This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by International Solutions Group Foreign Agricultural Service, United States Department of Agriculture Photo: Students in a classroom with materials Senegal McGovern-Dole Food For Education Project Final Evaluation August 2018 DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Final Evaluation McGovern Dole International Food for Education and Child Nutrition Program Saint Louis Region, Senegal With funding support from the United States Department of Agriculture (USDA), Counterpoint International (CPI) implemented the McGovern-Dole Food For Education project in the Saint Louis Region of Senegal from 2014 to 2018. The key objectives of the project were to improve the literacy of school-age children and increase the use of health and dietary practices. Agreement Number: FFE-685-2014-029-00 Project Duration: 2014-2018 Implemented by: Counterpart International Evaluation Authored by: International Solutions Group Evaluation Team Leader: Jason Wares Co-Evaluator: Christopher Root Technical Specialist: Noreen Mucha 1 Final Evaluation McGovern-Dole International Food for Education and Child Nutrition Program Saint Louis Region, Senegal PREPARED BY INTERNATIONAL SOLUTIONS GROUP PREPARED FOR Counterpart International 14 August 2018 2 TABLE OF CONTENTS TABLE OF ACRONYMS 6 EXECUTIVE SUMMARY 1 FINDINGS SUMMARY 2 STRATEGIC OBJECTIVES 2 ACTIVITY LEVEL RESULTS 3 RECOMMENDATIONS SUMMARY 4 METHODOLOGY 5 DOCUMENT REVIEW 5 QUALITATIVE RESEARCH 5 QUANTITATIVE RESEARCH 6 ANALYSIS 7 IMPROVEMENTS IN PRIMARY SCHOOL STUDENTS’ LITERACY 7 SCHOOL FEEDING SUSTAINABILITY 7 LIMITATIONS OF THE EVALUATION 8 FINDINGS 8 STRATEGIC OBJECTIVE RESULTS 8 SO1: IMPROVED LITERACY OF SCHOOL-AGED CHILDREN 8 IMPROVED LITERACY 9 REDUCED SHORT-TERM HUNGER 14 IMPROVED STUDENT ATTENDANCE 15 INCREASED GOVERNMENT SUPPORT 16 SO2: IMPROVE HEALTH AND DIETARY PRACTICES RESULTS 17 PERCENT OF CHILDREN RECEIVING A MINIMUM ACCEPTABLE DIET AT THE SCHOOL LEVEL 17 INCREASED KNOWLEDGE OF SAFE FOOD PREPARATION AND STORAGE PRACTICES 18 IMPROVED KNOWLEDGE OF HEALTH AND HYGIENE PRACTICES 19 ACTIVITY LEVEL RESULTS 20 SCHOOL SELECTION 20 ACTIVITY 1: CAPACITY BUILDING AT THE LOCAL, REGIONAL, AND NATIONAL LEVEL 22 ACTIVITY 2: CONSTRUCTION OF SCHOOL INFRASTRUCTURE 24 ACTIVITY 3: CONSTRUCTION OF LATRINES AND WATER STATION SYSTEMS 24 ACTIVITY 4: EQUIP SCHOOLS WITH ENERGY SAVING STOVES, CANTEEN EQUIPMENT, AND MATERIALS 26 ACTIVITY 5: GOOD HEALTH AND NUTRITION PRACTICES 27 ACTIVITY 6: TRAINING IN FOOD PREPARATION AND STORAGE PRACTICES 32 ACTIVITY 7: PROVIDE ACCESS TO HEALTH INTERVENTIONS 35 ACTIVITY 8: THE SCHOOL FEEDING PROGRAM 36 ACTIVITY 9: ESTABLISHING COMMUNITY FARMS 42 ACTIVITY 10: IMPROVED TEACHER ATTENDANCE 48 ACTIVITY 11: PARENT ASSOCIATION TRAINING 49 MONITORING AND EVALUATION OF PROGRAM ACTIVITIES 49 3 EVALUATION QUESTIONS 50 IMPLEMENTATION AND RELEVANCE 50 EFFECTIVENESS AND PERFORMANCE 52 EFFICIENCY 53 SUSTAINABILITY 54 LESSONS LEARNED 57 MGD PROVIDES AN IMPORTANT OPPORTUNITY FOR TESTING NEW IDEAS. 57 MGD’S COMPLETE DESIGN APPROACH WAS CRUCIAL FOR ITS SUCCESS. 57 GENDER ISSUES REQUIRE CLOSER EXAMINATION 57 THE MGD PROGRAM DOESN’T HAVE A CLEAR UNDERSTANDING OF THE SUFFICIENCY OF STUDENTS’ DIETS 57 STUDENT ATTENDANCE IMPROVES BECAUSE THERE IS FOOD AT SCHOOL 58 TEACHER ATTENDANCE MAY NOT BE AN ISSUE 58 SCHOOL NEEDS VARY 58 SUSTAINABILITY INITIATIVES SHOULD START EARLY IN THE PROGRAM 58 RECOMMENDATIONS 59 ANNEX 1: DOCUMENT REVIEW LIST 62 ANNEX 2: DATA COLLECTION TOOLS 63 ANNEX 3: SELECTED SAMPLES BY TYPE AND LOCATION 87 ANNEX 4: SURVEY RESPONDENT CHARACTERISTICS 89 ANNEX 5: SCHOOL SAMPLE 90 ANNEX 6: EVALUATION TEAM BIOS 92 ANNEX 7: INDICATOR FRAMEWORK – BASELINE VS MID-TERM DATA 94 ANNEX 8: EVALUATION TERMS OF REFERENCE 96 4 TABLE OF FIGURES Figure 1 Reading Score by Gender (Average) ..............................................................................................................11 Figure 2 Proportion of Students Reading At or Above Level By Grade ........................................................................12 Figure 3 CM1 Reading Level Distribution.....................................................................................................................13 Figure 4 CE2 Reading Level Distribution ......................................................................................................................13 Figure 5 CE1 Reading Level Distribution ......................................................................................................................14 Figure 6 Frequency with which students are hungry during school.............................................................................15 Figure 7 Minimum Acceptable Diet .............................................................................................................................18 Figure 8 Percentage of Beneficiaries Using Appropriate Hand Washing Practices .....................................................19 Figure 9 Student knowledge of ways to prevent intestinal worms..............................................................................19 Figure 10 Situations when Teachers Washed Their Hands (%) (Multiple Responses Possible)....................................30 Figure 11 The Amount of MGD Food that Is Currently and Potentially Covered by Granaries and School Farms.......45 Figure 12 Contributions to Primary School Canteen Sustainability..............................................................................46 5 LIST OF TABLES Table 1 Total Interviews Conducted 5 Table 2 Improvement in reading level by grade 10 Table 3 Change in Cohort Reading Level 10 Table 4 Expected Reading Level by Grade 11 Table 5 Maximum Possible Student Attendance 2018 16 Table 6 Minimum Acceptable Diet Food Groups 18 Table 7 School Categorization 21 Table 8 Training Dates, Themes, and Participants 22 Table 9 Classroom Construction Targets and Progress 24 Table 10 Progress toward School Latrine 24 Table 11 Schools with latrine and water 25 Table 12 Cleanliness of School Latrines (n = 159) 25 Table 13: Latrine's condition (mid-term n = 96, Final Evaluation n = 159) 26 Table 14 Health and Nutrition Trainings 28 Table 15 The degree to which teachers have received and use training (Mid-Term n =312, Final = 356) 29 Table 16 Degree to which teachers instruct on nutrition and hygiene (n=312, N=356) 30 Table 17 Student Situtions for washing hands (Mid-Term n = 1028, Final n = 787) 31 Table 18 Access to Hand Watching Facilities (mid-term n = 96, final n = 159) 32 Table 19 Condition of Food Storage Facilities % (Mid-Term n =111, Final N = 159) 33 Table 20 Conditions for storage in warehouse 34 Table 21 Director knowledge of food storage facility maintenance (mid-term n = 108, final n = 159) 34 Table 22 Directors knowledge of inventory control (mid-term n = 108, Final = 160) 34 Table 23 Directors knowledge of serving safe food 35 Table 24 McGovern-Dole Meal Schedule 37 Table 25 Food required to sustain the School Feeding Activity 38 Table 26 School Granary Inventory - Oct 2017 to Feb 2018 (KGs unless noted) 39 Table 27 Criteria for School Categorization 40 Table 28 Days providing meals without USDA assistance by school category 41 Table 29 Number of meals provided without USDA assistance by School Category 42 Table 30 School farm 2017 performance with one and two hectare projections 44 Table 31 Contribution to School Canteen Sustainability 47 6 Table of Acronyms Acronym Definition CE1 3rd Grade CE2 4th Grade CFA West African Frank CM1 5th Grade CPI Counterpart International DCMS-MEN School Medical Control Division ISG International Solutions Group IYCF infant and young child feeding JICA Japan International Cooperation Agency LOP Life of Project MAD Minimum Acceptable Diet MDD Minimum Dietary Diversity MGD McGovern-Dole MMF Minimum Meal Frequency MoA Ministry of Agriculture MoE Ministry of Education SMC School Management Committee SPI Social Progress Index TOR Terms of Reference USD United States Dollar USDA United States Department of Agriculture WHO World Health Organization 1 Executive Summary In Senegal, only 72.3 percent of primary school-aged children are enrolled in school. 1 Of the children that enroll, 58.9 percent continue to complete primary school. 2 The 2017 Social Progress Index (SPI) ranks Senegal 116th out of 128 countries in access to primary education3. In comparison, the SPI ranks Namibia 91st in the world and Ghana 19th. 4 More generally, Senegal is substantially below the level of countries at a similar level of economic development as defined by the SPI. 5 The low enrollment and completion in primary education translates to low literacy rates for adults (42.8% for adults, 52.8% for men, 33.6% for women).6 Approximately 40 percent of adult women in Senegal are literate, and 66 percent of Senegalese adult men are literate. While these statistics describe Senegal as a whole, the problem of insufficient school access may be especially acute in the Saint Louis region. One obstacle to school access in Saint Louis is the region’s dependence on agriculture. In Saint Louis, 60 percent of the population depends on agriculture for their income. 7 The agriculture sector in Saint Louis has experienced increased food insecurity due to irregular rainfalls and shortfalls in investment. 8 In Senegal, 85 percent of children that work are employed in the agriculture sector. When food insecurity increases primary school-aged children work more and attend school less. 9 The McGovern–Dole International Food for Education and Child Nutrition Program (MGD) works around the world to address the types of access to education and literacy challenges that Senegal faces. In September of 2014, USDA FAS awarded a grant to Counterpart International to implement a program that would “improve food security, reduce the incidence of hunger, and improve literacy and primary education” in Senegal’s Saint Louis region. After the completion of the baseline report, the project was launched in 2016. Counterpart International hired International Solutions Group (ISG) to conduct a final evaluation of the McGovern-Dole program as a follow up to the mid-term evaluation ISG conducted in May 2017. The purpose of the final evaluation was to “gain an understanding of what the project has achieved, whether the project has achieved its targets and objectives and what can be learned from it and provide recommendations for future programming to improve implementation and contribute to the MGD learning agenda.” The evaluation sought to answer questions related to the project’s implementation, effectiveness and performance, efficiency, and sustainability. The evaluation questions are listed on page 10 of the terms of reference, included as Annex 8. This document presents the results of the final evaluation. Results were gathered through a rigorous mixed￾methods data collection methodology. To conduct the evaluation, the evaluation team surveyed 787 students, 355 teachers, and 159 school directors. The team also recorded observations of facilities at 159 of the 270 schools that participated in the program. These observations were of school infrastructure, latrine access, 1 World Bank’s Databank “World Development Indicators” database. 2015 Data series 2 Ibid. 3 https://www.socialprogressindex.com/?tab=2&code=SEN 4 Ghana and Namibia chosen for points of comparison because the SPI includes them with Senegal in the “Lower Middle Social Progress” category, they are sub-Saharan African countries, and have similar scores in the “Foundations of Wellbeing” dimension of the index. 5 https://www.socialprogressindex.com/?tab=2&code=SEN 6 2013 data; World Development Indicators Databank; http://databank.worldbank.org/data/source/world￾development-indicators. 7 http://www.fao.org/emergencies/countries/detail/en/c/161500 8 FAO Emergencies – Senegal brief 9 World Bank’s Databank “World Development Indicators” database. 2015 Data series 2 hand-washing facilities, food storage facilities, and school canteens. The student, director, and school observation sample sizes correspond to a 95 percent level of confidence and a margin of error of 5 percent. The statistically significance of the teacher survey corresponds to a 95 percent level of confidence and a 6 percent margin of error. The team also conducted qualitative information gathering to add context and narrative around the quantitative data collection results; conducting key informant interviews and informal focus groups that with 93 stakeholders. These stakeholders were students, teachers, parents, school cooks, school farm presidents, school directors, project staff, and government officials at the local, regional, and national levels. Findings Summary The final evaluation presents findings at the strategic objective and activity indicator levels. It also answers specific questions included in the terms of reference (TOR) regarding program implementation and relevance, effectiveness and performance, efficiency, and sustainability. Strategic Objectives McGovern-Dole in Senegal is focused on achieving two strategic objectives (SO). These are: • SO1: Improved Literacy for School Aged Children • SO2: Increased Use of Health and Dietary Practices Under SO1, McGovern-Dole seeks to improve the literacy of children by achieving four results: 1. Short-term hunger and literacy gains: The evaluation found that students who said that they were not hungry during the school day were 61 percent more likely to read at or above grade level than students who said they were hungry more often than from time-to-time. 87 percent of students said that they were not hungry at school except “from time to time.” 2. Improved literacy of school-aged children: Final evaluation ASER test results demonstrate that the McGovern-Dole program has improved student literacy. The ASER test scores a student’s reading ability on a scale of twelve reading levels. Students who score at level 0 or 1 cannot do more than identify letters, while students who score at level 11 have advanced reading ability. Students in the CE1 grade are expected to read at level 5, CE2 students are expected to read at level 7, and students in CM1 should read at level 9. On an absolute basis, the average reading level for CE1 students improved from 3.31 to 4.88 between baseline and final evaluation, an improvement of 1.57 levels. Students in CE2 improved on average from 4.89 to 6.39, an improvement of 1.5 levels. Finally, CM1 students improved on average from 6.19 to 7.69, an improvement of 1.5 levels. In the full report, the evaluation team adjusts the baseline score because the final evaluation was delivered later in the school year than the baseline report. Even with the penalty on baseline scores that the adjustment applies, a clear improvement in reading is evident. Importantly, the proportion of students that read at their grade level or higher increased dramatically between baseline and final evaluation. The percentage of CE1 students that read at their grade level or higher increased from 33 percent to 53 percent between baseline and final evaluation. CE2 students reading at their grade level or higher increased from 27 percent to 48 percent, and for CM1 students the increase was from 28 percent to 43 percent. 3. Improved student enrollment and attendance: Student attendance has increased since the baseline study. Teachers and directors that the evaluation team interviewed said that enrollment had increased and that students stayed at school longer during the school day as a result of the school feeding program. Student attendance records support an increase in school attendance. However, the evaluation team suspects that official school records overstate attendance, which show a 99.5 3 percent attendance rate. This issue is further discussed in the findings and recommendations sections. 4. Increased government support: Increased government support, defined as the percent of teachers who had received pedagogical training in the three months prior to the survey, had declined between baseline and mid-term. The indicator improved significantly between mid-term and final evaluation. At final evaluation, 32 percent of teachers reported that they had received pedagogical training in the three months previous to the survey compared to 13 percent at mid-term. The evaluation team suspects that the 32 percent average understates actual training as teachers may have not counted monthly peer to peer trainings that many teachers attend. Under SO2, McGovern-Dole seeks to improve health and dietary practices through achieving three results: 1. Increased percentage of children receiving a minimum acceptable diet: The final evaluation found that 58 percent of students received a minimum acceptable diet over school. This is an increase over the 43 percent at midterm and 2 percent at baseline who receive a minimum acceptable diet at school. 2. Increased knowledge of safe preparation and food storage practices: The strategic objective￾level measure for this indicator is the percentage of beneficiaries who use appropriate hand washing practices before meals, before food prep, after using the latrine, and after diaper changing. The final evaluation team calculated that 71 percent of boys and 75 percent of girls use appropriate hand washing practices, a clear improvement over the baseline finding of 50 percent for boys and 55 percent for girls 3. Improved knowledge among students of health and hygiene practices: The final evaluation estimates that 53 percent of boys and 51 percent of girls can identify at least two ways to prevent intestinal worms (the central measure for progress under this objective). This result is a significant improvement over both the 8 percent of boys and 10 percent of girls recorded at baseline, and the 45 percent of boys and 41 percent of girls could do the same at mid-term. Activity Level Results The McGovern-Dole program aims to produce outputs in 11 activity categories and track achievement through 52 indicators. The mid-term evaluation found that the program was on track to achieve nearly all of the required results under these indicators. Some highlights of the program’s achievement include: • 15,151,085 meals served in school over the course of the program against a life of project (LOP) target of 13,750,000. 10 • 573 Parent Association Members trained against a LOP project target of 540. • 808 Cooks trained against a LOP target of 540. • 90,576 doses of de-worming medication distributed compared to an LOP target of 99,998. • 20 community farms established against an LOP target of 20. • 270 durable, culturally appropriate energy efficient stoves produced and distributed against an LOP target of 270. 10 Targets reflect figures included in USDA’s amendment III to the McGovern-Dole agreement signed by Counterpart on June 6, 2017 and USDA on June 1, 2017. 4 Overall, the evaluation team found that Counterpart’s strategy for program implementation, which creates locally sustainable assets around the complete school feeding and literacy program, is an important component of the program’s success. The full report describes the quality of these outputs, reasons for successes, and areas where improvements could be made. Recommendations Summary The report concludes with a discussion of recommendations for future McGovern-Dole programs. We include a summary of the recommended actions here. 1. Continue the program: USDA should continue the program in Saint Louis for another 2-3 years to provide time for schools to implement sustainability programs. Ending the program now would be pulling food from thousands of kids who have just started on an educational path equipped with the energy that USDA commodities provide. 2. Overcome the diminishing marginal utility of the program: The program showed significantly larger gains between baseline and mid-term than between mid-term and final evaluation. To overcome the diminishing utility of the program, USDA could initiate second phase initiatives that may have a large impact on primary school literacy and improved health practices. 3. Focus school selection on high and medium food insecurity schools: McGovern-Dole’s school selection process and mid-term reorientation toward need-based food provision worked well. In future McGovern-Dole programs, McGovern-Dole should use the needs identification process that Counterpart developed to categorize schools during the school selection process. The process should incorporate means of identifying school needs that include level of food security as well as other criteria. 4. Work with school directors to plan for maximizing each school’s resources: McGovern-Dole should work with directors to develop school feeding sustainability strategies based on different levels of food availability. 5. Create a School Feeding Board: School feeding programs need to be run more professionally if they are to become more sustainable. McGovern-Dole Senegal should consider forming a school feeding program board of directors to guide decision making, resource allocation, and school feeding coordination. 6. Identify ways to overcome school farms’ major constraints: School farms potential is limited by three major constraints; insufficient access to finance, labor shortages, and inappropriate pump provision. School farms would also benefit through forming model farms in partnership with the MoA. 7. Use random spot checks as well as school data to track increased attendance: To truly understand the way the program influences school attendance, McGovern-Dole should have an independent research plan for studying the topic from the beginning of the program. 8. Research whether infrastructure construction activities provide sufficient value: Interviews and surveys revealed that the school feeding and health initiatives were primarily responsible for improvements in reading. Infrastructure projects were less successful and created unfulfillable expectations among some project beneficiaries. School infrastructure projects were also the single biggest direct cost line item in the program's budget. The funds allocated to school infrastructure projects could have funded school feeding programs for over 100 new schools and over 17,000 additional students. Schools certainly require repair and upgrading. USDA could use the data it has on hand from previous McGovern-Dole programs to see if there is a correlation between schools that benefit from infrastructure projects and improved reading scores. It could also allocate funds for school maintenance and upgrading rather than new construction, which would cost less, and use the savings to serve more schools. 5 Methodology The evaluation team carried out a final evaluation in April and May of 2018 with fieldwork and data collection in Senegal taking place between April 20 and May 5. The team used mixed methods including document review, qualitative research including key informant interviews (KIIs), focus group discussions (FGDs), and quantitative data collection through surveys. The mixed methods approach allows the evaluation team to strengthen qualitative findings through statistically representative survey data, and to provide depth and context to survey findings. Document review Document review informed all stages of the evaluation. The team used the document review initially to learn about the project and inform data collection tools. Later the review was useful in explaining different features of program design and tracking the evolution of the program over time. McGovern-Dole’s monitoring and evaluation team provided data and documents that served as source data for the evaluation team used to compare performance results against LOP targets, assess the frequency of trainings, and changes in student and teacher attendance. Finally, project records were used to analyze school farm and granary performance and potential. Annex I includes a full list of documents the team reviewed. Qualitative research The evaluation team conducted interviews with a total of 93 respondents to provide context and depth to survey results. The team asked questions intended to expand its understanding of the McGovern-Dole project, answer evaluation questions, and confirm information gleaned from other sources. Question guides were used to lead semi-structured interviews. The guides ensured that the evaluation team asked the same key questions across similar respondent types while still allowing the freedom for open-ended, conversational￾style interviews. These guides are included in Annex 2. Qualitative field research was conducted by the evaluation team in the three departments (Dagana, Podor and Saint Louis) that make up the project area. In total, the evaluators visited 18 schools. The schools chosen for the evaluators to visit were all part of the random sample of schools chosen for the survey. One evaluator focused primarily on schools with school farms to allow for evaluation of this pilot program (Activity 9 in the findings section). Nine schools with school farms were visited. Other interviews happened at schools where enumerators were conducting surveys. Within schools, the evaluators conducted interviews with the directors, teachers, school farm presidents, cooks and parents, based on availability. The evaluation team conducted informal focus groups with students at primary schools. In addition to interviews at schools, the evaluation team conducted KIIs with education administrators and other government officials at the national, regional and departmental levels. Finally, the team interviewed McGovern-Dole project staff. A summary of KIIs and FGDs conducted is presented in table 1 below. Table 1 Total Interviews Conducted CATEGORY NUMBER Directors 17 Teachers 15 Students 24 Parents 8 Cooks 9 Government 9 Project Staff 7 Total 93 6 Quantitative Research The final evaluation utilized student, teacher, and school director surveys. The questionnaires largely followed those used for the mid-term evaluation. Questionnaires are available in Annex 2. In addition, enumerators completed a school observation checklist at each school in the sample. Sampling for the surveys followed a two-stage cluster design, with the schools comprising the first cluster schools and the subject of the survey (students, teachers, or directors) comprising the second. In order to achieve necessary statistical strength, the first cluster required 159 randomly selected schools of the 270 project affiliated schools. Student Sample For the student sample, we randomly selected 8 students from each of the 99 primary schools included in the first cluster. This created a total student sample of 787 students, corresponding to a confidence level of 95% and a margin of error of 5%. In conducting the survey, this sample size target was reached. Teacher Sample To reach the same confidence level and margin of error, the required teacher sample was required 404 teachers across 135 of the 159 schools chosen in the first cluster. As a teacher strike was underway and there was a lack of certainty around which teachers would be present at any school, enumerators were instructed to interview 3 available teachers per school. As it turned out, fewer than three teachers on average were actually available to serve as survey respondents in each school, and 355 teacher surveys were completed. Accordingly, results are statistically significant at 95 percent level of confidence and 6 percent margin of error. Director and Checklist samples Director interviews and checklists were completed at all 159 schools in the sample to achieve a confidence level of 95% and a margin of error of 5%. Note on Comparability Please note that at mid-term, information collected on school facilities through the mid-term evaluation team’s checklist instrument, teacher surveys, and school director surveys had wider margins of error and lower levels of confidence than the final evaluation instruments. The section indicates the margin of error and level of confidence where necessary for appropriate comparison. EXHIBIT 1 MAP OF 159 SURVEYED SCHOOLS 7 Analysis Improvements in Primary School Students’ Literacy The evaluation team measured the reading skills of primary school students. This measurement was based on the same ASER reading test conducted in support of the baseline report.11 The evaluation team compared reading scores at baseline, midterm, and final evaluation by grade to determine if the program caused a change in student’s reading skills. The reading scores for CE1 (3rd Grade) students from before the McGovern-Dole program started are compared with CE1 students at mid-term evaluation and at the time of final evaluation, as were scores for CE2 (4th Grade) and CM1 (5th Grade) students. These differences describe the change in primary school students’ reading skills for each of the three grades. Note that the literacy analysis includes a correction to account for the difference in timing of the baseline and midterm surveys. The baseline survey was conducted in January and February of 2016 whereas the midterm survey in May and June of 2017 and the Final Evaluation in May of 2018. The difference in timing means that students at midterm and final evaluation had nearly half a school year longer to improve their reading in comparison with their baseline cohorts. Naturally, the extra half year of school should cause the reading results from the midterm to be better than those at baseline, independent of the project’s impact. The evaluation team corrects for the different timing of the reading tests by adding half of the difference in reading scores between grades that are observed to the baseline. For example, the difference between the average reading score for CE1 and CE2 at baseline is 1.58. Thus, 0.79 is added to the baseline score for CE1 to estimate what that average score might be if it were collected at the end of the school year in May or June instead of January or February. Likewise, the difference between CE2 and CM1 is 1.3, so 0.65 is added to grades for CE2. Because no data is available for CM2, for the CM1 baseline, the adjustment is made by adding 0.65 – (0.79-0.65) = 0.51 to baseline data for CM1. This attempts to project the decreasing trend in intra-grade reading improvements shown in the difference between reading improvements between CE1 and CE2 and then between CE2 and CM1. The report also looks at gender differences in literacy outcomes by comparing ASER scores for boys and girls in each grade. This comparison is made at baseline and at final evaluation to assess if gender differences in reading may have changed with the FFE program’s implementation. School feeding sustainability The evaluation team identified the quantity of food and resources required to conduct a school feeding as defined by McGovern-Dole’s operational plan. The team valued products using market prices to estimate the resources a community would need to sustain the program after McGovern-Dole concludes. Then the team identified the food and cash contribution to these requirements made by the school granaries, school farms and by the government. These results are used to identify how much of FFE’s meal provision can be expected to be replaced after the program ends. The evaluation team also made estimates based on a more optimistic scenario by extrapolating granary contribution by a factor of two and increasing school farm size to two hectares. 11 http://www.asercentre.org/ 8 Limitations of the Evaluation Statistical strength of the teacher survey: The teacher survey had 355 respondents, short of the target of 404 required to be statistically representative at the 95 and 5 percent levels. The shortcoming may be explained by the occurrence of teacher strikes in some areas during the survey. School granary data incomplete: Complete school granary data was not available, which may affect the accuracy of school feeding program sustainability projections. Increased government support - This indicator is difficult to measure because teachers appear to receive pedagogical training through many channels, and it is unclear which qualify as “government support.” Mid-term Survey Statistical Strength: The mid-term surveys for teachers, school directors, and school checklists were not statistically representative, which affects the validity of comparisons between those mid￾term and final evaluation survey results. Minimum acceptable diet measurements – Unlike the baseline, neither the midterm nor end of project evaluations included a survey of mothers. As a result, minimum acceptable diet measurements are only for food consumed at school and not at home. Findings In this section, we describe a) the progress that the McGovern-Dole program has made toward strategic objectives; b) the program’s activity level results; c) the process of school selection; and d) McGovern-Dole’s monitoring and evaluation system. Strategic Objective Results McGovern-Dole in Senegal is focused on achieving two strategic objectives. These are: • SO1: Improved Literacy for School Aged Children • SO2: Increased Use of Health and Dietary Practices SO1: Improved Literacy of School-Aged Children Under SO1, McGovern-Dole seeks to improve the literacy of children through achieving three results: 1. Improved student attentiveness through reduced short-term hunger 2. Improved student attendance 3. Increased government support 9 Improved Literacy The final evaluation supported the mid-term evaluation’s finding that the McGovern-Dole program has improved student literacy. Following the methodology used in the baseline study, the evaluation team utilized the ASER literary assessment to compare students’ reading ability to baseline and mid￾term evaluation results. The assessment found that on average, students had improved in relative reading ability and proportion of students reading at grade level. The evaluation team attributes reading improvement to the program because the team could not come up with another reason, other than the program’s effectiveness, that school children across Saint Louis would significantly improve at reading between the baseline study and final evaluation, holding grade level constant.12 Therefore, we believe it is fair to attribute the literacy improvement to the project . Table 2 shows the average reading level that students achieved at baseline (using the adjusted baseline score), the reading level measured at midterm, and the reading level measured at final evaluation for each grade. The table also shows the difference between reading levels at baseline and final evaluation, which describes students’ overall improvement. Reading scores improved for Grades 3, 4, and 5 students by 1.58, 1.5.and 1.5 levels respectively. The program appears to have experienced diminishing marginal utility between mid-term and final evaluation as students’ reading levels did not improve during that period as they did during the first part of the program. The final evaluation showed a slightly lower score for grade 3 (CE1) students and slightly higher scores for grades 4 (CE2) and 5 (CM1) students. However, the final evaluation scores were not statistically different from the mid-term evaluation scores. 13 12 As a point of comparison, the St. Louis regional department of education measured a 4% increase in the proportion of CE1 students who read at grade level between 2016 and 2017. The MGD program saw over 82% increase between 2015 and 2016. (Rapport Annuel De Performance (2017); Inspection d’Academie de Saint-Louis; March 2018. Pg 8. 13 Two-sample t tests showed p values of 0.46 for CE 1, 0.53 for CE 2, and 0.15 for CM 1. EXHIBIT 2 BLACKBOARD SHOWING LESSONS IN FRENCH AND ARABIC 10 Table 2 Improvement in reading level by grade GRADE ADJ. BASELINE** MIDTERM FINAL EVALUATION BASELINE/FINAL EVALUATION DIFFERENCE GRADE 3 (CE1) 3.31 5.04 4.88 1.57 N 356 366 326 GRADE 4 (CE2) 4.89 6.25 6.39 1.5 N 341 379 278 GRADE 5 (CM1) 6.19 7.34 7.69 1.5 N 339 321 194 * statistically significant at p<0.05 The evaluation team also looked at cohort groups that participated in the program. The Baseline CE 1 group are students that were in CE 1 at the beginning of the program and now are in CM 1. The Baseline CE 2 group was in CE 2 at baseline and CM 1 at midterm. The Baseline CM 1 group was in CM 1 at the time of the baseline study. Table 3 shows how cohort reading scores changed as they progressed through the McGovern-Dole program. Students that were in CE 1 at the beginning of the program had a mean reading score of 7.69 by the end of CM1. This achievement represents an entire level improvement over students who were in CM 1 at the beginning of the program. Students who were in CE 2 at the beginning of the project had a mean reading score of 7.34 at the end of the program or 0.6 levels above students in CM 1 at baseline. Table 3 Change in Cohort Reading Level GRADE ADJUSTED BASELINE MIDTERM FINAL EVALUATION BASELINE CE1 4.1 6.25 7.69 BASELINE CE2 5.54 7.34 BASELINE CM1 6.7 Reading level results differ for boys and girls by grade. At baseline and mid-term, boys scored higher than girls on the reading assessment. This trend held at Final evaluation for CE1 students. However, for CE 2 and CM1 students, girls scored higher than boys. Most significantly, CM 1 girls scored 0.44 higher than boys. 11 Figure 1 Reading Score by Gender (Average) Table 4 Expected Reading Level by Grade Grade Appropriate Reading Level Grade 3 (CE1) 5 or higher Grade 4 (CE2) 7 or higher Grade 5 (CM1) 9 or higher At baseline, only 33.13 percent of CE 1 students read at an appropriate level for their grade. At final evaluation, that number had increased to 53.67 percent. CE 2 students showed a marked improvement, increasing from 27.27 percent reading at or above grade level to 48.47 percent. CM 1 students also improved, from 28.02 percent reading at or above grade level to 43.3 percent by final evaluation. The graphics below demonstrate reading level improvement. The black vertical line represents the appropriate reading level at each grade. The graphs illustrate how the proportion of students reading at or above level (to the right of the line) has increased. CE1 students show a decrease between mid-term and final evaluation in the proportion of students reading at or above reading level. CM1 students show an improvement. For CM1 students, the improvement makes sense, as those students had participated in the program longest of all cohort groups. 0 1 2 3 4 5 6 7 8 9 Baseline Midterm Endline Baseline Midterm Endline Baseline Midterm Endline grade 3 (CE1) grade 4 (CE2) grade 5 (CM1) Male Female Reading Level 12 Figure 2 Proportion of Students Reading At or Above Level By Grade In fact, when looking at the distribution of CM1 reading scores, not only did the proportion of students reading at or above level improve, but that proportion improved most at the highest reading levels. Figure 3 below shows the percentage of CM1 students that scored at each ASER test reading level. The vertical line is at reading level 9, which is the appropriate reading level for CM1 students. The graph shows that a higher proportion of CM1 students reading at or above grade level after the project than at baseline. It also shows two important findings: an increase in the number of students reading near grade level, and a large increase in students who are reading at the highest levels. The percentage of students reading at level 10 increased from 13.1 percent at baseline to 17.5 percent at final evaluation, and students reading at level 11 increased from 8.21 percent to 16.5 percent. The data also shows a general shift to higher reading levels. At baseline, about 20 percent of CM1 students read at levels 0 – 2, while at final evaluation that proportion was approximately 11 percent. 0 10 20 30 40 50 60 70 Baseline Mid-term Endline Percentage of Students CE 1 CE2 CM1 13 Figure 3 CM1 Reading Level Distribution CE 2 showed a similar trend, as demonstrated in figure 4. The proportion reading at or above level 7 increased between baseline and final evaluation. At baseline, 40.48 percent of CE2 students were reading at level 4 or below. By final evaluation, that number had decreased to 27.44 percent. Figure 4 CE2 Reading Level Distribution Figure 5 shows that by the end of the project there nearly 13 percent fewer students reading at levels 0 and 1 compared to baseline. Nearly 19 percent more students were reading at or above level 7 compared to at baseline. 0 5 10 15 20 25 30 0 1 2 3 4 5 6 7 8 9 10 11 Baseline Endline Percentage of Students 0 5 10 15 20 25 30 35 0 1 2 3 4 5 6 7 8 9 10 11 Baseline Endline Percentage of Students 14 Figure 5 CE1 Reading Level Distribution Reduced Short-Term Hunger The McGovern-Dole program is reducing hunger for its beneficiary students. Senegal is a food insecure country with 53 percent of Senegalese families report that there have been times in the past 12 months that they did not have enough money for food. 14 The program’s primary measure of short-term hunger is the percentage of students that report feeling hungry at school. 15 A section of the students’ survey asked students whether or not they became hungry during the school day. If a student responded that they became hungry during the school day, they were asked if they became hungry always, often, about half the time, or just from time to time. The baseline study found that 66 percent of students reported that they do not feel hungry at school. The final evaluation found this proportion of students had not changed significantly, with 65 percent indicating that they are not hungry during the school day. Though the question was not offered at baseline, the mid-term evaluation allowed students to respond to a survey question about short-term hunger that they “are not hungry at school, except for time to time.” The mid-term evaluation survey found that 89 percent report that they are only hungry “from time to time (figure 6). At final evaluation, this figure had not changed significantly, with 87 percent choosing that answer. Boys and girls chose that answer in approximately equal proportions. At final evaluation, 21 percent of students said that they were hungry about half the time, an increase from 4.3 percent at mid-term. However, the proportion of students who said that they were hungry “always” or “often” dropped from 33.54 percent at mid-term to 14.48 percent at final evaluation. 14 2016 Gallup World Poll 15 See MGD Indictor 1.2.1 0 5 10 15 20 25 30 35 40 0 1 2 3 4 5 6 7 8 9 10 11 Baseline Endline Percentage of Students 15 Figure 6 Frequency with which students are hungry during school Eight-five percent of students said that they usually ate breakfast at school compared to 78 percent at mid￾term. Forty-five percent said they eat at least one other time at school, compared to 56 percent at baseline. At mid-term, 65 percent of students mentioned eating something before school without prompting from the mid-term evaluation interviewer. At final evaluation, 97 percent said that they eat before school. While the increase in numbers eating before school is encouraging, the evaluation team believes that the dramatic change may have to do with the timing of the survey. At mid-term, the survey was conducted during Ramadan, which was not the case at final evaluation. Relationship between reduced short-term hunger and literacy improvement At the core of McGovern-Dole design is the assumption that less hungry children will perform better in school. We test this assumption using logistic regression. Controlling for location (commune), school student￾teacher ratio and gender, we find that students who report not being hungry during the school day16 are 61% more likely to read at or above their grade level than those who are hungry. This result is statistically significant at the P <.05 level. Improved Student Attendance Student attendance has increased since the baseline study. As during the mid-term evaluation, teachers and school directors that the evaluation team interviewed indicated that more students were attending school and that students were staying at school longer during the day. Teachers said that students stay longer because they no longer have to return home to eat or use the latrine. A school director in Podor said that he had to hire three new teachers, bringing his teacher total to five, because of increased enrollment and attendance. Teachers and school directors say that students rarely miss school since the school feeding activity began, and they stay at school longer. Directors report growing class sizes and schools operating at capacity. 16 Or only hungry from time to time. 0 10 20 30 40 50 60 70 Always Often Half the time From time to time Mid-term Endline 16 Counterpart maintains records on student attendance. The data that Counterpart collects comes from the attendance records of schools participating in the program. These records support the statements that school directors and teachers made regarding the increase in student attendance. That said, the evaluation team believes that data does not accurately reflect that true rate at which students attend school. The team has two primary reasons for this judgment. First, the unit of measurement that the school system uses to track enrollment seems difficult to track. For example, in the 2017/2018 school year, the maximum possible attendance by month was as follows: Table 5 Maximum Possible Student Attendance 2018 MONTH MAXIMUM POSSIBLE STUDENT ATTENDANCE October 99 November 118 December 87 January 128 February 116 March 99 The unit of measurement is the hour, with Mondays, Wednesdays, and Fridays allocated five hours, and Tuesdays and Thursdays allocated seven hours. It seems unlikely that teachers and directors have the capacity to track hourly attendance for students, especially given the large class sizes and lack of administrative personnel at some schools. Secondly, the data indicates that students attended school 99.5 percent of the time on average during the 2017/2018 school year. As a point of comparison, a UNICEF survey conducted on school attendance rates worldwide calculated Senegal’s rate at 54 percent overall and 43 percent in rural areas. 17 Another point of comparison is the US, where school attendance averages about 92 percent. 18 The World Food Program estimates that primary school completion rate is 66.7 percent in Senegal. 19 The evaluation team believes that McGovern-Dole collected data on school attendance exaggerates student attendance rates. That said, survey results combined with interviews and McGovern-Dole’s food consumption records suggest that student attendance rates have increased and are high. Increased Government Support Increased government support, defined as the percent of teachers who had received pedagogical training in the three months prior to the survey, had declined between baseline and mid-term. The indicator improved significantly between mid-term and final evaluation. At final evaluation, 33 percent of teachers reported that they had received pedagogical training in the three months previous to the survey compared to 13 percent at 17 https://data.unicef.org/topic/education/primary-education/ 18 https://nces.ed.gov/programs/digest/d11/tables/dt11_043.asp 19 https://docs.wfp.org/api/documents/WFP-0000023416/download/ 17 mid-term. The evaluation team suspects that the 32 percent average understates actual training as teachers may have not included monthly peer to peer trainings that many attend. This indicator is difficult to measure because teachers appear to receive pedagogical training through many channels, and it is unclear which qualify as “government support.” Additionally, the three-month requirement may not be most appropriate. As discussed in the mid-term report’s recommendations, McGovern-Dole should work with local government to determine how often trainings are required and to make sure delivering quality training at regular intervals is established among local education inspectors. SO2: Improve Health and Dietary Practices Results Under SO2, McGovern-Dole seeks to improve health and dietary practices through achieving three results: 1. Increased percentage of children receiving a minimum acceptable diet 2. Increased knowledge of safe preparation and food storage practices 3. Improved knowledge among students of health and hygiene practices Percent of children receiving a minimum acceptable diet at the school level Dietary diversity is a proxy for adequate nutrient-density of foods, including micro-nutrients. Commonly, the minimum acceptable diet (MAD) measurement is one of eight core indicators for assessing infant and young child feeding (IYCF) practices as developed by the World Health Organization. This indicator ordinarily reflects both the minimum dietary diversity (MDD) and minimum meal frequency (MMF) of a population. In the context of FFE Senegal, MAD is also important because undernourished kids have difficulty learning20 . MGD adopted the MAD indicator to analyze if school-age program participants are eating sufficiently to ensure appropriate growth and development. Without adequate diversity and meal frequency, school-age children are vulnerable to undernutrition, especially micronutrient deficiencies which increase chances of increased morbidity and mortality. The indicators itself measures which food groups (table 6) are being consumed with a 24-hour dietary recall. The data collection instruments asked children about their breakfast and lunch intake since this is what they ate as part of the school program. Minimum dietary diversity means child is eating from at least four food groups. The cut-off of four food groups is associated with better-quality diets. 21 20 In it’s more common usage, MAD is used for infants 6-23 months of age and the mother is usually asked in a 24 hour dietary recall. 21 In most studies of MAD, the data differentiates between Vitamin A rich fruits and vegetables and regular fruits and vegetables. This differentiation was not made in FFE Senegal data collection 18 Table 6 Minimum Acceptable Diet Food Groups 1. Grains or roots 2. Legumes and Nuts 3. Dairy Products (milk, yogurt, cheese) 4. Flesh foods (Meat, Fish, and Poultry) 5. Eggs 6. Vegetables 7. Fruits Other (a lot of bread and coffee) (does not count as a food group) Source: “Indicators for Assessing Infant and Young Child Feeding Practices: Part I Definitions.” Washington, D.C. USA: World Health Organization (WHO), 2008. http://apps.who.int/iris/bitstream/10665/43895/1/9789241596664_eng.pdf. The McGovern-Dole final evaluation indicates that 58 percent of the targeted Senegalese school-age children (both sexes) usually eat at least 4 of the seven food groups at school (figure 7) for the minimum acceptable diet. it is a significant increase over the mid-term evaluation’s result. Figure 7 shows the baseline results for children’s diet at home at baseline versus at school for the midterm and at the end of the project. By comparison, the baseline result for children’s minimum acceptable diet at school was two percent.22 We calculate that 58 percent of students receive a minimum acceptable diet in school at the final evaluation, a substantial improvement over the mid-term evaluation’s 43 percent estimate. The final evaluation percentage is slightly higher for females (58.2%) compared to males (55.3%). Figure 7 Minimum Acceptable Diet Increased Knowledge of Safe Food Preparation and Storage Practices The strategic objective-level measure for this indicator is the percentage of beneficiaries who use appropriate hand washing practices before meals, before food prep, after using the latrine, and after diaper changing. The final evaluation calculated that 74 percent of beneficiaries use appropriate hand washing practices, a clear improvement over the baseline finding of 50 percent for males and 55 percent for females. The final 22 Note that the midterms and endline evaluations did not include a mothers’ survey necessary to assess children’s diet at home 0 10 20 30 40 50 60 70 80 Baseline result Midterm result Endline result 19 evaluation average dropped somewhat from the mid-term overall average (77%). However, this result is within the mid-term’s margin of error. Figure 8 Percentage of Beneficiaries Using Appropriate Hand Washing Practices Improved Knowledge of Health and Hygiene Practices The measure of this indicator is the percentage of students that can name at least two ways to prevent intestinal worms (MGD Indicator Framework, Annex 7). The final evaluation showed a significant improvement in this indicator over the baseline and mid-term. At baseline, 8 percent of boys and 10 percent of girls could identify at least two ways to prevent intestinal worms. At mid-term, 45 percent of boys and 41 percent of girls could do the same. By final evaluation, students ability to identify two ways to prevent intestinal worms had improved to over 51 percent, with boys and girls scoring about equally (figure 9). Figure 9 Student knowledge of ways to prevent intestinal worms 0 10 20 30 40 50 60 70 80 90 Baseline Mid-Term Final 0 10 20 30 40 50 60 All Boys Girls Baseline Mid-Term Final 20 Activity Level Results The following section reports on final evaluation results for each McGovern-Dole program activity. This section compares final evaluation results to mid-term results for information that was collected during mid￾term and final evaluation but not at baseline, for example in reporting on the cleanliness of latrines that were not yet constructed at baseline. Please note that at mid-term, information collected on school facilities through the mid-term evaluation team’s checklist instrument, teacher surveys, and school director surveys had wider margins of error and lower levels of confidence than the final evaluation instruments. The section indicates the margin of error and level of confidence where necessary for appropriate comparison. School Selection The McGovern-Dole program’s school selection process was transparently and efficiently managed. It worked to secure the participation of the school communities. While participation was slow to develop for many schools at the beginning of the program, McGovern-Dole staff took steps to deepen school community participation between mid-term and final evaluation. McGovern-Dole’s selection process focused on selecting schools that were the most likely to have success in the program as well as those with great need. Many very needy schools were included in the program. However, practical considerations led to a bias toward including schools that were more feasible to serve over schools that had the most needs. The McGovern-Dole program selected eligible schools based on six criteria: 1. A minimum of 30 students. 2. A sufficient number of teachers given the number of students, though no specific ratio was given. 3. Low enrollment statistics, which were provided by the inspector. 4. Accessibility, meaning within a reasonable distance to major towns and accessible by trucks. Schools that were only reachable by crossing a river were ineligible. 5. A kitchen and food storage building. 6. Schools with a community that has demonstrated a commitment and willingness to contribute to previous projects, such as the Development Activity Proposal project23; including contributions in cash or in-kind support. McGovern-Dole asked the school inspector’s office for a list of schools in Saint Louis that met these criteria. Then, Counterpart and School Inspector’s Office staff visited the schools to make sure the project could serve them. After visiting the schools, the Inspector’s Office and Counterpart realized that a number of the schools were not a good match for the program, so Counterpart asked the Inspector’s office to identify replacement schools. At the end of school selection, the inspector’s office initially identified 180 primary schools and 44 preschools that met the program’s criteria. The program chose to serve all 44 preschools and 163 of the primary schools from this first round of school selection. Currently, the program supports 204 primary schools and 66 pre-schools.24 As reported in the mid-term evaluation, the program changed two of its initial school selection criteria. The first was the requirement that schools have a canteen, as these were relatively rare in primary schools and prohibitively expensive for many schools to build. The second was the community commitment criterion. 23 The Development Activity Proposal project was a school feeding project that came before MGD-Food for Education. 24 Information on school selection derived from Counterpart staff interviews and school election documentation. 21 Counterpart realized many schools needed more help rallying commitment than the program originally anticipated. In the mid-term evaluation, the evaluation team recommended that the McGovern-Dole program consider allocating resources to schools according to need, rather than following a one size fits all approach. In response to this recommendation, Counterpart segmented the schools selected for the project by each school’s level of food insecurity. All of the schools participating in the project were grouped into one of three categories as shown below. Table 7 School Categorization Category Percent Low food insecurity 15% Medium food insecurity 45% High food insecurity 40% This categorization of schools led to realization about the difference is the needs of high food insecurity schools versus the other categories, and the difficulties in maximizing benefit to the neediest schools. The most food insecure schools also had the most basic infrastructure needs. At these schools, temporary structures may serve as classrooms. They have no auxiliary structures, such as a barrier fence/wall, library or school director’s office. When asked “What else could the McGovern-Dole program provide your school to improve results?” school directors and teachers at low food insecurity schools replied that they needed “computers.” Low food insecurity schools also appeared to have greater community contributions. At the Mbilor primary school in Dagana, for example, the community had built classrooms with community contributions independent of the McGovern-Dole program. Two fathers that the evaluation team spoke with said that they had attended the school in 1978 and alumni had contributed to the school’s growth. One parent at a low food insecurity school said that she liked the program because “I used to have to give money (for food), and now I save money.” This comment expresses a much different type of need than is seen in the high food insecurity schools where parents and teachers commented that they liked the program because students were no longer hungry. The Regional School Inspector Office in Saint Louis mentioned that another way to categorize schools was to distinguish those in the diéri and those in the wallo, a factor that was considered in Counterparts classification system. The diéri is an area in Saint Louis that is outside of the rivers flood zone and is thus drier and more dependent on rainfall. Schools in the diéri are unlikely to qualify for a school farm and are generally in poorer communities. Many are also harder to reach. While school and student needs differed across McGovern-Dole’s school categories, students in all of the categories benefited. Students at schools in each category improved reading scores, nutrition, and hygiene. In interviews, school directors across the three categories said that attendance was up, hygiene had improved, and the students seemed healthier. Clearly, the McGovern-Dole program did not focus only “on selecting the schools that had the most needs,” although certainly a great many of the program’s schools fell into that category. The program focused on schools that had serious needs, that showed potential for achieving McGovern-Dole objectives and potential EXHIBIT 3 CONTRASTING SCHOOL BUILDINGS 22 to graduate and that were accessible for commodity delivery. The program also chose a portfolio of low and high food security schools that mitigated the risk of serving only the neediest schools. Activity 1: Capacity building at the local, regional, and national level Counterpart International delivered trainings to 1,956 participants since 2016 (table 8). This number includes 1,211 teachers, 447 school directors, 262 school management committee (SMC) and community members, and 37 school inspectors and government officials. Table 8 Training Dates, Themes, and Participants25 Date Title Purpose Teachers Directors Gov. Community members/ Parents 28 April – 2 May 2016 Curriculum for Basic Education and innovative reading instruction. Training of trainers in literacy curriculum. 18 7-10 May 2016 10-13 May 2016 Curriculum for Basic Education and innovative reading instruction. Teacher training basic education curriculum and innovative reading instruction. 827 24 – 25 Nov 2016 8-9 Dec 2016 10 – 13 Jan 2017 17 – 20 Jan 2017 CGE members training on governance and management. 270 256 14-15 March 2017 13-14 April 2017 Promote the presence of teachers Improvement of literacy of school-aged children. Promote teacher attendance. 4526 22-23 May 2017 Advocacy workshop on the regular holding of teaching trainings. To identify the real obstacles to monthly teaching training and to find practical, sustainable solutions adapted to the realities of the field. 2 17 6 24– 25 January 2018 Training of trainers in innovative reading instruction Strengthen the capacity of school inspectors in reading instruction. 14 7-9, 13-15, and 20- 23 February 2018 Teacher training in innovative reading instruction methods Build the capacity of teachers in innovative reading instruction. 337 143 28,29 March 2018 Management of multi-grade classes 1727 5 25 Data provided by Counterpart International 26 Attendees included teachers, but school directors, education inspectors and local administrative authorities and representatives from the regional Pas’ Association. Participants include 5 teachers, 7 inspectors, 11 PAs members and local administrative authorities. Categories were not broken down in the training report. 27 It is unclear from Counterpart’s report if attendees were school directors or teachers. 23 Teacher Training McGovern-Dole’s most common training focused on improving reading instruction. McGovern-Dole and the national and regional government education ministries designed training that conformed to the current National education instruction methodologies. The training also includes assessment tools and lesson plans that assist teachers in designing classes. McGovern-Dole tested the effectiveness of these trainings with pre￾and post-test surveys. Learners scored in the 40 percent range on the pre-test and in the low 90 percent range on posttests. New literacy teaching skills are formulated at the central government level and then disseminated throughout the educational system through a training-of-trainers model. McGovern-Dole worked with the Ministry of National Education’s Directorate of Elementary Teaching to revise Senegal’s recently established curriculum and design training tools around the curriculum. The Directorate then trained local school inspectors, who trained the teachers of targeted schools. The trainings covered the new literacy curriculum, the importance of reading skills, reading assessment tools, curriculum and lesson plans. During the final evaluation, the evaluation team asked teachers whether they had received training. Teachers indicated that they had received training in health and nutrition and improved techniques for teaching reading. Teachers either directly attended trainings or were trained by the school director or another teacher that had attended a training. Teachers easily discussed new tools they had learned for teaching. Particularly, they discussed methods for teaching the new government curriculum, which focused on teaching the five components of reading. One teacher said that this method made the learning process clearer for students. One new technique that teachers learned was a method that teaches reading by emphasizing the sound of the word rather than piecing letters together. Teachers also said that they now start teaching reading to younger children than before McGovern-Dole. In interviews, several teachers and directors reported report that they were very pleased with the new, more interactive teaching approach they had learned and felt it was more effective. The survey reveals that of the teachers receiving pedogeological training, 60.4% applied what the learned “always”, 23.8% “often”, 15.8% “sometimes” and 0% “never”. In addition to training in literacy instruction, McGovern-Dole held advocacy workshops and assisted school teachers in organizing monthly performance meetings which are mandatory in Senegal but were not held with regularity in Saint Louis. The monthly meetings allow teachers to discuss work issues, and also to deliver peer-to-peer trainings on improved teaching techniques. Counterpart worked with the Saint Louis Academic Inspector to form teacher units at schools across the region. The purpose of the units is to address classroom instruction issues and promote student performance. Director Training School directors received training on managing the school feeding program. These trainings included proper food storage and granary maintenance, courses in training teachers about new reading instruction methods, training in the School Health and Nutrition Guide (SH&NG), school farm management, and canteen management. School directors that the evaluation team interviewed found the trainings useful. The trainings’ effectiveness is revealed in the high survey scores on knowledge of food storage practices and hygiene practices discussed in the Activity 6: Training in food preparation and storage practices section. 24 Activity 2: Construction of school infrastructure Table 9 Classroom Construction Targets and Progress Construction of School Classrooms # of classrooms LOP Target 43 Completed 30 In Progress 13 The McGovern-Dole Program aims to build classrooms for schools that have the greatest need. As indicated in the mid-term evaluation, schools in Saint Louis have a wide range of infrastructure needs. Some school buildings, usually those further from urban areas, suffer from lack of infrastructure maintenance. Many schools in more remote locations need complete construction of classrooms as they currently consist of a temporary structure built from scrap wood and metal with a thatched roof. As of the final evaluation, McGovern-Dole had not met its LOP target under this activity, having completed 30 of the 43 classrooms indicated as the LOP target. Delays with a current contractor and the rebidding of the work caused delays in the first half of 2018, but McGovern-Dole staff indicate that the remaining 13 classrooms will be complete before the program ends in September 2018. School directors whose schools had received new facilities were generally satisfied with their new facilities. One director said that the quality of the classroom and latrine were very good compared to the old ones that the new buildings replaced. There were however some problems with new construction. A school in Saint Louis mentioned that they had issues with their latrine not draining properly. Others who had not received new facilities either said that they were promised new facilities and they hadn’t been built yet or had questions about how schools were chosen for construction. One school director said that he thought it wasn’t “equal how classrooms and toilets are assigned.” The communication and construction issues that the program has faced were likely caused by the need to replace the program’s construction contractor. Counterpart International is planning to address these issues in the program’s final quarter. The mid-term evaluation team recommended that McGovern-Dole reallocate resources in this area to ensure that budget allocated for infrastructure was appropriate for wealthier and poorer school needs. However, by the mid-term evaluation, the program had already allocated its budget for this activity. In its response to the mid-term evaluation report’s recommendations, McGovern-Dole acknowledged the recommendation and indicated that a new infrastructure allocation policy would be put in place for future programs. Activity 3: Construction of latrines and water station systems As part of developing the school feeding system, Counterpart builds latrines and water station systems. As of the final evaluation, Counterpart completed construction of latrines and water stations at 24 schools. According to Counterpart staff, construction has started on the remaining 19 sites, and they expect to meet the activities goal before the project ends in September. Table 10 Progress toward School Latrine and Water Station Construction Targets Construction of School Latrines # LOP Target 43 Completed 24 In Progress 19 25 One hundred and forty two of the 159 schools (89%) that the evaluation team visited have some kind of facility where students can relieve relieve themselves. This is about the same percentage as at mid-term (86.5%). Only 53 of them are connected to water. If a school has water, it is likely it is connected to the latrine: 42 of the schools that had a water connection also had a water connection for their toilet. One school director in Dagana that the evaluation team interviewed did complain that the lack of water for the latrine made the latrine “not very good.” A director at a school in Dagana with 53 enrolled students that did not have a latrine said that created a hygiene problem for the school. Table 11 Schools with latrine and water infrastructure Status of school # Schools with latrines 142 Connected to water 53 Water connected to latrine 42 The evaluation team graded school latrines as very clean, mainly clean, a little clean, or not clean. As shown in table 12, a majority of those latrines are at least mainly clean. Still, 40 percent of latrines were either “a little clean” or “not clean.” This percentage is excessively high. Students that find latrines unfit will still walk to unsafe outdoor areas to relieve themselves and possibly avoid coming to school or will leave school early. Table 12 Cleanliness of School Latrines (n = 159) How clean are the latrines? % Very clean 21 Mainly clean 39 A little clean 30 Not clean 10 Since the issue was identified at the mid-term evaluation, Counterpart has taken steps to improve the cleanliness of latrines. Counterpart developed an infrastructure maintenance guide for use at the school level. The guide covered maintenance of classrooms, latrines, and other infrastructure. In 2017 they also carried out trainings in use of the guide. However, follow up visits revealed that latrines were still not up to standard. Counterpart has instructed its facilitators to continue to raise awareness at schools and in communities about the importance of maintaining latrines. The final evaluation found that 83 percent of schools that had latrines had separate facilities for girls and boys, which is an improvement over mid-term (75%). The improvement is the likely result of Counterpart’s efforts since the mid-term to establish proper signs on latrine doors. Counterpart worked with the Ministry of National Education to procure Female and Male signs for restroom doors and had them installed in early 2018. EXHIBIT 4 UNFINISHED LATRINE FOR BOYS AND GIRLS 26 Table 13 below shows structural conditions of latrines at mid-term and final evaluation. Both mid-term and final evaluation asked about community contribution to latrine maintenance. While we found that 44 percent of schools report community contribution to this activity, we also found through interviewing teachers and school directors that schools organize students to participate in this activity as well. School directors report that they divide the students up, generally into boys groups and girls groups, and the students clean the schools on a regular schedule. The evaluation team asked school directors how they decided which tasks the boys would do, and which tasks the girls would do. At one school, the school director said that some jobs were for boys and others for girls, and the girls cleaned the latrines. However, at every other school we interviewed, the director said that the boys and girls switched tasks during each session and that tasks were not assigned by gender. Table 13: Latrine's condition (mid-term n = 96, Final Evaluation n = 159) Mid-term % Final % Very well maintained 22 16 Well maintained 44 51 Somewhat maintained 29 30 Not maintained 4 4 Latrines available to boys and girls 75 83 Parents and community contribute to latrine maintenance 40 44 School Directors that the evaluation team interviewed said that they were generally pleased with the new latrines. In interviews, beneficiaries highlighted the importance of access to latrines. However, there were issues at some locations. One school director in Dagana said the lack of a water connection to the latrine created a problem. Students said that before the program they relieved themselves outside, but now they use the latrine which is much better. Students also mentioned that having separate boy and girl latrines did make female students more comfortable. Activity 4: Equip schools with energy saving stoves, canteen equipment, and materials McGovern-Dole equips schools so that they have tools they need to prepare and serve meals to students. These tools include bowls and utensils to prepare and serve food, pots to cook the food, and an energy efficient stove. Eighty-four percent of schools that the evaluation team visited have energy efficient stoves compared to an estimated 58 percent at mid-term. The final evaluation also found that 84 percent of schools had sufficient utensils for cooking and eating. The development and distribution of the energy efficient stove was the most important and complicated element of this activity. Making an energy efficient stove available to the schools was important for two reasons. First, energy efficient stoves reduce the environmental impact of the school feeding program. Secondly, energy efficient stoves reduce the work volunteers, primarily women must perform to gather firewood for cooking the school lunches. 27 Initially, the project intended to manufacture the stoves in cooperation with community members and Peace Corps volunteers. Counterpart brought in experts to research how to make a stove using local materials. After considerable effort, Counterpart concluded that the kind of stove that the Program could construct under its initial plan was not feasible. These types of stoves, those usable with large pots, broke down after a few months. As a next step, Counterpart researched firms that specialized in making energy efficient stoves. They found businesses with this specialty outside of the Saint Louis region. They met with three of the specialists and asked them to manufacture sample stoves. Once Counterpart staff verified that the samples worked, the team considered having communities make the stoves, but the communities’ manufacturing costs were too high. Counterpart then researched firms that could construct the stoves within the program’s budget. Following that step, Counterpart had volunteer cooks use the stoves for three weeks. After the test period, Counterpart collected their feedback. The cooks made several suggestions to improve the base models. For example, they suggested that a door with a latch be placed where they inserted the wood. The cooks also made recommendations for the placement of holes to speed up cooking and create less smoke. Using cooks’ recommendations, the samples were re-tested and cooks clearly preferred a stove manufactured by a Senegalese firm. This stove incorporated all of their suggested adjustments. Counterpart distributed this final model to schools. At the final evaluation, enumerators observed stoves at 135 of the 159 schools that the evaluation team visited. As at mid-term, the final evaluation team interviewed cooks about the energy efficient stoves. As at mid￾term, cooks say that its primary benefits are that it requires less wood, which means that they spend less time gathering the wood. It also shields the fire from the wind, and it is very heavy, making it difficult to steal or misuse. The mid-term evaluation identified three challenges with the stoves: 1) one stove per school was often not sufficient; 2) the stoves could not accommodate the various types of pots and pans that school cooks used to prepare meals; and 3) preschools did not use the stoves because of limited access to charcoal or wood. Some schools using the stove commented during final evaluation interviews that “one is not enough,” and that they “would like another stove,” even though it was not the perfect size for other cooking implements. Following recommendations made at mid-term, Counterpart procured an additional 204 stoves, 102 of which had been distributed to elementary schools, which allowed some schools to have two stoves. The second batch was due for distribution in late May, 2018. Activity 5: Good health and Nutrition Practices The good health and nutrition practices activity promotes the use of the School Health and Nutrition Guide (SH&NG). The guide aims to improve the health and hygiene practices of the school community including students, teachers, and directors. Counterpart promotes the guide by training school directors on SH&NG. The directors then train teachers on how to use the guides, and the teachers build lessons into the curriculum to improve students’ practices. Counterpart conducted trainings for the 204 McGovern-Dole primary school directors, leading to the training of 1,045 teachers. Table 14 below lists the trainings. Counterpart also trained 270 cooks who went on to train other volunteer cooks in their communities. FIGURE 5 AN FFE STOVE AND TRADITIONAL POT USED TO COOK SCHOOL LUNCH 28 Table 14 Health and Nutrition Trainings28 Date Title Directors Cooks Government 25-26 March Guidance for IEFs on training managers in the use of the Health / Nutrition / Environment Guide 7 15-20, 23-26 April 2017 Training elementary school principals in using the Health / Nutrition / Environment guide 204 (Directors trained 1,045 teachers) 6 March 2018 Retraining of Cooks in 270 School Canteens 270 Counterpart also printed 3,500 copies of the SH&NG. At mid-term, teachers and directors acknowledged that they had received the guide, and most had read it. During teacher interviews, teachers could pull out the guide and show sections of it to evaluators upon request. The teachers and directors that evaluators asked about the guide found it useful and practical as an educational resource. Counterpart reports that the program will distribute the last 500 guides between June and September of 2018. The guide was also distributed to the Regional Training Center for Education Personnel in Saint Louis, and 200 copies went to schools in the region not covered by the McGovern-Dole program. Counterpart is currently training new teachers and directors on the guide, and they will receive a copy for their classrooms. School directors trained teachers at their schools in hygiene and nutrition using the SH&NG. One director at a large school reported that he trained 13 teachers. Counterpart also conducted a “Hygiene Week” at each of the McGovern-Dole schools. Hygiene weeks were popular with parents and teachers. At the mid-term evaluation, the director of the Ehmane Racine Sy primary school said that hygiene weeks worked well. She had noticed that students hand washing had increased before meals and after using the bathroom as a result of hygiene week activities. Teacher Knowledge and Practices The mid-term evaluation attempted to measure the degree to which teachers were themselves trained in nutrition and hygiene, and the degree to which they instruct students in nutrition and hygiene. The teacher results are shown in table 15. 28 Data taken from Counterpart training reports. 29 Table 15 The degree to which teachers have received and use training (Mid-Term n =312, Final = 356) Mid-term % Final % Teachers that have received nutrition training in the past 3 months 33.87 13.88 Teachers who learned new tools and techniques to teach nutrition 96.2 97.96 Teachers who use new nutrition tools and techniques "always" or "often" 73.53 62.50 Teachers who have received training on hygiene in the past 3 months 38.46 17.56 Teachers who learned new tools and techniques to teach hygiene 95.83 98.39 Teachers who use new tools and techniques always or often 81.74 89.16 Fewer teachers that responded to the final evaluation survey said that they had received training in nutrition or hygiene in the past three months. Teachers that had received training found it useful, saying that they frequently used the new tools and techniques that they learned. One reason for the low percentage of teachers that say they received pedagogy training in the last three months may be a lack of clarity in what officially constitutes “training.” Teachers may not have received training from a school inspector, but may have received peer to peer training through regular teacher meetings, or received in-school training from the school director. It is unclear whether teachers responding to this survey question counted these events as training. Another reason may be that, while training in the SH&NG was delivered to all teachers last year, this year it was only delivered to new teachers, and was not scheduled until after the final evaluation, resulting in one fewer training for most teachers scheduled within three months of the evaluation. Table 16 shows how often the teachers provide instruction to their students. Teachers more often teach about hygiene in their classes, with 99 percent saying that they teach it “usually.” Seventy-seven percent said they will teach about hygiene in the current week, indicating that it is a frequent topic of discussion. Nutrition is taught less frequently. Slightly over 83 percent of teachers say that they teach nutrition with 54 percent saying they will teach in the coming week. 30 Table 16 Degree to which teachers instruct on nutrition and hygiene (n=312, N=356) Mid￾Term Final % usually teach hygiene in their class 99.04 99.43 % usually teach about nutrition 88.14 83.29 % have or will teach hygiene in current week 76.28 77.33 % Have or will teach nutrition in current week 54.49 54.11 When enumerators asked teachers if they had washed their hands the day of the survey, 94 percent said they had. Of the 94 percent that said they had washed their hands on the day of the survey, figure 10 shows the situations in which they had washed their hands that day. The most common answer at final evaluation “Before eating as at mid-term. The second most common response at mid-term and final evaluation was “after using the latrine.” However, while 44 percent of teachers gave this response at mid-term, only 29 percent gave the response at final evaluation. These percentages are surprisingly low given that the surveys were given in the middle of a school day. Figure 10 Situations when Teachers Washed Their Hands (%) (Multiple Responses Possible) Student Knowledge and Practices The final evaluation sought to measure the degree to which students engaged in good hygiene practices. The final evaluation used the mid-term team restructuring of the baseline survey evaluation questions to overcome some of the social desirability bias found in the baseline report. Rather than asking questions such as “did you wash your hands today before eating?” enumerators first asked students, “did you wash your hands today?” Of those that said yes, enumerators asked: “why did you wash your hands?” The intention of rewording the question was to remove prompts for when hands should be washed, and also lessen social desirability bias that comes when prompting specific instances of hand washing. 0 10 20 30 40 50 60 After touching an animal After changing a diaper Before giving food to someone else When you have earth on your hands Before touching or preparing food After touching something dirty After using the latrine Before eating Mid-Term Final 31 When asked if they had washed their hands today, 94 percent still said that they had, not significantly fewer than at baseline (95.3 percent). As students who are regularly taught lessons on hygiene, this result probably still reflects bias as all students knew that they should have washed their hands at least once. Table 17 shows student explanations of why they had washed their hands. A majority of students said that they washed their hands “before eating” (67%). As in the mid-term, the second most common answer was “after using the latrine” (30%). Table 17 Student Situtions for washing hands (Mid-Term n = 1028, Final n = 787) When did you wash your hands today? (multiple answers possible) Mid￾term % Final29 % Before eating 75.58 67.33 After using the latrine 36.48 30.00 After touching something dirty 19.84 12.00 When you have earth on your hands 6.03 5.07 Before touching or preparing food 8.17 1.60 After touching an animal 5.16 1.07 Before giving food to someone else 13.33 0.13 After changing a diaper 2.92 0.07 Hygiene Infrastructure One hundred and fifty-seven of the 159 schools of the schools that enumerators visited had a place to wash hands before eating. At mid￾term, many schools used buckets supplied by the program, and water brought from outside the school as hand washing stations. Between mid-term and final evaluation, Counterpart launched an initiative to place portable hand washing stations at each school. Counterpart collaborated with the School Medical Control Division (DCMS￾MEN) to design and procure one hand washing station for each school. The evaluation team saw these hand washing stations in action at the schools the team visited. Eighty-five percent of school’s hand washing areas were at least “mainly clean” at the final evaluation, which is a large improvement over the 46 percent judged clean at mid-term. Sixty-one percent had soap at final evaluation, another large improvement over the 40 percent with soap at mid-term. Fifteen percent had a hand towel for drying. The gross numbers are provided in table 18 below. These improvements reflect the distribution of portable hand washing facilities that Counterpart distributed to schools (exhibit 14), and the new trainings provided to school directors and Counterpart’s facilitators. 29 This excludes “Other” which accounts for 26.5 percent of responses. FIGURE 6 A MCGOVERN-DOLE PROVIDED HAND WASHING STATION WITH SOAP 32 Table 18 Access to Hand Watching Facilities (mid-term n = 96, final n = 159) 30 Hand washing facilities Mid-term Final w/ place to wash hands before eating 93 158 w/sink 33 100 Very clean 13 37 Mainly clean 31 100 A little clean 21 19 Not clean 31 6 % w/soap 39 99 % w/hand towel 5 24 About the same percentage of schools at mid-term and final evaluation had a place to wash hands before eating (99% at final). However, the percentage of schools with a sink tripled between mid-term and final evaluation. The final evaluation team witnessed school hand washing before lunch. Teacher’s took turn supervising the hand washing line. Hand washing was orderly, and students were diligent in doing it correctly. Activity 6: Training in Food Preparation and Storage Practices As a key part of the school feeding program, Counterpart works with schools to form school feeding management committees. McGovern-Dole’s facilitators work with these committees; comprised of teachers, parents, school administrators and other community members; to oversee the acquisition, storage, preparation, and delivery of food for the school feeding program. McGovern-Dole provides training to committee members. The two main types of training are the management of the food storage system, and skills in the sanitary preparation of nutritious school meals. Training Volunteer Cooks McGovern-Dole provided training to 808 school cooks from 270 schools over the course of the program, about three cooks per school. The cooks then returned to their communities and train other cook volunteers. Through interviews, the evaluation team learned that school cooks conducted the trainings, and the volunteers that were trained valued the instruction. In interviews, cooks revealed that they learned how to combine foods to make meals more nutritious. They most enjoyed learning new recipes and about proper nutrition. They also learned to tone down the spiciness so that the food would be more palatable to children. Cooks that the evaluation team interviewed were proud of the cleanliness of the cooking area and of washing their hands before cooking. Some cooks expressed a need for more and better equipment. Cooks that did not have structures to cook in said that they needed a kitchen. One factor that makes a kitchen necessary is the high wind in many school 30 Results statistically significant at final but not mid-term. EXHIBIT 7 STUDENTS WASHING THEIR HANDS BEFORE LUNCH 33 locations. Cooks working in the open or in temporary shelters face safety issues as open flames that are used when cooking with traditional implements flare and lurch unpredictably. Another issue that cooks face is the additional burden of unpaid care work faced when they volunteer for the McGovern-Dole program. The cooks that participate in the program are all women who cook for the school in addition to the care work they do at home. In some communities, where there were few volunteers, the few women who volunteer end up cooking every day at the school. The evaluation team interviewed one volunteer cook who said: “it’s difficult to do this and my other work.” Storage and Inventory Management School granaries were established in the second year of the program to store community food contributions for school canteens. Granaries are buildings in the community that are appropriate and available to store grain and other food for schools. Food contributions come from the community and may also come from other sources, including in at least one case a local military base. As of March 2018, 189 schools had served nearly 646 thousand meals to students from school granaries. The section on Activity 9: School farms, discusses the potential of contributions from granaries and school farms to sustain the McGovern-Dole program. School inventory management is handled efficiently. Table 19 shows the condition of food storage facilities at the schools we visited. 96.84 percent of the storage facilities had a secure roof, and 95.57 percent were clean and well maintained. Food storage security was at 91.77 percent a large improvement over mid-term, and 13.29 percent had some pest damage, which is also a considerable improvement. Table 19 Condition of Food Storage Facilities % (Mid-Term n =111, Final N = 159) Food storage Mid-term Final % food storage areas clean 92.79 95.57 % food storage w/ sealed roof 97.3 96.84 % food storage secure 80.18 91.77 % food with signs of pest damage 20.72 13.29 When asked how food should be stored, directors knew that the area should be well aerated (83%) and clean (86%). Many also knew that the food should be stored a specific distance from the walls. Fewer mentioned specifics about the materials the storage facility should be made of or other factors, but given the other issues they manage, not having these specifics on the tip of their tongue is not surprising. FIGURE 8 USING THE STOVE IN A TEMPORARY SHELTER 34 Table 20 Conditions for storage in warehouse How food should be stored in the warehouse? (Multiple answers possible) Mid-term % Final % Aerated 83.33 82.91 Adequate size 21.3 30.38 Clean 78.7 86.08 Safe 25.93 27.22 Accessible 31.48 20.89 Wall and cement floor 32.41 13.92 Distance between walls and food 57.4 27.21 Table 21 Director knowledge of food storage facility maintenance (mid-term n = 108, final n = 159) How should the storage space be maintained? (Multiple answers possible) Mid￾Term % Final % Clean interior and exterior 72.22 84.18 Sweep the soil weekly 50.00 73.52 Clean the roof, walls, and pallets once a week 53.70 59.63 Clean the outside of the warehouse once a week 40.74 59.63 Remove weeds and trash that attract rats and insects 34.26 45.57 Other 25.93 25.95 A higher percentage of directors had a proper understanding of taking inventory at final than mid-term (table 22). They knew about the inventory stock card and forms and to compare the quantity present against the amount listed on the card and theoretical stock (79%). Most also knew about completing the inventory form (73%). Table 22 Directors knowledge of inventory control (mid-term n = 108, Final = 160) How to keep inventory (Multiple answers possible) Mid-term Final Compare the quantity inventoried with the quantity on the stock card 77.78 86.71 Record the date of the inventory and sign your initials 20.37 30.38 Compare the physical inventory to the theoretical stock 76.85 78.48 Report reasons for loss in case of discrepancies 17.59 6.33 Completing the inventory form 69.44 72.78 Make a Loss allowance 7.41 3.16 35 Similar to the mid-term measurement, directors had a good understanding of how to ensure food was sanitary before serving it to students, as shown in table 23. Table 23 Directors knowledge of serving safe food How to serve safe food (Multiple answers possible) Mid-term Final Wash hands with clean water and soap before preparing food or before eating 98.15 98.73 Protect food from flies, cockroaches, dust 69.44 69.62 Respect the shelf-life dates 23.15 24.05 Avoid food with mold 32.41 20.25 Avoid breaking the cold chain 13.89 3.16 All raw consumed food (fruits and vegetables) must be cleanly washed before consumption 45.37 57.59 School feeding management committees’ work with McGovern-Dole facilitators to track inventory, order more food when required, and solicit community contributions. Food generally arrives on time, except for a delivery due in January 2018, which was delayed due to customs clearance issues. This delay and Counterpart’s handling of it is discussed further in the section on Activity 8: School Feeding. Incoming and outgoing food is tracked adequately. Food committee members could easily demonstrate inventory tracking forms and knew the schedule. Activity 7: Provide Access to Health Interventions Access to health interventions aimed to reduce absenteeism and increase student attentiveness through three activities; provision of vitamins, provision of de-worming medication, and the establishment and use of moringa plantations. Deworming Medication and Vitamin A Supplements McGovern-Dole received and processed two shipments of de-worming medication and Vitamin A supplements. The first shipment was distributed to students in January 2017, when 44,095 students in all of the program’s 270 schools received deworming medication. The second shipment of the medication was divided into thirds: 1/3rd for distribution to McGovern-Dole schools in Saint Louis now, 1/3rd for distribution to other schools in Senegal now, and 1/3rd set aside for distribution in December 2017. Over the course of the program, 90,726 doses of deworming medication were distributed by the McGovern-Dole program. 3,097 doses for teachers and 9,039 doses for volunteer cooks were also distributed. 10,931 doses of vitamin A supplements were distributed to preschool students. After Counterpart’s last deworming campaign, Counterpart interviewed school cooks to see if they had noticed any changes in student behavior. Cooks mentioned that students were eating more food after the campaign. A parent at a primary school said that his son had parasites before the program and no longer does. He also has a daughter who had no appetite before the program, and her appetite has grown. He attributed the improvement to the vitamin supplements but also, and more likely that they’ve reduced the spiciness of the food they cook at home, which they learned from the McGovern-Dole program. Counterpart’s Health officer mentioned that the biggest struggle in the deworming program was spreading correct information about how parasites are acquired. While students and parents had many correct ideas about how parasites were acquired and could be prevented, some myths persist. The most common myth that the evaluation team heard was that kids get parasites from eating too much sugar. 36 Healthy Message Murals The McGovern-Dole program promoted healthy behavior through mural competitions. Students submitted ideas for murals that promoted healthy messages, and the best one was selected to be painted on a wall at the student’s school, which was both an honor for the student and an effective means of communicating messages about healthy behavior to the community. Examples of mural topics include the importance of avoiding unclean water and the reasons it is necessary to keep common areas swept and clean. Moringa Plantations The McGovern-Dole program piloted moringa plantations at schools as a way of promoting student health. Moringa trees grow nutritious leaves that can be ground into a powder and added to food. Counterpart purchased 12,114 moringa plants over the course of the program. School teachers and directors demonstrated the plants to the evaluation team and could explain the benefits of the plant as well as the way it is processed and used in food. Counterpart reports that 43 of the 270 McGovern-Dole schools regularly use moringa powder in school meals. The moringa plantation program produced some of the most questionable results of all of the program. Moringa trees are difficult to maintain in the Saint Louis region. They require rainfall to be productive where they could be of the most use. In years like FY2018, an unusually dry year, the trees are likely to die. Additionally, many schools lack fences or barrier walls, which exposes the trees to goats and other animals that feed on them. Lastly, in some areas where schools are located, the soil isn’t appropriate for moringa cultivation. Activity 8: The school feeding program School feeding is the core of McGovern-Dole. As reported at mid-term, McGovern-Dole Senegal’s school feeding program was highly successful. The program provided over 15 million meals over the course of the program, greatly exceeding its LOP target. From October 2017 to March 2018, the program motivated its beneficiary schools to provide an additional 645,841 meals without the assistance of USDA provided commodities. Stakeholders attribute the achievement of the program’s higher-level objectives, namely improved literacy and increased student attendance, to the school feeding activity. The McGovern-Dole program’s primary activity is to provide food to schools in food insecure regions. Distributing the food through schools provides incentives for students to attend school and to make sure they have the energy required to be attentive and focus on their studies. Many of McGovern-Dole’s other activities are implemented in support of the school feeding program. School farms and school granaries have been implemented through McGovern-Dole to replace McGovern￾Dole’s food donations to schools. The section on Activity 9, School Farms, attempts to address the question of feeding program sustainability and replacement of McGovern-Dole donations through farms and granaries. FIGURE 9 TWO MURALS ON MCGOVERN￾DOLE SCHOOLS 37 Table 24 McGovern-Dole Meal Schedule School Meal Portions Preschool Breakfast 5 days a week 100 grams rice or maize meal; 20 grams lentils or peas; 15 grams vegetable oil Primary school Breakfast 5 days a week 100 grams rice or maize meal; 20 grams lentils or peas; 15 grams vegetable oil Lunch 2 days a week 150 grams rice or maize meal; 25 grams lentils or peas; 20 grams vegetable oil The program requires successful coordination of the following activities, some of which are discussed in more detail under their own activity heading below: • Coordination with government entities including local and regional inspectors and the Ministry of National Education • The establishment of successful food storage facilities at each beneficiary school • Training of school directors in food storage requirements • Training of school committees in inventory management • Secure and sanitary storage of food used in the program • Oversight by McGovern-Dole facilitators • The transfer of USDA commodities to school facilities • The organization of volunteers to oversee and maintain food in storage • The organization of local volunteers into groups of cooks that prepare food for the students • Raising funds and food contributions from parents to complement the commodities provided through the program. Food Acquisition and Community Contribution The school feeding program relies on USDA-provided commodities (Milled Rice, soy-fortified cornmeal, Peas, lentils and vegetable oil). Communities also contribute cash and locally produced food to compliment the commodities. As discussed under Activity 6 in this document, USDA’s commodities are well managed. The schools are consistently adequately supplied. Food loss is rigorously accounted for. School management volunteers demonstrated in interviews that they are well versed in proper storage and distribution protocol. School feeding requirements Table 25 below shows the volume of food and cost of food required to provide meals for the school feeding activity. The table presents requirements per day and per year. It also shows the volume and cost of food per student and per school. Note that throughout we exclude food that is served with meals but not provided by MGD, including vegetables, salt, milk and animal protein. These supplementary foods are sometimes provided through contributions from parents, or in the case of vegetables, through school farms. Costs are estimated using the market price information (see footnotes to table 25 for details). The cost of food to provide preschool students (91 students per school on average) one meal a day for the whole school year is 950 thousand CFAs or just under 1,679 USD. The annual cost of providing 7 meals a week for a primary school is nearly 3 million CFA or 5,125 USD. 38 Table 25 Food required to sustain the School Feeding Activity Day Year31 Student School32 Student School Preschool – Volume (KG) Grain 0.1 9.1 18 1,638 Oil 0.02 1.8 3.6 328 Pulses 0.02 1.8 3.6 328 Primary school - Volume (KG) Grain 0.25 50.6 28.8 5,846 Oil 0.04 7.1 4.1 840 Pulses 0.05 9.1 5.4 1096 Preschool – Cost (CFA) Grain33 22 2,002 3,960 360,360 Oil34 28 2,548 5,040 458,640 Pulses35 8 728 1,440 131,040 Total 58 5,278 10,440 950,040 Primary school – Cost (CFA) Grain 55 11,165 6,336 1,286,208 Oil 49 9.947 5,796 1,76,588 Pulses 18 3.654 2,160 438,480 Total 122 24,766 14,292 2,901,276 Granaries From October 2017 to February 2018, 140 primary schools with granaries had used or still had in stock, on average, 673 kilograms of grain (mostly rice), 35 kilograms of oil, and 30 kilograms of pulses. In addition, they had spent over 51,000 CFAs (Approximately $90). These figures are significantly lower for the 49 preschools with granaries (see table 26 below). Note that these figures are not representative of the total stock accumulated or spent throughout the year as the school year runs from October to July, and food may come into the granary throughout the year. Also note that the program’s data showed opening stocks for October 2017, but it is not clear when this stock was received and over how long a period it was accumulated. 31 Based on 180 school days per year. 32 School estimates are based on the 91 average students per preschool and 203 per primary school found for FFE schools. 33 The price used for maize is 220 CFA/KG. This is the midway point between the school sale price of 164 and the March 2018 retail price of 274 CFA/KG (Bulletin Mensuel d’Information sur les, Marchés Agricoles N°360 – Mars 2018). This is an estimate of a wholesale price, due to schools buying in bulk. 34 The price for vegetable oil used is 1400/KG. (http://apanews.net/index.php/news/senegal-legere-baisse-du-prix￾du-litre-dhuile-vegetale-en-fevrier-2018). 35 Cowpea prices are used to cost pulses. The price used is 400 CFA/KG. This is slightly less than the reported February retail price of 410 to account for buying in bulk. 39 Table 26 School Granary Inventory - Oct 2017 to Feb 2018 (KGs unless noted) USE STOCK36 TOTAL PRESCHOOL (49) TOTAL GRAIN (RICE IN PARENTHESIS) 94 (76) 162 (157) 256 (233) OIL 5 9 13 PEAS 10 1 11 CASH (CFA) 8,772 7,013 15,786 PRIMARY SCHOOL (140) TOTAL GRAIN (RICE) 352 (287) 322 (314) 673 (601) OIL 20 15 35 PEAS 25 6 30 CASH (CFA) 29,517 21,886 51,403 Meal Preparation and Delivery McGovern-Dole’s facilitators work with schools to organize parents into school feeding management committees. These committees recruit mothers in the community to serve as volunteer cooks. The volunteer cooks serve in groups of two or more. Each group prepares lunch for the school for a week at a time on a rotating schedule. For example, for a school that has six groups of volunteer cooks, each group will prepare lunch for one week out of every six. Through interviews and focus groups with students, the final evaluation and mid-term evaluation teams found that the food was popular with students and parents. Students often said that they prefer the food at school to what they ate at home. Lentils are particularly popular. Directors report that the popularity of the food has increased attendance considerably. Examples include the Ndiakhaye school, where enrollment has increased by 30 students, the Ndiawgone preschool, who said that enrollment has approximately doubled since before the program’s implementation, and the EE de Thiago school, where the director mentioned that his CE level classes went from 20 students to 47 students per class. One school director reported that he even had one student from Mauritania, which he attributed to the school feeding program. In addition to increased enrollment, the school feeding program has improved the quality of attendance. This improvement takes three forms. First, students from poor families would often eat very simple breakfast or no breakfast. These students now receive sufficient food at school, which improves their attentiveness. Parents, directors, and teachers reported that the students were healthier and now have more energy for learning. Additionally, students would often go home for lunch. Teachers report that few of these students would return for afternoon classes. Now, students have no reason to leave, so they are more likely to stay for afternoon classes. 36 As of the end of February, 2018. 40 Building School Feeding Sustainability The mid-term evaluation recommended that Counterpart change its formula for calculating how much donated food to distribute to each school. Counterpart distributed a specific quantity of food per student to each school. The mid-term evaluation pointed out that McGovern-Dole schools varied widely in the wealth of their communities and school needs. The mid-term evaluation recommended distributing food, and other resources, based on school need rather than taking a one size fits all approach. In response, Counterpart developed a system for making sure that schools with the greatest need got the most food, while schools with lesser need were encouraged to rely more on community resources (table 8, school selection section). Schools were categorized based on the score their assigned facilitator gave the school. Scores were calculated by indicating how the school ranked on 16 indicators grouped into 4 categories (table 27). The higher the score, the higher the level of the school’s food security. Schools scoring higher than 8 were deemed as having “low food insecurity,” schools scoring between 4 and 8 were categorized “medium food insecurity, and schools lower than 4 had “high food insecurity. Table 27 Criteria for School Categorization CRITERIA POINTS Criterion Agricultural Perimeter POINTS Community Fields Developed By Project 3 Community Field Developed By The Population 2 School Garden 1 No Agricultural Perimeter 0 Criteria Location And Economic Opportunities POINTS School Located In The Chief Town Of Region And Department (Strong Economic Opportunity) 3 School Located In The Chief Places Of Communes (Average Economic Opportunity) 2 School Located In Rural Areas And Easy To Access (Low Economic Opportunity) 1 School Located In Rural Areas And Difficult To Access (Little Economic Opportunity) 0 Criteria Contribution to the GRANARY POINTS Strong Contribution In Loft With More Than 15 Bags Of 50kg 3 Average Contribution In Loft Between 10 And 15 Bags Of 50kg 2 Low Contribution Attic With Less Than 10 Bags Of 50kg 1 No Contribution 0 Criteria For Access To Irrigation Points Located Less Than 1 Km From The River / Lake 3 Located 1 To 3 Km From The River / Lake 2 Located More Than 3 Km From The River 1 Located In The Diéry 0 41 In McGovern-Dole’s original work plan, the program was scheduled in its last year to reduce food distributed to school’s equivalent to one meal per week per student every quarter. As the final year approached, the program had two challenges to address. The first was how to change this original plan so that it worked toward sustainability for schools based on each school’s need rather than implementing an across the board reduction in food distribution. The second was that Counterpart’s commodities shipment for the last quarter of 2017 was delayed, so the program was restricted to distributing the food it had on hand from October through November. Counterpart addressed these challenges by asking schools to provide food without McGovern-Dole assistance for a specific number of days during the school year based on the school’s categorization. High food insecurity schools were asked to provide 5 days of meals to students, medium insecurity schools were asked to provide 10 days of food to their students, and low food insecurity schools were asked to provide 15 days of meals for their students. All categories of schools exceeded the targets that Counterpart set for them, demonstrating their ability to work toward program sustainability. Table 28 shows the achievement for each school group when McGovern-Dole served them based on school need and sustainability goals. Achievement occurred between January and the final evaluation. Table 28 Days providing meals without USDA assistance by school category Category Target number of days Average number of days Median Minimum Number of days Maximum Number of days 1 st Quartile 3rd Quartile Low Food Insecurity 15 7.07 2 037 51 0 12 Medium Food Insecurity 10 9.59 5 038 74 0 12 High Food Insecurity 5 10.56 5 039 73 0 10.25 The results show a high level of variability within categories. For low food insecurity schools, half of the schools were unable to provide food for more than two days without using USDA commodities. For medium food insecurity schools, half of the schools could provide meals for five days, or half the target set by the program. 56% of the high food insecurity schools provided meals for their students for five days or less, less than a third of the target. However, schools that were motivated to provide meals for their students greatly exceeded the target that Counterpart set for them. Twenty-seven high food insecurity schools (more than doubled the target number of days set to provide meals without USDA assistance, and at the highest level, those schools performed as well or better than the other two categories. For the low food insecurity category, only 8 schools met or exceeded the target 15 days. Two of those schools more than doubled the target, with one feeding students 37 11 low food insecurity schools did not provide meals without using USDA commodities. 38 49 medium food insecurity schools did not provide meals without using USDA commodities 39 37 high food insecurity schools were unable to provide meals without using USDA commodities 42 for 34 days and the other for 51 days. The remaining 6 schools fed students for 15 – 22 days. Thirty-eight medium food insecurity schools met the target, with 16 of those more than doubling the target number of days. Table 29 Number of meals provided without USDA assistance by School Category Category Target number of meal days Average number of meals Median Minimum Number of meals Maximum Number of meals 1 st Quartile 3rd Quartile Low Food Insecurity 5 1,218 175 040 10,578 0 1,558 Medium Food Insecurity 10 2,250 495 041 36,206 0 2,800 High Food Insecurity 15 1,642 349.5 042 24,371 0 1531.5 The number of days that a school can provide food for its students is related to the number of students enrolled in the school. Clearly, larger schools have to provide more food than smaller schools. When looking at the number of meals that schools in each category provided the sustainability picture becomes more surprising. Though medium and high food insecurity schools provided meals for a similar number of days on average, medium food insecurity schools provided many more meals per school, indicating that medium insecurity schools are larger than high food insecurity schools and low food insecurity schools. Medium schools provided an average of 235 meals per day while high insecurity schools provided 156 and low insecurity schools provided 173. In each category 50 percent of the schools provided a fraction of the average for the category, indicating that when the schools in the top 50 percent became motivated, they highly overperformed. For high food insecurity schools, a couple of highly motivated schools brought the average number of meals very high. However, high insecurity schools still have a median number of meals that outperforms low insecurity schools and is comparable to the apparently larger medium insecurity schools. Determining next steps toward sustainability should factor in the motivating forces that led some schools to perform well and others unable to perform at all. Activity 9: Establishing Community Farms School farms were a pilot program of MGD. The pilot’s intention was to test how effectively farms managed on behalf of schools could generate food and revenue to support school feeding programs. McGovern-Dole established twenty school farms near schools. To qualify for a school farm, schools had to procure at least half a hectare of arable land in their community, have access to water for irrigation, and demonstrate community enthusiasm to support the project through the provision of volunteer labor. 40 11 low food insecurity schools did not provide meals without using USDA commodities. 41 49 medium food insecurity schools did not provide meals without using USDA commodities 42 37 high food insecurity schools were unable to provide meals without using USDA commodities 43 In the first year of the program, MGD gave the 20 selected school farms pumps and other inputs, such as seeds and fertilizer. MGD along with the Ministry of Agriculture (MoA) also provided technical and managerial training to school farm presidents and helped to mobilize labor in communities to work on the farms. School farms were instructed to save part of their harvest each year to buy inputs for the following season. MGD and the MoA provided follow-up technical support and monitoring in subsequent seasons. Some school farms started production in 2016. The bulk, however, began producing in 2017. Table 30 below shows that in 2017, eleven farms produced rice on an average of half a hectare of land. Paddy (unmilled rice) yields were over 5 metric tons per hectare, exceeding the national average of 4.14. This national average, however, is deflated by low-yielding rainfed rice in the southern part of the country that accounts for nearly a third of national production.43 According to the MoA staff, the yields observed on the MGD school farms were lower than on other farms in the same area. Profit from rice cultivation - rice kept or sold for use by the school canteen - was 32 percent. This figure is lower than the 39 percent that the Japanese International Cooperation Agency (JICA) calculated in 2013 for their study of rice production in Podor and Dagana. 44 This difference may result from the JICA report not having included the cost of “free” family labor provided for family farms. Labor appears to be a significant issue for school farms and is discussed further below. Nine school farms produced vegetables and fruit, including tomato, onion, okra, and watermelon. Vegetables and fruit were produced on an average of 0.39 hectares in 2017. In 2018, some school farms are also cultivating groundnuts. Profit on vegetable cultivation is also 32 percent with only 7 percent consumed by school canteens and the remaining 25 percent sold to benefit the canteen coffers. This year, CPI is encouraging school farms to produce multiple crops per year. The main rice season is from March to July while vegetables are mostly cultivated from September to February, which allows for the cultivation of both crops on the same land in the same year. Cultivating both rice and vegetables is a way to maximize the productivity of the school farm and provide the school canteen with fresh vegetables for a good portion of the school year. School farms make an important contribution to the schools that manage them and show potential to make a substantial contribute to the school feeding program’s sustainability. In order to achieve true sustainability however, schools would have to expand the size of their farms. The last two columns of table 30 show projections for average school farm sizes of one and two hectares, each cultivating both rice and vegetables. Increasing average farm size to one hectare, cultivated with both rice and vegetables, would increase the total school farm profit by a factor of 4, from 95,480 to 403,184 CFA. Increasing to two hectares with both rice and vegetables would double this profit to over eight hundred thousand CFA. Through discussions with school farm presidents, the evaluation team found that in many cases land was available for farm expansion, and in fact, school farms had access to more land than they could cultivate. 43 CGIAR. Ricepedia: Senegal. Accessed May 2018. 44 JICA. 2014. Project on Improvement of Rice Productivity for Irrigation Schemes in the Valley of Senegal in Republic of Senegal: Final Report 44 Table 30 School farm 2017 performance with one and two hectare projections45 2017 RICE OR VEG 2017 20 FARM AVG.46 1 HECTARE PROJECTION 2 HECTARES PROJECTION RICE Farms Producing Rice (%) 100 58 100 100 Average Hectares Rice 0.51 0.29 1 2 Yield (Kg/Ha) 5,248 5,248 5,248 5,248 Production 2,650 1,534 5,248 10,496 Price (Cfa/Kg) 164 164 164 164 Profit (%) 32 32 32 32 Consumed (%) 29 29 29 29 Consumed - Paddy(Kg) 769 445 1,522 3,044 Consumed – Milled (Kg)47 515 298 1,522 2,039 Sold (%) 3 3 3 3 Value Of Sales (CFA) 13,039 7,549 25,820 51,640 Total Value Of Profit (CFA) 97,490 56,441 193,049 386,097 VEGETABLES48 Farms Producing Vegetables (%) 100 47 100 100 Average Hectares Vegetables 0.39 0.19 1 1 Yield (Kg/Ha) 3,998 3,998 3,998 3,998 Production 1,568 743 3,998 7,996 Price (Cfa/Kg) 164 164 164 164 Profit (%) 32 32 32 32 Consumed (%) 7 7 7 7 Consumed (Kg) 110 52 280 560 Sold (%) 25 25 25 25 Value Of Sales (CFA) 64,386 30,499 164,168 328,336 Total Value Of Profit (CFA) 82,414 39,039 210,135 420,270 TOTAL CASH INCOME (CFA) 77,425 38,048 189,988 379,976 TOTAL VALUE OF PROFIT (CFA) 179,904 95,480 403,184 806,367 However, expanding farms and producing higher yields faces two important constraints: lack of capital for inputs and labor constraints. For example, rice yields were only 3,292 KG/HA in 2016. Low yields may have 45 Note that there is data for 11 rice producing farms and 9 vegetable producing farms. Only one farm, Themeye Thiago, produced both. 46 This is the weighted averaged across the 20 school farms. 47 Conversion rate of 67% (CGIAR. Ricepedia: Senegal. Accessed May 2018.) 48 Tomato, onion, okra and watermelon. This year some farms report planting groundnuts too. 45 been the result of pumps that were too small and relied on costly fuel. Several school farm presidents noted the low output and high cost of fuel for pumps. One president said it was the reason he was not producing vegetables this season. Another noted that while they had more land available for cultivation, the pump constrained their ability to expand. CPI is aware of the pump problem and is in the process of replacing them. Access to capital is another constraint to school farms’ ability to cultivate multiple crops per year or expand their area under cultivation. In the program’s first year, school farms received free inputs including seeds and fertilizers from CPI. In the subsequent years, school farms were required to pay for inputs through savings from the previous year’s harvest. Because of pump issues, discussed above, the first year’s yields were low, and therefore savings for 2017 inputs was also low. As a consequence, the area under rice cultivation per farm declined, and only one farm produced both rice and vegetables in 2017. Labor availability is another a constraint to farm production and expansion. Many schools have a difficult time mobilizing volunteer, productive labor. If school farms expand, the burden of mobilizing volunteer labor will increase. Some schools that have difficulty mobilizing sufficient volunteer labor either pay for labor with rice at the end of the season or use a mix of free and remunerated labor. In interviews, school farm presidents said they had their first priority is to manage their own family farms, which likely represents the sentiment of many potential volunteers. A Summary of Contributions to School Canteen Sustainability Table 31 below shows the potential of school granaries and farms to replace the McGovern-Dole donated food for school feeding. For primary schools, the table also includes the 375,000 CFA the government provides to schools with a canteen. Total 1 shows the actual total contribution based on 2017/2018 data of school granary and farm production. Totally 1 describes schools with both granaries and school farms. Total 2 shows the total contribution of granaries and farms if granary contribution were to double and school farms were to expand to two hectares on average. The total also includes the contribution of two-hectare farms producing both rice and vegetables. Like total 1, the second total is only for schools with both granaries and farms. Figure 11 The Amount of MGD Food that Is Currently and Potentially Covered by Granaries and School Farms Preschool 2017/2018 Preschool 2 HA farm Granary, School Farm, Deficit 10.0% 10.9% 79.1% 19.9 % 87.2% 46 For preschools, current school farms and granaries could replace 20.9 percent of donated food, or one meal per week. The contribution of granaries and school farms is almost equal. If preschools could cultivate 2- hectare farms, and the contribution from granaries was to double (Total 2), all of the preschools’ food requirements would be met (see figure 11). Primary schools with granaries and school farms that receive government funds could currently replace 25.5 percent of McGovern-Dole donated food. This contribution comes from granaries (9%), school farms (3.6 %) and the government (12.9 %). If granary contribution is doubled and school farms size increased to 2 hectares (rice and vegetables), 59.4 of the food provided by FFE would be replaced (figure 12). Figure 12 Contributions to Primary School Canteen Sustainability Primary school 2017/2018 Primary school 2 HA farm Granary, School Farm, Deficit, Government 9.0% 9.0% 12.9% 69.1% 18.0% 28.6% 12.9% 40.6% 47 Table 31 Contribution to School Canteen Sustainability 49 The value of vegetables consumed from the school farm is excluded. Required for FFE Granary Farm (2017)49 Farm (2 HA) Government Total 1 2017/18 Total 2 2 HA Total % Total % Total % Total % % % Preschool Grain (KG) 1,638 256 15.6 298 18.2 2,039 124.5 33.8 155.8 Oil (KG) 328 13 4 4 7.9 Pulses (KG) 328 11 3.4 3.4 6.7 Cash (FCA) 15,786 1.7 38,048 4 379,976 40 5.7 44.3 Total value (FCA) 950,040 94,706 10 103,635 10.9 828,638 87.22 20.9 107.2 Primary school Grain (KG) 5,846 673 11.5 298 5.1 3,044 52.1 16.6 57.9 Oil (KG) 840 35 4.2 4.2 8.3 Pulses (KG) 1,096 30 2.7 2.7 5.5 Cash (FCA) 51,403 1.8 38,048 1.3 379,976 13.1 375,000 12.9 3.1 29.6 Total value (FCA) 2,901,276 260,463 9 103,635 3.6 828,638 28.6 375,000 12.9 25.5 59.4 48 The evaluation team’s analysis shows that in a best-case scenario, preschools could meet their feeding requirements, but primary schools’ could only meet 60 percent of their feeding requirements. Most likely, not all schools will have both a granary and a well-functioning two-hectare farm. Therefore, many schools would have to find an alternative solution if McGovern-Dole in Saint Louis ends in September. The school farm pilot program and the establishment of granaries have the potential to make major contributions to a sustainable school canteen program. This opinion was also expressed in MoA expressed in interviews. Also, the Saint Louis regional school inspectors office stated that “generalizing granaries to the whole region” was among its hopes for building on the program’s success. However, with farms’ current limited size and planting schedule, which results in one crop sewn per year, school farms make only a minimal contribution to school canteen sustainability (<4% for primary schools). Access to three types of capital be addressed before the significant expansion of the school farm pilot project is considered: access to finance, labor, and management. Activity 10: Improved Teacher Attendance Under this activity McGovern-Dole was tasked with promoting teacher attendance through the reduction of administrative duties. The activity requires organizing advocacy workshops that would result in the reduction in time and resources teachers required to collect their salaries. The McGovern-Dole Program conducted two advocacy workshops, both taking place before the mid-term evaluation. One workshop that hosted 25 participants from IEFs, the Regional Agency for Early Childhood Education, and one that hosted 45 teachers from Dagana and Podor. The evaluation team calculates 92.5 percent teacher attendance rate for 2018 through the end of April, based on Counterpart’s teacher attendance data. This figure seems reasonable considering that teachers often live close to the school’s they serve. As a point of comparison, teachers in the US are present for 94 percent of the school year. 50 Also, in interviews, directors said that teachers arrive on schedule and on time. Another school director said that she had no problem with teacher attendance and “except for the strike, they are here.” Teachers said that they had no obstacles to attending work. Teacher attendance is a complicated issue in Senegal. During the final evaluation, teachers across the country were on strike to demand better working conditions and to protest what the teacher’s union saw as an overly strong response by the government to a previous teacher strike. Teacher strikes complicate education in Senegal. The Saint Louis School Inspectors office said that a primary school teachers’ strike in 2016 lasted three months. The next year, CM 2 (6th grade) students were behind, and it was difficult for the school system to provide resources for students to catch up to level. The evaluation team asked teachers whether they felt their administrative duties were onerous or whether they had any difficulty collecting their salary. None of the teachers that we interviewed indicated that they had any problem in these areas. The evaluation team also asked parents, students, teachers, and directors whether teacher attendance was a problem at their school. While teachers are absent from time to time, no interviewee indicated a concern about teacher attendance. There are other issues that McGovern-Dole could focus on that might have a greater impact on teacher performance. For example, while teachers praised the school feeding activity, several mentioned that they 50 https://www.nctq.org/dmsView/RollCall_TeacherAttendance - data from the 2012 – 2013 school year. 49 would also like lunches provided for teachers. They also said they would prefer their own latrines rather than using the same latrines as the students. Activity 11: Parent Association Training McGovern-Dole relies heavily on community volunteers. The program supports government created parent associations called School Management Committees, to build community ownership of the program, keep the program aware of community needs, and ensure program transparency. The Associations also serve as a means through which schools set parent contribution policy and communicate that policy to community members. PAs were formed in 2016. Between November 2016 and January 2017, 526 PA members received training in governance, leadership, and management. In later periods, SMC members were also trained in infrastructure maintenance SMCs also serve as the contact point for directors to organize the community. At some wealthier schools, the SMCs raise funds to build classrooms, provide electricity, and assist the school director in making decisions. Monitoring and Evaluation of Program Activities As reported in the mid-term report, McGovern-Dole appears to have an excellent and efficiently functioning monitoring and evaluation system. Facilitators work with community members, school directors, and teachers to collect data important to the program. This data includes all output level data reported in semi-annual progress reports. As at mid-term, the evaluation team met with the program’s monitoring and evaluation officer to test their system. The monitoring and evaluation system could quickly bring up information about specific students in randomly chosen schools. When the evaluation team visited schools, the information brought up in the monitoring and evaluation system proved accurate. The system had adequate data entry staff which allowed quick reporting on program activities. At mid-term, the evaluation team reported on the monitoring and evaluation system’s important role in fine￾tuning the project’s energy efficient stoves and serving the program’s management in monitoring the performance of staff. Since mid-term the monitoring and evaluation system proved its worth in responding to the mid-term evaluation’s recommendations. At mid-term, the evaluation team recommended that the McGovern-Dole program provide resources based on need rather than follow a one size fits all approach. Counterpart’s first step towards a needs-based approach was to use its monitoring and evaluation system to identify differences in school status, such as relative level of wealth, the stability of the local population, economic potential of the community, and needs. The program’s monitoring and evaluation staff developed a data analysis tool that took in information provided by school facilitators and created a school ranking based on level of food security. The development and implementation of the tool demonstrated the strength of the program’s monitoring and evaluation system. The tool was developed quickly and rolled out to facilitators who could swiftly understand its purpose and use it, demonstrating Counterpart’s commitment to learning and adapting to new lessons. At mid-term, the evaluation team recommended that three indicators merited reconsideration: 1. 100 percent of school-age children receive a minimum acceptable diet at the school level. 2. 90 percent of schools have energy-saving stoves. 3. 20 percent increase in teacher attendance rates compared to baseline. We note that USDA dropped the minimum acceptable diet indicator and the teacher attendance indicator from its current standard indicator definitions. 50 The final evaluation brings forward three other recommendations regarding indicators: Improve Student Attendance The LOP target for this indicator is the number of students that regularly (80%) attend USDA supported classrooms, disaggregated by gender. McGovern-Dole Senegal’s monitoring and evaluation system adequately collects school attendance records for more than 45,000 students and enters the information into a central database where the monitoring and evaluation staff can analyze the data and uncover trends. However, the data collected through school records may not accurately reflect attendance rates. The high reported rates of attendance, including very high rates reported at baseline, and the insufficient administrative staff at schools makes the data seem suspect. Also, though records show an increase in student attendance increased of 2%, from 97% at baseline to over 99% by final evaluation, school teachers and directors report a much bigger increase over the course of the program. Much of this increase is due to kids attending school more regularly and staying at school longer during the day because they do not need to return home to eat. To truly understand the way the program influences school attendance, McGovern-Dole should have an independent research plan for studying the topic from the beginning of the program. The plan could include random spot checks once per quarter at a sample of schools, which would generate a clearer picture of the influence of McGovern-Dole on student attendance. Understanding children who do not attend school McGovern-Dole works to overcome barriers to school attendance and measures the benefits of the program for children who regularly attend McGovern-Dole schools. McGovern-Dole should also consider researching reasons that some kids do not attend school despite the school feeding incentive. Collecting data on this population would lead to innovative activities that reduce barriers to attending school. Understanding the status of children who are not in school would also provide a clearer picture of McGovern-Dole’s impact. For example, the final evaluation team believes it is possible that boy students who become frustrated with learning to read are more likely to drop out of school than girls. Using McGovern￾Dole’s current data collection methodology, only boys in school take the reading test conducted by the external evaluator, so boy’s average reading scores may be inflated by the non-participation of boys who are poor at reading and have dropped out of school. Improve the collection of the Minimum Acceptable Diet (MAD) indicator. The program’s measurement of the percent of school-age children receiving a minimum acceptable diet at the school level differs from the definition used commonly by international health organizations. The indicator could be improved by using 24-hour recall consistently and ensuring that the survey uses common definitions of food groups: 1. Grains, roots and tubers, 2. Legumes and nuts, 3. Dairy products (milk, yogurt, and cheese), 4. Flesh foods (meat, fish, and poultry), 5. Eggs, 6. Vitamin A rich fruits and vegetables and 7. Other fruits and vegetables. Evaluation Questions Implementation and Relevance Implementation compared to design: The McGovern-Dole program’s design is its strongest characteristic. The best source of evidence of the program’s success in design and implementation is the results it has achieved in student attendance, student literacy, and stakeholder’s improved knowledge of nutrition and 51 hygiene. As discussed in the mid-term evaluation report, the program creates a holistic system using improved health and nutrition to increase literacy. Rather than simply delivering food to schools and making demands on teachers, the program creates structures to manage various aspects of the program. McGovern-Dole sees that schools are prepared to receive, store, and prepare food and that they contribute to the program to ensure local ownership over achievement and outcomes. The program also realizes that if people are going to eat, they will also need a latrine to relieve themselves including the sanitation infrastructure that goes along with latrines. Program interventions are designed to minimize the impact of new activities that fuel demand on the local environment. Teachers and school directors praised the high quality, practical, and professional training that they received. Parents recognized positive changes in their children. While Counterpart has implemented the program as they have designed it, the organization’s management has also responded flexibly when activities have required adjustment. Since the mid-term evaluation, Counterpart has experimented with a needs-based approach to resource delivery where possible, including commodity distribution and re-allocation of energy efficient stoves. The program has also assessed how it can more effectively engage with communities to improve hygiene practices and school maintenance. The development of the energy efficient stove presented in the report and at mid-term is another example of the Counterpart’s flexibility and willingness to learn. As discussed at mid-term, Counterpart’s systematic vision includes exercising restraint in activity implementation where there are no options for sustaining the activity after the program ends. Thus, McGovern-Dole does not provide soap to schools for handwashing because schools could not provide the soap themselves if the program were to end. Similarly, toilets do not include water connections where communities would be unable to maintain them. Counterpart’s judgment in these program components is that it is not beneficial to the community to set up unsustainable assets even if the assets might produce short-term benefits. The evaluation team believes this judgment is correct, with the caveat that Counterpart should be careful not to use “unsustainability” as an excuse to not looking for creative solutions to difficult problems. The evaluation team also noted that Counterpart’s inclusion of innovative pilot projects in the overall program was effective. Pilots include the establishment of school farms, the provision of energy efficient stoves and the planting moringa plantations. Though the pilot projects achieved varying levels of success, McGovern-Dole learned a great deal from them that will assist future program. Importantly, pilot projects appropriately balanced the potential payoff of success with the opportunity cost of administering a project that failed. Design compared to needs of beneficiaries: As reported in the mid-term evaluation, the program design reflected the needs of beneficiaries in most important respects. As the projects results at mid-term prove, delivery of food matched with appropriate training does deliver the program’s intended results. The Senegalese government, which is used to responding to requests for assistance, has benefited from the demonstration of an organization taking the lead to organize communities and achieve goals. McGovern￾Dole has taught teachers and directors how to organize and leverage resources in their communities, and many have adopted those practices. Additionally, McGovern-Dole has advanced GOS reading objectives by providing trainings so teachers could more effectively teach the Government’s new reading curriculum. As reported at mid-term, there were a few activities that were not as relevant to stakeholders as others. These included: • Improvements in teacher attendance –It is unclear that teacher attendance in McGovern-Dole schools is a significant problem or that the programs identification of administrative burdens that the program identified are truly an obstacle to teacher attendance or effectiveness. 52 • School Construction – As noted in the mid-term evaluation, the school construction element of the program should be re-oriented to the specific needs of schools, ranging from patching up roofs to complete classroom construction. The evaluation team notes that the program’s construction budget was allocated by the mid-term evaluation, so no changes could be made in response to this recommendation. Interestingly, Counterpart’s experiment with allocating food based on school need revealed that some school needs are not a factor of community wealth or local food security. As discussed under Activity 8: School Feeding, many high food insecurity schools outperformed many low food-insecurity schools in providing student meals without the assistance of USDA. Some medium food insecurity schools provided school meals self-sufficiently despite the large student populations that those schools were feeding. Clearly, there is something that motivates some schools to perform at a high level when challenged to do so, while many others sink into complete inactivity. This factor appears to be independent of the relative wealth of the school’s community, or possibly has an inverse relationship to local wealth. Program relevance given the economic, cultural, gender, and political context: The program provides food and education in a region that is short on both assets. The program is in line with the national government’s health, nutrition, and education plans. Counterpart’s leadership has coordinated with government and political entities to ensure support for the project. In fact, the GOS has followed up on the success of McGovern-Dole school feeding with funding allocated for school feeding for schools with existing canteens. However as noted above in table 31, this is likely to provide about 13 percent of school feeding needs. Going forward the balance will have to come from community contributions including through school granaries and farms. Communities are in many cases willing to contribute labor, time and money for the school feeding program. However, school feeding self-sufficiency could have been accelerated with more emphasis from inception of the three-year program. The mid-term evaluation noted that one area where the project’s relevance requires further research is in its handling of gender issues. Counterpart has taken steps by initiating a study into gender issues that is still in draft form. At final evaluation, the evaluation team found that the way gender interacts with education for McGovern-Dole students is still unclear. As noted, the program’s reliance on volunteers exacerbates the unpaid care work burden for women in McGovern-Dole communities. The final evaluation supported the mid-term’s estimate that more girls (60%) than boys (40%) attend school. However, the imbalance is not true for all schools. Where the imbalance existed, explanations included that boys were required at home to help breed animals, that a national campaign to promote education for girls meant that boys felt it wasn’t as important for them, and that boys were more likely to get frustrated and drop out of school when they struggled with reading or math. Effectiveness and Performance Changes in student enrolment, attendance and drop-out rates since the start of the project: As mentioned in the SO1 section of this document, enrollment, and attendance have increased. Increasing student literacy outcomes: As noted in the SO1 section of this document, student literacy has significantly improved since baseline, and that students’ improvement between baseline and mid-term was much greater than between mid-term and final evaluation. 53 Improved student knowledge and behavior about nutrition, health, and sanitation: Please see SO2: Improve health and dietary practices results and Activity 5: Good health and Nutrition Practices. Teacher and mothers’ knowledge and behavior about nutrition, health, and sanitation: This document addresses teacher knowledge and behavior under Activity 5: Good health and Nutrition Practices. To the extent that the McGovern-Dole program addresses mother’s behavior, it does so in mother’s capacity as volunteer cooks or as part of SMC which this document addresses under Activity 6: Training in Food Preparation and Storage. Otherwise, the FFE program does not serve mothers as a direct beneficiary group. Impact of new infrastructure on school enrollment and attendance: The evaluation found limited information that supports the idea that school enrollment or attendance is influenced by new infrastructure. The evaluation team encountered one parent who said that she had stopped sending her child to the school because she thought the structure was unsafe, and teachers at two schools that said students were more likely to attend in inclement weather because of new facilities. Beneficiaries appreciated new facilities, and particularly connections to water, new classrooms, and useable latrines. However, at mid-term and final evaluation, when the evaluation team asked teachers and school directors if enrollment or attendance had changed, nearly all replied that it had increased and attributed the increase to the school feeding program. Beneficiaries attribute increased enrollment, increased attendance, and students spending more hours per day at school to the school feeding program. Improvement in reading fluency and comprehension skills due to new/renovated school infrastructure: Too few schools benefited completed or renovated construction at the time of the final evaluation to know whether infrastructure influenced reading achievement. Also, unless the construction was allocated randomly, too many other factors could have influenced student’s improvement in reading skills. Efficiency Could other implementation strategies have achieved more with the same resources? Counterpart’s overall implementation strategy is effective and efficient. Counterpart does an excellent job of ensuring that stakeholders at all levels are engaged and informed and that the program is implemented with a complete school feeding system strategy. Counterpart’s adjustment to allocating resources based on need shows that there are other strategies that might achieve more than the McGovern-Dole program in its original design. These strategies include: • Phase in sustainability measures earlier in the project: When Counterpart challenged schools to provide food without the assistance of USDA, many schools stepped up to the challenge and exceeded targets. If initiated earlier in the program, these schools may have served as models for other schools that had difficulty mobilizing. Additionally, working with schools and communities to establish and work toward canteen sustainability should start in year one. A crucial part of this is to work with schools to calculate the quantity of food they need to achieve sustainability and setting specific targets to achieve that sustainability within a reasonable time frame. • Taking a needs-based approach: In response to the mid-term evaluation, Counterpart began allocating resources according to need rather than quantity per student. The evaluation team believes this is a more effective means of achieving McGovern-Dole results. As Counterpart’s foray into needs-based activities demonstrated, not all needs are a function of relative wealth. Some low food insecurity schools would have benefited from more assistance motivating community engagement for example. Counterpart made important strides in the redirection of the program’s approach, and the evaluation team believes it should be carried into future phases of the program. 54 • An Alternative: Implementing fewer activities: The evaluation team believes that activities related to school feeding and teacher training make the greatest contribution to McGovern-Dole’s strategic objectives. An alternative strategy could be to focus on those activities that were related to the provision of food, the preparation of food, organizing communities in managing and sustain the feeding program, and training teachers in innovative reading, nutrition, and hygiene instructional methodologies. In light of the difficulties faced by the construction activities and other activities not directly related to school feeding or instruction, focusing resources on activities that appear to be the most effective may have allowed more schools to participate and permitted greater achievement in current schools. Focusing on the program’s core activities rather than school infrastructure,51 McGovern-Dole Senegal could have served approximately 100 additional schools and an estimated 17,600 students. 52 Sustainability Steps/actions required to realize full sustainability of activities beyond the life of project: At the highest level, ensuring sustainability requires improved transparency among USDA, the Senegalese government, and Counterpart International. Government officials at the ministry of school feeding said that their mission is to take over the school feeding program and make it sustainable. They are currently formulating policies and developing a monitoring system to make that happen. At the local level, regional ministries of education and school inspectors’ offices must take over the duties of Counterpart’s facilitators to continue to drive community engagement and continuous improvement among school directors, teachers, and students. A second step involves training directors to create a detailed budget of the school feeding program’s items and costs, and develop a plan for working with their communities to fund that budget. During final evaluation interviews, few directors could articulate the programs costs and describe how they could sustain school feeding in a realistic way. School feeding programs are funded currently through USDA donations, granaries, school farms (in some cases), and government allocations. In the absence of USDA donations, the other categories will have to expand to sustain the program, or new revenue-generating activities will have to be developed. As mentioned at mid-term, the workload that the program creates for school teachers, directors, and female volunteers threatens the program’s sustainability. The program should measure the work burden that the program produces for women and school directors, and identify means of reducing the work burden or otherwise compensating workers. This could involve encouraging men in communities to devote more time to volunteering in addition to making cash or food contributions. One possibility might involve encouraging men to provide cooking fuel to schools to reduce women’s burden of collecting firewood. Another might be for men to work more closely with school directors to more easily oversee aspects of the school feeding program, so it does not interfere with their other school management duties. A final important step involves planning for the sustainable management of the volunteer component of the program. Currently, volunteers run most aspects of the program. They do so in close coordination with Counterpart’s facilitators. Even with the facilitator’s regular oversight, school results in terms of food storage and preparation, acquisition of community contributions, and facilities management are less than perfect. There is a strong possibility that the Senegalese government entity that assumes ownership of the project will 51 School infrastructure refers to the construction of classrooms. It does not include latrines or water stations. 52 The estimate divides the school infrastructure budget for 2018 but the per school cost of the school feeding program, stoves and equipment for cooking and serving students, training in food preparation and storage, good health and nutrition practices initiatives, and warehousing costs. 55 not have Counterpart’s expertise in volunteer management and underestimates its importance to the program. As part of its transition plan, USDA should conduct a cost-benefit analysis for the Senegalese government to calculate whether maintaining the program as largely volunteer-run is worthwhile given the losses and quality control difficulties it implies, or if the government should offer communities and school directors incentives to assume the program’s duties. The challenges and successes of the project activities: As discussed in the sections covering SO1 and SO2, the program has made great strides toward improving student literacy and improving health and dietary practices. Reading scores show that literacy has improved greatly, and interviews with parents and school personnel reveal that children are healthier, cleaner, and more energetic. The program’s challenge is revealed by the diminishing returns it demonstrated in the period between mid￾term and final evaluation. While students reading has improved, a large portion of students still have not achieved grade level. 53 Future iterations of the program will have to develop new activities or partner with organizations specializing in literacy or hygiene initiatives to create significant beneficiary improvement. These initiatives could include maternal literacy, supply more books to schools and communities, or bringing other more advanced literacy and hygiene programs to communities. The challenges and successes of project activities are addressed throughout the Findings section of the document. Lessons for future sustainability: The strongest direct lesson is that to achieve sustainability, schools require a relatively large source of food or income, like from 1 to 2-hectare farms, increased granary inventory, and increased government assistance to achieve school feeding program sustainability. The details of this calculation are covered under Activity 9. How the program incorporates lessons learned from the Matam McGovern-Dole program exit strategy: The evaluation team understands that the Matam MGD program did not have an exit strategy; that it ended abruptly. When the program ended in Matam, the school feeding program also ended at many schools, though it continued at many more preschools than primary schools. Where the program was terminated, it was for several reasons, including54: • Lack of will in the communities. • Directors that preferred to focus on technical issues, like pedagogical training, rather than managing the school feeding program. • Disengagement by the government precipitated by the lack of notice that the program was coming to an end. Schools that had school feeding programs that persisted exhibited the following characteristics: • An innovative culture among teachers that worked to make the most of resources on hand. • Consistent cash contributions from a community of parents eager to continue the program. • Canteen management not dependent on Counterpart or the McGovern-Dole program. 53 57% of CM1 students, 52% of CE2, and 46% of CE1 students read below grade level. 54 Evaluation Report McGovern-Dole – Matam, June 2016. 56 The current McGovern-Dole program incorporated these lessons by focusing on stakeholder training and engagement, to varying levels of success. • The school created official positions for community members, such as SMC members, food program managers, school farm presidents, and other positions intended to build ownership and permanent engagement with the program. • McGovern-Dole built a culture of contributing and supporting the program in communities and challenged them to take responsibility for feeding and educating students. • The program built the capacity of school directors to engage with communities and raise funds and resources. • Multiple stakeholders in schools and the community that understand the school feeding program, including volunteer scheduling, inventory management, and dietary diversity. The extent to which project activities will continue with the absence of support from both USDA and Counterpart: Under a best-case scenario, schools could sustain 60 percent of the primary school feeding program and 100 percent of the preschool feeding program. This does not take into consideration the effort required to maintain engaged volunteers or perform the other functions that McGovern-Dole facilitators carry out. Other program activities will be more difficult to sustain. Regular pedagogical training, for example, will require initiative from school inspectors and a leader to maintain monthly teacher meetings and regular skills development. If Counterpart can identify that leader and work with him/her to plan for continued updates to the professional skills of teachers in the region, the activity has a reasonable chance of sustaining itself. This outcome requires clear communication about the programs progress and end date. The government’s School Health Office is interested in assuming ownership of the program’s health-related activities. McGovern-Dole deworming and hand washing campaigns are similar to campaigns that the office already runs. However, as in the case of the central government, time and clear transition plan are required to successfully hand the program over. 57 Lessons Learned This section details particular lessons learned during the project. Recommendations highlighting lessons learned appear in the next section. MGD provides an important opportunity for testing new ideas. Over the course of MGD Senegal, Counterpart tested several new and valuable ideas, including new energy efficient stove designs, new means of establishing school farms, and the planting of moringa trees to improve student health. Counterpart’s approach to managing these new ideas emphasized using them as a learning opportunity without exaggerating their successes or failures. Counterpart also kept the cost of trying new approaches as a reasonable part of the program’s budget, allowing them to promote ideas that succeeded while dropping ideas that failed. MGD’s complete design approach was crucial for its success. The MDG program’s central aim was to increase literacy and health outcomes by providing nutritious food in sufficient quantities to students in schools. The positive results that the program achieved were amplified by Counterpart’s strategy for program implementation, which creates locally sustainable assets around the complete school feeding and literacy program. This approach ensured that community structures were in place to take ownership of the food and manage its storage, safe handling and preparation. Trainings prevented new bouts of illness in student communities that can come with a new communal feeding system. Latrines were available so that students had a safe, clean place to relieve themselves after eating. This total systems approach was crucial to the program’s success. Gender issues require closer examination Among the goals of the evaluation was to “pay close attention to gender and social inclusion dynamics to ensure that considerations related to gender and social inclusion are factored into future programming.” The way the McGovern-Dole program’s initiatives interacts with gender and social inclusion is still unclear. There are two areas where further investigation is needed. First, with respect to students, different populations in the Saint Louis region of Senegal appeared to have different traditional norms with respect to boys and girls’ education. Across the region, more girls (60%) than boys (40%) attend school. However, this imbalance certainly didn’t represent all schools. As noted in this report, reasons given for why girls or boys may or may not attend school seemed to vary from community to community, with no obvious, single factor tipping the balance one way or the other. To understand student attendance and performance as a function of gender, a gender specific study is required. Secondly, the program’s reliance on volunteer labor creates an unpaid work burden for women that the program should address. Interviews with female volunteers indicated that women took responsibility for most of the new labor requirements that MGD puts on communities, and that very often this burden fell on a small number of volunteers. The MGD Program doesn’t have a clear understanding of the sufficiency of students’ diets While the program made an effort to measure the sufficiency of students’ diets, the measurement did not include a clear picture of what students were eating outside of school and did not emphasize the consumption of foods rich in micronutrients and other components of more traditional Minimum Acceptable Diet definitions. It is possible that taking a comprehensive survey of students’ diets is beyond the scope of the program, in which case a different estimate of SO2: Improved Health, should be adopted. 58 Student attendance improves because there is food at school Student enrollment and attendance at school improved. Interviews with stakeholders indicates that the single biggest reason for the improvement was the school feeding program. Teacher attendance may not be an issue The evaluations ToR asked that the evaluation team examine improvement in teacher attendance due to reductions in teachers administrative duties. The evaluation team did not find that school administrators, students, or parents found teacher attendance to be lacking, and teachers did not report that administrative duties were burdensome. School needs vary Understandably, the original design of the MGD program sought to maximize efficiency by applying a one￾size-fits all approach to schools. Accordingly, the amount of food provided to schools was the same per student, infrastructure projects focused on new construction, and capacity building projects focused on building the same community structures. The MGD program changed its approach to provide interventions for schools based on each school’s need rather than continue with the previous approach. MGD found that schools’ needs were not always obvious and that they were not always a function of relative wealth. Some relatively poorer schools provided more food to their school canteens, for example, because they had more developed community structures. Some relatively wealthier schools required more help than expected motivating community engagement. Sustainability initiatives should start early in the program Achieving sustainability requires that a number of complicated objectives are achieved. For example: • The government requires a budget allocation and policy in place for assuming ownership of the program. • School inspectors and other government staff have to take over roles previously performed by project staff, including the management of the volunteer program • School directors must have capacity to create a detailed budget of the school feeding program’s items and costs, and develop a plan for working with their communities to fund that budget. • To achieve school feeding sustainability, schools require a relatively large and diverse source of food or income, including from 1 to 2-hectare farms, increased granary inventory, and increased government assistance. • Communities have to assume responsibility for monitoring school feeding programs and providing food and resources when other sources fall short. Achieving the objectives takes time. The program would benefit from drawing stakeholders’ attention to them as early in the program as possible. 59 Recommendations As the prior sections of this document have shown, McGovern-Dole is a successful project. It has improved the literacy of primary school students, improved knowledge and practices of hygiene and nutrition among its stakeholders, built the capacity of schools and communities to sustain school feeding programs, and brightened the future prospects for tens of thousands of children. It has also built the capacity of dozens of school communities to support and manage school feeding programs. That said, every program, no matter how successful, can be improved. In this section, the report recommends the following 8 priority actions that the program should incorporate into its next phase. Continue the program: USDA should continue the program in Saint Louis for another 2-3 years. Ending the program now would be pulling food from thousands of kids who have just started on an educational path equipped with the energy that USDA commodities provide. The next phase of the program should focus on building school, community and governmental capacity to sustain and scale up the school feeding program. Steps to achieving this include: • Convening a workshop of existing school directors and community leaders to identify and consolidate school feeding program best practices with regard to sustainability. • Focusing on self-sufficiency from the outset of the program. This includes investing in school farm and granaries and formation and capacity building for school feeding boards (see below). • Have a steeper annual phase out of food provided by the program, forcing schools to learn lessons about school feeding sustainability with plenty of time left in the program. For example, a new program could provide new schools with 100, 67 and 33 percent of school feeding requirements in years 1, 2 and 3 respectively. For existing schools, the phaseout could look more like 75, 50 and 25 percent respectively. • Not only would a more rapid phasing out of food accelerate sustainability, it would reduce costs per school, allowing for expanded outreach. Focus school selection on high and medium food insecurity schools: The school selection process and reorientation of need-based food provision worked well. However, allowing the regional Ministry of Education to have final say over school selection may have led to a higher number of relatively wealthy schools participating in the program that otherwise would have been the case. In future McGovern-Dole programs, McGovern-Dole should use the process that Counterpart developed to categorize schools in the school selection process. School selection should minimize the number of low food insecurity schools that the program serves. McGovern-Dole in these schools is more likely to replace meals rather than provide additional meals and benefit less from the program than the larger medium insecurity schools or, the needier high insecurity schools. Work with school directors to plan for maximizing each school’s resources: McGovern-Dole should work with directors to develop school feeding strategies and schedules for different levels of food availability. These strategies should be aimed at maximizing the nutritional and educational benefits of school feeding within the resources that schools can acquire. For example, preschools with only 60 percent of the recommended school feeding budget may serve breakfasts on Monday, Wednesday, and Friday. Or a primary school only able to meet 75 percent of the MGD feeding recommendations may choose to serve lunch but not breakfast on Tuesdays and Thursdays. More difficult or important parts of the curriculum may be scheduled on these days when school attendance and student concentration may be highest. 60 Support establishment of School Feeding boards at school and regional levels: School feeding programs need to be run more professionally if they are to become more sustainable. At the school level, McGovern-Dole Senegal should consider forming a school feeding board to guide decision making, resource allocation, and school coordination. This will also help to reduce the burden on directors that will likely increase with more self-sufficient school feeding programs. For example, the board of directors may decide that for the school farm to expand, they need to hire the farm president as a full-time manager. They may also help set school feeding schedules or assist schools in sharing assets and resources. Members of the school level feeding board should include the director, the school farm president, the lead cook, parents and a community leader. A regional school feeding board should also be formed for St Louis. The purpose of this would be to identify and disseminate best practices in building self-reliant school feeding programs and to advocate for policy and resource support for school feeding within the government and private sector. Members would include the MoE and MoA at departmental and regional levels, food and agriculture companies, as well select members of school level feeding boards. Identify ways to overcome school farms’ major constraints: School farms potential is limited by four major constraints: • Insufficient access to finance • Labor shortages • Lack of sufficient equipment and inputs • Lack of a model farm to learn from School directors and community members are already contributing a good deal by taking on extra work and volunteering to support extracurricular activities. The next phase or iteration of the program should work to help school farms to overcome these constraints so that farms can reach their potential in contributing to a sustainable school feeding program. Part of this activity could involve partnering with the MoA to create school farm demonstration plots. These plots would introduce improved technologies and approaches to the community and increase the productivity of school farms. In the evaluation team’s interview, The Ministry of Agriculture expressed interest in pursuing this idea. Overcome the diminishing marginal utility of the program: To overcome the diminishing utility of the program, USDA could look at second phase initiatives that may have a large impact on primary school literacy and improved health practices. Potential second phase initiatives to improve literacy include improving the reading skills of mothers with preschool children, developing reading events or competitions, establishing libraries or otherwise increasing access to a wide variety of books, developing summer learning and reading initiatives, and generally providing more resources for reading and raising the profile of its importance. For health and hygiene practices, the program could promote improved household food processing, storage and preservation to retain nutrient value, shelf life, and food safety; conduct community-level cooking training and demonstrations and advocating for improved household level hand-washing practices. Use random spot checks as well as school data to track increased attendance: The system currently does a good job of collecting school attendance records and entering the information into a central database, where the team can analyze the data and uncover trends. However, particularly in the case of student attendance, the high reported rates of attendance and the insufficient administrative staff at schools makes the data seem. suspect. 61 To truly understand the way the program influences school attendance, McGovern-Dole should have an independent research plan for studying the topic from the beginning of the program. One approach could be use random spot checks to estimate a difference between reported attendance rates and actual attendance rates. To conduct spot checks, Counterpart’s Program Manager would collect student enrollment data for each school at the beginning of the school year. Then, the Program Manager would randomly choose days for school project officers to take a student and headcount at each school. The Program Manager would choose at least one day each quarter of the school year, and would take into consideration fluctuations due to seasonal attendance and the time of day. On spot check days, school project officers would take a head count of students for each grade in the school. Comparing reported attendance against attendance taken during spot checks would show whether or not official school attendance records were accurate. If they are not accurate, the spot check head counts divided by total class enrollment would give a more accurate estimate of attendance. Over the course of a three to five-year program, USDA could analyze the spot check data to see if the MGD project produced any attendance trends. Research whether infrastructure construction activities provide sufficient value: Interviews and surveys revealed that improvements in nutrition and health initiatives were primarily responsible for improvements in reading. Infrastructure projects were less successful and created unfulfillable expectations among some project beneficiaries. School infrastructure projects were also the single biggest direct cost line item in the programs budget. The funds allocated to school infrastructure projects could have funded school feeding programs for over 100 new schools and over 17,000 additional students. Schools certainly require repair and upgrading. USDA could use the data it has on hand from previous McGovern-Dole programs to see if there is a correlation between schools that benefit from infrastructure projects and improved reading scores. It could also allocate funds for school maintenance and upgrade rather than new construction. These savings could be used to serve more schools through other project activies. 62 Annex 1: Document Review List 1. CPI Performance Indicators Attachment E 2. CPI Performance ME Plan Revised 12/09/15 3. CPI Plan of Operation Attachment A 4. CPI MGD Midterm Report - Updates to USDA April 2018 5. CPI MGD Program Spending Report 6. CPI Reports on health and nutrition activities 7. CPI Reports on literacy training and capacity building activities 8. CPI School Categorization document 9. CPI Students, teachers, schools, and other monitoring and evaluation system databases 10. FFE Sustainability Evaluation Report Sept 2016- Matam 11. Food For Progress and McGovern-Dole Indicators and Definitions 12. FFE Semiannual Performance Report April 2017 - September 2017 13. FFE Semiannual Performance Report April 2016 - September 2016 14. FFE Semiannual Performance Report April 2015 - September 2015 15. FFE Semiannual Performance Report Oct 2014 - March 2015 16. FFE Semiannual Performance Report Oct 2015 - March 2016 17. FFE Semiannual Performance Report Oct 2016 - March 2017 18. FFE Semiannual Performance Report Oct 2017 - March 2018 19. Food For Education Activity Management guides and protocols. 20. Food for Education Annual Workplans 21. Fuchs, Marek. "The Reliability of Children’s Survey Responses: The Impact of Cognitive Functioning on Respondent Behavior." Symposium 2008: Data Collection: Challenges, Achievements and New Directions (2008): 1-9. Web. 22. Kampman, Haile et. al. “How Senegal created an enabling environment for nutrition: A story of change,” Global Food Security, pgs 57 – 65, 2017. 23. Memos of Understanding between CPI and Communities 24. MGD Indicator Framework 25. Monitoring and Evaluation Databases and Data Collection Tools 26. Performance ME Plan Revised 12/09/15 27. prnewswire.com/news-releases/global-handwashing-day-focuses-on-need-for-universal-hand￾hygiene-300159521.html 28. Raw Baseline Data 29. School Canteen Guide 30. School Hygiene and Nutrition Guide 31. School Selection Criteria Document 32. Senegal School Curriculum Guides 33. Social Progress Index Senegal Report 34. Staff Field Visit and Training Reports 35. World Bank’s Databank “World Development Indicators” database. 36. World Food Program “Senegal Transitional Interim Country Strategic Plan 2018.” 63 Annex 2: Data Collection Tools Student Survey 1. Start time 2. End time 3. Date of interview 4. Device ID 5. subscriber id 6. username 7. Telephone number 8. SIM serial number 9. Introduction: "You have been selected to participate in a survey on health, nutrition and education as part of the School Canteen project. Your participation in this study is entirely voluntary. You are under no obligation to participate. You have the right to refuse to answer questions and to withdraw from the study at any time. If you agree, please answer all questions as honestly as possible. If you are unable to answer one of the questions, you can ignore the question. All your answers are strictly confidential." 10. Please select the interviewer name from the list bellow 11. Commune 12. Name of school 13. Select preschool or elementary level 14. Type de structure 15. "Ask for the respondent's consent” If the person agrees to participate, go with the interview." 16. What is your first name 17. What is your last name? 18. How old are you? 19. Gender? 20. Grade? 21. On school days, do you usually eat breakfast? a. Yes b. No 22. Why do not usually eat at breakfast time? a. I do not have enough food at home b. I do not have enough time to go home c. I do not like food in the canteen d. I do not have money to buy food e. other (Specify) 23. What do you usually eat for breakfast? a. Tubers, or grains (such as millet, rice, maize, sorghum), or roots (such as cassava) 64 b. Nuts or legumes (such as beans like niebe) c. Milk, yogurt or cheese d. Meat, fish, poultry or liver/organ meats e. Eggs f. Vegetable red palm oil, or fruits and vegetables including pumpkin, carrot, squash, sweet potato, dark green leafy vegetables, ripe mango, cantaloupe melon, apricot, ripe papaya, peach, red peppers, moringa leaves, bean leaves g. Fruits and vegetables, such as onion, eggplant, watermelon, oranges, green peppers, cabbage, tomoatoes, dates h. Other 24. Where do you usually get your food from at breakfast time? a. Home b. School canteen c. Snack vender d. Other (Specify) 25. Do you usually eat something at lunch time while you're in school? a. Yes b. No 26. Why don't you usually eat something at lunch time? a. I don't have enough time to eat b. I don't have enough time to go home c. I don't like the food in the canteen d. I do not have food e. Other 27. What do you usually eat at lunch time? a. tubers, or grains (such as millet, rice, maize, sorghum), or roots (such as cassava) b. nuts or legumes (such as beans like niebe) c. milk, yogurt or cheese d. Did you eat meat, fish, poultry or liver/organ meats e. Eggs f. vegetable red palm oil, or fruits and vegetables including pumpkin, carrot, squash, sweet potato, dark green leafy vegetables, ripe mango, cantaloupe melon, apricot, ripe papaya, peach, red peppers, moringa leaves, bean leaves g. fruits and vegetables, such as onion, eggplant, watermelon, oranges, green peppers, cabbage, tomoatoes, dates h. Other 28. Where do you usually eat at lunch time? a. Home b. School canteen c. Other (Specify) 29. Do you usually eat any other time at school besides breakfast and lunch? a. Yes b. No 30. What time of day do you eat besides breakfast and lunch when you're at school? a. after school starts but before lunch b. after lunch but before leaving school c. Other 31. What do you eat at this time normally? a. tubers, or grains (such as millet, rice, maize, sorghum), or roots (such as cassava) 65 b. nuts or legumes (such as beans like niebe) c. milk, yogurt or cheese d. Did you eat meat, fish, poultry or liver/organ meats e. Eggs f. vegetable red palm oil, or fruits and vegetables including pumpkin, carrot, squash, sweet potato, dark green leafy vegetables, ripe mango, cantaloupe melon, apricot, ripe papaya, peach, red peppers, moringa leaves, bean leaves g. fruits and vegetables, such as onion, eggplant, watermelon, oranges, green peppers, cabbage, tomoatoes, dates h. Other 32. What is your opinion of the food served at the school canteen? a. Big Smiley Face b. Smiley Face c. Frown Face d. Big Frown face 33. Some pupils are hungry during the school day, other pupils are not hungry during the school day. Are you ever hungry during the school day? a. Yes b. No 34. How often are you hungry during the school day? a. I'm always hungry during the school day b. I am often hungry during the school day c. I'm hungry about half the time d. I'm hungry from time to time e. do not know Now, I'm going to ask you some questions about health and cleanliness 35. Intestinal worms are parasites that live in people's bodies and make them sick. Can you think of any ways to prevent intestinal worms? a. Avoid walking barefoot (wear shoes) b. Don't bathe or swim in stagnant water c. Eat meat that's cooked well done (especially beef and pork) d. Avoid direct contact with contaminated water, but if necessary wear boots and gloves e. Wash hands with water that's safe to drink and soap before preparing food, serving food or eating f. Wash hands with water that's safe to drink and soap after using the toilet g. Protect food from flies, cockroaches, and dust h. Keep food in a pantry, refrigerator, or place that's clean and well-ventilated i. Other 36. Some people wash their hands a lot, others almost never wash their hands Thinking about your day so far today, have you washed your hands today? a. Yes b. No 37. When did you wash your hands today? a. Before eating b. Before touching or preparing food c. Before giving food to someone else 66 d. When you have earth on your hands e. After touching something dirty f. After touching an animal g. After using the latrine h. After changing a layer i. Other (Specify) 38. What did you use to wash your hands? a. Water b. Ashes c. Sand d. Soap e. Other 39. In your opinion, when do you think a person should wash their hands? a. Before eating b. Before touching or preparing food c. Before giving food to someone else d. When you have earth on your hands e. After touching something dirty f. After touching an animal g. After using the latrine h. After changing a layer i. Other (Specify) 40. Now I’m going to ask you a few questions about healthy food. In your opinion, what food that can help you grow? a. Meat b. Fish c. Eggs d. Cheese e. Milk f. Nuts g. Other 41. What food can give you energy? a. Butter b. Oil c. Sugar d. Bread e. Rice f. Millet g. Other 42. What foods can protect you from diseases? a. Oranges b. Apples c. Mangoes 67 d. Tomatoes e. Carrots f. Watermelon g. Melon h. Cabbage i. Leaves of bissap j. Beans k. Moringa l. Other 43. What do you take when you are thirsty? a. Water b. Other 44. What is the reading level of the learner? Thank you so much for answering my questions. Please take the GPS coordinates of the interview location. Teacher Survey 1. Start time 2. End time 3. Date of interview 4. Device ID 5. subscriber id 6. username 7. Telephone number 8. SIM serial number You have been selected to participate in a survey on health, nutrition and education as part of the School Canteen project. Your participation in this study is entirely voluntary. You are under no obligation to participate. You have the right to refuse to answer questions and to withdraw from the study at any time. If you agree, please answer all questions as honestly as possible. If you are unable to answer one of the questions, you can ignore the question. All your answers are strictly confidential. Please select the interviewer name from the list below INTERVIEWER: PLEASE TYPE IN YOUR NAME 9. Do you agree to participate in this survey? a. Yes b. No 10. Department 11. Preschool or Primary school? 12. What is your name? 13. What is your last name? 14. What is your age? 68 15. Gender 16. What is the highest level of education you have achieved? 17. Have you completed a teaching certification or formal training in teaching children? a. Yes b. No c. Don’t Know 18. Did you receive literacy training this school year? a. Yes b. No c. Don’t Know 19. Were you consulted or considered before the selection of training modules? a. Yes b. No c. Don’t Know 20. After the training was completed, were you given the opportunity to provide feedback? a. Yes b. No c. Don’t Know 21. What grades do you mostly teach? a. CE1 b. CE2 c. CM1 d. petite section e. moyenne section f. grande section g. Multigrade class h. other 22. How long have you been working as a teacher? 23. How many pupils are in your main class? 24. What is the size of the CE1 25. What is the size of the class of CE2 26. What is the size of the CM1 class 27. What is your professional status? a. Full Time b. Part time (50% to 90% of full hours) c. Part time (less than 50% of full hours) 28. Do you regularly talk about nutrition in your class? a. Yes, already have b. Yes, I’m planning to c. No 29. Did you teach or plan to teach about nutrition in your classes this week, which means between Monday and Saturday? a. Yes b. No 30. Have you received training on teaching nutrition in the last three months? a. Yes b. No 31. Do you think the training you received provided you with new tools or techniques to teach nutrition? a. Yes 69 b. No 32. How often do you use the new tools or skills to teach nutrition? a. Always b. Often c. Sometimes d. Never 33. Do you usually teach hygiene in your class? a. Yes, already b. Yes, planning c. No, does not and does not plan 34. Did you teach or plan to teach about hygiene in your classes this week, which means between Monday and Saturday? a. Yes b. No 35. Have you received training on hygiene education in the last three months? a. Yes b. No 36. In your opinion, did the training you receive provide you with new tools or techniques to teach hygiene? a. Yes b. No 37. How often do you use the new tools or hygiene skills? a. Always b. Often c. Sometimes d. Never 38. Q27 Have you attended a course in pedagogy supported by the government in the last three months? a. Yes b. No 39. Q28 Do you think the training she has provided you with new tools or techniques to teach? a. Yes b. No 40. Q29 How often do you use the new tools and skills in teaching? a. Always b. Often c. Sometimes d. Never 41. In your opinion, what foods can help children grow? a. Meat b. Fish c. Eggs d. Cheese e. Milk f. Nuts g. Other 42. What foods can give you energy? a. Butter b. Oil c. Sugar 70 d. Bread e. Rice f. Millet g. Other 43. What foods can protect you from diseases? a. Oranges b. Apples c. Mangoes d. Tomatoes e. Carrots f. Watermelon g. Melon h. Cabbage i. Leaves of bissap j. Beans k. Moringa l. Other 44. Have you washed your hands today? a. Yes b. No 45. When did you wash your hands today? a. Before eating b. Before touching or preparing food c. Before giving food to someone else d. When you have earth on your hands e. After touching something dirty f. After touching an animal g. After using the latrine h. After changing a layer i. Other (Specify) 46. What did you use to wash your hands? a. Water b. Ashes c. Sand d. Soap e. Other 47. In your opinion, when do you think a person should wash their hands? a. Before eating b. Before touching or preparing food c. Before giving food to someone else d. When you have earth on your hands e. After touching something dirty f. After touching an animal g. After using the latrine h. After changing a layer i. Other (Specify) 71 Thank you very much for answering my questions. Director Survey 1. Start time 2. End time 3. Date of interview 4. Device ID 5. subscriber id 6. username 7. Telephone number 8. SIM serial number You have been selected to participate in a survey on health, nutrition and education as part of the School Canteen project. Your participation in this study is entirely voluntary. You are under no obligation to participate. You have the right to refuse to answer questions and to withdraw from the study at any time. If you agree, please answer all questions as honestly as possible. If you are unable to answer one of the questions, you can ignore the question. All your answers are strictly confidential. 9. Please select the interviewer name from the list bellow 10. INTERVIEWER: PLEASE TYPE IN YOUR NAME 11. Ask for the respondent's consent. If the person agrees to participate, go with the interview. 12. Select school 13. Select preschool or elementary level 14. What's your first name? 15. What is your last name? I would like to ask you some questions about your background 16. How old are you? 17. Gender 18. What is the highest level of education you have completed? 19. Have you ever completed a teacher certification program or received formal training in teaching children? 20. Have you received training on teaching literacy since the school year started, that is since last October? 21. Do you teach at this school ? 22. What grade do you teach? 23. How long have you been the principal of this school? Since you are a school principal I would like to ask you a few more questions. Let's start with how the school stores food and prepares meals… 24. In your opinion, what is a way to ensure that the storage facility at the school is safe to store the food? a. Airy b. Adequate size c. Clean d. afe e. Accessible f. Walls and floor cemented 72 g. Distance between walls and food (30cm) h. Don’t Know 25. What is a way to maintain the storage facility at the school? a. Clean inside and out b. Sweep floor every week c. Clean roof, walls, and piles once per week d. Completely clean outside warehouse once a month e. Remove weeds and waste that attracts rats and insects f. Other g. Don't Know h. Other, please specify 26. What is a way to track the stock/food in your storage facility? a. Compare the inventoried quantity with the quantity listed on the stock sheet b. Note the date of the inventory and sign your initials c. Compare the physical stock to the theoretical stock d. Report in case of discrepancy the reasons for the loss e. Fill in the inventory sheet f. Make a loss PV 27. What is a way that food can be safely prepared? a. Wash hands with clean water and soap b. Protect food from flies, cockroaches, dust c. Respect the expiry dates of foods d. Avoid foods with mold e. Avoid breaking the cold chain f. Any food consumed raw (fruits and vegetables) must be washed cleanly before consumption g. Do not Know Great! Now, I'd like to ask you a few questions about the community farm in this community… 28. From your knowledge, does the community have a community farm? a. Yes b. No 29. Who manages the community farm? a. Community Farm Management Committee b. CGE c. APE d. Principal e. Others f. Don't Know 30. Does the community support the community farm? 73 a. Yes b. No 31. What are some of the ways the community supports the farm? 32. Has the received any of the following? a. Agricultural training b. Small scale farm equipment c. Other d. Don’t Know 33. Has the community farm produced any yields so far? a. Yes b. No 34. Are the yields used to make school meals at the school? a. Yes b. No 35. Are the yields sold at local markets? a. Yes b. No 36. How are the proceeds from the sale used? a. Support management of school canteens b. Support school feeding c. Pay school lunch cooks d. Pay members <> e. Other f. Don't Know 37. What else are the yields used for? Thank you! Now, I would like to ask you a few question about Moringa… 38. Have you heard of moringa? a. Yes b. No 39. Can you give me an example of a benefit of moringa? a. Provides vitamin C or helps with growth and repair of tissues and healing wounds b. Provides protein or is needed for growth and adequate nutrition c. Provides calcium or is needed for bone growth d. Provides vitamin A or helps improve immune system function e. Provides iron to treat "tired blood" (anemia) f. Other g. Don’t Know 40. Does your school grow moringa? a. Yes b. No 41. Does your school use moringa to prepare the school meals? a. Yes b. No 42. How often do you use moringa for school feeding during the week, that is Monday through Friday? a. Always b. Very Often 74 c. Sometimes d. Rarely e. Never f. Don’t Know 43. Q33 Why does your school not use moringa for school feeding? a. Moringa is too expensive b. Do not have access to moringa c. Do not like the taste of moringa d. Do not think it is beneficial e. Other 44. Q34 Does your school have a classroom built by Counterpart? a. Yes b. No 45. Q35 Is the classroom in good condition and well equipped? a. Yes b. No 46. Q36 Is the classroom used regularly? a. Yes b. No 47. Q37 How often does the community contribute to the maintenance of the classroom? a. Always b. Very Often c. Sometimes d. Rarely e. Never f. Don’t Know Thank you so much for answering my questions. Observation Checklist 1. Start time 2. End time 3. Date of interview 4. Device ID 5. subscriber id 6. username 7. Telephone number 8. SIM serial number INTERVIEWER: PLEASE TYPE IN YOUR NAM 75 Q1 Select Department Q2 Select preschool or elementary level Q3a Dagana primary school Q3b Pété primary school Q3c Podor primary school Q3d Dagana Pre-school Q3e Pété Pre-school Q3f Podor Pre-school Q3g Saint Louis Pre-school # Latrines and hand washing Q5 Does the school have a latrine? Q6 Is the latrine connected to water? Q7 Is the water connection working? Q8 How clean is the latrine? Q9 Is the latrine well maintained in terms its structure? Q10 Is it available to boys and girls? Q11 Is it equipped with a sink for washing hands? Q12 Is the sink area clean? Q13 Is there soap? Q14 Is there a clean towel or other material for drying? Q15 Is there evidence that the school and parents contribute to latrine management? What is the evidence? # Food storage Q16 Is the storage area clean? Q17 Is the roof covered? Q18 Is the establishment secure? Q19 Are there signs of damage from rodents or other pests? Q20 Which of the following is offered in the canteen? Other, specify # Cooking area Q21 Is the cooking area generally clean? 76 Q22 Is there soap or detergent? Q23 Are there adequate utensils for cooking and eating? Q24 Is there an energy efficient stove? Q25 Is there a way to wash your hands before eating? # observations Interview Guide Introduction for the Interviewer ISG is conducting research on behalf of Counterpart International. The objective of the research is to analyse the impact that Counterpart International’s Food For Education program has had on the beneficiaries and communities where it has been implemented. ISG will conduct its research via a survey, focus group discussions, and key informant interviews. This interview guide serves as a manual for data collectors conducting the key informant interview portion of the research. The interviews will cover five specific areas: 1. The degree to which FFE has met the needs of students, schools, and communities. 2. The effectiveness of FFE’s activities aimed at improving nutrition, hygiene practices, literacy, school infrastructure and community involvement in schools. 3. FFE’s success at increasing student enrolment and attendance as well as teacher attendance. 4. The relevance of FFE’s interventions to achieving its intended objectives and providing for the needs of targeted beneficiaries. 5. Examples of improvements FFE could make to activities and services to improve community satisfaction and improve student achievement. ISG’s researchers will conduct face to face interviews with FFE beneficiaries. While this guide is intended to assist the interviewer in setting key areas for research and orienting the discussions, the interview should happen as a conversation, allowing the interviewer to ask follow up questions that may lead to discovering new and important information. NOTES FOR THE INTERVIEWER • You should ensure that you cover all the major issues noted in this guide. However, some questions may elicit long discussions and others short answers depending on the relevance of individual topics for discussion to individual respondents. Exploring the most relevant areas (and others that may be important to the respondent, but not necessarily covered in this guide) is more important than covering every question in the guide. • FFE’s activities operate in a dynamic and complex environment. It is likely that there are key issues that the author of this guide did not anticipate. The interviewer should feel free to ask questions that the interviewer feels are fruitful whether or not they are included in the guide. • Please take notes in a separate document. • Make sure to record your notes at the end of each day. Too much time between the interview and recording notes can lead to loss of salient information that will have a negative impact on the rest of the research. 77 INTERVIEW QUESTIONS Introduction Hello! Thank you for participating in this interview. My name is _________ and I am working on a project for International Solutions Group, a research organization based in the United States. ISG is conducting research on behalf of Counterpart International. The purpose of our research is to assess the effectiveness of Counterpart International’s Food for Education program. Counterpart is interested in understanding if the program has achieved its intended goals. I will ask you questions about your life and experience working with Counterpart and the FFE program. The interview today will last about half an hour. During the interview, I’ll take notes so I can remember what you said. I want to mention a few important points before we start. • There are no ‘right’ or ‘wrong’ answers to any of the questions. We are interested in hearing your opinions, whatever they are. • Your participation in this interview is voluntary. If you don’t want to answer any question, you don’t have to. The responses that you provide are confidential: we won’t mention your name in any report we write and we will never connect your name with anything you say. Your responses will be combined with responses from other people. Do you have any questions before we begin? Now that I’ve told you about the topics we’ll discuss and answered your questions, are you comfortable proceeding with the interview? 1. IF NO: ASK IF YOU COULD PROVIDE ANY OTHER INFORMATION THAT WOULD MAKE THE RESPONDENT MORE COMFORTABLE. IF RESPONDENT PREFERS NOT TO CONTINUE, TERMINATE INTERVIEW. 2. IF YES: CONTINUE TO THE NEXT SECTION. DIRECTOR INTERVIEWS Respondent Background 1. First, I’d like to understand your history working at the school where you currently work. 2. Did you ever teach? FFE Program Responsiveness 3. The FFE program has several components – school feeding, infrastructure, hygiene, community farm construction, training, and others – which has been the most successful at this school? Why? 4. Did the food the program provided arrive on time? Did you have any trouble with logistics? 5. How does your school use Moringa? 6. Have the project’s de-worming and vitamin A interventions worked at your school? How do you know? 7. Which has faced the greatest challenges? Why? Teacher and Student Attendance 8. Some schools have problems with teach attendance. Has the FFE program had any impact on teacher attendance at your school? How did the change happen? 78 9. Also, some communities see obstacles to students enrolling in or attending school. Have any aspects of the FFE program changed student enrolment or attendance? Which and how? Community Involvement 10. How involved is the community in this school in terms of contributing to the community garden, infrastructure, or other activities? Where does the community contribute the most? How? 11. What’s the biggest challenge in engaging the community? 12. Who contributes most from the community? Training 13. Have you received any training through the program? What kind of training? 14. What was the most important thing you took away from the training? 15. Has it changed the way you work at all? If so, how? Sustainability 16. How does the school ensure proper maintenance of the program and/or facilities? 17. How will the program sustain itself in the future? Conclusion 18. Do you have any final comments or any questions? TEACHER INTERVIEWS Respondent Background 1. First, I’d like to understand your history teaching at the school where you currently work. How long have you been a teacher? 2. How long have you worked at your current school? 3. What grade do you teach? School Enrolment and Attendance 4. One of the purposes of the FFE project is to increase student enrolment and attendance. In your opinion, why do some students not enrol or attend school? Why do some students face obstacles to attending or enrolling in school? 5. Are there some obstacles that boys face that girls don’t? 6. Are there some obstacles that girls face that boys doing? 7. Has the FFE program helped improve enrolment? Which parts of it, such as school feeding or access to latrines has contributed the most? 8. What could it do to further improve enrolment and attendance? 9. Sometimes, teachers also face obstacles in attending school. Have you noticed a change in teacher attendance? Why did the change occur? Hygiene and Nutrition Practices 10. The project has carried out a number of health and hygiene projects, such as building latrines and delivering trainings. Have you noticed a change in student’s hygiene or nutrition practices since the program began? Why did the changes occur? 11. Teachers were also trained in nutrition and hygiene practices. Has their behaviour changed? How? Why did it occur? 12. Have you received any training in hygiene or nutrition? What was the most important skill or piece of knowledge you took away from the training? Were you satisfied with the training? 13. Have the project’s de-worming and vitamin A interventions worked at your school? How do you know? 79 14. Does the school utilize the School Nutrition and Health Guide? How? General Questions 15. What do you like most about teaching at your school? 16. Are there any difficulties you face in your daily duties that the FFE program could address? Conclusion 17. Do you have any final comments or any questions? 80 PTA MEMBERS/Parents Respondent Background 1. First, I’d like to understand your history in this community. Do you have kids at this school? 2. How long have you served on the PTA? School Enrolment and Attendance 3. One of the purposes of the FFE project is to increase student enrolment and attendance. In your opinion, why do some students not enrol or attend school? Why do some students face obstacles to attending or enrolling in school? 4. Are there some obstacles that boys face that girls don’t? 5. Are there some obstacles that girls face that boys doing? 6. Has the FFE program helped improve enrolment? Which parts of it, such as school feeding or access to latrines has contributed the most? 7. What could it do to further improve enrolment and attendance? 8. Sometimes, teachers also face obstacles in attending school. Have you noticed a change in teacher attendance? Why did the change occur? Community Involvement 9. Does this school have a community farm? Is the farm important to the school? Why? 10. Does the community participate in the farm? How does it contribute? 11. Are there any other ways community members contribute to the school? How? Does the community contribute to school infrastructure? 12. What does the community think of the Moringa plantations? 13. [if the school has a farm] Does the community like the school farm? Is it effective? How will they maintain it in the future? Training 14. Did you receive any training as part of the program? What was the best part of the training? How did the training change your knowledge or behavior? 15. What other kinds of training would you like to receive? General 16. In your opinion, does the FFE program respond to the school and community’s needs? What is the most successful thing the project has done? What has been its greatest challenge? Conclusion 17. Do you have any final comments or any questions? COOKS Background 1. How long have you cooked at this school? 81 Nutrition and Hygiene 2. The FFE program seeks to improve nutrition and hygiene at schools. Did you receive any training in nutrition or hygiene? What did you learn? What was the best part of the training? 3. Has the FFE program changed the way you work? How? 4. What is the best thing about the FFE program? 5. What is the most difficult thing about the program? Food delivery and storage 6. How does the process for food delivery work? When does it come? Where does it come from? 7. How do you store the food when it arrives? Is it easy to store? How do you know proper storage practices? 8. Are there ever any problems with delivery? What are they? Stoves 9. Do you use the cookstoves that the program provided? 10. How have the cookstoves changed the way you provide food for the students? Conclusion 11. Do you have any final comments or any questions? 82 COMMUNITY FARM MANAGERS Respondent Background 1. First, I’d like to understand your history in this community. Do you have kids at this school? 2. How long have you managed the farm? Community Farm Management 3. How does the farm work? Who provides the labor? Who provides seeds and other inputs? 4. What happens to the food the farm produces? How much goes to the school? Is any sold? Who markets the produce? How is the profit distributed? 5. How does the farm relate to the community? Are there strengths and weaknesses in community ownership? 6. What has been the farm’s biggest success? 7. What are the farm’s challenges? 8. What could FFE do to make the program more successful? Conclusion 9. Do you have any final comments or any questions? Project Staff55 Background 1. Name 2. Position 3. How long have you worked on this project? Implementation 4. What do you think is the most successful aspect of the project? What contributes to its success? Has the management of that aspect changed in any way? 5. How do you ensure that the project’s activities meet the needs of its beneficiaries? How were schools, communities, and beneficiaries selected? 6. What aspect of the project has struggled the most? Why? Efficiency 7. What are the best parts of how the project is managed? 8. Do you feel like the project has sufficient resources? 9. Is there anything you would have done differently in retrospect? Sustainability 10. How does the project work toward graduating its communities? 11. Where is the project most sustainable? Least sustainable? Conclusion 12. Do you have any other comments or thoughts you’d like to share? 55 Questions asked will differ according to the position of the staff member interviewed. 83 Government Officials56 1. Name, position, department. 2. Why is the Food For Education important to [official’s department or scope of work]? 3. What have been the program’s most important successes in your opinion? 4. Every project, no matter how successful, has areas where it can improve. In your opinion, what would have improved the design, implementation, or other aspect of the Food For Education project? 5. What’s your vision for the future of this project? How likely is it that the vision will be achieved? What resources will be required to achieve the vision? 6. If the project were starting over today, what advice would you give the new management team? 7. Do you have any other comments or thoughts you’d like to share? Focus Group Guide Introduction for the focus group leader ISG is conducting research on behalf of Counterpart International. The objective of the research is to understand the impact of the Food for Education and Child Nutrition program. Particularly, we want to know how the program has worked to meet you expectations, where it has succeeded, and where it could improve. ISG’s research will be conducted via a survey, focus group discussions, and key informant interviews. This focus group guide serves as a manual for the focus group leader and note taker conducting focus groups. ISG will conduct focus groups with mothers of students, PTA members, teachers, and school directors. Focus groups will cover five topics: 1. The degree to which project activities served the needs of beneficiaries. 2. The attitude of community members toward ownership and maintenance of infrastructure and asset building projects 3. The effectiveness of program in increasing student enrolment and attendance. 4. The importance of community farms. 5. The success of school feeding programs. The purpose of these focus groups is to present broad issues and to let participants generate ideas and build on each other’s ideas. The focus group leader’s goal is to encourage participation and ask questions that build on participants thoughts. This guide provides follow up questions that the leader may use. The leader may ask specific participants to compare their opinion or thoughts to those of other participants through prods such as “Participant B, how does your experience compare to the experience of participant A that we just heard?” or “Participant C, what would you add to participant B’s comments?” The interaction that the focus group leader encourages will provide depth to the information we are gathering through the survey and interviews. 56 The interviewer will also ask questions specific to the government officials area of expertise. Ministry of education officials will be asked about training and student achievement. Government of health officials will be asked about the SH&NG, moringa, and de-worming and vitamin A programs, for example 84 Preparation • Make sure there is a sign-in sheet, pen, and paper for each participant. • Make sure the room is set up with flip charts or whatever materials you need. • Set up refreshments if appropriate. • If available, set up small gifts to thank the focus group participants. Agenda Introduction, Ground Rules, Logistics and Sign-in Sheet [FOCUS GROUP LEADER: Introduce yourself and the note taker.] Thank you all for participating in this focus group. My name is _________ and I will lead the group, and this is ____________ who will take notes. Our discussion will last no more than an hour and a half. Before we get started, let me mention that there are drinks and refreshments available (point them out), and the bathroom is (indicate location of bathroom). Please feel free to get up and move around or help yourself to refreshments during the discussion. The purpose of this focus group is to understand the impact of the Food for Education and Child Nutrition program. Particularly, we’d like to understand: • How project has served your community’s needs. • Your opinion of the project’s construction projects, such as buildings, latrines, or community farms. • Whether the program has increased school attendance and why or why not. • The project’s food and hygiene programs. I wanted to mention a few important points before we start. • The purpose is for us to learn from you, both your positive and negative experiences and opinions. • We’re gathering information, not trying to achieve consensus. Please contribute your experience and opinion. It’s ok for there to be disagreement or different ideas. We have a few ground rules that we hope you’ll agree with: • Please participate. Your contribution is valuable and will improve the program’s activities in the future. • Please agree to keep the information provided in the focus group confidential. We won’t discuss any sensitive topics. Still, people may share ideas that they’d rather not be discussed privately. • Stay with the group and please don’t have side conversations. • Turn off cell phones if possible. Are there any questions before we get started? FOCUS GROUP LEADER: Go around the table and ask people to introduce themselves. 85 Questions 1. Thinking about the students who go to your school and their families, how have students most benefited from the Food for Education program? Follow Up Questions a. What has been the biggest improvement in the school since the program began? b. Does the school, its students, or its teachers have any needs that the program hasn’t addressed? Which? 2. I’d like to talk about how the program has changed student attendance. Has student attendance at the school increased or decreased since the Food for Education program began? Follow Up Questions a. What are the main obstacles to attending school for students in your community? b. What are the things that the program has done to overcome those obstacles? Have they been effective? c. Is there anything else the program could do to help students attend school? 3. The project has worked with the community to build many assets at the school. These might include latrines, classrooms, energy efficient stoves, or community farms. Which has been the most important? Why? Follow Up Questions a. Thinking about community farms, is there one at your school? How does your community contribute to the farm? Why is the farm important? b. Some schools have also built canteens. Is there one at your school? Why is it important? How does the community contribute to its maintenance? c. What other assets has the program brought that you value? Why? 4. One of the program’s central activities is providing improved access to nutritious food and improving nutrition education. Does your school have these programs? Follow Up Questions a. What is the most successful part of the school food program? How has it helped students and the community? b. How has the program changed student or teacher behavior? c. What is the school food program’s biggest challenge? d. Does the program as a whole meet the community’s needs? How could it improve? 86 5. Another program activity is improving health, including hygiene and deworming medication Follow Up Questions a. What is the most successful part of the hygiene program? How has it helped students and the community? b. How has the program changed student or teacher behaviour? c. What is the school health and hygiene program’s biggest challenge? d. Does the program as a whole meet the community’s needs? How could it improve? 6. One last question: If you were the boss of the program and could change it in one way, how would you change the program? 87 Annex 3: Selected Samples by Type and Location Student Sample Dagana Pete Podor Total CE1 70 113 147 320 CE2 83 76 119 278 CM1 55 56 83 194 Total 208 245 349 802 Dagana Pete Podor Total Girls 108 166 97 371 Boys 100 79 252 431 Total 208 245 349 802 Sample - Teachers Dagana Pete Podor St. Louis Total Primary 68 72 114 255 Preschool 28 16 17 40 101 Total 96 88 101 356 Female Male Total Dagana 43 53 96 Pete 27 61 88 Podor 35 96 131 Saint Louis 35 5 40 Total 140 215 355 88 Sample - Directors Dagana Pete Podor St. Louis Total Primary 30 23 25 78 Preschool 10 2 4 14 30 Total 40 25 29 14 108 Female Male Total Primary 2 76 78 Preschool 23 7 30 Total 25 83 108 Sample - Checklists Dagana Pete Podor St. Louis Total Primary 32 38 49 119 Preschool 13 6 6 16 41 Total 45 44 55 16 160 89 Annex 4: Survey Respondent Characteristics 90 Annex 5: School Sample Dept School Code Type Dept School Code Type Podor 12047 Primary Saint Louis 24028 Preschool Dagana 13066 Primary Saint Louis 24012 Preschool Dagana 23006 Preschool Saint Louis 24026 Preschool Dagana 13010 Primary Saint Louis 24027 Preschool Pete 13073 Primary Saint Louis 24025 Preschool Dagana 23007 Preschool Saint Louis 24009 Preschool Dagana 13029 Primary Podor 12062 Primary Dagana 13009 Primary Podor 12069 Primary Dagana 23001 Preschool Podor 12026 Primary Pete 13065 Primary Podor 12048 Primary Pete 13067 Primary Podor 22014 Primary Dagana 23010 Preschool Podor 12031 Primary Dagana 13001 Primary Podor 22004 Preschool Pete 13049 Primary Podor 22014 Preschool Pete 13068 Primary Saint Louis 24006 Preschool Pete 13074 Primary Saint Louis 24010 Preschool Dagana 13037 Primary Saint Louis 24015 Preschool Dagana 13015 Primary Dagana 23011 Preschool Dagana 11040 Primary Pete 13057 Primary Dagana 13064 Primary Podor 12044 Primary Dagana 13008 Primary Dagana 13042 Primary Pete 13054 Primary Dagana 13045 Primary Pete 13047 Primary Podor 12074 Primary Pete 13071 Primary Podor 12061 Primary Dagana 13075 Primary Pete 11003 Primary Pete 13052 Primary Pete 11037 Primary Pete 13053 Primary Pete 11007 Primary Dagana 23016 Preschool Pete 11015 Primary 91 Pete 13069 Primary Podor 11018 Primary Dagana 13023 Preschool Podor 12066 Primary Dagana 13028 Primary Podor 12017 Primary Dagana 13012 Primary Pete 11023 Primary Pete 13058 Primary Podor 12036 Primary Pete 13076 Primary Podor 12028 Primary Pete 13051 Primary Podor 12039 Primary Dagana 13013 Primary Podor 22017 Preschool Dagana 23003 Preschool Podor 12011 Primary Dagana 23005 Preschool Pete 22006 Preschool Dagana 13026 Primary Podor 12003 Primary Dagana 13077 Primary Podor 12005 Primary Dagana 13014 Primary Pete 22018 Preschool Dagana 13011 Primary Podor 12004 Primary Dagana 13027 Primary Podor 12006 Primary Dagana 23017 Preschool Pete 22013 Preschool Dagana 13033 Primary Pete 11022 Primary Dagana 13044 Primary Pete 22008 Preschool Dagana 13041 Primary Podor 12002 Primary Dagana 13017 Primary Pete 11012 Primary Dagana 13032 Primary Pete 11035 Primary Dagana 23009 Preschool Pete 11026 Primary Podor 12018 Primary Pete 11016 Primary Dagana 13019 Primary Podor 12033 Primary Dagana 13039 Primary Pete 11033 Primary Podor 22001 Preschool Pete 22015 Preschool Podor 12020 Primary Podor 11024 Primary Podor 12024 Primary Pete 11006 Primary Saint Louis 24029 Preschool Podor 22010 Primary Saint Louis 24008 Preschool Podor 12056 Primary 92 Dagana 13036 Primary Podor 12078 Primary Podor 12075 Primary Pete 11011 Primary Podor 12073 Primary Pete 11009 Primary Podor 12064 Primary Dagana Primary Podor 12051 Primary Pete 11041 Primary Podor 12050 Primary Pete 11008 Primary Dagana 13024 Primary Pete 11013 Primary Podor 12025 Primary Podor 12035 Primary Podor 12055 Primary Podor 12029 Primary Podor 12012 Primary Podor 12042 Primary Podor 12068 Primary Podor 12052 Primary Podor 22016 Preschool Podor 12040 Primary Dagana 13040 Primary Pete 11025 Primary Saint Louis 24018 Preschool Pete 22007 Preschool Podor 12065 Primary Pete 11036 Primary Dagana 23018 Preschool Pete 11027 Primary Dagana 23013 Preschool Pete 11002 Primary Saint Louis 24020 Preschool Pete 11044 Primary Saint Louis 24023 Preschool Podor 12046 Primary Saint Louis 24022 Preschool Podor 22010 Preschool Saint Louis 24002 Preschool Podor 12080 Primary Podor 12079 Primary Podor 12041 Primary Annex 6: Evaluation Team Bios Mr. Jason Wares, Team Leader, 93 Mr. Jason Wares, a Director of Programs at ISG, is an international development expert with 15 years’ experience specializing in catalyzing development and aid programming focused on livelihoods, competitiveness, and market development. Of most relevance to this project, Mr. Wares is currently leading baseline evaluation for ActionAid that is focused on Climate Smart Agriculture, women’s empowerment, and agriculture development. Other relevant experience includes an assessment of FINCA International’s Food for Progress (USDA-funded) projects in Haiti and Tanzania. Both assessments sought to understand how FINCA’s microfinance clients use loans to create sustainable microenterprises that translate into improved quality of life for their communities. Previously, Mr. Wares served as a senior economic development advisor at RTI International. At RTI, Mr. Wares led the institute’s business environment improvement technical area. Mr. Wares holds an MBA from the Thunderbird School of Global Management. Mr. Chris Root, Co-Evaluator Mr. Root is an agricultural economist with ten years of experience including in M&E, and impact evaluation. He recently worked with ISG leading the Final Evaluation of FINCA’s Tanzania program (USDA-funded) as well as for the Mid-term Evaluation of the McGovern-Dole USDA-funded program in Senegal. While working in Sudan, Mr. Root assessed the use of crop byproducts in livestock feed production. He made recommendations on how these byproducts, which are currently underutilized, could be better used to feed the country's huge livestock population. For a Somalia project he produced a brief on the camel milk value chain making recommendations on how a new project could work in the sector. While working at Land O'Lakes, Chris conducted research on livestock lending. Mr. Root has experience and expertise in propensity score matching (PSM) and difference in difference evaluation methodology. In Nepal he led a large agricultural value chain impact evaluation, conducting propensity score matching to identify impact. This built off his MA research titled "Assessing the Impact of Agricultural Value Chain Interventions" which developed the approaches he used in Nepal and built off his coursework. For RTI, he created guidelines on assessing the impact of economic development projects which detailed the PSM methodology. Recently at Michigan State he updated these skills through coursework on these quasi-experimental evaluation methodologies for his MS in Agricultural Economics. Ms. Noreen Mucha, Technical Specialist Ms. Mucha brings over 15 years’ experience implementing and evaluating child nutrition, sanitation and education projects. Ms. Mucha has been engaged with ISG since 2014 conducting evaluations, most recently for the Irish AID Program Fund recipient GOAL in Kenya and Zimbabwe centered on nutrition practices, child protection and water, sanitation and hygiene. 94 Annex 7: Indicator Framework – Baseline vs Mid-Term Data Strategic Objectives Indicator Indicator description Final Evaluation Result Baseline Result57 SO 1: Improve Literacy of School Aged Children Percent of students, who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text All 53.68 Boys 66 31 Girls 57 33 1.2.1: reduced Short-Term Hunger Percent of students in target schools who indicate that they are not hungry during the school day All 64.54 66 Boys 61.9 Girls 65.9 1.2.1: reduced Short-Term Hunger Percent of students in target schools who indicate that they are not hungry (or hungry from "time to time") during the school day All 86.72 66 Boys 85.35 Girls 87.43 1.4.3: Increased Government Support Percent of teachers who received government supported training in pedagogy in the past three months All 31.73 28 SO 2: Increased Use of Health and Dietary Practices Percent of school-aged children receiving a minimum acceptable diet at the school level All 57.89 2 2.2: Increased Knowledge of Safe Food Prep and Storage Practices Percent of beneficiaries (students, cooks) who use appropriate handwashing practices29 (i.e. with soap, before meals, before food prep, after latrine use, and diaper changing) All 73.68 Boys 71.43 50 Girls 74.86 55 All 51.88 57 The baseline study did not disaggregate all indicators. The table includes baseline indicators at the level each was reported, resulting in some blank cells in the table. 95 2.1: Improved Knowledge of Health and Hygiene Practices Percent of students in target schools who can correctly identify at least 2 ways to prevent intestinal worms Boys 52.75 8 Girls 51.43 10 Difference between baseline, adjusted baseline, mid-term and final evaluation reading levels Baseline Adj. baseline58 Midterm^ Final Grade 3 (CE1) 3.31 4.1 5.04 4.88 Grade 4 (CE2) 4.89 5.54 6.25 6.39 Grade 5 (CM1) 6.19 6.7 7.34 7.69 COHORT ANALYSIS Baseline Adjusted Baseline Midterm Final BASELINE CE1 COHORT 3.31 4.1 6.25 7.69 BASELINE CE2 COHORT 4.89 5.54 7.34 BASELINE CM1 COHORT 6.19 6.7 58Adjusted baseline accounts for the fact that children in the midterm have had more schooling than those at baseline. The baseline data was collected in Jan-Feb, the midterm data in May-June and the endline in April-May. Therefore kids in each grade have had nearly half a year longer to learn and should be expected to do better on reading tests, independent of the project. Adjustment made by dividing the difference betweren grades at the baseline by two (half a school year) and adding to the preveious year. I.e adjusted grade 3 = grade 3 + ((grade 4 - grade 3)/2) For grade 5, the project increase in reading score to grade 6 is based on the difference between the difference from grade 3 to 4 and grade 4 to 5. That is 1.58 - 1.30 = .28. 1.3 - .28 = 1.02. 96 Annex 8: Evaluation Terms of Reference Please find the terms of reference here