Mid-Term Evaluation McGovern-Dole International Food for Education and Child Nutrition Project Saint Louis Region, Senegal PREPARED BY INTERNATIONAL SOLUTIONS GROUP PREPARED FOR Counterpart International 9 August 2017 DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared independently by International Solutions Group. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. FY 2014 McGovern-Dole Saint Louis Region, Senegal Midterm Evaluation Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-685-2014/029-00 Funding Year: Fiscal Year 2014 Project Duration: 2014-2019 Implemented by: Counterpart International Evaluation Authored by: International Solutions Group Jason Wares Chris Root Safiétou Kane Contents Abbreviations ................................................................................................................................... 4 Executive Summary ..........................................................................................................................1 Findings Summary .................................................................................................................................................. 2 Strategic Objectives ........................................................................................................................................... 2 Activity Level Results ........................................................................................................................................ 3 Recommendations Summary ........................................................................................................................... 3 Methodology .................................................................................................................................... 4 Document review .................................................................................................................................................... 4 Qualitative research ................................................................................................................................................ 4 Quantitative Research ............................................................................................................................................ 5 Analysis 6 Measurement of Student Literacy ............................................................................................................................ 6 Dietary diversity ................................................................................................................................................. 6 Handwashing ...................................................................................................................................................... 6 Limitations of the Mid-Term Evaluation ............................................................................................................ 7 Statistical Strength .............................................................................................................................................. 7 Timing of the study ........................................................................................................................................... 7 Findings ........................................................................................................................................... 8 Strategic Objective Results .................................................................................................................................... 8 SO1: Improved Literacy of School Aged Children ........................................................................................... 8 SO1 Results ......................................................................................................................................................... 8 Improved Literacy .............................................................................................................................................. 8 Reduced Short-Term Hunger ........................................................................................................................ 11 Improved Student Attendance ....................................................................................................................... 13 Increased Government Support .................................................................................................................... 13 SO2: Improve Health and Dietary Practices Results ...................................................................................... 14 Percent of children receiving a minimum acceptable diet at the school level ........................................ 14 Increased Knowledge of Safe Food Prep and Storage Practices ............................................................. 15 Improved Knowledge of Health and Hygiene Practices ........................................................................... 16 At result levels, the MTE shall provide information on the status of the following indicators: ....... Error! Bookmark not defined. Activity Level Results ........................................................................................................................................... 18 School Selection ............................................................................................................................................... 18 Activity 1: Capacity building at the local, regional, and national level ..................................................... 18 Activity 2: Construction of school infrastructure ....................................................................................... 19 Activity 3: Construction of latrines and water station systems ................................................................. 20 Activity 4: Equip schools with energy saving stoves, canteen equipment and materials ..................... 22 Activity 5: Good health and Nutrition Practices ........................................................................................ 23 Activity 6: Training in Food Preparation and Storage Practices .............................................................. 26 Activity 7: Provide Access to Health Interventions ................................................................................... 29 Activity 8: The school feeding program ....................................................................................................... 29 Activity 9: Establishing Community Farms ................................................................................................. 31 Activity 10: Improved Teacher Attendance ................................................................................................. 32 Activity 11: Parent Association Training ...................................................................................................... 32 Monitoring and Evaluation of Program Activities ..................................................................................... 33 Evaluation Questions .................................................................................................................... 34 Implementation and Relevance .......................................................................................................................... 34 Effectiveness and Performance .......................................................................................................................... 36 Efficiency ............................................................................................................................................................... 37 Sustainability .......................................................................................................................................................... 38 Recommendations ......................................................................................................................... 39 Issues for consideration: ...................................................................................................................................... 39 Priority Actions ..................................................................................................................................................... 44 Annex 1: Document Review List.................................................................................................... 46 Annex 2: Data Collection Tools ..................................................................................................... 46 Annex 3: Sampling Methodology and Selected Samples by Type and Location .......................... 47 Annex 4: Survey Respondent Characteristics ................................................................................ 49 Annex 5: Schools Included in Sample............................................................................................ 50 Annex 6: Evaluation Team Bios .................................................................................................... 53 Annex 7: Indicator Framework – Baseline vs Mid-Term Data ..................................................... 54 Annex 8: Evaluation Terms of Reference ...................................................................................... 56 List of Tables, Figures, and Exhibits Table 1 Total interviews conducted including KIIs and FGDs (number of female in parenthesis) .............. 5 Table 2 Improvement in reading level by grade ..................................................................................................... 9 Table 3 Expected Reading Level by Grade ............................................................................................................. 9 Table 4 Reading Level By Department and Grade .............................................................................................. 11 Table 5 Attendance: Baseline vs Mid-Term........................................................................................................... 13 Table 6 School-age children receiving a minimum acceptable diet at Mid-Term............................................ 15 Table 7 Hand washing disaggregated by Department ......................................................................................... 16 Table 8 Classroom Construction Targets and Progress ...................................................................................... 19 Table 9 Progress toward School Latrine Construction Targets ......................................................................... 20 Table 10 Progress Toward Water Station Construction Targets ....................................................................... 20 Table 11 Schools with latrine and water infrastructure (n = 111) ..................................................................... 21 Table 12 Cleanliness of School Latrines ................................................................................................................ 21 Table 13: Latrine's structural condition (n = 96) .................................................................................................. 21 Table 14 The degree to which teachers have received and use training (n =312) ........................................... 24 Table 15 Degree to which teachers instruct on nutrition and hygiene (n=312) .............................................. 24 Table 16 Student reasons for washing hands ........................................................................................................ 25 Table 17 Access to Hand Watching Facilities ....................................................................................................... 26 Table 18 Condition of Food Storage Facilities % (n =111) ................................................................................ 27 Table 19 Conditions for storage in warehouse (n = 108) ................................................................................... 27 Table 20 Director knowledge of food storage facility maintenance .................................................................. 28 Table 21 Directors knowledge of inventory control (n = 108) .......................................................................... 28 Table 22 Directors knowledge of serving safe food ............................................................................................ 28 Figure 1 Reading Level by Gender (adjusted Values) ............................................................................................ 9 Figure 2 Proportion of Students Who Read At Or Above Grade Level (adjusted Values) .......................... 10 Figure 3 Proportion of students who read at or above grade level - baseline vs mid-term ........................... 11 Figure 4 Percent of students who indicate that they are not hungry during the school day ........................ 12 Figure 5 Short-term hunger disaggregated by department at Mid-Term* ........................................................ 12 Figure 6 Students Minimum Acceptable Diet ....................................................................................................... 15 Figure 7 Appropriate Hand Washing Practices..................................................................................................... 16 Figure 8 Percentage of Students that Can Name Two Ways to Prevent Intestinal Worms .......................... 17 Figure 9 Percentage of Students that can name two ways to prevent intestinal worms disaggregatd by department* ................................................................................................................................................................ 17 Figure 10 Reasons That Teachers Washed Their Hands (%) (Multiple Responses Possible) ....................... 25 Exhibit 1 Students in Dagana .................................................................................................................................... 1 Exhibit 2 Students Attend Class in Dagana ............................................................................................................. 8 Exhibit 3 Members of a PA demonstrate holes in need of Repair .................................................................... 19 Exhibit 4 Cooks at a School with 32 Students using an energy efficient stove to cook lentils, and a traditional fire to cook cous cous. ........................................................................................................................... 22 Exhibit 5 Teacher in Dagana showing his copy of the SH&NG ....................................................................... 23 Exhibit 6 STUDENTS WASHING HANDS BEFORE LUNCH .................................................................. 26 Exhibit 7 An evaluator shows a container of moringa powder used at a pre-school in Dagana .................. 29 Exhibit 8 A CPI Community Facilitator Introducing evaluators to a community farm volunteer .............. 31 Exhibit 9 A typical classroom visited by Interviewers, clearly in need of roof repair .................................... 35 Exhibit 10 The blackboard is all that is left of a classroom destroyed by the wind ........................................ 40 Exhibit 11 Hand washing implements stored out of reach ................................................................................ 41 Abbreviations CE1 3rd Grade CE2 4th Grade CM1 5th Grade EE Ecole Elémentaire FAO Food and Agriculture Organization FFE Food for Education FGD Focus Group Discussions IEF Inspection l' education et de la Formation ISG International Solutions Group KII Key Informant Interview MGD McGovern Dole PA Parent Association SH&NG School Health and Nutrition Guide SMC School Management Committee SO Strategic Objective TOR Terms of Reference USDA United States Department of Agriculture 1 Executive Summary In Senegal, only 71 percent of primary school-aged children are enrolled in school1. Of the children that enroll, 60 percent of girls and 53 percent of boys continue to complete primary school2. The 2017 Social Progress Index ranks Senegal 116th in the world in access to primary education3. The low enrollment and completion in primary education translates to low literacy rates for adults. Approximately 40 percent of adult women in Senegal are literate and 66 percent of Senegalese adult men are literate. While these statistics describe Senegal as a whole, the problem of insufficient school access is especially acute in The Saint Louis region. One obstacle to school access in Saint Louis is the region’s dependence on agriculture. In Saint Louis, 60 percent of the population depends on agriculture for their income. The agriculture sector in Saint Louis has experienced increased food insecurity due to irregular rainfalls and shortfalls in investment4. In Senegal, 85 percent of children that work are employed in the agriculture sector, meaning that increased food insecurity in agricultural regions means more work and presumably less school5. The McGovern–Dole International Food for Education and Child Nutrition Program (FFE) works around the world to address the types of access to education and literacy challenges that Senegal faces. In September of 2014, FFE awarded a grant to Counterpart International to implement a program that would “improve food security, reduce the incidence of hunger, and improve literacy and primary education” in Senegal’s Saint Louis region. After the completion of the baseline report, the project was launched in 2016. In May of 2017, Counterpart International hired International Solutions Group (ISG) to conduct a mid￾term evaluation of the Food for Education program. The purpose of the mid-term evaluation was to “assess the program’s performance to date,” with respect to the program’s “goals, objectives and activities.” In conducting a mid-term evaluation, Counterpart sought to obtain an independent assessment of the program’s achievement at the half way point of the program, an estimate of whether the program was on-track to achieve its targets and objectives, and to receive recommendations on how Counterpart might improve the program’s performance. This document presents the results of the mid-term evaluation. Results were gathered through a mixed￾methods data collection methodology. To conduct the evaluation, we surveyed 1,079 students, 312 teachers, and 108 school directors at 111 schools. The evaluation team also took observations at 111 schools. These observations were of school infrastructure, latrine access, hand-washing facilities, food storage facilities, and school canteens. The student sample size corresponds to a confidence level of 95% and a margin of error of 5.01%. For teacher survey data, the margin of error increases to 6% at a 95% level of confidence. Director interviews and school observations were not intended as a statically representative sample of the 270 project schools but rather as a best effort concession to budgetary and 1 World Bank’s Databank “World Development Indicators” database. 2015 Data series 2 Ibid. 3 https://www.socialprogressindex.com/?tab=2&code=SEN 4 FAO Emergencies – Senegal brief 5 World Bank’s Databank “World Development Indicators” database. 2015 Data series EXHIBIT 1 STUDENTS IN DAGANA 2 logistical realities.6 The results of the director and school observation surveys set a minimum number to inform indicators rather than a representative average. The team also conducted qualitative information gathering to add context and narrative around the quantitative data collection results; conducting key informant interviews and focus groups that engaged 84 stakeholders. These stakeholders were students, teachers, parents, school cooks, school directors, project staff, and government officials at the local, regional, and national levels. Findings Summary The mid-term evaluation presents findings at the strategic objective and activity indicator levels. It also answers specific questions included in the TOR regarding program implementation and relevance, effectiveness and performance, efficiency and sustainability. Strategic Objectives Food for Education in Senegal is focused on achieving two strategic objectives(SO). These are: • SO1: Improved Literacy for School Aged Children • SO2: Increased Use of Health and Dietary Practices Under SO1, Food for Education seeks to improve the literacy of children through achieving four results: 1. Improved literacy of school aged children: Mid-term ASER test results demonstrate that the Food for Education program has improved student literacy. Grades 3, 4, and 5 students improved their reading levels by 1.79, 1.42.and 1.29 levels respectively in terms of the total literacy improvement over the baseline. 2. Reduced short-term hunger: The evaluation team found that the Food for Education program is reducing hunger for its beneficiary students. The mid-term evaluation survey determined that 70 percent of student beneficiaries report that they are not hungry at school and 89 percent report that they are not hungry at school except “from time to time,” compared to 66 percent at baseline who said they were not hungry during the school day. 3. Improved student attendance: Student attendance has increased since the baseline study. During January, February, and March of 2017, 98 percent of students were present at least 90 percent of each month’s school days. 94 percent of students had perfect attendance records during those months. Additionally, school enrollment has increase by 4.2 percent since July, 20167. 4. Increased government support: The one area that showed a decrease from baseline to mid￾term was the measurement for increased government support, defined as the percent of teachers who had received pedagogical training in the three months prior to the survey. At baseline 28 percent of teachers said that they had received government supported training in pedagogy in the last three months. At mid-term, this measure had fallen to 13 percent. Under SO2, Food for Education seeks to improve health and dietary practices through achieving three results: 1. Increased percentage of children receiving a minimum acceptable diet: The mid-term evaluation team found it difficult to compare baseline and mid-term results for this indicator. The team felt that the baseline methodology resulted in biased and overly optimistic estimates of the proportion of students that were eating a minimum acceptable diet. While the mid-term methodology was far from perfect, it erred toward deriving a more conservative estimate. The mid-term survey estimated that 43 percent of students receive a minimum acceptable diet compared to 72 percent at baseline. 6 Note that in cluster sampling the clusters sampled (in this case schools) are not necessarily representative of the population of clusters. 7 Data taken from Counterpart Senegal’s monitoring and evaluation records 3 2. Increased knowledge of safe preparation and food storage practices: The strategic objective-level measure for this indicator is the percentage of beneficiaries who use appropriate hand washing practices before meals, before food prep, after using the latrine, and after diaper changing. The mid-term evaluation team calculated that 77 percent of beneficiaries use appropriate hand washing practices, a clear improvement over the baseline finding of 50 percent for boys and 55 percent for girls 3. Improved knowledge among students of health and hygiene practices: The mid-term evaluation also measured improvement in this indicator over the baseline. At baseline, 8 percent of boys and 10 percent of girls could identify at least two ways to prevent intestinal worms (the central measure for progress under this objective). At mid-term, 45 percent of boys and 41 percent of girls could do the same. Activity Level Results The Food for Education program aims to produce outputs in 11 activity categories and achieve 52 indicators. The mid-term evaluation found that the program was on track to achieve nearly all of the required results under these indicators. Some highlights of the program’s achievement include: • 10,496,309 meals served in school compared to an end of FY2017 target of 9,050,0008 • 526 Parent Association Members trained against a total project target of 540 • 539 Cooks trained against a target of 540 • 44,095 students receiving a first cycle of deworming medication compared to a target of 42,055 • 20 community farms established • 270 durable, culturally appropriate energy efficient stoves produced, with more than half distributed and in use Overall, the evaluation team found that Counterpart’s strategy for program implementation, which creates locally sustainable assets around the complete school feeding and literacy program, is an important component of the program’s success. The full report describes the quality of these outputs, reasons for successes, and areas where improvements could be made. Recommendations Summary The report concludes with a discussion of issues for consideration and recommendations of priority actions the program should take. We include a summary of the recommended actions here. 1. Work with the Senegalese government to plan for the program’s sustainability: The Food for Education program has done an excellent job of coordinating program implementation with the government at all levels. The next step is to assist these governmental partners to plan and acquire resources to maintain the program after USDA and Counterpart step away. 2. Conduct research into gender issues: Particularly, the research should focus on the obstacles that boys and girls face to attending school and achieving in school. 3. Improve communication with stakeholders: This communication includes information about construction project requirements and timelines, community contribution requirements, why schools are eligible for specific benefits such as community farms, and why others are denied eligibility for certain benefits such as construction projects. 4. Ensure that teacher trainings continue: Teachers’ enthusiasm for the pedagogical trainings that Food for Education facilitated were an important contributor to gain in student literacy. Food for Education should make sure that trainings continue under the program, and that inspectors have a plan for continuing training after the program concludes. 5. Allocate construction and food resources according to need rather than equal distribution: Needs vary greatly across Food for Education schools The Food for Education 8 Targets reflect figures included in USDA’s amendment III to the Food for Education agreement signed by Counterpart on June 6, 2017 and USDA on June 1, 2017. 4 program was designed to distribute resources as equally as possible among eligible schools. However, a need based approach would make more efficient and effective use of Food for Education resources. 6. Systematize sustainable Latrine maintenance: Though Food for Education did not build all of the latrines at its beneficiary schools, nearly 87 percent of schools had latrines. It is unlikely that volunteer work alone will ensure that latrines are maintained in an acceptable state. Food for Education should come up with a solution to this problem, which could involve providing stipends or incentives to community members that would appreciate the work. 7. Provide equal access to latrines for boys and girls: The majority of existing school latrines provide equal access for boys and girls. Still, Mid-term evaluation enumerators recorded that 25 percent of latrines were not accessible by boys and girls. This number is unacceptably high, though the solution is probably simply a matter of proper signage or scheduling. 8. Improve access to handwashing implements: Many schools have no access to soap which is a crucial component of proper hand hygiene. Food for Education should work with schools to ensure that a complete set of handwashing implements are accessible whenever required. Methodology The Evaluation Team carried out the mid-term evaluation in May and June of 2017. Data collection took place in Senegal between May 19 and June 16. The team used several methods to collect data including, a document review, qualitative research through key informant interviews (KIIs) and focus group discussions (FGDs), and quantitative data collection through surveys. This mixed methods approach allowed the evaluation team to strengthen quantitative results through the depth and context that qualitative research provides. Document review The document review informed all stages of the evaluation. The team used the document review initially to learn about the project and design data collection tools. Later the review was useful in explaining different features of program design. It was also the source of most of the data the evaluation team used to compare performance results against targets. Annex 1 includes a full list of the documents the team reviewed. Qualitative research The evaluation team conducted KIIs and FGDs with a total of 97 respondents to provide context and depth to findings. The team developed interview and focus group discussion guides with the assistance of Counterpart staff. The guides asked questions intended to expand the team’s understanding of the Food for Education project, answer evaluation questions, verify the team’s tentative hypotheses, and confirm information gleaned from other sources. The question guides were used to lead semi-structured interviews. The guides ensured that the evaluation team asked the same key questions across similar respondent types while allowing the freedom for open ended, conversational-style interviews. These guides are included in Annex 2. Qualitative field research was conducted in the three departments (Dagana, Podor and St. Louis) that make up the Saint Louis region. In total, the evaluators visited 20 schools. These were a subset of the 1119 randomly sampled schools where interviews were conducted by the survey team. Each day the evaluators went in a separate vehicle with an enumerator team and conducted KIIs and FGDs while enumerators conducted surveys. The assignment to enumerator teams was random, and so was the selection of schools for qualitative research. Within each school, the evaluators conducted interviews with the director and one or more teachers. In many cases interviews also included cooks or parents or school 9 The sampling methodology indicates that the team should have sampled 112 schools rather than 111. This discrepancy is due to a last minute downward adjustment in the number of teachers per school based on feedback from the survey team. Given lower than expected teacher response rate, this difference is immaterial. 5 management committee members. The evaluation team conducted FGDs with students at primary schools and with some teachers and parents at primary and preschools. In addition to interviews at schools, the evaluation team conducted KIIs with education administrators and other government officials at the national, regional and departmental levels. Finally, the team interviewed FFE project staff. A summary of KIIs and FGDs conducted is presented in table 1 below. TABLE 1 TOTAL INTERVIEWS CONDUCTED INCLUDING KIIS AND FGDS (NUMBER OF FEMALE IN PARENTHESIS) Dagana Podor St. Louis National Level Directors – Primary 9 (0) 1 (0) Directors - Preschool 3 (0) 2 (2) 3 (2) Teachers – Primary 9 (1) 3 (2) Teachers – Preschool 1 (1) 1 (1) Students - Primary 6 (5) 9 (6) Parents 8 (4) 2 (1) Cooks 5 (5) 2 (2) Government 3 (0) 3 (0) 7 (2) Project Staff 4 (0) 3 (1) Total 47 16 11 10 Quantitative Research The midterm evaluation included student, teacher and director surveys. In addition, enumerators completed a school observation checklist at each school in the sample. Sampling for the student survey followed a two-stage cluster design with the first cluster schools and the second students. In brief, the first stage involved randomly selecting 111 schools from the 270 project-affiliated schools. In the second stage, we randomly selected 14 students from within each primary school (81 primary schools) for a total student sample of 1,133 out of a student population of 44,912. This corresponds to a confidence level of 95% and a margin of error of 5%. In actuality, 1,079 students were interviewed across 81 schools. With a confidence level of 95%, this only increases the margin of error by 0.01 percentage points to 5.01%. Sampling methodology is detailed in annex 3. The teacher sample was initially calculated as 444 teachers across the 111 schools in the survey. There was no list of randomly selected teachers by name for enumerators to follow; instead enumerators were instructed to interview 4 teachers per school. However, in practice 312 teachers were available to take the survey; less than three per school on average. Therefore, the margin of error increases to 6% at a 95% level of confidence10. Director surveys and school checklists were to be completed at all 111 schools in the sample. In actuality, 108 directors were available to participate in the survey. This was not intended as a statically representative sample of the 270 project schools but rather a concession to budgetary and logistical realities.11 The results of these surveys set a minimum number to inform indicators rather than a representative average. 10 Please see Annex 3 for a description of the survey sample. 11 Note that in cluster sampling the clusters sampled (in this case schools) are not necessarily representative of the population of clusters. 6 The mid-term evaluation questionnaires were modified versions of the questionnaires used for the baseline study. While many key questions were maintained to allow for comparison between baseline and mid-term, several questions were altered to reduce the social desirability bias12 that skewed baseline results, as highlighted by the baseline report. Questionnaires are available in Annex 2. Characteristics of survey respondents are included in annex 4. Analysis Measurement of Student Literacy A key outcome this report measures is improvement in student literacy. This measurement is based on the same ASER reading test conducted for the baseline study.13 The mid-term evaluation team compared reading scores at baseline and midterm by grade to identify the program’s achievement. In other words, the reading scores for CE1 students from before the Food for Education program started are compared to CE1 students after one year of the program. We also compared CE2 and CM1 students at baseline and midterm. The difference between the scores is the improvement in student literacy that the program has achieved at the mid-term point for each of the three grades (See Findings, SO1). Note that the literacy analysis includes a correction to account for the difference in timing of the baseline and midterm surveys. The baseline survey was conducted in January and February of 2016 whereas the midterm survey in May and June of 2017. The difference in timing means that students in the midterm had nearly half a school year longer to improve their reading in comparison with their baseline cohorts. Naturally, the extra half year of school should cause the reading results from the midterm to be better than those at baseline, independent of the project’s impact. The evaluation team corrects for the different timing of the studies by adding half of the difference in reading scores between grades that are observed in the baseline. For example, the difference between the average reading score for CE1 and CE2 at baseline is 1.58. Thus, 0.79 is added to the baseline score for CE1 to estimate what that average score might be if it were collected at the end of the school year in May or June instead of January or February. Likewise, the difference between CE2 and CM1 is 1.3, so 0.65 is added to grades for CE2. Because no data is available for CM2, for the CM1 baseline, the adjustment is made by adding 0.65 – (0.79-0.65) = 0.51 to baseline data for CM1. This attempts to project the decreasing trend in intra-grade reading improvements shown in the difference between reading improvements between CE1 and CE2 and then between CE2 and CM1. The report also looks at gender differences in literacy outcomes by comparing ASER scores for boys and girls in each grade. This comparison is made at baseline and at midterm to assess how gender differences in reading may have changed with FFE. Dietary diversity Dietary diversity is calculated based on its definition at baseline. That is, it is the share of students who report eating at least 4 of 7 major food groups at school. These seven groups are: tubers, grains and root crops; nuts or legumes; dairy products; eggs; and two different groupings of fruits and vegetables. Handwashing Correct handwashing practices is defined as washing hands with soap and water before eating and preparing food, and after using the latrine. 12 Social desirability bias is the tendency of survey respondents to provide answers that they believe others will approve of rather than reflect their true opinion or experience. The baseline and mid-term surveys had particular risk of this bias as they often took the form of an adult asking questions of a child. 13 http://www.asercentre.org/ 7 Limitations of the Mid-Term Evaluation Statistical Strength Of the three surveys plus the checklist, only the student survey is statistically representative. While the director survey and school observation checklists were not designed to be representative, the teacher survey was initially designed to be representative. However not enough teachers were available to interview at each sampled school to reach the targeted sample size. The evaluation team does not know why fewer teachers than the calculated average per school were available. Timing of the study The evaluation was initially scheduled for April/May 2017. However, contracting issues and time required to develop data collection instruments delayed the start of field work. Hence, the evaluation team conducted most of the surveys and interviews during Ramadan, which may have influenced the study in several ways. First, many schools that ordinarily might have had lunch prepared were not serving food because volunteers did not want to cook during the holiday. Secondly, many respondents were not eating or drinking, so surveys and interviews that were conducted late in the afternoon may have been cut short to respect respondents’ energy levels. Thirdly, the lack of food and water may have affected the expression of attitudes and perceptions that survey questions requested. 8 Findings In this section, we describe the progress that the Food for Education program has made toward strategic objectives, the program’s activity level results, the process of school selection, and discuss Food for Education’s monitoring and evaluation system. Strategic Objective Results Food for Education in Senegal is focused on achieving two strategic objectives. These are: • SO1: Improved Literacy for School Aged Children • SO2: Increased Use of Health and Dietary Practices SO1: Improved Literacy of School Aged Children Under SO1, Food for Education seeks to improve the literacy of children through achieving four results: 1. Improved literacy of school aged children 2. Improved student attentiveness through reduced short-term hunger 3. Improved student attendance 4. Increased government support SO1 Results Improved Literacy The evaluation team believes that the Food for Education program has improved student literacy. Following the methodology used in the baseline study, the evaluation team utilized the ASER literary assessment to determine whether Food for Education students’ reading ability had improved since baseline and had made progress toward reading at grade level on average. Additionally, the evaluation team could not come up with another reason, other than the program’s effectiveness, that school children across Saint Louis would significantly improve at reading from one year to the next holding grade level constant. Also, we have come across no evidence to suggest this result should be expected in the absence of the program. Therefore, it's very fair to attribute the literacy improvement to the project. Table 2 shows the average reading level that students achieved at baseline, the adjusted baseline score, and the reading level at midterm for each grade. The table also shows the raw difference, which describes students’ overall improvement since baseline, and the adjusted difference, which is difference that is assignable to the Food for Education program. Grades 3, 4, and 5 students improved their reading levels by 1.79, 1.42.and 1.29 levels respectively in terms of the total literacy improvement over the baseline. In terms of adjusted difference, grade 3 students improved by 1 level, grade 4 by .77 levels, and grade 5 by .78 levels. EXHIBIT 2 STUDENTS ATTEND CLASS IN DAGANA 12 65 percent of students mentioned eating something before school without prompting from the mid-term evaluation interviewer. In interviews and focus groups, students and teachers said the Food for Education provided meal was more nutritious and diverse than what students usually eat at home. FIGURE 4 PERCENT OF STUDENTS WHO INDICATE THAT THEY ARE NOT HUNGRY DURING THE SCHOOL DAY 19 Figure 5 shows the short-term hunger results disaggregated by department. Statistically, the results show a difference among the three departments. FIGURE 5 SHORT-TERM HUNGER DISAGGREGATED BY DEPARTMENT AT MID-TERM* * we can reject the hypothesis that all three means are the same at the p<0.05 level 19 The Mid-Term excluded students who reported that they only felt hungry “from time to time.” 0 20 40 60 80 100 Mid-Term Baseline n=1079 0.7 0.75 0.8 0.85 0.9 0.95 1 Dagana Pete Podor 14 SO2: Improve Health and Dietary Practices Results Under SO2, Food for Education seeks to improve health and dietary practices through achieving three results: 1. Increased percentage of children receiving a minimum acceptable diet 2. Increased knowledge of safe preparation and food storage practices 3. Improved knowledge among students of health and hygiene practices Percent of children receiving a minimum acceptable diet at the school level It is difficult to compare the change in student’s diet between the baseline and mid-term evaluation. Figure 6 shows a decline in the percentage of students that consume a minimum acceptable diet between baseline and mid-term. The mid-term evaluation team believes that the baseline study presented an overly optimistic estimate of the sufficiency of students’ diets. The baseline study relied heavily on children’s memory about food they ate days before the survey was administered. It also asked them to remember how they felt after eating, and asked them yes/no questions about whether they had eaten foods in each food group. Children’s memories are very often unreliable in these types of questionnaires21. The students also may have answered yes to questions in order to impress the enumerator. The evaluation team believes that relying on children’s memory and asking about specific food groups resulted in the baseline study significantly overstating the percentage of children that had access to a minimum acceptable diet. The mid-term team did attempt to take a more accurate, though certainly not perfect, measurement of student’s diets at the time of the mid-term evaluation. First, the survey asked students how they prepared for school in the morning. The purpose of the question was to see if students would mention eating as part of their daily routine as they prepare for school. As described above, approximately 65 percent of students mentioned eating unprompted as part of their routine. However, when those that did not mention eating were asked if they did eat, the percentage of students who said that they ate before school rose to 90 percent. These results are burdened by the circumstances of the survey. First, the survey took place during Ramadan, so some students may not have eaten and did not mention eating before school, even though it may be part of their routine when it is not Ramadan. Conversely, some students may only eat before school during Ramadan. Respondents to the second question may have forgotten to mention that they ate, or they may have felt that there was social pressure to say that they ate before school. The students were then asked to describe what they ate as part of their daily breakfast and lunch. Using the formula recommended in the baseline study, we calculate that 43 percent of students receive a minimum acceptable diet22 (Figure 6). That percentage is almost exactly the same for boys and girls (table 6). Whereas the baseline study erred toward an optimistic estimate of the quality of student’s diet, the mid-term estimate errs toward the conservative. 21 See Fuchs, Marek. "The Reliability of Children’s Survey Responses: The Impact of Cognitive Functioning on Respondent Behavior." Symposium 2008: Data Collection: Challenges, Achievements and New Directions (2008): 1-9. Web. 22 Dietary diversity is calculated based on its definition at baseline. That is, it is the share of students who report eating at least 4 of 7 major food groups at school. These seven groups are: tubers, grains and root crops; nuts or legumes; dairy products; eggs; and two different groupings of fruits and vegetables. 18 Activity Level Results School Selection The Food for Education program selected eligible schools based on six criteria: 1. A minimum of 30 students. 2. A sufficient number of teachers given the number of students, though no specific number was given. 3. Low enrollment statistics, which were provided by the inspector. 4. Accessibility, meaning within a reasonable distance to major towns and accessible by trucks. Schools that were only reachable by crossing a river were ineligible. 5. A kitchen and food storage building. 6. Schools with a community that has demonstrated a commitment and willingness to contribute to previous projects; including contributions in cash or in-kind support.. Food for Education asked the school inspector’s office for a list of schools in Saint Louis that met this criteria. Then, Counterpart and School Inspector’s Office staff visited the schools to make sure the project could serve them. After visiting the schools, the Inspector’s Office and Counterpart realized that a number of the schools were not a good match for the program, so Counterpart asked the Inspector’s office to identify replacement schools. At the end of school selection, the inspectors office initially identified 180 primary schools and 44 preschools that met the program’s criteria. The program chose to serve all 44 preschools and 163 of the primary schools from this first round of school selection. Currently, the program supports 204 primary schools and 66 pre-schools. Two criteria for school participation have changed; one before the program launched and one after. The first was the requirement that schools have kitchens in place. As this document discusses in Activity 8: The School Feeding Program, kitchens are relatively rare, though more common at preschools than primary schools. Requiring schools to build kitchens in advance of the program would have been prohibitively onerous. Counterpart felt that a secure food storage facility was necessary, but that the program could work around lack of a kitchen23. The second involves community engagement. As the Food for Education relies on volunteers, school level success requires that schools actively partner with Food for Education to identify volunteers, acquire inputs, and engage in management activities. Counterpart found that at the beginning, schools could talk about the importance of community engagement, but were slow to act when it came time to start the program. Counterpart program managers say that in the next round, they will require more proof of activity from schools before they start contributing resources. Activity 1: Capacity building at the local, regional, and national level Like the other components of the program, Food for Education’s methodology for improving literacy stresses improving the system rather than focusing on direct delivery of services. Food for Education consults with the national Ministry of National Education, Regional governor’s office and education departments, and local level inspectors. Together, Food for Education and the various government bodies develop a training that conforms to the most current training curriculum. The training also includes assessment tools and lesson plans that assist teachers in designing classes. New literacy teaching skills are formulated at the central government level and then disseminated throughout the educational system through a training-of-trainers model. Food for Education worked with the Ministry of National Education’s Directorate of Elementary Teaching to revise Senegal’s recently 23 The information about criteria changes comes from staff interviews. 23 counterpart stoves. Cooks that used gas stoves said that gas was easier to acquire. Some acknowledged that the Counterpart prototype stove cooked faster, but they did not have access to charcoal or firewood so they stuck with the gas stoves. Some cooks could not use the stoves if the firewood they had would not fit in the stove. At one preschool that the team visited, five mothers all said their kids were not eating at school and that food had not be served at the preschool for three years. The parents said they didn't know who to report this to. Cooking equipment at the preschool looked unused. However the director, cook and community association president all said the preschool feeding program was in operation. When this information was shared with Counterpart, Counterpart sent two staff members to conduct an unannounced audit of the site. Counterpart provided the evaluators with a report of this audit. Counterpart’s report stated that the canteen has operated at the school since March 2016 that the canteen has operated continuously, and that students said they were well-fed. Counterpart also supplied records of food delivery and consumption. The evaluation team does not doubt the information provided in Counterpart’s report. Still, the report contradicts the information gathered by the evaluator through key informant interviews, and observations that the evaluator made while on-site. In the evaluator’s opinion, neither the evaluator’s visit nor Counterpart’s report provides evidence of how effectively this school manages the program. Activity 5: Good health and Nutrition Practices The good health and nutrition practices activity promotes the use of the School Health and Nutrition Guide (SH&NG). The promotion of the guide aims to improve the health and hygiene practices of the school community including students, teachers, and directors. Counterpart promotes the guide by training school directors in the SH&NG’s lessons. The directors then train teachers in the guide’s use, and the teachers build lessons into the curriculum to improve students’ practices. Counterpart conducted trainings for the 204 Food for Education primary schools between March and May of 2017. They also printed 3,000 copies of the SH&NG. The guide was distributed to the 204 primary schools, the Regional Training Center for Education Personnel in Saint Louis, and 200 copies went to schools in the region not covered by the Food for Education program. In interviews, teachers and directors acknowledged that they had received the guide, and most had read it. During teacher interviews, teachers could pull out the guide and show sections of it to evaluators upon request. The teachers and directors that evaluators asked about the guide found it useful and practical as an educational resource. Counterpart also conducted a “Hygiene Week” at each of the Food for Education schools. Hygiene weeks were popular with parents and teachers. The director of the Ehmane Racine Sy primary school said that hygiene weeks worked well. She had noticed that students hand washing had increased before meals and after using the bathroom as a result of hygiene week activities. Teacher Knowledge and Practices The mid-term evaluation attempted to measure the degree to which teachers were themselves trained in nutrition and hygiene, and the degree to which they instruct students in nutrition and hygiene. The teacher results are shown in table 14. EXHIBIT 5 TEACHER IN DAGANA SHOWING HIS COPY OF THE SH&NG 25 reflect the fact that most teacher surveys were conducted during Ramadan and interpretation of when the day begins could have influenced responses. FIGURE 10 REASONS THAT TEACHERS WASHED THEIR HANDS (%) (MULTIPLE RESPONSES POSSIBLE) Student Knowledge and Practices The mid-term evaluation sought to measure the degree to which students engaged in good hygiene practices. The mid-term team restructured the baseline survey evaluation questions to overcome some of the social desirability bias found in the baseline report. Rather than asking questions such as “did you wash your hands today before eating?” enumerators first asked students, “did you wash your hands today?” Of those that said yes, enumerators asked “why did you wash your hands?” The intention of rewording the question was to remove prompts for when hands should be washed, and also lessen social desirability bias that comes with specific instances of hand washing. Though the redesigned questions still suffer from bias, we believe our results reflect student behavior somewhat more accurately. When asked if they had washed their hands today, 95% still said that they had. As students who had recently participated in lessons on hygiene and nutrition, this result probably still reflects bias as all students knew that they should have washed their hands at least once. However, table 16 shows student explanations of why they had washed their hands. TABLE 16 STUDENT REASONS FOR WASHING HANDS When did you wash your hands today? (%) multiple answers possible (n = 1028) Before eating 75.58 Before touching or preparing food 8.17 Before giving food to someone else 13.33 When you have earth on your hands 6.03 After touching something dirty 19.84 After touching an animal 5.16 After using the latrine 36.48 After changing a diaper 2.92 0 20 40 60 After touching an animal I had dirt on my hands After changing a diaper Before touching or preparing food Before giving food to someone else After touching something dirty After using the latrine Before eating n = 280 29 The program does experience some losses. When commodities arrive at the ports, inspectors collect a sample of commodities to conduct tests for phytosanitary inspections. Inspectors may refuse entry to some commodities if they do not pass the phytosanitary inspection. Some is lost to pests or in transit. However, these losses are not material and are reported to USDA regularly. Activity 7: Provide Access to Health Interventions Access to health interventions consists of three activities; provision of vitamins, provision of de-worming medication, and the establishment and use of moringa plantations. Deworming Medication and Vitamin A Supplements As of the mid-term evaluation Food for Education had received and processed two shipments of de￾worming medication and Vitamin A supplements. The first shipment was distributed to students in January 2017, when 44,095 students in all of the programs 270 schools received deworming medication. Counterpart will use a different distribution strategy for the second shipment. The second shipment of the medication was divided into thirds; 1/3rd for distribution to Food for Education schools in Saint Louis now, 1/3rd for distribution to other schools in Senegal now, and 1/3rd set aside for distribution in December 2017. In interviews and focus groups, parents and teachers credited the deworming medication with improved health in the children. They said they could see the effects of it very quickly. The evaluation team also met with the national department for school health and spoke with a deputy administrator. She reported that the deworming program fits within their scope of work and they look forward to planning for the transfer of the deworming activity to their office when the project ends. Moringa Plantations The Food for Education program has purchased 9,434 moringa plants of which 6,750 have been planted at schools. School teachers and directors demonstrated the plants to the evaluation team and could explain the benefits of the plant as well as the way it is processed and used in food. Although Counterpart’s most recent progress report (October 1 2016 to March 31 2017) says that no schools are regularly using moringa in meals, the evaluation team did find at least one school that was utilizing the plants products. One school director in Dagana said that moringa is beneficial because “moringa never dies. The whole plant is useful and now households in the community have begun to grow it!” Though moringa plants are very hardy, they do require some maintenance that makes raising them a challenge. Areas that have poor access to water find it difficult to provide the water that young moringa plants require. A school director who managed a facility with poor infrastructure said that he received 25 moringa trees but “they all died.” If the school had a fence, they might have lived but the animals ate them. Activity 8: The school feeding program The school feeding program is the core of Food for Education. As part of Food for Education Senegal, the school feeding program is highly successful as of the mid-term evaluation. The program has exceeded its targets and produced verifiable results in the higher level objectives reported above, namely improved literacy and increased attendance. The Food for Education program’s primary activity is to provide food to schools in food insecure regions. Distributing the food through schools provides incentives for students to attend school and to make sure they have the energy required to be attentive and focus on their studies. Many of Food for Education’s other activities are implemented in support of the school feeding program. EXHIBIT 7 AN EVALUATOR SHOWS A CONTAINER OF MORINGA POWDER USED AT A PRE￾SCHOOL IN DAGANA 30 The program requires successful coordination of the following activities, some of which are discussed in more detail under their own activity heading below: • Coordination with government entities including local and regional inspectors and the Ministry of National Education • The establishment of successful food storage facilities at each beneficiary school • Training of school directors in food storage requirements • Training of school committees in inventory management • Secure and sanitary storage of food used in the program • Oversight by Food for Education facilitators • The transfer of USDA commodities to school facilities • The organization of volunteers to oversee and maintain food in storage • The organization of local volunteers into groups of cooks that prepare food for the students • Raising funds and food contributions from parents to compliment the commodities provided through the program. Directors, teachers, and local volunteers report that schools receive food on time and have what they need to provide daily meals to students. Schools provide lunch on different schedules. Primary schools only have afternoon classes on Tuesdays and Thursdays. Those schools provide breakfast every day of the week, and lunch only on Tuesdays and Thursdays when kids stay through the afternoon. Food Acquisition and Community Contribution The school feeding program relies on USDA provided commodities (Milled Rice, soy fortified cornmeal, Peas, lentils and vegetable oil). Communities also contribute cash/money and locally produced food to compliment the commodities. As discussed under Activity 6 in this document, USDA’s commodities are well managed. The schools are consistently adequately supplied. Food loss is rigorously accounted for. School management volunteers demonstrated in interviews that they are well versed in proper storage and distribution protocol. Community contributions are not managed quite as rigorously. Parents and communities make contributions in two ways, through food donations and cash donations. One issue that was apparent from interviews was that there is no set standard for community contribution requirements. There are three variables to the school feeding program; the amount USDA will contribute, the amount the government contributes (approximately $1.30/student/year26), and the gap that each community should contribute to make sure students have a balanced diet. However, the evaluation team found that community contributions varied greatly from school to school. Some schools had no trouble acquiring 1 kilogram of rice per student. Others, particularly in nomadic communities, rarely received any food from the community. At some schools students contribute cash, food, or cash and food. The amounts of cash that parents contribute for their children ranges from school to school. Amounts discovered during interviews ranged from 50 francs a day, or 1000 francs/month if the student attended every school day in a month to 2000 francs per month. Several issues came up during interviews and focus groups regarding the reliance of the program on community contributions. The first is that families in wealthier areas have an easier time making contributions than families in poorer areas. It appears from mid-term evaluation team observations that wealthier areas were also more likely to have access to a source of water, community farm, and are easier to access to construction materials. Nonetheless, USDA commodities are distributed to schools evenly so students at poorer schools end up with a less diverse diet. 26 Figure from interview with school feeding official in the Ministry of National Education. 31 Lastly, several teachers and directors expressed concern that the contribution requirement discouraged poorer students from attending school. PA participants said that, “kids can be shamed if they can’t contribute and others can.” Meal Preparation and Delivery Food for Education’s facilitators work with schools to organize parents into school feeding management committees. These committees recruit mothers in the community to serve as volunteer cooks. The volunteer cooks serve in groups of two or more. Each group prepares lunch for the school for a week at a time on a rotating schedule. For example, for a school that has six groups of volunteer cooks, each group will prepare lunch for one week out of every six. Through interviews and focus groups with students, the mid-term evaluation team found that the food was popular with students and parents. Students often said that they prefer the food at school to what they ate at home. Lentils are particularly popular. Directors report that the popularity of the food has increased attendance considerably. Examples include the school, where enrollment has increased by 30 students, the preschool, who said that enrollment has approximately doubled since before the program’s implementation, and the EE de school, where the director mentioned that his CE level classes went from 20 students to 47 students per class. One school director reported that he even had one student from Mauritania now, which he attributed to the school feeding program. In addition to increased enrollment, the school feeding program has improved the quality of attendance. This improvement takes three forms. First, students from poor families would often eat very simple breakfast, or no breakfast. These students now receive sufficient food at school, which improves their attentiveness. Parents, directors, and teachers reported that the students were healthier and now have more energy for learning. Additionally, students would often go home for lunch. Teachers report that few of these students would return for afternoon classes. Now, students have no reason to leave so they nearly universally stay for afternoon classes. Kitchen Requirement The original school feeding program plan required that schools provide a kitchen for preparation of school meals to qualify for the program. The evaluation team did not find any primary schools that possessed a kitchen, although several pre-schools did. At primary schools, cooking happened either outdoors or in a temporary shelter. Activity 9: Establishing Community Farms The Food for Education project established twenty community farms. The purpose of the farms is to support the school feeding program by allowing community volunteers to grow food that will complement the commodities donated through Food for Education. Communities also believe that farms have income generating potential for the schools. The community farms are a popular component of the program. A broad spectrum of stakeholders see the farms as a key piece of the school feeding program’s sustainability. The deputy governor of Saint Louis said that the establishment of community farms creates independence for the schools in several ways. Most clearly, the farms produce food that will sustain the school feeding program. Additionally, he felt that the community management of the farms, which requires community members to put resources into working the farm, focuses community attention on sustaining the school feeding program. The deputy governor also felt that community farms had income generating possibilities, such as through the establishment of poultry farms. EXHIBIT 8 A CPI COMMUNITY FACILITATOR INTRODUCING EVALUATORS TO A COMMUNITY FARM VOLUNTEER 32 The mid-term evaluation team also met with officers at the Dagana department of agriculture. The department of agriculture worked with the Food for Education program to provide training for community farms. The officer we spoke with pointed to several aspects of the community farming program that work in its favor. He mentioned that a failure of other community farming projects was the requirement that the volunteer farmers be paid, either in cash or crops. The Food for Education volunteer farmers are dedicated to the mission of feeding the school, and the department of agriculture officer believes their dedication will continue. He also believed that checks on resources donated to the farm were appropriate.27 Activity 10: Improved Teacher Attendance Under this activity Food for Education was tasked with promoting teacher attendance through the reduction of administrative duties. The activity requires organizing advocacy workshops that would result in the reduction in time and resources teachers required to collect their salaries. As of the mid-term evaluation, The Program had conducted two workshops; one that hosted 25 participants from IEFs, the Regional Agency for Early Childhood and the Case of the Petits, and one that hosted 45 teachers from Dagana and Podor. The mid-term evaluation team asked teachers whether they felt their administrative duties were onerous. The team also asked teachers in urban and very remote locations whether they had any difficulty collecting their salary. None of the teachers that we interviewed indicated that they had any problem in these areas. The evaluation team also asked parents, students, teachers, and directors whether teacher attendance was a problem at their school. While teachers are absent from time to time, no interviewee indicated a concern about teacher attendance. The school feeding program is at least a component of good teacher attendance. When asked why teachers had such a good record of coming to work, one teacher responded, “If the teacher isn’t here, the kids don’t eat!” Activity 11: Parent Association Training Food for Education relies heavily on community volunteers. The program creates parent associations to build community ownership of the program, keep the program aware of community needs, and ensures transparency among program managers, the school, and the community. The Associations also serve as a means through which schools set parent contribution policy and communicate that policy to community members. At most schools, the parent association takes the form of the School Management Committees. PAs were formed in 2016. Between November 2016 and January 2017, 526 PA members received training in governance, leadership, and management. In interviews, members of PAs expressed thorough knowledge of the Food for Education program, the issues that it aims to address, the benefits it brings to their community, and the difficulties it faces. Engaging parents in PAs improves their knowledge of the value of education. Parents expressed the idea that having better educated children in the house improves the level of knowledge of the whole household. One parent exclaimed “When they learn, I learn,” when asked about the benefits of education. PA members also expressed that education expanded children’s hopes for the future. PA members frequently expressed the idea that better education could lead to children becoming “a pilot or a doctor.” 27 At the time of field data collection, the farms that the evaluation team visited had either just planted crops, had very young plants, and/or were producing very small amounts of greens and okra. The evaluation team did see that granaries housed donations from communities, but it was unclear if these came from private or community farms. 33 The idea that education could improve quality of life in the future found its way into children’s minds as well. Children were asked in focus groups “what do you want to be when you grow up.” These children frequently mentioned possibilities such as school director, doctor, president, teacher, and others. The profession that children listed often did not exist anywhere in the school’s community. Children provided these responses no matter how wealthy, poor, urban, or rural the school, demonstrating that students saw education as a clear component of improving their prospects for the future. Monitoring and Evaluation of Program Activities Food for Education appears to have an excellent and efficiently functioning monitoring and evaluation system. Facilitators work with community members, school directors, and teachers to collect data important to the program. This data includes all output level data reported in semi-annual progress reports. The evaluation team met with the program’s monitoring and evaluation officer to test their system. The Monitoring and Evaluation system could quickly bring up information about specific students in randomly chosen schools and could quickly show progress on program activities. Project stakeholders, such as school inspectors, directors, and teachers were happy to produce information on the state of schools and students when asked. The Monitoring and Evaluation system is well run and informs decisions made by program managers. Two examples demonstrate its usefulness. First, the Monitoring and Evaluation system assisted in capturing information regarding the energy efficient stoves (see activity 4), and allowed program managers to rethink their plan and redesign the activity so that it still had an impact on the program despite the initial plan’s failure. The second is in the connection between the Food for Education Country Director and the ground level facilitators. FFE’s Monitoring and Evaluation system allows executive staff to see program performance, identify specific trouble spots to the school level, and to work with facilitators to identify means of addressing any problems that arise. While the system runs well and is up to FFE’s Monitoring and Evaluation task, there are two areas for which it does not capture information that would improve management’s knowledge of the program and program results. The first area for which the system does not collect any data is information on kids that do not attend school. The quality of the program’s success would be more meaningful if student and school achievement were comparable to the whole, local potential student population. For example, schools that have seen enrollment double success would be more meaningful if the program knew that that the increase in local enrollment meant that the total population of kids who do not attend school was halved. Further, if Food for Education knew why those kids were not attending school, it might influence the design of interventions. Secondly, the system misses differences in school status, such as relative level of wealth, stability of the local population, and needs. As this document discusses in the evaluation questions and recommendations sections, the programs one-size-fits-all approach to construction, sanitary facility needs, and food distribution often results in the allocation of limited resources to schools that do not maximize their value. For example, relatively wealthy schools whose stakeholders have an easier time contributing to the school feeding program may not need as large a USDA donation as schools located in areas where water and farmland is scarce. Conversely, some construction funds may be better spent repairing hole￾ridding roofs at several schools rather than building a new classroom for one school. Three indicators and targets that merit reconsideration. The first is “100 percent of school age children receive a minimum acceptable diet at the school level.” The current measurement of this indicator is 43 percent. Many very rural communities have little or no access to water and contribute little to the school feeding program. It is unlikely that students in these areas will all achieve this goal. Food for Education should either redefine what it is trying to achieve or find ways to supply a diverse diet in addition to USDA donated food and the scant community contribution. 34 The second indicator is “90 percent of schools have energy saving stoves.” Instead, Food for Education should determine which schools would benefit the most and allocate more than one stove to those schools. For example, preschools, which use gas stoves, or schools that have plentiful access to other fuels may not require the stoves. The stoves could then be distributed to large schools that require more than one stove, or schools in particularly environmentally sensitive areas. Last is 20 percent increase in teacher attendance rates compared to baseline. The baseline found that teachers were present at school over 94% of the time. Teacher attendance does not appear to merit the attention that it may have during the project’s design. Evaluation Questions Implementation and Relevance Implementation compared to design: The Food for Education program’s design is its strongest characteristic. As the Officer in charge of school feeding at the central government’s department of education pointed out, the program covers the whole system around using improved nutrition to increase literacy. Rather than simply delivering food to schools and make demands on teachers, the program recognizes that delivering donations and training into a community has the potential to create burdens in the community if there aren’t structures in place to manage all aspects of the program. Food for Education sees that schools are prepared to receive, store, and prepare food and that they contribute to the program to ensure local ownership over achievement and outcomes. The program also realizes that if people are going to eat, they will also need a latrine to relieve themselves including the sanitation infrastructure that goes along with latrines. Program interventions are designed to minimize impact of new activities that demand fuel on the local environment. Teachers and school directors praised the high quality, practical, and professional training that they received. Parents recognized the improvement in their children. Perhaps the best source of evidence of the program’s success in design and implementation is the results it has achieved in student attendance, student literacy, and all stakeholder’s improvement in knowledge about nutrition and hygiene. While Counterpart has implemented the program just as they have designed it, the organization’s management has also responded flexibly when activities have required adjustment. Examples of these types of adjustments include redesigning the energy efficient stoves when it was clear that the initial plan was insufficient and striking the kitchen requirement from school participation after receiving evidence that it wasn’t a practical criterion for school eligibility. An example Counterpart’s systematic vision includes exercising restraint in activity implementation where there are no options for sustaining the activity after the program ends. Thus, Food for Education does not provide soap to schools for handwashing because schools could not provide the soap themselves if the program were to end. Similarly, toilets do not include water connections where communities would be unable to maintain them. Counterpart’s judgment in these program components is that it is not beneficial to the community to set up unsustainable assets even if the assets might produce short-term benefits. The evaluation team believes this judgement is correct, with the caveat that Counterpart should be careful not to use “unsustainability” as an excuse to not looking for creative solutions to difficult problems. Even if soap cannot be sustainably supplied in the short-term, it is still a key component of hygiene. What possibilities exist for procuring it in the longer term? Design compared to needs of beneficiaries: In the most important respects, the program design reflected the needs of beneficiaries. As the projects results at mid-term prove, delivery of food matched with appropriate training does deliver the program’s intended results. The Senegalese government, which is used to responding to requests for assistance, has benefited from the demonstration of an organization taking the lead to organize communities and achieve goals. Food for Education has taught teachers and directors how to organize and leverage resources in their communities. As one director said when asked if 35 the project could sustain itself, “we know how to do this now. We have food contributions and meals are cooked, we could carry on if the program ended.” There are few areas where activities are not as relevant to their stakeholders needs as they should be. The clearest example is around teacher attendance. In interviews, no teachers agreed that the administrative burdens that this activity is designed to relieve were a problem. Likewise, we did not find any school directors that said they struggled with teacher attendance. The second example is school construction. Certainly, several schools required new classrooms. However, a number of schools that we visited would benefit only marginally from new classrooms space. All of the primary schools that the evaluation team visited had large holes in their roofs, often in several places. Schools that require space for many students would benefit much more from new roofs than a new classroom, which would provide adequate space for a portion of the students leaving the rest subject to the elements. School directors often said that, rather than have new classrooms, they would prefer that their current infrastructure be repaired, or other types of infrastructure, such as fences, be constructed. One area in which project design did not meet the needs of many beneficiaries is stakeholder communication. Many stakeholders, and particularly school directors, lacked crucial information about the program. Several mentioned that they had put in requests for infrastructure but didn’t know the status of their application. Some that had schools where infrastructure had begun did not know the timeline for completion, which had a direct impact on students that wanted to attend class but did not know if there would be adequate space available. Other directors did not know protocol around food use, such as how frequently students were supposed to receive USDA contributed rice or lentils. Most importantly, national government officials felt that they needed more information from USDA if they were to take over the project at some point. National government officials in the department of education said that they fully intended to assume ownership over the program, but that the transfer required time and resources, for which they would have to lobby their government. We cover this topic in more detail in the sustainability part of this section. Program Relevance given the economic, cultural, gender, and political context: As national government officials pointed out, the Northern section of Senegal is a difficult place to work and in dire need of assistance. The program provides food and education in a region that is short on both of these assets. The program is in line with the national governments health, nutrition, and education plans. Counterpart’s leadership has coordinated diligently with government and political entities to ensure support for the project. One area where the project’s relevance requires further research is in its handling of gender issues. There are many touch points between gender, nutrition, and education that bear exploration. First is the burden of the unpaid work that the program creates for women in Food for Education communities. Women in West Africa suffer from a heavy work load, much of it unpaid care work in service to their families and communities. The Food for Education project creates a new layer of unpaid care work that women shoulder. Food for Education should explore the seriousness of this burden and whether or not there is some means to ameliorate it while still ensuring the successful implementation of the program. EXHIBIT 9 A TYPICAL CLASSROOM VISITED BY INTERVIEWERS, CLEARLY IN NEED OF ROOF REPAIR 36 The second important issue related to gender is the higher school enrollment of girls than boys. The proportion of girls (60%) to boys (40%) is true across Senegal28 when the mid-term evaluation asked parents, teachers, and directors for explanations of this imbalance we received a variety of opinions. First, we noted that the imbalance isn’t true for all schools. Some directors could demonstrate in their attendance records that schools comprised 50 percent boys and 50 percent girls. Where the imbalance existed, explanations included that boys were required at home to help breed animals, that a national campaign to promote education for girls meant that boys felt it wasn’t as important for them, and that boys were more likely to get frustrated and drop out of school when they struggled with reading or math. The issue merits further research The project’s accounting for the current government strategic framework: The project is in close coordination with national, regional, and local government agencies. The mid-term evaluation team interviewed high level officials in the department of education and school health as part of the evaluation, as well as the Deputy Governor for Saint Louis, and local school inspectors. All officials affirmed that the project matched their approach to improving student performance through a systematic approach that included school feeding, improved infrastructure, improved teacher performance, and community engagement. Effectiveness and Performance The root causes children did not enroll or attend school and root causes for dropout rates: The mid-term evaluation did not have the resources required to understand the root causes that prevent children from enrolling or attending school. However, the team did uncover some of the causes through interviews and focus groups. Parents, teachers, and directors pointed to a number of reasons why children did not enroll or attend school prior to the Food for Education program. These included that they were needed at home to work on the farm, or that they lived too far away. Almost every school we visited said that before the program students would go home for lunch and not return for afternoon classes. While comprehensive reasons for low attendance are unknown, nearly all teachers and directors agreed that that the school feeding program was creating an incentive to come to school. Teachers and directors report that drop outs are less frequent, attendance is more reliable, and enrollment is up. Some directors mentioned that boys were more likely to drop out than girls. Sometimes boys drop out to attend to duties at home, such as animal rearing. Teachers and directors also mentioned that boys are more likely to drop out because they become frustrated and embarrassed at their inability to read. Changes in student enrolment, attendance and drop-out rates since the start of the project: As mentioned in the SO1 section of this document, enrollment and attendance have increased. Increasing student literacy outcomes: As noted in the SO1 section of this document, student literacy has significantly improved. Beneficiary perception of community farm activities: Stakeholders from the national government to community volunteers have a positive view of the community farms and are optimistic about their ability to contribute to school feeding program sustainability. The officer in charge of school feeding at the national department of education sees the farms as an important step toward community engagement and independence. School directors that have access to farms put a lot of faith in them. For example, one school director of a larger school in Dagana believes that the farm will provide for all of the school’s rice needs. School 28 World Bank data for Senegal shows that about 54% of males and 60% of females complete primary school (2015 data) 37 directors and government officials also believe that farms will provide income generating possibilities through the sale of vegetables or the establishment of poultry production. Improved student knowledge and behavior about nutrition, health, and sanitation: Please see SO2: Improve health and dietary practices results and Activity 5: Good health and Nutrition Practices. Beneficiary perception of Moringa plantations: Please see Activity 7: Provide Access to Health Interventions. Teacher and mothers’ knowledge and behavior about nutrition, health, and sanitation: This document addresses teacher knowledge and behavior under Activity 5: Good health and Nutrition Practices. To the extent that the Food for Education program addresses mother’s behavior, it does so in mother’s capacity as volunteer cooks or as part of SMC which this document addresses under Activity 6: Training in Food Preparation and Storage. Otherwise the FFE program does not serve mothers as a direct beneficiary group. Unintended or indirect impacts: As a result of the school feeding program, there were several unintended or indirect impacts. These are: • The school feeding program has increased school enrollment and attendance. One consequence of this is that student/teacher ratios have grown. Many schools serve 50-80 students now with only one or two teachers. The average student/teacher ratio in Senegal for 2015 was 3229. Several schools that we visited said that they had to turn students away because of lack of teachers and classroom space. • At several schools that the mid-term evaluation team visited, students appeared older than usual for primary school students. Teachers mentioned that students will lie about their age to qualify for primary school. Many times these students are attending school for the first time. Other times they are returning after an extended absence. Teachers believed that they are attracted to school both for a chance at education as well as access to food. Across Senegal about 7% of primary school students are overage30. • At some urban schools, food vendors would sell food to kids at lunch and after school as a snack. These vendors have disappeared from schools that have the school feeding program. • A number of teachers speculated that that FFE schools are drawing students away from schools that do not have school feeding programs. This unintended impact merits further research. Efficiency Project return on USDA investment: Return on USDA’s investment is very difficult to calculate. The most valuable product of the investment so far is a generation of students in the Saint Louis region that are becoming literate and believe that working for an education greatly increases the possibility of a brighter future. The value of that certainly exceeds the value of the commodities and the time USDA dedicated to the project many times over. In terms of output-level measurements, USDA’s investment has resulted in the following key results thus far: • 44,912 students enrolled in school, of which 26,523 are girls and 18,389 are boys. These students have near perfect attendance and have access to nutritious food, a rarity in many of their communities. They have also significantly improved in literacy compared to before the program launched. • 539 people trained in food preparation and storage, improving the health of students and communities. • Over 10 million meals served to children so that they have the energy to succeed in school. 29 World Bank Databank – World Development Indicators for Senegal 30 Ibid. 38 • 44,095 children have received de-worming medication and/or Vitamin A supplements, making them healthier, stronger, and more attentive in class. Could other implementation strategies have achieved more with the same resources? Counterpart’s overall implementation strategy is effective and efficient. Counterpart does an excellent job of ensuring that stakeholders at all levels are engaged and informed, and that the program is implemented with a complete school feeding system strategy. There are some adjustments to the program that might improve its efficiency, as mentioned in other locations of this document: • Food and stove distribution: Rather than distribute commodities and stoves equally to all schools, Food for Education should consider distributing based on need. In this system, poorer school communities that have a harder time contributing to the program would receive more food or equipment, where wealthier schools that are more capable of sustaining themselves based on community contributions would receive less. • Construction: While some schools need new classrooms, a significant number would benefit more from repairs to current structures, particularly roofs. Food for Education could improve the learning environment for more students by setting aside some construction funds for schools that require repair or maintenance. Sustainability Steps/actions required to realize full sustainability of activities beyond the life of project: At the highest level, ensuring sustainability requires improved transparency among USDA, the Senegalese government, and Counterpart International. Government officials explained during mid-term evaluation interviews that they fully expect to take ownership and responsibility for the program when it ends. However, assuming responsibility takes time. Responsible government officials must rally support in their legislature, provide the President with the information and materials he needs to raise a budget, and put the structures in place to manage the program. The school official in charge of school feeding at the department of education estimates that this process will take 3 to 5 years. This official mentioned that the sudden conclusion of the program in Matam without adequate advanced notice for the government is one of the reasons the project died so abruptly. USDA should use this benchmark to plan for a smooth transition to Senegalese ownership of the program. A second step involves calculating a detailed estimate of what the program’s costs will be after the program ends. Currently, USDA’s contribution is known. The Senegalese Department of Education dedicates approximately $8 per student per year, of which $1.30/year goes to school feeding. The rest is made up through other means such as school farm production or community contributions. These figures should serve as a basis for estimating the project’s cost after it ends to provide more accurate guidelines and expectations for stakeholder groups at all levels. The program should also measure the work burden that the program produces for women and school directors, and identify means of reducing the work burden. This should involve encouraging men to devote more time to volunteering in addition to the money or food they contribute. One possibility might involve encouraging men to provide cooking fuel to schools to reduce women’s burden of collecting firewood. Another might be for men to work more closely with school directors to more easily oversee aspects of the school feeding program so it does not interfere with their other school management duties. A final important step involves planning for the sustainable management of the volunteer component of the program. Currently, volunteers run most aspects of the program. They do so in close coordination with Counterpart’s facilitators. Even with the facilitator’s regular oversight, school results in terms of food storage and preparation, acquisition of community contributions, and facilities management are less than perfect. There is a strong possibility that the Senegalese government entity that assumes ownership of the project after Counterpart’s involvement is complete will not have Counterparts expertise in 39 volunteer management. As part of its transition plan, USDA should conduct a cost benefit analysis for the Senegalese government to calculate whether maintaining the program as largely volunteer run is worthwhile given the losses and quality control difficulties it implies, or if the government should offer communities and school directors incentives to assume the program’s duties. The extent to which project activities will continue with the absence of support from both USDA and Counterpart: The majority of Food for Education schools could maintain the school feeding program on some level if USDA and Counterpart were to withdraw. Their ability to maintain that component of the program depends on how effectively USDA and Counterpart manages the transition, as stated in the response to the previous question. In interviews, school directors, teachers, and parents expressed a belief that they could manage the program on their own. They recognized their new capacity as planners and managers, and believed their communities were dedicated to maintaining the program. With clear communications and time taken to plan and train the program’s new owners, the feeding activities have a high likelihood of continuing. Other program activities will be more difficult to sustain. Regular pedagogical training, for example, will require initiative from school inspectors and a leader to maintain the program. If Counterpart can identify that leader and work with him/her to plan for continued updates to the professional skills of teachers in the region, the activity has a reasonable chance of sustaining itself. This outcome requires clear communication about the programs progress and end date. The government’s School Health Office is interested in assuming ownership of the program’s health related activities. Food for Education deworming and hand washing campaigns are similar to campaigns that the office already runs. However, as in the case of the central government, time and clear transition plan are required to successfully hand the program over to the School Health Office. Recommendations As the prior sections of this document has shown, Food for Education is a successful project. It has improved the literacy of primary school students, improved knowledge and practices of hygiene and nutrition among its stakeholders, built the capacity of schools and communities to sustain school feeding programs, and brightened the future prospects for tens of thousands of children. The mid-term evaluation team’s primary recommendation is that the program should keep doing what it is doing. That said, every program, no matter how successful, can be improved. In this section, the report lists issues that Food for Education should consider under some activities and then recommends eight priority actions that the program should take. Issues for consideration: School Selection Target a balanced portfolio: The Food for Education program in Senegal serves a wide range of schools across different demographic characteristics. Some schools are in relatively wealthy areas that have access to water and farmland, while others are extremely poor with very limited access to natural resources. Some schools are located among stable farming populations, while other schools see their enrollment fluctuate with the movement of nomadic peoples. Serving a diversified population is a good management strategy as it could allow sharing of resources among differing school needs. As is discussed throughout the document, Food for Education must change some of its one-size-fits all approaches to program activities to take full advantage of this balanced portfolio approach. Activity 1: Capacity building at the local, regional, and national level Continuing Pedagogical Training: Food for Education’s training in May 2016 covered 813 teachers. This training emphasized systematic sustainability in that it empowered inspectors, who have the resources to conduct the training at regular intervals, with the skills needed to replicate the training repeatedly without 40 further input from Food for Education. The Deputy Governor of Saint Louis who agreed that training inspectors instituted local government “autonomy in implementing the program31.” Allowing trainees to provide feedback about training, and incorporating that feedback into future training sessions, is an important component of professional training sustainability. Seventy-five percent of the teachers we surveyed said that they had the opportunity to provide feedback regarding the trainings that they had received. This result shows Food for Education’s focus on sustainability of capacity building, though it should consider how to move that number closer to 100 percent. The challenge for the Food for Education program and the educational system is to continue providing high quality training. Food for Education’s initial training built the skills of 816 teachers. As of the mid￾term evaluation, 1,411 teachers work at primary schools that participate in the Food for Education program in Saint Louis, meaning that 595 teachers did not benefit from the training either because they are new to the system or for some other reason could not attend the inspector-led training. 34 percent of teachers that responded to our survey said that they had not received training this year. Food for Education should work with educational authorities to create a protocol and incentives for continuing to regularly conduct trainings and to ensure that their quality is maintained. Activity 2: Construction of school infrastructure Flexibility with Construction Resources: Building classrooms for the neediest schools is an important contribution to improving student educational achievement. However, it may not ensure that the resources are used to efficiently serve the greatest number of students. Many schools require roof repair to maximize use of already sufficient space. Repairing the roof of one classroom costs approximately $24032. Some schools had infrastructure that was constructed relatively recently, but was heavily degraded. The degradation was due to high winds coming across desert areas and sandblasting concrete buildings. In the case of these schools, a protective wall may be a more efficient expenditure of funds than new facilities which high winds would quickly damage. In future iterations of the Food for Education project, Counterpart should determine the number of its schools which most direly need new classrooms to serve growing student populations. If that number is less than 43, the current target for new classroom construction, the remainder of the projects funds should be allocated to new infrastructure and maintenance needs that would serve the greatest number of students. Activity 3: Construction of latrines and water station systems Latrine Maintenance: At the schools that Food for Education supports, a high percentage of latrines are insufficiently clean and structurally maintained. Food for Education did not build the great majority of these latrines. To fully realize the benefit of having latrines at school for children’s hygiene and education, Food for Education should make sure that school administrators and the larger community are engaged in latrine maintenance and cleanliness. This engagement should happen at schools that receive new construction from the program and those that do not. 31 Interview with Deputy Governor of Saint Louis – 24 May 2017 32 A 10-pack of corrugated metal sheets costs approximately 33,500 CFA, or $60. A school director that was raising money to repair his school’s roof reported that 4our packs are required to roof a typical classroom EXHIBIT 10 THE BLACKBOARD IS ALL THAT IS LEFT OF A CLASSROOM DESTROYED BY THE WIND 41 That said, there is a good deal of social stigma attached to cleaning sanitary areas around the world. The Food for Education community may find that relying on volunteers is not a sustainable way to ensure maintenance and cleanliness. If this is the case in Food for Education communities, the program should utilize a portion of Food for Education community contributions, either in the form of food or money, to create incentives or a part time job for an available community member. Ensure equal access to latrines: Seventy-five percent of schools have equal access to latrines for girls and boys. Still, 25 percent of students don’t have equal access. Food for Education should work with school administrators and communities to remedy this issue. Activity 4: Equip schools with energy saving stoves, canteen equipment and materials Not enough of a good thing: Where the stoves are successful they save cooks time and effort both in reduced time required to collect firewood and time required to cook meals. However, one stove per school is insufficient to prepare a meal. Food for Education should provide a sufficient number of stoves to fully reduce wood collection and cooking time. Some schools may not need stoves: Schools that use gas for cooking may not require an efficient wood burning stove. The stoves that would be allocated to these schools might be better used as a second stove for large schools that use wood to cook. Create attachments to accommodate different cooking tools: Clearly, creating tailored stoves for each cooking purpose is not possible given the program’s budget and complexity. If USDA is considering a second phase that will serve these same schools, it should evaluate whether it is possible to create a stove attachment that would allow cooks to use the various sizes and shapes of pots and pans that the cooks commonly utilize. Activity 5: Good health and Nutrition Practices Turning knowledge into practice: Food for Education has performed excellently teaching directors, teachers, and students about hygiene and nutrition. In interviews, teachers and directors could easily pull out their copy of the SH&NG, and show the sections that they found most useful. Students could easily discuss nutritious foods, ways to avoid getting sick, and why hand washing is important. However, turning hygiene and nutrition knowledge into practice is notoriously difficult. Even in developed countries, such as the United States, only 67% of people wash their hands sufficiently33. As difficult as it is, there are ways that Food for Education could improve its results in the area of hand washing. First, finding a sustainable supply of soap for schools would help a lot. As the Food for Education Country Director mentioned in his interview, it is not worthwhile to provide soap through the program only for the supply to end when the program is turned over to the government. While that is true, washing hands with only water is also not terribly worthwhile. Food for Education should work with communities and governments to find a supply of soap that is affordable and accessible. Secondly, hand washing facilities need to be more accessible to students, so that they can wash their hands after they relieve themselves or play in areas contaminated with animal feces. Designating a location outside of a storage area or classroom to keep basins and water designated for hand washing could increase hygiene considerably. 33 http://www.prnewswire.com/news-releases/global-handwashing-day-focuses-on-need-for-universal-hand￾hygiene-300159521.html EXHIBIT 11 HAND WASHING IMPLEMENTS STORED OUT OF REACH 42 Activity 6: Training in Food Preparation and Storage Practices Planning for Handover: Counterpart has built a tightly managed system for managing food storage and ensuring that responsible volunteers and school staff are properly maintained. As part of its sustainability strategy, Counterpart should develop a plan for handing this component over the government. The plan should include details about the systems that the government will require as well as new staff that it will have to put in place. Counterpart should also prepare a cost benefit analysis to assist the government in deciding whether continuing volunteer management is more or less effective than having professional staff manage storage and inventory. As the baseline survey showed, requiring that the program be managed through adding additional duties to already overtaxed school directors, and relying on volunteer performance means that about 75 to 80% of schools will manage their inventory and storage correctly. Activity 8: The school feeding program While the school feeding program is clearly exceeding its goals, there are still several issues that merit attention. Unequal needs: The level of poverty in communities that Food for Education serves varies considerably. Some, primarily those close to urban areas, are relatively wealthy, have decent infrastructure, and more ready access to resources. Others, particularly those in very rural areas, have hardly a basic structure for holding classes. Despite these unequal needs, the program calculates students’ needs uniformly, distributing the same amount of food per student to each school and determining community contribution requirements as if all schools and students were equal. However, poorer schools have much greater needs and participation in the program creates a much larger burden. Poorer schools are less likely to be able to contribute fresh food to Food for Education’s commodities, so the total food their students receive is less. It is also much more difficult for poorer schools to acquire the wood and water needed for cooking. In severe cases visited by the mid-term evaluation team, the villages where schools were located hadn’t had access to water for several weeks. The program should allocate more resources to poorer schools while helping relatively wealthy schools make efficient use of their resources. Inconsistent community contribution requirements: Schools asked families to contribute money for each student that took advantage of the school feeding program. For the schools at which we conducted interviews, these amounts ranged from approximately 1000 francs to 1700 francs a month. These contributions caused some consternation in communities, especially where a portion of students felt shame because their parents could not pay. The Food for Education program should provide guidelines to school about limits on these requirements and ensure that the program contributes enough to keep community cash requirements low. Unequal distribution of volunteer work: The school feeding program’s success relies heavily on volunteer cooks. These cooks are always women. Women in West Africa already bear much of their family’s and community’s unpaid care work. The Food for Education program should explore how it can encourage their participation in school feeding, without worsening their workload34. Some communities had difficulty recruiting volunteer cooks. For example, at the Niary School in Dagana that had 76 students, the volunteer cook mentioned that she is often the only cook that shows up because the other women in her group say that they have no time. Another pre-school mentioned that it had two cooks for 50 students. These two cooks rotated, each cooking every other week. 34 The mid-term evaluation team notes that the program did not increase the work burden for all women. Women with polygamous marriages that the team interviewed noted that other wives took care of the housework while they cooked at school. 43 Communication with stakeholders about food use: Volunteers and school administrators frequently mentioned that the rice they received was insufficient. They often ran out very early in the month. They wondered why their supply from the program was so low. The Food for Education program has a policy that the schools should serve rice only twice a week, but most school food managers and directors were unaware of the policy. The mismanagement of rice is one example of miscommunication in the program, a theme that the evaluation will discuss in greater detail below. The kitchen requirement: The evaluation team does not believe that it is practical to require schools to have a kitchen as a prerequisite for program participation. Though they would make safety and sanitation easier, it is uncommon for schools to have kitchens and building one to participate in the program would be an onerous cost on any school. Most schools have other priorities, such as repairing porous roofs or treacherous walkways. Activity 9: Establishing Community Farms Community Farm’s Intended Contribution: An outstanding question is how the farms could best contribute to the sustainability of the school feeding program. The likely productive potential of a community farm should be possible to calculate. However, stakeholders were confused about the percentage of a school’s food that would come from the farm. Also, there was uncertainty about whether, when the farms were fully operational, whether they could contribute to a single school or many schools. While Counterpart estimates that there is potential for more than 100 farms, the Program currently has funds to develop only twenty. As schools had to meet specific criteria, such as access to water, that many schools could not meet, it seemed that more than one school should benefit from the farms produce. School’s that don’t meet criteria necessary to manage a farm should have options for other income generating activities, such as poultry production. Appropriateness of donated equipment: Each of the farms that the mid-term evaluation team visited seemed to have an issue with contributed equipment. One farm in Dagana was working with an irrigation pump that was too small for the field it was to irrigate. Consequently, it was frequently broken as the community pushed it beyond its capacity to water the entire field. Another farm had an irrigation pipe that was broken when they received it. The Food for Education program should ensure that processes are in place to review the appropriateness and quality of donated equipment. Activity 10: Improved Teacher Attendance Is teacher attendance insufficient? The evaluation team does not believe that encouraging teacher attendance through the means mandated in the grant agreement is relevant to the program. We believe that in 2011, when the program was initially designed, teacher attendance may have been a relevant problem that required addressing. However, between the time the program was designed and its actual implementation in 2016, the problem of inadequate teacher attendance seems to have diminished considerably. This appears to be the result of increased inspector focus on holding teachers accountable for teacher attendance and teachers having less of an administrative burden. Student to Teacher Ratios: School visits revealed that student/teacher ratios are a much larger problem. At the schools that the evaluation team visited, it was common to find two teachers for 70 or more students, and for one of those teachers to also serve as school director. These teachers usually had two to four grade levels in a single class. As the school feeding program grows enrollment, the student/teacher ratio is likely to grow. Food for Education should work with the government and communities to provide more teachers as the program achieves its goals. Activity 11: Parent Association Training Standards and Consistency: At this stage in the program, Parent Associations (PAs) seemed impromptu. At many schools that the mid-term evaluation team visited, it was unclear if the PA and school management committee were separate entities, or whether they needed to be. Some PAs were well organized with a clear balance between male and female members. Others seemed to divide responsibility 44 with women serving as cooks and men forming the management portion of the PA. Counterpart should consider whether it is necessary to have separate school management committees (SMC) and PAs, or whether they should exist within the same entity. It should also set some clear guidelines given school size and the complexity of issues the schools face. Priority Actions The following are the evaluation team’s recommendations for priority actions that the program should take, although we do not assign any order of priority to them. 1. Work with the Senegalese government to plan for the program’s sustainability: The Food for Education program has done an excellent job of coordinating program implementation with the government at all levels. These governmental partners require assistance and advice regarding the timing of the program and longer-term planning for the resources required to maintain it after USDA and Counterpart steps away. The sooner that the program begins this discussion, the more likely the Senegalese government will ensure Food for Education activities are sustained. The handover to the central government should include a plan for managing Food for Education systems, a cost-benefit analysis of maintaining the program’s reliance on volunteers, a description of where resources come from and how much communities should be required to give. 2. Conduct research into gender issues: Particularly, the research should focus on the obstacles that boys and girls face to attending school and achieving in school. Included in that question is why some schools have a much larger number of girls than boys and why, if true, boys drop out at a higher rate than girls. A second issue that requires attention is the burden of unpaid care work on female school volunteers and how greater participation of men and more efficient systems may lessen that burden. 3. Improve communication with stakeholders: This communication includes information about construction project requirements and timelines, community contribution requirements, why schools are eligible for specific benefits such as community farms, and why others are denied eligibility for certain benefits such as construction projects. 4. Ensure that teacher trainings continue: Teachers’ enthusiasm for the pedagogical trainings that Food for Education facilitated were an important contributor to gains in student literacy. The mid-term evaluation survey indicated that these trainings hadn’t happened since last year. Food for Education should make sure that trainings continue under the program, and that inspectors have a plan for continuing training after the program concludes. 5. Allocate construction and food resources according to need rather than equal distribution: As this document demonstrated, needs vary greatly across Food for Education schools. Some school benefit greatly from new classroom space, although a large number would be well served by some basic and possibly lower cost maintenance. Some schools are very poor and require plentiful food donations while others have an easier time accessing community contributions. The Food for Education program was designed to distribute these resources as equally as possible among eligible schools. However, a need based approach would make more efficient and effective use of Food for Education resources. As part of the needs identification process, Food for Education should clarify how much community contribution is required to meet the program’s goals. The specification of program costs for the government, Counterpart, and community contributors will provide a formula for the program’s sustainability. It will also set expectations for what communities can achieve given their ability to contribute and access to resources. 6. Systematize sustainable Latrine maintenance: Though Food for Education did not build all of the latrines at its beneficiary schools, nearly 87 percent of schools had latrines. More than 1/3rd of these latrines are not sufficiently clean or well maintained, reducing their contribution to 45 school attendance or the learning environment. Considering the state of many latrines and social stigma attached to cleaning latrines, it is unlikely that volunteer work alone will ensure that latrines reach an acceptable state. Food for Education should come up with a solution to this problem, which could involve providing stipends or incentives to community members that would appreciate the work. 7. Provide equal access to latrines for boys and girls: The majority of existing school latrines provide equal access for boys and girls. Still, Mid-term evaluation enumerators recorded that 25 percent of latrines were not accessible by boys and girls. This number is unacceptably high, though the solution is probably simply a matter of proper signage or scheduling. 8. Improve access to handwashing implements: Many schools have no access to soap which is a crucial component of proper hand hygiene. Food for Education had decided not to provide soap to schools because the supply would end with the program and schools would be unable to sustain soap procurement on their own. Still, the effort put into training students and teachers in hygiene is lost without proper supplies to put the training into practice. Food for Education should work with community members to find a creative solution to this issue. Additionally, schools that rely on water in containers and buckets for hand washing often store those implements where students can only access them at specific times, such as lunch, and not as needed, for example after using the bathroom or touching an animal. Food for Education should work with schools to ensure that handwashing implements are accessible whenever required. 46 Annex 1: Document Review List 1. 2016 Gallup World Poll 2. Activity and Indicator Report March 2017 - May 2017 3. Aser Center literacy testing methodology 4. Baseline Data Collection tools and Instructions 5. CPI FFE Senegal Annual Report 6. CPI Grant Agreement with USDA 7. CPI Performance Indicators Attachment E 8. CPI Plan of Operations Attachment A 9. CPI Senegal Baseline Attendance Analysis 10. CPI Senegal FFE Baseline Report 11. Data Collection Coverage Plan Preschool 12. Data Collection Coverage Plan Primary School 13. Evaluation Report FFE Matam 14. FAO Senegal Reports on Food Security 15. FFE Semiannual Performance Report April 20116 - September 2016 16. FFE Semiannual Performance Report April 2015 - September 2015 17. FFE Semiannual Performance Report Oct 2014 - March 2015 18. FFE Semiannual Performance Report Oct 2015 - March 2016 19. FFE Semiannual Performance Report Oct 2016 - March 2017 20. Food For Education Activity Management guides and protocols. 21. Food for Education Annual Workplans 22. Fuchs, Marek. "The Reliability of Children’s Survey Responses: The Impact of Cognitive Functioning on Respondent Behavior." Symposium 2008: Data Collection: Challenges, Achievements and New Directions (2008): 1-9. Web. 23. Memos of Understanding between CPI and Communities 24. MGD Indicator Framework 25. Monitoring and Evaluation Databases and Data Collection Tools 26. Performance ME Plan Revised 12/09/15 27. prnewswire.com/news-releases/global-handwashing-day-focuses-on-need-for-universal-hand￾hygiene-300159521.html 28. Raw Baseline Data 29. School Canteen Guide 30. School Hygiene and Nutrition Guide 31. School Selection Criteria Document 32. Senegal School Curriculum Guides 33. Social Progress Index Senegal Report 34. Staff Field Visit and Training Reports 35. World Bank’s Databank “World Development Indicators” database. Annex 2: Data Collection Tools 47 Annex 3: Sampling Methodology and Selected Samples by Type and Location Sample design Sampling follows a two-stage cluster design for students and teachers. The first stage is randomly selecting the schools, which are the clusters. The next stage is randomly selecting students within each school. For both student and teacher samples, the evaluation uses a confidence level of 95% and confidence interval or margin of error of 5%. With simple random sampling (SRS) for a large population, the sample size is 384. To account for the use of clusters, the design effect needs to be calculated. The formula for the design effect is as follows. Design effect = 1 + ICC(c-1) Where: ICC = The intra cluster correlation. This compares the correlation of elements within clusters to the correlation of elements between clusters. Here we use 0.15, which is higher (more conservative) than values found for literacy.35 This means that elements (in this case students) within a cluster are 15 percent more likely to have the same value on the key outcome (in this case literacy tests) as elements in different clusters. c = The average cluster size. Note that the smaller the cluster, the smaller the design effect which implies a smaller sample size. Therefore, to optimize budget, the intra-cluster costs (travel related costs for enumerators) must be weighed against the costs of additional interviews. We use a cluster size of 14 students per school and an ICC = 0.15. The design effect is then: Design effect = 1 + 0.15(14 - 1) = 2.95 Multiplying the design effect (2.95) by the SRS sample size (384) results in an n of 1,133. This is our student sample size. With 14 students per school, this means that we need 1,133/14 = 81 schools. Note that we are only interviewing students in primary schools. Our calculation for the teacher sample size is similar. Using a teacher population of 1,620 (6 teachers x 270 schools), we get an SRS sample size of 310. We estimate that we can interview 4 teachers per school, meaning the cluster size is 4. The design effect is below. Design effect = 1 + 0.15(4 - 1) = 1.45 1.45 x 310 = 450 this is our teacher sample size. 450/4 = 112, which is the number of schools. It is not logistically feasible to interview a statistically representative sample of directors. We will interview the director at all 112 schools in the sample, as available. 35 See for example: https://educationendowmentfoundation.org.uk/public/files/Evaluation/Writing_a_Protocol/ICC_2015.pdf 49 Annex 4: Survey Respondent Characteristics 52 St - Louis St -Louis Dept CTP Dagana Dagana MAT St - Louis St -Louis Com CTP St - Louis St -Louis Dept CTP Podor Pété CTP St - Louis St -Louis Dept CTP St - Louis St -Louis Com CTP Dagana Dagana CTP Dagana Dagana MAT Dagana Dagana CTP Podor Pété CTP St - Louis St -Louis Com MAT Podor Podor MAT St - Louis St -Louis Com MAT St - Louis St -Louis Dept CTP Dagana Dagana CTP Dagana Dagana CTP Dagana Dagana MAT Podor Podor CTP Dagana Dagana MAT New schools with sample revision St - Louis St -Louis Dept MAT Podor Podor CTP St - Louis St -Louis Dept MAT St - Louis St -Louis Dept CTP Dagana Dagana CTP Podor Podor CTP Dagana Dagana CTP 53 Annex 6: Evaluation Team Bios Mr. Jason Wares, Team Leader, Mr. Jason Wares, a Director of Programs at ISG, is an international development expert with 15 years’ experience specializing in catalyzing development and aid programming focused on livelihoods, competitiveness, and market development. Of most relevance to this project, Mr. Wares is currently leading baseline evaluation for ActionAid that is focused on Climate Smart Agriculture, women’s empowerment, and agriculture development. Other relevant experience includes an assessment of FINCA International’s Food for Progress (USDA-funded) projects in Haiti and Tanzania. Both assessments sought to understand how FINCA’s microfinance clients use loans to create sustainable microenterprises that translate into improved quality of life for their communities. Previously, Mr. Wares served as a senior economic development advisor at RTI International. At RTI, Mr. Wares led the institute’s business environment improvement technical area. Mr. Wares holds an MBA from the Thunderbird School of Global Management. Mr. Chris Root, Co-Evaluator Mr. Root is an agricultural economist with ten years of experience including in M&E, and impact evaluation. He recently worked with ISG leading the Final Evaluation of FINCA’s Tanzania program (USDA-funded) as well as for the Mid-term Evaluation of the McGovern-Dole USDA-funded program in Senegal. While working in Sudan, Mr. Root assessed the use of crop byproducts in livestock feed production. He made recommendations on how these byproducts, which are currently underutilized, could be better used to feed the country's huge livestock population. For a Somalia project he produced a brief on the camel milk value chain making recommendations on how a new project could work in the sector. While working at Land O'Lakes, Chris conducted research on livestock lending. Mr. Root has experience and expertise in propensity score matching (PSM) and difference in difference evaluation methodology. In Nepal he led a large agricultural value chain impact evaluation, conducting propensity score matching to identify impact. This built off his MA research titled "Assessing the Impact of Agricultural Value Chain Interventions" which developed the approaches he used in Nepal and built off his coursework. For RTI, he created guidelines on assessing the impact of economic development projects which detailed the PSM methodology. Recently at Michigan State he updated these skills through coursework on these quasi-experimental evaluation methodologies for his MS in Agricultural Economics. Dr. Safiétou Kane, National Specialist and Co-Evaluator Dr. Kane brings over 10 years’ of research experience in Senegal. This includes undertaking research and studies related to health, education and nutrition. Dr. Kane serves as a consultant for several relevant programs funded by USAID, including for the health and gender program evaluation in Dakar, and an education program evaluation as well. She recently worked with ISG on and evaluation of the Global Program on Increasing Accountability in Financing for Gender Equality for UN Women. Dr. Kane is a native French speaker and speaks fluent Wolof. Centre de Recherche pour le Développement Humain (CRDH) CRDH is a Senegalese research firm that provided research study design advice as well as 15 enumerators and 2 supervisors to conduct the mid-term evaluation’s surveys. The majority of CRDH team members have more than 20 years of experience in numerous countries where they managed multiple projects. Countries include Guinée Bissau, Cameroun, Guinée Conakry, Mali, Sénégal, Burkina Faso, Niger, Cap￾Vert, Togo, Bénin, Centre Afrique, Congo, Mauritanie, Guinée Équatoriale. In these countries, CRDH has conducted large surveys across a wide variety of technical areas. The permanent personnel of CRDH is about of 15 members composed of demographers, statisticians, an economist, a sociologist and data processing/analysing team. Three experts, working part-time on surveys, data processing and data analysis are also working on CRDH’s team. 54 Annex 7: Indicator Framework – Baseline vs Mid-Term Data Strategic Objectives Indicator Indicator description Midterm result Baseline result SO 1: Improve Literacy of School Aged Children Percent of students, who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text28 All 60 Boys 66 31 Girls 57 33 1.2.1: reduced Short-Term Hunger Percent of students in target schools who indicate that they are not hungry during the school day All 69.88 66 Boys 67.74 Girls 71.15 1.2.1: reduced Short-Term Hunger Percent of students in target schools who indicate that they are not hungry (or hungry from "time to time") during the school day All 88.6 66 Boys 86.85 Girls 89.64 1.4.3: Increased Government Support Percent of teachers who received government supported training in pedagogy in the past three months All 13 28 SO 2: Increased Use of Health and Dietary Practices Percent of school-aged children receiving a minimum acceptable diet at the school level All 43 2 2.2: Increased Knowledge of Safe Food Prep and Storage Practices Percent of beneficiaries (students, cooks) who use appropriate handwashing practices29 (i.e. with soap, before meals, before food prep, after latrine use, and diaper changing) All 77 Boys 77 50 Girls 77 55 2.1: Improved Knowledge of Health and Hygiene Practices Percent of students in target schools who can correctly identify at least 2 ways to prevent intestinal worms All 43 Boys 45 8 Girls 41 10 DIFFERENCE BETWEEN MID-TERM AND BASELINE 56 Annex 8: Evaluation Terms of Reference USDA/FAS Grant: FFE-685-2014/029-00B Counterpart International Senegal McGovern Dole International Food for Education and Child Nutrition Program (MGD-FFE) MID-TERM EVALUATION TERMS OF REFERENCE Counterpart International is seeking an experienced Consultant/Firm to conduct a mid-term assessment for its McGovern Dole International Food for Education and Child Nutrition Program (MGD-FFE) program in Senegal. Although the three-year agreement was signed on September 30, 2014, activities effectively started in February 2016 due to delays with the baseline study. The purpose of the mid-term evaluation is to assist Counterpart International in the systematic collection and analysis of information about the characteristics and outcome of the MGD/FFE program, to improve effectiveness, and to inform management decisions moving forward. This mid-term evaluation will assess intermediate characteristics and outcomes that can be used to improve the program quality and potentially be integrated into future planning. The mid-term evaluation will take place between April and June and will be conducted by a third party evaluator that will meet the criteria laid out in the program’s Evaluation Plan, USDA’s Monitoring and Evaluation Policy36, the program’s agreement, 7 CFR 1499.13 and 7 CFR Part 1599.13 2. The team will have expertise in conducting evaluations using mixed methods, particularly of school feeding and education programs or other similar USGfunded/donor-funded programs, preferably in Western Africa. I. BACKGROUND Counterpart was awarded funding from the U.S. Department of Agriculture (USDA) to implement a Food for Education (FFE) Program in Northern Senegal, in the St. Louis region. Since 2001, Counterpart International, with generous funding from the USG, has been addressing issues related to food security, child health and nutrition, and access to education in Senegal. Counterpart has implemented FFE programs in the Matam and Saint-Louis regions in Senegal and is currently implementing a FFE program in 270 schools in the Saint-Louis region. Over the years, Counterpart has built relationships and worked closely with national and local authorities in Senegal, collaborating with National Agency of Early Childhood and Community Based Pre-schools (ANPECTP), the Ministry of Education and the Departmental Inspector of National Education (IDEN). The goal of the MGD/FFE is to improve literacy and primary education, food security, reduce the incidence of hunger, and thereby contribute to a more self-reliant, productive society in Senegal (MGD results framework #1 and #2 listed in the annex section). The project has the following objectives:  Improve the literacy of school age children by: • Increasing student attendance rates by providing daily breakfasts and lunches for school children and supporting income-generating projects, such as community farms and granaries; 36 USDA Monitoring and Evaluation Policy. https://www.fas.usda.gov/sites/default/files/2014- 03/evalpol.pdf 2 U.S. Government Publishing Office (2012). 7 CFR 1499.13 – Recordkeeping and Reporting Requirements. Retrieved from: http://www.gpo.gov/fdsys/granule/CFR-2012-title7-vol10/CFR-2012-title7-vol10-sec1499-13 57 • Increasing skills and knowledge of school administrators and teachers by providing access to certification materials and capacity building for schools’ staff and Ministry of Education staff; • Improving teacher attendance through advocacy workshops and introduction of new policies in collaboration with education authorities, community members, and teachers; • Improving policy and the regulatory framework by advocating for the mainstreaming of the School Health and Nutrition Guide into the local teacher college curriculum and ensuring that the School Health and Nutrition Guide is available and widely distributed as a pedagogical resource to educational authorities (emailed nationwide).  Improve health and dietary practices by: • Improving school infrastructure and increasing access to clean water and sanitation by building latrines and water station systems; • Increasing knowledge of safe food preparation and storage practices by conducting training on commodity management, cooking techniques, food safety, and hygiene; • Increasing access to preventative health interventions by distributing deworming medication to school children and building the capacity of Parent Associations (PA) members and teachers in using moringa powder in school meals; and • Increasing access to requisite food preparation and storage tools and equipment by equipping school canteens and kitchens and providing energy-saving stoves. The main program activities include capacity building at the local, regional, and national level; construction of latrines and water station systems; equipment of schools with energy saving stoves and canteen equipment materials; establishment of community farms; training on good health and nutrition practices; capacity building of parent associations; promotion of teacher attendance; construction of school infrastructure; training in food preparation and storage practices; access to preventive health interventions and school feeding. The FFE program targets 33,811 students (16,567 boys and 17,244 girls) across 270 beneficiary schools (204 primary schools and 66 preschools) in the departments of Dagana, Podor and Saint Louis in the Saint Louis region of Senegal. Though the FFE program in Saint Louis was awarded at the end of September 2014, there were significant delays in the project startup, mainly with the start of the baseline evaluation which led to delays with the implementation of activities. Despite these delays, the program has been able to distribute millions of meals to students, to finalize the revision of the School Health & Nutrition Guide, to set up 20 community farms, and to construct 20 classrooms, 20 latrines and 20 water station systems among other achievements. CPI developed a Monitoring and Evaluation Plan and undertook a baseline study. Counterpart International is now seeking the services of a third-party, independent consulting firm to carry out a mid￾term review/evaluation of the program. The evaluation team will conduct this midterm evaluation in accordance with the framework laid out in the Monitoring and Evaluation Plan, the indicators listed in the program agreement, and subsequent amendments, relying on the program’s baseline evaluation and monitoring data. II. GOAL OF THE EVALUATION The mid-term evaluation aims to review and assess the program’s performance to date as it relates to program implementation, goals, objectives and activities. The goal is to gain an understanding of what the 58 program has achieved so far, whether it is likely that the program will achieve its targets and objectives and what can be learnt from it, and provide recommendations on how to improve program’s performance. The mid-term evaluation should also pay close attention to gender and social inclusion dynamics to ensure that considerations related to gender and social inclusion are factored into recommendations made to improve the program. III. SCOPE OF WORK The mid-term assessment design should accommodate and address the following elements: 1. Assess performance to-date with respect to relevance, effectiveness, efficiency, impact, and sustainability as defined below; 2. Identify particular trends and potential relationships between data collected; 3. Assess the performance of stakeholders (Ministry of Education (MOE), Departmental Inspector of Education and Training, Government Agency in Charge of Early Childhood Development (ANPECTP), Services Departementaux de l’Agriculture (SDA), Parent Associations (PAs), etc.); 4. Provide a clear picture of project successes and failures to-date; 5. Examine the strength of the relationships between program activities and results; 6. Assess the pertinence of project design to the problems the project is aiming to solve through the achievement of its objectives; 7. Assess the likelihood of attainment of targets and objectives; 8. Develop lessons learnt and strategies for performance improvement; 9. Provide a quality check of the M&E system and monitoring activities. Relevance: the extent to which the project interventions met the needs of the project beneficiaries and is aligned with Senegal’s and US Government’s development goals, objectives, and strategies. Relevance should also address the extent to which the project was designed taking into account the economic, cultural, and political context and existing relevant program activities. Effectiveness: the extent to which the project is likely to achieve its objectives. Effectiveness should also assess the extent to which the interventions are contributing to the expected results or objectives. Efficiency: the extent to which the project resources (inputs) have led to the achieved results. An assessment of efficiency should also consider whether the same results could have been achieved with fewer resources or whether alternative approaches could have been adopted to achieve the same results. Impact: assessment of the medium and long-term effects, both intended and unintended, of a project intervention. Effects can be both direct or indirect and positive or negative. As this is a mid-term evaluation, attribution of effects to project interventions is not required, but attempts should be made to prove any likely contribution. Sustainability: assessment of likelihood that the benefits of the project will endure over time after the completion of the project. Sustainability should also assess the extent to which the project has planned 59 for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships. IV. APPROACH AND METHODOLOGY Counterpart is looking for innovative suggestions regarding the methodology design of this evaluation. Generally, Counterpart anticipates that this evaluation will take a mixed-methods approach, utilizing quantitative and qualitative methods and tools appropriate to the evaluation’s research questions. The qualitative methods will help address some of the limitations of the quantitative methods and provide contextual understanding and interpretation of the quantitative results. The evaluation should use participatory approaches to involve key stakeholders, including implementing partners, subcontractors, program participants, and USDA. The evaluation must show how gender integration and social inclusion are being applied to the approach and methodology. The data collection tools for this mid-term evaluation should be consistent with the tools developed during the baseline evaluation, the indicators outlined in Attachment E of the Award agreement with USDA (taking into account disaggregation by gender where indicated), and the USDA-approved M&E plans (including reading and numeracy assessments). As outlined in the approved M&E plan, the methodology section should include a cohort comparison for student beneficiaries and a pre-post comparison for teacher beneficiaries:  A Cohort Comparison Method will be used to assess beneficiary students’ literacy outcomes and health and nutrition knowledge and practices. This methodology measures improvement (change) over time of beneficiaries at later point in time relative to the initial state of non￾beneficiaries before the program started. The student cohorts at baseline will serve as a comparison group for later cohorts of students.  A Pre-Post Comparison Method will be used to assess beneficiary teachers’ and mothers’ health and nutrition knowledge and practices. This methodology will attempt to establish the impact of the program by tracking changes in outcomes for the same program beneficiaries over time using measures both before and after the program interventions. The methods and tools used in this evaluation may include a combination of the following: Document review: The evaluator will find it useful to consult a broad range of background documents including project proposal, workplan, USDA-approved M&E Plan, USDA’s Monitoring and Evaluation Policy, etc. Besides project documents, Counterpart will support the evaluator’s efforts in obtaining secondary data. The package of briefing materials may include but are not limited to: 1. USDA-approved Plan of Operation and Performance Indicators (Attachments A and E of the program agreement) 2. Semiannual reports submitted to USDA 3. Program Baseline Evaluation report 4. Monitoring & Evaluation plan submitted to USDA 5. Field visit reports from program staff and training reports 6. FFE annual workplans 7. M&E data base 8. M&E data collection Tools 9. MOU between Counterpart and communities 60 10. School Hygiene, and Nutrition Guide 11. National School Feeding Guidelines 12. Annual reports 13. School Selection Report 14. Evaluation Report Food for Education – Matam Key Informant Interviews (KIIs): KIIs are suggested to be conducted with key stakeholders, including program staff, relevant USDA staff, relevant local government staff, relevant local partners staff, Counterpart relevant staff, and the community. Focus Group Discussion: It is anticipated that the evaluator will conduct multiple rounds of focus group discussions with local government staff, school administrators, teachers, parents, relevant local partners staff, Counterpart relevant staff, and the community (Ministry of Education, Departmental Inspector of Education and Training, Government Agency in Charge of Early Childhood Development (ANPECTP), Services Departementaux de l’Agriculture (SDA), PAs). Reading and Numeracy Assessments: The evaluator will develop and use an adaptation of the ASER￾Reading Test (and the test’s administration instructions) to measure the program’s effects on students’ literacy outcomes. Survey: The evaluator will carry out a survey of beneficiaries to test and verify the integrity of the data collected. The evaluator will detail in the Methodology Plan whether an in-depth survey, more relevant for qualitative information, or a broad quantitative survey will be conducted, with supporting reasons. V. DELIVERABLES The evaluator is expected to provide Counterpart the following deliverables: 1) Mid-term Assessment Workplan: The evaluator will prepare/formulate the final evaluation questions, finalize the assignment timeline, clarify potential team members’ roles and responsibilities, and develop data collection methods, tools, and guidelines over the course of the two weeks following signature of the contract. Counterpart will support the evaluator’s efforts in obtaining information about the logistical and administrative procedures for the assignment. 2) Methodology Plan: A strategy document that details the methodology that will be used, geographic and participant sampling structure, evaluative procedures and a quality assurance plan. The document will include a data analysis plan that details what procedures will be used to analyze data. Each data collection, analysis, and/or presentation tool, including the specific instruments, needs to be approved by Counterpart prior to the start of the evaluator’s field work. 3) Draft Evaluation Report: Prior to the mid-term assessment debriefing, the evaluator will submit a rough draft of the evaluation report to Counterpart Headquarters, who will then provide preliminary comments. Based on the draft report and comments, potential meetings among evaluator and Counterpart will be scheduled to discuss the results. The report will outline the methodology used, overall analysis, and the conclusions of the mid-term assessment, and specific programmatic recommendations for ongoing or future programming. 61 4) Debriefing: The evaluator will present the major findings of the final evaluation to Counterpart’s HQ and field program staff, USDA and other stakeholders. The debrief should also include recommendations, challenges encountered throughout the process and explain how they were overcome or not, as well as lessons learnt. The debriefing will also include pending data validation issues, if any, in order to ensure that all data has been collected and no further data collection is necessary. The debriefing will take place at a time and place agreed upon by both parties or via Skype, if needed. 5) Final Report: The final report, not to exceed 40 pages (not including annexes, table of contents, and acronym list), should be concise and to the point, utilizing charts, graphs and diagrams where appropriate. In producing the final evaluation report, the evaluator will link findings to the data analyzed (all findings must be evidence-based). Those links must be clearly articulated in the report. Additionally, the conclusions will clearly relate to the findings. It may require back and forth consultation; payment will not be rendered until the report is approved. The following table outlines requirements for the final report: Final Report Requirements Report Length Maximum of 40 pages, excluding the Table of Contents, Acronym List, and Annexes and should be written in Times New Roman font size 12. Executive Summary Include an Executive Summary that provides a brief overview of the evaluation purpose, project background, evaluation questions, methods, findings, and conclusions. Questions Address all evaluation questions in the SOW. Methods • Explain evaluation methodology in detail. • Disclose evaluation limitations, especially those associated with the evaluation methodology (e.g. selection bias, recall bias, etc.). NOTE: A summary of methodology can be included in the body of the report, with the full description provided as an annex. Findings • Present findings as analyzed facts, evidence and data supported by strong qualitative evidence. • Include findings that assess outcomes and impacts on males and females. • Table with all custom and standard indicator values Recommendations • Support recommendations with specific findings. • Provide recommendations that are action-oriented, practical, and specific. 62 Annexes Include the following as annexes, at a minimum: • Statement of Work. • Full description of evaluation methods. • All evaluation tools (questionnaires, checklists, discussion guides, surveys, etc.). • A list of sources of information (key informants, documents reviewed, other data sources) • Clean, English-language quantitative data sets in Microsoft-Excel • Qualitative transcripts and notes of focus group discussions and key informant interviews Only if applicable, include as an annex Statement(s) of Differences regarding any significant unresolved differences of opinion on the part of funders, implementers, and/or members of the evaluator. Quality Control Assess reports for quality by including an in-house peer technical review with comments provided to the evaluator. Transparency An English report should be submitted in electronic version to Counterpart for approval. All deliverables must be approved by Counterpart. VI. TIMELINE Counterpart anticipates the preparatory work and review of relevant reports and documents to be completed and the Mid-term Assessment Workplan submitted by April 20, 2017. The evaluator should expect to conduct the field work between April 25 and May 16, 2017. The first draft of the evaluation report is due to Counterpart by June 5, 2017. Once the draft is submitted, Counterpart has seven business days to review the report, raise concerns, provide comments, and send it back to the evaluator no later than close of business on June 15, 2017. The evaluator will then address Counterpart’s comments and concerns and submit a revised report to the Counterpart for Counterpart and the donor’s review. The finalized report is due no later than June 21, 2017. If Counterpart is still not satisfied with the final report, then both CPI and the evaluator will negotiate a no cost extension to ensure both parties are satisfied with the final result. Item Deadline Selection of consultant and signing of contract April 7, 2017 63 Submission of Mid-term Assessment Workplan: (Literature review, evaluation design including data collection and analysis methodology; draft sampling strategy, and intended respondents /key informants; quality assurance plan; Draft Evaluation Schedule; and Draft Data Collection Tools) April 20, 2017 Field Work (Training of data collectors, testing of the data collection tools and calibrate, data collection and analyses) April 25 to May 16, 2017 Debriefing on preliminary findings May 26, 2017 Submission of first draft report June 5, 2017 Review and comment of first draft (subsequent drafts as necessary) by Counterpart International followed by review and comments by USDA June 15, 2017 Submission of final report June 30, 2017 Throughout the mid-term assessment, there must be open communication between the evaluator and Counterpart through phone calls, emails, text messages, Skype calls, or face-to-face meetings for effective coordination between both parties and to ensure that potential issues are resolved in a timely manner. Any anticipated changes to the plan during the course of the evaluation must be a submitted in writing and be approved by Counterpart. VII. KEY RESEARCH QUESTIONS Some key evaluation questions are organized around the five program dimensions to be evaluated (relevance, effectiveness, efficiency, performance, and sustainability) and presented below. Appropriate disaggregation must be applied throughout all research questions such as disaggregation by sex and other factors in-line with the program’s M&E plan. A. Questions related to implementation and relevance: • To what extent is the program implemented as designed (e.g., lunches and trainings delivered)? • To what extent did the program design and activities reflect the needs of the project beneficiaries? • How relevant is the program given the economic, cultural, gender, and political context? • To what extent did the project take into account the existing government strategic framework in terms of the thematic areas tackled by the project? B. Questions related to effectiveness and performance: 64 • What were the root causes children did not enroll in school (disaggregated by sex and department)? • What were the root causes for student low attendance (disaggregated by sex and department)? • What were the root causes for drop-out rates (disaggregated by sex and department)? • Has there been a change in student enrollment, attendance and drop-out rates since the start of the project for male and female students per school, per department? • To what extent is the program effective at increasing student enrollment and attendance? At decreasing student drop-out rates? (disaggregated by sex, school and department) • To what extent is the program effective at increasing student literacy and numeracy outcomes? • To what extent do beneficiaries perceive the community farm activities as helpful in supporting the management and maintenance of school canteens and school feeding activities? • To what extent is the program successful in improving student knowledge and behavior about nutrition, health, and sanitation? • To what extent do beneficiary perceive the establishment of Moringa plantations as helpful in improving nutrition among school-aged children? • To what extent is the program successful at improving school teacher and mothers’ knowledge and behavior about nutrition, health, and sanitation? • Were there any unintended or indirect impacts? C. Questions related to efficiency: • How much return, in terms of project results, did USDA resources yield? • Could other implementation strategies have achieved more with the same resources? D. Questions related to sustainability: • What steps/actions/ inputs are required to realize full sustainability of activities beyond the life of project? • What are the challenges and successes of the program activities and what lessons can be drawn for the future sustainability of the program? • To what extent will project activities continue with the absence of support from both USDA and Counterpart? VIII. SPECIFIC ELEMENTS OF THE EVALUATION The following points below highlight areas of specific interests for the evaluation. The questions listed below are not exhaustive and do not necessarily cover all of the evaluation objectives. They are meant to guide the evaluator in setting the focus of this evaluation. Beneficiary School Selection Evaluate whether or not the selection of the beneficiary schools by the project followed a sound and efficient process, specifically in terms of: a) selecting the schools that had the most needs, b) time and resource efficiency of the process, and c) identifying and securing adequate participation from the schools, and make recommendations for future programs. 65 Revisit the selection criteria of beneficiary schools, the strengths and weaknesses of the governing structures and sustainability potentials and risks. Important Questions: 1. How effective was the school selection process? 2. What are the contributions of the different actors (school administrators and teachers, cooks, inspectors, PA,) to program implementation and sustainability? 3. What are the perceptions of the different actors on program implementation and sustainability? 4. What are the difficulties faced by these actors in program implementation and sustainability? 5. How can the present program be reoriented to meet the aspirations of these actors? Activity 1: Capacity Building of Education Authorities at the Local, Regional, and National Level Revisit performance through needs identification, training, monitoring and evaluation of teachers with effect on teaching for a higher quality education. Important Questions: 1. How effective was the process of identifying training needs? 2. How relevant were the identified needs to program objectives? 3. Did the different training modules tie with identified needs? 4. What has been the impact of the training on the quality of teaching? 5. How appropriate is the follow-up mechanism of trained teachers? 6. What other difficulties do these trained teachers still encounter in their daily duties? 7. What measures need to be taken to ensure sustained improvement in teaching quality within FFE beneficiary schools? Activity 2: Construction of School Infrastructure: Revisit program performance and evaluate the degree of achievements in relation to expressed needs, beneficiary participation, the quality of infrastructures and the degree of use of the infrastructures. Also revisit strengths and weaknesses identified and make recommendations for improvements in management of the program and implementation strategy. Important Questions: 1. To what extent was the number of infrastructures realized both in terms of quality and in quantity helpful with regards to programming during the period under review? 2. How effective is the selection process of beneficiaries of project infrastructures? What are the strengths and weaknesses? 3. To what extent were the infrastructures ready for use within the required period in relation to the stipulated provisions? What lessons can be learnt with regards to the control and supervision of construction works? 4. What are the strategies put in place by the various schools to ensure proper maintenance of the new infrastructures? What are the chances of durability of these infrastructures? 5. What significant differences exist in terms of cost and quality between program infrastructures and those of other partners in the schools? What lessons can be learnt? 66 6. What is the degree of use of the new infrastructures? 7. Did the construction of the new infrastructures have any impact on school enrollment and attendance? 8. Has there been an improvement in reading fluency and comprehension skills due to new/renovated school infrastructure? 9. What is the attitude of the communities vis-à-vis their role and responsibilities in the realization of these activities? What are the pertinent lessons that were learnt from this experience? 10. What changes could the program have made to maximize chances of success? Activity 3: Construction of latrines and water station systems Revisit program performance and evaluate the degree of achievements in relation to expressed needs, beneficiary participation, the quality of latrines and water station systems and the degree of use of the latrines and water station systems. Also revisit strengths and weaknesses identified and make recommendations for improvements in management of the program and implementation strategy. Important Questions: 1. To what extent was the number of latrines and water station systems realized both in terms of quality and in quantity helpful with regards to programming during the period under review? 2. How effective is the selection process of beneficiaries of project latrines and water station systems? What are the strengths and weaknesses? 3. To what extent were the latrines and water station systems ready for use within the required period in relation to the stipulated provisions? What lessons can be learnt with regards to the control and supervision of construction works? 4. What are the strategies put in place by the various schools to ensure proper maintenance of the new latrines and water station systems? What are the chances of durability of these latrines and water station systems? 5. What significant differences exist in terms of cost and quality between program latrines and water station systems and those of other partners in the schools? What lessons can be learnt? 6. What is the degree of use of the new latrines and water station systems? 7. Did the construction of the new latrines and water station systems have any impact on school enrollment and attendance? 8. Has there been an improvement in reading fluency and comprehension skills due to new/renovated school latrines and water station systems? 9. What is the attitude of the communities vis-à-vis their role and responsibilities in the realization of these activities? What are the pertinent lessons that were learnt from this experience? 10. What changes could the program have made to maximize chances of success? Activity 4: Equip schools with energy saving stoves, canteen equipment and materials Revisit the execution calendar and evaluate activities in relation to initially expressed needs; give explanations for every difficulty identified, make recommendations for changes in the implementation and sustainability strategy of energy saving stoves, canteen equipment and materials provided by the project. Delays to this activity were in part due to the fact that Counterpart was unable to find local expertise to implement the activity as planned and therefore decided to procure and equip all 270 school canteens with improved energy saving stoves. This activity started in FY17 and is expected to be completed around April 2017. The midterm evaluation should seek to address the delays in installing the energy saving stoves and possible impacts to the program. 67 Important Questions: 1. To what extent were the energy saving stoves, canteen equipment and materials ready for use within the required period in relation to the stipulated provisions? 2. What are the strategies put in place by the various schools to ensure proper maintenance of the energy saving stoves and canteen equipment and materials? What are the chances of durability of these energy saving stoves, canteen equipment and materials? 3. What significant differences exist in terms of cost and quality between program energy saving stoves, canteen equipment and materials and those of other partners in the schools? What lessons can be learnt? 4. What is the degree of use of the energy saving stoves, canteen equipment and materials? 5. What are the expected or unexpected effects of the sustainability plan in the schools? 6. What are the possible impacts of the delays with the installation of energy saving stoves to the program? Activity 5: Good health and nutrition practices Revisit program performance and evaluate the implementation of the training on the existing School Health and Nutrition Guide (SH&NG) for directors and teachers in the schools; the use of SH&NG as tools for the teachers and as a pedagogical resource in the St. Louis teacher training school administration; “health, nutrition, and hygiene weeks” in schools; and mural competitions promoting healthy behavior in good nutrition, hygiene and sanitation practices in schools. Identify strengths and weaknesses and make recommendations towards improving the management and the implementation strategy of good health and nutrition practices. Important Questions: 1. To what extent did project interventions in the health domain (construction of latrines, training on the existing SH&NG, health, nutrition and hygiene weeks, mural competitions, hand washing facilities, etc.) bring significant changes to the hygiene and sanitation status of the school students and cooks? 2. To what extent is the SH&NG being implemented? 3. What modifications are needed to be made by the program in order to improve the quality of program delivery? Activity 6: Training in Food preparation and storage practices Revisit program performance and evaluate the rules of the school feeding management committee on managing and securing donated commodities, organizing cooking schedules and procuring supplemental food and cooking supplies; training of trainers (TOT) trainings on the topics of commodity inventory management, food preparation, and food storage practices; capacity building in appropriate food preparation hygiene ; distribution of copies of the National School Feeding Guidelines developed jointly by Counterpart International and other development partners ; training in preparing nutritious meals with recipes using the donated commodities, measuring rations, applying good food hygiene practice, and effectively planning school meals. Important Questions: 1. To what extent are the TOT trainings on the topics of commodity inventory management, food preparation, and food storage practices and capacity building in appropriate food preparation hygiene being implemented? 2. To what extent did the distribution of copies of the National School Feeding Guidelines bring significant changes to the commodity inventory management, organizing cooking schedules? 68 3. What modifications are needed to be made by the program in order to improve the quality of program delivery? Activity 7: Provide access to Health Interventions Revisit program performance and evaluate the deworming medicine and multi-vitamins distribution in schools by the Government of Senegal (GoS) Saint-Louis Regional Medical office during deworming campaigns that are organized every six months throughout the region, moringa plantations and the process of moringa powder from dried moringa in schools with our partner Peace Corps Volunteers (PCV) in Senegal and food fortification with moringa powder. As there were delays of the deworming campaign by the GoS Saint Louis Regional Medical office, the midterm evaluation should seek to address those delays and possible impacts to the program.37 Identify strengths and weaknesses and make recommendations towards improving moringa added-value chain activities from the floor to food. Important Questions: 1. To what extent did project interventions in the deworming medicine and multi-vitamins bring significant changes to the access to preventive health by the school students, teachers and cooks? 2. To what extent are the moringa plantations being implemented? 3. To what extent did the delays in the distribution of the deworming medicine and the Vitamin A supplements impact the program? 4. What modifications are needed to be made by the program in order to improve the quality of program delivery? 5. What are other suggestions for PCV to improve their service delivery? Activity 8: School Feeding Revisit the program and evaluate the process and distribution of food items in relation to initial needs expressed; the management of stock in the warehouses; the mobilization of community contributions in school granaries; and identify strengths and weaknesses and make recommendations for improvement of management of the program. Important Questions: 1. To what extent did the food items reach the schools on time? What are the lessons to be drawn from the transportation process? 2. Were the food items distributed on time and in sufficient quantities to school students? 3. Are the stocking conditions, management and control of food items in the schools adequate? What are the improvements that can be made to the system? 4. How relevant was the training on commodity management? 5. How relevant were the trainings on food preparation, storage and hygiene? 6. What difficulties are faced by PAs and school authorities in food preparation and how can improvements be made? 7. How acceptable are the donated food commodities to the beneficiaries? 8. Have school feedings had a significant influence on enrollment, attendance and school success rates? 37 On July 27, 2016 Counterpart International handed over 300,000 tablets of deworming medication and Vitamin A to the Government of Senegal’s Saint Louis Regional Medical Office to be used during deworming campaigns that are organized every six months throughout the Region. However, due to administrative setbacks, the deworming campaigns took place January 3-6, 2017. 69 9. What are the strengths and weaknesses in the relationship between the PA, community, Ministry of Education (MOE) and the FFE Team in the execution of school feeding program activities? 10. What changes can the program make to increase community and Ministry of Education National Division for School Canteen participation with regards to moving towards the implementation of a local and indigenous school feeding program? Activity 9: Establish Community Farming Revisit program emphasizing on selection criteria, final choice of community farming. As concerns the execution of the sustainability plan and the mobilization of community in the farming, revisit the execution calendar and evaluate activities especially in relation to initially expressed needs; give explanations for every difficulty identified, make recommendations for changes in the sustainability strategy and the mobilization of local labor in farming. Important Questions: 1. How were the selection criteria developed? How transparent was the selection process? To what extent was the sustainability plan executed? 2. Are there moves to put in place a sustainable program? 3. What are the expected or unexpected effects of the sustainability plan in the schools? Was the exploitation plan put in place understood by the population? 4. Was the mobilization of community labor convenient? What are the lessons to be learnt? What strategies can be put in place to ensure success? 5. What are the strengths and weaknesses in the relationship between the communities, PA, and FFE team in the execution of program activities? 6. What changes can be made by the program to increase community /government or local council’s participation in the program? 7. What other elements could be integrated to foster an indigenous school feeding program? Activity 10: Promote Teacher Attendance Revisit program performance and evaluate advocacy workshops to reduce teacher absences related to teacher’s administrative duties; recognition events for exemplary teacher. Important Questions: 1. How effective was the process of identifying advocacy workshop needs? 2. How relevant were the identified needs to program objectives? 3. Did the different topics in advocacy workshop tie with identified needs? 4. What has been the impact of the advocacy workshop on increasing teacher attendance? 5. How were the criteria developed for selecting the recipient of teacher recognition? 6. How transparent was the selection process? 7. How appropriate is the follow-up mechanism of trained or sensitized teachers for reduce absences? 8. What other difficulties do these trained or sensitized teachers still encounter in their administrative duties? 9. What measures need to be taken to ensure sustained improvement in promoting teacher attendance within FFE beneficiary schools? 70 Activity 11: Parent Association training Revisit performance through needs identification, training, monitoring and evaluation of Parent Associations (PA) with effect on governance and management of participating schools, assessment of competition among PAs created in order to share best practices in the design and management of sustainable activities that benefit the schools, such as granaries, canteens, and community farms. Important Questions: 1. How effective was the process of identifying training needs? 2. How relevant were the identified needs to program objectives? 3. Did the different training modules tie with identified needs? 4. What has been the impact of the training on the quality of the design and management of sustainable activities? 5. How appropriate is the follow-up mechanism of trained PAs? 6. What other difficulties do these trained PAs still encounter in their daily duties? 7. What measures need to be taken to ensure sustained improvement in design and management of sustainable activities within FFE beneficiary schools? 8. How were the criteria developed for selecting the competition winner? 9. How transparent was the selection process? Crosscutting Activity: Monitoring and Evaluation of Program Activities Revisit progress indicators for all program components and data collection tools with relations to their appropriateness, reliability and validity to the program and their ability to measure and capture changes. Examine progress made as per indicator taking into consideration certain constraints like logistics, time and available resources. Evaluate the use of program management of information from baseline, and that of program M&E. Make recommendations for changes, fine-tuning or addition of indicators or the data collection tools. Important Questions: 1. How appropriate and timely was the baseline study carried out prior to program startup? 2. How relevant are the monitoring tools and the indicators table for the measurement and demonstration of changes? What are some of the changes that can be made to make them more user-friendly? 3. How effective and efficient are the data collection tools being used by project with respect to reliability, accuracy, timeliness, validity and integrity? 4. To what extent are project stakeholders comfortable with the data collection tools? 5. How efficient is the data /indicator analyses process? 6. To what extent were the monitoring tools for data collection available in schools on time and in sufficient quantities? 7. Was the frequency of data collection, supervision of activities sufficient and well planned? 8. Was data collected transmitted on time and in the sufficient quantities for processing? What are the lessons to be learnt? 9. To what extent did the teachers and PAs participate in data collection and transmission to FFE program? 10. What is the perception of teachers involved in M&E activities vis-à-vis their normal roles and responsibilities? 71 11. To what extent is the information collected pertinent to program management and the measurement of changes? 12. To what extent is the information collected used in program management? How is the data collected currently being used? How can the usage be improved upon? 13. How efficient is the information exchange mechanism between program staff, different parties, the communities and the main beneficiaries at different levels? 14. How is data storage done vis-à-vis its availability for future decision making? IX. ROLES AND RESPONSIBILITIES The evaluator will coordinate with Counterpart HQ’s Program Officer, Regional Director, Monitoring and Evaluation Officer and Counterpart Senegal Country Representative in regards to the overall scope, direction, and completion of this assignment. USDA will provide directives as needed along with feedback on the initial draft, to be included in the final report. Counterpart HQ Regional Director, Program Officer, Gender Technical Specialist, Monitoring and Evaluation Officer and Counterpart Senegal Country Representative will provide all relevant reports, data and related information necessary to prepare the evaluator for the assignment. And as required, Counterpart HQ staff will coordinate with program staff to facilitate potential meetings with all relevant stakeholders during the field visit in Senegal. Counterpart HQ Regional Director is responsible for approving evaluation deliverables. Payment will be based on the following milestones: Milestone Payment Mid-term Assesment Workplan and Methodology Plan Submission 20 % of the total Draft Evaluation Report Submission 30 % of the total Debriefing 15% of the total Final report submitted and approved; all data handed over to CPI 35 % of the total X. PROPOSAL REQUIREMENTS The applicant shall submit a full technical proposal to Counterpart via an electronic submission with the following documents: 1. Qualifications: a. Resume/CV of the lead consultant that demonstrates at least 5-7 years of solid experience working on or leading USG-funded program/project mid-term and/or final evaluations or similar work, preferably in the region. Prior experience working in Senegal is a plus. b. Resume/CV of the lead consultant that demonstrates expertise in literacy and/or school feeding programs. c. To the extent possible, the evaluator will be gender-balanced. 72 d. Ability to hire experienced enumerators that are proficient in French and Wolof. e. Ability to work within the international and multicultural environment. f. Native French speaker, or near-native level command of French is required and Wolof speaker is highly desirable. If the proposal is submitted by a consortium of partners, qualifications of each proposed partner should be presented. Consultant must also demonstrate familiarity and experience with designing and managing qualitative research designs. 2. References: The applicant is required to submit three references with email and telephone contact information related to past experiences of evaluation research. 3. Proposed methodology and structure of the evaluation, which will include:  Proposed sampling methodology • Proposed evaluation design with a detailed description of tools to be used • Team composition and structure • Quality control method and tools 4. Workplan, which will include: Detailed timeline of activities – in days, required for each stage of the mid-term assessment. 5. Budget: An itemized budget in US dollars. XI. PROPOSAL EVALUATION Proposals for this Mid-term assessment will be evaluated on the basis of the following criteria: Evaluation Criteria Score Consultant prior experience in similar work 25 points Proposed overall methodology 25 points Proposed sampling method 15 points Quality of data-collection and data-entry procedures 10 points Timeframe for delivery of Evaluation deliverables 5 points Methods of quality control 10 points Budget 10 points Total Score: 100 points When drafting the proposal, the Consultant/Firm should be careful to include all information requested above. Failure to submit a complete application will result in the rejection of the proposal. All questions pertaining to this RFP must be submitted by 12:00 noon EST on March 13th , 2017. Questions may be submitted, in written form, to: ffecounterpartsenegal@counterpart.org. Questions 73 will be answered within three (3) business days via email. Questions and answers will be posted publically. XII. INSTRUCTIONS FOR SUBMITTING THE PROPOSAL Please submit the proposal via email as a Word and/or PDF document to: ffecounterpartsenegal@counterpart.org. Please include in the subject “Senegal MGD-FFE Mid-term assessment.” The proposal submission deadline is March 24th, 2017 at 11:59 p.m. EST. Proposals received after this date will not be considered. The selected Consultant/Firm is expected to start work on the mid-term assessment on or around April 7th, 2017. 1 ANNEX A: MGD Results Frameworks #1 and #2 2 20 McGovernMcGovern--Dole Dole