DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Foreign Agricultural Service, United States Department of Agriculture McGovern-Dole International Food For Education And Child Nutrition Project Baseline Evaluation August 2019 t DISCLAIMER: The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. McGovern-Dole International Food for Education and Child Nutrition Project Baseline Evaluation This report comprises a baseline study intended to support the McGovern-Dole program called Sukaabe Janngo “Children of Tomorrow” in Senegal. This four-year program is funded through a cooperative agreement with the United States Department of Agriculture (USDA). The program aims to improve student performance in part through school feeding programs. Counterpart, along with partners Save the Children and Associates for Research and Education for Development, implement the McGovern￾Dole project in the Saint Louis region of Senegal. The program will work with the same 270 schools Counterpart worked with from 2015 to 2018, continuing to work towards achieving the two McGovern￾Dole strategic objectives of 1) improved literacy of school-age children (MGD SOI), and 2) increased use of health, nutrition, and dietary practices (MGD S02). The purpose of the baseline study is to: (a) produce high-quality data to compare progress on the midterm, endline, and impact evaluations; (b) make methodological recommendations for subsequent evaluations; and (c) give input on the program’s implementation strategy. This report was informed by a mixed methods approach which included quantitative and qualitative methods: (a) document review of relevant documentation; (b) in-person interviews and surveys with a wide variety of stakeholders; and (c) one-to-one administration of reading assessments among students in selected schools. Agreement Number: No. FFE-685-2018/009-0 Project Duration: 2019-2023 Implemented by: Counterpart International Evaluation Authored by: IMC Worldwide Table of Contents 1. INTRODUCTION......................................................................................................................................1 2. BACKGROUND AND CONTEXT ...............................................................................................................1 2.1 Country Context ...........................................................................................................................1 2.2 Program Overview........................................................................................................................3 2.2.1 Strategic Objectives and Results Frameworks.........................................................................3 2.2.2 Primary Program Activities per Strategic Objective ................................................................5 3. METHODOLOGY.....................................................................................................................................6 3.1 Instruments..................................................................................................................................6 3.1.1 Surveys, Checklist, and Interviews...........................................................................................6 3.1.2 The Annual Status of Education Report (ASER) Reading Tool .................................................7 3.2 Sample design ..............................................................................................................................7 3.2.1 Comparison group school selection.........................................................................................7 3.2.2 Sampling...................................................................................................................................8 3.2.3 Qualitative interviews............................................................................................................10 3.2.4 Fieldwork................................................................................................................................10 3.2.5 Limitations..............................................................................................................................10 4. Results..................................................................................................................................................11 4.1 Operationalized Research Questions.........................................................................................11 4.1.1 Sukaabe Janngo project level performance...........................................................................11 4.1.2 School feeding and nutrition .................................................................................................13 4.1.3 Education and Literacy...........................................................................................................14 4.1.4 Health.....................................................................................................................................14 4.1.5 Methodological......................................................................................................................15 4.1.6 School Feeding Sustainability ................................................................................................15 4.2 MGD SO1 - Improved Literacy of School Aged Children ............................................................17 4.2.1 MGD 1.1: Improved Quality of Literacy Instruction ..............................................................18 4.2.2 MGD 1.2: Improved Student Attentiveness...........................................................................20 4.2.3 MGD 1.3 Improved Student Attendance ...............................................................................21 4.2.4 MGD 1.3.2 Reduce Health Related Absences........................................................................22 4.2.5 MGD 1.3.4 Improved Student Enrollment.............................................................................23 4.3 MGD SO2 Increase use of health, nutrition, and dietary practices ...........................................23 4.3.1 MGD 2.4 Increased Access to Clean Water and Sanitation...................................................24 4.3.2 MGD 2.6 Increased Access to Requisite Food Prep and Storage Tools.................................26 4.4 School Graduation from Sukaabe Janngo ..................................................................................27 5. Recommendations and Conclusions....................................................................................................29 5.1 MGD S01: Improved Literacy of School Aged Children..............................................................29 5.2 MGD 1.1: Improved Quality of Literacy Instruction...................................................................29 5.3 MGD 1.2: Improved Attentiveness.............................................................................................29 5.4 MGD 1.3: Improved Student Attendance ..................................................................................29 5.5 MGD S02: Increased Use of Health, Nutrition and Dietary Practices........................................30 5.6 Measuring effectiveness of extra-curricular reading activities in improving literacy ...............30 5.7 Measuring Outputs and Outcome Achievement .......................................................................31 5.8 Teacher Training Effectiveness...................................................................................................31 5.9 School Graduation......................................................................................................................31 Annex A: Baseline Values for McGovern-Dole Standard Indicators............................................................33 Annex B: Data Collection Instruments.........................................................................................................47 Annex C: Baseline Study Scope of Work......................................................................................................48 Tables Table 1: Survey Topics ...................................................................................................................................7 Table 2: Sukaabe Janngo Baseline Sample sizes............................................................................................9 Table 3 Sex Composition of the Student Sample ..........................................................................................9 Table 4: Qualitative Interviews....................................................................................................................10 Table 5: Shares of recommended per student feeding requirements by source (%) .................................13 Table 6: Average Reading Level by Grade and Treatment/Comparison .....................................................17 Table 7: Teacher training experience ..........................................................................................................18 Table 8: Director rating of literacy instruction ............................................................................................19 Table 9: (Directors) Extent to which reading instruction needs to improve...............................................19 Table 10: what percentage students pay attention in class?......................................................................20 Table 11: attendance rates by grade ...........................................................................................................21 Table 12: Reasons students miss school (n = 285) ......................................................................................22 Table 13: Average number of students per school by grade and Treatment/Comparison ........................23 Table 14: Cleanliness of Latrines .................................................................................................................24 Table 15: State of hand washing facilites....................................................................................................25 Table 16: Do you wash your hands at school? ............................................................................................25 Table 17: Why don't you wash your hands at school? ................................................................................26 Table 18: Degree to which treatment schools are equipped for school feeding........................................26 Figures Figure 1: Percentage of chidren of primary school age (6-11) out of school................................................2 Figure 2: Literacy rates among youth and adults..........................................................................................3 Figure 3: Results framework for improved literacy of school-aged children................................................4 Figure 4: Results framework for increased health, nutrition, and dietary practices ....................................5 Figure 5: Map of Project and Comparison Schools........................................................................................7 Figure 6: Reading Level Distribution by GRade and Treatment/Comparison .............................................17 Figure 7: Attendance rates with and without school feeding .....................................................................21 Figure 8: Out of service hand washing equipment......................................................................................25 Figure 9: MGD provided equipment in use .................................................................................................26 Abbreviations ASER The Annual Status of Education Report CPI Counterpart International CRDH Centre de Recherche pour le Développement Humain DCS Division of School Canteens LRP Local and Regional Procurement LPT Lecture Pour Tous MGD McGovern-Dole RTI Research Triangle Institute SMC School Management Committee SO Strategic Objective TACSS Transition d’Alimentation dans les Confines Scolaires au Senegal 1 1. INTRODUCTION This report is the final product of a baseline study conducted to support the newly launched McGovern￾Dole program in Senegal. The program is called Sukaabe Janngo, or “Children of Tomorrow” (hereinafter, “Sukaabe Janngo” or “the program”). The Sukaabe Janngo program is funded through a cooperative agreement with the United States Department of Agriculture (USDA). Sukaabe Janngo’s aim is to improve student literacy and increase student use of appropriate health, nutrition, and dietary practices. Counterpart International will implement this four-year program in the Saint Louis Region, with certain activities targeting national actors, from 2018 to 2022. IMC Worldwide served as an external independent team between April and May of 2019 to conduct this baseline study. The study team coordinated with Counterpart International’s Senior Program Officer, Regional Director, Director of Program Quality and Learning and Counterpart Senegal Chief of Party regarding the overall scope, direction, and completion of the baseline study. Counterpart’s Senior Program Officer was the point of contact for the baseline study. The purpose of the baseline study was to: (a) produce high-quality data to compare progress on the midterm, endline, and impact evaluations; (b) make methodological recommendations for subsequent evaluations; and (c) give input on the program’s implementation strategy. This report was informed by a mixed-methods approach, which included quantitative and qualitative methods including: (a) review of relevant documentation; (b) in-person interviews and surveys with a variety of stakeholders; and (c) one-to-one administration of reading assessments among students in selected schools. This report is presented in five sections. This first section, this introduction, provides a brief overview of the evaluation, including its purpose, objectives and methodology. Section 2 provides a description of the country context and a program overview. The baseline study methodology is presented in Section 3. Baseline study findings and recommendations are presented in Sections 4 and 5. 2. BACKGROUND AND CONTEXT 2.1 Country Context Senegal is a lower middle-income country. In recent years it has exhibited strong economic growth. Still, 39 percent of its 15 million inhabitants live below the poverty line. Subsistence farming dominates the agriculture sector and reliance on traditional and sometimes unsustainable practices leads to vulnerability to climate shocks. For example, 70 percent of crops in Senegal are rain-fed, which means that climate unpredictability directly impacts their availability and price.1 In Senegal, 85 percent of children that work are employed in the agriculture sector, and data indicates that when food insecurity increases, primary school-aged children work more and attend school less.2 The education system of Senegal is split between pre-primary (or preschool), primary, lower secondary, and upper secondary (see official school ages by level of education in the figure below). Law No. 2004- 37 of December 15, 2004, defines compulsory education for both sexes as a 10-year continuing 1 World Food Programme. (2019). Senegal. Accessed at: https://www1.wfp.org/countries/senegal. 2 World Bank Databank. (2015.) World Development Indicators Database 2015 Series. 2 curriculum. This fundamental school cycle requires parents to enroll children once they reach the age of six and lasts until the child is 16 years old.3 FIGURE 1: OFFICIAL SCHOOL AGES BY LEVEL OF EDUCATION Image source: UNESCO Institute for Statistics, Senegal. Available at: http://uis.unesco.org/country/SN The total percentage of out-of-school populations for primary school age children in Senegal is estimated at 41 percent. The difference in percentage of out-of-school primary school age boys and girls is slim (1 percent). The largest disparities are exhibited among rural versus urban populations, and the poorest compared to the richest children and youth.4 FIGURE 1: PERCENTAGE OF CHIDREN OF PRIMARY SCHOOL AGE (6-11) OUT OF SCHOOL Source: FHI360 (2018). Senegal National Education Profile 2018 Update. Available at: https://www.epdc.org/sites/default/files/documents/EPDC_NEP_2018_Senegal.pdf Data source: Extraction of DHS dataset 2016. Children and youth in Senegal who attend school perform more poorly on a range of education indicators compared to children in sub-Saharan African countries as well as low and middle income countries.5 Illustratively, as shown in Figure 2 below, literacy rates among Senegalese youth (ages 15-24) are significantly lower than the median rates for lower middle-income countries globally. 3 République du Sénégal (2018). Programme d'Amélioration de la Qualité de l’Équité et de la Transparence-PAQUET EF 2018-2030 (pp. 40-41). Available at: http://www.education.gouv.sn/sites/default/files/politique%20en%20vigeur/Programme%20d%27Ame%CC%81lio ration%20de%20la%20Qualite%CC%81%20de%20l%E2%80%99E%CC%81quite%CC%81%20et%20de%20la%20Tran sparence-PAQUET%20EF%202018-2030.pdf 4 FHI360 (2018). Senegal National Education Profile 2018 Update. Available at: https://www.epdc.org/sites/default/files/documents/EPDC_NEP_2018_Senegal.pdf 5 Lower middle-income countries as classified by the World Bank. 3 FIGURE 2: LITERACY RATES AMONG YOUTH AND ADULTS Source: FHI360 (2018). Senegal National Education Profile 2018 Update. Available at: https://www.epdc.org/sites/default/files/documents/EPDC_NEP_2018_Senegal.pdf Data source: UNESCO Institute for Statistics, 2017. 2.2 Program Overview The current Sukaabe Janngo program covers the 270 schools (204 primary schools and 66 preschools) that Counterpart International worked with in the Saint Louis region during Phase I from 2015-2018, serving 47,329 students. During the four-year life of the program, Sukaabe Janngo schools are expected to enroll at least 66,000 students total. Over the four years, the current program (Phase II) will continue to support the Government of Senegal’s education reform efforts, and has a goal of graduating 200 Sukaabe Janngo schools by providing technical assistance and ensuring effective structures are in place. The program’s graduation strategy combines establishing a national-level implementation strategy and budget that sustains school feeding by the Government of Senegal and expanding capacity and commitment at all levels: from schools to districts to the national level. Sukaabe Janngo will build on the successes and lessons-learned from Phase I via an accelerated capacity-building program to enable target schools in the Saint Louis region to achieve some level of sustainability by year three of the program. This level of sustainability is defined by schools’ graduation from McGovern-Dole assistance. Progress toward graduation will be tracked in the fourth quarter of each year and will also be discussed during the annual review of program activities.6 2.2.1 Strategic Objectives and Results Frameworks Implemented by Counterpart International and its partners, Save the Children (SCI) and Associates for Research and Education for Development (ARED), the program aims to achieve two strategic objectives: 1) improved literacy of school-age children, and 2) increased use of health, nutrition, and dietary practices. The achievement of the two strategic objectives is underpinned by four cross-cutting outcomes: a) increased capacity of government institutions, b) improved policy and regulatory framework, 6 Counterpart International. (2019). McGovern-Dole International Food for Education and Child Nutrition Project Sukaabe Janngo , Annual Work Plan October 1, 2018-September 30 2019, Revised and Submitted on January 15, 2019 (pp. 29-32). 4 c) increased government support, and d) increased engagement of local organizations and community groups. It is important to note that these strategic objectives are also aligned with the Government of Senegal’s reform initiatives, as detailed in its Program for Improving Quality, Equity and Transparency of the Education and Training Sector (PAQUET); the framework for the operationalization of the education policy for the period 2012-2025. Illustratively, the PAQUET highlights the importance of empowering local communities to manage the education system at the grassroots level and advocates for a move to decentralization of the management of educational programs (i.e. in line with the cross-cutting outcomes). Strategic Objective 1: Improved literacy of school-aged children As stated in the baseline evaluation’s scope of work, Counterpart and its partners will work to achieve Strategic Objective 1 (SO1) through improved quality of literacy instruction, greater student in-class concentration through school feeding, and increased attendance. The baseline evaluation’s scope of work also notes that: ▪ Improved literacy will be achieved through improved school supplies and materials, improved literacy instruction materials, and increased skill and knowledge of teachers and school administrators. ▪ Improved attentiveness will be achieved through increased access to food through school feeding. ▪ Improved quality of literacy education is also expected to increase student attentiveness. ▪ Improved student attendance and enrollment will result from increased access to food, reduced health absences, and improved school infrastructure. FIGURE 3: RESULTS FRAMEWORK FOR IMPROVED LITERACY OF SCHOOL-AGED CHILDREN 5 Strategic Objective 2: Increased Use of Health, Nutrition, and Dietary Practices Also outlined in the baseline study’s scope of work is how Counterpart and its partners will work to achieve Strategic Objective 2 (SO2). Specifically, SO2 will be achieved through improved knowledge of health and hygiene practices, increased knowledge of safe food preparation, increased knowledge of nutrition, increased access to water and sanitation, increased access to preventative health interventions, and increased access to food preparation and storage tools and equipment. The four foundational results (MGD 2.4.1 – 2.4.4), are common to all MGD SOs and work to support the sustainability of all results. FIGURE 4: RESULTS FRAMEWORK FOR INCREASED HEALTH, NUTRITION, AND DIETARY PRACTICES 2.2.2 Primary Program Activities per Strategic Objective The program will implement the following primary activities to support the achievement of the two strategic objectives: Strategic Objective 1: Primary Activities ▪ Provide school meals: In preschools and primary schools, breakfast will be provided five days a week for 170 school days. In primary schools, lunch will be provided two days a week. ▪ Take-home rations: These will be provided to boys in grades five and six who have stellar (90%) attendance rates. Cooks will also receive take-home rations as “an incentive for their labor “ as stated in McGovern-Dole’s Annual Work Plan, October 1, 2018-September 30, 2019. ▪ Training in commodity management: Counterpart will train master trainers who will then train school management committees (SMCs) on sustainable commodity management, using revised national guidelines. Counterpart will also provide refresher trainings to SMCs. ▪ Training for cooks and SMCs on food preparation and storage: Counterpart will train school cooks, SMCs, and master trainers on safe and nutritious food preparation and management. ▪ Teacher training: Counterpart will train designated government master trainers who will train teachers on improved literacy approaches. ▪ School administrator training: Counterpart will also train participating school directors on literacy, data collection, and other topics. ▪ Literacy extra-curricular activities: Counterpart will organize reading clubs at all targeted primary schools. 6 ▪ Building and rehabilitating classrooms: Counterpart will build or repair classrooms for at least 30 schools in most need.7 ▪ Providing furniture and equipment: Counterpart will provide furniture and equipment for at least 30 schools most in need.8 ▪ Training for government officials: Government officials will be trained on management and data collection and how to sustain the program. Strategic Objective 2: Primary Activities ▪ Provide latrines and water stations: Counterpart will build or rehabilitate up to 20 latrines and water station systems and train students on their proper use. Mobile hand washing stations will also be provided. ▪ Maintain or replace canteen equipment: Canteen equipment including energy efficient stoves and pots was provided in the previous program. Counterpart will replace equipment if needed. ▪ Train teachers on good health and nutrition practices: Government master trainers will be trained by Counterpart on these practices and provide training to teachers annually. ▪ Distribute deworming medicine, vitamins and minerals: Counterpart will support the Government of Senegal’s existing initiatives to ensure children, cooks and teachers are dewormed. 3. METHODOLOGY The baseline will be used in two types of future evaluations: before and after mid-term and final performance evaluations, and an ex post impact evaluation. The performance evaluations will only include Sukaabe Janngo schools, whereas the impact evaluation will include comparison schools. Since school selection for Sukaabe Janngo was not random, and comparison school selection aimed to mimic this non-random selection, the impact evaluation will use quasi-experimental design with a sample of children in grades 4, 5 and 6. 3.1 Instruments 3.1.1 Surveys, Checklist, and Interviews To operationalize Sukaabe Janngo’s research questions and indicators, the baseline study used three quantitative surveys: an ASER reading test, surveys including a school checklist, and qualitative interviews. The surveys were administered in French, the current language of instruction in Senegal’s schools. The surveys are of students, teachers and directors. The questionnaires for all of the surveys were implemented via tablet through Open Data Kit. Topics for each survey and school checklist are highlighted below. Full questionnaires and interview guides are found in Annex B. 7 Counterpart International. (2019). McGovern-Dole International Food for Education and Child Nutrition Project Sukaabe Janngo, Annual Work Plan October 1, 2018-September 30 2019, Revised and Submitted on January 15, 2019 (p. 25). 8 Counterpart International. (2019). McGovern-Dole International Food for Education and Child Nutrition Project Sukaabe Janngo, Annual Work Plan October 1, 2018-September 30 2019, Revised and Submitted on January 15, 2019 (p. 25). 7 TABLE 1: SURVEY TOPICS Student Teacher Director School checklist ASER reading test Trainings received Literacy related classes Attendance Literacy related classes Literacy related classes Cooks and training Latrines Extra-curricular reading Student attentiveness Canteen management School canteen Missed school days School management committee Latrines Handwashing practice Attentiveness The research team conducted qualitative interviews with departmental, regional, and national government officials, as well as school directors, teachers and school management committees. The interviews focused on school feeding sustainability, attendance, enrollment, student attentiveness, and teacher training. Interviews were semi-structured and conversational to ensure interviewers asked key questions. 3.1.2 The Annual Status of Education Report (ASER) Reading Tool The ASER reading assessment tool, developed in India and used globally, was used to assess reading for this baseline and in the previous phase of MGD in Saint Louis. The ASER has five possible scores: 1) cannot recognize letters; 2) can recognize letters but cannot read words; 3) can read words but cannot read sentences; 4) Can read text at grade 1 level but not higher; and 5) can read text at grade 2 level.9 Even though this is an early grade reading test, it is often used to assess reading for higher grades in countries with high levels of illiteracy. For example, a recent ASER assessment carried out in rural India found that 41, 50 and 60 percent of 4th, 5th and 6th graders read at the grade 2 level.10 This baseline report follows the India study’s precedent, assessing the share of students from grades 4, 5 and 6 able to read at 2nd grade level. 3.2 Sample design 3.2.1 Comparison group school selection Comparison group schools were selected from the schools in Saint Louis region that were not chosen for the Sukaabe Janngo project in 2015. The evaluation team aimed to select comparison schools following the same criteria as Counterpart originally used to select project schools in 2015. 9 These reading grade levels are calibrated for India. 10 http://img.asercentre.org/docs/ASER%202018/Release%20Material/aserreport2018.pdf FIGURE 5: MAP OF PROJECT AND COMPARISON SCHOOLS 8 Counterpart reports that the criteria used for the original Sukaabe Janngo school selection in 2015 were: ▪ A minimum of 30 students. ▪ Sufficient number of teachers. ▪ Low rates of school enrollment. ▪ Accessibility by trucks for food distribution in all seasons. ▪ Availability of adequate facilities for the storage and cooking, or ability to mobilize communities to quickly provide these facilities. (This was changed to having a suitable building for storage only). ▪ Engaged community and have demonstrated the will to contribute to the previous projects; including through in-kind or cash support. However, original Sukaabe Janngo school selection was ultimately left to government school inspectors with final verifications by Counterpart. It is not clear how closely these above criteria were followed as the baseline research team did not have access to original school selection process documentation. We used two of the above criteria in defining comparison school eligibility for this baseline study: ▪ A minimum of 30 students. ▪ Accessibility by trucks for food distribution in all seasons. After removing schools not meeting the above criteria, comparison schools were randomly selected as discussed below. Comparison schools could be further matched with Sukaabe Janngo schools ex post. 3.2.2 Sampling For student and teacher surveys the team used a two-stage cluster sampling design. The first stage comprises the random selection of the clusters, defined as individual schools, and the second stage comprises selection of students or teachers. For directors (and checklists) the sample is a simple random sample. The calculated sample sizes for all surveys allows the mid-term and final evaluations to utilize only the treatment group sample to assess Sukaabe Janngo performance with adequate statistical significance. For the performance evaluations that do not utilize the comparison group, treatment samples were designed to produce estimates with the standard 95 percent confidence level and a 5 percent margin of error.11 For the impact evaluation, the design of the treatment and comparison group sample size is based on a minimum detectable effect size (i.e. difference in outcome estimates between Sukaabe Janngo and comparison groups) of approximately 0.2 standard deviations (SD), with a power of 0.80 and a 0.05 significance level. This was calculated using Optimal Design software. Table 2 includes the sample design produced sample sizes. 11 Simple random sample sizes were calculated using the standard formula. For the student and teacher surveys, the simple random sample size was multiplied by the design effect to adjust for clustering. The design effect is calculated as DF = 1 + icc(c– 1) where the ICC is the intra class correlation coefficient and c is the cluster size. For ICC coefficient we used 0.15. This is consistent with the value used in another recent MGD baseline study: https://apps.fas.usda.gov/fais/public/files/Mozambique_McGovernDole_Final%20Baseline%20Report_2 .18.18.pdf 9 TABLE 2: SUKAABE JANNGO BASELINE SAMPLE SIZES Interview type Sukaabe Janngo schools Comparison schools # surveys per school Treatment surveys Comparison surveys Students – Primary 131 107 5 657 551 Teachers – Primary 131 107 2 12 262 (234)13 214 (164) Directors – Primary 131 107 1 131 107 Directors – Preschools 29 24 1 29 24 Checklists – Primary 131 107 1 131 107 Checklists - Preschools 29 24 1 29 24 The sex composition of the student sample is found in table 3: TABLE 3 SEX COMPOSITION OF THE STUDENT SAMPLE Type Sex n Proportion Comparison female 345 63.41% Comparison male 198 36.59% Project female 422 64.99% Project male 225 35.01% Schools were randomly selected from the 270 Sukaabe Janngo schools and all eligible comparison schools. At each primary school, students were randomly selected from attendance lists that day. The interviewer team supervisor numbered attendance lists and then chose five random numbers using a tablet-based application in order to select students for interview. Teachers were selected based on their availability to respond to surveys on the day that the evaluation was conducted at their school. In 78 schools, only one teacher was available on the day the survey was conducted. For both Sukaabe Janngo and comparison schools, replacement schools representing 10 percent of the sampled schools were also identified. These were used in several cases because preschools no longer existed or because the sampled schools did not have grades 4, 5 or 6. Note on the presence of the Lecture Pour Tous program in Sukaabe Janngo schools: Along with Sukaabe Janngo, the US government funds other literacy programs in Senegal. Specifically, USAID is implementing the Lecture Pour Tous (LPT) program, which supports the Government of Senegal’s efforts to improve early grade reading. The program is implemented nationwide. It was scheduled to reach Saint Louis during the 2018/2019 school year14. In Saint Louis, the program is implemented directly through the Ministry of Education The evaluation team did not see LPT operating in the schools visited for qualitative research. However, 75% of school directors said they’re school is participating in the project. According to the Ministry of 12 Some schools had only one teacher for grades 4, 5 and 6. 13 The numbers in parenthesis are the actual sample which is less than the design because some schools had fewer than 2 teachers in grades 4, 5 and 6. 14 https://www.sil-lead.org/senegal-acr 10 Education, the first training of teachers happened from Dec 18 – 24. There also appears to be some political opposition to the program15. We don’t know the degree to which implementation has occurred in Sukaabe Janngo schools. 3.2.3 Qualitative interviews Key informant interviews were selected through purposeful sampling, and conducted in Dakar, Saint Louis, Dagana, and Podor. Table 4 shows the distribution of qualitative interviews by interviewee type. TABLE 4: QUALITATIVE INTERVIEWS Qualitative Type Number Directors 13 Teachers 20 School management committees 7 Government - Education 4 Government - Agriculture 3 3.2.4 Fieldwork Fieldwork was carried out between April 16th and May 10th of 2019. Fieldwork started with two days of interviewer training followed by two days of piloting the instruments. Two questionnaire revisions were made based on feedback from interviewers and survey supervisors: one after training and one after piloting. Data collection was conducted between April 23rd and May 9th by 20 interviewers managed by four supervisors. Qualitative data collection was completed between April 23rd and May 4th . 3.2.5 Limitations There are two key limitations of the baseline study both relating to the impact evaluation that will follow. McGovern-Dole worked with the same schools as those included in Sukaabe Janngo from 2015 to 2018. The previous phase featured many similar activities and objectives. Therefore, the evaluation with the comparison group schools will capture differences associated with two iterations of the program and not just the most recent version. This limitation is an unavoidable part of this study. Since school selection for McGovern-Dole in 2015 was not random, the evaluation team attempted to replicate the McGovern-Dole selection process in choosing comparison schools. However original school selection was not completely transparent. There was no record of scoring of each possible school in Saint Louis on all the criteria that reportedly were used for school selection. Furthermore, it would not have been possible for the evaluation team to assess community engagement at all 900 plus schools in Saint Louis if indeed that criteria were used. The selection of comparison schools was based on two criteria - a minimum number of students and school accessibility – that may not completely capture how McGovern-Dole schools were originally selected. We have included some covariates including on school 15 https://www.senenews.com/actualites/politique/le-financement-de-lusaid-pour-la-lecture-pour-tous-lpt-au￾senegal-serait-il-un-cas-de-detournement-de-fonds-publics_280862.html 11 management committee engagement that the impact evaluation team can use to help control for these differences. 4. Results The baseline study’s tasks were to operationalize a set of research questions included in the study’s scope of work, operationalize Sukaabe Janngo’s outcome indicators, and provide baseline values for research questions (when possible), outcome indicators, and project management plan indicators. This section provides a description of the research questions and the baseline study team’s recommendations for operationalizing them. The section also details Sukaabe Janngo’s outcome indicators and results from the baseline study’s qualitative and quantitative research. The baseline study’s indicator set, which provides baseline measurements for Sukaabe Janngo’s project management plan are included as Annex A. 4.1 Operationalized Research Questions 4.1.1 Sukaabe Janngo project level performance Research question: Have program outputs and outcome targets been achieved? (The outcomes requiring baseline values are listed below). This section describes how we operationalized the project’s outcome indicators. The results of measuring these indicators for the baseline study are reported in section 4.2. MGD S01: Improved Literacy of School Aged Children: The baseline study measured ASER reading level scores for treatment and comparison groups. The study disaggregates these scores by treatment/comparison, grade, and gender. The study provides a mean score for each disaggregation, as well as a distribution of scores among ASER’s five reading levels. Performance evaluations can assess improvements in literacy by re-administering the ASER test and assessing if mean scores have improved for each group, and assessing if the distribution of students scoring at higher reading levels has changed. MGD 1.1: Improved Quality of Literacy Instruction: This indicator is one of three that the baseline study team was required to define. It is also included in two methodological research questions below. After considering options such as Stallings classroom observation methodology16, which requires interviewers to assess instruction quality in real time, the baseline study team decided to utilize respondent assessment of literacy instruction quality. The team made this determination because a) it was not possible to guarantee access to the number of observation sessions necessary to guarantee statistical strength of the Stallings methodology, and b) the baseline team could not ensure that current and future evaluation teams would have sufficient training to make reliable and consistent assessments. The surveys ask student, teacher and director respondents to subjectively assess the quality of reading instruction. Qualitative research triangulates data collected through surveys by asking teachers and school directors about why students have difficulty learning to read and how teachers are trained to teach reading. 16 See https://www.worldbank.org/en/programs/sief-trust-fund/brief/the-stallings-classroom-snapshot 12 MGD 1.2: Improved Attentiveness: As with MGD 1.1, this indicator was one of the three that the baseline team was required to define. The baseline team utilized a question from RTI International’s 2018 baseline assessment for McGovern-Dole Mozambique, which assessed teacher perception of student attentiveness.17 We complement this question with student self￾reported attentiveness, and through enumerator counts of the number of times students lost focus during their surveys. This latter method is an objective measure of student attentiveness which, while not accounting for quality of education, does assess attentiveness that can be attributed to school canteens. The evaluation team triangulated survey data through interview questions that asked teachers and school directors how student behavior changed with changes in school feeding activities, to describe their observations on student attentiveness, and how they keep students attentive in the classroom. MGD 1.3: Improved Student Attendance: Future evaluations will measure this outcome by utilizing official school and government attendance records. The baseline study aimed to verify the reliability of these attendance records by comparing official enrollment figures with a headcount of students present on the day of the survey (see the second methodological research question below in section 4.1.5). The evaluation team triangulated attendance record data through teacher and director interviews. Teachers and school directors were asked about attendance levels, their process for taking attendance, why students are sometimes absent, and how they encourage students to attend school. MGD 1.3.2: Reduced Health-Related Absences: This outcome is the last of the three that the baseline study team was required to define. The team measured this outcome by a) asking students how many days of school they have missed in the current semester and then b) how many of these days were due to illness. The team also asked teachers and school directors in interviews why students were most often absent. MGD 1.3.4: Improved Student Enrollment: This outcome is assessed through school and government data. The baseline study measures average enrollment in school by grade. In interviews, teachers and directors were asked about the percentage of students who could enroll but don’t, why some children don’t enroll in school, and how teachers and directors encourage enrollment in their communities. MGD S02: Increased Use of Health, Nutrition and Dietary Practices: This outcome is assessed through school director interviews, which ask the number of cooks and those involved in commodities that have been trained and are practicing improved techniques, and school checklists, which check to see if schools are adequately equipped and facilities are in sufficiently good condition to meet the outcome’s targets. Students were also surveyed about their hand washing practices. 17 https://apps.fas.usda.gov/fais/public/files/Mozambique_McGovernDole_Final%20Baseline%20Report_2.18.18.pdf 13 4.1.2 School feeding and nutrition Research question: What is the effect of school feeding on attendance, enrollment and attentiveness? For the performance evaluations, future evaluations may answer this question by comparing baseline and endline values for MGD 1.2, 1.3 and 1.3.4 above. For the impact evaluation, a difference in differences methodology can be used to compare the change over time that occurs in Sukaabe Janngo and comparison schools at baseline and the impact evaluation. Research question: How effective are take home rations at increasing attendance among boys in 5th and 6th grades? Future evaluations may answer this question by comparing attendance differences for boys in 5 th and 6th grades that have received take home rations with four groups: ▪ Girls in 5th and 6th grades that have received take home rations. ▪ Boys and girls in 5th and 6th grades at comparison schools that have not received take home rations. ▪ Boys and girls in 4th grade at comparison schools that have not received take home rations. Future evaluations should also interview teachers, school directors, and other stakeholders that observe children about their perspective on the distribution of food ration distribution’s effect on change in attendance. Research question: What share of the total recommended per student feeding schedule in McGovern￾Dole schools sourced from non-USG sources (disaggregated by source)? Future evaluators may assess this question based on data from the previous McGovern-Dole iteration’s final evaluation. This data should be regularly collected by school directors or school management committees with support from Sukaabe Janngo’s monitoring and evaluation team as it is vital for both Sukaabe Janngo and school canteen management. The final evaluation of McGovern-Dole last year found the following contributions to primary school recommended feeding requirements. In total, primary schools were able to fulfil 22 percent of requirements (or 26 percent with school farms) in 2018. This compares to 10 percent for preschools, which do not receive government contributions for school feeding. Note that these figures are likely to slightly underestimate actual contributions, since many schools charge student families small weekly or monthly fees that contribute to school feeding. TABLE 5: SHARES OF RECOMMENDED PER STUDENT FEEDING REQUIREMENTS BY SOURCE (%) School Type Granary Government School farm (where present) Total (w/ farm) Primary 9 13.1 3.6 22.1 (25.7) Preschool 10 0 10.9 10 (20.9) Sukaabe Janngo should work with schools to continue to collect information on school feeding contributions by different sources. This will help schools to manage their school feeding programs as well as Sukaabe Janngo to help schools’ transition towards self-sufficiency. 14 Categories to track include granary cash contributions, granary food contributions (by type), student family fee cash contributions, government cash contributions and school farm food (by type) and cash contributions. This data should be tracked by schools on an ongoing basis and submitted to Sukaabe Janngo quarterly. 4.1.3 Education and Literacy Research question: How effective are reading-oriented extra-curricular activities in improving literacy? The baseline survey asks students if they have participated in extra-curricular reading activities. If the response is no, they are asked if they would have liked to participate. These two questions will allow future researchers to control for self-selection by comparing reading scores of those participating who like participating with those who are participating but do not like participating, those not participating but wishing to participate, and those not participating who do not want to participate. Research question: How effective are teacher trainings? The baseline study did not take a measurement regarding the effectiveness of teacher trainings as CPI has not yet implemented any trainings in the current iteration of the program. Future evaluations may compare improvements in reading scores at schools where teachers received training, controlling for other variables such as school’s relative level of food security, location, and size to understand the effectiveness of teacher trainings. Evaluators should also conduct contribution analysis to uncover the degree to which teacher training contributes to changes in student achievement. Contribution analysis is effective because it uncovers alternative explanations for observed changes and qualifies the degree to which the measured variable created change. 4.1.4 Health Research question: What is the effect of deworming medicine on student attendance? The government provides deworming medicine once annually to all schools. CPI will be providing deworming medicine one additional time during the year to Sukaabe Janngo schools. Therefore, the control, to be made through the impact evaluation, is in attendance for Sukaabe Janngo school students receiving two doses of deworming medicine annually versus attendance for comparison school students receiving only one annually. Required attendance data is found in government attendance records. Research question: What is the effect of latrine quality on student attendance, especially for girls? The baseline study asked students to assess how comfortable students are using school latrines. Additionally, school checklists feature interviewer assessments of latrine quality. Performance and impact evaluators may relate these results to school attendance, especially for girls, through regression or simply disaggregation. Research questions: How closely are students following handwashing recommendations? Are they practicing at home too? The student questionnaire asks about handwashing behavior at home and at school. 15 4.1.5 Methodological Research questions: What is the best way to measure the three undefined Sukaabe Janngo outcome indicators (MGD 1.1, 1.2 and 1.3.2)? This research question has been addressed above. Research question: How reliable is school and government-collected attendance and enrollment data? How can the accuracy be improved? The baseline team took attendance for grades 4, 5 and 6 at each school and compared it with the official enrollment in each grade. Additionally, qualitative interviews asked how attendance was recorded and kept, as well as possible incentives for misreporting. As discussed in section 4.2, the baseline study found that attendance and enrollment data is reliable. The baseline estimated an average of 94 percent enrollment. Tardiness and leaving school early appear to be more important issues. Regarding enrollment, while the data is accurate, the source of new enrollment is unclear. As discussed later in this report, some teachers speculated that students move from schools that offer school feeding to those that don’t. Future evaluations should include a question that asks students if they have changed schools since the beginning of the program and the reason for that change. 4.1.6 School Feeding Sustainability Research question: What is government capacity to manage school feeding at regional and national levels? This research question is assessed through qualitative interviews with government officials. The baseline study found that government’s capacity to manage school feeding is extremely limited. One indication of the difficulty of managing school feeding programs is the rapidity which with school feeding was greatly reduced or ceased completely after McGovern-Dole ended in November 2018. The regular encouragement that Counterpart provided as well as management skills and holding school management accountable were drivers of McGovern￾Dole’s success. A teacher the baseline team interviewed said that they had a canteen coordinator who stopped volunteering, explaining that, “The counterpart guy disappeared, the school feeding program stopped.” Additionally, McGovern-Dole reaches schools. A school inspector that the baseline team interviewed commented that, “We can’t reach remote schools.” Government does not have the capacity to provide the leadership or coordination, including volunteer management, that Sukaabe Janngo provides. That said, Government may have the capacity to raise and distribute funds for use in school canteens. In the past it has allocated funds for school use and implemented anti-fraud mechanisms that help those funds go to their intended use. Unfortunately, as of the time of the baseline study, the central government had not allocated funds to schools in Saint Louis department. Some teachers and school directors said that the delay was caused by the recent presidential election, although it is unclear why an election should hold up a bureaucratic mechanism. A school director in Dagana suggested that, 16 without the benefit of Counterpart’s advocacy efforts, the school would not have received government funding last year. At the national level, The Division of School Canteens (DCS), has improved its capacity to manage its school feeding partners throughout the country. Working with the World Food Programme, DCS has developed a system for capturing and processing school feeding information from its many partners. That said, the DCS secretary feels that DCS’ role should grow from one of assisting international partners in setting and implementing the school feeding agenda, to developing a cohesive national policy driven by the Government of Senegal. Future evaluators may conduct interviews with government and school officials to determine if government’s capacity to support school feeding has changed from this baseline determination. Research question: What commitment has the government shown on school feeding? (e.g. do they have a school feeding policy, clearly defined roles for managing school feeding, plans to expand school feeding budget) This research question was assessed through qualitative interviews with government officials. The Government of Senegal maintains the DCS office, whose primary mission is liaising with school feeding partners, assisting in the coordination of school feeding programs, and maintaining data that school feeding partners collect. The Government also has a policy of setting aside 16 percent of the public education budget to be distributed to schools for providing food to students. This allocation amounts to around $500.00 per school for the school year. As mentioned above, the Government is working with the its school feeding partners, such as the World Food Program, Counterpart, and GRDR, to upgrade its database so that it will easily accept data from all school feeding partners. The DCS secretary felt that the lack of a school feeding advocacy platform softened the government’s commitment to school feeding. He said that lack of a communications campaign and platform for giving voice to the school feeding issue allowed politicians to minimize resources devoted to school feeding. He pointed to the Cote d’Ivoire government’s management of school feeding programs as a possible model that Senegal could follow. Particularly, the Secretary felt that whereas DCS is responsible for coordinating and responding to the agendas of its school feeding partners, Cote d’Ivoire’s school feeding management bodies had created an integrated national program that set the agenda and informed partner activities. Currently, the government understands the importance and political appeal of school feeding programs but has not provided sufficient budget or integrated policies18. Future evaluators may track new policies and initiatives to assess if the government’s commitment has changed. 18 These issues are described in more detail in the school graduation section below. 17 4.2 MGD SO1 - Improved Literacy of School Aged Children The ASER reading test that the research team administered produced the baseline measurement for determining if Sukaabe Janngo improved literacy of school aged children at the time of mid-term, final, and impact evaluations. This section provides ASER results in two ways: average reading level by grade for treatment and comparison schools, and the distribution of scores among five reading levels by grade19 . MGD SO1 seeks to measure an increase in the “percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text.” The baseline research team suggest that Sukaabe Janngo’s target should be that all students in grades 4, 5, and 6 read at ASER level 5, meaning that students can read text at grade 2 level. Table 6 shows average reading levels for students in grades CE2 (4th), CM1(5th), and CM2(6th) in treatment and comparison groups as measured by the baseline study. There are five levels that a student could demonstrate: 1) cannot recognize letters; 2) can recognize letters but cannot read words; 3) can read words but cannot read sentences; 4) Can read text at grade 1 level but not higher; and 5) can read text at grade 2 level. TABLE 6: AVERAGE READING LEVEL BY GRADE AND TREATMENT/COMPARISON Grade All Treatment Comparison CE2 (4th) 3.64 3.62 3.66 CM1 (5th) 4.2 4.23 4.17 CM2 (6th) 4.57 4.65 4.49 The baseline team conducted a two-tailed t test on the mean reading scores of the comparison and treatment populations. The test produced a p-value of 0.474, indicating that there is likely no difference between the treatment and comparison populations20 . The baseline study translates SO1 into the increase between baseline and subsequent evaluations in the percentage of students who read at level 5. At the mid-term and final evaluation, one means of measuring progress is to determine if the percentages of students' reading at higher levels has increased, and the percentages of students read at lower levels has decreased. Figure 6 below illustrates the distribution of reading levels by grade and comparison/treatment. In treatment schools, 35.22 percent of CE2 students, 57.64 percent of CM1 students, and 74.4 percent of CM2 students 19 Annex A includes the average reading level disaggregated by comparison/treatment, grade, gender, and location. 20 By convention, p-value less than 0.05 indicates a strong possibility that populations are significantly different. For example, at the end of the program, when the impact evaluation is conducted, a p-value of less than 0.05 would indicate that the studied populations had become significantly different in their reading scores during the program’s FIGURE 6: READING LEVEL DISTRIBUTION BY GRADE AND TREATMENT/COMPARISON 18 read at level 521. In other words, 64.78 percent of CE2 students, 42.36 percent of CM1 students, and 25.6 percent of CM2 students could show progress by improving their reading level by mid-term and final evaluations. The full distribution of reading scores is included in Annex A. 4.2.1 MGD 1.1: Improved Quality of Literacy Instruction The research team designed questions for teachers, school directors, and students to measure the change in quality of literacy instruction. Teachers The teacher survey asked teachers, in their opinion, how much literacy instruction needed to improve at their school to best teach students. 62.3 percent of teachers said that literacy instruction had to improve “to a large extent” or “to a very large extent” to best serve students. 26.6 percent said that instruction had to improve only “to a moderate extent. The survey also asked teachers what they thought the best part and worst part of teaching reading class was. Teacher’s said they enjoyed teaching grammar (35%) and reading (30%) the most. They struggled the most with spelling (21%) and vocabulary (20%). The highest-level indicator is the percentage of teachers that say they received instruction in teaching reading in the past school year. Teachers that received training were asked how useful they found the training. We also asked teachers how well students were learning to read and the degree to which reading instruction could be improved at their school. Overall, 43 percent of teachers said that they’ve received training during the previous school year, comprising 46 percent of treatment school teachers and 38 percent of comparison group teachers. Though the average number of teachers saying that they’d received training was different, a t test resulted in a p-value of .12, indicating that the null hypothesis, that the populations were equally likely to have received training, cannot be rejected. Of the teachers that received training, 85.4 percent said that they found it very useful and 14.6 percent said that found it somewhat useful. No teachers said that the training was not at all useful. TABLE 7: TEACHER TRAINING EXPERIENCE duration. The lower the p-value, the greater confidence one may have that the populations are significantly different in regards to a measurement. In this case, the very high p-value of 0.474 indicates likely no difference. 21 Mean reading levels between boys and girls do not appear statistically different. Comparing the data sets for boys and girls resulted in a p-value of 0.8537, indicating that there is a high level of statistical similarity between mean reading scores for boys and girls. Response Percent of Teachers Teachers that received training in literacy instruction 43% Teachers that found the training very useful 85.4 % Teachers that found the training somewhat useful 14.6% Teachers that said literacy instruction had to improve “to a large extent” or “to a very large extent” 62.3% 19 In interviews, the research team asked teachers and directors why some students had difficulty learning to read. Teachers had a variety of answers, but three themes stood out. The first was that there is no culture of reading in students’ homes. A school director in Podor put it like this, “One of the reasons is, once back home they don't read or learn at all. Students need to study at home, parents don't feel it's important.” Another common theme was that teachers felt the system for reading instruction changes frequently and it is difficult to keep up. A teacher in Podor said that the government changes the reading instruction system yearly, and the change was confusing for the students and difficult for teachers to manage. Lastly, some teachers stated that lack of reading and instruction materials make it difficult for students to learn to read. A teacher in Podor pointed out that lack of electricity in the schools meant that teachers could not create their own instructional materials. Directors The baseline survey asked directors two questions related to literacy instruction: ▪ How would you rate the quality of reading instruction for 4th, 5th and 6th graders at this school in the current school year? ▪ To what extent does reading instruction at this school need to improve to best teach students? ▪ 40.79 percent of school directors said that reading instruction for CE2, CM1, and CM2 was fair or poor. 51 percent rated reading instruction for those grades as good. Treatment school directors thought better of their reading instruction. 33.5 percent of treatment school directors rated the quality of literacy instruction as fair or poor, and 54.2 percent rated it as good. Only 12.23 percent rated literacy instruction as very good or superior. TABLE 8: DIRECTOR RATING OF LITERACY INSTRUCTION Director Rating of Reading Instruction for CE2, CM1, and CM2 All Treatment Comparison Fair or Poor 40.79% 33.55% 51.07% Good 51% 54.2% 45.7% Similar to the teacher results, 57 percent of all school directors said that literacy instruction had to improve “to a large extent” or “to a very large extent” to best serve students. 22.4 percent reported that literacy instruction had to improve to a moderate extent. TABLE 9: (DIRECTORS) EXTENT TO WHICH READING INSTRUCTION NEEDS TO IMPROVE Option All Treatment Comparison A very small extent 4.93% 5.34% 4.35% A small extent 15.70% 16.80% 14.10% A moderate extent 22.40% 19.80% 26.10% A large extent 40.40% 42.70% 37% A very large extent 16.60% 15.30% 18.50% 20 Students The student survey asked three questions related to literacy instruction: • How much do you like your reading class in school? • What's the best part of reading class? • What's the worst part of reading class? Students on the whole seem to like reading class. 83.4 percent reported that they liked it “a lot” or “pretty much.” Only 3 percent said that they didn’t like reading class. Those proportions remained approximately consistent for the total population, treatment, and comparison groups. 4.2.2 MGD 1.2: Improved Student Attentiveness To measure student attentiveness, surveys asked teachers about their students’ attentiveness and students about their own attentiveness. The enumerators who conducted student surveys also made observations of the respondent’s attentiveness. Teachers The baseline survey asked teachers to estimate the percent of the class that is usually paying attention in class in the current semester. About 26 percent of teachers responded that half or less than half of students usually pay attention, and about 74 percent responded that more than half or all students usually pay attention. These proportions remained nearly the same for treatment and comparison groups despite 94 percent of treatment school directors saying that they had a school feeding program, while only 8 percent of comparison group school directors said they did not have a school feeding program. The unintuitive result could arise because, while treatment school directors have a school feeding program, qualitative data collection suggests that it may not have been operating during the current semester or may have only operated on a limited basis. TABLE 10: WHAT PERCENTAGE STUDENTS PAY ATTENTION IN CLASS? Option All Treatment Comparison Less than half 7.29% 6.41% 8.54% Half 18.80% 17.50% 20.70% More than half 60.30% 59.80% 61% All 13.60% 16.20% 9.76% Students The baseline survey asked students whether they usually found it very easy, easy, hard, or very hard to pay attention in reading class. 86.4 percent of students said paying attention was easy or very easy. This percentage was consistent across comparison and treatment groups. Enumerator observation As a check on student and teacher reporting about attentiveness, the baseline team asked enumerators to observe how often students became distracted during the survey. Enumerator observation supports 21 the level of attentiveness that students and teachers reported. In 1105 student surveys, enumerators reported that 853 students did not become distracted at all during the survey. Of the remaining 252 students, 116 were only distracted one time. 4.2.3 MGD 1.3 Improved Student Attendance The baseline study sought to discover levels of student attendance by doing a headcount at schools in the sample and asking teachers and school directors through interview questions their perception of student attendance. The baseline team conducted the headcount by first checking the school director’s enrollment registry to record the official number of students enrolled in each grade, and then counting the number of students at school on that day to see how it compared to the number enrolled. The baseline team found that attendance rates are high for all three grades and for treatment and comparison schools, as shown in table 11. For CE2 and CM1, p-values were 0.262 and 0.717 respectively, meaning that the team could not reject the hypothesis that there was no difference in mean attendance between comparison and treatment groups. For CM2, the p-value was 0.041, indicating that there may nor may not be a difference in mean between the comparison and treatment groups. TABLE 11: ATTENDANCE RATES BY GRADE Grade Total Treatment Comparison CE2 94% 96% 91% CM1 95% 96% 94% CM2 96% 98% 92% The baseline study found that there are two aspects of attendance; the number of days attended, and the quality of those days. Whether or not meals are provided does not appear to change the quantity of days attended. Of the 14 schools that the baseline team visited to conduct qualitative research, 8 were not conducting any school feeding activities and 5 were only providing food on Tuesdays and Thursdays. One school was providing five breakfasts, consisting of millet and sugar, and two lunches every week. The director at this school, named the Kuer Mbaye Peul school, reported that the meals provided by the school have less nutritional value than McGovern-Dole supported meals, and that the students don’t like the food as much. Despite the great reduction in the provision of meals for school feeding, none of these schools saw a decline in attendance. The high levels of attendance in treatment and comparison schools supports the finding that school feeding does not drive attendance, as displayed in figure 9, which shows the attendance rates for schools with FIGURE 7: ATTENDANCE RATES WITH AND WITHOUT SCHOOL FEEDING 22 and without feeding programs. As the figure demonstrates, there does not appear to be a correlation between the existence of a feeding program and attendance rates for this population. Aside from receiving a meal, students that are enrolled in school appear to have several other incentives for attending. Some students come because they are eager to learn. Others come because there is not much to do in rural areas other than farm or go to school, and school is where their friends are. Several teachers that we spoke with said that they go to households to collect children and make sure their parents understand the importance of education. One school director noted that at his school, “We have a list for tracking who is missing. If we find a missing student, we go to the house to know why the student isn't in school.” Interviews with teachers, school directors, and management committee members revealed that, while attendance does not depend on the existence of a food program, promptness and attention do. When asked how school feeding changes the behavior of students, the most common answers were that students were often late to school when the school feeding program wasn’t operating, and that students had less energy and were less focused. Consequently, students’ level of achievement suffers. A teacher in Podor commented that, “Their (students) energy is lower. When Counterpart was here, kids were stronger. When they stopped (school feeding) activities, the level of achievement is lower. Energy is lower when there is no food.” A teacher at a different school in Podor explained how the absence of school feeding activities related to tardiness, “With the school feeding program, students didn't wait for breakfast at home, when [the program] ended, students had to wait at home and were late for school.” Another teacher in Podor explained how lack of a school feeding program affects afternoon classes: “Most of the population is poor. They don't have means to contribute. When school feeding was here, there was no delay. The kids live far away, and it takes a long time to come after eating when they wait to eat at home.” One teacher in Podor mentioned that attendance at his school was seasonal. He commented, “Sometimes [students] miss school during harvest months. Harvest happens around opening day, during the first month of the school year.” 4.2.4 MGD 1.3.2 Reduce Health Related Absences Schools do not keep records on why students are absent when they miss school. To discover a baseline measurement of health-related absences, the baseline study asked students how many days of school they missed in the previous semester. Few students reported missing days. The average number of days students estimated they missed was about half a day. Only 285 of the 1208 students surveyed reported missing at least one day of school. Of those, the most common reason they missed school was illness. TABLE 12: REASONS STUDENTS MISS SCHOOL (n = 285) Reason Percent I was sick 90% Someone in my family was sick 2.11% I needed to work on the farm 0 I needed to do other work 0.70% Other 6.32% 23 The 6 percent of students that chose “other” had a variety of reasons why they had missed school. These responses included travel, not realizing the teachers had returned to work, having to renew civil status papers, an absent teacher, and oversleeping. 4.2.5 MGD 1.3.4 Improved Student Enrollment The baseline study measured average school enrollment by grade. Table 13 presents these results. TABLE 13: AVERAGE NUMBER OF STUDENTS PER SCHOOL BY GRADE AND TREATMENT/COMPARISON Grade All Treatment Comparison CE2 29 30.6 27 CM1 25.5 25.4 25.5 CM2 20 20.1 19.8 The research team asked school directors and teachers if the rate of enrollment in their school was high or low and the reasons students enrolled or did not enroll in school. Directors and teachers thought that the enrollment rate compared to the population of eligible children in their school’s community was high. Some felt that school feeding was a motivator for enrolling in school. One school director mentioned that, “Sometimes little kids say that he can't wait for school because of the school feeding.” A teacher in Podor mentioned that school feeding draws some students away from schools that don’t serve food to those that do. Several teachers mentioned that they go to households in their communities to encourage parents to send their students to school. A teacher in Guede said, “The rate of children who enroll in this village is high. The teachers meet with the population and convince them, and make sure they know the date of enrollment.” Directors and teachers mentioned some common reasons that students might not attend school. A school director mentioned that at his school they have trouble enrolling girls in CM2 (6th grade) because girls in their area often marry at that age. Other reasons for not enrolling included going to religious school or staying home to work on a farm. 4.3 MGD SO2 Increase use of health, nutrition, and dietary practices The study tracked baseline measurements under SO2 that were relevant to the activities that the program will implement. These activities include: ▪ Provision of latrines and water stations: this activity includes promotion of good hand washing practices among students. ▪ Maintenance or replacement of canteen equipment: includes energy efficient stoves and cooking structures. ▪ Training teachers on good health and nutrition practices ▪ Distribute deworming medicine, vitamins and minerals The last two activities are output based activities that require Sukaabe Janngo to count the number of teachers trained or amount of medicine distributed, and therefore do not require a baseline measurement. The baseline study focused on the first two activities. 24 4.3.1 MGD 2.4 Increased Access to Clean Water and Sanitation School Checklist The school checklist that enumerators administered at each of the schools identified the following related to water and sanitation: ▪ The number of latrines at the school ▪ Whether the latrines were marked for boys and girls ▪ Whether the latrine was connected to water ▪ Whether water connections were in working condition ▪ The cleanliness of the latrine ▪ The degree to which latrines were well maintained ▪ Whether or not there was a sink for washing hands and whether it was clean ▪ If soap and a towel was available The average number of latrines at a school was 2.6 for all schools. There was not a meaningful difference between treatment schools, with 2.83 latrines available, or comparison schools, with 2.43 latrines available. Overall, 59 percent of schools had latrines marked for girls or boys. However, there was a considerable difference between treatment schools, with 70 percent marketed appropriately, and treatment schools, which only had 21 percent marked appropriately. For all schools that had a latrine, 57.1 percent were rated somewhat clean or not clean by the baseline study’s enumerators, with 23 percent counted as not clean. The level of cleanliness differed considerably between comparison and treatment groups and shown in table 14. Nearly half of treatment schools were at least mostly clean, whereas only about one third of comparison schools were at least mostly clean. TABLE 14: CLEANLINESS OF LATRINES Level of Cleanliness – Latrines Total Treatment Comparison Very clean 15.40% 18.50% 10.60% Mostly clean 27.50% 30.80% 22.30% Somewhat clean 34.20% 32.20% 37.20% Not clean 22.90% 18.50% 29.80% In terms of the maintenance of latrines, there was also a difference between treatment and comparison schools. 63.7 percent of treatment schools had latrine structures that were well maintained or very well maintained, while 53.2 percent of comparison school latrines received the same rating. 35 percent of treatment schools and 15 percent of comparison schools were equipped with a sink for washing hands. Table 15 shows the state of these hand washing areas. 25 TABLE 15: STATE OF HAND WASHING FACILITES Question Options Total Treatment Comparison Is the sink area clean? Very clean 30% 31.60% 25% Mostly clean 40% 42.10% 33.30% Somewhat clean 22% 21.10% 25% Not clean 8% 5.20% 16.70% Is there soap? y/n 20% 27% 11% Is there a clean towel or other material for drying? y/n 7% 1% 11% Student Survey To understand student hygiene behavior, we asked students whether or not they wash their hands at school. We asked students who said they don’t wash their hands at school the reason for not washing their hands22 . Table 16 shows the number of students that said they wash their hands at school. TABLE 16: DO YOU WASH YOUR HANDS AT SCHOOL? Response All Treatment Comparison Yes 79.20% 89.10% 66.00% No 20.80% 10.90% 34.00% We asked students who said they do not wash their hands at school why they do not wash. The highest percentage said that there was no place to wash hands at their school. As mentioned above, only 26 percent of schools overall and 35 percent of treatment schools had a place to wash hands, this answer makes sense. Others mentioned that there was no water or soap. Several treatment schools that the evaluation team visited had out of service hand washing equipment that McGovern￾Dole had provided during the previous phase of the program (figure 8). At other schools, McGovern-Dole provided equipment was well supplied and in use (figure 9). 22 We also asked students whether they washed their hands at home. 100 percent of students said they did, which did not reveal much about their behavior FIGURE 8: OUT OF SERVICE HAND WASHING EQUIPMENT 26 TABLE 17: WHY DON'T YOU WASH YOUR HANDS AT SCHOOL? Option All Treatment Comparison No place to wash hands 47.40% 41.00% 50.00% No water 28.00% 26.5% 28.6% No Soap 23.5% 30.10% 20.90% Wash basin isn’t clean 1.04% 24.10% 0.49% 4.3.2 MGD 2.6 Increased Access to Requisite Food Prep and Storage Tools As one would expect, treatment schools were much more equipped for school feeding programs than comparison schools. 97.42 percent of treatment schools had a canteen, compared to just 10.66 percent of comparison schools. In this section, the report includes information on treatment schools and the degree to which they have requisite conditions to prepare, store, and serve food. Table 18 below shows the degree to which treatment schools are equipped to conduct school feeding programs, disaggregated by location: TABLE 18: DEGREE TO WHICH TREATMENT SCHOOLS ARE EQUIPPED FOR SCHOOL FEEDING Dagana Podor Saint Louis Does this school have a food storage area? 96% 99% 84% Is the storage area clean? 82% 79% 81% Is the roof covered? 96% 100% 94% Is the food storage area secure? 96% 99% 100% Are there signs of damage from rodents or other pests? 7% 17% 19% Does this school have a cooking area? 100% 96% 95% Is there a structure for the cooking area? 74% 79% 84% Is the cooking area generally clean? 79% 84% 89% Is there soap or detergent? 34% 35% 83% Are there adequate utensils for cooking and eating? 98% 98% 100% Is there an energy efficient stove? 98% 98% 100% Is there a way to wash your hands before eating? 98% 96% 94% FIGURE 9: MGD PROVIDED EQUIPMENT IN USE 27 4.4 School Graduation from Sukaabe Janngo The Sukaabe Janngo 2019 workplan details the program’s strategy for graduating schools. The document does not specifically define graduation. However, it indicates that graduation involves no longer depending on McGovern-Dole contributions to support school feeding.23 The Work Plan also states that Sukaabe Janngo’s school graduation strategy has two components: 1. Establishing “a national-level implementation strategy and budget that sustains school feeding by the GoS,” and, 2. “Expanding capacity and commitment at all levels: from schools to districts to the national level.”24 National Level Implementation Strategy and Budget It does not seem likely that Sukaabe Janngo will create the effective advocacy and communications assets that the Ministry of Education requires to establish a strategy and attract a sufficient budget to support school feeding before the end of the current program. As discussed in section 4.1.6, school feeding sustainability, this first component of the graduation strategy is a complex endeavor that will require dedicated time and resources. Sukaabe Janngo’s workplan does not mention time or resources dedicated to advocacy or communications toward the goals of establishing a strategy or obtaining budget allocation. It will be difficult to obtain a budget commitment from the government beyond its current commitment. While Senegal’s economy is slightly more stable than average compared to those in its risk group25, it still carries debt amounting to 61 percent of its GDP, and a budget deficit of about 4 percent. These figures indicate that while funds are available for public projects, the government will spend them carefully and work to ensure the highest return on investment both in terms of results and votes. In order to obtain funding for school feeding, the Ministry of Education, through DCS, will have to convince the government that school feeding meets its investment criteria. Persuading the government will require having access to data and creating a communications plan that increases urgency for school feeding among politicians and the public. As the DCS official we met with indicated, and as we reported in section 4.1.6, budget goes to the departments that have adequate communications platforms and strong advocacy channels. The mid-term evaluation of the last iteration of McGovern-Dole, finalized in August 2017, stated that, “Responsible government officials must rally support in their legislature, provide the President with the information and materials he needs to raise a budget, and put the structures in place to manage the program.” The first step in the process, providing the information that the President requires, is still a work in progress as of this baseline study finalized two years after the mid-term evaluation. If the information gathering component is completed, time remains short to quickly follow with by rallying support in the legislature, receiving a budget commitment from the government, and distributing those funds to schools before the end of the current Sukaabe Janngo program. 23 The closest the Work Plan comes to defining graduation is when it states, “(Sustainability Action Plans) developed under Activity 1 will identify and incorporate the necessary steps towards graduation from USDA-funded school feeding” (page 16). Presumably meaning that graduated schools will no longer receive USDA commodity contributions. 24 MGD Annual Work plan, Revised and Submitted on January 15, 2019 , page 29. 25 Oxford Economics Senegal Forecast, January 25, 2019. 28 Expanding capacity and commitment at all levels The graduation strategy as articulated in the revised workplan focuses on transitioning schools from obtaining commodities through Sukaabe Janngo to purchasing commodities through the TACSS LRP initiative. The work plan states, “It is anticipated that the TACSS LRP project will assist the transitioning process and support in meeting the objective of graduating up to 200 Sukaabe Janngo participating schools." However, food availability is not a crucial obstacle to school graduation, rather the central constraint is finding resources to purchase food. The final evaluation of the last iteration of Sukaabe Janngo found that granaries and government contribution provided approximately 22 percent of Sukaabe Janngo feeding requirements. It remains unclear how schools that graduate from Sukaabe Janngo will acquire resources to make up for the 78 percent that the program supplies. During the baseline study, teachers and school directors reflected on the constraints poverty in their school community put on obtaining school feeding resources. As one teacher put it, “Most of the population is poor. They don't have means to contribute.” Another teacher reflected that, “the teachers now kick in for food for school feeding. Parents contribute 600 CFA per month. Some poor parents can't give this amount. Teachers and directors support fees for food. They add money for kids that can't bring anything.” Another constraint is the program’s reliance on volunteers. The work plan says, “SMC purchasing groups will plan and purchase rations to prepare daily meals at school for 30,000 beneficiaries in 200 McGovern-Dole transitioned schools by IY3.” The baseline study team understands that SMCs will attend trainings that will give them the skills required to implement these activities. However, a discussion of incentives is missing from the strategy. The program will ask community members, teachers, and school directors to take on additional work and responsibility, but it is unclear why they will be glad to do so. As the School Categorization, Graduation and Sustainable Community-Led School Feeding study that Sukaabe Janngo commissioned in January 2019 concluded, “Without strong school leadership, school feeding programs are not sustainable…26 ” The evaluation makes recommendations about how these constraints might be managed in section 6.9. 26 Lane, Katheryn, “School Categorization, Graduation and Sustainable Community-Led School Feeding”; January 28, 2019; pg 7. 29 5. Recommendations and Conclusions 5.1 MGD S01: Improved Literacy of School Aged Children All students in grades 4, 5, and 6 should read well enough to reach level 5 on the ASER test27. Progress toward achieving Sukaabe Janngo’s literacy goal is measured two ways; first by improving the mean reading scores of targeted populations, and also by shifting the distribution of students reading at lower levels to higher levels. At mid-term evaluation, the degree to which this distribution has shifted will indicate how close the program is to achieving its targeted result. 5.2 MGD 1.1: Improved Quality of Literacy Instruction The baseline study measured students’, directors’, and teachers’ perception of instruction quality, and triangulated that information with qualitative interviews. Future evaluations should ask the baseline questions about instruction quality and also the degree to which quality could improve. Future evaluations should add questions about the degree to which instruction quality has increased since the launch of the program. Qualitative research should triangulate by asking teachers, directors, and students if they think instructional quality has improved, and if so, why they think it has improved. Examples of specific improvements will increase the program’s understanding and feed into the contribution analysis discussed below. 5.3 MGD 1.2: Improved Attentiveness The baseline study attempted to collect an objective measure of attentiveness by asking enumerators to note the number of times a student’s attention strayed during the survey. This question failed to produce a meaningful measure of student’s attentiveness. Still, teacher’s estimation of student attentiveness combined with descriptions of classroom behavior in qualitative interviews provides a baseline for the program. Sukaabe Janngo’s targeted students are mostly attentive but struggle somewhat during the school day. Teachers and SMC members report that when school feeding activities cease, students have visibly less energy, greater difficulty focusing on their studies, and consequently school achievement suffers. An objective measure of attentiveness, without investing time and resources into a sophisticated assessment methodology remains elusive. The Sukaabe Janngo program could choose proxy indicators if it wished to track increases, like completeness of student assignments, or average grades on regularly taken tests. Tracking these indicators would also require time and resources. In lieu of a better alternative, teacher estimation of student attentiveness compared to data collected through interviews appears sufficient to measure improvement. 5.4 MGD 1.3: Improved Student Attendance The final evaluation of the previous iteration of McGovern-Dole reported high levels of student attendance based on school and government attendance data. The levels were so high that the evaluation questioned whether the data was accurate. Accordingly, this baseline study compared enrollment to a headcount at the time of the evaluation. The results confirmed that school attendance rates for Sukaabe Janngo schools in Saint Louis are very high. The study estimates that attendance rates 27 The ASER has five possible scores: 1) cannot recognize letters; 2) can recognize letters but cannot read words; 3) can read words but cannot read sentences; 4) Can read text at grade 1 level but not higher; and 5) can read text at grade 2 level. 30 are around 94 percent. The student survey and interviews with teachers and directors support this finding. A very small number of students reported missing even one day of school, and teachers and directors in every school that the baseline team visited to conduct interviews reported high attendance levels. The baseline study showed that attendance has no relationship to whether or not a school has a school feeding program. However, attendance does not describe the degree to which students are in the classroom learning. Interviews indicated that when school feeding programs aren’t active, students are more frequently late to school and do not stay at school as long during some days. The Sukaabe Janngo program should consider substituting an indicator that measures promptness and time spent in the classroom for Sukaabe Janngo 1.3. Promptness could be measured by asking program coordinators to spot check schools on a regular basis. The coordinator could arrive before school opens and count the number of students that arrive after a set time, such as after class has begun. A decrease in the number of students arriving late, or for impact evaluation, fewer students arriving late at treatment versus comparison schools, would indicate the effectiveness of Sukaabe Janngo. 5.5 MGD S02: Increased Use of Health, Nutrition and Dietary Practices The baseline study checklist revealed that Sukaabe Janngo schools are still well equipped to promote good food handling, nutrition, and dietary practices. One exception was with hand washing facilities. At final evaluation in 2018, 98 percent of McGovern￾Dole schools had a place for students to wash hands28. This baseline study’s enumerators found that 35 percent of schools still had a place for students to wash their hands, about 25 percent of those weren’t clean, and only 27 percent of those had soap. This finding was supported by qualitative researcher visits to schools, which revealed equipment that the McGovern-Dole program had distributed during the previous school year was currently in disrepair. In the current phase of the project, Sukaabe Janngo should dedicate resources to discovering why equipment became unavailable. Interviews suggest that one reason is the scarcity of water. Several teachers mentioned that they or someone else associated with the school had to carry water from an inconvenient location every day. A teacher in Dagana said that there was no water at his school, and either a cook or student had to carry it every day. Another teacher said that lack of water made the environment difficult to teach in. Another possibility is that no stakeholder has clear responsibility for maintaining and promoting use of hand washing equipment. After the reasons are discovered, the program should promote solutions to ensuring maintenance and use of hand washing equipment. Schools could be grouped according to common problems with this issue and supported to efficiently address their difficulties accordingly. 5.6 Measuring effectiveness of extra-curricular reading activities in improving literacy In addition to asking students questions included in the baseline survey, future evaluations should include questions about the quality of extra-curricular activities and whether or not materials were always available. Interviews should ask teachers, students, and directors about the consistency with which activities were organized. 28 McGovern-Dole International Food for Education and Child Nutrition Program Final Evaluation page 31 31 5.7 Measuring Outputs and Outcome Achievement Annex A presents the outcome and output level data that the baseline data gathered. Some indicators are made up of several measurements. For example, the baseline study assessed the quality of literacy instruction through questions asked in the student survey, the teacher survey, and the director survey. Future evaluations will have to assess the relative importance of changes in response to these questions to judge whether the program has met its targets. Rigorous qualitative research will contribute greatly to making those judgements. Interviews, document reviews, and other qualitative research tools allow researchers to estimate the answer to “why” questions to support the “what” information most surveys provide. Sukaabe Janngo should make sure that future evaluations have ask consistent qualitative questions, and pursue avenues discovered during previous evaluations and research. 5.8 Teacher Training Effectiveness The effectiveness of teacher trainings is difficult to define and measure. The means through which effective training is translated into results is influenced by many variables, as is the means by which children improve their reading skills. In a future evaluation, Sukaabe Janngo should consider asking the evaluator to conduct a contribution analysis. Contribution analysis attempts to assess the causes of a result and attempts to attribute the degree to which each cause contributed to the result. It attempts to create understanding of why a result occurred and how much each factor contributed. Contribution analysis would indicate the effectiveness of teacher trainings and the degree to which those trainings are translated into improved student achievement. 5.9 School Graduation While schools are unlikely to reach 100 percent of their school feeding needs through government and community contributions, there are a number of measures that Sukaabe Janngo could take to sustain school feeding at graduated schools. Adjust the school feeding schedule – Some schools maintained school feeding by limiting the feeding schedule. For example, since students leave school around noon on Mondays, Tuesdays, and Wednesdays, some schools only provide lunch on Tuesdays and Thursdays, when students attend class into the afternoon. This schedule stretches school resources and provides energy for students during the longer study periods. Keep procurement as simple as possible – So as not to burden SMC’s, Sukaabe Janngo should make purchasing food as easy as possible, which in this case may mean relying on local markets and producers rather than the LRP. Work with parents on food budgets – A campaign to raise awareness among parents on how to allocate money or food that is used to feed students at home could be donated to schools for the school feeding program and lead to improved education for their children. This approach might change parent’s perspective that they are too poor to contribute, or that contributing food or money to the school means less food or money at home. Income generating activities – Sukaabe Janngo could explore school-based income generating activities to provide additional resources to the school feeding program. If Sukaabe Janngo takes this approach, it 32 should keep in mind that income generating activities mean increased workloads and should provide incentives accordingly. Event-based fundraising – A lower risk approach could be to organize school-led fundraising events or initiatives. Sukaabe Janngo should shape the specifics of these events, and they might include trying to get local farmers to commit a small share of their harvest to school canteens and/or events with local traders/merchants. 33 Annex A: Baseline Values for McGovern-Dole Standard Indicators Strategic Objective 1: Improved literacy of school-aged children Reading levels by grade (project vs comparison) School level Reading level Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 CE2 (4th) 1 Project mean 8.91% Comparison mean 6.53% 2 Project mean 16.19% Comparison mean 19.60% 3 Project mean 13.77% Comparison mean 12.06% 4 Project mean 25.91% Comparison mean 24.62% 5 Project mean 35.22% Comparison mean 37.19% CM1 (5th) 1 Project mean 3.45% Comparison mean 4.35% 2 Project mean 5.42% Comparison mean 4.35% 3 Project mean 13.30% Comparison mean 14.67% 4 Project mean 20.20% Comparison mean 22.83% 5 Project mean 57.64% Comparison mean 53.80% 34 CM2 (6th) 1 Project mean 0.48% Comparison mean 2.37% 2 Project mean 1.45% Comparison mean 1.18% 3 Project mean 5.31% Comparison mean 9.47% 4 Project mean 18.36% Comparison mean 19.53% 5 Project mean 74.40% Comparison mean 67.46% Reading levels by grade and sex (project vs comparison) School level Reading level Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 CE2 (4th) 1 Project mean (female) 9.94% Comparison mean (female) 6.30% Project mean (male) 6.98% Comparison mean (male) 7.04% 2 Project mean (female) 17.39% Comparison mean (female) 20.47% Project mean (male) 13.95% Comparison mean (male) 18.31% 3 Project mean (female) 14.29% Comparison mean (female) 8.66% Project mean (male) 12.79% Comparison mean (male) 18.31% 35 4 Project mean (female) 21.74% Comparison mean (female) 23.62% Project mean (male) 33.72% Comparison mean (male) 26.76% 5 Project mean (female) 36.65% Comparison mean (female) 40.94% Project mean (male) 32.56% Comparison mean (male) 29.58% CM1 (5th ) 1 Project mean (female) 3.82% Comparison mean (female) 5.22% Project mean (male) 2.78% Comparison mean (male) 2.90% 2 Project mean (female) 5.34% Comparison mean (female) 4.35% Project mean (male) 5.56% Comparison mean (male) 4.35% 3 Project mean (female) 14.50% Comparison mean (female) 14.78% Project mean (male) 11.11% Comparison mean (male) 14.49% 4 Project mean (female) 21.37% Comparison mean (female) 19.13% Project mean (male) 18.06% Comparison mean (male) 28.99% 36 5 Project mean (female) 54.96% Comparison mean (female) 56.52% Project mean (male) 62.50% Comparison mean (male) 49.28% CM2 (6th ) 1 Project mean (female) -- Comparison mean (female) 0.93% Project mean (male) 1.39% Comparison mean (male) 4.92% 2 Project mean (female) -- Comparison mean (female) 1.85% Project mean (male) 4.17% Comparison mean (male) -- 3 Project mean (female) 7.41% Comparison mean (female) 10.19% Project mean (male) 1.39% Comparison mean (male) 8.20% 4 Project mean (female) 14.07% Comparison mean (female) 21.30% Project mean (male) 26.39% Comparison mean (male) 16.39% 5 Project mean (female) 78.52% Comparison mean (female) 65.74% Project mean (male) 66.67% Comparison mean (male) 70.49% 37 Indicator 1.1: Improved quality of literacy instruction Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 % of Teachers w/ gov. provided training in literacy in the past year Project mean 46% Comparison mean 38% (Teacher) Usefulness of training Very useful Project mean 83.33% Comparison mean 88.89% Somewhat useful Project mean 16.67% Comparison mean 11.11% (Teacher) How well are students learning reading? 1-5; 1 = Very well, 5 = Very poor Project mean 2.44 Comparison mean 2.49 (Teacher) To what extent does literacy instruction at this school need to improve to best teach students? 1-5; 1 = A very small extent, 5 = A large extent Project mean 3.75 Comparison mean 3.80 (Director) How would you rate the quality of literacy instruction for 4th, 5th and 6th graders at this school in the current school year? 1-5; 1=Superior, 2=Very good, 3=Good, 4=Fair, 5=Poor Project mean 3.21 Comparison mean 3.49 (Director) To what extent does literacy instruction at this school need to improve to best teach students? 1-5; 1=A very small extent, 5=A large extent Project mean 3.46 Comparison mean 3.51 (Student) How much do you like your reading class in school? A lot Project mean 53.90% Comparison mean 55.30% Pretty much Project mean 29.50% Comparison mean 25.70% It’s ok Project mean 13.60% Comparison mean 26.50% Don’t like it Project mean 2.22% 38 Comparison mean 2.11% Hate it Project mean 0.79% Comparison mean 0.42% Indicator 1.2: Improved attentiveness Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 (Teacher) Approximately what percent of your students pay attention in class this semester? Less than half Project mean 6.41% Comparison mean 8.54% Half Project mean 17.50% Comparison mean 20.70% More than half Project mean 59.80% Comparison mean 61.00% All Project mean 16.20% Comparison mean 10.00% (Student) Do you usually find it very easy, easy, hard, or very hard to pay attention in reading class? Very easy Project mean 33.30% Comparison mean 30.40% Easy Project mean 54.50% Comparison mean 54.20% Hard Project mean 10.50% Comparison mean 13.30% Very hard Project mean 1.74% Comparison mean 2.11% 39 Indicator 2/MGD 1.3: Average student attendance rate in USDA-supported classrooms/schools Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Students present/students enrolled CE2 (4th) Project mean 96% Comparison mean 91% CM1 (5th) Project mean 96% Comparison mean 94% CM2 (6th) Project mean 98% Comparison mean 92% Indicator 9: Number of students enrolled in schools receiving USDA assistance Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Target for 2022 Number of students enrolled in school receiving USDA assistance (total) preschool • Male 2,737 • Female 3,237 primary • Male 16,567 • Female 24,788 Indicator 1.3.4: Improved student enrollment Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Average number of students enrolled in each grade per school CE2 (4th) Project mean 30.6 Comparison mean 27 CM1 (5th) Project mean 25.4 Comparison mean 25.5 CM2 (6th) Project mean 20.1 Comparison mean 19.8 40 MGD Indicator 27: Number of schools using an improved water source Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Target for 2022 Number of schools using an improved water source Total N/A 3329 MGD Indicator 28: Number of schools with improved sanitary facilities Measure Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Target for 2022 Number of schools with improved sanitary facilities Total N/A 43 0 0 0 20 Strategic Objective 2: Increased use of health, nutrition and dietary practices Indicator Scale Disaggregation Baseline Target for 2019 Target for 2020 Target for 2021 Average number of latrines per school Integer Project mean 2.83 Comparison mean 2.43 Percentage marked for girls and boys Y/N Project mean 79% Comparison mean 21% Percentage connected to water Y/N Project mean 34% Comparison mean 27% Percentage connected to water and water connection functions Y/N Project mean 64% Comparison mean 48% 29 MGD Indicators 27 and 28 reflect totals as of the end of the previous phase of the project. 41 How clean is the latrine? Very clean Project mean 18.50% Comparison mean 10.60% Mostly clean Project mean 30.80% Comparison mean 22.30% Somewhat clean Project mean 32.20% Comparison mean 37.20% Not clean Project mean 18.50% Comparison mean 29.80% Is the latrine well maintained in terms of structure? Very well maintained Project mean 29.50% Comparison mean 18.10% Well maintained Project mean 34.20% Comparison mean 35.10% Somewhat maintained Project mean 24% Comparison mean 31.90% Not maintained Project mean 12.30% Comparison mean 14.90% (Student) How comfortable are you using the school latrines? I am very comfortable using them Comparison mean 35.10% Project mean (total) 49.10% Project mean (female) 46.28% Project mean (male) 54.40% I don’t mind using them if I have to Comparison mean 26.28% Project mean (total) 28.99% 42 Project mean (female) 31.00% Project mean (male) 25.20% I don’t like to use the latrines Comparison mean 12.66% Project mean (total) 9.99% Project mean (female) 10.40% Project mean (male) 9.20% I try to avoid using the school latrines Comparison mean 11.22% Project mean (total) 9.15% Project mean (female) 9.77% Project mean (male) 8.00% There is no latrine at this school Comparison mean 14.74% Project mean (total) 2.77% Project mean (female) 2.55% Project mean (male) 3.20% Why aren’t you comfortable using the school latrines? They are not clean Comparison mean 63.08% Project mean (total) 61.72% Project mean (female) 63.64% 43 Project mean (male) 57.50% I don’t feel like I have privacy Comparison mean 13.08% Project mean (total) 10.94% Project mean (female) 13.64% Project mean (male) 5.00% There is not a place to wash my hands Comparison mean 2.31% Project mean (total) 4.69% Project mean (female) 2.27% Project mean (male) 10.00% They are not safe Comparison mean 3.08% Project mean (total) 2.34% Project mean (female) 2.27% Project mean (male) 2.50% Other Other 18.46% Project mean (total) 20.31% Project mean (female) 18.18% Project mean (male) 25.00% 44 Is the latrine equipped with a sink for washing hands? Y/N Project mean 35% Comparison mean 15% Is the sink area clean? 1=Very clean 2=Mostly clean 3=Somewhat clean 4=Not clean Project mean 2 Comparison mean 2.33 Is the sink area clean? Very clean Project mean 31.60% Comparison mean 25% Mostly clean Project mean 42.10% Comparison mean 33.30% Somewhat clean Project mean 21.10% Comparison mean 25% Not clean Project mean 5.20% Comparison mean 16.70% Is there soap? Y/N Project mean 27% Comparison mean 11% Is there a clean towel or other material for drying? Y/N Project mean 1% Comparison mean 11% (Student) Do you wash your hands at school? Yes Project mean 89.10% Comparison mean 66.00% No Project mean 10.90% Comparison mean 34.00% (Student) Why don’t you wash hands at school? (Mark all that apply.) There is no place to wash hands Project mean 41.00% Comparison mean 50.00% There is no water Project mean 26.50% 45 Comparison mean 28.60% There is no soap Project mean 30.10% Comparison mean 20.90% The wash basin isn’t clean Project mean 2.41% Comparison mean 0.49% Does the school have a school feeding program? Yes Project mean 94.34% Comparison mean 7.76% No Project mean 5.66% Comparison mean 92.24% Does the school have a school canteen? Yes Project mean 97.42% Comparison mean 10.66% No Project mean 1.94% Comparison mean 85.25% N/A Project mean 0.65% Comparison mean 4.10% Does this school have a food storage area? Y/N Project mean 99% Comparison mean 69% Is the storage area clean? Y/N Project mean 80% Comparison mean 78% Is the roof covered? Y/N Project mean 98% Comparison mean 100% Is the food storage area secure? Y/N Project mean 99% Comparison mean 100% Are there signs of damage from rodents or other pests? Y/N Project mean 14% Comparison mean 22% Food present in food storage area? Y/N Project mean 68% 46 Comparison mean 60% Does this school have a cooking area? Y/N Project mean 97% Comparison mean 100% Is there a structure for the cooking area? Y/N Project mean 80% Comparison mean 62% Is the cooking area generally clean? Y/N Project mean 82% Comparison mean 100% Is there soap or detergent? Y/N Project mean 38% Comparison mean 62% Are there adequate utensils for cooking and eating? Y/N Project mean 99% Comparison mean 92% Is there an energy efficient stove? Y/N Project mean 95% Comparison mean 54% Is there a way to wash your hands before eating? Y/N Project mean 98% Comparison mean 77% How many cooks were trained and practicing safe food prep? Y/N Project mean 41% Comparison mean 52% Annex B: Data Collection Instruments Student Survey Teacher Survey Director Survey Checklist Interview Guide Annex C: Baseline Study Scope of Work Scope of Work- McGovern-Dole Vendor shall provide to Counterpart professional services as follows: Background Counterpart is seeking a qualified consultant/firm to implement the baseline study for the newly funded McGovern-Dole program called Sukaabe Janngo “Children of Tomorrow”. This four-year program is funded through a cooperative agreement with the United States Department of Agriculture (USDA) and will start in January 2019. The program aims to improve student performance in part through school feeding programs. The baseline study will be performed in March and April of 2019 and will feature surveys as well as recommendations on indicators for the evaluations that follow. USDA requests that baseline information be collected by the project within 6 (six) months of project award date, in order to set accurate and realistic targets and to enable the project to monitor progress and performance throughout the project. Counterpart expects this baseline study to be fully integrated in the body of information that the project will use for performance monitoring and evaluation, and for learning. These services will be implemented from on/about between March and June 2019. 2. Goal of the Evaluation The baseline study, to be carried out in first quarter of 2019, will produce quantitative data used to compare progress on the midterm, endline and impact evaluations. It will also produce qualitative findings that will themselves be used for comparison but also to help guide program strategy and implementation. Counterpart expects the baseline study to provide information on contextual factors that may slow or accelerate the changes that the program expects to make. This information should enable project staff to validate the design of the project and, if needed, recalibrate its interventions. The contextual factors should focus on, among others, governance at the school level, community level, and national level, perceived nutritional and health needs of the beneficiaries, as well as school management committees’ needs in capacity building. The potential for “graduation” of the school feeding program should also be a main focus of the baseline study. The evaluation should also pay close attention to gender and social inclusion dynamics to ensure that considerations related to gender and social inclusion are factored into future programming. The results of this baseline evaluation should be curated for feedback to project stakeholder groups including beneficiaries. 3. Dissemination The executive summary of the baseline reports will be shared with stakeholders including the Ministry of National Education (Minister, Secretary General, DEE, DC AS), Ministry of Agriculture, Governor of Saint Louis Region, Academic Inspector and all the inspectors of the Saint Louis Region, School Management Committees, and USDA local mission. Per the USDA Monitoring and Evaluation policy, the baseline evaluation report will be made publicly available. The publicly available version of the report will be free from personal identifying information. Vendor Agreement IMC Worldwide FY2019 8 Counterpart, along with partners Save the Children and Associates for Research and Education for Development, will implement the McGovern-Dole project for four years in the Saint Louis region of Senegal (see map). The program will work with the same 270 schools Counterpart worked with from 2015 to 2018, continuing to work towards achieving the two McGovern-Dole strategic objectives of 1) improved literacy of school-age children (Sukaabe Janngo SOI), and 2) increased use of health, nutrition, and dietary practices (Sukaabe Janngo S02). The previous McGovern-Dole project, which concluded activities in September 2018, was a three-year project that served over fifteen million meals to over 45,000 students across the Saint- Louis region. Despite delayed construction activities, the project’s final evaluation concluded that the McGovern￾Dole program had improved student literacy, student enrollment and attendance, knowledge of safe storage practices, and knowledge among students of health and hygiene practices. In addition to continuing to work towards achieving these two objectives, McGovern-Dole will foster the capacity of regional and community-level stakeholders in implementing their own graduation and sustainability plans. Results and Activities Improved literacy of school-aged children (Strategic Objective 1) is to be achieved through improved quality of literacy instruction, greater student in class concentration through school feeding and increased attendance. Improved literacy will be achieved through improved school supplies and materials, improved literacy instruction materials, and increased skill and knowledge of teachers and school administrators. Improved attentiveness will be achieved through increased access to food through school feeding. Improved quality of literacy education is also expected to increase student attentiveness. Improved student attendance and enrollment will result from increased access to food, reduced health absences and improved school infrastructure. S02 will contribute to the broader goal of improving literacy by reducing health related absences, increasing attentiveness and providing incentives for parents to enroll their children in school and encourage attendance. This causal pathway is illustrated in the results framework below. Results Framework for Improve Literacy of School-Age Children Vendor Agreement IMC Worldwide FY2019 9 Policy results which support the implementation and sustainability of the strategic objective are increased capacity of government institutions, improved policy and regulatory framework, increased government support, and increased engagement of local organizations and community groups. The following are the primary activities that will be undertaken by Counterpart to achieve SO 1: • Provide school meals: In preschools and primary schools, breakfast will be provided five days a week for 170 school days. In primary schools, lunch will be provided two days a week. • Take home rations: These will be provided to boys in grades five and six who have stellar (90%) attendance rates. • Training in commodity management: Counterpart will train master trainers who will then train school management committees (SMCs) on sustainable commodity management, using revised national guidelines. Counterpart will also provide refresher trainings to SMCs. • Work with farms to supply food to schools: Counterpart will continue to support the 20 school farms established previously and work to identify food donations from other farms. • Training for cooks and SMCs on food preparation and storage: Counterpart will train school cooks, SMCs and master trainers on safe and nutritious food preparation and management. • Teacher training: Counterpart will train government master trainers who will train teachers on improved literacy approaches. • School administrator training: Counterpart will also train school directors on literacy, data collection and other topics. Vendor Agreement IMC Worldwide FY2019 10 • Literacy extra-curricular activities: Counterpart will organize reading clubs at all primary schools. • Building and rehabilitating classrooms: Counterpart will build or repair classrooms for at least 30 schools. • Providing furniture and equipment: Counterpart will provide furniture and equipment for at least 30 schools. • Training for government officials: Government officials will be trained on management and data collection and how to sustain the program. Increased use of health, nutrition and dietary practices (Strategic Objective 2) will be achieved through improved knowledge of health and hygiene practices, increased knowledge of safe food preparation, increased knowledge of nutrition, increased access to water and sanitation, increased access to preventative health interventions and increased access to food preparation and storage tools and equipment. The effectiveness and sustainability of these results are supported by the same policy results as for SO 1. Results Framework for Increased Health, Nutrition and Dietary Practices In addition to the related activities described under SO 1 above, the following activities will support the achievement of SO 2. • Provide latrines and water stations: Counterpart will build or rehabilitate up to 20 latrines and water station systems and train students on their proper use. Mobile hand washing stations will also be provided. • Maintain or replace canteen equipment: Canteen equipment including energy efficient stoves and pots was provided in the previous program. Counterpart will replace equipment if needed. • Train teachers on good health and nutrition practices: Government master trainers will be trained by Counterpart on these practices and provide training to teachers annually. • Distribute deworming medicine, vitamins and minerals: Counterpart will support the GoS’ Increased Use of Health, Nutrition and Dietary Practices (Sukaabe Janngo SO 2) i I i k i k i k t t Improved Knowledge of Health and Hygiene Increased Knowledge of Safe Food Preparation (Sukaabe Janngo 2.2) Increased Knowledge of Nutrition (Sukaabe Janngo 1ncreased Access to Clean Water and Sanitation Increased Access to Preventative Health Interventions (Sukaabe Janngo 2.5) Increased Access to Requisite Food Prep and Storage Tools Practices (Sukaabe Janngo 2.1) 2.3) (Sukaabe Janngo 2.4) and Equipment (MGD2.6 Increased Capacity of Government Institutions (MGD 2.4.1) Improved Policy and Regulatory Framework (MGD 2.4.2) Increased Government Support (MGD 2.4.3) Increased Engagement of Local Organizations and Community Groups (MGD 2.4.4) existing initiatives to ensure children, cooks and teachers are dewormed. Beneficiaries McGovern-Dole will work in 270 schools, including 204 primary schools and 66 pre-schools. In total, 66,000 students are expected be enrolled in McGovern-Dole -supported schools throughout the life of the project with nearly all receiving meals and deworming medicine through McGovern-Dole. The project will also improve infrastructure at 70 schools, including repairing or replacing old class rooms and installing new latrines and hand washing facilities. 5. Scone of Work To successfully complete the baseline study, the awarded firm must complete the following tasks. • Operationalize research questions and indicators. Of special interest is Sukaabe Janngo 1.1, 1.2 and 1.3.2, the effectiveness of teacher trainings, as well as the research questions on school farm sustainability that will be assessed in terms of government support and capacity. • Conduct a survey to collect high quality data from treatment (McGovern-Dole) and comparison schools to provide baseline values for the research questions and for PMP indicators. Data quality is to be ensured through data logic consistency checks and validation rules, as well as ongoing monitoring of data collected during the survey. Key aspects of fieldwork include the following tasks: o Identifying comparison schools o Operationalizing research questions for measurement • Produce a baseline report containing a) descriptive statistics of baseline values including, where appropriate, disaggregated by gender, department, grade and pre vs primary school. In addition to the descriptive statistics, the report will include recommendations on methodology for implementing subsequent evaluations. • The firm will also provide Counterpart with all data sets in both original and final (cleaned) version used for analysis. The firm will also provide all tools used as well as a survey manual that will inform survey implementation for subsequent evaluations. 5. Research Questions The following research question will be answered cumulatively by the midterm, final, and impact evaluation. The baseline study therefore must assess the baseline status of these questions. The awardee will be responsible for operationalizing these research questions so that they can be consistently and accurately measured or assessed in subsequent evaluations. The operationalization of the research questions will be informed by review of similar evaluations and research, including other McGovern-Dole evaluations. McGovern-Dole project level performance • Have program outputs and outcome targets been achieved? (see “Sukaabe Janngo Outcome Indicators that Require Baseline Values” table below) School feeding and nutrition • What is the effect of school feeding on attendance, enrollment and attention? • How effective are take home rations at increasing attendance among boys in 5th and 6th grades? • What are the shares of the total recommended per student feeding schedule are McGovern-Dole schools from non-USG sources (disaggregated by source)? Vendor Agreement IMC Worldwide FY2019 12 Education and Literacy • How effective are reading-oriented extra-curricular activities in improving literacy? • How effective are teacher trainings? Health • What is the effect of deworming medicine on student attendance? • What is the effect of latrine quality on student attendance, especially for girls? • How closely are students following handwashing recommendations? Are they practicing at home too? Methodological • What is the best way to measure the three undefined McGovern-Dole outcome indicators (Sukaabe Janngo 1.1, 1.2 and 1.3.2)? • How reliable is school and government-collected attendance and enrollment data? How can the accuracy be improved? School Feeding Sustainability • What is government capacity to manage school feeding at regional and national levels? • What commitment has the government shown on school feeding? (e.g. do they have a school feeding policy, clearly defined roles for managing school feeding, plans to expand school feeding budget) Sukaabe Janngo Outcome Indicators that Require Baseline Values Result # Title in Result Framework Indicator Sukaabe Janngo SOI Improved Literacy of School Aged Children Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Sukaabe Janngo 1.1 Improved Quality of Literacy Instruction TBD Sukaabe Janngo 1.2 Improved Attentiveness TBD Sukaabe Janngo 1.3 Improved Student Attendance Average student attendance rate in USDA supported classrooms/schools Sukaabe Janngo 1.3.2 Reduced Health-Related Absences TBD Sukaabe Janngo 1.3.4 Improved Student Enrollment Number of students enrolled in school receiving USDA assistance Vendor Agreement IMC Worldwide FY2019 Design overview The survey conducted for the baseline study will include treatment and comparison schools. Treatment schools will be those in which McGovern-Dole is working. Comparison schools will be schools in Saint Louis where McGovern-Dole is not working but which are similar to McGovern-Dole schools. The awardee will select comparison schools using the same criteria that were used for choosing Sukaabe Janngo schools so that they are as similar as possible to McGovern-Dole schools. Counterpart will provide these criteria to the awardee. This selection process will be documented in baseline study. The awardee should also note the presence of USAID-funded Lecture Pour Tous (LPT) in primary schools in Saint Louis starting October 2018. LPT will work to improve reading for students in grades 1 to 3, whereas, McGovern-Dole will focus on grades 4-6. The project (LRP) aims to work in all schools in Saint Louis and will end in 2021. The awardee must note whether or not each sampled school has received support from LRP by the time of the survey. Within sampled schools, interviews will be conducted with students, directors and teachers. Interviews with students and teachers will only be carried out at primary schools whereas interviews with directors will be at both primary and pre-schools. There are 204 McGovern-Dole primary schools and 66 McGovern-Dole pre￾schools. In addition, interviews will be conducted with government to assess capacity and commitment for school farm sustainability. A list of potential respondents for these interviews will be provided to the awardee. Document review Documents to be reviewed as part of the baseline study include the evaluations of the previous phase of McGovern-Dole, the monitoring and evaluation plan as well as baseline studies and evaluations for other McGovern-Dole or similar projects worldwide. Pilot survey The awardee is required to conduct a pilot survey to test all questionnaires. The pilot is especially important because question phrasing will be used for the subsequent evaluations and must elicit accurate responses. Sufficient time must be for updating all instruments based on pilot findings. Survey Sampling for the student surveys will use a two-stage cluster design with the first stage being school selection and the second students election. The total sample size of students for treatment and comparison schools combined is expected to be around 1,400. This total sample size will be divided approximately 60:40 between treatment and comparison groups, as is consistent with best practices. The exact sample sizes will be left to the firm conducting the baseline study. However, all sampling will comply with the 95 percent confidence level, 5 percent margin of error standard. Sample designs for teachers and directors will also meet this 95 percent, 5 percent standard. Reading Tools The evaluations will use Early Grade Reading Assessment (EGRA) methodologies. EGRA reading assessment tool that has been used with McGovern-Dole evaluations in other countries. EGRA is being used in the current USAID-funded Lecture Pour Tous (LPT) project, implemented by Chemonics International. Vendor Agreement IMC Worldwide FY2019 14 MGDS02 Increased Use of Health, Number of individuals who demonstrate use of new safe Nutrition and Dietary food preparation and storage practices as a result of Practices USDA assistance 6. Approach and Methodology. Data analysis Survey data analysis will be carried out using an appropriate statistical package such as R or Stata. Analysis will include means and means comparisons by gender, grade and department, along with tests of statistical significance. For ASER reading scores, this includes mean reading scores by grade as well as the share of students reading at or above their grade levels. 7. Assignment Timeline. Place of Performance, and Other Conditions. Counterpart anticipates the preparatory work and review of relevant reports and documents to be completed and the Baseline Evaluation Workplan submitted by April 1, 2019. The evaluator will conduct the field data collection between April 5 and May 5,2019, and completed before the start of Ramadan, May 6. Once the draft is submitted, Counterpart will have seven business days to review the report, raise concerns, provide comments, and send it back to the evaluator. The evaluator will then address Counterpart’s comments and concerns and submit a revised report to Counterpart for Counterpart and the donor’s review. The final version of the report is due no later than June 25,2019. Should Counterpart still not be satisfied with the quality of the final report, then both Counterpart and the evaluator will negotiate a no-cost extension to ensure both parties are satisfied with the final result. Throughout the baseline evaluation, there must be open communication between the evaluator and Counterpart through phone calls, emails, text messages, Skype calls, or face-to-face meetings for effective coordination between both parties and to ensure that potential issues are resolved in a timely manner. Any anticipated changes to the plan during the evaluation must be submitted in writing and be approved by Counterpart. 8. Staff Structure. Vendor Agreement IMC Worldwide FY2019 15 Item Deadline (subject to change) Submission of Baseline Evaluation Workplan: (Literature review, evaluation design including data collection and analysis methodology; draft sampling strategy, and intended respondents /key informants; quality assurance plan; Draft Evaluation Schedule; and Draft Data Collection Tools) April 1 Finalize data collection tools and evaluation schedule No later than April 5 Field work: travel and training of data collectors On or about April 5 to April 8 Field Work: testing of the data collection tools and calibrate, data collection and analyses On or about April 5 to May 5*** Debriefing on preliminary findings at CPI Field Office and/or CPI HO On or about May 20 Submission of first draft report On or about May 3 Review and comment of first draft (and subsequent drafts as necessary) by Counterpart International, followed by review and comments by USDA May 20-June 20 Submission of final report, datasets and tools On or about June 25, 2019 *** Field work must take place on or around the suggested dates. Tl evaluation team must complete field work before May 6, 2019, which marks the beginning of the month of Ramadan. The evaluation team will comprise one international program evaluator (Team Leader), and three or more local or international consultants or members of a consulting firm selected for their technical expertise. To the extent possible, the evaluation team will be gender-balanced. 9. Deliverables. The awardee will be responsible for the six deliverables below. 1. Inception report: Before starting fieldwork, the awardee will submit an inception report for approval by Counterpart. The inception report will contain a plan for fieldwork with a description and schedule for all activities. It will also include all tools to be used for data collection, including questionnaires. 2. In country debrief: Before the fieldwork is completed, the awardee will complete a debrief with the MGD team in Senegal. The debrief presentation will cover what changes were made to questions based on the pilot, if there were any survey challenges and preliminary findings. 3. Draft evaluation report: The report shall contain the following sections: 1) Introduction, 2) Methodology, 3) Results and 4) Recommendations. The Results section will include both summary baseline values as well as description of qualitative results. The Recommendations will be both a) programmatic based on relevant results and b) methodological for the subsequent evaluations. The report should detail and justify how key indicators and research questions were operationalized and measured. 4. Presentation to field and HO staff: The awardee will prepare a PowerPoint presentation of the draft evaluation report and present it to Counterpart HQ and MGD field staff. 5. Datasets, tools and survey manual: All datasets and tools will be shared with Counterpart. Additionally, a survey manual explaining how the survey/s were carried out will be provided. 6. Final report: The final report will incorporate written feedback and comments from the presentation. It should not exceed 40 pages, exclusive of annexes. The following table outlines requirements for the Final Baseline Report: Report Length Maximum of 40 pages, excluding the Table of Contents, Acronym List, and Annexes and should be written in Times New Roman font size 12. 16 Illustrative Report Outline Acknowledgement Table of Contents Table of Exhibits Acronym List Executive Summary (in English and French) Chapter 1. Evaluation Purpose and Research Questions Chapter 2. Project Background Chapter 3. Evaluation Method 3.1 Methodologies 3.2 Sampling Framework 3.3 Data Sources and Data Collection Methods 3.4 Field Work 3.5 Analysis Plan 3.6 Strengths and Limitations Chapter 4. Findings Chapter 5. Conclusion and Recommendations 1.1 Summary of Key Findings 1.2 Lessons Learned 5.2 Recommendations References Annexes Executive Summary Include an Executive Summary that provides a brief overview of the evaluation purpose, project background, evaluation questions, methods, findings, and conclusions. Questions Address all evaluation questions in the SOW. Methods • Explain evaluation methodology in detail. • Disclose evaluation limitations, especially those associated with the evaluation methodology (e.g. selection bias, recall bias, etc.). NOTE: A summary of methodology can be included in the body of the report, with the full description provided as an annex. Findings • Tables with baseline results, disaggregated by gender, department, grade and/or pre-school, as appropriate • Brief description of each table, including any context or explanation needed to aid the reader in interpreting and understanding. • • Detailed description of findings for methodological research questions and operationalizing of key indicators as described in the scope of work Recommendations • Support recommendations with specific findings. • Provide recommendations that are action-oriented, practical, and specific. 10. Roles and Responsibilities The evaluator will be responsible for all the deliverables listed in the TOR and will be free to draw its own conclusions free from political or organizational pressure. The evaluator will coordinate with Counterpart HQ’s Senior Program Manager, Program Specialist, Director of Program Quality and Learning and Counterpart Senegal Chief of Party with regards to the overall scope, direction, and completion of this assignment. USDA will provide guidance as needed along with feedback on the initial draft, to be included in the final report. USDA will also be consulted as a key informant prior to evaluation fieldwork. Counterpart HQ staff including Senior Program Manager, Program Specialist, Director of Program Quality and Learning, Monitoring and Evaluation Specialist and the Chief of Party will provide all relevant reports, data and related information necessary to prepare the evaluator for the assignment. And as needed, Counterpart HQ Senior Program Officer will coordinate with the Chief of Party to facilitate field logistics, including potential meetings with all relevant stakeholders during the field visit in Senegal. The HQ point of contact will be the Program Specialist The in-country point of contact will be the Chief of Party. Counterpart HQ Senior Program Manager is responsible for approving evaluation deliverables. Vendor Agreement IMC Worldwide FY2019 Annexes Include the following as annexes, at a minimum: • Terms of Reference • All evaluation tools (questionnaires, checklists, discussion guides, surveys, etc.). • A list of sources of information (key informants, documents reviewed, other data sources) Only if applicable, include as an annex Statement(s) of Differences regarding any significant unresolved differences of opinion on the part of funders, implementers, and/or members of the evaluation. Quality Comparison Assess reports for quality by including an in-house peer technical review with comments provided to the evaluator. Transparency An English report should be submitted in an electronic version to Counterpart for approval. All deliverables must be approved by Counterpart. 18