CRS FOOD FOR EDUCATION MID-TERM EVALUATION OF FFE Intervening in a demanding context MAURICE GARNIER1 , PIERRE GBENOU2 and SERGE MARCEL LOUKPE3 1 Emeritus Professor of Sociology. Indiana University, Garnier@indiana.edu 2 Retired Primary school inspector Benin, gbenoupierre7174@gmail.com 3 Statistician, losema1@yahoo.fr April 2017 Midterm Evaluation for FY 2014 McGovern-Dole Food for Education Project in Benin - Intervening in a demanding context Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-680-2014/046-00 Funding Year: Fiscal Year 2014 Project Duration: 2014-2020 Implemented by: Catholic Relief Services Evaluation Authored by: Maurice Garnier Pierre Gbenou Serge Marcel Loukpe DISCLAIMER: This evaluation was conducted by an independent third party firm. The views expressed in this report do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this report can be made available upon request by emailing FAS.monitoring.evaluation@usda.gov. ii Mid-term Evaluation FFE | April 2017 Table of Contents List of Tables ............................................................................................................................. iv List of Abbreviations ................................................................................................................... v Executive Summary ................................................................................................................... vi Introduction ............................................................................................................................... 1 FFE CONCEPTUAL APPROACH .............................................................................................................. 2 1. Quality of reading instruction ............................................................................................................................ 2 2. Nutrition and health .......................................................................................................................................... 2 3. Learning resources ............................................................................................................................................. 3 4. The educational system ..................................................................................................................................... 3 5. The community .................................................................................................................................................. 3 6. Primary Education in Benin ............................................................................................................................... 3 Evaluation Methodology ............................................................................................................ 5 INTRODUCTION .................................................................................................................................. 5 DESIGN OF THE INSTRUMENTS ............................................................................................................ 6 SAMPLING .......................................................................................................................................... 6 FIELD WORK ....................................................................................................................................... 7 INTERVIEWS AND FOCUS GROUPS....................................................................................................... 8 METHODOLOGICAL APPENDIX ............................................................................................................ 8 Results ..................................................................................................................................... 11 STUDENTS’ READING LEVELS ............................................................................................................. 11 1. EGRA and its Characteristics ............................................................................................................................ 11 2. FFE’s usage of EGRA ......................................................................................................................................... 14 3. EGRA Results .................................................................................................................................................... 14 4. Conclusion ....................................................................................................................................................... 18 QUALITY OF READING INSTRUCTION ................................................................................................. 20 1. Time spent on reading instruction ................................................................................................................... 20 2. Attentiveness and Classroom Activities ........................................................................................................... 21 LEARNING CONDITIONS .................................................................................................................... 26 1. Classroom Size and Availability of Supplies ..................................................................................................... 26 2. Teachers’ Academic and Professional degrees ................................................................................................ 28 3. Teachers’ attendance ...................................................................................................................................... 33 4. The teacher training program .......................................................................................................................... 34 5. Training situation and the outcomes ............................................................................................................... 35 6. Relevance of the teacher training program ..................................................................................................... 36 7. Efficiency of the training program ................................................................................................................... 37 8. Conclusion concerning the Improvement of learning instruction ................................................................... 37 iii Mid-term Evaluation FFE | April 2017 THE SCHOOL ENVIRONMENT ............................................................................................................. 42 1. Hunger ............................................................................................................................................................. 42 2. Illness ............................................................................................................................................................... 43 3. Washing Stations ............................................................................................................................................. 44 KNOWLEDGE ABOUT EDUCATION ..................................................................................................... 47 IMPLICATION OF RELEVANT MINISTRIES AND OFFICES ....................................................................... 51 EVALUATION QUESTIONS ................................................................................................................. 53 1. Relevance:........................................................................................................................................................ 53 2. Effectiveness: ................................................................................................................................................... 55 3. Efficiency: ......................................................................................................................................................... 57 4. Sustainability: .................................................................................................................................................. 59 5. Impact: ............................................................................................................................................................. 60 Recommendations ................................................................................................................... 63 Acknowledgements .................................................................................................................. 66 Appendices .............................................................................................................................. 68 SEVERAL SUGGESTIONS CONCERNING READING ................................................................................ 68 UNDERSTANDING THE PURPOSE OF EDUCATION ............................................................................... 71 iv Mid-term Evaluation FFE | April 2017 List of Tables Table 1. Dates and sample sizes of FFE EGRA surveys 14 Table 2. Percentage of First and Second Grade students by level of performance 15 Table 3. Percentage of the maximum score in November 2015 and June 2016 2nd grade 16 Table 4. Percentage of 2nd graders who received 0 on EGRA sections in June 2016 17 Table 5. Teachers’ and students’ activities during the lesson 22 Table 6. Average number of students in the first two grades 26 Table 7. Classroom condition for the first two grades (%) [Extrapolated results] 27 Table 8. Average % of Students who do not have all the required supplies [Extrapolated results] 27 Table 9. Teachers’ perception of their didactic material (%) [Extrapolated results] 28 Table 10. Teachers’ status [Extrapolated results] 29 Table 11. Teachers’ highest professional degree [Extrapolated results] 30 Table 12. Educational background of first and second grade teachers (%) [Extrapolated results] 30 Table 13. Professional degrees among First and Second grade teachers [Extrapolated results] 31 Table 14. Teachers degrees and students’ EGRA average score 32 Table 15. Directors’ report concerning their teachers’ attendance (%) [Extrapolated results] 33 Table 16. Parents’ answers: « has your child been hungry the previous week? » [Extrapolated results] 43 Table 17. Percentage of first and second grade students absent because of illness in October [Extrapolated results] 43 Table 18. Percentage of students who report a decrease in health related absences [Extrapolated results] 44 Table 19. Presence of washing stations in schools [Extrapolated results] 45 Table 20. Health and Hygiene Conditions November 2016 [Extrapolated results] 46 Table 21. Percent of parents who consider their children education important [Extrapolated results] 47 Table 22. Percent of parents who can name benefits of education [Extrapolated results] 47 Table 23. Percent of fathers and mothers who can name number of benefits of education [Extrapolated results] 48 Table 24. Percent of school with an APE or an AME [Extrapolated results] 49 v Mid-term Evaluation FFE | April 2017 List of Abbreviations CCS Head of School Districts (Chef de Circonscriptions Scolaires) CI Cours d’Initiation (first grade) CP Conseillers Pédagogiques (Pedagogical Advisors) CP Course Préparatoire (second grade) CRS Catholic Relief Services ACE Agent Contractuel de l’Etat APE Agent Permanent de l’Etat EGRA Early Grade Reading Assessment ENI Ecole Normale des Instituteurs INFRE Institute National Pour la Formation et la Recherche en Education UNICEF United Nations Children's Fund USDA US Department of Agriculture WEI World Education, Inc. vi Mid-term Evaluation FFE | April 2017 Executive Summary Catholic Relief Services, Benin (CRS Benin) is implementing a four-year project to improve literacy among first and second grade students in the Northern part of the country. Improved literacy requires that students be healthy and “ready to learn”. That readiness to learn will be attained by providing school lunches and by improving access to clean water and sanitation. The project also aims to strengthen the learning environment and to improve the quality of instruction. Community involvement is built-in the planned activities. Improved health will increase student attentiveness and reduce students’ absences, thus increase learning. Trainings for teachers and school administrators, improved instructional materials, school facilities and infrastructure will also increase the quality of education. The project will improve health and dietary practices by implementing awareness campaigns for parents, by trainings teachers and food preparers in hygienic practices, and by improving infrastructure and sanitation facilities The mid-term evaluation seeks to ascertain whether the intervention has improved the following:  Reading level of students;  Quality of reading instruction;  Overall student enrollment/attendance and students' attentiveness;  Attendance of teachers;  School environment; and  Knowledge of the community on the importance of education. The intervention takes place in 4 “communes” in Northern Benin. 144 schools are involved in the intervention area, 98 in the control zone located in nearby “communes”. These control schools do not benefit from any intervention. The mid-term evaluation drew a sample of 50 schools in the intervention zone and another sample of 50 in the control area. The data come from a survey of teachers, principals and parents in the first and second grades of the 100 schools. In addition, focus group discussions with parents also took place. Observations of first and second grade classrooms focusing on reading instruction were made. The evaluation also used the data from two surveys of first and second grade students’ reading proficiency that had been carried out in November 2015 and June 2016. The school samples were independent of the November 2016 samples. This test measures the extent to which students are able to perform various reading tasks, such as comprehension of a simple text. The data files were made available to the evaluation team which carried its own analyses and produced results identical to those produced by the M & E team. In addition, interviews with district supervisors and senior officials also took place. FFE’s first objective is to make students competent readers at the appropriate level by the end of the second grade. This objective has not been attained and is unlikely to be. Even if reading competence is defined as obtaining 2 out of 5 points (less than 50%) on the reading and comprehension part of the test, barely 2% of students obtain such a score. The average score on that part of the score represents 0.60% of the maximum (5 points). 97% of the students obtain zero on that same part of the test. viii Mid-term Evaluation FFE | April 2017 being 67% in urban treatment schools (these would be the teachers eligible for training). 39% of teachers in treatment rural areas are paid by the community and usually have had no prior training. WEI, the organization charged with running that part of the project, selected a unit (INFRE) responsible for pedagogical issues to carry out the training program. The expectation was that participation in the training would reinforce INFRE’s technical capacity. That unit, however, lacked the basic technical expertise to become a strong partner. Moreover, it is located 600 KM from the intervention area, thus making easy access to local supervisors and to the intervention classrooms difficult and costly. However, many factors make it virtually impossible to improve teachers’ effectiveness. Classrooms, number of students in them, availability of books create major obstacles to success and the removal of these obstacles does not depend upon FFE. Moreover, even after training has taken place, teachers or supervisors are transferred out of the area. They are replaced, when they are, by people who have not been trained and who do not know the project. The trusting relationships between instructors and teachers that contribute to a training program’s effectiveness disappear when transfers occur. This practice of transferring teachers and supervisors out of the project’s schools during the school year reduces the project’s impact. Learning to teach more effectively demands sustained relationships between instructors and teachers being trained. The only training workshop that took place was not sufficient to create the mentoring relationships that are important to change teaching behavior. FFE operates on the premise that an appropriately nourished student practicing good hygiene and who engages in healthy behavior will be “ready to learn”. That student will not usually miss school and, when in school, that student will be attentive, participate and learn. FFE has met these objectives or will in the very near future. It has done so to a large extent because, besides the project’s effective management and committed staff, FFE has built relationships with all relevant local, regional and national actors, particularly in education, but also in agriculture and health. These working relationships are essential when water supply, or school gardens are planned in an arid part of the country. The web of cooperative relationships between local, regional and national officers is wide. CRS handles the complex management of the food supply. At lunch time, hundreds of student have access to large quantities of food that has been prepared by trained volunteer cooks. In addition, CRS’ partner, WEI (responsible for the education component) works with communities to strengthen the organizational skills of AMEs. A very large part of the canteens’ success depends upon the training of the cooks and secretaries who keep the records and who volunteer their time. Management committees, assisted by the school director and at least one teacher, insure that the bookkeeping is properly done. WEI, being responsible for the education component, also runs a mentoring program designed to assist young girls’ transition to school by having an older girl living in the same village mentoring her. Future activities will involve helping teachers with their housing problems (a major difficulty in rural areas). Communities are significantly involved in a large number of activities and are acquiring organizational skills that will remain after the project leaves. Parents find it difficult to name three advantages of education. However, they are convinced that education is very important for their children (92%) and they overwhelmingly cite the economic advantages of education (a job and taking care of the parents when they get old). The advantages of education cited by parents appear indifferently as first, second or third reason. Two other indicators of the importance given to education can be cited: parents came in very large numbers to the focus group meetings and participated. In addition, they pay fees to hire community teachers because the state does ix Mid-term Evaluation FFE | April 2017 not assign enough of them. In poor communities, that is a significant testimony to the importance parents attach to their children’s education. The expectation that improved nutrition and hygiene will lead to better health has been fulfilled. The difference between treatment and control schools concerning being hungry is large (in rural areas, control school parents report their child as having been hungry during the previous week at 3 times the rate (39%) of the treatment areas (13%). Absenteeism is lower in treatment schools than the control zone. Thus, the activities under FFE’s control have been realized, or are in the process of being realized. Members of the communities appreciate the benefits their children derive from the project and they demonstrate their appreciation by volunteering their time and efforts. The organizational skills (canteen committees in which cooks are active), the knowledge of nutrition and safe food preparation will stay with them. Of course, it is difficult to imagine that these communities will be able to purchase, or even grow, such quantities of food. It would also be prohibitive for the GOB whose canteen budget has remained constant for at least 5 years and who cannot provide the basic resources that are required for children to read. 1 Mid-term Evaluation FFE | April 2017 Introduction “We wish to have productive Beninese citizens through making all girls and boys healthy and with a quality and complete primary education.” 1 Food for Education believes that children will learn to read more easily if they are well nourished and healthy. Students will require more than food and health, however. FFE organizes its activities along two major axes, one centering on making the individual child “ready to learn”, the other on helping the child acquire reading skills. Making the child “ready to learn” involves nutrition and health while learning to read involves training teachers, and making reading materials available. Each of these categories of intervention is itself composed of several activities. The project has been operational since October 2015 in the Northern part of Benin (an arid part of the country), in 144 primary schools located in four communes (counties). Approximately 39 000 students benefit directly from the intervention. The project has been managed by Catholic Relief Service, assisted by World Education International, two US based NGOs. Each is responsible for several but distinct activities designed to make the attainment of the objectives possible. The mid-term evaluation focuses on the extent to which progress has been made on strategic objective 1, improved literacy at the end of the second grade, and strategic objective 2, increased use of health and dietary practices. This study seeks to ascertain whether the intervention has improved the following:  Reading level of students;  Quality of reading instruction;  Overall student enrollment/attendance and students' attentiveness;  Attendance of teachers;  School environment; and  Knowledge of the community on the importance of education. This report follows seeks to answer the questions listed above and in that order. It is logical to begin with the reading level of the students since that constitutes strategic objective 1. Quality of reading instruction follows and the presentation turns to enrollment and attentiveness. The 4th question, teachers’ attendance closes the elements of the intervention pertaining to strategic objective 1. The last two questions are concerned with the individual student’s nutrition and health as well as the activities of the community. These activities center on insuring that the nutrition and health objectives are attained effectively and that the project’s achievements will be sustainable. 1 Terms of Reference. Problematic. p. 20 2 Mid-term Evaluation FFE | April 2017 FFE CONCEPTUAL APPROACH The selection of reading as the ultimate objective is strategic because reading conditions a student’s entire school career. Thus, a successful entry into the word of reading by the end of the second grade will significantly increase the probability that each child will successfully complete primary school. The FEE project seeks to make second grade students in the experimental (or treatment) schools capable of reading upon completion of that school year. To reach that objective, the project organizes a number of “enabling conditions” which complement each other. For example, being well nourished leads to being attentive in class and that, in turn, fosters learning. The intervention sees the child as embedded in several sources of influence. First, nutrition and good health are required and many of the project’s activities are designed to improve these two conditions. In addition, teachers need to be proficient and the classroom must not include too many students. The classroom itself is part of a school which, in turn, is part of an educational system. That system is managed by administrators at different levels whose tasks vary: designing the curriculum, supervising, etc. Each teacher, thus each student, is affected by the actions of these officials and, therefore, the project cannot reasonably ignore these actors. Lastly, the school is also part of a local community. Parents must support what takes place in the school and in the classroom. From its inception2, FFE has been conceived as a series of inter-related activities involving many local, regional, national, and even international actors. The activities are highly diverse, from receiving the food that arrives from the US, to warehousing and dispatching it to the schools, to insuring that it is safely stored and prepared. It involves building kitchens and storerooms, training cooks and community members who are responsible for managing the day-to-day actions needed to provide nutritious meals on time to a several hundred children in most schools. The interventions can be divided into the following categories of “enabling conditions”, i.e. conditions that contribute to the objectives’ attainment. 1. Quality of reading instruction Teachers are indispensable. That, too, is the state’s responsibility but FFE can assist by organizing training workshops. Teachers must be in the classroom and spend time on reading instruction. Learning to read is a highly complex and difficult activity, the more so when children must learn to read in a language that is foreign to them. If children are “ready to learn”, the teachers’ task will be easier. 2. Nutrition and health This category of intervention increases the likelihood that the students will be “ready to learn”. Good health depends upon nutrition (hence, the importance given to the acquisition and preparation of balanced meals by trained persons adhering to norms of hygiene which are also part of the project), but it is not the only factor. Hygiene is also essential. Hand-washing before eating prevents the development of diseases and so does the appropriate disposal of human waste. These types of enabling conditions will increase attendance because the child will be healthy and attendance will enhance the opportunity to learn. Moreover, once in the classroom, the student will be more likely to concentrate on learning because of the absence of hunger. The safe disposal of human waste can also contribute to girls’ enrollment since facilities can be reserved for girls’ exclusive usage, an important consideration in a Muslim area. 2 The TOR include a presentation of the change theory that underpins the project’s conception. 3 Mid-term Evaluation FFE | April 2017 3. Learning resources Individual good health, however, is not sufficient. Children must also have access to appropriate pedagogical inputs in order to learn. The project’s activities include the distribution of some learning materials. The project is not responsible for the availability of basic textbooks (that is the state’s responsibility), but it makes supplementary readers available so that students will be able to practice what they have learned. Parents can also be made more sensitive to the need for supplies through community meetings and discussions. Funds to purchase school supplies can be made available through the organization SLIC by community members. 4. The educational system Teachers do not work alone but rather within a system that goes beyond the classroom. Being part of an educational system affects what they do and how they discharge their duties. That is why the project actively involves local supervisors and national decision-makers in several types of activities relevant to the daily tasks of teachers. 5. The community Lastly, the community (the parents) should participate in certain school activities – such as storing and preparing food – because these activities will reinforce their commitment to the school and will also enable parents to develop new organizational skills. Such skills could be increase the sustainability of the project’s accomplishments. In addition, they will derive satisfaction from being able to contribute to their children’s welfare. Parents can show through their volunteer work that they are committed to their children’s education. If these conditions are met, a child should experience a significantly increased probability of learning to read when attending a FFE school compared to children who attend a school without the intervention. 6. Primary Education in Benin In Benin, the availability and skills of teachers represent a major challenge that could affect the project’s success. Previous evaluations of student learning in Benin have established that only a small proportion of children attain the expected competencies.3 Several of these include the second grade and all have found that the proportion of students who perform at grade level hovers at the 20% level. 4 This low level of achievement is partly the result of enrollment growth, itself the outcome of rapid demographic growth and of the schooling rate. The growth in enrollment has stretched resources (including the organizational capacity of the highly centralized system). The TOR characterize the educational system as “dysfunctional”. Even if this characterization should be only partially correct, it means that the system’s day-to-day practices will affect the project’s success. For example, if a teacher, or a supervisor, who has been trained by FFE is transferred to another school, the new skills will no longer be available to that school. Yet, the project cannot prevent such a transfer. Even when relationships between a project and local authorities are cooperative and cordial, constraints exist (maternity leaves, career opportunities, for example). If the number of teachers involved becomes 3 PASEC 2014 Performances des systèmes éducatifs en Afrique subsaharienne francophone compétences et facteurs de réussite au primaire. CONFEMEN. See also: Ministère des Enseignements Maternel et Primaire, 2011. Les Acquis des élèves du CP et du CM1 en mai 2011. 4 Evaluation à mi-parcours du Plan décennal de développement du secteur de l’éducation du Bénin. Février 2012. 4 Mid-term Evaluation FFE | April 2017 significant, the project’s success can be compromised. Thus, FFE faces significant obstacles which were mentioned in the baseline study, particularly in the case of teachers’ skills and availability. The evaluation report presents the methodology used to collect the evidence. The presentation of the evidence starts with the students’ reading level which is followed by a discussion of the teachers’ qualifications. This discussion includes observations of teachers’ teaching behavior in the classroom. The report then turns to the enabling conditions involving nutrition and health. 5 Mid-term Evaluation FFE | April 2017 Evaluation Methodology INTRODUCTION In order to fulfill the terms of reference, two methodological conditions had to be met: the mid- term evaluation had to make comparisons between the experimental and the control schools possible. Such comparisons would establish whether the intervention had an impact. The instruments developed by the baseline team thus had to be used. However, the mid-term evaluation had to rely on measurements that were not included in the baseline study simply because the interventions had not yet occurred. In order to come close to an experimental design, schools that matched the characteristics of the “experimental” schools were selected to constitute a “control” group selected from schools in nearby communes. The baseline study established that significant differences between the “experimental” and the “control” groups were not significant. Thus, in the strictest sense, the original design did not fully adhere to the requirements of random assignment, but the evidence suggests that valid and reliable comparisons can be made between experimental and control groups. These remark, however, demand that the analysis take them into account. That is why the analysis does not strictly adhere to the customary before and after comparisons. Rather, it adheres to the following model: Experimental schools : ￾￾￾￾￾ Control schools : ￾￾ ￾￾ Effects resulting from treatment : (￾￾ − ￾￾) − (￾￾ − ￾￾) Where, O1 is the value obtained for a specific indicator at the baseline study for the experimental schools O2 is the value obtained for a specific indicator at the MTE for the experimental schools O3 is the value obtained for a specific indicator at the baseline study for the control schools O4 is the value obtained for a specific indicator at the MTE for the control schools This method eliminates selection biases, even if the two groups were originally different. The schools selected to be included in the experimental or control groups are part of an educational system, thus subjected to the consequences of policy decisions that can have consequences for the schools in the project and for the interpretation of the evidence. One involves the “disappearance” of certain schools in the samples because these schools merged with another for policy reasons having nothing to do with the project. Yet another difficulty stems from the practice of assigning teachers during the school year to another school that might not be part of the project. The key actors in this experimentation are thus not always 6 Mid-term Evaluation FFE | April 2017 stable. When a teacher is assigned to another school not in the sample, the benefits that resulted from training, supervision, better health of the children, fade away. The new teacher to a FFE school, on the other hand needs to be socialized into the new ways of doing things and should also benefit from training. These two factors can reduce the measured effects of the experimentation. It should also be noted that teachers are sometimes not assigned at all and that classrooms are sometimes vacant or staffed with persons recruited locally. Even if the school director and his colleagues patch together some instruction, it will not be nearly as complete as when a regular teacher is available. DESIGN OF THE INSTRUMENTS The school director, teachers, parents, district officials as well as selected senior policy makers were the relevant actors, in the baseline study as well as in the mid-term evaluation. Instruments had been designed instruments to collect data from these persons and these instruments constituted the basis for the ones used in the mid-term evaluation. Necessarily, the modified instruments reflected the various interventions that had taken place since January 2015. Questionnaires were used to collect evidence from directors, teachers, and parents. The other actors were interviewed individually. The parents’ questionnaire is long because it seeks to measure various health practices on the part of the child as well as information concerning the child’s recent eating habits. The purpose of these detailed questions is to ascertain the extent to which these children eat at school and enjoy a nutritious diet. After reading the abundant documentation available, the consultant studied the baseline study report and revised the instruments. Upon arriving in the field, the evaluation team sought the inputs of CRS and WEI monitoring and evaluation staff members who made themselves available to answer clarification questions. SAMPLING An important issue involved the sampling approach. After studying the baseline report and the voluminous documentation, the consultant had concluded that relying on the same sample of schools as that used in the baseline study would be both practical and methodologically satisfactory. The alternative, drawing new samples, was initially considered and rejected. 5 However, after discussing the issue with the statistician in more detail, doubts emerged on the desirability of relying on the same sample as that used in the baseline study. On several occasions, these discussions included members of the monitoring and evaluation staff who contributed their insights but who were not involved in the final decision (the team also benefitted from the input of a professional statistician who was involved in another part of the study and who joined the conversation). The final decision was to select new samples largely because these samples would better reflect the situation in November 2016. For reasons explained in the baseline study, and with which we concurred entirely, the experimental and control groups were composed of 50 schools each. Part of the reason for the decision to select new samples (the population of experimental and control schools remained identical in the baseline and mid-term evaluation studies) involved the fact that the EGRA tests were based on a sample of students who were not identified individually in each sampled school. Thus, the comparisons that could be made over time involve an aggregation of students. The decision not to 5 Additional technical details are included in an appendix. 7 Mid-term Evaluation FFE | April 2017 identify individuals makes it impossible to control for individual characteristics that may affect learning to read. The selection of new samples insures that the randomness of the selecting students is maintained. As previously mentioned, 100 schools were selected, 50 experimental and 50 from the control group. This high rate of representation insures that the variations in the completion of the activities would be included in the schools visited. FIELD WORK The training of the 21 previously selected interviewers was first devoted to the mastery of the content of each instrument and to its modification when such a revision was thought necessary. Virtually all the interviewers had been involved in EGRA before and were familiar with this type of survey. During the third and last day of the training program, a trial run of the instruments was organized, one group being responsible for one instrument and expected to report wording or other issues to the rest of the team. A debriefing then took place and several modifications were made. That trial of the research instruments took advantage of a meeting of many teachers and directors in a nearby school which made every type of actors readily available, except parents. The final version of each instrument was then loaded into each of the tablets. A session was devoted to insuring that the interviewers could use the tablets competently. A detailed calendar was established which took account of the linguistic competence of the interviewers. This calendar identified which school each team of 2 interviewers were to visit on each of the 10 days the field work lasted. 5 supervisors were selected. These persons were expected to answer questions and to collect the tablets of the other interviewers in the evening to download their content onto the website. The expert who had trained the interviewers in the tablets’ usage and who had loaded them with a version of each instrument was responsible for insuring that the data were in the appropriate format. The interviewers used 3 questionnaires, one for the school director, one for first and second grade teachers as well as one additional teacher, and one for parents. The interviewers contacted the school director the day before the visit so that s/he could ask parents to come to the school. In the first and second grade classrooms, the interviewers observed the teaching of reading. One interviewer was responsible for appraising the degree to which a subgroup of 10 students was attentive and the other recorded what activity took place, such as the teacher writing on the board, students responding to questions individually or as a group. An observation grid had been prepared and the relevant information was entered into the tablet. The usage of that observation grid, particularly the criteria to be used to judge the teacher’s activity as “excellent” or poor, had been discussed during the training sessions. This instrument had not been used in the baseline study. That baseline study had observed whether students were attentive during the reading session. That was also done in the mid-term evaluation, but modified by using “participation”. The measurement of attentiveness relies heavily on the students’ behavior. The mid-term evaluation observed also observed the behavior of students during the reading lesson. The teacher’s behavior was also recorded. Participation is highly relevant since reading represents an activity that requires action both by the teacher and the students. These actions should be directly recorded to know what took place in the 8 Mid-term Evaluation FFE | April 2017 classroom. Some practice with the instrument took place during the training sessions, but that practice did not last very long. An approach relying more heavily on the observation of activities seemed to hold sufficient promise of valuable information to be tried on an experimental basis. Interviewers selected a random sample of students in each class. These parents were then contacted to be interviewed. The response on the part of parents was overwhelming despite the fact that this period was one of intense agricultural work. In addition, members of the APE and AME stayed (if it happened that their child had been selected) in order to participate in the focus group led by one of the consultants. INTERVIEWS AND FOCUS GROUPS Other actors also participated: the regional director shared his views on the reading programs and their implementation during a long conversation. The CCS of each commune was also interviewed. Prior to these exchanges, the head of the nutrition program at the Ministry shared her experience with different types of feeding programs. The head of INFRE and his collaborators discussed EGRA. The head of WEI and his collaborators shared their experience not only with the administration of EGRA, but also their role in training the APEs and AMEs. The monitoring and evaluation teams were readily available to provide information. Documents were made available, such as progress reports, the report concerning the monitoring of the teachers’ training and the mentoring programs. The focus groups were limited to the intervention schools because the questions centered on issues of governance, training of the canteen management committees, and the day-to-day difficulties specific schools might have encountered in the several activities within the project’s scope. The visits to the schools afforded an opportunity to appreciate what had been achieved in these schools and to discuss issues informally. Thus, in some schools, access to clean water was an issue, in another it was the presence of cows in the school yard or the fact that some people would ride their motorcycles in the school yard, even when children were present. A frequently mentioned problem was the lack of teachers and the fact that the community had to recruit them and pay their salaries through a school wide assessment. During the focus-group, the national consultant visited classrooms to independently observe reading instruction at the appropriate time and to observe students’ reading skills. These visits reinforced the conclusions made on the basis of EGRA results. Meal preparation was observed as well as all other activities (food storage, record keeping, quantity and quality of the meals). The presence of hand washing facilities, and of soap, was verified as well as the latrines’ number and condition. These visits also make it possible to learn about other activities, such as gardens or savings groups. Most importantly, the field visits made it possible to learn about the communities’ concerns and to witness first -hand how grateful parents are and, also, how convinced they are that education is essential in today’s world. In fact, when the evaluation team asked to meet with parents who do not send their children to school, a frequent response was: “there are no such families”, a statement that may not be entirely accurate because, at certain times of the year, some children are officially enrolled, but they may not always attend. Teachers were asked how frequent such absences were. METHODOLOGICAL APPENDIX Sampling procedure used in baseline study 9 Mid-term Evaluation FFE | April 2017 In the schools, the goal was to interview all principals and all teachers in the 100 selected schools. However, because some teachers and principals were absent, a response rate for each sample was calculated. Similarly, the study aimed to interview ten parents at each school but often less than ten selected parents were present. Each sample was weighted by the inverse probability of selecting the school from the universe of schools provided, and by the inverse probability of selecting the respondent from the total respondents available within that school. In this way, sample weighting accounts for the selection of the school, but also for the selection within the school or classroom. Schools were randomly selected with probability proportional to size. The measure of size was the number of students enrolled in each school. Therefore the school weight is as follows: ￾￾ ￾ = ￾ ￾￾￾ Where ￾ is the number of schools sampled in each strata, ￾￾ is the number of students in each school and ￾ = ￾ ￾￾ ￾ ￾ Where ￾ is the number of schools in the explicit stratum. The school weight was adjusted in specific cases. There were 7 schools in the intervention area classified as urban; these schools received a weight of 1. In order to avoid some weight being less than unity, the size of large schools (schools with sizes larger than the sampling interval given by ￾ ￾￾ ), was set equal to 50. As a result, these large schools were sampled with equal probability without having to use an explicit stratification approach. Finally, the sampling weight was adjusted for the rural control stratum since one of the schools was closed. Adjustment Factors Weights for principals and teachers were adjusted based on response rate. The adjustment factors are listed as follows: ￾￾￾ = ￾￾ ￾￾ Is the adjustment rate for teachers, where ￾￾ is the number of teachers within school ￾ and ￾￾ is the number of teachers who completed the survey in school ￾. Thus the final weight for the teachers is as follows: ￾￾￾￾ = ￾￾ ∗ ￾￾￾ Similarly, the parents subsample was adjusted based on the probability of selecting the parents from the total available parents at school ￾. ￾￾￾ = ￾￾ ￾￾ 10 Mid-term Evaluation FFE | April 2017 ￾￾￾ is the adjustment factor for parents, where ￾￾ is the total number of parents at school ￾, and ￾￾ is the number of parents from school ￾ who participated in the survey. Thus the final weight for the parents is as follows ￾￾￾￾ = ￾￾ ∗ ￾￾￾ Classrooms were similarly adjusted for the probability of selection. The adjustment factor for classrooms is as follows: ￾￾￾ = ￾￾ ￾￾ Where ￾￾ is the total number of classrooms eligible for observation in school ￾ (note that only grades 3- 6 were eligible for selection), and ￾￾ is the number of classrooms that were observed in school ￾. Additionally, this sample is adjusted based on the probability of selecting a student within the classroom. This adjustment factor is as follows: ￾￾￾￾ = ￾￾￾ ￾￾￾ Where ￾￾￾ is the number of students in classroom ￾ in school￾, and ￾￾￾ is the number of students observed in classroom ￾ in school￾. Thus the final weight for classroom observation is as follows: ￾￾￾￾￾ = ￾￾ ∗ ￾￾￾ ∗ ￾￾￾￾ Finally, the principal sample was adjusted by the response rate of principals within the strata. ￾￾ = ￾ ￾ Where ￾ is the total number of principals within the strata, and ￾ is the number of principals who completed the survey in that stratum. Thus the full weight calculation for principals is: ￾￾￾￾ = ￾￾ ∗ ￾￾ 11 Mid-term Evaluation FFE | April 2017 Results STUDENTS’ READING LEVELS 1. EGRA6 and its Characteristics The project selected the French language version of EGRA (Early Grade Reading Assessment, a widely used instrument developed by RTI with funding from USAID) to measure the reading skills in the first and second grades. EGRA’s reliability and validity are well documented. However, experience with the French version is limited to Mali and Senegal in Africa7 . The EGRA evidence was collected in separate studies in January 2015 for the baseline study, in November 2015 and June 2016 for FFE monitoring. Eighteen of the 21 enumerators had participated in one or, in a few cases, two of the EGRA surveys, including the one done for the baseline study in January 2015. They were selected by the mid-term evaluation because they were known to be reliable workers and because they brought significant experience to the task. In order to verify whether EGRA was valid in Benin, i.e. that it measured the competencies specified in the official national curriculum, a comparison of EGRA’s content with the official reading program in the second grade in Benin was carried out by a member of the evaluation team with significant experience in the reading curriculum in Benin. The conclusion of that comparison is that all the components found in the Benin reading program are also present in EGRA. Thus, a second grader evaluated to be “competent”, hence qualified to be promoted to the 3d grade, should score at an adequate level on EGRA8 . EGRA measures several components of the student’s reading competencies: Vocabulary During the vocabulary section, the enumerator asked the students to point at different items. Each item was asked once to the student and it was marked as correct or incorrect according to the student’s response. The items were body parts (head, knee, ears, foot, stomach), school items (eraser, blackboard, notebook, chalk), things from nature (the sun, a stone, a bowl and a tree), and spatial terms (a door, the top of the table, a rag, the corner of the table). Oral comprehension 6 For further details on EGRA, see RTI International. 2015. Early Grade Reading Assessment (EGRA) Toolkit; Second edition. Washington, DC. United States Agency for International Development. 7 The large amount of evidence collected in the course of FFE could add valuable information to what is already known. 8 This assertion could not be empirically verified. INFRE is said to have been tasked with the development of a simple version of EGRA, as recommended by the baseline study, and to establish whether such a simplified version of EGRA would yield sufficiently informative data to be useful to teachers and to supervisors. The evaluation team did not see a short version of EGRA nor the metric that could be used to determine whether a student is a competent reader. The need to define what total score on EGRA should be reached for a student to be considered competent in reading still exists. 12 Mid-term Evaluation FFE | April 2017 The oral comprehension section is an extension to the vocabulary section. The enumerator asked students to perform some actions using a pencil or a rag. In total the students were asked four groups of actions. In Group A, the enumerator asked students to pick a pencil and put it on top of a sheet of paper, behind them, on the soil, under the paper, by their right side, and in a box. In Group B the enumerator ask students to take the pencil with their hand and put it on the table on their right side. In Group C, the enumerator asks students to take a chalk with their right hand and put it under the table. Knowledge of letter names In the section on knowledge of letters, the students were asked to read as many letters as they could in a minute. The enumerator presented a grid containing 90 letters organized in table of 9 rows by 10 columns. Letters were capital or lower case and some had accents. If students could not read any of the letters in the first row correctly they were asked to stop this section of the test and move to the next one. Simple words reading In this section, the enumerator presented a grid with 50 words and asked student to read as many words as possible in one minute. The majority of the words have one or two syllables. The section is stopped if the student cannot read the first five words correctly. Invented words reading Similar to the previous section, the enumerator presented a grid with invented words to the student who should read as many as possible in a minute. Most words have one or two syllables and have no meaning in the French language. The section is stopped if the student cannot read the first five words correctly. Text reading and comprehension This section had two parts. In the first part, students were asked to read a short text, or as many words as possible in a minute. In the second part, a series of questions are asked to the student on what they were able to read. There are a total of 5 questions, but students are asked the questions in proportion to how much they were able to read. Oral comprehension In the section of oral comprehension, the enumerator read a story twice in French and then asked 5 questions about the story. Dictation In this section, the enumerator read a sentence three times. The first time students listened; the second time the enumerator read slowly and students wrote as best as they can; the third time, the enumerator read the text so students could correct what they wrote. The enumerator then graded the text by classifying each word as correct, partially incorrect or incorrect. A correctly written word is worth twice as a partially correct one. The students did not get any credit if they wrote the word incorrectly or nothing at all. 13 Mid-term Evaluation FFE | April 2017 The test measures group outcomes, i.e. how groups of students (classrooms, schools, groups of schools) performed on each part of the test. This procedure makes it possible to ascertain whether reading skills, and which ones, are improving. Improvements can be identified by using standard statistical tools, such as an average, mode or median. However, in this case, the objective is to determine whether students are competent readers, or how far they are from that level. This is substantively crucial because, if students leave the second grade being competent readers, it is highly likely that they will develop higher reading skills at a normal rate because they will have mastered the basic tools. The selection of reading mastery at the second grade level is unarguably strategic. However, a description of the situation does not address the issue of reading competence because competence implies a level of performance. This characteristic applies to all tests designed to insure that recipients of a diploma meet previously determined minimal standards of performance. For example, in most countries, a candidate for the driver’s license may not make more than a certain number of errors in order to be deemed a minimally “competent” driver. The baseline study recommended that such a threshold be developed, but, as far as the consultant knew, that recommendation had not been implemented. Thus, the early versions of the evaluation report relied on several analyses that were similar to those used in the baseline study. However, in late February 2017, a report was sent to the mid-term evaluation team, report that included the minutes of a meeting that preceded the teacher training workshop described later in this report. The officials responsible for running the workshop realized that a standard of competence needed to be defined and they relied on the competency approach that is used in Benin. Usually, 3 points out of 5 is considered to be the lowest acceptable performance to be considered “competent” in any academic performance in Benin. Thus, the team settled on a threshold of 3 out of 5 points in the reading comprehension portion of EGRA. This decision was pragmatic, but entirely consistent with the standards used in Benin. The EGRA surveys were administered at three time points: in January 2015, in November 2015 and in June 2016. The instruments used in the baseline study in January 2015 were not identical to the other two. The specific differences between the January 2015 instrument and the others could not be ascertained because the January instrument was not included in the baseline report appendix. For example, section one included 17 items in January 2015 while there were 15 in the later tests and, in section 2, there were 19 items in January 2015 but 18 in the later ones. The differences pertaining to the other sections of the test are unknown because the report does not specify how many items were included. The EGRA tests were not administered in the control schools in November 2015 nor in June 20169 . In the following analyses, we present descriptive statistics concerning students’ performance. Then, we rely on a definition of competence that is slightly more generous than the one developed by the training workshop. Our justification was that 50% of the maximum score constitutes failure in the Benin system (and in the US as well). Thus, in order to be considered competent, a student has to score at least 2.5 points out of a possible 5. Instead of rounding up to 3 points, we decided that, in this context, a slightly 9 The baseline study established that no significant difference existed between treatment and control schools. After that, the only change of interest concern what happens to students in the treatment schools as a result of the intervention. 14 Mid-term Evaluation FFE | April 2017 less stringent level of performance would be appropriate. It turns out that using 2 or 3 as constituting “competence” does not change the proportion of students who are defined as being competent. 2. FFE’s usage of EGRA FFE administered EGRA in January 2015, in November 2015 and in June 2016. In addition, a small experimental survey was also administered to a small sample of schools. These surveys yielded the following samples: Table 1. Dates and sample sizes of FFE EGRA surveys 15-Jan 15-Nov 16-Jun CI CP CI CP CI CP Sample size 1346 1331 1960 1890 1647 1514 Number classrooms 88 93 135 128 138 129 Source: EGRA data base for the relevant dates 10 The November 2015 and June 2016 surveys relied on identical instrument which were administered to two different samples of students selected from all 144 schools. The sample size is high and, thus, the conclusions can be assumed to be reliable and valid since EGRA measures the competencies officially expected at the end of the second grade in Benin. The project management decided early that the selected individual students would not be followed over time. This decision was made to avoid the possibility that teachers would coach these particular students, thereby making the results invalid. Such a fear is realistic. Moreover, following individual students creates a substantial risk of attrition that could bias the results. However, a panel design offers some advantages. It makes it possible to control for the influence of individual factors since they remain constant. Following the same students over time also creates the opportunity to study whether the mastery of one part of the test affects others. In other words, a panel design can help determine the correlation over time between parts of the test. 3. EGRA Results A student who obtains the highest number of points possible can accumulate 366 points. With such a high score, a student can be assumed to be a competent reader at the second grade level. That is also likely to be the case at the 90% level as well. However, a student who reads at the 60% level could have some deficiencies.11 For the policy maker, knowing that the vast majority of the students read at the 30% of the maximum level implies that significant remediation should be organized. When a metric is available, it would become easier to know where the majority of the sample stands in relation to the objective. That information would then assist decision makers in the allocation of resources, for example, schools A and B should receive assistance to organize summer programs in order to bring students to the required level by the time the new school year begins. A project such as FFE could significantly contribute to such an effort if food were available during that summer program. 10 Some schools do not have a first grade while others do not have a second, hence the dissimilar numbers of first and second grade classrooms. 11 It would be useful to compare EGRA results with those obtained by teachers when they evaluate their students. It is possible that the results would be identical, but that EGRA would provide additional details that could assist the teacher in designing remediation. Such an activity would make EGRA useful at the systemic level (the educational system) as well as the classroom. 15 Mid-term Evaluation FFE | April 2017 The following table presents the percentage of learners who fall within a specific performance level. For example, no one performed at 75% of the maximum possible performance in June 2016. Then, the performance level at the 50% of the maximum level, then at the 25% and, finally, at the 10% levels are presented. It then becomes possible to determine how many students fall above and below these performance levels. In the case of the two surveys, the overwhelming proportion of all students fall at an extremely low level of reading competence, confirming the findings of the baseline study. EGRA RESULTS IN NOVEMBER 2015 AND JUNE 2016 The maximum number of points that can be obtained is used as a point of reference. We then compute the performance of students at specific cut-off points, 75%, 50%, etc. It can fairly be assumed that 10% of the maximum means that the students who fall in that category cannot read at the expected level. Table 2. Percentage of First and Second Grade students by level of performance Nov-15 Jun-16 Rural Urban Rural+Urban Rural Urban Rural+Urban CI CP CI CP CI CP CI CP CI CP CI CP Girls (%) < 75% maximum score 100 100 100 100 100 100 100 99 100 100 100 100 < 50% maximum score 99.6 99.5 100 98.6 99.8 99 100 98.9 100 97.7 100 98.3 < 25% maximum 97.8 89.8 98.6 88.6 98.2 89.1 99.3 87.9 98.5 85.6 98.9 86.7 < 10% maximum 75.4 41.7 73.3 38.2 74.3 39.8 79.5 37.2 75.7 39.5 77.6 38.4 Boys (%) < 75% maximum score 100 100 100 99.8 100 99.9 100 100 100 100 100 100 < 50% maximum 99.8 99.1 99.8 98.7 99.8 98.9 100 99.4 99.8 98.3 99.9 98.8 < 25% maximum 99 89.1 98.3 87.7 98.6 88.4 97.3 88.5 98.2 88.7 97.7 88.6 < 10% maximum 78.6 40.8 76.8 35.8 77.6 38.1 77 41.9 74.9 39.7 75.9 40.7 Source. EGRA survey 2015 and June 2016 This table shows that no student scored higher than 75% of the maximum score and that the situation was essentially identical at the 50% of the maximum score level. It is only when the level reaches a low of 10% of the maximum score, or 37 points out of 366, that 38% of the CP students in 2015 fall in that performance level in 2015 and 41% in 2016. Differences between boys and girls are very small. The evolution between November 2015 and June 2016 is essentially non- existent. However, the difference between first and second grade results are large. The following table provides additional details concerning the performance of second grade students at the end of the school year. “Average score in November 2015” and “Average scores in June 2016”represent average scores for each of the 10 components of EGRA. “Maximum possible score” 16 Mid-term Evaluation FFE | April 2017 represents the maximum number of points it was possible to obtain on each component of the test. “% maximum” represents the % of the maximum the average score for that part of the test represents. Thus, on the first part of the test, the average score was 10.85 out of 15 possible points or 72% of the maximum. Table 3. Percentage of the maximum score in November 2015 and June 2016 2nd grade Test Component Maximum points November 2015 CP June 2016 CP Average score N=1889 % maximum Average Score N=1514 % maximum 1. vocabulary 15 10.85 72.33 10.66 71.07 2. oral comprehension 18 12.46 69.22 11.71 65.06 3. letters names 90 13.72 15.24 13.60 15.11 4. letters sounds 60 6.11 10.18 6.12 10.20 5. initial sounds 10 1.71 17.10 1.66 16.60 6. familiar words 50 1.35 2.70 1.37 2.74 7. invented words 50 1.24 2.48 1.31 2.62 8 a. words in context 45 2.02 4.49 2.58 5.73 8 b. Text comprehension 5 0.03 0.60 0.06 1.20 9. oral comprehension 5 0.49 9.80 0.48 9.60 10. dictation 18 2.03 11.28 3.01 16.72 Source. EGRA survey 2015 and June 2016 The table shows stability across the 2 time periods. For example, in items 6, familiar words, second graders were able to score 2.7% of the maximum in November 2015 and 2.74 % in June 2016. Only in the dictation, which measures the ability to write (a skill related to reading, but nevertheless different), were the average scores higher in June 2016 by slightly more than 5%. An examination of the different components of the test shows where the performance was relatively high and where, on the contrary, it was low. For the first item in the test, the average score represented 72% of the maximum number of points possible in November 2015; for the second item, the performance decreased to 69% in 2015 and to 65% in 2016. After that, the scores plummeted to reach a low of less than 1% on the words in context. In the case of “Text comprehension” (8.b), the desired objective, the average score was of 1.2% of the maximum, an increase over the 2015 score. Such a difference could fall within the measurement error. The following table shows the proportion of second graders who scored 0 on the various components of EGRA in June 2016. That percentage increases as the difficulty of the test also increases, reaching 97% for text comprehension. 17 Mid-term Evaluation FFE | April 2017 Table 4. Percentage of 2nd graders who received 0 on EGRA sections in June 2016 Component Maximum points Average Score N=1514 % zero 1. vocabulary 15 10.66 1.67 2. oral comprehension 18 11.71 13.98 3. letters names 90 13.60 12.77 4. letters sounds 60 6.12 15.73 5. initial sounds 10 1.66 70.59 6. familiar words 50 1.37 75.61 7. invented words 50 1.31 78.27 8 a. words in context 45 2.58 65.65 8 b. Text comprehension 5 0.06 97.11 9. oral comprehension 5 0.48 71.81 10. dictation 18 3.01 36.78 Source. EGRA survey June 2016 Second grade students obtain better results than first graders, thus showing that some learning had taken place, but the majority of students actually obtain 0. The stability and consistency of the results over 2 time periods (the baseline study shows results consistent with the ones presented for November 2015 and for June 2016) forces the conclusion that second graders are reasonably competent at the very simplest level (items 1 and 2), but that they falter thereafter, except in dictation (writing). This suggests that most 3d graders will begin the new phase of their schooling with a significant handicap. Indeed, during the site visits, the national consultant asked students in several 3d grade classes to read some simple words he had written on the blackboard or on a slate. In those classes, virtually no 3d graders could recognize letters. These students had begun the third grade about 6 weeks earlier after almost 4 months of time away from school since many schools essentially cease to function after the sixth grade school leaving exam has taken place in early June. These results show that an extremely high proportion of the students who leave the second grade are far from having mastered the expected reading competence. In a sample of 1514 second grade students, 24 scored above 2 out of 5 possible points in the comprehension part of the test, or 1.6%. Differences between boys and girls are negligible, except that the highest scoring student was a girl. CCS are well aware of the situation and so is the regional director. However, they are overwhelmed by difficulties that cannot readily be tackled, among them the lack of teachers and the deficient training these teachers received. Teachers use a phrase to explain why they are not able to produce better results: they claim that “the program is too dense” (is the sequencing poorly organized, are steps missing?), or “the students do not understand”. In other words, teachers blame either the reading program or the students. The 18 Mid-term Evaluation FFE | April 2017 implication is that the match between the two, the program and the student, is far from ideal. It is interesting that hardly any teacher mention teaching skills or the need to be better trained. 4. Conclusion When first and second grades are compared, EGRA results show that learning does take place. That learning, however, is insufficient given the objective. Students do well on the easiest parts of the test, but the ability to comprehend what has been read is essentially nil. They are on their way to being able readers, but they are far from being at the expected level. The evidence reveals no evolution between November 2015 and June 2016. The evidence also shows that there has been no evolution between January 2015 and June 2016. That comparison, however, must be made with caution because the instruments used in January 2015 differed from the one used in the two subsequent surveys. However, despite this difficulty, it is likely that the early and later findings are extremely close. Some components of EGRA are timed, i.e. the student must perform within a period of time. If time is exceeded, the student is not allowed to proceed and obtains zero. This requirement leads to a question: would students perform better if no time constraints existed? That is an empirical question that the next administrators of EGRA could investigate on a small sample of students. These students usually come to school knowing no French at all. School is where they learn that language, but the opportunities to learn it are limited because children speak the local language among themselves. In class, they learn phrases by heart and eventually comprehend their meaning. Whether students can learn a new language and to read simultaneously is open to question. Children who do relatively well on EGRA tend to have parents who speak at least some French and who may use that language at home part of the time. The analysis done by the WEI team shows that having parents who speak French improves EGRA results. These are circumstances that the school does not control. Exposure to the language seems essential and the opportunity to practice the language are limited because of the conditions that prevail in the classrooms: number of students, a reading program that may not be totally appropriate for the population, and qualifications of the teachers. EGRA results are satisfactory for the early part of the test. What characterizes this part of the test? Is the progression from one skill to another more gradual than it is in the subsequent parts, is the time devoted to it longer, does it rely more heavily on actions on the part of students and less on abstract symbols? The early part of the test involves understanding simple commands and these have been learned by the second grade. This relatively good performance suggests that students should reach a level of French similar to their level in their own language before they begin to read. Experimentation in Senegal and in Mali has shown that learning to read in a native language facilitates learning to read in that language. However, the transfer of the ability to read in Wolof (the dominant language in Senegal) for example, to the ability to read in French causes difficulties. It is thus not surprising that children who are asked to learn to read in a language they do not know experience difficulties. Children who hear some French at home learn to read faster and that exposure implies that the time currently devoted to reading may be too short given the complexity of the task or that, prior to reading, students should spend a significant amount of time just speaking. In short, there is need for an effective immersion program. It may also be that the teaching method does not rely sufficiently on the principle of discovery, i.e. on figuring out what the words might mean, given that some elements of the words are already known to 19 Mid-term Evaluation FFE | April 2017 the learner 12. Lastly, students have few opportunities to practice reading. They may not bring books home, even if there were enough of them at school for all students. It may also be that the currently available texts do not provide a sufficiently wide range of opportunities to practice their newly acquired skills. Children learn in different ways and it may be that the current teaching tools are too monolithic to yield the expected results. This analysis suggests several options to remedy the current situation: provide more opportunities to learn French (that implies more time in the curriculum for that activity), more opportunities to practice French in the classroom, more opportunities to read at home, and different teaching methods. How can such practices be put in place when no teachers are available or when classrooms are extremely overcrowded? If reading comprehension is the objective, the overwhelming majority of second graders do not fall within an acceptable range. If 50% of the maximum number of points is defined as a minimal level of performance, very few students meet that criterion. As stated above, a tiny minority of students score more than 50% on the comprehension component of EGRA. Both in the US and in the French inspired educational systems, less than 50% usually means that the expected competencies have not been acquired. These results should be placed in a comparative context. A recent observational study done in 131 French second grades showed that the time allotted to reading was slightly above 7 hours a week (compared to under 4 hours in Benin). In that sample, teachers had an average of 22 students, all of whom had attended preschool for 2 years and whose native language was French. Despite these favorable circumstances, the range of individual outcomes was large13. EVALUATION QUESTION The TOR specify the following indicators as a measurement of the students reading levels:  Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (male)  Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text (female) The percentage of boys and girls who are competent to read can be estimated at 2%, using a score of 2/5 on the reading and comprehension portion of the test as a criterion. No student scored above 50% of the maximum number of points which is considered to constitute a “passing” grade in Benin. 12 That approach to reading is presented briefly in an appendix. 13 Bishop, Marie-France; Cebe, Sylvie; Pique, Celine. 2016. L’enseignement de la compréhension dans les classes de CP aujourd’hui : temps consacre et choix didactique. Reperes 52: 15:38. 20 Mid-term Evaluation FFE | April 2017 QUALITY OF READING INSTRUCTION 1. Time spent on reading instruction The indicators selected to measure the quality of reading instruction included two related measures : Improved quality of literacy instruction (“70% of teachers who devote at least an average of 45 minutes a day to literacy instruction”) and improved attentiveness (“75% of students in target schools who are identified as attentive during class/instruction”). While the indicator of quality of literacy instruction is empirically straightforward, such is not the case for attentiveness on the part of students. The idea is that, in order to learn, students should pay attention to what the teacher is doing or asking them to do. Students should be “primed” for learning or receptive. It is obvious that they cannot be attentive, ready to engage in behavior that fosters learning, if they are hungry. The connection between the removal of hunger and learning is direct. However, such a psychological state is difficult to measure, for students could be day-dreaming or giving the impression that they are “attentive” while, in reality they don’t know what’s happening in the class. Moreover, African students, in primary school especially, are remarkably well behaved compared to their counterparts in rich countries. Thus, obviously disruptive behavior in the presence of an adult, particularly of a teacher, is rather rare. Quality of reading instruction refers to what teachers do in the classroom to make it possible for students to learn. Key determinants of what happens in the classroom are classroom’s condition and the initial training the teacher received. Reading demands not only listening and carrying out instructions on the part of the student, but it also involves practicing the skills in order to eventually master them. The opportunity to practice the new skill, reading, and being helped when struggling depends on the relationship that develops between the teacher and the student. In turn, the development of that relationship will be affected by the classroom conditions, among which the number of students is an important component. That number affects what can, and cannot, be done during class time. Other factors are obviously involved, such as in-service training, the kind of relationship that exists between the director and teachers and whether teachers feel appreciated by the hierarchy. The selected indicators, devoting time to reading instruction and students being attentive (the operational definition will be discussed below) raise several issues. The selection of the indicator assumes that 45 minutes a day represents an appropriate length of time. Yet, the French study mentioned earlier reports that 7 hours a week are allocated to reading instruction, or more than 80 minutes a day. Despite the vastly more favorable learning conditions in French classrooms, learning is not homogeneous, a finding that will surprise no teacher. FFE had planned to train 144 teachers (although, typically, there is a CI and a CP in each school) and supervisors as well, in order to improve their skill level. The ability of teachers to perform effectively in the classroom was not measured in the baseline study. However, that ability could be assumed to be quite low. Supervisors said, then and now, that teachers lack skills and the results of the January 2015 EGRA provide ample evidence that such is the case. However, improving teachers’ skills as an intervention mechanism takes time and it would be possible to successfully train teachers and still get poor results because of the learning conditions. The baseline study did not consider a number of these variables in detail, but the mid-term evaluation suggests that contextual variables are crucial to understand what took place between January 2015 and November 2016. Such an understanding should provide a partial explanation of the results. 21 Mid-term Evaluation FFE | April 2017 Teachers report spending more than 45 minutes on reading instruction, 60 minutes in treatment schools, 57 in the control zone. The baseline study reported that “on average, teachers spend 100 minutes per day teaching literacy.” (p. 20). The difference between the two studies may seem large, but it may reflect localized adaptation to learning situations. The time devoted to reading is significantly longer than the officially mandated 45 minutes. Time does not seem to be the issue. 2. Attentiveness and Classroom Activities Another indicator selected to measure the quality of reading instruction was whether students were attentive during instruction. In order to learn to read, students need to concentrate on their tasks. That is the idea behind the importance accorded to nutrition: if students are not hungry, they will be able to focus on learning in the short term and, if properly nourished, they will be able to maintain a level of physiological well￾being that will insure the normal physical and psychological growth of each student. If the teacher has to spend a significant amount of time on reminding students that they should be focused on their work, or correcting disruptive behavior, learning will be reduced and, in the worst cases, learning will not take place. The observations realized for the baseline study showed that 74% of the students were attentive. That is not a surprising number because students in Benin are remarkably disciplined, i.e. they do what they are asked to do, largely because they have been taught that adults must be obeyed. The problem is that apparent attentiveness and day-dreaming are not far apart in terms of overt behavior. Moreover, in this case, the student may conform to the criteria used to identify attentiveness, but that student may understand little of what is taking place. The overt behavior in the baseline assessment of attentiveness is taking notes, doing what the teacher asks, and raising hand to answer question. The other criteria, listening to the teacher for example, constitutes an inference that is difficult to verify. Behavior implies not only attentiveness, but also involvement and action. Taking notes (not relevant in this context) and asking questions are observable. In the early grades in Benin, it is likely that actions by students, opening the book after the teacher asked students to do it, may simply constitute an imitation of what the other students do. Later, the words in the command will acquire meaning. Because EGRA was taking place at the same time as the other parts of the baseline study, first and second grade students were not observed, but grade 3 and up were. In the mid-term evaluation, first and second grade students were observed. Criteria for classifying students as attentive in the baseline study. Attentive:  All or most of student behaviors are related to actively completing the task that the teacher has assigned, or  Student is listening to teacher or other students as they participate, or  Student is taking notes that appear to be related to classroom content  Student raises hand to answer question in a sincere manner 22 Mid-term Evaluation FFE | April 2017 Inattentive:  Student is singled out by the teacher to remind him/her of the focus of the class  Student makes comments that are disruptive to the class  Student is speaking to other students during a time that s/he should be listening  Student is “fidgeting” in his or her seat, to the point that it is disruptive of others around him or her  Student is asleep  Is forgetful in classroom activities The bold type points to participation, to actual behavior. The assumption is that these behaviors are relevant to learning to read. The emphasis is on overt behavior that can be observed and recorded. Thus, the concept of attentiveness, in this case, emphasizes activity and avoids more inferential aspects that attentiveness often implies. The mid-term evaluation emphasizes involvement, what students actually do in response to the teacher. If students do not respond to the teacher’s commands, do not participate, then they are assumed not to be attentive. However, another possibility exists: they might not understand what is being said. Since the evaluation took place in mid -November, first graders might still be unfamiliar with what a student should do in class. Enumerators proceeded as had been done in the baseline study by randomly identifying 10 students. One enumerator focused on what the teacher did, the other on the students. The outcome of these observations show a substantial degree of activity on the part of the teacher and participation by the students. The findings shown below are entirely consistent with the 74% reported in the baseline study. This method is more detailed than the one used in the baseline study because it records a wider range of actions, from doing a lot to doing virtually nothing. That instrument also records what did not take place. For example, in 79% of the observations, students did not write. Participation Poor Mediocre adequate good Excellent NA Participation 1% 10% 39% 39% 9% 2% The participation level is high since it is rated at good or excellent for 48% of the students. The following shows a significant level of activity during a reading lesson. Average of observed activities by the teachers and students during reading lessons. Table 5. Teachers’ and students’ activities during the lesson Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Teacher Speaks, explains Poor 1,7 0,0 0,0 0,0 1,4 0,0 23 Mid-term Evaluation FFE | April 2017 Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment (%) Mediocre 0,0 1,6 0,0 0,0 0,0 1,3 Acceptable 15,5 10,9 8,3 7,1 14,3 10,3 Good 63,8 71,9 58,3 64,3 62,9 70,5 Excellent 13,8 15,6 25,0 28,6 15,7 17,9 NA 5,2 0,0 8,3 0,0 5,7 0,0 Teacher writes on the blackboard (%) Poor 1,7 0,0 0,0 0,0 1,4 0,0 Mediocre 1,7 3,1 0,0 0,0 1,4 2,6 Acceptable 19,0 20,3 25,0 14,3 20,0 19,2 Good 41,4 35,9 33,3 50,0 40,0 38,5 Excellent 8,6 7,8 8,3 7,1 8,6 7,7 NA 27,6 32,8 33,3 28,6 28,6 32,1 Teacher shows an example to the students (%) Poor 0,0 0,0 0,0 0,0 0,0 0,0 Mediocre 0,0 3,1 0,0 0,0 0,0 2,6 Acceptable 15,5 20,3 0,0 7,1 12,9 17,9 Good 56,9 50,0 66,7 71,4 58,6 53,8 Excellent 5,2 6,3 8,3 7,1 5,7 6,4 NA 22,4 20,3 25,0 14,3 22,9 19,2 The whole class repeats Poor 6,9 4,7 8,3 21,4 7,1 7,7 Mediocre 5,2 1,6 8,3 0,0 5,7 1,3 Acceptable 12,1 12,5 0,0 21,4 10,0 14,1 Good 8,6 1,6 25,0 7,1 11,4 2,6 Excellent 0,0 1,6 8,3 0,0 1,4 1,3 NA 67,2 78,1 50,0 50,0 64,3 73,1 Students repeat individually Poor 1,7 0,0 0,0 0,0 1,4 0,0 Mediocre 1,7 3,1 8,3 0,0 2,9 2,6 Acceptable 24,1 20,3 8,3 7,1 21,4 17,9 Good 41,4 56,3 75,0 71,4 47,1 59,0 Excellent 5,2 3,1 0,0 7,1 4,3 3,8 NA 25,9 17,2 8,3 14,3 22,9 16,7 Teacher asks questions to the students Poor 1,7 0,0 0,0 0,0 1,4 0,0 Mediocre 1,7 0,0 8,3 0,0 2,9 0,0 Acceptable 17,2 12,5 0,0 7,1 14,3 11,5 Good 55,2 54,7 66,7 71,4 57,1 57,7 Excellent 12,1 26,6 16,7 14,3 12,9 24,4 NA 12,1 6,3 8,3 7,1 11,4 6,4 Teacher answers students’ questions (%) Poor 0,0 0,0 0,0 0,0 0,0 0,0 Mediocre 0,0 0,0 0,0 0,0 0,0 0,0 Acceptable 15,5 7,8 8,3 0,0 14,3 6,4 Good 17,2 6,3 16,7 28,6 17,1 10,3 Excellent 1,7 0,0 0,0 0,0 1,4 0,0 NA 65,5 85,9 75,0 71,4 67,1 83,3 Teacher assists the students by using reading strategies Poor 0,0 0,0 0,0 0,0 0,0 0,0 Mediocre 1,7 3,1 8,3 0,0 2,9 2,6 Acceptable 17,2 23,4 16,7 28,6 17,1 24,4 Good 41,4 42,2 50,0 64,3 42,9 46,2 24 Mid-term Evaluation FFE | April 2017 Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Excellent 8,6 10,9 8,3 0,0 8,6 9,0 NA 31,0 20,3 16,7 7,1 28,6 17,9 Teacher follows up and provides additional assistance Poor 1,7 0,0 0,0 0,0 1,4 0,0 Mediocre 1,7 1,6 0,0 7,1 1,4 2,6 Acceptable 31,0 23,4 25,0 28,6 30,0 24,4 Good 41,4 34,4 33,3 57,1 40,0 38,5 Excellent 1,7 7,8 8,3 0,0 2,9 6,4 NA 22,4 32,8 33,3 7,1 24,3 28,2 Source. Classroom observation guide Each line represents the average of all observations of the activities during the lesson. If they read, for example, they are obviously attentive since they are doing what is required of them: 45% of the students read a word they already knew and 38% later read a phrase. 42% answer a comprehension question. It is also possible to focus on the merely adequate, i.e. students who did not participate significantly. The progression went from very simple to the more difficult. The association of letters seems a little more difficult. Writing, that day, was not emphasized. The observations also showed that 9% of the students asked questions. That may seem small, but most children come to school not knowing any French. Some families speak French at home. These parents are usually educated and use French because mother and father speak a different national language. These results raise a fundamental issue. If teachers and students are as active as this evidence suggests, then why are EGRA results not better? Several answers are possible, but the most important is that far several sessions in the same classroom. These sessions should be taped so they could be reviewed by highly experienced reading experts. If we assume that the evidence presented above translates the reality of the classroom reasonably reliably, then what explains the poor EGRA results? The first answer is that students do not learn the fundamental skills involved in reading, i.e. they do not learn how to use what they have learned before in a new situation. They cannot generalize from what they know to what they do not know. Thus, there is no building, each situation is considered novel, thus difficult. They learn by heart, are able to repeat, and nothing else. Finally, the opportunities to practice the newly acquired skills, if they are learned, are very limited. For example, group work is rare. Yet, to practice reading, grouping students can provide opportunities to practice reading, by following what another student is reading and, at another time, by being asked to read oneself. These ideas should be investigated in detail. If they are found to be valid, they might explain why students do not read adequately. Evaluation question: The percent of students who are identified as attentive during instruction is estimate at 87% in the control school and 91% in the intervention schools. 25 Mid-term Evaluation FFE | April 2017 The percent of students who repeat when asked to do so was estimated at 73% in the control schools and 81% in the intervention schools. These numbers suggest that, as was the case during the baseline study, students are attentive and participate. 26 Mid-term Evaluation FFE | April 2017 LEARNING CONDITIONS 1. Classroom Size and Availability of Supplies The baseline study documented the learning conditions by relying on CCS’ statements. As the following section will show, the baseline study’s assertions (see pp. 20-24) do not differ from the evidence presented in this report. The mid-term evaluation seeks to document the factors that may explain why reading is at such a low level. Among the basic requirements for success is a classroom with a number of students that makes some individual attention possible. Table 6. Average number of students in the first two grades CI CP Control Treatment Control Treatment Rural 43 74 61 51 Urban 40 59 47 69 Source. Teachers’ questionnaire In the first grade, rural treatment schools have a large average size of 74 students. Control first grades have smaller averages, 43 in rural areas and 40 in the urban. In the second grade, enrollments are always above the official norm which was set at 40 many years ago when the EQF idea was adopted (fundamental quality school); 69 in treatment urban and 51 in the rural areas. It should be noted that these are averages with significant variations around the mean. Some classes have more than 100 students! Some of these very large enrollments occur when no teacher is available for one class. Thus, two classes become one. Treatment schools are characterized by higher enrollments than control schools, with one exception (rural control CP). These numbers cannot readily be compared to the baseline study because its authors did not provide such details. The qualitative evidence they present is entirely consistent with the evidence presented here. There is no reason to believe that average enrollments in the 144 FFE schools has significantly changed. The important point here is that classrooms are overcrowded. The numbers presented in Table 6 do not exaggerate the crowded conditions. The field visits confirmed that many classrooms had many more students than the room could accommodate. It was not uncommon to see 5 children sitting at a desk designed for 2. The field visit afforded an opportunity to visit classrooms while the students were on recess or at lunch. Their names are often written on the desks and, very frequently, the number of students far exceeded the desk’s normal capacity. Providing some individualized attention is simply not possible in such classrooms. Yet, the official pedagogy in Benin is based on the competency approach which requires the student to be active participants in learning. The learning conditions, however, do not make it possible for teachers to have students practice what they have learned. Most of these students come to school speaking no French at all. They must learn all the basics of the language and simultaneously learn to read, a demanding task, even if they had an experienced and well trained teacher to guide them. In addition, the classrooms where such complex learning should take place are not always in good condition, as the following table shows which summarizes the teachers’ assessment of their classrooms in the first two grades. 27 Mid-term Evaluation FFE | April 2017 Table 7. Classroom condition for the first two grades (%) [Extrapolated results] CI CP Rural Urban Rural Urban Control Treatment Control Treatment Control Treatment Control Treatment Bad condition 30,8a 30,8a 33,3a 25a 29a 7,4b 50a 50a Average Condition 34,6a 53,8b 50a 75a 25,8a 51,9b 50a 50a Good condition 34,6a 15,4b 16,7a 0 45,2a 40,7a 0 0 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p < .05 in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source. Teachers’ questionnaire No pattern clearly emerges in the case of the first two grades. For the first grade, the percentage of classrooms in “good” condition” is similar to the percentage of classrooms in “bad” condition. For the second grade, rural treatment schools are somewhat less likely to be in good condition, but more likely to be in average condition. The proportion of classrooms in “poor” condition is high in urban areas. What about supplies? Table 8. Average % of Students who do not have all the required supplies [Extrapolated results] Rural Urban Control Treatment Control Treatment CI 41 21 19 48 CP 29 35 30 41 - Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. - Denominator is the average number of students in the respective classrooms Source. Directors’ questionnaires The percentage of students who do not have the required supplies varies between 19% in urban control CI to a high of 48% in the urban treatment CI. In the CP, the percentage varies between 29% in rural control CP and 41% in urban treatment CP. This evidence, heavily influenced by the size of the denominator, average enrollment, nevertheless shows that in the best of circumstances, 1/5 of the students do not have what they need, that percentage being close to 50% for the urban treatment CI. Teachers were also asked about the extent to which they thought they had adequate didactic materials to work with. 28 Mid-term Evaluation FFE | April 2017 Table 9. Teachers’ perception of their didactic material (%) [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Adequate 8,7a 15,4b 33,3a 23,1a 10,9 18,7 Insufficient 82,5a 80,3a 66,7a 76,9a 81,1 78,9 No material 8,7a 4,3a 0 0 8 2,4 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source: teachers’ questionnaire A large proportion of rural teachers, in treatment as well as in control schools, thought that the availability of didactic materials was insufficient. That proportion was also high in both types of schools in urban areas. On average, there were 2.5 students per reading textbook and 2 students per workbook. These are the workbooks where students practice writing. Each student is supposed to have one. Students must have access to basic learning materials. In the case of reading at the first and second grade levels, each child should have a textbook and an exercise book. These are normally made available by the state. In fact, according to the EGRA results of November 2015, 7% of students do not share a book with another student and 35% share a book with only one student. Officially, each student is supposed to have one book in the early grades. Official national statistics report that the average number of reading books per student is 1, a number considered to maximize learning opportunities. In 2014-2015, MEMP statistics report 505 267 students being enrolled in the first grade nationwide. These students were reported to have access to 540 496 French textbooks, i.e. more than one book was theoretically available to each first grader. In the second grade, 441 686 were enrolled and these second graders had 446 343 French textbooks available, or a ratio of 1 book per student. The distribution of these books is very uneven and directors often complain of receiving shipments that bear little relation to the actual enrollment. It is likely that the distribution system is unreliable and inefficient and some theft may occur. The issue is important enough to warrant a thorough external investigation by the Ministry. If the books are in fact bought in sufficient numbers but are not available to the students, the large financial investment is not producing the anticipated return: learning and all the benefits such learning brings about. 2. Teachers’ Academic and Professional degrees The baseline study insisted on the fact that teachers were unable to provide quality instruction. The authors state that “… teachers … do not have sufficient training or resources to provide quality instruction”. (p. 40). They did not elaborate, but their conclusion cannot be disputed, as the data presented below demonstrate. 29 Mid-term Evaluation FFE | April 2017 Besides the limited availability of books and less than optimal opportunities to read in class (and at home because books must remain in class), teachers’ skills loom large as a factor likely to explain the low level of reading skills. Two indicators are relevant: the status of teachers, i.e. whether teachers were employed by the state or not, and their training. Being employed by the state usually means higher income and stability of employment. Teachers are trained in teachers’ colleges over a two year period, the first being devoted to the mastery of the various disciplines (French, mathematics, history, and pedagogy), the second to supervised teaching in a classroom. In practice, the second year is supposed to involve supervised teaching. In fact, they work as regular teachers and may benefit from some supervision by the director or other teachers who are responsible for their own class as well. The shortage of teachers leads communities to hire their own and they are paid by the parents. These teachers may hold a teaching degree or not. Table 10. Teachers’ status [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment APE14 32a 34,2a 37,5a 34,6a 32,5 34,4 ACE 1a 6b 0 3,8a 0,9 5,1 Community 54,4a 32,5b 54,2a 15,4b 54,4 25,1 Student teachers 11,7a 27,4b 8,3a 46,2b 11,4 35,5 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source. Teachers’ questionnaires November 2016 APE teachers are regular civil servants who have been recruited on the basis of a competitive exam which they can sit if they hold a teaching degree. They are a small minority in the treatment schools and are essentially absent in the control schools. Community teachers represent 54% of teachers in rural control schools and almost 33% in rural treatment schools. Teachers hired by the community receive very low pay from the parents and they can leave at any time if they find a better opportunity since, often, payment is irregular. Student teachers who are doing their supervised (the supervision varies between serious and non- existent) student teachers represent 27% of the treatment school teachers in rural areas and 54% of all control schools. Student teachers have little say as to where they will be assigned and it is therefore not surprising to find such a large proportion of this category in rural areas. If student teachers had a scholarship while in a public training college, they continue receiving that stipend (33 000 FCFA per month or about $ 60). If not, they receive no compensation at all. For the state, they are free labor. 14 APE are teachers who are civil servants who will enjoy civil service retirement benefit. ACE are teachers who have signed an employment contract with the State. 30 Mid-term Evaluation FFE | April 2017 Table 11. Teachers’ highest professional degree [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment None 46,6a 28,2b 16,7a 34,6b 43,9 31 CEAP 32a 41,9b 50a 38,5b 33,6 40,4 CAP 21,4a 29,9b 33,3a 26,9b 22,4 28,6 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source: teachers’ questionnaire November 2016 In order to become a civil servant, a position has to be declared vacant by the state and candidates must compete for the available positions by sitting an examination whose outcome determines being accepted or not. The CAP, the highest professional degree, requires having the BAC, 12 years of schooling, and having passed an examination that includes being observed by a team of officials headed by the district supervisor (CCS). Preparation for the exam takes place in a state sponsored training college or in a private one. Then, the teacher must perform sufficiently well on the state administered examination to fall within the number of vacancies (if there are 100 positions, the successful candidate is someone who scored within the top 100 candidates in terms of number of points). Treatment schools have a slightly higher proportion of teachers with the highest certification (30% vs. 32%). 47% of rural control schools teachers have no professional degree, a much higher percentage than treatment schools, 28%. A possible explanation is that control schools have nothing special to offer, particularly no lunch program. It is often thought that schools that house a project will have better amenities. How teachers are recruited or assigned is officially straightforward, but informal factors are often involved, or so teachers claim. Community teachers are paid with funds collected from parents. Some pay, some do not. Thus, whether a teacher is paid depends on the community’s ability to raise funds. It is not unusual for teachers to leave before the end of the school year because they are no longer receiving their pay. During the field work, mid- November 2016, no community teacher reported having yet been paid during the school year that had begun in early October. 15 The situation concerning the teachers most relevant to this study, first and second grade, must be examined more closely. What are the status and qualifications of the teachers charged with teaching to read to an average of 50 students in the first grade and 47 in the second? Table 12. Educational background of first and second grade teachers (%) [Extrapolated results] Rural Urban All (Rural + Urban) 15 In Benin, it takes several months after having been appointed for the paycheck to arrive. After the waiting period, however, pay is regular. 31 Mid-term Evaluation FFE | April 2017 Control Treatment Control Treatment Control Treatment CI BEPC 69,2a 80,8a 50a 75a 67,5 78,8 BAC, DEAT 19,2a 11,5a 50a 25a 21,9 16,2 DEUG, DUES 7,7a 0 0 0 7 0 University 3,8a 7,7a 0 0 3,5 5 CP BEPC 90,3a 77,8b 75a 83,3a 89,5 80,2 BAC, DEAT 6,5a 18,5b 25a 0 7,4 10,5 DEUG, DUES 3,2a 3,7a 0 0 3,1 2,1 University 0 0 0 16,7a 0 7,2 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source. Teachers’ questionnaires The vast majority of teachers in the first two grades have the 9th grade school leaving examination (BEPC). However, those holding the BAC, with 3 additional years of schooling, are found in the control schools more often than in treatment schools in the first grade. These teachers are likely to have found no other employment and settle in the teaching profession while waiting for another job. In the second grade, the situation is different and more teachers hold the higher degree in treatment schools than in control schools. The most common educational attainment of teachers is the BEPC, 9th grade. Table 13. Professional degrees among First and Second grade teachers [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment CI None 53,8a 3,8b 16,7a 0 50,6 2,5 CAP 26,9a 65,4b 66,7a 50a 30,4 60,1 CEAP 19,2a 30,8b 16,7a 50b 19 37,4 CP None 67,7a 51,9b 25a 66,7b 65,6 58,2 CAP 16,1a 18,5a 25a 16,7a 16,6 17,7 CEAP 16,1a 29,6b 50a 16,7b 17,9 24 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source. Teachers’ questionnaires 32 Mid-term Evaluation FFE | April 2017 Among first grade teachers in control schools, 54% have no professional degree and, in the second grade, that proportion reaches 68%. In treatment rural first grade, 65% have the highest credentials. Overall, the Combining rural and urban areas, the proportion of teachers with highest professional qualifications are almost twice as high in treatment schools as in control schools.16 For the second grade, overall, 58% of treatment classes are under the guidance of teachers with no professional degree while a minority (17 and 18%) have the highest certification. Those holding the other professional degree, the CEAP, are slightly more represented in treatment schools overall, but they represent 50% of teachers in the second grade. It is worth underlining that, overall, 58% of second grade teachers have no professional qualification. It is likely that this outcome results from a deliberate decision: it is thought preferable, given the scarcity of teachers, to assign fully certified teachers to the first grade rather than to the second grade. For CCS and school directors, the options are limited because the last grade, the 6th, is when students take the national exam which makes access to secondary schools possible. That class is almost always assigned to the director. Thus, if the 6th grade should be in the hands of an experienced teacher (the director), and it is thought preferable to assign the first grade to a qualified teacher, that leaves fewer options for the other assignments, particularly if teachers are not available. The presentation of teachers’ backgrounds and professional degrees assumes that better educated teachers who have a professional degree, thus training, produce better results than their less well educated colleagues. That association in fact does not exist. Table 14. Teachers degrees and students’ EGRA average score 2015 2016 Rural Urban All (Rural + Urban) Rural Urban All (Rural + Urban) Mean N= Mean N= Mean N= Mean N= Mean N= Mean N= Degree ND 77a 15 24b 15 50 30 41 24 . 0 41,25 24 None 37a 619 48b 585 42 1204 38a 168 51b 166 44 334 CAP 28a 173 35b 268 32,5 441 41a 404 35b 609 38 1013 CEAP 38a 812 41b 1140 40 1952 36a 654 44b 675 40 1329 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source. EGRA survey 2015 and June 2016 The students’ scores were linked to their own teacher’s qualifications. Among all teachers in the EGRA test of November 2015, the highest average was obtained by the students of teachers who had no 16 One possible explanation is that teachers with the higher degrees were selected by district supervisors to be assigned to treatment schools, possibly at the request of the teachers who expected to enjoy more desirable working conditions in a school involved in a project. In a highly bureaucratic organization, supervisors have little to offer teachers. An assignment that a teacher wishes to have can constitute a reward for good performance or a favor to the teacher for professional or personal reasons. 33 Mid-term Evaluation FFE | April 2017 professional degree (score = 42 in 2015; 44 in 2016)). The lowest score in 2015 was obtained by the students of teachers holding the highest qualification, the CAP, the same situation prevailing in 2016. These results can seem surprising, in particular the score obtained by students whose teacher had no professional degree at all in 2015 and in 2016 and whose students scored an average of 42 and 44 respectively. This suggests that, in Benin, professional degrees do not predict students’ performance. In fact, that is not a unique situation because the students’ individual characteristics are often a better predictor of performance. An analysis of the factors that explain EGRA performance carried out by the M & E staff at WEI shows that individual factors such as having attended pre-school and having an older sibling who knows how to read improves performance. 17 The absence of a relationship between professional degree and students’ performance has important implications for the selection of policies that might improve the situation. These will be discussed below. 3. Teachers’ attendance Directors report a very high level of attendance. Official records are notoriously unreliable, in part because directors do not want to alienate the good will of their teachers, particularly now that the new government is starting to discipline some behavior. Table 15. Directors’ report concerning their teachers’ attendance (%) [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Yes, always present 100 92,3a 85,7a 85,7a 98,7 89,5 Yes, but not enough 0 5,1a 14,3a 14,3a 1,3 9,1 Not at all 0 2,6a 0 0 0 1,5 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variance. Source. Directors’ questionnaire The difference between the two types of school is non-existent. It might seem easy to record whether teachers are present or not. It is not. If a teacher arrives late, is she absent? How late does one have to be before being considered absent? Aside from the fact that directors have their own class and are under significant pressure to get good results at the 6th grade school leaving exam (otherwise, they might lose their position and the prestige and small additional remuneration that go with the job). Informal “understandings” are common in all places of work. Punishing a teacher by reporting absences cannot be done without impunity because the teacher might be “protected” by a politician or a union leader and that protection could then bring some difficulties to the director. Thus, directors often are content with a verbal reminder that punctuality is important. Some directors come to their school very early and teachers know that their late arrival will be noticed, possibly officially recorded depending on the circumstances, such as heavy rain that makes roads difficult. Schools vary a great deal in the punctuality of teachers. If they live nearby, something that is not always possible, they can reasonably be expected to be on time. If they live far, that is more 17 This is also the case in the US where parents’ educational level is a high predictor of the child’s educational achievement. Similar results obtain in most rich countries. 34 Mid-term Evaluation FFE | April 2017 problematic because many obstacles can hinder the intention to arrive on time. The number of variables that affect punctuality and being present at work is large and, often, not totally under the control of the teacher. As noted earlier, relationships external to the school can raise or lower the cost of being late or absent. 4. The teacher training program Since one objective of the project is to improve the professional skills of the educational system’s leadership and of the teachers in the intervention zone, the project logically turned to the educational system’s training unit to work on the elaboration of a training program that would improve reading outcomes among FFE first and second grade teachers. INFRE was tasked to develop a training program, in collaboration with WEI. This collaboration was to increase INFRE’s competence in the assessment of reading skills. While the decision to rely on INFRE was logical and consistent with the project’s objectives, it did not take account of the specific skills available among INFRE’s staff. It was assumed that a certain level of skills existed and that external assistance would improve them. A lengthy conversation with the revealed a number of difficulties that hampered the full realization of the intention. INFRE has no reading specialist as such and has not been able to recruit the expertise that would have filled the gap. The WEI technical director is not a reading specialist. In addition, INFRE is located in Porto Novo, very far from the intervention zone. This geographical location made it impossible to monitor closely the progress made toward increased teacher competence in the intervention area. The local supervisors were involved in the training program, but these individuals have demanding official duties. They may gain some competencies as a result of the training, but they may not be able to significantly increase the supervision of, or the assistance provided to, first and second grade teachers. In short, the project organized a relatively short training program, but the day-to-day work of teachers and school directors, and of local supervisors, was not affected. There seems to have been no detailed diagnosis of the strength and weaknesses of the FFE teachers as far as reading was concerned, nor was a remediation plan developed to improve the situation over the duration of the project. It is, however, more accurate to say that a plan existed when the project was launched. For reasons that are not entirely clear, this plan was not implemented. Part of the reason may have been the difficulties of working with INFRE whose director was relatively new to the job and whose structure was scheduled to be re-organized. WEI found itself in a difficult position: it had to work with an official structure that ought to have had some expertise, but did not in reality. WEI could not compel INFRE to do anything, short of bypassing it, a move that would have been criticized, or even reprimanded. It is virtually impossible to bypass the official structure, even if that structure cannot carry out the planned activities. The training program took place in March 2016 and lasted 4 days. A group of 5 trainers spent time finalizing the training documents prior to the arrival of the trainees. Among the trainers were the people who normally organize the in-service training groups that meet twice a month during the school year (the UP). 137 Directors, 8 CCS and CP, as well as CI and CP 249 teachers then spent 2 days studying the documents that had been previously prepared and discussing their content. The last two days were 35 Mid-term Evaluation FFE | April 2017 spent preparing their lessons and actually teaching. Not every trainee actually taught since there was an average of 35 teachers per classroom. 18 This kind of organization is not unusual in Benin, but the number of participants in each room seems high for professional training, particularly since teacher training requires individual feedback from the trainers. The report does not specify whether students were present nor does it explain why only 249 teachers were involved since the EGRA survey carried out in June 2016 included 267 classrooms. It is surprising that no distinction seems to have been made between teachers in the first and teachers in the second grade. Yet, the problems each teacher faces in the first or the second grade are not identical. The report does not specify whether the training was defined as the first installment in a comprehensive program or whether the March workshop was defined as sufficient to improve the participants’ teaching skills. The discussions with WEI staff members suggest that organizing the workshop was difficult in part because INFRE is very far and also because it was experiencing staffing problems. 19 A follow- up was organized in June 2016, at a time when the CCS and the CP were busy with the CEP administration. Why such a date was selected is not clear. The workshop did not rely on EGRA to ascertain whether, in fact, reading outcomes had improved over that of November 2015 (EGRA is costly to administer). The report highlights the assumption on the part of INFRE and WEI that training will necessarily produce the desired outcome. As we will argue below, that is not necessarily the case. The follow up evaluation relied on observation grids, but a before and after measurement of every participant’s teaching behavior did not take place. The 47 teachers included in this follow-up were selected from a pool of candidates for the professional degree who, presumably, had been involved in the March training program. The follow-up did not refer to the performance of the trainees before and after the training. It is possible that some trainees significantly improved their performance as a result of additional professional experience and the maturation of the theoretical background acquired during the training while others would not have. The report written after the sole follow up workshop does not provide the evidence needed to determine whether teaching skills, and which ones, had improved. Yet, directors and teachers rate the training positively and as responding to their needs. It is not clear whether their perception of the usefulness of the training took account of EGRA results for their classroom. 20 5. Training situation and the outcomes The training program assessed the teachers’ knowledge, but not their teaching behavior in their classroom.21 Teachers have few resources they might use to present material in a different way and 18 The report authored by the technical director mentions 170 teachers. The discrepancy between the two sources of information, the technical director and the M & E staff, was not fully explained. 19 Two sources of information exist for this section of the report: one from the person charged with the M & E program who based his statements on the workshop attendance records; the other was from the technical director who organized the workshop. In his report, he asserts that 170 teachers participated. The source of the discrepancy has not been explained. 20 Workshop are usually evaluated positively. They provide an opportunity to learn something, to meet with colleagues and to collect a small payment. 21 The EGRA study of June 2016 asked teachers how far along they were supposed to be in the official curriculum and how far along they actually were. Most reported having completed about 50% of the official program. They provided two types of explanations: “the program is too demanding” or “the students are not able to “follow”, i.e. 36 Mid-term Evaluation FFE | April 2017 their limited training has equipped them with few professional tools. As a consequence, every new school year, teachers face a group of students who have not acquired the expected competencies in the previous grade. Being ill equipped to assess cognitive achievement, and facing bureaucratic mandates to follow the prescribed schedule, some proceed according to official plans while others move at far slower pace than officially planned. Either decision is dysfunctional. If the prescribed schedule is adhered to, many students will not be able to grasp what is being done; if the pace does not follow the official schedule, the curriculum will not be respected and many students will enter the following grade poorly prepared. A modular approach might be a solution, but teachers would have to have access to resources and they would have to be trained extensively. The cost of such training would be very high. The training program used in the training workshop had been designed to refresh previously learned skills that might have become somewhat rusty. It was not intended to train a highly diverse group of teachers, some of whom had no previous training at all. The evidence presented above showed that teachers differ significantly in their academic background, their professional training, and experience.22 Of course, the workshop may have been composed of teachers who did have prior training and who only needed some new ideas and some feedback on their own teaching. It is important to note that the training workshop and the follow-up included the people who supervise the UP, the in-service training activity that groups several schools twice a month. The usage of the specific skills acquired, or improved, during the workshop by in-service trainers was not ascertained in the follow up. In other words, the specific tasks of the participants in the workshop, teaching for the teachers and supervision for the CP and CCS, were not taken into consideration. If they were, the report makes no mention of them. The diversity of background constitutes a fundamental characteristic of the teaching corps in Benin and any teacher training program must take such diversity into account. To that characteristic should be added the wide range of conditions under which professional degrees are obtained and, as in all educational programs, the wide range of learning that actually occurs among the trainees. These fundamental features demand highly flexible training programs that must take the variety found among teachers into account. A modular approach broken into many steps might be effective to solve the diversity issue. Such a modular approach would have to include assessment techniques that would enable teachers to know whether their students had mastered the skills sufficiently to justify proceeding to the next skills. 6. Relevance of the teacher training program The two units used in the training workshop were analyzed by the national consultant. These documents present the six steps involved in a French teaching unit and reiterate the officially recommended practice. As a refresher for trained teachers having a similar level, it could have been effective, particularly if the trainees had benefitted from comments by their peers and the trainers concerning their teaching. It would have been even better if they had been video- taped. Being able to watch specific aspects of their teaching performance several times can produce significant improvements in teaching competence. In addition, outstanding performances (preferably with EGRA results) could be they cannot master the expected skills. It is noteworthy that insufficient training, or skills, on the part of the teacher are never mentioned. 22 See Tama, Clarisse. 2014. Etre Enseignant au Benin: Les mutations d’un groupe professionnel. Koln. Rudiger Koppe Verlag. This study shows how diverse the teaching corps is. One source of diversity is the recruitment methods which have changed several times over the years. This creates significant problems for the design of training programs since the kind of professional background that can be assumed to be present in fact is not. 37 Mid-term Evaluation FFE | April 2017 saved for future training programs. In this instance, can the reliance on a training program designed to be a refresher course be justified? Before starting any training program, it is essential to ascertain the existing competencies among the trainees. That was not done. Trainees read material and discussed the content, but systematic measurement of learning and of teaching skills did not take place. Then, the goal of the training should specify what specific competencies should have been acquired at the end of the training (training objectives), given the resources that have been made available. In other words, it is essential to ascertain the existing competencies, to identify the training objectives and then to identify the steps that will make it possible for trainees to reach the training objective. The training program will usually require that different modules be developed to match training and existing competencies among the trainees. In some cases, it may be necessary to complement the theoretical background of the trainees by providing the trainee with self-learning modules (that was done partly in the March training program). The report written by the technical director does not answer many questions, such as how participants were selected, and how their teaching skills were specifically measured, before and after training. In retrospect, the training workshop and the follow up can be criticized as being insufficient in number and for not taking place in the teachers’ classroom at the end of the training. It should be emphasized, however, that teachers in Benin have no incentive to improve their performance (except for their own satisfaction) and this is particularly true of community teachers who might not be retained the next year or who might decide that being a community teacher is too demanding and too poorly compensated. Organizing a customized program would be very difficult, in part because the project has no authority over teachers and also because teachers have no compelling reason to invest significantly in their professional skills. Under the circumstances, it may well be that what was organized was the best possible outcome. 7. Efficiency of the training program INFRE is located in Porto Novo, about 600 km from Kandi. If it had the necessary personnel, it could have acted as a skilled designer of training that would improve teachers’ competence. Local supervisors might act as disseminators of new methods if they themselves were trained. Indeed, some were, but the numbers involved are very small. Who would be the most effective supervisors to improve teachers’ skills, the CCS, the CP, or the RUP? All of them working cooperatively in their respective zone? Cooperation with INFRE involves significant costs. WEI had no choice in the matter because it had to presume that INFRE had the relevant personnel. It did not. In hindsight, it seems that an approach based on the classroom, possibly a small number of classrooms on an experimental basis, would have been more effective and efficient. 8. Conclusion concerning the Improvement of learning instruction The very low proportion of children who are able to read at the appropriate grade level requires an explanation. Four factors can be invoked: 1. The context. a. Children enter school not knowing French. They use the local language in their daily interaction among themselves and at home. 38 Mid-term Evaluation FFE | April 2017 b. Previous evaluations have documented a similarly low level of achievement at the second grade level. c. Essential supplies are insufficient. d. Many classrooms are overcrowded, some severely. Opportunities to practice reading are very limited. 2. Teachers. a. The shortage of teachers, and the poor results, are the most frequently heard complaint by parents and by CCS as well. b. The level of training is low, frequently inexistent c. Formal professional training does not predict the students’ performance, suggesting that the training and the award of degrees need to be evaluated. d. No incentives reward teachers for improving their skills. 3. The training program designed to improve the low teachers’ skill level did not produce the expected results a. Only one took place b. It did not involve all FFE teachers c. The design of the training may not have been optimal for the highly diverse target audience d. It did not document the learning outcomes. Evaluation Question. Quality of reading instruction; The TOR specified the following indicators: 1. 75% average teacher attendance rate for each school and aggregated by district. This objective is largely met. No evidence shows that, even if the 92% attendance rate is “generous”, that the attendance is as low as 75%. That would imply that, on an average day, 25% of the teachers are not present. The field observations do not suggest that that is the case. 92% of teachers attend regularly. There is no difference between treatment and control schools. Transfers during the school year are said to be very common, among teachers and among supervisors. How many teachers spend the entire school year with the same students might constitute a more relevant indicator. Of course, even if they remain in the classroom for the entire year, teachers’ attendance would have to be monitored. 39 Mid-term Evaluation FFE | April 2017 2. 4230 textbooks and other teaching and learning materials provided as a result of USDA assistance. Supplementary booklets have been distributed in January 2017 because of delays in printing. Reports are that students use them little. The content of these booklets should be analyzed to determine whether they are sufficiently interesting for Benin children in that age group. 3. 144 teachers using the national literacy curriculum and the related instructional materials Teachers adhere to the national curriculum and rely on the instructional materials contained in the teacher’s manual. However, the pace of learning is slow and thus the official schedule cannot be respected. If it were, most students would be lost and probably learn even less. The assumption made in the indicator is that “the national literacy curriculum and the related instructional materials” are appropriate to the learning situations. The enrollment in most classes far exceeds what would insure learning and the training of most teachers does not enable them to use the instructional materials effectively. Teachers report that they adhere to the national curriculum and respect the time officially allotted to reading instruction (45 minutes). Teachers report exceeding the time allotted to reading instruction. 45 minutes for 5 days a week comes to 225 minutes or almost 4 hours. That is a small amount for so complex a task. A recent French study cited above reported that, in a sample of 131 CP classes, teachers spent an average of over 7 hours a week on reading in classrooms far better equipped, with an average of 22 students who all had participated in 2 years of preschool before and whose teachers had been trained in a university. Teachers in Benin report not being able to complete the official number of lessons and, in June 2016, they had on average completed 60% of the total number of lessons specified in the official program. Overwhelmingly, they claim that the program is “too dense” which could mean that it includes too many skills or that the progression from skill to skill is insufficiently gradual. Such a statement could also be read as meaning that the program expects students to learn at a much faster rate than is feasible, particularly given the limited opportunities to practice. Teachers also claim that the students “do not follow” because they do not understand. In other words, teachers see a disconnect between the official expectations of the reading program and the ability of students who come to school not knowing French at all to master the expected skills in the planned amount of time. Some supervisors believe that the time allotted to learning to read is too short. Moreover, interruptions in the school year are common, teachers’ strike in particular. These interruptions shorten the time available during the school year. The official curriculum is based on a 180 days school year and on an efficient usage of the time. In fact, the actual teaching time hovers around 50% of the official time. Thus, several issues concerning the curriculum exist: rate of progression toward the goal, speed of the expected learning, appropriateness of the curriculum for these students. The recommended pedagogy may also be not fully appropriate for the students and for the conditions under which learning takes place. 4. 144 teachers trained or certified as a result of USDA assistance 40 Mid-term Evaluation FFE | April 2017 The relevance of the teacher training program has been extensively discussed in this evaluation report. Training or certification should be synonymous with the demonstration of minimal skills. That is not the case in the FFE zone. A multiple step wise regression showed that the only significant relationship between teachers’ characteristic and EGRA results came from years of experience. 5. 144 teachers in target schools who demonstrate use of new and quality teaching techniques or tools Some ambiguity exists concerning the phrase “…use of new and quality teaching techniques or tools”. The modules used in the training was “new” to many of the teachers since some of them had no previous training. These modules were not “quality” because they were not specifically designed for first or second grade teachers. “New and quality teaching techniques” cannot solve the problem because many of the elements that, together, produce good results were lacking. 394 persons (137 directors, 8 CCS and CP, and 249 teachers) were trained during one 4-day workshop. This workshop relied on already existing modules that did not take account of the diverse teachers’ skills. The follow up to the workshop included 31 directors and 52 teachers. It is unreasonable to expect one workshop to have a significant impact on teaching skills, particularly since, as the evidence shows, many teachers had no prior training. A systematic training program has not been developed. Other circumstances mitigate against training having an impact on a large number of teachers: a. Teachers are not necessarily recruited when the school year begins. Thus, training time would be truncated. b. Some teachers who were trained are transferred out of the FFE schools c. New teachers may come to replace them. They were not trained d. Supervisors who were involved in the workshop are also transferred to other areas. e. These conditions make it very difficult to develop the kind of working relationships between project staff, supervisors, directors and teachers that could result in improved teaching skills. f. The working conditions, the failure to recruit teachers in sufficient numbers and at the appropriate time, and the prevalence of transfers among supervisors and teachers are beyond FFE’s control. g. If training constitutes a key intervention variable, that intervention must be given time to show results. The minimal conditions for effective training were not present. Quality teaching skills 41 Mid-term Evaluation FFE | April 2017 The classroom observations showed that teachers are active and that students participate. However, the time available to practice skills is limited by the absence of a sufficient number of books and by the large enrollments in many classrooms. Many teachers have at best a superficial understanding of the steps involved in learning to read. When asked why the official schedule is not adhered to, teachers never invoke insufficient training. They refer to the program (essentially too demanding) or the students’ inability to proceed at the expected rate, presumably because they know no French). The conditions that would enable teachers to become more professional would require that they be treated as professionals (incentives, progression in the profession, for example). The large number of community teachers paid by parents does not encourage the creation of a teaching corps seeking to reach learning goals. WEI can be criticized for not having adhered to the well- designed original plan and to have organized only one workshop. It can also be criticized for not having documented the specific outcomes of that workshop so its lessons could then be used in subsequent workshops. However, a large number of crucial factors are not under WEI’s control, but are the responsibility of the Ministry. 42 Mid-term Evaluation FFE | April 2017 THE SCHOOL ENVIRONMENT The evidence presented above pertains to SO1, reading and the closely related issue concerning the improvement of teaching to read. The evidence suggests that the low reading skills measured by EGRA can to a large extent be explained by poor learning conditions and an insufficiently skilled teaching personnel. The differences between treatment and control schools are generally small and reflect no particular pattern. The sole training program could not have had a significant impact on the teachers’ competence, not only because there was only one workshop that did not include all FFE teachers, but also because it did not correspond to their needs, even though teachers claimed that it did. The evaluation report now turns to another aspect of the project which was subsumed under SO 2, increased use of health and dietary practices. These practices involve many changes in the school environment, such as the construction of kitchens and storage rooms, the repair or construction of latrines, the availability of drinking water as well as facilities that make it possible for students to wash their hand with soap. The community plays a key role in making all these changes possible. Mothers are trained as cooks, members of the APE or AME handle the bookkeeping involved in managing food supplies, all of them serving as volunteers. Other activities are also organized, for example a mentoring program that relies on older girls to accompany very young ones to school and to serve as role models; support to the reading objective by organizing reading groups, and the creation of a rapidly growing savings system to generate resources that could be used to purchase school supplies. The support provided by the communities can be seen in a few communities’ providing free rent to the school director. These activities benefit the students, but they also benefit the community itself because the organizational skills, the experience with successful endeavors will remain in these communities. The local NGOs that train the AMEs have developed considerable expertise in assisting communities to organize their own affairs. These NGOs can continue doing such work, even if financing such activities will not be easy in a resource poor environment. 1. Hunger The baseline study reported that 55% of the parents had indicated that at least one child had been hungry during the school day. That statement does not accord with the question asked of parents in the questionnaire which referred to “the previous week”. The other question asked of parents involved having eaten during the school day yesterday. Table 14 in the baseline study concerning “…being hungry during the school day”, a question that was not asked, shows 58% of the parents saying yes in the control schools and 55% in the treatment schools. The percentage of parents saying that their children had eaten during the school day, were respectively 97% in the control group and 98% in the treatment schools. The following questions were asked of parents in the course of the baseline study. 18. Did ${name} say that he or she was hungry during the school last week? 0 No 1 Yes 99 Don't know 19. Did ${name} eat during the school day yesterday? 0 No 1 Yes 99 Don't know 43 Mid-term Evaluation FFE | April 2017 In the mid-term evaluation, parents were asked about a child having been hungry the previous week, as had been done in the baseline study. Table 16. Parents’ answers: « has your child been hungry the previous week? » [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment No (%) 60,2a 86,1b 80,3a 92,2b 64,1 87,1 Yes (%) 39,8a 13,9b 19,7a 7,8b 35,9 12,9 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: Parents database, November 2016 13% of rural treatment parents reported that their children had been hungry compared to the 39% in the control group or 3 times more. In urban areas, the yes response was more than twice as prevalent among control parents. In the baseline study, 58% of parents in the control zone and 55% in the treatment zone reported a child as having been hungry the previous week (as noted above, the table and the questionnaire differ). The reduction of hunger in the control zone is 19% while, in the treatment zone, it Thus, compared to January 2015, the incidence of hunger decreased among students attending schools with no intervention, but it decreased much more in the treatment schools. It is possible that January is the time when food can run out in rural areas (in urban areas control schools, the “yes” response is 19% compared to 39% in rural areas). Canteens clearly reduce hunger. The issue of hunger was raised during the focus group conversations. Parents emphatically stated that hunger had “disappeared”. The results of the survey show that parents were exaggerating since almost 13% of treatment rural parents and 7% of treatment urban ones said that their child had been hungry. Thousands of children benefit from a food program every school day and that the project contributes in a major way, as shown by the difference between treatment and control group parents’ response to the question concerning their child having been hungry. In July and August 2016, CRS received delivery of almost 716 MT of commodities. This very large shipment demonstrates the magnitude of the food program, a programs that serves about 39000 students. 2. Illness Improved nutrition should contribute to good health, hence to a low level of illness among students. Table 17. Percentage of first and second grade students absent because of illness in October [Extrapolated results] CI CP Control Treatment Control Treatment Girls 15,26a 7,04b 14,30a 7,09b Boys 17,17a 6,91b 17,91a 10,67b 44 Mid-term Evaluation FFE | April 2017 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column means. Cells with no subscript are not included in the test. Tests assume equal variances. Source. Teachers’ questionnaire The baseline study reported a statistically significant difference between treatment and control schools, overall and for boys and girls. Almost 5% of all students had been absent for health reasons in the treatment areas, but that percentage was 6.4% in the control zone. Differences between boys and girls were very small in the control area. (Table 16, p. 28) Table 17 shows a low level of absence, lower in treatment schools than in the control zone. Teachers report whether students are absent, a piece of information that is required to compute how much food will be required each day. The differences are significant, essentially by factor of one to two, for example 7% for CI girls in the treatment schools and 15% in the control schools. Table 18. Percentage of students who report a decrease in health related absences [Extrapolated results] Rural Urban All (Rural Urban) Control Treatment Control Treatment Control Treatment Girls 11,33a 7,46b 11,33a 5,04b 11,33 6,42 Boys 13,85a 10,61b 10,33a 5,55b 13,53 8,43 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column means. Cells with no subscript are not included in the test. Tests assume equal variances. Source. Parents’ survey Table 18 shows that the decrease in health related absences was smaller in treatment schools than in the control area. It must be remembered that treatment schools are characterized by an initially lower rate of absence caused by illness. An examination of the raw data showed that the vast majority of students had never missed class during October. In short, absences caused by illness are very infrequent. The logic of the intervention leads to the expectation that absences resulting from illness should decrease in the treatment area. That logic must confront the fact that absences are low to begin with and even lower in the treatment area. The floor effect has probably been reached. 3. Washing Stations Clean water and the sanitary disposal of human waste are essential in promoting healthy living. Another measure is also important, especially in a cultural context in which people eat with their hands: washing one’s hands with soap. During the field visits, the consultants observed that no project school lacked the ingenious arrangement that makes it possible to press a small board with the foot so that water can flow from a recycled bottle with a hole near the top. This arrangement makes it possible for someone to wash both hands at the same time and to use soap. This kind of arrangement costs virtually nothing because it relies on available recycled bottles, a piece of string connected to the bottle and to the piece of wood that is pressed with one foot. It should be relatively easy to continue using this system after the project is no longer present. 45 Mid-term Evaluation FFE | April 2017 Information concerning the presence of washing stations were not discussed in the baseline report and for a very good reason: they would not have existed in January 2015 because the project had not built them. There would have been some latrines, but in limited numbers, except in schools previously sponsored by an international project. The situation in control schools reflects the situation in areas that did not benefit from the intervention: 80% of control rural schools have no washing stations, 38% of rural schools have access to clean water, and 43% have functional latrines (Table 20). The situation outside the FFE zone had some facilities (using the evidence from control schools in November 2016), but FFE schools enjoyed better facilities, a situation that should continue to improve since the construction of additional facilities has been planned. Given the considerable experience gained during the life of the project, it can reasonably be expected that the plan will be quickly realized. Table 19. Presence of washing stations in schools [Extrapolated results] Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: directors’ questionnaire The difference between treatment and control schools is stark. 74% of the rural treatment schools have such a station, and 86% of the urban ones. 80% of rural control schools lack such stations, that number reaching 86% in urban areas. The ubiquity of washing stations was corroborated by the evaluation team’s field visits. None of the treatment schools visited lacked a washing station. Indeed, in one case, the treatment school had a large number of washing stations outside the classrooms, but a school not in the project, but no farther distant than 200 meters had none. Despite seeing these simple to create facilities daily, the neighboring school did not manage to create washing stations. In addition, while the treatment school yard was clean, the neighboring school yard was strewn with papers, even the area immediately around the flag pole. This contrast between a treatment and another school suggests that an intervention, even if it involves simple changes in daily routine, such as the creation of washing station using readily available materials, can have important consequences in the students’ daily routine. The installation of these simple facilities can demonstrate that change is possible with a modest, but organized, effort. Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Yes, for all classrooms 10a 74,4b 0 85,7a 9,1 79,3 Yes, but no for all classrooms 10a 20,5a 14,3a 14,3a 10,4 17,8 No, for any classroom 80a 5,1b 85,7a 0 80,5 2,9 46 Mid-term Evaluation FFE | April 2017 Table 20. Health and Hygiene Conditions November 2016 [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Access to clean drinking water (%) of schools 37,5a 48,7a 28,6a 57,1a 36,7 52,4 Functional latrines (%) 70a 79,5a 100 71,4a 72,7 76 Sufficient number of latrines (%) 42,9a 51,6a 14,3a 40a 39,4 46,9 Girls have their own latrines (%) 25a 38,7a 14,3a 20a 23,7 31,1 Number of girls’ own latrines is sufficient (%) 71,4a 91,7a 100 100 73,5 93,8 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source. Directors’ survey November 2016 Generally, treatment schools enjoy better hygienic conditions than control schools. Rural and urban treatment schools enjoy better access to clean drinking water than control schools. 52% of the rural treatment schools have a sufficient number of latrines while that percentage is 43% for control schools. The differences between treatment and control schools can be large. For example, 40% of urban treatment schools have a sufficient number of latrines while that number is 14% for control schools. Only 20% of urban treatment schools have latrines reserved to girls, a number that is lower than the situation prevailing in rural areas. Treatment rural schools enjoy a higher level of facilities promoting health than control schools. One difference characterizes urban schools and that involve functional latrines which are more prevalent in control schools than in treatment schools. The prevalence of latrines reserved for girls is high in all cases, falling to a low of 71% among control schools while urban areas boast a 100% prevalence. In conclusion, FFE has significantly improved the school environment. Students eat well (the baseline study showed that nutrition was not totally satisfactory in January 2015), can adhere to satisfactory standards of hygiene, drink clean water and have access to adequate disposal of human waste. Some improvements remain to be realized, but the expertise that has been demonstrated strongly suggests that the, at the end of the project, FFE’s school environment will be appropriate. 47 Mid-term Evaluation FFE | April 2017 KNOWLEDGE ABOUT EDUCATION It will be impossible to sustain nearly universal schooling if parents are not convinced that sending their children to school will provide important benefits of many kinds, despite the sacrifices such attendance entails. Parents are unanimous in considering their children’s education important. The large numbers who came to the focus groups and participated in the conversations attest to that commitment, particularly given that November was a busy time for farmers. Table 21. Percent of parents who consider their children education important [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment 98,2a 99,2b 100 98,4a 98,5 99,0 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: Parents database, November 2016 Table 22. Percent of parents who can name benefits of education [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Only one benefit named 58,5a 58,5a 46a 71,4b 56,1 60,9 Two benefits named 35,4a 34,9a 46a 22,2b 37,5 32,6 Three benefits named 3,6a 5,6b 7,9a 4,7b 4,4 5,4 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: Parents database, November 2016 The baseline study reported that 39% of parents could name at least 3 benefits of education. That number is vastly superior to that of the mid-term evaluation, with 6%. The percentage who can name 2 is close to the baseline results (35%). The percentage of parents who can name one benefit of education is the same in rural areas treatment and control schools. However, a difference emerges between treatment and control schools in urban areas. For those able to list two benefits, again no difference emerges in rural areas, but urban parents whose child attends a control school list two benefits at twice the rate of treatment schools. The percentage of parents who are able to list three benefits of education is small. Are there differences between fathers and mothers’ ability to name the benefits of education? 48 Mid-term Evaluation FFE | April 2017 Table 23. Percent of fathers and mothers who can name number of benefits of education [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment Father Only one benefit named 57,3a 61,5b 43,4 62,1 54,7 61,6 Two benefit named 36,9a 32b 47,2a 31b 38,9 31,9 Three benefit named 4a 5,7b 9,4a 3,4b 5,0 5,4 Mother Only one benefit named 65,5a 51,9b 60a 79,4b 64,4 59,6 Two benefit named 27,6a 41,7b 40a 14,7b 30,1 34,1 Three benefit named 1,7a 5,6b 0,0 5,9a 1,4 5,6 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: Parents database, November 2016 In the baseline study, 39% of parents could list 3 reasons why education was important. 47% of fathers and 26% of mothers could list 3 advantages of education. In the mid -term evaluation, the numbers are much lower: 6% of fathers and mothers in the rural treatment zone vs 6% of fathers and 6% of mothers. Rural treatment schools had slightly higher percentages for men and women than control schools. The baseline study did not report the differences between urban and rural areas nor did it report the proportion of parents who listed only one advantage of education. Yet, parents clearly perceive the advantages education brings. These are economic, a job, and the consequences of the job, in particular being able to look after the parents when they get old. They also mention the ability to talk to others, to avoid or escape the problems illiteracy creates. Parents perceive education as beneficial to their child’s future, but also to themselves since, in that part of the word, children look after their parents when they get old. Many are willing to invest in their child’s education by volunteering and that is true of fathers and mothers. Some parents are well aware of the intrinsic advantages of education, such as being knowledgeable about many subjects, or being able to communicate (in French, probably). Nevertheless, the economic advantages dominate, but that is not unique to Benin. Some parents rely on one theme but with different specifics, find a job, be independent, help the parents later while others cite different kinds of advantages, develop the village, avoid teasing (because the child did not attend school). An examination of what fathers and mothers say suggests that the number of reasons is not crucial. If the reasons given by parents who cite 3 reasons why children’s education is important are examined, the same items appear as 1, or 2 or 3 among different parents. The number of reasons is not as important as the type of reason cited. Parents are keenly aware that, in today’s world, being educated is essential, to secure a job and to enjoy the benefits of literacy. Whether that is cited as number 1 or number 3 is less crucial than the fact that not sending one’s child to school is considered to be a 49 Mid-term Evaluation FFE | April 2017 dereliction of parental duty. Parents who refuse to send their children to school are said to be “bad” or “irresponsible” parents. Such parents violate what has become a societal norm. These statements are not simply moral, for they reflect a clear self- interest. The complaints parents expressed during the focus groups concerning the often deplorable learning conditions were based on the expectation that their children should be able to learn, i.e. to benefit from learning. Parents know that that promise is not being honored. International partners can be reassured that the benefits of education are known and understood. These convictions are put in practice: most parents are willing to spend scarce cash to pay school fees to insure that teachers will be hired when the state does not do so. Such action is compelling. COMMUNITY PARTICIPATION FFE considers parental involvement very important. Most schools in Benin have an APE (parent-teacher association), but their level of involvement in school affairs is highly variable. Ever since parental financial contributions were no longer required, the role played by the APE decreased, even more so when financial transactions were assigned to accountants who handled all school expenses in the district. Table 24. Percent of school with an APE or an AME [Extrapolated results] Rural Urban All (Rural + Urban) Control Treatment Control Treatment Control Treatment School with an APE 94,8a 94,4a 81a 95,2b 92,1 94,5 School with an AME 15a 70,1b 15,9a 76,2b 15,2 71,2 Note: Values in the same row and sub table not sharing the same subscript are significantly different at p<5% in the two-sided test of equality for column proportions. Cells with no subscript are not included in the test. Tests assume equal variances. Source: Parents database, November 2016 Many schools have both an APE, and an AME, composed of women only. Some women volunteer to be cooks and they are trained to discharge that role safely. Usually, they take turns because the work is demanding and these women all have many household obligations. They participate in the canteen management committee which plans menus and organizes the storage of food, the record–keeping and the daily outflow of supplies. The records are verified by project personnel, but, now that the project has gained experience, the routine activities are in the hands of the volunteers who wish they would be paid, but are not. The same holds for the management committee secretaries who maintain the records. Directors supervise and one teacher is also assigned to help keep records. WEI uses a tool, ERO that measures the progress the PTO or AME is making toward complete organizational functioning. This tool makes it possible to identify areas of organizational function that have reached a satisfactory level, for example the members know the organization’s rules, and those 50 Mid-term Evaluation FFE | April 2017 where progress is required. Members decide what score they should receive on a particular issue.23 The NGO representative leads a discussion to reach a consensus concerning an appropriate score. Then, these numbers are entered and the members can see where they stand concerning a particular issue, for example whether they should take more initiatives to promote education, particularly of girls. Then, specific plans will be made and responsibilities assigned to specific members. Some of the most important categories are as follows:  The leaders of the AME hold meetings  Documents are kept (minutes of meetings, for example)  The AME organizes activities to promote the children’s education  The AME maintains relationship with the PTO  The AME meets with the school director  The AME has a bank account in its own name to handle the association’s expenses  Members pay their dues in cash. Some of these items involve simple organizational practices, such as keeping records. Others pertain to their mission: contributing to their children’s education. By measuring where the organization stands in relation to the maximum score, members can understand what they should do to improve the functioning of their organization. Learning how to run an organization, learning how to hold meetings, to discharge the obligations of a particular position, treasurer for example, are skills that are useful in many kinds of endeavors and that will stay with the community’s members. This formalization of community life should make the attainment of other community goals easier to attain. 23 This discussion among community members strengthens the knowledge of organizational functioning and insures that the agreed upon improvements will be known to all participants. These discussions have a clear pedagogical benefit. 51 Mid-term Evaluation FFE | April 2017 IMPLICATION OF RELEVANT MINISTRIES AND OFFICES FFE requires the maintenance of a large network of partners that span all levels, national, regional, districts, and schools. The construction of latrines, or the improvement of a school’s water supply, involve municipal authorities who bring specific expertise to the project. Some of these contacts are with civil society, such as mayors; others with representatives of various ministries, such as health or agriculture. Many such contacts necessarily involve the educational system’s administrators at all levels, from the Minister to the school. Meetings with school directors are frequent because field agents representing FFE visit the schools at least once a month, more often if something is being built or changed. Exchange of information is frequent between the project’s leadership and the district administration. The regional director is also informed of what is planned and is invited to meetings when appropriate. These individuals are defined as essential partners who contribute to FFE’s mission. The help is mutual. For example, CCS seek to maximize enrollment. They can see that a canteen increases enrollment. Since FFE helps in attracting children to school, the CCS’ task is made easier. That applies also to the municipal engineer who wished to bring water to several hundred children but who did not have the means to finance the drilling of a well. These people can see some of their ambitions realized and that achievement surely motivates them. A symbiosis develops where each partner needs the other. It was noted earlier that FFE is well aware of, and sensitive to, the societal context within which it operates. Many members of the senior staff are nationals with the appropriate educational and professional credentials who bring a special sensitivity to local preferences. For example, the canteen management committees must raise funds to purchase some food items. This has three purposes: to insure that the community will contribute something to the children’s food, to enrich nutrition, but also to insure that the food will be to the children’s liking because it will include items they know prepared in a way that is familiar to them. FFE also maintains relationships with other organizations that share a similar purpose, such as Caritas and UNICEF. These organizations are active in the same zone and cooperate. They interact, either in the course of meetings to which several organizations have been invited, or because they need to coordinate an activity in an area where both are active for different purposes. FFE includes another American NGO with a long history of involvement in Benin. WEI is responsible for a number of activities, particularly teacher training and the strengthening of the AME, an area where it has significant experience. WEI worked directly with INFRE to prepare the administration of EGRA, a key component of the project since it measures progress on the SO1. WEI relies on local NGOs to carry out some of its activities under the supervision of its own staff. A review of the semester reports shows the extent of the contacts just described. Some of them are units of ministries, education often, but also agriculture (for school gardens) and health. Some of these contacts involve the organization of special committees, such as the one that regroups the Ministry or Education with the regional and local levels as well as the regional PTO. Some involve more local activities, such as the involvement of midwives to run a girls’ hygiene program in collaboration with the regional branch of the Health Ministry. 52 Mid-term Evaluation FFE | April 2017 Despite all these contacts and access to local, regional and national expertise, FFE sometimes calls upon more highly specialized competencies. This was the case in the construction of WASH stations in schools which was assisted by a specialist who came from CRS headquarters. CRS’ regional headquarters are located in Togo and the interaction with that part of the organization is frequent. The same applies to WEI whose activities benefit from the support of Boston headquarters. 53 Mid-term Evaluation FFE | April 2017 EVALUATION QUESTIONS 1. Relevance: The answers to the evaluation questions are designed to provide an understanding, a perspective, on the project. Relevance:  To what extent are canteen activities equally relevant for both urban and rural schools? According to the person responsible for the Ministry of Education’s service charged with the supervision of food programs, lunch programs are available only in rural schools because of the distance children have to cover to reach the school. If they go home for lunch, they often do not come back and miss the afternoon’s instruction. In addition, poverty is more prevalent in rural areas. Borgou and Alibori are arid and poor. The need exists in both rural and urban areas because of the nutrition issue. If a choice must be made, rural schools should be prioritized. The evidence in Benin shows that attendance increases when a lunch program is introduced. Part of the reason for the increased attendance is that food is provided which reduces the parents’ expenditure. Food also partly compensates the loss of the students’ labor when they go to school since children work in the fields at an early age. Attending school represents a significant opportunity cost for families, particularly in rural areas that are usually poorer than urban ones. Mothers also work in the fields and need the assistance of their older daughters for domestic activities. A food program constitutes a significant incentive for children and for parents. The interviews with parents made that fact abundantly clear. The need for school lunch programs exist in both areas because the areas are characterized by very high rates of poverty. School lunch programs are particularly essential in rural areas because of the distance some children must cover and also because of the high poverty level.  To what extent is the construction/rehabilitation of a limited number of teacher housing relevant to increasing teacher attendance in the whole project’s intervention area? The construction of teacher housing has not begun. This is because parents must provide 20% of the cost (often by contributing labor). Parents decided that repairing classrooms was a higher priority than teachers’ housing. It is commonly believed by education officials that housing difficulties make the rural postings of teachers problematic. A few communities have undertaken to grow corn in a community field, and to sell the harvest to subsidize the school director’s rent. Such a practice assumes that housing is available. If the GOB made the required number of teachers available, communities might be convinced to contribute to teachers’ housing.  To what extent does the project meet community and government priorities? Interviews with parents reveal a high level of satisfaction with the project’s activities and parents in the control schools often ask why their community is not eligible. The project thus corresponds to a community priority because they could not purchase such quantities of food. 54 Mid-term Evaluation FFE | April 2017 The willingness of parents to volunteer shows how important education is to parents. Canteens improve attendance and, without attendance, there would be no learning. The budget available for government- sponsored canteens has not increased in several years despite the fact that new canteens have been created. Thus, what is available to each canteen is very small. Attendance for both genders is a government priority as stated in PDDSE, for example. Communities have been forced to recruit teachers with their own funds. The insufficient funding of basic educational needs, teachers for example, suggests that the GOB cannot fund a lunch program in most rural areas.  To what extent does the project align with and complement programs funded by other donors and/or managed by other international nongovernmental organizations (INGO) or local organization as well as with local, national and regional policies? UNICEF, among other organizations, is active in this geographical area and engages in similar activities. Coordination occurs in the selection of the specific communities where various organizations will be active. International assistance takes many forms, building of teacher training colleges by Japan, France and Denmark, and support for school lunch programs by the World Bank. A coordinating committee exists.  Are stakeholders (PTAs, mothers’ associations, teachers, and local authorities) satisfied with their participation in the project? Why or why not? PTAs and mothers’ associations praise the project without being prompted. Teachers believe that the lunch program contributes to learning. Parents repeatedly lament the poor learning conditions, particularly the fact that there are not enough teachers. The same applies to local education officials. The very high level of participation in focus groups during the field work testifies to the communities’ commitment to the project’s success. The effective participation in the running of the lunch program demonstrates the importance of local commitment. The lunch program and its success has created several consequences: new activities can more easily be launched, for example the savings program or the mentoring of young girls by older ones. A cadre of responsible and experienced community leaders is emerging among men and women. These individuals will play key roles in harnessing the resources needed to continue the activities. AME can be expected to produce women leaders.  How well does the project complement and link to activities supported by other donors at the local level? The selection of the treatment schools originally occurred because some communities that should have had a lunch program did not have them. That is according to the head of the school food program at the Ministry who has been in her position for at least 5 years. Thus, FFE filled a significant gap. UNICEF and CRS cooperate. In addition, donors to education in Benin are grouped in a committee with one organization (the Swiss Development Agency) being the lead 55 Mid-term Evaluation FFE | April 2017 organization this year. The flow of information is thus insured between organizations. This PTF committee works with the relevant government agencies. 2. Effectiveness:  To what extent have the objectives of the project and the yearly benchmark indicators been achieved, or to what extent are they likely to be achieved? The objectives under the project’s control have been largely achieved, with minor gaps, such as the number of latrines that will be filled in the near future. The students are better nourished and practice good hygiene to a significant extent. However, the project operates in a zone that lacks teachers. Communities have to raise funds to hire teachers, many of them without the appropriate credentials. Even when they have the credentials, their ability to teach effectively under harsh conditions (supplies, books, appropriate level of cognition when students enter the grade) is limited. The educational component of the project, all second grade leavers being competent readers, has never been achieved in Benin in the past. The project undertook an ambitious objective at a time when the GOB seems unable to fund normal operations. The involvement of INFRE reveals some serious lack of basic competence in the government services that were to have been strengthened. Frequent rotations of teachers and supervisors is puzzling in a resource poor organization. Frequent rotations among teachers and supervisors reduce the effectiveness of the program.  How effective is mobilization of community counterpart and to what extent does the implementation of this measure impact the effectiveness of the project? Community mobilization is high. Volunteers have been trained to prepare meals in a healthful manner and records are kept by volunteers. Parents also volunteered to build the kitchens and the storerooms. Without such volunteers, the project would not operate. The teaching staff is also involved by helping with record keeping. For example, a method has been developed so that the number of children present (and who will eat lunch) is quickly done and transmitted to the kitchen without interfering with instruction. School directors are also key actors. The community mobilization contributes heavily to the success of those parts of the project under its control (nutrition and health in particular). The community makes an essential contribution to the effectiveness of the project not only because meals are available on time but also because members of the community develop important organizational skills and health knowledge while helping the community’s children.  To what extent have Take Home Rations led to improvements in students’ attendance so far, and/or to what extent are they likely to improve students’ attendance in the future? Attendance was high when the project began and, therefore, an increase in attendance rates is difficult to document. The take home rations act as an additional incentive that rewards a very high rate of attendance. It should be noted that school attendance carries significant opportunity costs for the family, particularly in the case of girls who play an important role in domestic work. Thus, an incentive like the Take Home Rations sustains high attendance. Is the management system for the project effective? 56 Mid-term Evaluation FFE | April 2017  Which strategies have been (or should be) put into place to effectively monitor teachers’ attendance? Has project implementation been effectively monitored? How well has the M&E mechanism in place helped the implementation of the project, and what improvements could be made, if any? Officially, teachers’ attendance is high. A system of incentives linked to student learning could be considered to insure teachers’ attendance. The M & E mechanism is well run. Requests by the evaluation team for information were quickly honored, indicating that the information was readily available. The M & E relies on quantitative and qualitative data. The staff is expected to visit each school at least once a month and that provides a familiarity with the particularities of each school and close working relationships between the school staff and the volunteers develop. The information thus collected is transmitted to project headquarters and acted upon. The working atmosphere in project headquarters is business-like, yet friendly. Easy communication and cooperation between co￾workers appear to be the norm. An experienced outsider becomes quickly aware that the staff understand the project’s mission and that it is committed to it.  To what extent has the project collaborated with other stakeholders, and to what extent has it taken their experiences into account? Project staff are well informed on stakeholders’ activities in their zone. Staff members are cognizant of the project’s work plan and schedule. School gardens, SILC, mentoring of young girls by older girls, WASH stations have been used in other projects and are being implemented in FFE, showing that relevant knowledge is being used. These experiments are promising. WEI has substantial experience in training AME in the North and that experience is being used in FFE.  What is the level of SILC process implementation so far? Is the SILC process likely to improve the targeted population’s resilience and to what extent could this contribute to improve their ability toward meeting the costs of student’s education? The SILC component began relatively recently. It has experienced very rapid growth because it fills an important need. The economic benefits of this activity contribute to the economic vitality of the community and it also generates income for the families, income that can be used to pay educational expenses. Almost $ 90 000 of loans have been made and $ 4 700 have been shared between members to handle emergencies, usually unavoidable expenses that would have to be covered by borrowing. The income is used for a wide range of family needs, including medical expenses. Learning to pool resources and to honor commitments contributes to the self￾empowerment and to the communities’ development. Given the trend toward a partial funding of education by families, the ability of SILC to generate resources will be even more important in the future.  To what extent are the WASH strategies likely to remove gender-linked barriers? WASH stations are ubiquitous in the project’s schools and children use them. 95% of rural experimental schools have them compared to 19% in control schools. 57 Mid-term Evaluation FFE | April 2017 Latrines reserved for girls exist, but not always in sufficient numbers. Plans exist to build or rehabilitate more. WASH stations are essential if girls are to complete primary schooling. This area is heavily Muslim. In the past, girls used to drop out of school often at the end of the 4th grade. This is no longer so prevalent and WASH stations contribute to girls’ remaining in school. It should be noted that, even if parents were willing to have their daughter attend a school without separate facilities, religious authorities could discourage such behavior very strongly. The presence of separate latrines and WASH stations makes it possible to organize girls’ hygiene programs. These require separate facilities. 3. Efficiency:  What indications are there that the activities thus far have been cost efficient? The evaluation team did not examine financial information. The field work consumed all the time available. An assessment of the cost efficiency would require information on the alternative methods for completing activities, salaries paid to similarly qualified staff in other projects, for example. All observations suggest that the staff is cost conscious and uses sound judgment to minimize cost. For example, car- pooling takes place to reduce fuel expenses, travel is organized so as to minimize the distance travelled. The staff seems to work hard, thus reducing the need for additional personnel.  Have objectives been achieved on time? If not, what were the obstacles? The achievements of the components under the project’s control over a period of slightly more than one year are remarkable. These include the many physical facilities (kitchens, storage rooms, distribution mechanisms) and training (cooks, record keeping, PTOs and AMEs). The major obstacle mentioned on several occasions above, is the deleterious condition of the educational system which makes the attainment of the SO1 impossible. The TOR describe the educational system as “dysfunctional”. That description remains valid and the learning conditions elicit bitter complains by parents and local officials.  Has the project been implemented in the most efficient way possible compared to alternatives? In particular is there another way to get the food management committees more efficient considering their literacy level? Food management committees have been trained and perform their duties without remuneration. Members perform a highly valuable service, not only because food is safely prepared, but also because the management committee uses local resources to adapt the available supplies to make the meals more palatable to children who are used to food being prepared in a certain way. This insures that meals are highly popular with the children. The project staff maintains close relationships with the community members and jointly work out appropriate solutions. The reliance on members of the community to carry out essential services is efficient and effective because the costs are low and the skills learned in the process substantial. 58 Mid-term Evaluation FFE | April 2017 Given the objective of strengthening governmental structures, the choice of INFRE as a partner to become familiar with EGRA was initially reasonable. However, that structure lacks the specialists required and no new appointments to fill the gaps seem likely in the near future. In addition, EGRA is very costly and it is unlikely that, in the near future, the Ministry will have the resources to use such an instrument. Monitoring could be done locally, a practice that would make it possible to organize small workshops on very specific issues with a dozen teachers. That would require a degree of local autonomy that does not exist, despite GOB’s claim that decentralization is a priority. So far, 52 CI and CP teachers have been followed up. When will the others be trained? The frequent rotation and the reliance on community teachers who are poorly remunerated and who serve without contract makes it difficult to work with teachers over a period of time. Frequent rotations of teachers and supervisors mean that the benefits of the training “disappear”. New personnel comes, but these individuals have not been trained. In many respects, FFE has been forced to rely on the centralized approach that prevails in Benin. That is not, in this case, the most effective approach.  How has the project responded to date in terms of key community stakeholders’ requests for material incentives? Is there a better approach to do this? Interviews with the kitchen staff and with record keepers revealed that they had asked to be remunerated. All of them indicated that such requests had not been honored and that they continue to serve without pay. They do receive some in-kind supplies, such as oil, but the quantities are small. This solution seems satisfactory. The message that “these are your children” could be emphasized and volunteers could be praised and given prizes on a regular basis. There is a problem of “free riders” (some parents contribute little). That is an area where important local leaders could contribute their influence.  How has the project responded to any internal and/or external factors that have hindered the efficient implementation of project activities? The improvement of reading instruction has not yet taken place. This is due in part to the condition of the educational system. The contractor responsible for that component of the project used a “top down” approach which was, in retrospect, ineffective for several reasons: the number of people trained was too small, the adequacy of the training program and its organization questionable. That choice of “top down” was by no means capricious, but motivated by a desire to share know-how with the official training unit. That unit, however, by the Director’s own admission, lacks expertise and resources. One option would have been for the project to provide the missing expertise. However, that would have negated an important component of the project: to improve national competencies. The choice of INFRE led to delays in the implementation of training programs, and to the usage of a training program that did not take the teachers’ existing competencies into consideration. The training program was supposed to fill too large a variety of training needs. Teachers need assistance in the classroom on a very frequent basis given their level of training. That is not available because there are few supervisors. Only one training session was organized and its outcome was evaluated three months later, but that evaluation was insufficiently systematic. The reliance on a national agency has turned out to be ineffective. A local approach could be tried. The approach should 59 Mid-term Evaluation FFE | April 2017 also be pragmatic and realistic, i.e. take account of the teachers’ skill levels. Limited interventions in the classroom with frequent evaluations of the outcomes combined with dissemination of positive results might have produced better results. However, the teaching conditions and the absence of incentives for teachers make successful interventions difficult. 4. Sustainability:  What are the major factors that are likely to influence the achievement or non-achievement of project sustainability? The project fills an important need and the children are ready to learn in FFE schools. However, it is unlikely that such quantities of food could be purchased either by the communities or the GOB. Other school lunch programs in Benin receive funds from international partners which make it possible to purchase supplies locally. The communities are simply too poor to be able to do so on their own. Valuable skills have been learned, and will be perfected over the time left in the project. The skills developed in the course of the project may be sufficient to manage the reception and distribution of food if USDA should continue to make it available to the GOB. Partial sustainability is possible, not only because valuable skills have been acquired, but also because several experiments point to greater autonomy: school gardens could become fields, SILC could benefit schools, and labor could continue to be donated. However, if the GOB cannot provide some effective assistance to rural schools, communities on their own will continue to require international help. Benin is characterized by a rapidly growing population which implies large primary school enrollments in the future.  How do government’s capacities, policies, procedures, and priorities contribute to sustainability? The GOB is currently not financially capable of purchasing the food. Whether that lack of capacity stems from a lack of financial ability to collect revenue, to insufficient commitment to education or to a poor usage of available resources is unclear. As observed above, the total national budget for school lunch programs has been stagnant for several years. The policy favors rural and border areas (in part because neighboring countries are said to have generous school lunch programs which lead to Benin children attending school in the neighboring country). The resources needed to operate schools seem inadequate at the moment, as can be seen in the fact that many schools do not have the required number of teachers. Whether this results from a lack of resources or an ineffective usage of the resources is unclear. Officially, primary education is free and parents complain that they must cover the cost of community teachers. The funds necessary to pay these teachers, despite a very low level of pay, are often not raised and teachers leave the school because they have not been paid. That situation necessarily leads to children not learning and abandoning school. The local officials who witness the benefits of the project are enthusiastic. However, they experience every day the consequences of inadequate resources.  What strategies should be used to obtain long-lasting support from communities and local/central administration that extends beyond the life of the project? Namely support for: 60 Mid-term Evaluation FFE | April 2017 o Extension of the school gardens into community fields to sustain the canteen This is a promising initiative. However, the amount of food required far exceeds what a school could grow. Scaling up school gardens could be tried, but that would require a gardener and that person would require supervision. Experimentation should be part of future plans. o permanent availability of water in a dry, semi desert area The viability of gardens is highly dependent on the availability of water, and large quantities of it. Wells can be drilled, but such drilling can be costly. Some funds will have to be available to maintain the pump. The ability of the water table to sustain high rates of water consumption is unknown, but it could be limited if several years of drought occurred. This area might benefit from drip agriculture which requires small quantities of water. That is where CRS could play a useful role by making international expertise available to determine whether new forms of agriculture could be successfully introduced. Demographic growth presents a huge challenge. o maintenance/servicing of all infrastructures constructed/rehabilitated by the project It is conceivable that, with appropriate training, the community could maintain what has been built. WEI has experience in training community members in building maintenance. In the past, some international partners have helped with maintenance costs. The practice of paying fees to create and sustain a fund that will make maintenance possible is beginning to emerge. However, these are subsistence economies where cash is rare.  When FFE is over, are the APEs/AMEs likely to continue applying the organizational competencies the project has passed on to them? The APEs/AMEs are well implanted and active. The available evidence suggests that parents understand the importance of organizing the community, particularly the schools. The importance given to education leads to the reasonable expectation that APEs/AMEs will continue to play their role. However, the practice of having the school budget under the control of a financial manager working at the commune level (and not at the school level) may reduce the willingness of school board members to devote their time to school affairs, particularly if they suspect, correctly or not, that the school managers do not manage the funds honestly. As educational level rise, it might be possible to organize training programs and activities for local leaders, women in particular. The GOB could define again its policies concerning decentralization so that communities would have an incentive to run their own affairs. 5. Impact:  Which issues should the project team particularly focus its efforts on in order to achieve greatest impact? One of the most significant contribution the project could make, now that children are essentially ready to learn, would be to run some experiments with local official support in order to determine what determines learning to read. These experiments could take several forms: limited classroom enrollment, more time allotted to teaching instruction, more varied reading material, hiring tutors to have children practice, relying more on play, using recess time to reinforce what has been learned in class (children are free from about 1230 to 3 PM), possibly 61 Mid-term Evaluation FFE | April 2017 while playing. These experiments, even if only a few of them were successful, would show the benefits of experimentation. The development of teacher assessment in order to match training needs with the level of skills ultimately needed would contribute to the entire system as well. The evaluation of the teacher training workshop mentioned two exceptional teachers whose students could read well. What do these teachers do? Their practice should be recorded in great detail so lessons could be learned. They might also become “demonstration teachers” who would coach other teachers. Action in the classrooms could improve the current situation. Videos of their performance could provide useful models for other teachers. Thus, in order to achieve the greatest impact, the project could seek to achieve SO1 in a limited number of schools in order to learn what works best. These lessons, even negative ones (what does not work), would be communicated to local and national authorities for dissemination and refinement. In this way, the project could have a significant impact on the future development of the educational system.  How likely is the project to achieve its strategic objectives and contribute significantly to its intended purpose? Under current conditions (resources and teachers’ competence), the likelihood of having second graders reading at the appropriate level is extremely low. More modest outcomes might be identified, concentrating meaningful efforts on the first grade, so that students would be better prepared for the second grade. What is the contribution of the radio broadcasts and the listening groups in the awareness raising on the importance of education? The evaluation team had little opportunity to evaluate this aspect of the project. Parents mentioned listening to radio programs, but they seemed rather vague about what had actually taken place. Other evaluations have shown that radio broadcasts are highly effective in raising the awareness of the importance of education. Radio could be used to improve reading. Students could receive short texts that would be read over the radio, perhaps with some play component.  To what extent are the activities and outputs of the project to date consistent with the intended outcomes, effects, and impacts? So far, getting students ready to learn (health, nutrition hygiene), have been largely realized. The food is available in the schools and meals are prepared on time. The children eat well. Management committees perform their obligations. Latrines are not yet in sufficient number, but plans are to build them. Water is more readily available. Children are healthy and relatively few miss school. However, they do not read at close to the appropriate level. This outcome is primarily the result of the poor learning conditions. The project could probably have been more aggressive in implementing some remediation. However, it is not the project’s responsibility to hire teachers, to train them, or to build classrooms. The project can only support what should already be present. Given that working with senior officials has produced little, a different approach could 62 Mid-term Evaluation FFE | April 2017 be attempted that would rely far more on local personnel: supervisors, school directors and teachers, possibly additional persons as well. If the project were able to promote experimentation and a critical and analytic approach to the learning situation, the lessons could benefit the entire educational system. These lessons would constitute a lasting FFE legacy. 63 Mid-term Evaluation FFE | April 2017 Recommendations The recommendations take the form of possibilities to be explored and discussed among the various actors who will decide which suggestions are the most promising. The implementation of these suggestions will require an experimental frame of mind on the part of the team. In turn, such a frame of mind demands a simple but rigorous M & E that will make it possible to abandon some practices if they should turn out to be ineffective, i.e. produce no additional learning. At the end of FFE, some resources should be allocated to the production of a document that systematizes the experiments and their outcomes. This document would contain materials, description of activities, successful training mechanisms and copies of the videos. MODIFY THE OBJECTIVE The new SO1 would specify that students should reach the level expected at the end of the first grade. All activities would focus on one grade only. If such a modification were impossible, the first grade could become a high priority. The second grade would benefit from interventions requiring fewer resources. This concentrated effort would take place at the classroom level. Thus, the focus would be on learning resources, using such resources effectively, and support for teachers, particularly in the organization of work groups. The goal is to help teachers select among the most effective newly available resource. This assistance would require hiring a small team of experienced and effective teachers, probably recently retired, who would act as a support team. The teacher responsible for the class would have to agree to such intervention and so would the school’s director. Monitoring would have to be organized in order to ascertain whether the approach is effective. The following activities involve teachers. Several of these activities will be assigned to the pedagogical team). These suggestions could be used in the second grade if the decision is to retain the original objectives. 1. Identify currently successful first grade teachers and identify what they do in the classroom that produces results. What are the best results, i.e. what goals are reachable in the context? 2. Determine whether these practices can be generalized to other classrooms. What conditions must be met? 3. The performance of effective teachers could be video-taped and shared with other teachers in a small group setting. Generally, foster the development of teaching communities by increasing professionally relevant communication between teachers (identify the problem faced by a teacher and discuss solutions with colleagues). 4. Effective teachers could be eligible for meaningful awards based on performance. 64 Mid-term Evaluation FFE | April 2017 5. Train teachers on the usage of groups in the classroom so students will teach each other by practicing their skills. Such activity reinforces previous learning. 6. Identify the types of difficulties students experience when learning to read. This step may require the participation of an external reading expert. 7. Identify how these difficulties have been, or can be, overcome (more time devoted to reading. More books, more practice, more time spent on French before proceeding to reading?). 8. Review teaching resources to ascertain whether they are effective and whether, if the resources are effective, they are used effectively. 9. Produce a guide concerning effective usage of resources and have teachers practice effective utilization in their own classroom with assistance (pedagogical team). 10. Develop materials to complement what is already available (exercises, group work, games) 11. Involve first grade teachers in the innovation process. Invite teachers to share the materials, activities they produced. Establish a prize for the 5 best examples. The winners could participate in decisions for later awards. 12. Have master teacher (successful teachers), experiment with the hiring of assistant teachers who would work with small groups to complement what the regular teacher does in large classrooms. 13. Organize local workshops, similar to UP, devoted exclusively to solving reading problems in the first (and second?) grades. Master teachers would be the lead participants. Videos of effective practices could be shown, problems discussed. 14. Encourage, if authorities approve, the emergence of grade specialists who would be rotated to another grade if they request it. This would foster the development of specific expertise. Several types of intervention are suggested concerning reading: 1. Adapt EGRA to local conditions by simplifying the instrument, making it easy for teachers to use it as an evaluation tool. This should be done by reading specialists. Rely on a pragmatic approach that builds on what exists: Why do students do well on the first components of the test, is it because it takes the entire year to complete it (i.e. considerable time is required to achieve early learning), or is it because these early parts of the program are organized differently? 2. Is the reading program sufficiently progressive, i.e. are there enough steps? 3. Is the time allotted to study reading sufficient? If reading is truly an essential skill, then that importance should be reflected in the time allotted to it. 4. Is the opportunity to practice skills sufficient? 65 Mid-term Evaluation FFE | April 2017 a. In the classroom? What is the time allocation for learning as opposed to practicing? b. Organize a short summer school before and after the completion of the first grade to increase the opportunity to speak and read French. Classrooms are empty during that time. c. Organize a similar summer school for students who completed the second grade. d. Can radio programs be used to get students to read interesting stories? This could take the form of reading clubs that would meet once a week. Small booklets with the text would have to be available. 5. Are the supplemental readers distributed by WEI interesting to the intended audience? In other words, are these readers something 7 year old African children who do not know much French would be motivated to read? Could national languages support the learning of French? 6. Ask international experts familiar with reading programs in Africa to examine the reading materials, including those used in training teachers. Such an examination could be done at a distance. These recommendations will require the creation of a small team composed of a reading specialist and of a coordinator committed to the importance of SO1. This team could begin its activities by focusing on a small number of schools in order to identify what works in the first grade (possibly in the second as well). If FFE could, upon completion of the project, leave behind a series of effective teaching practices, its legacy would be highly significant for thousands of students. This legacy would be transmitted to the future teachers who will be trained in the ENI. A relatively short follow-up to train ENI trainers would have to be planned. These ideas focus on the classroom. However, other activities at the community level could continue. Supporting teachers, organizing reading clubs, insuring that parents are involved in some school activities and in important decisions will strengthen the connection between the community and the school. SO2 should continue since it is designed to create students who are “ready to learn”. That objective remains essential. SO2 conditions the attainment of SO1. LOBBYING FFE could usefully discuss the classroom and teacher situation with the Ministry. It might be possible to reach an agreement that would commit the GOB to improvements in the current situation. 68 Mid-term Evaluation FFE | April 2017 Appendices SEVERAL SUGGESTIONS CONCERNING READING It is essential to systematically train supervisors and teachers in the strategies that make it possible to learn the basic approach to reading. We use the following definition of reading: to discover the meaning of written words by relying on various strategies that rely on the usage of what is already known. In order to be able to read and to understand, thus to analyze the information that is in the text he is trying to decipher, the learner must call upon prior knowledge. This knowledge can have several sources: 1. First, the knowledge that is related to the topic of the text (cultural knowledge and what the learner already knows on the basis of his own experience) 2. Then, the knowledge of the language: a) The vocabulary or the knowledge of the individual words in their spoken version and in their written version. b) The syntax, or the way in which words are placed in a sentence and the sentence in a text by looking at punctuation, capital letters, beginning of a sentence, identification of the gender and of the singular and plural of the words in the text. c) Discovery of the connection between what is written and the pronunciation of words. d) Semantic, or the meaning to give to a word, in relation to the meaning that emerges from the context and from the relationship between words. 3. Finally, it is also possible to use what the reader knows about the physical environment in which the reading session takes place, the time of day, the people who are involved in the reading situation. The learner mobilizes all he already knows and he will use different strategies to recognize words in their context to understand what is written. In a reading situation, words are always part of a context, be it linguistic or not. Teacher training must take these principles into account and use the following strategies:  Some words can be recognized through a global approach, i.e. the reader has seen this word before and recognizes it.  Linguistic context: The reader discovers words in terms of the linguistic context. The word is included in a sentence. What might the unknown words say? Discovery of the word is made possible by the words that come before and those that come after the one that is already known. The learner uses this to make sense of the sentence.  Word scan be split into smaller units that are meaningful. 69 Mid-term Evaluation FFE | April 2017  Split the words that are unknown into syllables. This is a phonetic analysis of the word in order to discover the pronunciation of a word by putting together what is written with the pronunciation of the word;  Guessing of the word by partially analyzing the word, without taking account of the context or of the specific letters in the word.  Analyze the strictly visual aspects of the word: its length, the kind of letters, for example the dot on the i, the accents.  Nonlinguistic context. For example, looking at the pictures in the book;  Ask for help from someone. A reader does not use only one strategy to decipher a word, he can begin reading a word by trying to decipher and then figure out what comes next by taking the context into consideration. In short, in order to decipher a word in its context, a reader must have strategies and be able to use them, to combine them and to ensure that the word, once identified, is the correct one. In short, the reader checks on what he has done.  A training program must necessarily take account of what teachers need. Training is a dynamic and complex system.  This will require that CCS and CP learn how to establish what the training needs are.  These needs lead to the elaboration of a training program;  This program to the development of training modules; Some of these can be developed as self￾study guides.  Monitoring of and support for teachers will need to be put in place;  Evaluation of students is essential. Teachers should be trained in remediation; The teacher acquires a tool kit.  The different mistakes learners make should be labeled and made into categorized. This will make the acquisition of remediation tools easier.  Hypotheses should be proposed to explain these errors. The hypotheses should be tested.  A set of remediation measures should be put in place.  Experiment with action research  Hire recently retired teachers whose competence has been documented in order to assist CP and CCS in the classrooms (this is not the project’s responsibility).  Assign a sufficient number of teachers and insure that the number of students in a classroom 70 Mid-term Evaluation FFE | April 2017 never exceeds the official limit.  Hire a specialist to manage the training program and a reading specialist to carry out diagnostic analyses of the reading issues facing teachers in this particular context. It is likely that a person capable of diagnosing the major issues will have to be recruited internationally.  The outcomes of the steps taken should be regularly evaluated.  These steps will be more easily taken if the actors enjoy a sufficient degree of autonomy to apply what they have learned. Then, teachers and supervisors will share their new methods and the sharing will make it possible to benefit from other teachers’ inputs.