This report was prepared for Nascent Solutions, Inc. by the et4d team, Dr. Brenda Sinclair, Dr. Michel Rosseau and Ms. Isabelle McMahon-Duston. USDA McGovern-Dole Food for Education and Child Nutrition Program USDA FFE-631-2015/001-00 Advancing Literacy through Good Nutrition (ALIGN) project – Cameroon Final Evaluation Report Submitted July 2018 Et4d ALIGN Cameroon Endline Survey Report 2 FY 2015 McGovern-Dole Advancing Literacy through Good Nutrition (ALIGN) project – Cameroon Final Evaluation Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: [FFE-631-2015/001-00] Funding Year: Fiscal Year 2015 Project Duration: FY 15- FY 19 Implemented by: Nascent Solutions Evaluation Authored by: et4d McMahon-Duston, Isabelle Sinclair, Brenda Rosseau, Michel DISCLAIMER: This publication was produced at the request of the United States Department of Agriculture. It was prepared by an independent third-party evaluation firm. The author’s views expressed in this publication do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this document can be made available by contacting fas.monitoring.evaluation@usda.gov Et4d ALIGN Cameroon Endline Survey Report 3 Table of Contents Executive Summary ..........................................................................................................6 Purpose and Key Evaluation Questions..........................................................................12 Evaluation Methodology..................................................................................................12 Question 1 Results and Discussion: What outcome-level changes are attributable to the project? ...........................................................................................................................20 Question 2 Results and Discussion: For each selected indicator, how much progress has been made against the baseline targets? .......................................................................43 Question 3 Results and Discussion: Qualitatively, what changes do stakeholders associate with ALIGN’s programming?...........................................................................46 Improved literacy .............................................................................................46 Question 4 Results and Discussion: In which programming areas, do stakeholders anticipate that ALIGN’s programming is sustainable? ....................................................51 Conclusions.....................................................................................................................54 ANNEX A: EGRA Scores by Student Characteristics .....................................................56 ANNEX B: Factors Associated with Stallings Observations............................................58 ANNEX C: EGRA Instrument..........................................................................................61 ANNEX D: STALLINGS Instrument ................................................................................74 ANNEX E: DDS Instrument.............................................................................................80 ANNEX G: Biographical summary of evaluation team ....................................................84 Figures Figure 1: EGRA results by reading category.......................................................22 Figure 2: Proportion of students meeting literacy indicator by gender ................23 Figure 3: Proportion of students meeting literacy indicator by geographic location .....................................................................................................................24 Figure 4: Proportion of students meeting the indicator by school type: private vs. public ...........................................................................................................25 Figure 5: Project and comparison mean fluency scores (correct letters/words per minute).........................................................................................................27 Figure 6: Project and comparison mean comprehension scores (percent correct) .....................................................................................................................27 Figure 7: Proportion of students scoring zero on each EGRA subtask ...............28 Figure 8: Mean scores by gender .......................................................................29 Figure 9: Mean scores by school-age of student ................................................30 Figure 10: Mean scores by father literacy ...........................................................31 Figure 11: Mean scores by mother literacy .........................................................31 Figure 12: Proportion of teachers meeting quality of instruction indicator...........33 Figure 13: Stallings results by teaching activity ..................................................34 Figure 14: Percent of classroom time devoted to each active teaching activity ..35 Figure 15: Time devoted to classroom management by activity .........................36 Figure 16: Percent of classroom time devoted to off-task behavior ....................37 Figure 17:Proportion of classrooms meeting student attentiveness indicator .....38 Figure 18: Pupil off-task behavior by activity.......................................................39 Figure 19:Food group consumption ....................................................................42 Figure 21: Students using donated oil cans as trash bins...................................48 Figure 23: Predictors of improved quality of instruction ......................................58 Et4d ALIGN Cameroon Endline Survey Report 4 Figure 24: Predictors of student attentiveness....................................................58 Tables Table 1: Outcome-level changes of ALIGN project...............................................8 Table 2: Endline results against baseline and targets of selected output indicators .......................................................................................................................9 Table 3: Question 1 Data Collection Instruments................................................12 Table 4: EGRA minimum score to achieve literacy indicator ..............................13 Table 5: Stallings categories of active instruction ...............................................13 Table 6: Endline results for selected output indicators........................................15 Table 7: Sampling for quantitative methods........................................................16 Table 8: Percentage of students absent in Class 3.............................................20 Table 9: Definition of MGD performance Indicator for improved literacy.............21 Table 10: Regression coefficients for fluency and comprehension .....................32 Table 11: Predictors of pupil off-task behavior....................................................40 Table 12: Proportion of pupils consuming a minimum acceptable diet ...............41 Table 13: Proportion of pupils meeting the minimum meal frequency.................41 Table 14: Proportion of students meeting food group diversity requirements. ....41 Table 15: Mean number of food groups and meals consumed ...........................42 Table 16: Predictors of pupils who have a minimum acceptable diet..................43 Table 17: Endline results against targets and baseline for outcome indicators...44 Table 18: Endline results against targets and baseline for selected output indicators .....................................................................................................45 Table 19: Annex A: EGRA Mean Scores and Zero Scores by Student Characteristics .............................................................................................56 Table 20: Teacher and Classroom-level factors associated with pupil off-task behavior.......................................................................................................59 Et4d ALIGN Cameroon Endline Survey Report 5 List of Abbreviations and Acronyms ALIGN Advancing Literacy through Good Nutrition CI Confidence Interval DDS Dietary Diversity Survey EGRA Early Grade Reading Assessment FA Field Agent M&E Monitoring and Evaluation MGD McGovern-Dole Food for Education and Child Nutrition Program Nascent Nascent Solutions Inc. PTA Parent-Teacher Association SO Strategic Objective Stallings Stallings Observation System USDA United States Department of Agriculture Et4d ALIGN Cameroon Endline Survey Report 6 Executive Summary The USDA-funded Advancing Literacy through Good Nutrition (ALIGN) project is a McGovern-Dole Food for Education and Child Nutrition project. The $12 million project is implemented in the Bui division of Cameroon by Nascent Solutions Inc (Nascent). Its primary goal is to improve the literacy of school-age children in 92 primary schools through improved: quality of literacy instruction, student attentiveness, and student attendance. Its secondary objective is to increase the use of positive health and dietary practices through improved knowledge of hygiene, safe food preparation, and nutrition, as well as increased access to clean water and preventative health interventions. The health objective will also be achieved by increasing the consumption of a minimum acceptable diet through provision of a daily lunch and periodic take-home rations using food commodities provided by the USDA. The ALIGN project, which began in July 2015, ends in June 2018. This report presents the results of the final performance evaluation, which provides final measures of ALIGN’s outcome-level progress relative to the 2015 baseline values and project targets. The endline results are based on data collected between April and May 2018 in 89 schools. A total of 354 teachers and 1,123 Class 3 children distributed in 43 project schools and 46 comparison schools participated in the endline surveys. Evaluation Questions The final performance evaluation aimed to respond to four key evaluation questions: 1. What outcome-level changes are attributable to the project? 2. For each selected indicator, how much progress has been made against the baseline targets? 3. Qualitatively, what changes do stakeholders associate with ALIGN’s programming? 4. In which programming areas do stakeholders anticipate that ALIGN’s programming is sustainable? Evaluation Methodology The final performance evaluation is an impact evaluation. Using difference-in-difference analysis, we compare the gains achieved by the treatment group with the comparison group from baseline to endline to determine the overall treatment effect (e.g., impact) of the project. To respond to the evaluation questions, we used a mixed-methods approach consisting of quantitative surveys--to assess the difference in performance of both groups between baseline and endline, and qualitative methods-- to gain stakeholder perspectives about the impact and sustainability of the project. Employing the same tools administered at baseline for consistency and reliability, data collectors administered three surveys: the Early Grade Reading Assessment (EGRA), the Stallings classroom observation system (Stallings), and a Dietary Diversity Survey (DDS), using a 24-hour recall. Thirty data collectors received four days of classroom-based training on tools, protocols and electronic equipment (Tangerine and tablets) for data collection. Following the training, enumerators conducted two practice administrations at primary schools. Et4d ALIGN Cameroon Endline Survey Report 7 Limitations There are numerous limitations potentially affecting endline results. First is the interruption of schooling, due to teacher strikes and school closures, which took place over the course of seven months during the project implementation period, from November 2016 to June 2017. This caused the suspension of school-based activities and school meals, which were replaced with the mobile reading activity and take-home food rations. Second is replication of the ALIGN Fantastic Phonics methodology in comparison schools, which was learned during interviews with Bui district Ministry of Basic Education representatives. This could have affected the performance of the comparison group and overall treatment effect of the project schools. Teacher transfers and spillover effects of other interventions (mobile reading activity and health and hygiene trainings) could also have affected performance of the comparison group. In terms of the endline study, the evaluation team encountered challenges with reaching the desired number of schools, students and teachers. Security risks led to high absenteeism rates among teachers and students in sampled schools, causing us to replace schools in the middle of data collection and to add more schools than anticipated. Weighting was applied to account for any biases during the analysis stage. Finally, also due to heightened security risks for expatriate staff, the evaluation team was unable to visit any schools during the enumerator training or fieldwork. It is important to bear these limitations in mind when interpreting the endline survey results. Results 1. What outcome-level changes are attributable to the project? Among the four outcome indicators listed in Table 1, the outcome with the greatest change since baseline, and a notable treatment effect attributed to the project, is MGD SO2 --increased use of health and dietary practices. For this outcome-level indicator, the proportion of students meeting the minimum acceptable diet indicator increased from 69% at baseline to 75% at endline. This difference minus any gains from the comparison group resulted in a treatment effect of 8.7% and was statistically significant. Therefore, there are 8.7% more students in the treatment schools consuming a minimum acceptable diet as a result of the ALIGN school-feeding program. The second outcome indicator that demonstrated positive results attributable to the project was MGDSO1-improved literacy of school-age children. The proportion of students meeting this indicator increased from 7.5% at baseline to 10.9% at endline with a treatment effect of 1.9% improvement over baseline. Despite this progress, mean scores and the overall treatment effect were quite low. There was very little difference between the performance of students in project and comparison schools. In fact, the majority of students (73% in both project and comparison schools) still cannot read. There was no significant improvement over the baseline results, which showed 75% of students unable to read. Female students, pupils in Kumbo sub-division, school-aged children, and students with literate parents produced higher mean scores, which are all factors unrelated to ALIGN interventions. The low project impact on literacy outcomes is most likely attributed to the seven-month period of school closures, teacher transfers, multi-grade classes, high student and teacher absenteeism, and the spillover effect of ALIGN interventions into comparison schools. Et4d ALIGN Cameroon Endline Survey Report 8 Table 1: Outcome-level changes of ALIGN project Outcome Target Baseline Endline Treatment Effect Factors associated with outcomes Improved literacy of school-age children 9% 7.5% 10.9% 1.9% Gender* Age* Parental literacy* Location* Improved quality of instruction 63% 52.0% 38.6% -15% Permanent private school teacher Grade 2 teachers Mean attendance ratio Improved student attentiveness 6% 31.1% 16.8% -2.6% Active teaching* Classroom* management Teacher outside of the classroom* Increased use of health and dietary practices 74% 69.1% 75.2% 8.7% Gender Location Age *Statistically significant (P<0.05) The improved quality of instruction outcome and improved student attentiveness outcome both showed negative treatment effects. For the improved quality of instruction indicator, the proportion of teachers engaged in active instruction for 50%+ of classroom time declined from 52% at baseline to 39% at endline. Meanwhile, the comparison group improved from 37.3% at baseline to 44% at endline, an increase of 6.7%. These differences in performance resulted in a treatment effect of -15% for the treatment group. While regression analysis did not reveal any statistically significant factors to explain this result, the decreased performance could be related to school closures, multigrade classes which reduced the amount of classroom time devoted to classroom instruction, teacher transfers reducing the number of ALIGN trained teachers, and replication of the ALIGN Fantastic Phonics training in comparison schools by the Bui district. The improved student attentiveness outcome is based on a reduction in the amount of class time in which more than five students were off-task. Hence, we expect to see a decline from baseline to endline, with a target of 6%. Performance on the student attentiveness indicator improved from 33% off-task at baseline to 17% off-task at endline in treatment schools, and from 30% off-task at baseline to 16% off-task at endline in comparison schools. The difference in performance between the comparison and treatment schools resulted in a treatment effect of -2.6% for the project schools. Factors positively associated with more students off-task were teacher outside of the room (e.g., to teach multigrade classes1) and classroom management (e.g., grading assignments, lesson planning, etc.). Meanwhile, use of active teaching pedagogy (e.g., reading aloud, discussion, demonstration), were associated with higher rates of pupil on-task behavior. With 25% of teachers managing multigrade classes, this could have affected students’ ability to stay on task. 1 In multigrade classes, an average of 48% of teachers were outside of the classroom compared 28% in mono-grade classes. Et4d ALIGN Cameroon Endline Survey Report 9 2. For each selected indicator, how much progress has been made against the baseline targets? Based on endline results in Table 1, the ALIGN project exceeded the targets for the two strategic objectives and outcome indicators: (MGD SO1) improved literacy and (MGD SO2) increased use of health and dietary practices. Simultaneously, ALIGN performed below the target for improved quality of instruction and improved student attentiveness. For the eight output indicators selected for analysis (See Table 2), ALIGN exceeded its targets for six indicators aimed to increase knowledge and skills of school administrators and community members for the various literacy and health-related activities. ALIGN particularly exceeded its target for training in child health and nutrition, culminating with 13,039 persons trained compared to the targeted 3,456. ALIGN also met its expected output for increased extracurricular activities engaging all 92 schools in after-school literacy activities. ALIGN fell short of meeting the indicator for improved skill and knowledge among teachers, which is measured by the percentage of teachers engaged in active instruction for 50%+ of classroom time. This is partially due to a reduction in the number of teachers in ALIGN schools, which decreased from 725 in 2016 to 648 in 2018, as a result of the crisis. Another contributing factor could have been the school closures, which created lag time between training and implementation, as well as disruption in teaching and learning. Table 2: Endline results against baseline and targets of selected output indicators Standard Output Target Baseline Endline MGD Standard 3 Increased skill and knowledge among school administrators 156 0 271 MGD Standard 5 Improved skill and knowledge among teachers 648 0 230 MGD Standard 9 Increased engagement of local orgs. and community groups 92 0 92 MGD Standard 18 Increased knowledge of good nutrition practices 3,456 0 13,039 Custom 5 Increased extracurricular activities 92 0 92 Custom 7 Increased community understanding of benefits 276 0 407 Custom 10 (& MGD Std. #18) Improved knowledge of health and hygiene practices 5,520 0 6,082 Custom 11 (& MGD Std. #18) Increased knowledge of safe food prep and storage practices 2,200 0 2,523 3. Qualitatively, what changes do stakeholders associate with ALIGN’s programming There was consensus among all stakeholders interviewed that the ALIGN project has infused positive changes in ALIGN-supported schools and communities. The six most noteworthy achievements were: 1. Improved literacy 2. Improved health and hygiene practices Et4d ALIGN Cameroon Endline Survey Report 10 3. Improved agricultural practices 4. Increased student attendance and enrollment 5. Improved school infrastructure, and 6. Increased community engagement. These accomplishments largely stemmed from the take-home food rations, the mobile reading activity, gardening techniques training, and the water and sanitation facilities. 4. In which programming areas, do stakeholders anticipate that ALIGN’s programming is sustainable? Stakeholders agreed that to a certain extent all ALIGN activities will be sustained, including the school library and mobile reading activity, parts of the school feeding program (e.g., management of school gardens and farms as well the agriculture practices learned), skills and knowledge gained from ALIGN trainings, and the water and sanitation facilities. However, all stakeholders noted the challenge to maintain school meals at the level of the ALIGN project (e.g., five days per week for all students). Most schools could manage no more than two meals per week with existing resources and stated that they would require additional land, inputs and support from the government or external donor to sustain the school feeding program. To provide meals for five days consumption would require the USDA and/or Cameroon government to provide additional inputs. Many stakeholders, including district representatives, believed that if the government made school-feeding a national priority, it would be a great achievement and the essential resources would become available. Conclusions Considering the three-year ALIGN project was condensed into two years due to the teacher strike and school closures, which stalled activities, it did achieve sustainable outcomes within that short timeframe in terms of improving knowledge, skills and creating a culture of reading. Although the project met its target for improving literacy of school-age children, the quality of instruction declined and the literacy interventions had low treatment effects. With more teacher refresher training and follow up coaching on evidence-based reading instruction, uninterrupted schooling (including summer catch-up programs), and appropriate reading materials, literacy results could be improved in future USDA McGovern Dole programs. In terms of the second objective, ALIGN achieved its target for increasing the percentage of students with a minimum acceptable diet and increasing the use of health and dietary practices. ALIGN also built the capacity of schools to establish and maintain school gardens while increasing the availability of food for school meals. Therefore, the foundation for maintaining the school-feeding program at the school level exists, and there is overwhelming support from schools and communities to continue meeting the minimum requirements for all pupils. However, in order to sustain the school meals at the level of the ALIGN project and maintain the positive health outcomes, more government and USDA intervention is needed. The next step, according to stakeholders, is to build on the commitment at the school-level with greater national and local government resources to bring ALIGN to full scale and sustainability. Et4d ALIGN Cameroon Endline Survey Report 11 Introduction This report presents the endline survey results for the Advancing Literacy through Good Nutrition (ALIGN) project. The three-year project runs from July 2015 to June 2018. ALIGN is implemented by Nascent Solutions Inc (Nascent) and funded by the USDA McGovern-Dole (MGD) Food for Education and Child Nutrition program. Project Description The ALIGN project is designed to achieve two strategic objectives: 1) to improve literacy of school-age children and 2) to increase the use of positive health and dietary practices. An activity that is central to achieving the literacy goal is the provision of a daily school lunch and periodic take-home rations using food commodities provided by the United States Department of Agriculture (USDA). According to the ALIGN project results framework, the literacy objective will be achieved via three intermediate results: improved quality of literacy instruction, improved student attentiveness and improved student attendance. The project sought to improve the quality of instruction through more consistent teacher attendance, better education materials, and training of teachers and administrators. ALIGN aimed to improve student attentiveness through the daily lunch program and after-school extracurricular activities. ALIGN planned to increase student attendance through school infrastructure improvements, enrollment outreach, an increased understanding of education benefits and reduced health-related absences. ALIGN also aimed to improve education systems through enhancing the capacity of local government and civil society to develop local education and nutrition policies. The health objective will be achieved by increasing the consumption of a minimum acceptable diet through provision of a daily lunch using food commodities provided by the USDA and homegrown fruits and vegetables from the school gardens established with technical assistance from ALIGN. The project aimed to increase the use of positive health and dietary practices through improved knowledge of hygiene, safe food preparation, and nutrition, as well as increased access to clean water and preventative health interventions. The project operates in 92 schools, both public and private, including Catholic, Baptist, Presbyterian and Islamic schools. All 92 schools are located in the Bui division in the Northwest region of Cameroon. The Bui division was selected because of its high rates of hunger and stunting, an indicator of chronic malnutrition, amongst school-age children in the area, and poor feeding and hygiene practices2. The 92 schools were selected for participation by government partners based on meeting at least four of the following criteria: 1) low enrollment, especially for girls; 2) lack of access to water and sanitation services; 3) lack of access to supplies and literacy materials; and 4) lack of trained teachers. All schools received the full menu of interventions, with the exception of pilot schools that received e-readers and after-school tutoring. Nascent implements the ALIGN project in partnership with the local authorities from the Ministries of Education, Public Health, Agriculture, and Water and Energy, and Caritas Kumbo. International partners include Books for Africa and WorldReader. 2 ALIGN Cooperative Agreement No.OGSM: FFE-631-2015/001-00, dated July 15, 2015. Et4d ALIGN Cameroon Endline Survey Report 12 ALIGN’s target beneficiaries comprised 157,000 students and their families as well as teachers, administrators, civil servants, parent-teacher associations, food preparers and other stakeholders associated with the 92 target schools. Purpose and Key Evaluation Questions The purpose of the final performance evaluation of the ALIGN project is to provide Nascent, USDA, implementing partners, and stakeholders with final measures of ALIGN’s outcome-level progress relative to the 2015 baseline values and project targets. The endline results will provide a snapshot of student achievement in relation to the literacy and health-related benchmarks for both the comparison and treatment groups to determine the outcome-level changes attributable to the project. It will further inform of ALIGN’s successes and challenges per stakeholder perceptions, which will enable ALIGN implementers to determine the most viable sustainability path going forward and to plan for future McGovern-Dole projects in Cameroon. Evaluation Questions The four key evaluation questions are: 1. What outcome-level changes are attributable to the project? 2. For each selected indicator, how much progress has been made against the baseline targets? 3. Qualitatively, what changes do stakeholders associate with ALIGN’s programming? 4. In which programming areas, do stakeholders anticipate that ALIGN’s programming is sustainable? Evaluation Methodology This section discusses the evaluation design and methodology for responding to each evaluation question, the sampling techniques, the rationale for the instruments selected, enumerator training, fieldwork and limitations. Evaluation design The endline evaluation is an impact evaluation, which compares the performance of students in a treatment group to the comparison group. It measures the difference in gains for each group since the baseline to determine the overall treatment effect or impact of the project. To respond to the evaluation questions, we used a mixed-methods approach consisting of quantitative surveys to assess the difference in performance of both groups between baseline and endline, and qualitative methods to gain stakeholder perspectives about the impact and sustainability of the project. Question 1 Evaluation Methods For evaluation question #1, three quantitative tools were used to measure four outcomes shown in Table 3. Table 3: Question 1 Data Collection Instruments Question 1 ALIGN Project Outcomes Instrument What outcome￾level changes are attributable Improved Literacy of School-Age Children Early Grade Reading Assessment (EGRA) Improved Quality of Instruction Stallings Observation Tool Et4d ALIGN Cameroon Endline Survey Report 13 to the project? Improved Student Attentiveness Stallings Observation Tool (Stallings) Increased Use of Health and Dietary Practices FAO’s Dietary Diversity Survey (DDS) The quantitative instruments (e.g., EGRA, Stallings, and DDS) were existing instruments from the baseline, while the qualitative instruments were specifically developed for the endline study to gather responses to questions 3 and 4. EGRA. The Early Grade Reading Assessment (EGRA) was administered one-on-one to students in Class 3 to assess English reading and comprehension skills learned by the end of two years of schooling3. The EGRA measures fluency in reading letters, words and passages as well as comprehension of stories read aloud by the assessor or the student. The four subtasks included in the EGRA instrument are: letter name knowledge, letter sound knowledge, non-word decoding, reading fluency, reading comprehension and listening comprehension. In order to meet the expected literacy outcome – ability to read and understand the meaning of grade level text by the end of two years of schooling, students must achieve a minimum score in the oral reading fluency and reading comprehension subtask4: Table 4: EGRA minimum score to achieve literacy indicator EGRA subtask Minimum score Oral Reading Fluency Greater than 0 Oral Reading Comprehension 40% of questions answered correctly Stallings Observation System. The Stallings observation system (Stallings) tool is based on a World Bank instrument adapted to the Cameroon context. The Stallings instrument was used to assess two project outcomes: 1) percentage of teachers in levels 1-6 spending at least 50%5 of their classroom time on active instruction and 2) percentage of classroom time in which five or fewer students are “off-task”. The Stallings tool is comprised of ten 15-second classroom observation snapshots spread evenly throughout the class period to capture what the teacher is doing, what materials are being used, and pupil involvement. The categories of instruction that qualify as “active” are: Table 5: Stallings categories of active instruction Activity Example of active instruction Reading aloud Teacher or students are reading aloud. One or more students are reading connected text from a textbook, trade book, periodical, their own writing, or reproduced material. When reading aloud, generally students take turns reading sections from the material. The teacher or student may also read aloud while the rest of the class follows along in their own texts.) 3 Class 1 students in Year 1 of the project were out of school during Year 2 of the project, and returned to school in Class 3 in Year 3 of the project. Thus, in order to assess students who had at least two years of schooling, Class 3 was selected for the EGRA rather than Class 2. 4 See results section for all reading categories and definition of literacy performance indicator. 5 Evidence from Stallings shows that teachers spending at least 50% of time engaged in “active instruction” is associated with high-achieving students. Et4d ALIGN Cameroon Endline Survey Report 14 Demonstration/ lecture The teacher, television or some form of media is informing or showing a student or groups of students about a subject i.e. reading, mathematics, science, history.) Discussion/debate/ question and answer Students and teacher interact in an academic discussion, i.e. a verbal exchange of ideas or opinions. This may be slow-paced questions and answers regarding reading material, science, mathematics, political science or any problem where there is not one right answer. This could also be a check by the teacher or student for comprehension and understanding of material. Practice/drill The objective here is to memorize material such as multiplication tables or spelling words. The teacher and students or several groups of students interact in oral question and answer sessions where there is one right answer. The teacher asks the students repetitive questions in order for the students to memorize the correct answer. The Stallings tool also tracks if students are off-task. The three categories that qualify as “off-task” are: 1) social interaction, 2) student uninvolved, or 3.) being disciplined. Both the EGRA and Stallings survey included a biographical survey to identify background characteristics of teachers and students that may be associated with teaching and learning outcomes (pupil age, student home environment, teacher qualifications, teacher training, prevalence of teaching and learning aids, etc.). Students who took the EGRA and DDS were given a student ID number that allowed us to track and analyze performance of the same students across both surveys6. Dietary Diversity Survey. The original Dietary Diversity Survey (DDS) used at baseline was administered at endline to the same students taking the EGRA. The DDS collected data to measure the percentage of school-age children receiving a minimum acceptable diet. According to the ALIGN MGD Performance Monitoring Plan, the “minimum acceptable diet” indicator measures both the minimum feeding frequency and minimum dietary diversity. 1. Minimum meal frequency for children is defined as three or more feedings of solid, semi-solid or soft food per day. 2. Minimum dietary diversity for children is defined as four or more food groups out of the following seven food groups: Grains, roots and tubers, Legumes and nuts, Dairy products (milk, yogurt, cheese), Flesh foods (meat, fish, poultry and liver/organ meats), Eggs, Vitamin-A enriched foods, including vegetable oil, fruits and vegetables, Other fruits and vegetables Based on the above indicators, the DDS tool collected data on the number of food groups consumed and frequency of meals within 24-hour recall. Question 2 Evaluation Methods In response to Evaluation Question #2, “For each selected indicator, how much progress has been made against the baseline targets?” eight output indicators (4 standard and 4 custom) were selected (See Table 6). For each indicator, we examined the progress achieved since the baseline and in relation to the targets using monitoring data provided by Nascent M&E staff. We further conducted qualitative interviews with project staff to gain insights into what contributed to or hindered achievement of the targets. In addition, 6 This analysis was conducted, but not included in the final report since we did not identify any pattern or relationship in performance across both surveys. Et4d ALIGN Cameroon Endline Survey Report 15 we assessed progress achieved against targets for each outcome indicator listed in Question 1. Table 6: Endline results for selected output indicators Questions 3 and 4 Evaluation Methods For Evaluation Questions #3 and 4, key informant interviews were conducted with government counterparts and implementing partners at the district level, and focus group discussions were facilitated with school staff and Parent-Teacher Association (PTA) members. At the government level, key informants included: local officials from Ministries of Basic Education, Partnership with Education, Public Health, Agriculture and Rural Development and Water and Energy. Additional interviews took place with Nascent project staff and their partner, Caritas Kumbo. At the school level, three focus groups were conducted with participants from different schools: one with head teachers, one with literacy-trained and health-trained teachers, and one with parents who were PTA members and involved in ALIGN project activities. A separate interview guide was developed for national-level stakeholders, for Nascent staff, and for each school-level informant. All stakeholders were asked about what changes have been observed in ALIGN-supported schools as a result of project interventions (Evaluation Question #3) and which project activities are likely to be sustained and why (Evaluation Question #4). Sample size We used the same sampling methodology employed at baseline in order to collect reliable and consistent data that could be compared against baseline results. Using a two-stage cluster sampling with project and comparison schools as the primary sampling unit and pupils/teachers as the secondary unit, the total targeted sample size was 80 schools (40 comparison and 40 project schools), 360 teachers and 1,120 students. Standard Output/Outcome Output Indicator MGD Standard 3 Increased skill and knowledge among school administrators Number of school administrators who demonstrate use of new techniques as a result of USDA assistance MGD Standard 5 Improved skill and knowledge among teachers Number of teachers who demonstrate use of now and quality teaching techniques or tools as a result of USDA assistance MGD Standard 9 Increased engagement of local organizations and community groups Number of Parent-Teacher Associations or similar school governance structures supported as a result of USDA assistance MGD Standard 18 Increased knowledge of good nutrition practices Number of people trained in child health and nutrition as a result of USDA assistance Custom 5 Increased extracurricular activities Number of schools organizing after-school literacy activities as a result of USDA assistance Custom 7 Increased community understanding of benefits Number of information sessions held regarding education benefits held as a result of USDA assistance Custom 10 (and MGD Std. #18) Improved knowledge of health and hygiene practices Number of individuals trained in hygiene as a result of USDA assistance Custom 11 (and MGD Std. #18) Increased knowledge of safe food prep and storage practices Number of individuals trained in safe food preparation and food storage as a result of USDA assistance Et4d ALIGN Cameroon Endline Survey Report 16 However, due to the political unrest and high student/teacher absenteeism, we collected data in 89 schools (43 project and 46 comparison), of which 22 were replacement7 schools. We surveyed 1,123 Class 3 pupils and 354 teachers with almost even distribution between project and comparison groups (186 project, 168 comparison). As the baseline found no issue with gender disparities, gender was not a stratification criterion for sampling. Table 7 lists the sample size and methods for each quantitative instrument. For the EGRA and DDS instruments, we initially planned to randomly sample 14 students from Class 3, but during fieldwork we found schools with fewer than 14 students due to high student absenteeism (especially in comparison schools), and thus increased the sample size to 16 students. In most cases, all students present were surveyed due to low attendance. For the Stallings instrument, 2-6 teachers were observed from Class levels 1-6, based on availability (See limitations section). A total of 1,123 students participated in EGRA and 1,122 in the DDS8. A total of 354 teachers were observed using the Stallings Observation tool. Table 7: Sampling for quantitative methods Instrument Sample Size Sampling method (project and comparison schools) Confidence interval and power EGRA 85 schools (41 Treatment; 44 Comparison) 1,123 students level 3 Up to 16 students based on student attendance from Class 3 per school (if there was more than one Class 3, one was randomly selected) 95% confidence interval with power to detect a 5.2% change in improved literacy outcome Stallings 89 schools (43 Treatment; 46 Comparison) 354 teachers Up to 6 teachers selected from class levels 1-6, based on availability of teacher For improved quality of instruction outcome, 95% confidence interval with power to detect a difference of 13.9%; For improved student attentiveness outcome, 95% confidence interval with power to detect a difference of 11.9% DDS 85 schools (41 Treatment; 44 Comparison) 1,122 level 3 students Up to 16 students based on student attendance from Class 3 per school (if there was more than one Class 3, one was randomly selected) 95% confidence interval with power to detect a 10.8% change in health and dietary practices outcome For the qualitative evaluation questions, we used a purposeful sampling methodology. The criteria for purposeful selection was to select the most knowledgeable stakeholders who had active involvement in the project and could best evaluate the outcomes, challenges and sustainability of the ALIGN project. To collect a representative sample of 7 Schools that were found to be unsafe due to security risks were replaced with alternate schools. Oversampling prior to data collection allowed us to identify replacement schools. 8 We had planned to assess the same students for the DDS and EGRA, but one student in a comparison school who took the EGRA refused to participate in the DDS. Et4d ALIGN Cameroon Endline Survey Report 17 teachers, we selected teachers from various grades and different schools. Head teachers from different school types (public, private, Christian, etc.) were selected. For the parents’ focus group, we requested to speak with parents who were PTA members and involved in diverse project activities (kitchen, commodity management committee, library, school garden, etc.). In total, we interviewed one national representative, six district stakeholders, 12 Nascent staff, and six head teachers, five teachers, and seven parents. Enumerator training The enumerator training workshop for the ALIGN project took place over the course of six days in Kumbo from April 16-21, 2018. A total of 30 enumerators participated in the training supported by Et4d facilitators, ALIGN Field Agents (FAs), all of whom served as enumerators in the baseline, and project technical staff. The purpose of the workshop was to train enumerators on the three surveys (EGRA and the associated student questionnaire, Stallings, and DDS) as well as on the protocols and equipment to be used during data collection. Unlike the baseline, which used paper instruments for data collection, enumerators entered data directly onto tablets using Tangerine software for the three quantitative tools9. Two school visits were organized in a nearby primary school to allow assessors to practice administering the tools with students and teachers, and using the technology. During the first school visit, each enumerator practiced the EGRA and student questionnaire with at least five students. On the second school visit, the team practiced administration of the DDS and Stallings classroom observation, and performed student sampling protocol. In addition to hands-on practice with facilitator feedback and practice school visits, Assessor Accuracy Measurement (AAM) simulations were conducted three times for EGRA and once for the Stallings and the DDS in order to familiarize the enumerators with the process and examine their accuracy levels. A score of 90% is considered passing the training. On average, trainees scored 94% accuracy for EGRA and 90% for Stallings and DDS. A few enumerators needed more training for DDS, which was provided to the DDS-specific enumerators. Training facilitators selected final enumerators, including Nascent Field Agents, according to AAM scores, quiz results and observations. Although we had planned to select the best 25 enumerators, all 30 trainees were retained as enumerators for two reasons. First, with the intensified political unrest, there was a need to shorten the data collection period and increase the number of teams. Secondly, all trainees passed the training with high levels of accuracy scores for each instrument. Therefore, it was possible to increase the number of teams from five to six. The enumerators received full training documentation, including the training program, enumerators’ manual, and assessor observation checklist. Challenges encountered. Due to heightened security risks during the training, the evaluation team was not able to oversee enumerator school practice. Instead, Nascent’s 9 Using tablets eliminates the need for data entry thereby minimizing error and facilitates cleaning of the data. Et4d ALIGN Cameroon Endline Survey Report 18 technical team conducted observations and provided feedback to the enumerators, while Et4d provided remote technical support during practice school visits and the first week of data collection. Additional challenges, according to trainee survey feedback and trainer observation, was the length and duration of the training. In order to allow every enumerator to experience administering all tools, it would be necessary to increase the number of training days. The training was successful as a result of the high standards of the trainees and the decision to specialize enumerators earlier than initially foreseen. Time for DDS survey training should also be budgeted carefully as DDS requires a lot of practice in order to be performed accurately. Data collection Fieldwork took place from April 23 through May 17 spanning four weeks. While we originally planned to visit 80 schools (similar to the baseline), we had to increase our final sample to 89 schools to reach the desired number of teachers and students due to high absenteeism rates. Although there were four weeks of data collection, there were 16 effective days of actual school visits due to holidays or security risks and the need to replace schools. Each team was composed of 2 enumerators for EGRA, 2 for Stallings, and 1 for DDS. One enumerator was designated as the supervisor for each team. In most cases, the supervisor was a FA, who was responsible for the Stallings observations. The supervisor was responsible for carrying out the introductions to Head Teachers, sampling the students, coordinating logistics, and providing directions to the driver. Prior to data collection, the evaluation team leader prepared weekly schedules for each data collection team and assigned schools randomly. Due to increased insecurity in a number of schools, we had to adjust the plan frequently and select replacement schools in 22 instances. During the first week of data collection, it was planned to have a Nascent or Et4d staff member supervise each team to assist with sampling and quality control. Due to security concerns for expatriate team members, Nascent staff oversaw fieldwork and ALIGN Field Agents served as enumerators. Their prior experience with the baseline was instrumental to ensuring high quality control and adherence to protocols. The Nascent Staff continued providing additional follow-up support to supervisors and teams as needed with remote technical assistance from Et4d. The data was analyzed by Et4d daily based on the school sheets completed by the supervisors and the data was uploaded to Tangerine Central. Daily feedback was communicated to the local M&E team, who relayed any data quality issue to the supervisors. Challenges and Limitations There are numerous limitations affecting endline results. One of the greatest challenges affecting ALIGN outcomes is the interruption in schooling that took place during the project. All 92 project schools in Bui Division closed for seven months (from November 2016 to June 2017) due to teacher strikes and political unrest. This caused the suspension of school-based activities and school meals, which were replaced with the mobile reading activity and take-home food rations. Teachers who were trained during the summer of 2016 therefore did not have the expected two years of implementation. Et4d ALIGN Cameroon Endline Survey Report 19 Following a refresher training in 2017 for the first cohort and an initial training for the second cohort, teachers had one full year of implementation. Thus, the assessment does not measure consecutive classroom instruction, but interrupted teaching and learning in treatment schools. This could account for the low treatment effect in the project schools. School closures could have also affected the performance of students in comparison schools. An impact-related limitation that helps to explain why there are no significant differences between the treatment and comparison group is the exposure to the ALIGN training to teachers in the comparison group. During discussions with the Ministry of Basic Education Divisional Pedagogical unit, we learned that the Basic Education staff replicated the ALIGN Fantastic Phonics training with all head teachers in Bui Division, and with all teachers at the sub-divisional level. Therefore, all teachers in comparison schools have received similar Fantastic Phonics training as teachers in project schools. However, it is likely that the project school teachers received a higher quality training than the comparison schools since it was provided directly by the Fantastic Phonics trainers, rather than through a cascade model. Furthermore, comparison schools did not receive teaching materials and the full methodology; however, there may have been a spillover effect caused by teacher transfers, the mobile reading activity, and community￾based activities. It is important to bear these challenges in mind when interpreting the results of the literacy outcome (MDG SO1 performance indicator). Another challenge related to data collection was reaching the desired number of schools, students and teachers. Because of the political unrest during the evaluation, about 25% of the schools selected were unsafe and therefore they had to be replaced10. The final school sample was 89 as opposed to the targeted 80 schools. Teacher absenteeism, especially in comparison schools, was very high, due to the security situation. To reach the targeted number of teachers, we asked the team to observe all teachers present whenever possible to make up the difference for schools in which the number of teachers present was below our target number. Consequently, we observed two teachers in some schools and up to six teachers in other schools. The situation was similar for students (See Table 8). Forty percent or more of enrolled students were absent in about half of the schools visited. In 28 schools, there were fewer than 14 students (the target number) present. We tested all students in these schools, but fell below the target number. To address this deficit, we increased the student sample size from 14 to 16 students in subsequent schools. Despite these measures, by the end of data collection, we were below the targeted 160 Stallings observations in comparison schools and project schools. Therefore, four additional schools were added to increase the number of Stallings observations. Unfortunately, we did not have enough replacement schools available, which were considered safe; so, in the end we were only able to collect 168 Stallings observations in comparison schools and 186 in project schools. Weighting has been applied to all datasets to control for any differences in probability to be selected in the sample. This statistical approach corrects the bias that is caused by the sample size differences. 10 A list of project and comparison schools that could serve as alternates had been identified prior to data collection, so we were easily able to identify replacement schools to substitute the original sampled schools. Et4d ALIGN Cameroon Endline Survey Report 20 Table 8: Percentage of students absent in Class 3 Percentage of students absent Number of schools Over 70% 6 60 to 69% 12 50 to 59% 16 40 to 49% 10 30 to 39% 12 20 to 29% 10 Less than 20% 19 Finally, it is important to note the limitations the evaluation team experienced with conducting school visits. Due to the heightened security risks in the project areas, the evaluation team was unable to visit any schools during the training, piloting or data collection. Focus groups with head teachers, teachers and PTA representatives were held at the ALIGN office in Kumbo. Additionally, ALIGN selected the participants. Consequently, these restrictions limited our ability to randomly select participants, conduct school observations, and validate the data collected from focus group discussions. Yet, we did meet face to face with multiple head teachers and teachers, which could have increased the level of accountability to provide credible responses. Data analysis Data from each instrument was analyzed using descriptive and inferential statistics to formulate responses to each evaluation question. For the quantitative tools, we report on the proportion of students meeting the outcome performance indicators using difference￾in-difference analysis to determine the treatment effect between baseline and endline. We also report on EGRA mean scores and zero scores, disaggregated by school type, gender and geographic location; on mean times for each type of activity category observed for the Stallings tool; as well as on the types of food consumed for DDS. Through examining student characteristics and EGRA mean scores, we aim to identify factors associated with results to help explain variations in scores. Regression analysis of teacher and classroom factors and mean scores was conducted to identify additional factors associated with literacy outcomes, student off-task behavior and food consumption. Finally, content analysis and triangulation of results was performed to identify key themes and responses to the qualitative questions. Question 1 Results and Discussion: What outcome-level changes are attributable to the project? This section reports the results for each outcome indicator and instrument. Using difference-in-difference analysis, we determine the treatment effect between the baseline and endline for each outcome indicator. Difference-in-differences is a statistical technique that compares the average change over time in the outcome variable for the treatment group compared to the average change over time for the comparison group. The difference between these gains is the “true effect” of the program, known as the “treatment effect”. The treatment effect represents how much the project schools improved compared to the comparison schools since baseline, which is a determinant of the impact of the program and is attributable to project interventions. In addition, we provide descriptive data, such as mean scores, to describe the overall performance of Et4d ALIGN Cameroon Endline Survey Report 21 the target group for each indicator. Finally, we examine characteristics associated with expected outcomes to identify which factors are possible predictors of higher performance. EGRA Results Proportion of students meeting improved literacy performance indicator. To achieve the ALIGN MGD SO1 performance indicator for improved literacy, students must fall into one of two reading categories: reads fluently and understands or reads with some understanding (See Table 9). Based on the definitions of each category in Table 9, the minimum score for achieving the MGD SO1 indicator is an ORF score greater than zero and a reading comprehension score of 40% correct. Table 9: Definition of MGD performance Indicator for improved literacy Reading Categories ORF Score Reading Comprehension score Meets ALIGN SO1 indicator Reads fluently and understands ≥35 cwpm ≥ 60% correct Reads with some understanding > 0 ≥ 40% correct of questions attempted Does not meet ALIGN SO1 indicator Read some words, but didn’t understand > 0 < 40% correct of questions attempted Cannot read 0 N/A At endline, 10.9% of students (95% CI(7.6, 15.3)) achieved the MGD SO1 indicator in project schools compared to 9.3% (95% CI(5.9, 14.2)) in comparison schools (See Figure 1). This is an increase from the baseline results, in which 7.5% of pupils met the benchmark. Project schools showed an average of 1.9% more improvement since baseline compared to the comparison schools. The difference-in-difference treatment effect is estimated at 0.019 and is not statistically significant (p = 0.716). The results in Figure 1 show that the majority of students, 73% in both project and comparison schools, cannot read. There is no significant improvement over the baseline results, which showed 75% of students unable to read, 16.7% reading some words, but did not understand, and 7.4% reading with some understanding. We do see slightly higher results in the percentage of students who can read fluently and understand (0.1% at baseline compared to 2.2% in comparison schools and 1.5% in project schools). Overall, the treatment effect of the project and improvement in reading is quite low. There are many explanatory factors contributing to these results. Chief among them, as mentioned in the limitations section, are school closures from November 2016 through June 2017 due to teacher strikes, the divisional basic education unit replicating the Fantastic Phonics methodology in comparison schools, and the spillover effect of bringing books home from the mobile reading activity, which could have reached students in comparison schools. Et4d ALIGN Cameroon Endline Survey Report 22 Figure 1: EGRA results by reading category 73.1% 73.1% 17.7% 16.0% 7.8% 8.7% 1.5% 2.2% 0% 20% 40% 60% 80% 100% Project Comparison Cannot read Read some words, but didn't understand Read with some understanding Read fluently and understands Et4d ALIGN Cameroon Endline Survey Report 23 Proportion meeting literacy indicator by gender. At endline, 7% of boys in project schools (95% CI (4.0,12.1)) met the MGD S01 indicator compared to 4.9% of boys (95% CI), (2.8, 8.1)) in comparison schools. For girls, this proportion is 14.8% in project schools (95% CI(10.0, 21.1)) and 13.3% in comparison schools (95% CI(8.2,20.8)). There were 10% more boys and 9% more girls in the project schools who met the indicator since the baseline11. Figure 2: Proportion of students meeting literacy indicator by gender 11 The difference-in-difference treatment effect for boys is 0.100 and is not significant (p=0.179.) For girls, the treatment effect is 0.093 and is not significant (p=0.303). 14.8% 7.0% 13.3% 4.9% 0% 10% 20% 30% 40% 50% Female Male Project Comparison Target Et4d ALIGN Cameroon Endline Survey Report 24 Proportion meeting literacy indicator by geographic location. Kumbo is the largest town of Bui division and tends to have a more peri-urban landscape than other project sites, which are primarily rural. Because of this geographical difference and assumption that students in urban areas tend to perform better than their rural counterparts, we separated results by students within Kumbo and students outside of Kumbo. Within Kumbo, 21.5% of students in project schools (95% CI(13.4,32.6)) reached the MGD S01 indicator compared to 12.2% in comparison schools (95% CI(6.1, 23.1)). Outside of Kumbo, 6.9% of students (95% CI(4.5, 10.4)) in project schools and 7.6% of students in comparison schools (95% CI(4.1,13.4)) achieved the literacy indicator. Therefore, a greater proportion of students within Kumbo met the benchmark than students in schools outside of Kumbo, and the treatment effect was greatest for project schools inside Kumbo. Within Kumbo, 29% more students (0.291 treatment effect) met the indicator at endline and it was statistically significant (p = .032). Meanwhile, only 4.3% more students met the indicator in schools outside of Kumbo (p=.495). Figure 3: Proportion of students meeting literacy indicator by geographic location 6.9% 21.5% 7.6% 12.2% 0% 10% 20% 30% 40% 50% Outside of Kumbo Kumbo Project Comparison Target Et4d ALIGN Cameroon Endline Survey Report 25 Proportion of students meeting literacy indicator by school type. Figure 4 shows no consistent patterns emerged from the results by school type. Among private schools, the ALIGN-supported schools performed better than comparison schools. Within government schools, the comparison schools performed better than project schools. There is no evidence to support the perception that private schools perform better than public schools. Figure 4: Proportion of students meeting literacy indicator by school type: private vs. public 15.6% 7.7% 7.5% 12.7% 0% 10% 20% 30% 40% 50% Private Public Project Comparison Target Et4d ALIGN Cameroon Endline Survey Report 26 Mean Scores Figure 5 provides a snapshot of the EGRA mean fluency scores for all schools sampled at endline. There were no significant differences in performance between the project and comparison groups. Both groups were better able to identify letter sounds than letter names. Both groups scored very low on the decoding (nonword) subtask and oral reading fluency subtask, indicating the majority of students have low fluency and decoding skills. The results for comprehension in Figure 6 were similar with both groups scoring equally low. Students comprehended an average of 3.8% of questions asked in the project group and 3.2% in the comparison group. Listening comprehension scores were higher for both groups, with each group answering an average of one-third of questions correctly. However, this is still quite low for class 3 considering listening comprehension is a pre-requisite to reading. The treatment group showed a 49% improvement in letter name and oral reading fluency at endline, and a 22% improvement in reading comprehension, but the results were not statistically significant12. Performance of the treatment group declined since baseline on two subtasks, in nonword (-38%) and letter sound (-33.6%). 12 Difference-in-difference treatment effect: Letter name (effect=0.493, p=0.600), Letter sound (effect=-0.336, p=.775), Nonword (effect=-0.382, p=.677), ORF (effect=0.490, p=0.450), Reading Comprehension (effect=-0.220, p=.758), Listening Comprehension (effect=3.173, p=.147). Et4d ALIGN Cameroon Endline Survey Report 27 Figure 5: Project and comparison mean fluency scores (correct letters/words per minute) Figure 6: Project and comparison mean comprehension scores (percent correct) 3.8% 34.2% 3.2% 33.8% 0% 10% 20% 30% 40% 50% Reading Comprehension Listening Comprehension Project Comparison 10.6 15.4 4.6 4.8 10.0 17.3 4.2 4.3 0 5 10 15 20 25 30 Letter Name Letter Sound Nonword Oral Reading Fluency Project Comparison Et4d ALIGN Cameroon Endline Survey Report 28 Zero Scores The proportion of students scoring zero on each EGRA subtask is illustrated in Figure 7. With 73% of students in both groups unable to read a single word of a story in the Oral Reading Fluency subtask, it is no surprise that many students across both groups could not complete many of the other subtasks. As illustrated, the proportion of zero scores in the project group ranged from 29.9% on listening comprehension to 88.6% in reading comprehension. Results for the comparison group were similar with zero scores ranging from 23.9% on letter sound to 90.4% on reading comprehension. Over 70% of students in both groups were unable to decode one unfamiliar word on the non-word subtask, which helps to explain the low oral reading fluency and reading comprehension scores. There was little improvement on the non-word and ORF subtasks since baseline13; and, an additional 9.2% of students in project schools scored zero at endline for the letter sound subtask. This result could be explained by the schools closures that took place during the project and could have caused learning regression. Figure 7: Proportion of students scoring zero on each EGRA subtask 13 Difference-in-difference treatment effect: letter name (effect=.019, p=.634), letter sound (effect=.092, p=.012), nonword (effect=-0.026, p=.441), orf (effect=-0.026, p=.441), reading comprehension (effect=-0.029, p=0.207), listening comprehension (effect=0.029, p=0.433). Letter Name Letter Sound Nonword Oral Reading Fluency Reading Comprehen sion Listening Comprehen sion Project 48.9% 33.2% 70.9% 73.1% 88.6% 29.9% Comparison 52.7% 23.9% 72.3% 73.0% 90.4% 24.9% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Proportion Scoring Zero Et4d ALIGN Cameroon Endline Survey Report 29 Discussion on factors associated with EGRA scores Using variables identified at baseline, we examined the relationship of EGRA scores with the following factors: gender, age, school type, home language, school location, household poverty score, and home literacy environment. There were stark and statistically significant differences in scores in relation to gender, age and parental literacy (described in detail below). Pupils whose home language was English had higher mean scores across all subtasks and a lower percentage of zero scores although there were no statistically significant differences. Similar to the indicator achievement results, the geographic location of the school was associated with EGRA scores. Pupils in schools within Kumbo had notably higher mean scores and lower zero scores. Students in the wealthiest quintile performed considerably better than students in other quintiles, while only listening comprehension was statistically significant. Other variables that showed positive relationships with mean scores, but not significant differences was eating before school, having time to read at home, and having reading materials at home. There were no significant trends with results by school type (public versus private schools) or for students who had homework help. Pupils who had time to read at home read 16.9 correct letter sounds per minute (clspm) while those who did not read 6.9 clspm, and the difference was statistically significant. Pupil gender. Similar to baseline results, pupil gender was found to be highly significant across all fluency timed tests. As Figure 8 illustrates, female pupils had consistently higher mean scores on letter name, letter sounds, non-word and oral reading fluency. The differences between female and male fluency scores were statistically significant. Girls also had a consistently lower percentage of zero scores compared to boys. At baseline, there were fewer girls enrolled in school, but girls who attended performed better than male counterparts. The endline results also found that female students are acquiring foundational literacy skills as a faster rate than male students. With regards to comprehension scores, there were no statistically significant differences between genders (similar to baseline results). Figure 8: Mean scores by gender 15.2 20.2 6.5 7 6.2 10.7 2.8 2.7 0 5 10 15 20 25 Letter Name* Letter Sound* Nonword* Oral Reading Fluency Female Male Et4d ALIGN Cameroon Endline Survey Report 30 A significant relationship exists between the age of the student and EGRA scores. The mean scores of students who were within the Grade 3 age range (7-9 years old) were significantly higher than students above or below the Grade 3 age. Students aged 7-9 read over twice as many letter names and letter sounds, and four times as many words and non-words as overage students (10+). Students within Grade 3 age range comprehended 5.2% of comprehension questions compared to 0.8% of comprehension questions among overage students. The differences were statistically significant for all subtasks, except for Listening Comprehension. Figure 9: Mean scores by school-age of student 12.4 18.4 5.9 6.3 5.1 37.7 7.8 9.4 1.8 1.5 0.8 28.5 0 5 10 15 20 25 30 35 40 At grade level (7-9 yrs) Beyond expected grade level Et4d ALIGN Cameroon Endline Survey Report 31 Parental literacy. Pupils with a mother or father who can read scored significantly higher across all EGRA subtasks (See Figures 10 and 11). For students with fathers who can read, the greatest differences were on the letter sound subtask (+8.5 clspm) and in listening comprehension, (+20.9 percentage points). Students with mothers who could read showed similar gains across all subtasks, averaging an increase of 9.3 letter sounds and 18.7 percentage points higher in listening comprehension than students with mothers who could not read. The scores for father literacy were statistically significant for all subtasks, while for mother literacy they were statistically significant for all timed fluency subtasks (letter name, letter sound, nonword and oral reading fluency). Figure 10: Mean scores by father literacy Figure 11: Mean scores by mother literacy 11.7 16.8 5.2 5.6 4.4 37.5 5.4 8.3 2.2 1.4 1 16.6 0 5 10 15 20 25 30 35 40 Father can read Father can't read 11.6 16.5 5.2 5.5 4.4 36.6 3.5 7.2 0.6 0.4 0 17.9 0 5 10 15 20 25 30 35 40 Mother can read Mother can't read Et4d ALIGN Cameroon Endline Survey Report 32 The regression coefficients in Table 10 further demonstrate the strong correlation between parental literacy and EGRA scores. Based on regression analysis controlling for differences in gender, students with a literate mother or father read 3.3 more words per minute on the oral reading fluency (ORF) subtask and correctly answered 2.5% more comprehension questions. Grade 3-aged (7-9 years) also showed positive correlations with ORF and comprehension scores. Students between ages of 7-9 read 2.5 more cwpm and comprehended 2% more than overage or underage students. The number of books checked out of the library did not show any substantial effect on fluency or comprehension scores. Table 10: Regression coefficients14 for fluency and comprehension Variable ORF Reading Comprehension Coefficients 95% CI Coefficients 95% CI In age (7-9 yrs) 2.48 1.04, 3.92 2.04 0.59, 3.49 Father can read 3.35 1.56, 5.14 2.66 0.83, 4.48 Mother can read 3.25 1,18, 5.31 2.43 0.34, 4.52 Number of books check￾out* 0.98 0.21, 1.75 0.51 -0.20, 1.22 *Effect of this variable has been controlled for gender, SES, help with homework and school type. The number of books checked out of the mobile reading activity was positively correlated with fluency scores. For every book checked out of the mobile library, fluency scores increased by 0.98 correct words per minute. This small increase could be due to the level of the books that were provided. Parents reported that the level of difficulty of the books was too high for Class 2 and Class 3 students, which may have impacted their ability to read the books. In fact, parents reported that they were more suitable for older siblings, but continued to check out books. When ALIGN received this feedback, they trained librarians to review the level of books and attempted to provide more age￾appropriate materials; however, there may not have been a sufficient quantity of materials or time to see impact on reading fluency levels. The treatment effect of the program was quite low (1.9% improvement since baseline) due to several limitations beyond the project’s control. Therefore, it is no surprise that the factors associated with improved literacy outcomes --school-age, gender, parental literacy, and geographic location-- are factors that existed prior to the project and are not related to ALIGN interventions. With more refresher training and follow up coaching, uninterrupted schooling (including summer catch-up programs), and appropriate reading materials, literacy results could be improved in future programs. Stallings Survey Results Proportion of teachers meeting the MGD 1.1 improved quality of instruction indicator. Based on research of effective classrooms, which found that effective learning takes place when teachers spend at least 50% of classroom time on active teaching, the MGD 1.1 indicator for improved quality of instruction is defined as the 14 All regression coefficients have been controlled for gender differences in performance. Et4d ALIGN Cameroon Endline Survey Report 33 percent of teachers engaged in “active instruction” for 50%+ of classroom time (See methodology section for definition of “active instruction”). Figure 12 shows the proportion of teachers in project and comparison schools meeting the improved quality of instruction indicator based on Stallings survey results. At endline, 39% (95% CI (32.2, 45.4)) of project schools teachers met this indicator compared to 44% of teachers (95% CI(35.8, 52.2)) in comparison schools. There was a significant decline in the quality of teaching in project schools compared to baseline results, which showed 52% of project schools meeting the standard indicator. Meanwhile, the comparison group improved from 37.3% at baseline to 44% at endline, an increase of 6.7%. As a result, the difference-in-difference treatment effect was -15% (-0.154) and is significant (p=0.031). Despite refresher trainings, the school closures likely explain this negative result in project schools. In comparison schools, the improvement from baseline to endline (from 37.3% to 44%) could be a result of the replication of the ALIGN training in comparison schools by the Bui divisional basic education staff as well as transfer of ALIGN teachers to non-ALIGN schools. It is important to note that these results are based on a single classroom observation and the subjects and level of the classes varied. Thus, the results could vary depending on the period, subject, grade or other factors. Figure 12: Proportion of teachers meeting quality of instruction indicator Stallings results by teaching activity. Four types of classroom activities were observed during Stallings snapshot observations: 1) active teaching, 2) passive teaching, 3) classroom management, and 4) teacher off-task. The percentage of 38.6% 43.8% 0% 10% 20% 30% 40% 50% Project Comparison Meet MGD 1.1 Indicator Et4d ALIGN Cameroon Endline Survey Report 34 classroom time devoted to each activity by school type is shown in Figure 13. Both project and comparison schools spent over 80% of the time on active teaching or classroom management. Comparison schools spent a slightly higher percentage of time on active teaching (43% compared to 41% for project schools), while project schools spent more time on classroom management (45% versus 40% for the comparison schools). Passive teaching was rarely observed in both types of schools, but was more often observed in comparison schools than in project schools. Unlike the baseline which found teachers off-task six percent of the time in both school types, there were significantly more teachers off-task at endline with a higher proportion in comparison schools (10.1 off-task) compared to project schools (8% off-task). Figure 13: Stallings results by teaching activity 40.9% 42.7% 5.7% 7.3% 45.4% 39.8% 8.0% 10.1% 0% 20% 40% 60% 80% 100% Project Comparison Active teaching Passive teaching Teacher classroom management Teacher off-task Et4d ALIGN Cameroon Endline Survey Report 35 Percent of classroom time devoted to active teaching by activity While the differences in classroom instruction time devoted to active teaching were not significant between project and comparison schools, there were some differences in types of activities teachers most often engaged in (See Figure 14). Teachers in the project group spent 15% of active instruction time on discussion and 10% of the time on practice and drill. Meanwhile, teachers in comparison schools spent 14% of the time on discussion and 15% on practice and drill. There were no significant differences on time spent on demonstration/lecture and reading aloud. Reading aloud, which is one of the most effective early grade reading strategies, accounted for 4% of active instruction time in both school types. About 50% of teachers observed were in the early grades (1-3) and of these, 35% used a read aloud strategy during the observation, while 28.8% of teachers in later grades used read aloud as a strategy. Figure 14: Percent of classroom time devoted to each active teaching activity 4.4% 4.3% 10.9% 9.3% 15.2% 13.6% 10.3% 15.3% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Project Comparison Practice & Drill Discussion/ Q&A Demonstration / Lecture Reading aloud Et4d ALIGN Cameroon Endline Survey Report 36 Percent of classroom time devoted to classroom management The classroom management category includes the composite activities highlighted in Figure 15. The most common activity observed in project schools was classroom management alone, representing 19% of observations. This includes activities in which no pupils are involved such as lesson planning, organizing the classroom and grading assignments. Classroom management involving pupils was the second most common activity observed in project schools accounting for 17.8% of snapshots. This can include distributing materials, facilitating clapping or songs intended to maintain order, and other similar activities. These two activities were also most commonly observed in comparison schools with 16.5% of the time spent on classroom management alone and 17.6% of classroom time spent on classroom management involving pupils. Seven percent of classroom time in project schools and 4% in comparison schools were spent on giving verbal directions to pupils, while less than 2% of classroom time was spent on disciplining students. Figure 15: Time devoted to classroom management by activity 6.8% 4.2% 1.7% 1.6% 17.8% 17.6% 19.1% 16.5% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% Project Comparison Classroom management alone Classroom management Discipline Verbal directions Et4d ALIGN Cameroon Endline Survey Report 37 Percent of classroom time devoted to off-task behavior Endline results found a substantial percentage of teachers off-task in both project and comparison schools. As Figure 16 demonstrates, the majority of this off-task time is spent with the teacher outside of the classroom. At baseline, 3% of classroom time was spent with the teacher outside of the room. This number has doubled at endline with 6% of classroom time in project schools and 7% in comparison schools spent with teachers outside of the classroom. Teacher social interaction was lower than or similar to baseline levels. While 2.7% of classroom instruction time was spent on teacher social interaction at baseline, at endline the figures were 1.8% in project schools and 2.6% in comparison schools. One reason teachers were “outside of the classroom” was to instruct other classes in multigrade classrooms or adjoining classrooms. Although they were still teaching, they were recorded as “teacher outside of the classroom” for the classroom being observed, following baseline protocol. Based on this finding, the significant increase in percentage of teachers outside of the classroom at endline could potentially be due to an increase in the number of multigrade classes. Multigrade classes existed in 25% of classrooms observed15. In multigrade classes, 48% of teachers were outside of the classroom compared to mono-grade teachers in which 28% were found outside of the classroom. Figure 16: Percent of classroom time devoted to off-task behavior 15 Data on multigrade classes was not collected at baseline; so, it is not possible to determine whether there was an actual increase since baseline. However, it is likely given the high proportion of student and teacher absences, resulting in smaller class sizes and fewer teachers to manage multiple classes. 0.4% 0.4% 1.8% 2.6% 5.8% 7.1% 0% 2% 4% 6% 8% 10% 12% Project Comparison Teacher out of the room Teacher social interaction or teacher uninvolved Social Interaction Et4d ALIGN Cameroon Endline Survey Report 38 Discussion of factors associated with improved quality of instruction. Regression analysis of teacher characteristics with improved quality of instruction results did not reveal any statistically significant findings (See Annex B). However, it appears that permanent private school teachers, grade 2 teachers and classrooms with higher mean attendance ratios were variables that had a greater probability of meeting the indicator. Teachers trained in the Fantastic Phonics approach showed a 28% greater probability of meeting the indicator compared to those not trained. These factors are not strong predictors of meeting the outcome indicator. Proportion of classrooms meeting the MGD 1.2 improved student attentiveness indicator. Based on the amount of classroom time in which more than five students were off-task (MGD 1.2 performance indicator), the endline results in Figure 17 show no significant differences between project and comparison schools (similar to baseline results). At baseline, 33.1%% of classrooms in project schools and 29.8% of comparison schools met the indicator. At endline, the off-task standard was met in 16.8% (95% CI(13.2, 23.1)) of classrooms observed in project schools, and in 16.4% (95% CI(12.1, 20.6)) of classrooms in comparison schools. The difference-in-difference treatment effect is -0.026 and is not statistically significant (p=0.492), which means the project schools performed 2.6% lower than comparison schools between baseline and endline. Figure 17:Proportion of classrooms meeting student attentiveness indicator 16.8% 16.4% 0% 20% 40% 60% 80% 100% Project Comparison Meet MGD 1.2 Indicator Et4d ALIGN Cameroon Endline Survey Report 39 Proportion of pupils off-task by activity. Three types of pupil activities are considered off-task: discipline, pupil uninvolved and social interaction. It is possible to observe several different types of off-task behavior simultaneously in any given classroom. Figure 18 shows off-task behavior separated by activity. The results are similar in both project and comparison schools. In both school types, social interactions were observed in 12% of observations, while pupils were uninvolved in 7-9% of classroom observations. Discipline was observed 1% of the time in project schools and 0.7% of classroom time in comparison schools. Figure 18: Pupil off-task behavior by activity16 16 Note: The total exceeds the total percentage of time off-task since students performed more than one off-task activity during the classroom observation. 12.3% 11.7% 7.3% 8.7% 1.0% 0.7% 0% 5% 10% 15% 20% 25% Project Comparison Discipline Pupil Uninvolved Social Interaction Et4d ALIGN Cameroon Endline Survey Report 40 Discussion of factors associated with student off-task behavior. Based on regression analysis of all activities in the Stallings classroom observation snapshot, several teaching techniques were found to be statistically and positively correlated to rates of pupil off-task behavior. Active teaching in general, and more specifically, reading aloud, demonstration and discussion teaching methods, were all associated with reduced rates of pupil off-task behavior. As illustrated in Table 11, all active teaching activities were associated with a 3.0 - 3.5 percent reduction in pupil off-task behavior. Conversely, classroom management and teacher being outside of the classroom were associated with higher rates of pupil off-task behavior. A ten percentage point increase in teacher out of the room or uninvolved is associated with a 2.9 percent increase in pupils off-task. These findings echo results from the baseline study. Table 11: Predictors of pupil off-task behavior Predictors of pupil off-task behavior Coefficient p-value Active teaching -0.032 <0.000 Reading Aloud -0.035 0.018 Demonstration -0.029 0.001 Discussion -0.035 <0.000 Practice & Drill -0.007 0.357 Assignment/Seat work -0.003 0.135 Copying -0.004 0.202 Verbal direction -0.041 0.005 Discipline 0.141 <0.000 Classroom Management 0.028 <0.000 Teacher out of the room 0.029 0.008 None of the student and teacher demographic variables analyzed were found to significantly influence student attentiveness (See Annex B). Thus, it appears teacher gender, class size, teacher school tenure, teacher, teacher training background, teacher place of origin, prevalence of teaching and learning aids and other classroom factors have little impact on pupil off-task behavior. While active teaching methods appear to have a positive correlation with on-task behavior, the relationship is quite small. Teachers’ off-task behavior may have also influenced students’ off-task behavior. More research is necessary to determine the best method for addressing off- task behavior of teachers and students. Dietary Diversity Survey (DDS) Results Proportion of students meeting ALIGN MGD SO2 Minimum Acceptable Diet indicator. In order to meet the ALIGN MGD SO2 minimum acceptable diet indicator, both the minimum frequency (3 or more feedings) and minimum diet diversity (4 or more food groups) must be met. Table 12 shows the proportion of pupils consuming a minimum acceptable diet based on DDS results. The proportion of pupils meeting the indicator at baseline was 69% in both project and comparison schools. At endline, this proportion increased to 75% for project schools and decreased slightly to 67% in comparison schools. The difference-in￾difference treatment effect is 0.87 and this is statistically significant (p=0.020), meaning that there are 8.7% more students in project schools consuming a minimum acceptable diet as a result of the ALIGN school-feeding program. Et4d ALIGN Cameroon Endline Survey Report 41 Table 12: Proportion of pupils consuming a minimum acceptable diet Project 95% CI Comparison 95% CI Pupils with minimum Acceptable Diet 75.2% (70.0,79.7) 67.4% (62.1,72.4) Female (n=556) 73.1% (66.5,78.7) 65.0% (57.7,71.7) Male (n=565) 77.3% (70.7,82.7) 69.7% (61.9,76.5) Proportion of students meeting minimum meal frequency. Per the results in Table 13, over 90% of students in both school types met the minimum meal frequency requirement, defined as 3 or more feedings of solid, semi-solid or soft food daily. A higher percentage of females met the indicator in the comparison group versus the treatment group, but this difference was not statistically. There was no difference in the meal frequency of males in both groups. Table 13: Proportion of pupils meeting the minimum meal frequency Project 95% CI Comparison 95% CI Pupils having 3 or more food daily 91.1% (87.1,94.1) 94.3% (91.4,96.3) Female (n=556) 88.9% (83.1,92.9) 95.2% (92.1,97.1) Male (n=565) 93.4% (88.7,96.2) 93.3% (89.3,95.9) Proportion of students meeting food group diversity requirements. As illustrated in Table 14, a significantly higher proportion of pupils in project schools met the food diversity requirements than pupils in the comparison group. In project schools, 80.5% of pupils ate four or more food groups compared to 69.4% of pupils in comparison schools. Girls and boys in project schools showed similar trends with 79% of girls and 80% of boys meeting the indicator compared to 70.9% of girls and 67.8% of boys in comparison schools. This difference is only statistically significant for boys. Table 14: Proportion of students meeting food group diversity requirements. Project 95% CI Comparison 95% CI Pupils having 4 or more food groups 80.5% (75.8,84.4) 69.4% (64.4,73.9) Female (n=556) 79.9% (74.1,84.7) 70.9% (63.1,77.5) Male (n=565) 81.0% (75.4,85.6) 67.8% (61.6,73.4) Mean number of food groups and meals consumed. According to the endline results in Table 15, pupils consumed an average of 4 food groups and ate three times a day. At baseline, students in both school types consumed 4.5 food groups and 3 meals daily. Hence, there was no significant change between baseline and endline results. Et4d ALIGN Cameroon Endline Survey Report 42 Results in Table 16 show no notable differences between the project and comparison schools for either indicator. Pupils who identified their consumption as normal (e.g., a typical day’s consumption) in project schools ate 4.5 food groups and consumed 3.4 meals, while students in comparison schools consumed 4.4 food groups and 3.5 meals. Pupils who described their previous day’s consumption as abnormal (e.g., unusual) ate 3.2 food groups and consumed 2.8 meals in project schools, and 3 food groups and 3.1 meals in comparison schools. There were no differences in consumption by gender. Table 15: Mean number of food groups and meals consumed Project Mean 95% CI Comparison Mean 95% CI Foods groups consumed 4.1 (4.0,4.2) 3.9 (3.8,4.1) Pupils with normal 4.5 (4.4,5.5) 4.4 (4.3,4.5) Pupils with abnormal 3.2 (3.1,3.4) 3.0 (2.9,3.1) Female (n=556) 4.2 (4.0,4.3) 4.0 (3.9,4.2) Male (n=565) 4.1 (4.0,4.2) 3.9 (3.8,4.0) Consumption occurrences 3.3 (3.1,3.4) 3.4 (3.3,3.5) Pupils with normal 3.4 (3.3,3.5) 3.5 (3.4,3.6) Pupils with abnormal 2.8 (2.6,3.0) 3.1 (3.0,3.3) Female (n=556) 3.2 (3.1,3.3) 3.4 (3.3,3.6) Male (n=565) 3.3 (3.2,3.4) 3.3 (3.2,3.4) Types of food groups consumed. According to the nine types of food groups in Figure 19, the most commonly eaten food groups were starchy staples, yellow or orange flesh foods, dark leafy greens, other fruits and vegetables, and legumes, nuts and seeds. Similar to baseline results, more than 60% of students consumed foods from each category. Also similar to baseline results, the least common food groups eaten were milk, eggs and organ meats. Figure 19:Food group consumption 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Starchy staples Yellow or orange flesh foods Dark leafy greens Other fruits and veg Legumes, nuts, seeds Flesh meats and fish Milk Eggs Organ meats Any consumption Standardized consumption Et4d ALIGN Cameroon Endline Survey Report 43 Discussion of factors associated with minimum acceptable diet. According to results of regression analysis in Table 17, no factor was found to be a significant predictor of pupils meeting the indicator. In this sample, the odds of boys having a minimum acceptable diet were about 20% less than girls. Students living in Kumbo had a 27.9% higher probability of having a minimum acceptable diet than students outside of Kumbo. Students who were within school age (7-9) had a 1% lower probability of meeting the indicator compared to older or younger students, which implies the differences are insignificant. The analysis did not reveal any relationship with school type, pupil class level, wealth, or whether the pupil had a normal day of consumption. Table 16: Predictors of pupils who have a minimum acceptable diet Predictors of pupils who have the minimum acceptable diet Coefficient Odds ratio p-value Boys 0.798 0.278 Live in Kumbo 1.279 0.407 Age of the student 0.998 0.688 Question 2 Results and Discussion: For each selected indicator, how much progress has been made against the baseline targets? Based on endline results in Table 18, the ALIGN project exceeded the targets for the two strategic objectives and outcome indicators: (SO1) improved literacy and (SO2) increased use of health and dietary practices. Simultaneously, ALIGN performed below the target for improved quality of instruction and improved student attentiveness. The ALIGN project aimed to improve literacy of school-age children from 7.5% meeting the indicator to 9% at the end of the project for MGD SO1. At endline, 10.8% achieved the literacy outcome, thus surpassing the target by 1.8%. For the improved quality of instruction outcome, the project aimed to engage 63% of teachers in “active instruction” for 50%+ of classroom time. At endline, only 39% of teachers met the target, which is a significant decline in teacher performance since the baseline (52%). This unexpected result could be explained by the school closures that took place between November 2016 and June 2017, and the lag time between teacher training and implementation of the new teaching methods. Additionally, multigrade classes further impact teachers’ ability to stay engaged in active instruction for half of the class period. For MGD 1.2 improved student attentiveness, the targeted percentage of classroom time in which more than five students were off-task was 6%. The proportion of classrooms meeting this indicator was 31% at baseline and 17% at endline. Thus, although the project missed the target, there was significant improvement in student attentiveness by the end of the project. The project also exceeded the MGD SO2 target (74%) by 1.2%, with 75.2% of school￾aged children receiving a minimum acceptable diet at endline compared to 69% of students at baseline. Et4d ALIGN Cameroon Endline Survey Report 44 Table 17: Endline results against targets and baseline for outcome indicators Standard Outcome Performance Indicator Target Baseline Endline MGD SO1 Standard 24 Improved Literacy of School-Age Children Percent of students who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text 9% 7.5% 10.8% MGD 1.1 Custom 1 Improved Quality of Instruction Percent of teachers engaged in “active instruction” for 50%+ of classroom time 63% 52% 39% MGD 1.2 Custom 4 Improved Student Attentiveness Percent of classroom time in which 5 or fewer students are “off-task” 6% 31% 17% MGD SO2 Standard 23 Increased Use of Health and Dietary Practices Percent of school-aged children receiving a minimum acceptable diet 74% 69% 75.2% Table 19 shows endline results for eight output performance indicators. The endline results, with the exception of MGD Standard 5 (which is based on the final evaluation data collection), are all from project monitoring data. Based on the data provided by ALIGN, the project met or exceeded all targets. MGD Standard indicators 3, 9, 18 and Custom indicators 7, 10 and 11. For outputs related to increasing knowledge and skills of school and community members, the project exceeded the targets. This was accomplished through involving the broader community in their training and awareness-raising activities. According to interviews with Nascent staff, “the school community was enthusiastic and engaged with our trainings more than we anticipated” (especially, for health, nutrition and hygiene trainings). For example, the project targeted 3,456 people to train in child health and nutrition and culminated with 13,039 persons trained. MGD Standard 5 indicator. ALIGN fell short of meeting this indicator, which is measured by the percentage of teachers demonstrating improved quality of instruction (active instruction for 50%+ of classroom time). Of the 648 teachers targeted, 230 (35%) meet the standard. This is partially due to a reduction in the number of teachers in ALIGN schools. In 2016-2017, there was an average of 725 teachers in ALIGN schools. However, due to the political unrest and insecurity, some teachers transferred to schools in safer areas, reducing the teacher count to about 550 in May 2018. Another explanation, also noted above for the MGD 1.1 standard indicator, is due to school closures, which created lag time between the training and implementation, as well as disruption in teaching and learning. Lastly, ALIGN’s training implementation focused heavily on class 1-3 teachers, as research demonstrates these are the most crucial grades for early grade reading development. Custom 5 indicator. In relation to this indicator, all 92 schools were very active in organizing after-school literacy activities. The project monitored whether schools had book clubs, reading competitions, after-school tutoring and other activities (mobile Et4d ALIGN Cameroon Endline Survey Report 45 reading activity). Reading clubs were established by trained library assistants and are functional in all 92 project schools. The reading clubs are well equipped with books through the mobile reading activity. The mobile reading activity, which was initiated in March 2017 to address school closures, saw a marked increase in participation. Following a slow launch in March 2017, by the end of March 2018, there were 14,789 students participating, representing 81% of all students enrolled in ALIGN schools. Additionally, reading competitions have been carried out in all 92 ALIGN schools, and 276 school reading champions were competing in sub-divisional competitions at the time of the evaluation. One champion in each division and class (levels 1-3) will be selected. Table 18: Endline results against targets and baseline for selected output indicators Standard Output Performance Indicator Target Baseline Endline MGD Standard 3 Increased skill and knowledge among school administrators Number of school administrators who demonstrate use of new techniques as a result of USDA assistance 156 0 271 MGD Standard 5 Improved skill and knowledge among teachers Number of teachers who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance 648 0 230 MGD Standard 9 Increased engagement of local orgs. and community groups Number of Parent-Teacher Associations or similar school governance structures supported as a result of USDA assistance 92 0 92 MGD Standard 18 Increased knowledge of good nutrition practices Number of people trained in child health and nutrition as a result of USDA assistance 3,456 0 13,039 Custom 5 Increased extracurricular activities Number of schools organizing after-school literacy activities as a result of USDA assistance 92 0 92 Custom 7 Increased community understanding of benefits Number of information sessions held regarding education benefits held as a result of USDA assistance 276 0 407 Custom 10 (& MGD Std. #18) Improved knowledge of health and hygiene practices Number of individuals trained in hygiene as a result of USDA assistance 5,520 0 6,082 Custom 11 (& MGD Std. #18) Increased knowledge of safe food prep and storage practices Number of individuals trained in safe food preparation and food storage as a result of USDA assistance 2,200 0 2,523 Et4d ALIGN Cameroon Endline Survey Report 46 Question 3 Results and Discussion: Qualitatively, what changes do stakeholders associate with ALIGN’s programming? Our key question to stakeholders was, “What changes have you observed in your school (or at the school-level) as a result of the ALIGN project?” Across all interviews with district-level stakeholders, Nascent staff, head teachers, teachers and PTAs, there was consensus that the ALIGN project has infused positive changes in ALIGN-supported schools and communities. The schools are grateful for the project interventions, particularly the take-home food rations, the mobile reading activity, gardening techniques, and the water and sanitation facilities. Considering the low number of stakeholders interviewed, it was not possible to aggregate responses. However, the responses have been grouped into six outcomes most often reported by stakeholders, which are also aligned with the project results: 1. Improved literacy 2. Improved health and hygiene practices 3. Improved agricultural practices 4. Increased school attendance and enrollment 5. Improved school infrastructure (including water and sanitation facilities) 6. Increased community engagement Improved literacy One major outcome of the ALIGN program, consistently reported by teachers and head teachers, was the culture of reading that was stimulated by the mobile reading activity. ALIGN introduced a temporary mobile reading activity in March 2017 as a mitigation measure to help children continue building their reading skills while schools remained closed. The enthusiastic buy-in of the mobile reading activity resulted in a culture of reading among children, parents and siblings. Assisted by ALIGN-trained library managers and school staff, parents would sign out library books for their children every two weeks. The mobile reading activity really increased children’s desire to read. Students, teachers and parents continued visiting the school library once schools reopened in September 2017. According to teachers and head teachers, the mobile reading activity also improved parents’ interest in reading. Since parents were asked to read the book with their child, parents became very much involved with reading and learning how read. Parents who did not know how to read would select picture books and tell stories based on the illustrations. To support parents with obtaining literacy skills, one school established an adult literacy program. Additionally, students increased their desire to read at school. One head teacher reported that students wanted to spend so much time reading that they increased the school time allocated for reading. There were numerous changes observed in the classroom as a result of applying the new Fantastic Phonics pedagogy. One teacher observed an improvement in her students’ ability to read sight words, especially three-letter words. Teachers claimed that through applying the approach, they improved their own knowledge and pronunciation of letter sounds. Both trained and untrained teachers benefited since the Ministry of Basic Education replicated the methodology in schools for all teachers. Et4d ALIGN Cameroon Endline Survey Report 47 The textbooks, notebooks and pencils were further cited as another contribution of the project to improved teaching and learning. In the past, many students did not have textbooks because their parents could not afford them. Now, according to one PTA member, “All children are holding a textbook. They are able to read; and, they are happy to read”. While e-readers were only piloted in four schools, several teachers and district representatives stated that the e-readers helped teachers and students to become more competent with using technology and that it was an exciting supplementary reading tool. Another notable change observed by Nascent program staff was improved teacher assignments. Due to increased awareness about the importance of the early grades, head teachers are now assigning the best teachers to Classes 1 and 2, while typically the most inexperienced teachers taught the lower grades. Additionally, student performance has improved on Class 6 exams. According to Ministry and ALIGN staff, ALIGN received a letter of appreciation from the Bui Division thanking them for improved scores on Class 6 exams. Improved health and hygiene practices Head teachers and teachers reported that children are healthier, more active, energetic and have fewer health-related absences. The school and community trainings on health and hygiene, provision of food at school and home, and the deworming medicine all contributed to these outcomes. School stakeholders reported that students have improved their hygiene practices, especially hand washing. With the new hand-washing stations provided by the ALIGN project and continued training, hand washing has become a daily habit among students compared to the past when it was a rare occurrence. Teachers and PTA members reported that students wash their hands before eating, after using the toilet, and at home. In one school, a student is in charge of distributing the soap to classmates and ensuring that every student washes his or her hands. As a result, there have been fewer incidences of intestinal illness. One PTA member stated, “In our environment, we used to have children reporting cases of stomach running, numerous times a year, but because of the new hygiene practices, we have not had any cases reported this year.” The good health and hygiene practices have transferred to the students’ homes, as they spread awareness about the importance of proper hygiene. According to stakeholders interviewed, parents are also applying what they have learned in the nutrition and food preparation workshops at home to create sanitary and healthy meals for their children. Parents who were trained how to prepare the food in a safe manner for the school’s kitchen are following the same safety guidelines when cooking at home. Parents are also introducing a variety of ingredients into their own cooking to create nutritious meals, and now children are bringing fruits and veggies to school to complete their diet. Children have learned what constitutes a diverse diet and are requesting fruit and vegetables from their parents to stay healthy. Et4d ALIGN Cameroon Endline Survey Report 48 Another change observed in schools is a clean school environment. According to a PTA member, the ALIGN field agent17 regularly checks the cleanliness of the school environment. They make sure everybody is making an effort to maintain the cleanliness of the school (e.g., using trash bins). Pupils, parents and teachers seemed proud of this change. The oil cans provided by USDA are now converted into trash cans and the rule of not throwing trash on the floor seems to be well enforced (See Figures 20 and 21). Figure 21: Students using donated oil cans as trash bins Improved agricultural practices at school and home As a result of participating in ALIGN school gardens, teachers and parents have improved their agricultural practices. They are now transferring the knowledge gained from school gardens to their home gardens and increasing the yield. Manure and seeds provided by ALIGN helped them to improve the yield of the school garden. The school gardens in some schools have been productive enough to provide food to supplement the USDA commodities. In another school, revenue generated from the school garden enabled them to pay for a night watch to guard the school grounds. Increased attendance and enrollment Annual focus group discussions with parents, teachers and traditional authorities in March 201718 discovered the most frequent reasons for student absences are: late, non-payment of school fees, lack of parental awareness of benefits of education, pupil health issues, attendance at funerals or traditional festivals, and helping family with chores or farm work at home. ALIGN has addressed most, if not all, of these issues. School-feeding, take-home rations for girls in Classes 5 and 6, enrollment campaigns and informal discussions with parents and teachers during the mobile reading activity all helped to increase attendance and student enrollment. The Ministry of Basic Education estimates that around 5% of students in the Bui division transferred to Nascent schools to benefit from the program (See 17 The evaluation team was not able to confirm this, as we did not visit the schools directly due to security precautions. 18 Nascent Solutions Inc. (2017). ALIGN Cameroon April 2017 – September 2017 Semi-annual report. Submitted to USDA on November 15, 2017. “The number of children in project schools has multiplied. Children have left other schools to attend Nascent schools. The impact is very visible. If you go to schools where children have transferred, you are going to discover they are wearing uniforms from another school.” Official at Ministry of Basic Education Figure 20: Clean school environment Et4d ALIGN Cameroon Endline Survey Report 49 textbox). The Ministry of Basic Education also informed that the dropout and absenteeism rates have reduced drastically. Note that this statistic was not supported with actual MOE data. Head teachers also claimed that absenteeism rates have been reduced and enrollment and attendance have increased. Teachers and head teachers attribute these results to the school meals and clean water, which motivate students to travel long distances to attend school. PTA members attribute the increased attendance and enrollment to the food as well as the school garden. Since the school provides food, students are able to attend school on time rather than wait for a meal to be prepared at home. Now that schools are operating a school garden, parents view education as relevant to them and are eager to send their children to school. The enrollment campaign and take-home food rations also significantly contributed to increasing enrollment for girls and tribal populations. The PTA educated parents about the importance of educating girls, rather than keeping them home to help with chores or farm work. This really helped girls in Muslim communities to attend school. According to Nascent staff, “The enrollment campaign went far beyond the project schools”. One of the unexpected outcomes of the project was a new community school that was established specifically for children in the Mbororo tribe, a semi-nomadic cattle￾rearing community that traditionally did not send their children to school, so they remained illiterate. At the request of the tribe, ALIGN supported the planned construction of the school, provided training to volunteer teachers, and 200 children were enrolled in the first year who had never attended school. This is a significant accomplishment of the ALIGN project. Mbororo children have also enrolled in other ALIGN schools (See textbox). Improved school infrastructure According to district representatives interviewed, the biggest success of the project was solving the water issue. The provision of water and sanitation facilities (boreholes/wells, solar pumping systems, hand washing stations, and water extension networks) increased access to clean water for drinking, cooking, hand washing, and removing waste. Beneficiaries stated that due to clean water, they are visiting the hospital less frequently. Children can wash their hands, drink clean water and keep their environment clean. Teachers are also teaching children how to use water to keep their body and environment clean. A district representative stated, “It is difficult to teach these messages if the children have no access to water”. As a result of the improved knowledge, children have become ambassadors of good hygiene for their family and community, spreading knowledge of hygienic practices. In one Islamic community with no access to water, the PTA allowed community members to use the borehole and organized a maintenance crew. This enabled the people to conduct off-season gardening. Despite these accomplishments, access to water is still a challenge frequently reported by teachers and head teachers. Unfortunately, there are many schools that did not meet the minimum criteria to receive water and sanitation facilities. In these schools, classes “We are in a community, where we are mixed with the Mbororo (tribe). Formally, they did not like to attend school, but now so many of them are coming, and they represent a quarter of our school. They have understood the importance of education, and they are even more punctual than other children.” PTA member Et4d ALIGN Cameroon Endline Survey Report 50 are disrupted because the students must fetch water and there is inadequate water to maintain the school garden or proper hygiene. In addition to water and sanitation facilities, some schools reported receiving school infrastructure repairs. At the beginning of the project, one ALIGN school was near collapsing. Now, it has been renovated thanks to the ALIGN project, and the head teacher is very grateful. Increased community engagement ALIGN trained 184 community mobilizers to conduct formal enrollment campaigns reaching an estimated 38,000 community members with messages on the benefits of education19. During the teacher strike and crisis, ALIGN conducted informal community￾based enrollment advocacy led by PTAs, community leaders, as well as teachers and head teachers. ALIGN’s holistic approach to community engagement involving multiple outreach strategies and stakeholders resulted in increased participation of parents and community members in ALIGN-supported schools. According to district representatives, parents are now more involved in school activities, such as assisting with the school farm, cooking, paying for community teachers and night watch guards, and bringing extra food to the schools. District representatives have also noticed massive attendance during PTA meetings and school assemblies in ALIGN￾supported schools compared to non-ALIGN schools. Head teachers played a key role in increasing community engagement. One head teacher shared his perspective, “After the training on school management, I realized my role as an administrator is to create committees and to coordinate their activities”. Teachers also played a role in encouraging parents to support their children’s education. When parents came to school to collect books from the mobile reading activity or to participate in other ALIGN activities, teachers took the opportunity to speak with parents about the importance of limiting chores, so that children have enough time to complete their homework, or reminded them to equip students with basic school supplies (e.g., a notebook or pencil). Many stakeholders stated that the take-home food rations and school meals helped parents to become more enthusiastic about sending their children to school, especially for girls in Muslim communities. Training parents in vegetable gardening techniques to work in school gardens also made parents eager to participate in trainings so they could apply it to their own farming practices. According to Nascent staff, the PTA played a significant role in promoting education by running the enrolment campaign. The PTA organized meetings to sensitize the community about the value of education. These meetings were well attended by all community members, including non-project school parents and family members. As a result of increased awareness and participating in ALIGN activities, parents are more supportive of the school and PTA. Some parents have 19 Nascent Solutions. (2017). ALIGN October 2016 – March 2017 Semi-annual Report. Submitted May 15, 2007 “The parents are happy with what they have learned (gardening techniques) because they can improve their own (farming) practice. It is a fantastic program!” Teacher Et4d ALIGN Cameroon Endline Survey Report 51 offered to pay for a schools’ electricity. Many parents who in the past would not pay for their children’s school fees are now paying. Parents have supplemented school meals with vegetables and fruit from their home gardens or farms, or provided cooking oil. Overall, there is a sense of community ownership of ALIGN-supported schools. Question 4 Results and Discussion: In which programming areas, do stakeholders anticipate that ALIGN’s programming is sustainable? All stakeholders were asked this key evaluation question, as well as what recommendations they had for future USDA McGovern-Dole programs. Stakeholders agreed that to a certain extent all ALIGN activities will be sustained, including the school library and mobile reading activity, parts of the school feeding program (e.g., management of school gardens and farms as well the agriculture practices learned), skills and knowledge gained from ALIGN trainings (in literacy, health and hygiene, nutrition), and the use of the water and sanitation facilities. However, all stakeholders noted the challenge to maintain school meals at the level of the ALIGN project (e.g., five days per week for all students). Most schools could manage no more than two meals per week with existing resources and stated that they would require additional land, inputs, and support from the government to sustain the school feeding program20. Stakeholders’ views on each of these components and their recommendations for future programs are highlighted below. Library initiative Head teachers, teachers, PTAs and Nascent staff all reported that the school libraries can be maintained since the school staff have been managing the libraries very well throughout the project. Currently, librarians maintain files on the books borrowed and if they are not returned within two weeks, the parents are charged a fine. Therefore, there is an incentive to return the books and it helps to maintain the school library. In addition to maintaining the books received from the ALIGN project, schools are taking the initiative to expand their libraries with book donations from other donors and NGOs. Stakeholders believe adding more books will make it sustainable. Teachers recommended that going forward the books should be leveled: “The librarian should level the books with color, and the children should be advised to take books at their level.” Some schools, which do not have a dedicated room for the library books (e.g., they are kept in the head teachers’ office) also suggested that a library be constructed to maintain the books. School-feeding program Based on discussions with stakeholders, the school-feeding program can be broken into three areas for sustainability: 1) improved gardening and kitchen management techniques, 2) management of school gardens, and 3) provision of school meals. Improved gardening and kitchen management techniques. All stakeholders believed the improved gardening techniques would be easily sustained. According to one PTA member, “The capacity building will remain. People have learned how to cook food in large volume, store the food, and grow some spices. Once you’ve got the skills, they 20 Communities did not express a need for labor, but for seeds and crops that are not locally available or that are not available year round. Et4d ALIGN Cameroon Endline Survey Report 52 cannot be unlearned.” As discussed in Question 3, the parents, PTA members and teachers who benefited from the garden training unanimously said that they have applied the new techniques and seeds to their home gardens, which has resulted in increased yields. Schools have also increased yields from school farms and gardens with the new methods they have learned from ALIGN. Thus, they will continue applying what they have learned. A district representative from the agricultural division suggested that the number of agricultural staff be increased in future programs so that they can further improve yields in all schools and expand the scope of activities to create a cooperative. This could make the schools more dynamic and generate more revenue. They could also plant fruit trees (e.g., guava, papaya, and oranges) to add to the diversity of school meals. Management of school garden. As noted above, schools have the requisite skills and capabilities to manage school gardens. To make the school-feeding program sustainable, they would need a larger plot of land to produce enough crops to feed all students. While some schools have acquired enough land for farming, others do not have any farmland and are renting plots. Nascent has been facilitating discussions with the local council and community to identify additional donated land for cultivation, but it appears to date this issue has not been resolved in many schools and government intervention is clearly necessary. Provision of school meals. As a result of the limited farmland and resources, most schools reported that they are only able to provide meals for about two days per week. To provide meals for five days consumption would require the USDA and/or Cameroon government to provide additional inputs and land. Some PTAs met with the local authority and discussed various options to increase the quantity of food available for the pupils. Possible solutions included increasing the size of the school farm or requesting a financial contribution from parents. PTA members felt that it was not possible to request additional contributions from parents since they are already paying salaries for community teachers. If the government paid community teachers, then the current revenue could be used for the school-feeding program. Additionally, there are some ingredients that are not locally available, such as cooking oil, which is very expensive to purchase, yet crucial. In fact, in the project area there is no production of palm oil or other cooking oil; so, even with an active school farm, cooking oil would still need to be procured from the market. Many stakeholders, including district representatives, believed that if the government made school-feeding a national priority, it would be a great achievement and the essential resources would become available to sustain the school-feeding program. At the national level, Nascent has been trying to support a change in the national policy and the ministry is recognizing that a school-feeding program would most likely improve learning outcomes. The ministry also feels that a “homegrown” policy would need to be in place, which requires the proper infrastructure. However, the government of Cameroon does not feel ready to maintain the school feeding program without external donor assistance. At the local and regional level, Nascent has been working actively with the local government to draft a school feeding policy framework, and the stakeholders are hoping that the ALIGN intervention will demonstrate the feasibility of such policy. The “If the government could take over the issue of feeding students and make it a national issue, it would be a great achievement!” Bui District official Et4d ALIGN Cameroon Endline Survey Report 53 stakeholders interviewed who have participated in the governance activity to draft an education and school meal policy have all reported on the importance of the activity and the steps taken so far to institutionalize it at the regional level. Skills and Knowledge Most stakeholders believe that the nutrition, health and hygiene practices acquired through ALIGN trainings will be sustained. According to Nascent, the health and hygiene training has been replicated at many levels, including the community. Numerous stakeholders (e.g., Nascent, Caritas, district representatives, teachers and head teachers) asserted that the knowledge and skills teachers have acquired from the literacy trainings will be sustained. The PTA provided two recommendations to improve literacy outcomes of future programs: 1) empower parents to be able to evaluate their children’s progress and train them so that learning can continue at home; 2) change the policy on transferred teachers so that the number of teachers assigned to a school is based on a teacher-student class size ratio. Teachers and head teachers requested additional ICT interventions, particularly more e￾readers, as well as books and teaching resources to enrich the learning environment for teachers and students. Water and Sanitation Stakeholders reported that the water systems installed are sustainable and will continue to be used for consumption, cleaning, washing, cooking and other activities. All the solar pumps provided by Caritas over the years are still functioning and the boreholes are providing good quality water. Some stakeholders thought maintenance could be an issue, despite communities having received a toolkit and user manual. Caritas assured that they would remain an active partner of schools beyond the life of the project. For future projects, teachers and head teachers recommended boreholes in every school and more infrastructure improvement for dilapidated schools. Additional observations The stakeholders seemed to have the capacity to maintain most of the activities, even if refresher training will remain necessary as new parents arrive to the school, or if teachers and head teachers are transferred. Some questions were raised on their capacity to monitor and manage the school-feeding program on their own even if the commodities were provided at no cost. Because a tremendous number of individuals were trained directly by Nascent staff, or by the training participants, most people at the community level are very well informed of the benefits of the project and are committed to continuing the activities after the ALIGN project ends. The methods used for each activity included in the ALIGN project have been extremely well documented. This is indeed one of the strong components of the project, along with the capacity building, which took place at the community level. ”The teachers have learned so much. Their skills will remain. They have taught other teachers.” Teacher Et4d ALIGN Cameroon Endline Survey Report 54 The training materials included all the technical information for each activity, such as commodity management, food preparation in large quantity, food safety, health and hygiene practices, school management, as well as methods for replicating the training with other teachers or PTA members. For the most part, the training material could probably be used with minimal modification for replication in non-project schools. Currently, the project documentation remains with Nascent and it is not clear if the local stakeholders were to assume responsibility for the project activities if they would be able to maintain the documentation and improve upon it. There is also a lot of knowledge exchange between schools and among communities, thanks to their collective participation in trainings. At this stage, unfortunately, there is no notion of a network between the schools, which could enable the dissemination of best practices; and, we fear that over the years the skills needed to manage such a project could be lost as PTA members rotate or teachers are transferred to other schools. Conclusions Considering the three-year ALIGN project was condensed into two years due to the teacher strike and school closures, which stalled activities, it did achieve sustainable outcomes within that short timeframe in terms of improving knowledge, skills and creating a culture of reading. ALIGN fell short of achieving all of its expected outcomes, particularly for improved quality of instruction and improved student attentiveness largely due to effects of the crisis, but it did achieve its two main strategic objectives for improving literacy and health outcomes, as measured by the targets established at baseline. Teachers faced many challenges throughout the project that hampered their ability to improve the quality of instruction, including the security situation which caused many teachers in ALIGN schools to transfer. For those who remained in ALIGN-supported schools, they struggled to manage multiple classes and to provide active instruction for 50% or more of the class time. The percentage of teachers found off task because they were “out of the classroom” (likely teaching other classes) doubled from baseline to endline, potentially resulting from an increased number of multigrade classes. We also found high teacher and student absenteeism rates in schools visited causing us to increase our sample size to reach the desired number of students and teachers. Thus, teacher shortages are likely an issue in the ALIGN project areas. While the project did exceed its target for the improved literacy of school-age children strategic objective, the mean scores and treatment effect of the program were quite low (1.9% improvement since baseline) due to the limitations described above. Therefore, it is no surprise that the factors associated with improved literacy outcomes --school-age, gender, parental literacy, and geographic location-- are factors that existed prior to the project and are not related to ALIGN interventions. Although the mobile reading activity was a success and an innovative way to keep students learning while out of school during the crisis, the effect on actual fluency scores was very low, with students gaining one more correct word per minute for every book read. Since the level of the books were reported as being too difficult for Class 3 students, this could explain why the books had a low impact, or perhaps the lack of parental literacy could have also contributed to these outcomes. Moreover, reading aloud in school, which is one of the most effective Et4d ALIGN Cameroon Endline Survey Report 55 early grade reading strategies, accounted for 4% of active instruction time in both school types, and was used by 35% of early grade teachers21. With more teacher refresher training and follow up coaching on use of evidence-based reading instruction, uninterrupted schooling (including summer catch-up programs), and appropriate reading materials, literacy results could be improved in future USDA McGovern Dole programs. Furthermore, the use of e-readers was considered to be an effective and attractive intervention that could be explored as a way to improve reading outcomes. In terms of the second strategic objective, ALIGN achieved its target for increasing the percentage of students with a minimum acceptable diet and increasing the use of health and dietary practices. The school feeding program, particularly the increased knowledge and skills of managing school gardens and the nutritious school meals, was very much appreciated by beneficiaries. Now the question is how to sustain the level of meals provided by the ALIGN program in order to maintain project outcomes (e.g., increased enrollment, improved attendance, improved nutrition and health of students, etc.). The findings from the qualitative interviews clearly indicate that more government and USDA intervention is needed to build on the foundational elements of the project and to bring ALIGN to full scale and sustainability with increased Cameroon government support. 21 About 50% of teachers observed were in the early grades (1-3) and of these, 35% used a read aloud strategy during the observation. However, on average the strategy was used for only 4% of classroom time. Et4d ALIGN Cameroon Endline Survey Report 56 ANNEX A: EGRA Scores by Student Characteristics Table 19: Annex A: EGRA Mean Scores and Zero Scores by Student Characteristics Letter name Letter sound Non-Word Oral reading Fluency Reading Comprehension Listening Comprehension Mean CLPM % Zero Score Mean CSPM % Zero Score Mean CWPM % Zero Score Mean CWPM % Zero Score Mean % Correct % Zero Score Mean % Correct % Zero Score Gender Female 15.2* 35.7% 20.2* 23.9% 6.5* 61.9% 7.0 64.9% 5.7 83.8% 36.6 24.7% Male 6.2 61.7%* 10.7 42.3%* 2.8 79.7%* 2.7 81.1%* 2.0 93.1% 31.6 34.9% Age At grade level (7- 9 years) 12.4 42.2% 18.48 27.3% 5.9* 63.8% 6.3* 66.3% 5.1* 85.2% 37.4 24.2% Beyond expected grade level 7.8 58.6% 9.4 38.2% 1.9 85.1%* 1.5 87.7%* 1.1 96.2%* 28.5 39.9% School Type Private (n = 134) 10.0 50.4% 17.8 25.8% 4.9 68.4% 4.4 70.7% 3.8 86.8% 38.2 17.3% Public (n = 432) 10.3 51.0% 16.3 27.2% 4.1 73.1% 4.5 73.9% 3.3 90.9% 32.4 29.7% Home Language English 14.9 38.1% 22.3 23.3% 6.8 62.2% 10.2 62.6% 7.7 79.1% 44.3 16.6% Other language 10.2 50.2% 14.7 34.1%* 4.4 71.6% 4.2 73.9% 3.4 89.6% 33.1 31.1% School Location Kumbo 15.5 40.3% 18.2 36.7% 7.1 61.4% 9.1 62.2% 7.5 77.1% 43.4 20.1% Outside of Kumbo 8.8 52.1% 14.3 32.0% 3.7 74.4% 3.3 77.2% 2.5 92.8%* 30.8 33.5% Household poverty score Wealthiest Quartile 13.7 41.9% 19.5 31.1% 6.4 63.4% 7.1 64.4% 6.4 81.1% 42.2* 22.7% Other quartile 9.3 51.9% 13.6 34.3% 3.8 74.2% 3.8 77.0% 2.7 91.9% 30.8 33.0% Ate before school Yes 10.8 47.7% 15.8 33.0% 4.9 69.8% 5.1 72.4% 4.2* 87.6% 34.7 29.7% No 9.0 56.9% 12.0 36.3% 2.7 78.6% 2.9 77.9% 0.8 85.9% 30.3 31.8% Time to read at school Yes 11.6 45.9% 16.9* 30.1% 5.1 67.9% 5.2 70.3% 4.2 86.9% 36.3 28.2% Et4d ALIGN Cameroon Endline Survey Report 57 No 7.4 58.3% 10.0 46.6% 3.0 80.4% 3.6 82.4% 2.7 94.5% 28.1 33.6% Homework help Yes 10.9 49.9% 15.9 30.0% 4.6 70.7% 4.6 75.1% 3.5 89.8% 38.0 19.1% Sometimes and No 10.6 47.4% 15.2 35.5% 4.7 70.6% 5.1 71.4% 4.1 87.7% 32.0 36.8% Reading material at home Yes 13.2 43.3% 16.9 29.2% 6.1 63.3% 6.5 68.8% 5.2 83.4% 38.6 22.7% Sometimes and No 9.7 50.9% 14.8 34.9% 4.1 73.7% 4.2 74.7% 3.3 90.5% 32.6 32.7% Father can read and write Yes 11.7* 46.4% 16.8* 31.8% 5.2* 68.2% 5.6* 70.6% 4.4* 87.3% 37.5* 25.2% No 5.4 62.6% 8.3 39.9% 2.2 82.8%* 1.4 87.4%* 1.0 94.8% 16.6 57.0%* Mother can read and write Yes 11.6* 46.6% 16.5* 31.8% 5.2* 68.5% 5.5* 70.1% 4.4* 86.9% 36.6* 25.8% No 3.5 64.8% 7.2 44.5% 0.6 88.3%* 0.4 95.2%* 0 100%* 17.9 57.6%* Et4d ALIGN Cameroon Endline Survey Report 58 ANNEX B: Factors Associated with Stallings Observations Figure 22: Predictors of improved quality of instruction Variables Odds Ratio p-value Teacher Gender Male 1.42 0.295 Type of employment contract Government school – Civil Servant - - Government school - Contract 1.52 0.640 Government school - PTA 0.89 0.895 Private school - Permanent 1.94 0.447 Private school - PTA 1.39 0.744 Other 8.33 0.046 Level of training Grade 1 - - Grade 2 2.4 0.346 None NA NA Other 1.42 0.359 Mean year of experience in this school 1.07 0.067 Training received Administration 1.11 0.827 E-reader 0.43 0.213 Fantastic Phonics 1.28 0.451 ICT 0.98 0.948 IST 0.64 0.249 Library 1.01 0.977 Other 1.44 0.257 None 1.33 0.464 Mean number of students enrolled 0.99 0.904 Mean number of students present 1.01 0.298 Mean attendance ratio 1.68 0.499 Blackboard in the classromm NA NA Charts, pictures, maps on the wall 0.97 0.914 Figure 23: Predictors of student attentiveness Variables Odds Ratio p-value Teacher Gender Male 0.055 0.158 Type of employment contract Government school – Civil Servant - - Government school - Contract 0.042 0.658 Government school - PTA -0.019 0.841 Private school - Permanent 0.086 0.353 Et4d ALIGN Cameroon Endline Survey Report 59 Private school - PTA 0.092 0.390 Other 0.205 0.061 Level of training Grade 1 - - Grade 2 0.143 0.179 None -0.267 0.024 Other 0.077 0.085 Mean year of experience in this school 0.002 0.609 Training received Administration 0.067 0.245 E-reader -0.032 0.642 Fantastic Phonics 0.010 0.781 ICT 0.075 0.031 IST 0.036 0.410 Library 0.027 0.546 Other -0.016 0.669 None -0.025 0.585 Mean number of students enrolled -0.003 0.004 Mean number of students present -0.005 0.000 Mean attendance ratio -0.194 0.023 Blackboard in the classroom NA NA Charts, pictures, maps on the wall 0.023 0.528 Table 20: Teacher and Classroom-level factors associated with pupil off-task behavior Predictors of pupils off-tasks behaviors Coefficient p-value Teacher Gender -0.004 0.167 Teacher Training (Grade 1 constrat level) Grade 1 - - Grade 2 -0.016 0.107 None 0.014 0.132 Other -0.008 0.340 Nbr. of years of experience in the school -0.006 0.081 From Bui division 0.004 0.481 Black board (100% of classroom) - - Charts, pictures maps on classroom walls 0.001 0.730 % of students who have textbooks 10% or less - - About 30% -0.006 0.166 About 50% -0.002 0.689 About 70% -0.003 0.571 90% or more -0.009 0.002 % of students who have notebooks 10% or less - - About 30% -0.010 0.523 About 50% -0.013 0.390 Et4d ALIGN Cameroon Endline Survey Report 60 About 70% -0.002 0.894 90% or more -0.001 0.913 Teacher receive no training 0.004 0.110 Teacher receive Administration training -0.008 0.141 Teacher receive eReader training 0.002 0.731 Teacher receive Fantastic Phonics -0.002 0.597 Teacher receive ICT technology -0.006 0.036 Teacher receive IST in service training -0.005 0.176 Teacher receive Library training -0.003 0.436 Teacher receive other training -0.001 0.762 Number of students in classroom 0.005 0.003 Multigrade classroom 0.049 0.088 Et4d ALIGN Cameroon Endline Survey Report 61 ANNEX C: EGRA Instrument EGRA – STUDENT QUESTIONNAIRE Please spend two minutes making light conversation with the pupil. Tell the child about yourself, and ask questions about things he/she likes to do, etc. Day month Year hour minute 1. Date: 2. Time at start 3. School Name: _________________________________________ 3a. School Code: 4. Data collector name: ____________________________ 4a. Data collector team #: 5. Tangerine ID _______________ Pupil Assent I work with Nascent Solutions, a group in Kumbo that is helping pupils. I am doing English testing to learn more about how to help pupils do better in English. If you decide to take part you will be asked to do some short games of English letters, words, and stories. It will take about 20 minutes. You can stop participating at any time. These games will not count as marks for you. Your teacher will not know how you did. You will not receive anything for doing the games ut you may be helping us to improve your school in the future. Your results will be secret. Your teacher and family are aware that you are taking part, but will not know how well you did. We will write a report about these exercises, but it will not include your name. The results will be erased from the tablet. Do you have any questions? Are you ready to get started? Yes No 6. Pupil gender (circle): Male Female Ask the pupil each question verbally, as in an interview. DO NOT READ THE ANSWER OPTIONS TO THE PUPIL UNLESS INDICATED TO DO SO. Wait for the pupil to respond to each question, then mark the answer that corresponds to his or her response. 7. What is your class? (select) 1 2 3 4 5 6 If the pupil is not in Class 2, contact your Team Captain. Et4d ALIGN Cameroon Endline Survey Report 62 8. How old are you? (select) 5 6 7 8 9 10 9. What Language do you speak at home most frequently English French Fulfulde Lamnso Noni Oku Ot he r Et4d ALIGN Cameroon Endline Survey Report 63 The following questions are asked after the EGRA evaluation 10. Did you eat before coming to school today? (Select) Yes No Don’t know/ no response 11. Did the pupil participate in mobile reading (See librarian answer on DDS paper) Yes No Not completed by librarian 12. How many times did the pupil check out books as part of mobile reading? (See librarian answer on DDS paper) Not completed by librarian (NA) 13. Did you like the books that you borrowed with the mobile library? Yes No Don’t know/ no response 14. Do you have an English reading textbook at your house? Yes No Don’t know/ no response 15. Do you have a math textbook at your house? Yes No Don’t know/ no response 16. Apart from school books, are there books, newspapers or other things for you to read at your house? Yes No Don’t know/ no response 17. Do you have time to read books in your classroom or school library every day? (select) Yes No Don’t know/ no response 18. How many minutes do you spend per day ready? none 1 to 30 min More than 30 Don’t know/ no respons e 19. Can your father read and write? Yes No Don’t know/ no response 20. Can your mother read and write? Yes No Don’t know/ no response 21. Does someone at home help you Yes/always/ Sometimes/ No/never Don’t Et4d ALIGN Cameroon Endline Survey Report 64 with your home work when you need help? (circle) Usually kind of / rarely know/ no response QUESTION ANSWER POINTS 22. How many people stay in your house? (A house is a socio-economic unit of one or more people— regardless of blood or marital relationship—who have lived (or plan to live) together for at least six months out of the previous twelve in the same compound, who share resources, who usually eat together, and who recognize the same head.) -Seven or more -Six -Five -Four -Three -One or two - Don’t know no response 0 10 12 16 20 29 NA 23. Does anyone in your house farm, keep animals, hunt, or fish? -Yes -No - Don’t know no response 0 2 NA 24. What type of floor does your house have? (you may read the answer options to help the pupil to remember, if needed, show the pictures to the child) -Dirt, ground, or other -Wood, cement, or tile - Don’t know no response 0 6 NA 25. What do you usually use for cooking at your house? (you may read the answer options to help the pupil to remember, if needed, show the pictures to the child) -Collected/gifted firewood, or other -Purchased firewood, kerosene/paraffin/petroleum , charcoal, sawdust/wood chips, electricity, or does not cook -Gas - Don’t know no response 0 9 19 NA 26. Does your house have an electric iron? -No -Yes - Don’t know no response 0 6 NA 27. Does your house have a radio or a television? -No -Only radio -Television (regardless of radio) - don’t know/no response 0 7 14 NA 28. Does your house have a wardrobe? (a piece of furniture to store clothes) -No -Yes - don’t know/no response 0 6 NA Comments (optional; to be used if there are unusual answers above): ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________________________________________________ ________________________________________________ Et4d ALIGN Cameroon Endline Survey Report 65 Letter Name Knowledge Show the child the sheet of letters in the student stimuli booklet as you read the instructions. Start the timer when the child reads the first letter. Follow along on your screen and mark any incorrect letters by touching that letter on the screen until it turns blue. Mark self-corrections as correct by touching the screen again until it returns to grey. Stay quiet, except if the child hesitates for 3 seconds. Then point to the next letter and say, “Please go on.” Mark the skipped letter as incorrect. If the student provides the letter sound rather than the name, say: “Please tell me the NAME of the letter”. This prompt may be given only once during the exercise. If the timer runs out before the last item is read, the screen will flash red and the timer will stop. Tell the child to stop. Mark the final letter read by touching it so that a bracket appears. Then press “Next.” If the child reaches the last item before the screen flashes red, STOP the timer as soon as the child reads the last letter. The red bracket will automatically be placed on the last letter. Verify that your scores are correct, then press “Next”. Early stop rule: If the child does not provide a single correct response for the first 10 items, the screen will flash red and the timer will stop. Say, “Thank you!”, discontinue this subtask, and go on to the next subtask. Example: A v L 1 2 3 4 5 6 7 8 9 10 F u e R y a c m V s (10) B Z I H k D J w X P (20) L q O N g T U z M j (30) t G W A l n O Q E f (40) i Y r h C K v p d b (50) x S h m E H A Y l G (60) A d j o a d N S b v (70) R E H S t h i o w F (80) Here is a page full of letters of the alphabet. Please tell me the NAMES of as many letters as you can – not the SOUNDS of the letters, but the names. For example, the name of this letter [point to A] is “A” Let’s practice: Tell me the name of this letter [point to v] If the pupil responds correctly, say: Good, the name of this letter is “VEE” If the pupil does not respond correctly, say: The name of this letter is “VEE” Now try another one: Tell me the name of this letter [point to L] If the pupil responds correctly, say: Good, the name of this letter is “ELL” If the pupil does not respond correctly, say: The name of this letter is “ELL” Do you understand what you are to do? When I say “Begin”, please name the letters as quickly and carefully as you can. Start here and continue this way. [Point to the first letter on the row after the example and draw your finger across the first line] If you come to a letter you do not know, move to the next letter. Ready? 60 second Et4d ALIGN Cameroon Endline Survey Report 66 U T r s I p K s L N (90) T n a t e I r O e c (100) Time remaining on stopwatch at completion (number of SECONDS): Number of Correct Letters: Et4d ALIGN Cameroon Endline Survey Report 67 Letter Sound Knowledge Show the child the sheet of letters in the student stimuli booklet as you read the instructions. Start the timer when the child reads the first letter. Follow along on your screen and mark any incorrect letters by touching that letter on the screen until it turns blue. Mark self-corrections as correct by touching the screen again until it returns to grey. Stay quiet, except if the child hesitates for 3 seconds. Then point to the next letter and say, “Please go on.” Mark the skipped letter as incorrect. If the student provides the letter name rather than the name, say: “Please tell me the SOUND of the letter”. This prompt may be given only once during the exercise. If the timer runs out before the last item is read, the screen will flash red and the timer will stop. Tell the child to stop. Mark the final letter read by touching it so that a bracket appears. Then press “Next.” If the child reaches the last item before the screen flashes red, STOP the timer as soon as the child reads the last letter. The red bracket will automatically be placed on the last letter. Verify that your scores are correct, then press “Next”. Early stop rule: If the child does not provide a single correct response for the first 10 items, the screen will flash red and the timer will stop. Say, “Thank you!”, discontinue this subtask, and go on to the next subtask.. Example: A v L 1 2 3 4 5 6 7 8 9 10 F u e R y a c m V s (10) B Z I H k D J w X P (20) L q O N g T U z M j (30) t G W A l n O Q E f (40) i Y r h C K v p d b (50) x S h m E H A Y l G (60) A d j o a d N S b v (70) R E H S t h i o w F (80) Here is a page full of letters of the alphabet. Please tell me the SOUNDS of as many letters as you can – not the NAMES of the letters, but the sounds. For example, the sound of this letter [point to A] is “AH” Let’s practice: Tell me the sound of this letter [point to v] If the pupil responds correctly, say: Good, the sound of this letter is “VUH” If the pupil does not respond correctly, say: The sound of this letter is “VUH” Now try another one: Tell me the sound of this letter [point to L] If the pupil responds correctly, say: Good, the sound of this letter is “LUH” If the pupil does not respond correctly, say: The sound of this letter is “LUH” Do you understand what you are to do? When I say “Begin”, please say the letter sounds as quickly and carefully as you can. Start here and continue this way. [Point to the first letter on the row after the example and draw your finger across the first line] If you come to a letter sound you do not know, move to the next 60 second Et4d ALIGN Cameroon Endline Survey Report 68 U T r s I p K s L N (90) T n a t e I r O e c (100) Time remaining on stopwatch at completion (number of SECONDS): Number of Correct Letters Sound Et4d ALIGN Cameroon Endline Survey Report 69 Nonword Decoding Show the child the sheet of non-words in the student stimuli booklet. Read the instructions. Start the timer when the child reads the first word. Follow along on your screen and mark any incorrect non-words by touching that word on the screen until it turns blue. Mark self-corrections as correct by touching again until it returns to grey. Stay quiet, except if the child hesitates for 3 seconds. Then point to the next word and say, “Please go on.” Mark the skipped word as incorrect. If the timer runs out before the last item is read, the screen will flash red and the timer will stop. Tell the child to stop. Mark the final letter read by touching it so that a bracket appears, then press "Next". If the child reaches the last item before the screen flashes red, stop the timer as soon as the child reads the last letter. Touch the last letter so the bracket appears, then press "Next". Early stop rule: If the child does not provide a single correct response for the first 5 non-words, the screen will flash red and stop the timer. Say, “Thank you!”, discontinue this subtask, and go on to the next subtask. Example: ut dif mab 1 2 3 4 5 hoy geg pim av ret (5) mog nin tup daw ul (10) saz mup ek pag ril (15) ev gam mib zet fos (20) lig el seb dof lub (25) tid wab hoz ped uf (30) gez daf pid von gax (35) wob ib fap heg nar (40) jek bew nep ut ren (45) faw vub pes gep vad (50) Here are some words with no meaning. Fake means they are not real words. Let’s play like they are real English words. I would like you to read as many as you can. Do not spell the words, but read them. For example, [point to the example word: ut ] this fake word is: “ut” Let’s practice: Please read this word [point to the next word: dif ] If the pupil says “dif”, say: Very good: “dif” If the pupil does not say “dif” correctly, say: This fake word is “dif” 60 second Et4d ALIGN Cameroon Endline Survey Report 70 Time remaining on stopwatch at completion (number of SECONDS): Number of Correct Non-Word Et4d ALIGN Cameroon Endline Survey Report 71 Reading Fluency and Comprehension Show the child the story in the stimuli booklet. Read the instructions to the child and conduct the examples. Start the timer when the child reads the first word. Follow along on your screen and mark any incorrect word by touching that word on the screen - it will turn blue. Mark self-corrections as correct by touching the word again - it will return to grey. Stay quiet, except if the child hesitates for 3 seconds. Then point to the next word and say, “Please go on.” Mark the skipped word as incorrect. If the timer runs out before the last item is read, the screen will flash red and the timer will stop. Tell the child to stop. Mark the final word read by touching it so that a red bracket appears, then press “Next”. If the child reaches the last item before the screen flashes red, stop the timer as soon as the child reads the last word. If the child reaches the last item before the screen flashes red, stop the timer as soon as the child reads the last letter. The bracket will automatically appear on the last word. Verify your scores, and then press "Next". When the reading fluency is complete, REMOVE the stimuli give the following instructions, give the pupil 15 sec to answer the question. Mark the question as ‘no response’, and go to the next question. Now I am going to ask you a few questions about the story you have read. Try to answer the questions as best as you can. Correct Incorrect No Response Bonje likes school. She is happy to walk with Seka to school in the mornings. (15) What does Bonje like? [school, walking with Seka to school] One day, Seka pretends to see a snake. Bonje is afraid. (11) Why did Bonje feel afraid? [Seka made her think there was a snake; she was afraid of the snake] Seka laughs at her. Bonje gets angry and walks faster to be alone. (13) What did Seka do that made Bonje angry? [he laughed at her] Here is a short story. I want you to read it aloud. When you have finished, I will ask you some questions about what you have read. Do you understand what you are to do? When I say “Begin”, read the story as quickly and carefully as you can. I will keep quiet and listen to you. Do you understand what you are to do? Ready? Begin. 60 secon Et4d ALIGN Cameroon Endline Survey Report 72 Early stop rule: If the child gives no correct answers on the first line, the screen will flash red and the timer will stop. Say, “Thank you!”, discontinue this subtask, and go on to the next subtask. No comprehension questions will appear on the next screen. Seka feels bad. In the afternoon, Seka gives Bonje a flower to say sorry. (14) What did Seka give Bonje to say sorry? [a flower] She smiles. They walk home together. (6) How do you know Bonje forgave Seka? [she smiled, she walked home with him] Time remaining on stopwatch at completion (number of SECONDS): Number of Correct Words: Number of Questions Correct: Number of Questions Asked: Et4d ALIGN Cameroon Endline Survey Report 73 Listening Comprehension Read the following passage to the pupil ONLY ONE TIME, slowly (about 1 word per second): Ask the first question below. Mark the pupil’s response as Correct or Incorrect and move on to the next question. Give the pupil 15 seconds to answer each question. If the pupil does not answer within 15 seconds, mark the question as “no response” and move on to the next question. Correct Incorrect No Response Who did Chung go with to fetch water? [Buhven] What did the boys first do at the river? [swam, went swimming] What problem did Buhven have? [he lost his bucket, couldn’t find his bucket] How did the bucket get lost? [they were careless when they got to the river, they were busy playing, they went swimming] Where did Chung find the bucket? [behind the bush] Number of Correct Answers: I am going to read you a short story aloud ONCE and then ask you some questions. Please listen carefully and answer the questions as best you can. Do you understand what you are to do? “Chung and Buhven go to fetch water. They swim in the river. When it is time to go, Buhven can’t find his bucket. Chung finds the bucket behind a bush. The boys bring the water home.” Et4d ALIGN Cameroon Endline Survey Report 74 Closing Remarks This marks the end of the EGRA exam. Thank the pupil for his/her participation by saying, “I want to thank you for working so hard today!” Hand the pupil a pencil and explain that it is a gift in appreciation of his/her effort. Say goodbye to the pupil and send him/her back to class. ANNEX D: STALLINGS Instrument Et4d ALIGN Cameroon Endline Survey Report 75 Classroom Observation Snapshot Please spend five minutes introducing yourself and building rapport with the teacher. Teacher Informed Consent I work with Nascent Solutions, an organization based in Kumbo. I am researching teacher and classroom habits in primary schools in Bui division. I wish to observe your classroom environment and make notes. The information will be used to evaluate ALIGN project and to improve the design of future interventions If you decide to participate you do not need to do anything special. Please continue with your normal teaching duties as if I’m not in the room. Please also encourage your pupils to continue as if I’m not in the room. The observation will take about 50 minutes. You can stop participating at any time. This study likely does not involve any significant risks to you, your pupils, or classroom. You will not receive any compensation for participating, though the information will improve future training in which you may be invited to participate. The information collected will be completely confidential. I will not record your name or the names of your pupils. Information about your individual classroom will not be shared with your head teacher, inspectors, pupils, or their parents. Any report including this information will not identify your specific classroom. The results will be erased from the tablet Participation in this study is voluntary. You have the right not to participate at all or stop in the middle of the observation. Deciding not to participate or choosing to stop early will not result in any anticipated negative result for you, your job, or your relationship with Nascent Solutions. Do you have any questions? Do you give your consent to be observed in your classroom today? Yes No 1. Teacher ID _______________ day month year 2. Date of observation: 3. School name: ___________________________________________ 3a. School Code: 4. Data collector name: ___________________________ 4a. Data collector team #: 6. Class observed (circle): 1 2 3 4 5 6 6b. Are you observing a multi-grade classroom? Yes No Et4d ALIGN Cameroon Endline Survey Report 76 7. Teacher’s gender: Male Female 8. What type of employment contract do you have? Government school – Civil Servant Government school - Contract Governme nt school - PTA Private school - Perman ent Private school - PTA O t h e r 9. What is your level of training? Grade 1 Grade 2 None O t h e r 10. How many years have you taught in this school? __________ years 11. Which training did you receive as part of the ALIGN project in 2016 and 2017  Administration  E-reader  Fantastic Phonics  ICT  In-service training (IST)  Library  Other  None 12. Are you from Bui division? Yes No 13. What languages do you speak?  English  French  Fulfulde  Lamnso  Noni  Oku  Other Girls Boys Total 14. How many pupils are enrolled in this class? 15. How many pupils are present in this class today? 16. Is there a blackboard in the classroom? Yes No 17. Are there any charts, pictures, maps on the wall, etc.? Yes No 18. For the lesson you are observing, what percentage of pupils have textbooks or other printed material? To answer question 18 and 19 ask the pupils to raise their textbooks and notebooks and count how many pupils have it. If the textbooks are with the teacher, ask the teacher to distribute them and then count them 90% or more About 70% About 50% About 30% 10% or less 19. For the lesson you are observing, what percentage of pupils have notebooks or other writing material ? 90% or more About 70% About 50% About 30% 10% or less Comments (optional; to be used if there are unusual answers above): ____________________________________________________________________________________ ____________________________________________________________________________________ ____________________________________________________________________________________ Et4d ALIGN Cameroon Endline Survey Report 77 ____________________________________________________________________________________ ________________________________________________________________ Observe 50 minutes of class and perform 10 snapshots. Finally select all the subjects that you have observed in 50 minutes. 20. Subject(s) observed:  Arabic  Citizenship/Civics  English Language/Reading  Environmental Education  French Language  General Knowledge  Geography  Handiwork/Art + Crafts/creativity  Health Education  History  Home Economics  ICT  Manual Labor  Mathematic  Moral Education  Religion  Sports  Other Et4d ALIGN Cameroon Endline Survey Report 78 Number: ___ Classroom Observation Snapshot Time: Locate the teacher, identify what he/she is doing, what materials are being used, and pupil involvement. Continue to scan the classroom to the right, noting pupil activity until you come back to the teacher. The observation should take around 15 seconds. Circle the appropriate codes below. Note anything out of the ordinary in Additional Comments. T = Teacher P = Pupils 1 = One M = Multiple A = All Material Activity No Material Textbook Notebook Blackboard Learning Aides 1. Reading Aloud T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 2. Demonstration / Lecture T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 3. Discussion / Question & Answer T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 4. Practice & Drill T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 5. Assignment / Seat Work T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 6. Copying T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M 7. Verbal Directions T 1 M A 1 M A 1 M A 1 M A 1 M A P 1 M 1 M 1 M 1 M 1 M T = Teacher P = Pupils 1 = One S = Small Group (5 or less) L = Large Group (6 or more) A = All Activity People Involved 8. Social Interaction T 1 S L A P 1 S L 9. Pupil(s) Uninvolved P 1 S L 10. Discipline T 1 S L A 11. Classroom Management T 1 S L A P 1 S L Et4d ALIGN Cameroon Endline Survey Report 79 12. Classroom Management Alone T 13. Teacher Social Interaction or Teacher Uninvolved T 14. Teacher Out of the Room T Additional Comments: Et4d ALIGN Cameroon Endline Survey Report 80 ANNEX E: DDS Instrument Pupil Assent I work with Nascent Solutions, a group in Kumbo that helps pupils. I am asking what you eat to help pupils become healthier. If you want to take part you will be asked questions about what you and your family ate yesterday. There is no right or wrong answer. It will take about 10 minutes. You can stop at any time. Your answers will be secret. Your teacher and family are aware that you are doing the exercises, but will not know your answers. We will write a report about this, but it will not include your name. Your answers will be erased from the tablet Do you have any questions? Do you have any questions? Are you ready to get started? Yes No Dietary Diversity Please spend two minutes making light conversation with the pupil. Tell the child about yourself, and ask questions about things he/she likes to do, such as: What do you like to play? What do you like to eat? What did you eat this morning? Yesterday’s dietary intake Now I want to talk to you about what you ate yesterday. Can you think of what you ate? I will ask you some questions. If you cannot remember I will help you. [Ask the pupils about what they ate yesterday. Remind them constantly that you are asking about yesterday, for the full day. You do not have to follow a script. Be sensible and think of questions that are necessary to get the information. If they cannot remember, you can help them. Start with colors: ask if they had something green or white, if they had any fruits or snacks, etc. You can also ask them what they normally eat. Only fill out the Dietary Recall sheet during the interview, and do not fill out the final column until you have finished all your interviews for the day.] Example Questions: A. What was the first thing you ate yesterday? a. Can you show me the size? [point to the volume measurements that approximate size. If they show you with their hand or fist, please approximate with the measurements provided.] b. What time was that? [morning, afternoon, evening. Also ask if it was at school, or before] c. Did you have anything to drink? Any fruit? [think of typical morning items] d. Think of any items that were not mentioned but are commonly eaten together [milk with tea, jam on bread, etc.]? e. Fruits, meats, etc.? B. When was the next time you ate? C. Did you eat again in the evening or before bed? D. Did you have any other snacks throughout the day? [picked mangoes from a tree, groundnuts, etc.] E. Did you buy any food on the street? [if they purchased street food, ask how much they paid for it, and who they bought it from] F. Was this a “normal amount” you ate? [We want to make sure that this food intake was like any typical day - no holiday celebration, wedding, or fasting] Et4d ALIGN Cameroon Endline Survey Report 81 Date: ____._____ School:_________________________________________ ___ Child’s Name________________________________________________________________ Tangerine Id: ___.___.___.___.___.___ EGRA Surveyor initial____________________ DDS Surveyor Name ____________________ Number of people in household ___________ Did the pupil participate in mobile reading Yes No How many times did the pupil check out books as part of mobile reading? Number of meals: Group number Food Group Examples Standards Meet Standard Yes/No Any consumed Yes/No 1 cereals, white roots, tubers, and plantains corn, fufu, Irish potato, plantain, white yam, sweet white yam, cassava (gari), cocoyam, rice, bread. 1 cooking spoon 1 medium, 1 slice. 2 yellow or orange flesh vegetables, tubers, oils, and fruits sweet potato, sweet yellow yam, carrots, pepper sauce, pumpkin, squash, sweet red pepper; Fruits: papaya, mango, cantaloupe, 100% fruit juice; Other: red palm oil, yellow soup (red palm) 1 cooking spoon, 1 medium 1/2 cup. For red palm oil, the standard is 1 tbs. For any soup, the standard is 1 cup 3 dark leafy green vegetables Leaves: huckleberry, eru, pumpkin leaf, bitter leaf, spinach, Soup: cocoyam leaf. 1 cooking spoon 1/2 cup. For any soup, the standard is 1 cup. 4 other fruits and vegetables Fruits: pineapple, orange, guava, tree tomato, watermelon, banana, pear, sugar cane Vegetables: young elephant grass, garden eggs, white carrot, onion, garlic, plum, Chinese cabbage, corn; Soups: tomato stew, okro soup, mixed vegetables. 1 cooking spoon 1 medium 1/2 cup. For any soup, the standard is 1 cup. 5 organ meats heart, liver, brain, lungs 1 tbsp 6 flesh meats and fish beef, chicken, fish, snake, bush meat, pork, edible insect 1/2 palm 7 eggs eggs from any bird. 2 medium eggs 8 legumes, nuts, and seeds cowpea, black beans, green beans, peanut butter 1 cooking spoon, 1 pack. Et4d ALIGN Cameroon Endline Survey Report 82 Soup: groundnut, melon seed (equsi), cowpea. For equsi, the standard is 2 tbs. For any soup, the standard is 1 cup 9 milk and milk product milk, cheese, yogurt, cream For liquid milk: 8 tbsp. For condensed: 2 tbsp. For powdered: 1 tbsp. 1-8 Mixed foods Corn and bean, potato and bean 2 cooking spoon Et4d ALIGN Cameroon Endline Survey Report 83 Et4d ALIGN Cameroon Endline Survey Report 84 ANNEX G: Biographical summary of evaluation team Et4d’s evaluation team has extensive experience working on large-scale complex evaluations, including RCTs, for government-funded and multilateral clients such as, the World Bank, USAID, and DFID, and is adept at collaborating with multiple stakeholders from, country offices, agency headquarters, and ministries of education. Project Director and Evaluation Lead Isabelle Duston, MBA Ms. Isabelle Duston, Et4d’s CEO, has 25 years of experience in managerial positions. Her work history is characterized by an ability to handle pressure and creatively solve problems, analyze situations and develop viable solutions and/or alternatives, and integrating technology where effective, while maintaining the highest levels of client satisfaction. Ms. Duston has substantial knowledge of data collection and quality control protocols. She has led the EGRA evaluation for the PEARL project implemented by the World Bank in five countries of the South Pacific (Kiribati, Tuvalu, Tonga, Samoa and the Solomon Islands), which included instrument design, instrument rendering in Tangerine, enumerator training, data collection, data analysis, and result dissemination. For Nascent, she acted as project manager and be the main point of contact and lead the enumerator training and the data collection. Ms. Duston also led the focus groups and ensured all key stakeholders can play an active role in the evaluation process. Senior Evaluation and EGRA Specialist Brenda Sinclair, Ed.D. Dr. Brenda Sinclair is an International Education, Gender and Evaluation Specialist. She brings 18 years of experience designing, implementing and evaluating education development projects across the globe. Dr. Sinclair served as a teacher for five years -- including as a Peace Corps Volunteer in Africa, Senior Education Advisor in Indonesia, and program manager for education programs in Latin America prior to beginning her consulting practice in 2009. Over the past nine years, Dr. Sinclair has led performance evaluations of education and early grade reading programs in Africa, Asia, the Middle East and Latin America for multiple donors, including USAID, DFID, and the World Bank. Throughout her evaluations, she has analyzed results frameworks; designed and pilot tested data collection instruments, including EGRA; conducted interviews, surveys, focus group discussions, and classroom observations; and performed data analysis and report writing. With Et4d, Dr. Sinclair supported the World Bank Pearl project with EGRAs in the Pacific Islands and assisted with five country reports. For the Cameroon final evaluation, Dr. Brenda Sinclair developed the evaluation plan and methodology, developed the qualitative instruments, and drafted the final evaluation report with inputs from the evaluation team. Data Analysis Lead Michel Rousseau, Ph.D. Professor Michel Rousseau has been a university professor since 2005. He has taught psychometrics, quantitative methods, and classroom assessment. Parallel to his work as a professor, he has also worked as a consultant for international development projects in many countries, mainly in Africa (Guinea, Senegal, Nigeria, Morocco, and Zambia). In those projects, he has worked on the development of assessment tests for impact evaluation as well as conducted test equating and other statistical analyses. With Et4d, he performed the data analysis for the PEARL project in the South Pacific for the early grade reading evaluations. Finally, he is the founder of GAUSS, a consultant company in Et4d ALIGN Cameroon Endline Survey Report 85 Quebec that offers services in the research and psychometrics fields. Professor Michel Rousseau guided the Et4d team on all issues related to sampling and conducted the quantitative analysis. EGRA Trainer, Quality Assurance and Survey Coordinator Lead Alice Michelazzi Ms. Alice Michelazzi has led numerous data collection projects using technology in several African countries with leading firms such as Chemonics, Edintersect, STS and Montrose. With STS, she led qualitative instrument design workshops and lead the qualitative data collection using focus groups. In Tanzania she has been part of the baseline data collection for the first phase of the USDA FFE intervention in Mara Region implemented by PCI (2014). She also carried out the midline evaluation (2015). For Et4d, Ms. Michelazzi performed three EGRA enumerator training sessions for the PEARL project, and coordinated the data collection in two countries. As the instrument design assistant, Ms. Michelazzi worked with Ms. Duston, Dr. Sinclair and the NASCENT M&E team to develop or adapt the questionnaires and interview protocols. As the lead trainer, she trained and evaluated the enumerators in Cameroon in collaboration with Ms. Isabelle Duston. As the survey coordinator, she also defineed the quality assurance protocols to be followed by the survey coordinator during data collection and supervise the data collection for EGRA, as well as the focus groups.