IMPAQ International, LLC Page i Beoog Biiga II Midline Report July 2017 McGovern-Dole International Food for Education Beoog Biiga II Program Midline Performance Evaluation Report July 2017 Authors: Elnaz Safarha, M.S. Sarah Pedersen, M.P.P. Edoxi Kindané, M.S. Paula Mian, M.P.P. Submitted to: Chief of Party: Mourad Aidi Catholic Relief Services – Burkina Faso Submitted by: Project Director: Elnaz Safarha IMPAQ International, LLC McGovern-Dole International Food for Education FY 2014 Beoog Biiga II Program Midline Performance Evaluation Report Program: McGovern-Dole International Food for Education and Child Nutrition Agreement Number: FFE-686-2014/047-00 Funding Year: Fiscal Year 2014 Project Duration: Implemented by: Catholic Relief Services Evaluation Authored by: IMPAQ International, LLC Elnaz Safarha Sarah Pedersen Edoxi Kindane Paula Mian DISCLAIMER: This evaluation was conducted by an independent third party firm. The views expressed in this report do not necessarily reflect the views of the United States Department of Agriculture or the United States Government. Accessibility Note: An accessible version of this report can be made available upon request by emailing FAS.monitoring.evaluation@usda.gov. IMPAQ International, LLC Page ii Beoog Biiga II Midline Report July 2017 ACKNOWLEDGEMENT Redacted from public view. IMPAQ International, LLC Page iii Beoog Biiga II Midline Report July 2017 TABLE OF CONTENTS ACKNOWLEDGEMENT............................................................................................................ II TABLE OF CONTENTS ............................................................................................................ III TABLE OF EXHIBITS ..............................................................................................................VII ACRONYM LIST.....................................................................................................................IX EXECUTIVE SUMMARY...........................................................................................................X Outcomes for Key Indicators............................................................................................. xi Additional Key Findings ...................................................................................................xiii Recommendations for the Project....................................................................................xiv SECTION 1. INTRODUCTION................................................................................................... 1 1.1 Program Background.................................................................................................... 1 1.2 Evaluation Background................................................................................................. 3 SECTION 2. EVALUATION METHODOLOGY ............................................................................. 4 2.1 Quantitative Approach................................................................................................. 4 2.1.1 Quantitative Research Questions.................................................................................. 4 2.1.2 Quantitative Sampling Design........................................................................................ 7 2.1.3 Quantitative Data Sources............................................................................................. 8 2.2 Qualitative Approach ................................................................................................. 10 2.2.1 Qualitative Research Questions................................................................................... 11 2.2.2 Qualitative Data Sources.............................................................................................. 14 SECTION 3. FIELD WORK AND ANALYSIS............................................................................... 16 3.1 Quantitative Fieldwork and Analysis .......................................................................... 16 3.1.1 Quantitative Data Collection........................................................................................ 16 3.1.2 Quantitative Analysis................................................................................................... 17 3.2 Qualitative Fieldwork and Analysis............................................................................. 17 3.2.1 Qualitative Data Collection .......................................................................................... 17 3.2.2 Qualitative Analysis...................................................................................................... 18 SECTION 4. EVALUATION SAMPLES ...................................................................................... 19 4.1 Students .................................................................................................................... 19 4.2 Parents...................................................................................................................... 20 4.3. Household Environment............................................................................................ 21 IMPAQ International, LLC Page iv Beoog Biiga II Midline Report July 2017 4.3.1 Household Composition............................................................................................... 21 4.3.2 Household Access to Basic Services............................................................................. 21 4.4 Teachers.................................................................................................................... 22 4.5 Classroom Observations............................................................................................. 23 4.6 Parent-Teacher Associations ...................................................................................... 24 4.7 School District Administrators.................................................................................... 24 SECTION 5. QUANTITATIVE OUTCOMES ............................................................................... 25 5.1 School Outcomes....................................................................................................... 28 5.2 Student Outcomes ..................................................................................................... 28 5.2.1 Nutrition Knowledge.................................................................................................... 29 5.2.2 Hygiene Knowledge and Practices............................................................................... 30 5.2.3 Food Security ............................................................................................................... 32 5.2.4 Student Attendance..................................................................................................... 34 5.2.5 Literacy Activities......................................................................................................... 34 5.2.6 Student Reading Outcomes......................................................................................... 36 5.3 Parent Outcomes....................................................................................................... 38 5.3.1 Nutrition knowledge .................................................................................................... 38 5.3.2 Attitude towards Girls’ Education................................................................................ 39 5.3.3 Parents’ Engagement in School ................................................................................... 40 5.4 Teacher Outcomes..................................................................................................... 42 5.4.1 Nutrition Knowledge.................................................................................................... 42 5.4.2 Attendance................................................................................................................... 43 5.4.3 Classroom Practices and Teaching Techniques ........................................................... 44 5.4.4 Classroom Observations .............................................................................................. 48 5.4.5 Attentiveness............................................................................................................... 53 5.5 PTA Outcomes ........................................................................................................... 53 5.6 School District Administrator Outcomes..................................................................... 55 SECTION 6. QUALITATIVE FINDINGS ..................................................................................... 57 6.1 Relevance .................................................................................................................. 58 6.1.1 Strengths and Weaknesses of Project Design.............................................................. 58 6.1.2 Alignment with Economic, Cultural, and Political Contexts ........................................ 60 IMPAQ International, LLC Page v Beoog Biiga II Midline Report July 2017 6.1.3 Alignment with Burkina Faso and U.S. Government Priorities.................................... 60 6.1.4 Stakeholder Satisfaction .............................................................................................. 60 6.2 Effectiveness.............................................................................................................. 61 6.2.1. Achievement of Objectives......................................................................................... 61 6.2.2 Successes and Challenges of Program Implementation.............................................. 66 6.2.3 Internal Collaboration and Management .................................................................... 67 6.2.4 Collaboration with Project Stakeholders..................................................................... 68 6.3 Efficiency ................................................................................................................... 68 6.3.1 Efficiency of Use of Project Resources......................................................................... 68 6.3.2 Achievement of Project Timeline................................................................................. 68 6.3.3 Response to Factors That Could Hinder Implementation ........................................... 69 6.4 Perceived Impact ....................................................................................................... 69 6.4.1 Perceived Impact of Program ...................................................................................... 70 6.4.2 Activities with the Greatest and Least Impact............................................................. 70 6.4.3 Respondents’ Recommended Strategies to Increase Impact...................................... 71 6.5 Sustainability ............................................................................................................. 72 6.5.1 Sustainability Factors................................................................................................... 72 6.5.2 Current Sustainability Activities................................................................................... 72 6.5.3 Respondents’ Recommended Strategies to Sustain Activities.................................... 73 SECTION 7. CONCLUSION..................................................................................................... 74 7.1 Key Findings........................................................................................................... 74 7.1.1 Students’ Outcomes..................................................................................................... 74 7.1.2 Parents and Parent-Teacher Associations’ (PTA) Outcomes....................................... 75 7.1.3 Teachers’ Outcomes .................................................................................................... 75 7.1.4 Qualitative Findings ..................................................................................................... 76 7.2 Limitations................................................................................................................. 76 7.3 Recommendations..................................................................................................... 77 7.3.1 Recommendations for the Project............................................................................... 77 7.3.2 Recommendations for the Evaluation ......................................................................... 79 7.3.3 Recommendations for the Evaluation with Budget Implications................................ 80 REFERENCE LIST................................................................................................................... 81 IMPAQ International, LLC Page vi Beoog Biiga II Midline Report July 2017 APPENDICES ........................................................................................................................ 82 APPENDIX A. BBII KEY PERFORMANCE INDICATORS ............................................................. 83 APPENDIX B. BBII KEY PERFORMANCE INDICATORS ............................................................. 89 APPENDIX C. ADDITIONAL TABLES AND COMPLEMENTARY OUTCOMES ............................... 97 APPENDIX D. ASER-READING TEST [REDACTED].................................................................. 102 APPENDIX E. ASER-READING TEST INSTRUCTIONS [REDACTED]........................................... 104 APPENDIX F. QUESTIONNAIRES [REDACTED]....................................................................... 106 APPENDIX G. PERFORMANCE EVALUATION SCHOOLS [REDACTED]..................................... 191 IMPAQ International, LLC Page vii Beoog Biiga II Midline Report July 2017 TABLE OF EXHIBITS Exhibit 1. Bam and Sanmatenga Provinces...................................................................................................2 Exhibit 2. Quantitative Research Questions .................................................................................................4 Exhibit 3. Performance Evaluation Sample...................................................................................................8 Exhibit 4. ASER-Reading Test Structure ......................................................................................................10 Exhibit 5. Qualitative Research Domain Descriptions.................................................................................10 Exhibit 6. Qualitative Research Questions..................................................................................................11 Exhibit 7. Respondent Types and Relevant Research Domains..................................................................15 Exhibit 8. Sample Distribution by Province and Type of Respondent ........................................................19 Exhibit 9. Sampled Student Gender Distribution by Grade ........................................................................20 Exhibit 10. Parents’ Educational Attainment..............................................................................................20 Exhibit 11. Household Size..........................................................................................................................21 Exhibit 12. Household Access to Basic Services..........................................................................................21 Exhibit 13. Classroom Size...........................................................................................................................22 Exhibit 14. Teachers’ Educational Attainment............................................................................................23 Exhibit 15. Teacher Training........................................................................................................................23 Exhibit 16. Overview of Baseline and Midline Levels of Key Performance Indicators................................26 Exhibit 17. Student Nutrition Knowledge ...................................................................................................29 Exhibit 18. Student Hygiene Practices ........................................................................................................30 Exhibit 19. Students’ Knowledge of Handwashing versus..........................................................................31 Exhibit 20. Students’ Food Intake ...............................................................................................................33 Exhibit 21. Minimum Acceptable Diet ........................................................................................................33 Exhibit 22. Student Attendance ..................................................................................................................34 Exhibit 23. Student Reported Classroom Activities ....................................................................................35 Exhibit 24. Student Reading Groups...........................................................................................................35 Exhibit 25. Distribution of Reading Skills for Second Grade Students........................................................36 Exhibit 26. Reading Proficiency at Second Grade Level..............................................................................37 Exhibit 27. Parent Nutrition Knowledge .....................................................................................................38 Exhibit 28. Parents’ Opinions of Girls’ Schooling........................................................................................39 Exhibit 29. Parents’ Reasons for Supporting Girls’ Schooling.....................................................................40 Exhibit 30. Parents’ Involvement in School and in Savings Cooperatives...................................................41 Exhibit 31. Parents –Teacher Individual Meetings......................................................................................41 Exhibit 32. Teachers’ Nutrition Knowledge ................................................................................................43 Exhibit 33. Teacher’s Attendance over the Past Three Months.................................................................44 Exhibit 34. Utilization of Classroom Activities for the Day of Data Collection............................................45 Exhibit 35. Teachers Using New Techniques or Tools.................................................................................46 Exhibit 36. Time Spent by Teachers on BBII Literacy Instruction................................................................47 Exhibit 37. Implementation of Literacy Activities.......................................................................................47 Exhibit 38. Observed BBII Techniques Using Reading Groups....................................................................49 Exhibit 39. Observed Reading Lesson Elements .........................................................................................49 Exhibit 40. Key Elements Observed Reading/Decoding Lessons by Grade.................................................50 Exhibit 41. Other Observed Teacher Practices by Grade............................................................................51 Exhibit 42. Students’ Attentiveness Reported by Teachers........................................................................53 Exhibit 43. Parent-Teacher Association Meetings......................................................................................54 Exhibit 44. PTA Representative Involvement in the Past 12 Months.........................................................54 Exhibit 45. Support for Canteens................................................................................................................55 IMPAQ International, LLC Page viii Beoog Biiga II Midline Report July 2017 Exhibit 46. Important Reading Instructions Reported by District Administrators......................................56 Exhibit 47. Gender Composition of Students..............................................................................................97 Exhibit 48. School District Staff’ Educational Attainment...........................................................................97 Exhibit 49. Minimum Acceptable Diet Reported by Parents......................................................................98 Exhibit 50. Teacher Reported Utilization of Classroom Activities for this Week........................................98 Exhibit 51. ASER-Reading Test Results, Disaggregated by Sex (Grade 2) .................................................100 Exhibit 52. Reading Proficiency at Second Grade Level............................................................................101 IMPAQ International, LLC Page ix Beoog Biiga II Midline Report July 2017 ACRONYM LIST AGIR Action, Gouvernance, Intégration, Renforcement ASER Annual Status of Education Report BBI Beoog Biiga I BBII Beoog Biiga II BEPC Brevet d'études du Premier Cycle (Certificate of Studies of the First Cycle) CCEB Chef de Circonscription d’Education de Base (School District Administrator) CRS Catholic Relief Services FAO Food and Agriculture Organization of the United Nations FAVL Friends of African Village Libraries IMPAQ IMPAQ International, LLC MENA Ministère de l’Education Nationale et de l’Alphabétisation (Ministry of Basic Education and Literacy) OCADES Organisation Catholique pour le Développement et la Solidarité PTA Parent-Teacher Association PMP Performance Monitoring Plan RCT Randomized Control Trial SILC Savings and Internal Lending Community TOR Terms of Reference USDA United States Department of Agriculture IMPAQ International, LLC Page x Beoog Biiga II Midline Report July 2017 EXECUTIVE SUMMARY This report presents the midline results of the performance evaluation of the Beoog Biiga II (BBII) McGovern-Dole International Food for Education and Child Nutrition program in the provinces of Bam and Sanmatenga in Burkina Faso. In this report, we measure changes in outcomes related to BBII core objectives, examine implementation of the BBII program thus far, and provide recommendations to strengthen the program and the evaluation. BBII is a four-year program (2014 – 2018) implemented by Catholic Relief Services (CRS) and funded by the United States Department of Agriculture (USDA). In BBII, CRS aims to improve the literacy1 , health, and dietary practices2 of primary school-age children in Burkina Faso. BBII implementation activities include provision of school meals and take-home rations; distribution of deworming pills, vitamins, and minerals; health and nutrition training; provision of school supplies and materials; school district administrator training; capacity building at the local, regional, and national levels; mentorship of girls; teacher training; creation of community saving and lending groups; establishment of a standardized reading assessment; establishment of libraries; and efforts to raise awareness of the importance of education. CRS selected IMPAQ International, LLC (IMPAQ) to conduct impact and performance evaluations of the BBII program. The performance and impact evaluations were designed in parallel to maximize comparability in the outcome indicators and findings, but they follow different timelines. The impact evaluation, which measures the causal effect of the mentoring intervention on literacy of school-aged girls in a subset of BBII program schools, spans only two data collection periods: baseline (2015) and follow up (2018). The performance evaluation spans three data collection periods: baseline (2015), midline (2017), and endline (2018). The baseline, midline, and endline rounds of the performance evaluation are structured to measure changes in outcomes over time and to inform the overall evaluation results on BBII core objectives. To accurately capture program performance over time, IMPAQ measures the same program indicators at all three data collection points. The key research questions for performance evaluations are: • To what extent has students’ knowledge of nutrition and dietary practices changed? • To what extent have students improved their hygiene-related practices? • To what extent has students’ knowledge of health and hygiene practices changed? • To what extent do the program interventions help address student hunger and attentiveness? • What percentage of children receiving a minimum acceptable diet has changed, if any, compared to baseline? • What is the current student attendance rate? Has the percentage of students (girls and boys) that regularly (80%) attend schools changed compared to the baseline? • What percentage of students (boys and girls) have increased their reading comprehension compared to baseline? • Have parents who are members of saving and internal lending communities used their savings on education costs? • Has the parents’ level of contribution to the school canteen changed? • To what extent has teachers’ attendance changed? • How much time per day do teachers devote to literacy instruction? 1 Strategic Objective 1: Improved Literacy of School-Age Children 2 Strategic Objective 2: Increased Use of Health and Dietary Practices IMPAQ International, LLC Page xi Beoog Biiga II Midline Report July 2017 • To what extent are teachers implementing literacy teaching techniques acquired through the project? To answer evaluation questions and provide evidence addressing the McGovern-Dole indicators, IMPAQ conducted the following quantitative data collection activities: ▪ Student survey including a reading assessment (the Annual Status of Education Report, ASER3 ) ▪ Parent survey ▪ Teacher survey ▪ Parent-teacher association (PTA) leader survey ▪ School district administrator survey The survey questionnaires were designed to ask questions about health, nutrition, literacy, and community involvement in schools. IMPAQ also integrated a complementary qualitative analysis by conducting key informant interviews and focus group discussions to address some of the limitations of the quantitative methods, to provide contextual understanding of the program, and to aid in interpretation of the quantitative results. The qualitative study assessed the following principles: (1) the relevance of all interventions, (2) the effectiveness of implementation strategies and activities, (3) the efficiency of the project, (4) the impact of the project, and (5) the likely sustainability of initiated actions. In addition, the team conducted classroom observations to assess implementation and uptake of project activities and triangulated this data with the qualitative and survey findings. This report presents the midline levels of key project indicators and measures progress in their achievement over time. At midline, IMPAQ collected data in May 2017 on more than 500 variables from 258 primary school students in grades 2 to 6, 262 parents, 105 teachers, 22 PTA members, and 30 school district administrators. The data provide interesting insights into student, parent, and teacher knowledge of nutrition and hygiene, student academic performance, and community involvement. The data also point to the need for projects like this one to improve food security, hygiene and nutrition knowledge, and student literacy. Key findings are summarized below. Outcomes for Key Indicators Below is a snapshot of the key outcomes for midline performance evaluation. More details are provided in Section 5, and Section 6. Student Outcomes • The proportion of students who achieved a passing score on a food nutrition test increased by 2 percentage points from baseline to midline (statistically significant at the 5 percent level) but remained low (3 percent). Surveyed data reported by teachers show that teachers are well￾informed on knowledge necessary to increase students’ nutrition awareness. However, low 3 The Annual Status of Education Report (ASER) Center pioneered in 2005 a nationwide survey, composed of a reading and math test, to measure the achievements of children in primary school in rural India in reading and math. Since their inception, adapted versions of the tests have been implemented in Pakistan, Kenya, Mali, Senegal, Tanzania, Uganda, Ghana, and now Burkina Faso (Pratham 2015; USAID 2012). IMPAQ International, LLC Page xii Beoog Biiga II Midline Report July 2017 nutrition knowledge of students may suggest that teachers need support on how to teach it effectively. • There was a slight (4 percentage point) increase in the proportion of children who reported washing their hands at critical moments compared to baseline although the change was not statistically significant. The proportion of children who achieved a passing score on a test of good health and hygiene practices remained relatively low with no significant change at midline (4 percent). The lack of significant changes from baseline to midline could be explained by the fact that some schools lack reliable water sources and handwashing devices sometimes fall into disrepair, as reported by school-level interviewees. • At midline, the proportion of children who reported that they were hungry during the school day (after eating breakfast before going to school) remained low (7 percent) with only a slight increase (3 percentage points) from baseline. However, the change was not statistically significant. Although this increase is trivial, it could be explained by fewer children (4 percentage points) eating before going to school (not statistically significant at any level) at midline. Students most commonly cited insufficient food (81 percent) as the reason why they could have eaten more after breakfast before going to school at midline. • The percentage of children who consumed a minimum acceptable diet increased significantly from baseline to midline. Students reported an 11 percentage point increase, and parents reported a 20 percentage point increase (statistically significant at the 1 percent level). The increase in consumption of a minimum acceptable diet was slightly higher for boys than for girls. This increase in the diversity of household’s diet might be affected by potential different socio￾economic backgrounds between baseline and midline samples or seasonality. • Almost all students attended school regularly (at least 80 percent of the time). However, there was a slight decrease in children’s regular attendance from baseline to midline (by 1 percentage point); the result was marginally statistically significant (Statistically significant at the 10 percent level) at conventional levels. Boys’ attendance decreased by 2 percentage points at midline, a slightly greater decrease than in girls’ attendance. The fact that student attendance data were collected from teachers who might not record attendance consistently could confound measurement of the attendance rate. • The proportion of second-graders with reading proficiency at grade level increased by 9 percentage points from baseline to midline. The improvement in reading proficiency was higher for girls (15 percentage points) than for boys (4 percentage points). However, the increase was not statistically significant at conventional levels for either group. Parent and Parent-Teacher Association Outcomes • At midline, parents’ participation in community savings groups established as part of the BBII program increased by 32 percentage points compared to baseline (statistically significant at the 1 percent level). Although more parents reported at midline that they used their savings to cover school expenses, fewer parents said that the savings helped “a lot” with school expenses, with a significant 33 percentage point decrease compared to baseline. On the other hand, the proportion of those who said that the savings helped “some” increased by 38 percentage points. Both findings are statistically significant at the 5 percent level. A possibility of increase in education expenses or a decrease in parents’ purchasing power to afford school costs might be potential explanations to this shift in parents’ responses. IMPAQ International, LLC Page xiii Beoog Biiga II Midline Report July 2017 • The total number of months in which the canteen was covered by MENA, CRS, parents, and other responsible parties decreased by 2 months. This finding is statistically significant at the 10 percent level at midline. This outcome should be interpreted with caution, because there is a possibility that canteen operations were misrepresented at baseline Teacher Outcomes • The proportion of teachers who attended at least 90 percent of school days did not change significantly from baseline to midline, increasing only slightly by 3 percentage points. • Teachers in target schools employed new teaching techniques at a lower rate: 44 percent at midline compared to 53 percent at baseline. This slight, statistically insignificant decrease should be interpreted with caution, because the training component of the project had not started at baseline. More than half of the surveyed teachers at baseline reporting using new techniques may suggest that teachers were already aware of these techniques even in the absence of the training. It is also important to note that teachers at midline were less educated compared to teachers at baseline. As discussed in Section 6.2, teachers with limited education found the training materials more difficult to understand, which may have led them to continue using traditional approaches— approaches they were more accustomed to using. • A larger proportion of teachers at midline than at baseline reported that they devoted at least 45 minutes a day to literacy instruction, specifically reading instruction (including, for example, phonological awareness, phonetics, and vocabulary). However, this increase (11 percentage points) was not statistically significant at any conventional levels. • Teachers identified a slight increase (3 percentage points) in the proportion of students who were attentive during class from baseline to midline. This change was not statistically significant at conventional levels. However, given the difficulty and subjectivity of measuring attentiveness, this outcome should be interpreted with caution. Additional Key Findings Classroom observation outcomes were more or less in line with the results of student and teacher surveys. In observed lessons, teachers mostly focused on reading skills, especially decoding. Below is a summary of the main findings by the qualitative research domain. Summary of Main Qualitative Findings Relevance Overall, respondents reported satisfaction with the project, saying they that believed the project interventions are meeting the needs of beneficiaries and are aligned with Burkina Faso’s and the U.S. government’s development objectives. Respondents also believed that the project adequately considers economic, cultural, and political contexts. According to respondents, the primary strengths of the project design include its holistic approach to child development, its focus on evidence-based literacy techniques and provision of teaching resources, and its focus on community and government engagement. The primary weaknesses, according to stakeholders, include little ability to deal with lack of motivation on the part of teachers, lack of key equipment and resources for teacher monitoring and support, and the design of reading clubs. Effectiveness IMPAQ International, LLC Page xiv Beoog Biiga II Midline Report July 2017 Summary of Main Qualitative Findings Respondents reported that immediate outcomes were reached in most dimensions of the project and that the interventions have contributed to expected results. Teachers reported that they have observed improvements in students’ literacy skills and motivation to learn. Teachers and students also reported that hygiene education has changed students’ behavior, as they are now washing their hands more frequently. Students appreciated the school meals and take-home rations received from the canteens, and teachers believed school attendance was closely tied to the presence of canteens. Though challenges exist, respondents said that CRS has effectively managed partners and that the program has coordinated and collaborated effectively with other stakeholders. Efficiency Respondents reported that the project resources have been sufficient to implement scheduled activities and, for the most part, objectives have been achieved on time. Some activities, including the provision of food in canteens and the provision of training materials, have been delayed due to slow administrative processes. Further, respondents noted that partners’ competing agendas and priorities sometimes exacerbate these delays. Despite challenges, respondents reported that CRS has adequately responded to internal and external factors that have hindered the efficient implementation of project activities. Perceived Impact Respondents believed that the project is on target to meet its medium- and long-term goals, particularly those related to students’ literacy outcomes, and that the effects are due to the project interventions. As for hygiene-related outcomes, respondents noticed positive changes at some schools related to students’ hygiene habits. Respondents believed that, if delivered properly, the educational activities related to hygiene and nutrition will strengthen students’ skills and knowledge. However, respondents described that limitations such as teachers’ lack of motivation to teach hygiene and nutrition knowledge and the lack of clean water sources and latrines may result in less significant outcomes. Respondents recommended several strategies to improve the impact of the project. Recommendations related to literacy included modifications to the teacher trainings to ensure that they are long enough to cover all content and allow sufficient time for practice, development of a motivational framework for teachers that focuses on incentives to implement new literacy techniques, and provision of sufficient materials and resources to enable teachers and students to implement these techniques. Recommendations related to health activities included engaging more teachers in health trainings to further promote the spread of information and ensuring that infrastructure, such as handwashing stations, are operational. Sustainability Respondents believed that the engagement of communities and the government are key factors to sustainability and that CRS and its partners have successfully engaged these parties. There is still work to be done, however, to ensure project activities can be sustained after USDA funding ends. Respondents reported a need to organize a workshop with all relevant players to develop sustainability strategies and define how all parties can play a role in sustainability. Recommendations for the Project Below is a summary of our recommendations to CRS based on both lessons learned from our experience in the field and our findings after analyzing the collected data. The recommendations are grouped by category. A complete list of recommendations is provided in Section 7. Project Management and Sustainability • Reevaluate how project activities are planned and communicated to ensure that relevant partners are available to conduct activities. Some project implementers noted that there have IMPAQ International, LLC Page xv Beoog Biiga II Midline Report July 2017 been instances in which project activities have been delayed because a partner is simultaneously rolling out other activities and is therefore unavailable. • Reevaluate staffing plan to ensure enough capacity to implement and monitor projects effectively and efficiently. Some implementers reported that the limited size of their staff sometimes impacted their ability to roll out field activities and effectively monitor their activities. When developing activity roll-out and monitoring plans, implementers should take into account the realities of staffing and resource limitations. • Provide additional training to parents on use of student performance tools and to SILC groups on management. Respondents described that students’ parents experience challenges with the use of performance tools distributed by teachersthrough a pilot activity at five of the BBII schools. Though these parents received training to read and use these tools to support their children, respondents explained that many of these parents still do not know how to interpret the tools. If CRS plans to sustain and/or expand this activity to other schools, they should consider enhancing the training to ensure that parents receive the support they need to effectively interpret and use the tools. Respondents also reported that SILC groups often struggle with management of group activities and require further support to run the groups effectively. Implementers should reevaluate their community sensitization and support structure to ensure that stakeholders are receiving the support and guidance they need to effectively initiate and manage project activities. • Reevaluate the sustainability strategy and develop a robust plan to engage all stakeholders. Respondents believed that the engagement of communities and the government are key factors to sustainability and that CRS and its partners have successfully engaged these parties. However, there is still work to be done to ensure that project activities can be sustained after USDA funding. CRS should continue engaging stakeholders and organize additional workshops with all relevant players to develop sustainability strategies and define how all parties can play a role in sustainability. Literacy Activities • Explore methods to increase teacher motivation and engagement. Interview respondents reported that teacher motivation is lacking in many schools. In order to motivate teachers to implement new teaching techniques, stakeholders could consider developing a framework in which teachers are incentivized for using the new techniques and tools in their classrooms. For example, during observation visits, school district administrators should continue evaluating teachers’ use of the techniques using a standardized evaluation process and assessing the level of students’ literacy skills using a standardized test. Teachers who excel could be offered awards for their performance. Further, some respondents reported that some teachers have refused to participate in CRS trainings because they believe the per diem payment for training is inadequate to cover all of their expenses. Implementers could consider distributing a short survey to obtain feedback directly from teachers about factors that motivate or dissuade teachers from participating in trainings, and use this feedback to modify practices and policies. • Consult with teachers to reevaluate length, timing, and content of teacher training. Interview respondents reported that trainings are too dense, causing teachers to be overwhelmed by the amount of information. Implementers may want to consider pilot testing a different training approach, in which teachers participate in two or three mini-training sessions that focus on different topics or techniques so as not to burden teachers with information overload. Teachers desired more time to practice the literacy techniques with their peers during trainings, and felt the trainings were lecture heavy. Some teachers also reported that they received literacy training late in the school year, giving them little time to apply the new approaches in their classrooms. Further, interview respondents noted that some teachers have a difficult time understanding the IMPAQ International, LLC Page xvi Beoog Biiga II Midline Report July 2017 training materials and therefore are not able to successfully implement the new techniques and tools in their classrooms. Implementers should obtain teachers’ input to reevaluate the length, timing, and content of teacher training to ensure that the trainings meet teachers’ needs. • Collaborate with MENA to ensure that teacher observations are fully meeting their objectives. For example, more real-time, hands-on support to teachers is needed through the use of more frequent student testing and summarized feedback. Teachers reported that they appreciated the feedback they received from administrators during observations but desired more detailed, data￾driven feedback. Implementers could consider implementing the ASER test more frequently and use the data to inform action plans with teachers that focus on ways to use the new teaching techniques and tools and support to teachers who face challenges. Implementers could consider having school-level administrators play a larger role in supporting teachers on a day-to-day basis. Further, the quantitative findings showed that teachers’ use of new techniques decreased by 9 percentage points over time, though the finding was not statistically significant. Implementers may want to use the observations as an opportunity to understand why teachers are not using the techniques, and how trainings and observations can be strengthened to better support teachers. Additionally, observation visits must occur early in the school year, as opposed to toward the end of the year, to give teachers adequate time to adjust based on the received feedback. • Reassess material distribution plan and further investigate why teachers are reluctant to use some tools. Teachers reported that they often did not have enough supplies to effectively use them in their classrooms. Implementers should ensure that teachers have more adequate quantities of materials to properly implement techniques. For example, teachers should receive one slate for every 5-7 students. Further, none of the teachers in classroom observations used Bananagrams, and word strips, and a very few of them used wood cubes, or phonemic tables as part of their lessons. In interviews, teachers reported only using large slates and blackboards. During trainings and observations by school district administrators, implementers may want to explore if and why teachers may be reluctant to use some tools and how trainings and observations can be strengthened to support teachers’ use of these materials. • Investigate why teachers’ use of new teaching techniques has decreased over time. At midline, teachers were less likely than at baseline to apply or plan to apply at least five of the seven teaching techniques on the day of data collection. Although the difference is not statistically significant, CRS might want to investigate why this may be the case and adjust trainings and teacher support accordingly. • Reconsider design of reading clubs with a greater focus on local context. Interview respondents described several challenges with the reading club design, including the difficulty to find qualified volunteers to facilitate groups and the fact that many parents require that their children help with domestic activities rather than attend reading club meetings. Implementers should reassess how the reading club component can be better tailored to the culture and stakeholders’ priorities. • Exhort school administrators to keep consistent records of teacher attendance and monitor data collection. Although the quantitative data showed high teacher attendance, the quality of the attendance data is uncertain given the sensitive nature of teacher absenteeism at schools. Some school district administrators or school administrators do not keep a record of absentees, and respondents reported varying levels of absenteeism. Given the uncertainty of the attendance data, implementers should work with schools to develop more robust tracking systems and conduct spot checks to confirm that schools are collecting the data. Health and Dietary Activities IMPAQ International, LLC Page xvii Beoog Biiga II Midline Report July 2017 • Work with schools to ensure that schools’ hand washing stations are in working order. Some respondents noted that hand washing stations are sometimes out of use and there is no effort made from the school nor the PTA to replace them. Implementers could develop a system whereby a designated teacher or the school-level administrator is responsible for managing the hand washing stations and reporting to CRS and its partners when it is out of order. • Further investigate why students’ hygiene and nutrition knowledge remains poor despite training. Very few children achieved a passing score on the test of hygiene knowledge and students’ level of nutrition knowledge remains very low. This area requires further probing by CRS to explore why student knowledge of good health and hygiene practices and nutrition are so low despite the training. A deeper analysis of why knowledge is so low would inform improvements to the program’s hygiene and health interventions, ideally improving students’ knowledge over time. • Investigate health issues related to dietary supplements. A few interviewed students reported that they stopped taking the micronutrients that teachers gave them because it caused them to experience nausea and/or headaches. Because the intensity of this issue is unknown, implementers should consult with students and parents to better understand why students are reacting this way to the dietary supplements and adjust as needed. IMPAQ International, LLC Page 1 Beoog Biiga II Midline Report July 2017 SECTION 1. INTRODUCTION Boog Biiga II (BBII) is a four-year project (2014 – 2018) implemented in the Bam and Sanmatenga provinces in Burkina Faso and funded by the United States Department of Agriculture (USDA) McGovern-Dole International Food for Education program. The project extends and expands on the previously implemented Beoog Biiga I (BBI) project. The purpose of this report is to assess the progress of BBII in achieving the desired project outcomes. The report also describes potential additional questions to be asked at endline for both the performance and impact evaluations and provides details about the implementation of BBII. This report consists of seven sections. This introduction (Section 1) provides a brief overview of the program context for the performance evaluation. Section 2 outlines the performance evaluation methodology, including research questions, sampling design, and data tools. In Section 3, we describe field work for data collection and data analysis. Section 4 describes the sample respondents and their key characteristics. Sections 5 and 6 present the quantitative and qualitative outcomes. Section 7 concludes with lessons learned, study limitations, and recommendations. 1.1 Program Background Catholic Relief Services (CRS), in partnership with the government of Burkina Faso and local organizations, implemented the BBI program, Tomorrow’s Child, from 2011 to 2014 in response to recurrent food crises, high levels of malnutrition, and low and unequal levels of education in the central north region of Burkina Faso. The program was funded by the International Child Nutrition and Food for Education program, which is managed by the Foreign Agricultural Service of the USDA. BBI has been expanded to BBII, a four-year project (October 2014 – August 2018) valued at $21,601,854 million, which includes the delivery of 8,910 metric tons of commodities: soy-fortified bulgur, lentils, cornmeal, and vegetable oil. In BBII, CRS continues to provide assistance to improve the literacy, health, and dietary practices of school-age children in Burkina Faso. BBII serves all primary schools (more than 900) and community-led preschools (bissongos) in the provinces of Bam and Sanmatenga in the central north region of Burkina Faso (Exhibit 1). The project is based on a participatory approach that reinforces the linkages between communities and schools. This approach aims to ensure program sustainability and to contribute to the achievement of the Burkina Faso education strategic plan and of the United Nations’ Millennium Development Goal objectives. IMPAQ International, LLC Page 2 Beoog Biiga II Midline Report July 2017 Exhibit 1. Bam and Sanmatenga Provinces For the implementation of BBII, CRS is partnering with the Ministry of Basic Education and Literacy (MENA), Organisation Catholique pour le Développement et la Solidarité (OCADES), and Friends of African Village Libraries (FAVL); it is also collaborating with the Burkina Faso Ministry of Health and Ministry of Social Action. CRS is leading the project’s implementation activities, including provision of school meals and take-home rations; distribution of deworming pills, vitamins, and minerals; health and nutrition training; provision of school supplies and materials; school district administrator training; capacity building at the local, regional, and national levels; mentorship of girls; teacher training; creation of community saving and lending groups; establishment of a standardized reading assessment; establishment of libraries; and efforts to raise awareness of the importance of education. CRS expects to reach approximately 387,000 direct beneficiaries by implementing these activities over the life of the project, distributed as follows: • 270,000 primary and preschool students (between 170,000 and 200,000 per year) • 3,600 teachers • 90 preschool caregivers • 9,900 parent-teacher association (PTA) members, school management committee members, and cooks • 51,120 savings and internal lending community (SILC) members • 2,250 community mentors • 20,000 community members (using libraries) • 40 local government officials (i.e., county mayors and deputy mayors) • 30,000 parents • 80 school administrators: district-level inspectors and pedagogical advisors IMPAQ International, LLC Page 3 Beoog Biiga II Midline Report July 2017 1.2 Evaluation Background The performance and impact evaluations were designed in parallel to maximize comparability in the outcome indicators and findings, but they follow different timelines. The impact evaluation measures the causal effect of the mentoring intervention on literacy of school-aged girls in a subset of BBII program schools. It spans only two data collection periods: baseline (2015) and follow up (2018). The performance evaluation spans three data collection periods: baseline (2015), midline (2017), and endline (2018). The baseline, midline, and endline rounds of the performance evaluation are structured to measure changes in outcomes over time and to inform the overall evaluation results on BBII core objectives. To accurately capture program performance longitudinally, IMPAQ measures the same program indicators at all three data collection points. This report describes the midline findings for the key performance indicators and studies their change since baseline (May–September 2015). The key performance indicators are the following: • Percentage of students in target schools, by gender, who demonstrate that they can read and understand the meaning of grade-level text by the end of two grades of primary schooling • Percentage of teachers who devote an average of at least 45 minutes a day to literacy instruction • Percentage of teachers in target schools who attend and teach school at least 90 percent of scheduled school days each year • Percentage of school district administrators who demonstrate use of new techniques or tools as a result of USDA assistance • Percentage of teachers, educators, and teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance • Percentage of students in target schools who are identified as attentive during class • Percentage of students in target schools who indicate that they are hungry or very hungry during the school days • Percentage of students, by gender, who regularly (80 percent of the time) attend USDA-supported classrooms and schools • Percentage of school-age children, by gender, who receive a minimum acceptable diet • Percentage of students who wash their hands at critical moments • Percentage of students in target schools who achieve a passing score on a test of good health and hygiene practices • Percentage of students in target schools who achieve a passing score on a test on food nutrition and dietary practices • Number of months of community and/or government support for school canteens To evaluate these key indicators and measure progress toward achieving them, IMPAQ collected survey data using the same instruments used at baseline. In addition, IMPAQ used qualitative key informant interviews, focus group data, and classroom observations to assess the relevance of BBII interventions; the appropriateness of BBII strategies; project management structures; and the effect of activities on project effectiveness, efficiency, impact, and sustainability. The same quantitative indicators will be collected and reported at endline, but some qualitative performance evaluation questions about lessons learned and sustainability will be different at endline. At midline, the qualitative research questions focused on learning what has occurred to date, including a formative assessment and suggestions for program process improvement. At endline, the qualitative research questions will focus on learning more about the potential for program sustainability and about promising practices and lessons learned. IMPAQ International, LLC Page 4 Beoog Biiga II Midline Report July 2017 SECTION 2. EVALUATION METHODOLOGY 2.1 Quantitative Approach The performance evaluation measures changes in outcomes related to BBII core activities over the life of the project. To accurately reflect program performance over time, IMPAQ will measure the program indicators outlined in Section 1, using the same methodology and sampling strategy at all three data collection points — baseline, midline, and endline — and then compare their values. 2.1.1 Quantitative Research Questions The research questions for the quantitative performance evaluation of BBII focus on literacy, health, and nutrition outcomes for various beneficiaries, including teachers, students, and PTA members, with an emphasis on improving outcomes for girls. These research questions are derived from the program objectives, as well as guidelines from the McGovern-Dole International Food for Education program. To answer the research questions, the performance evaluation needs to develop evidence addressing the indicators described in Exhibit 2. Quantitative Research Questions. Exhibit 2 maps the strategic objectives (SO) of the CRS performance monitoring plan (PMP) to the quantitative research questions, their corresponding indicators, and the sources from which IMPAQ will collect data in accordance with the approved terms of reference (TOR). This exhibit summarizes the research questions; Appendix A4 provides a detailed table mapping all the research questions to the strategic objectives. Exhibit 2. Quantitative Research Questions Strategic Objectives and Results Research Questions Project Performance Indicators Data Source SO1. Improved Literacy of School-Age Children ▪ What percent of students (boys and girls) have increased their reading comprehension compared to baseline? Percentage of students (boys and girls) who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text. a Student literacy assessment. IR1.1. Improved Quality of Literacy Instruction ▪ To what extent are teachers implementing literacy teaching techniques acquired through the project? ▪ How much time per day do teachers devote to literacy instruction? Number of teachers, educators, and teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance. a Teacher survey, student survey, district administrator survey, and classroom observations. 4 In the approved terms of reference and performance monitoring plan (Appendix A) the data collection is referred to as “evaluation” when IMPAQ is collecting the data through qualitative and/or quantitative methods and as “CRS/monitoring” when CRS is responsible for collecting monitoring data on performance indicators. There are overlaps in some questions for which both IMPAQ and CRS will collect data. IMPAQ International, LLC Page 5 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Research Questions Project Performance Indicators Data Source ▪ ▪ To what extent teachers’ attendance has changed? ▪ To what extent are school administrators implementing training teaching techniques and pedagogical accompaniment and teacher monitoring/feedback techniques acquired through the project? Number of teachers who devote an average of at least 45 minutes a day to literacy instruction. a Number of teachers in target schools who attend and teach school at least 90% of scheduled school days per year. a Number of school administrators and officials who demonstrate use of at least one new technique, skill, or tool as a result of USDA assistance. a IR1.2. Improved Attentiveness Output 1.2.1. Reduced Short￾Term Hunger ▪ To what extent do the program interventions help address student hunger and attentiveness? Percentage of students in target schools who indicate that they are hungry or very hungry during school daysa . Percentage of students in target schools who are identified as attentive during class/instruction by their teachers. Student survey, teacher survey. IR1.3. Improved Student Attendance ▪ What is the current student attendance rate? Number of students (boys and girls) who attend USDA￾supported classrooms and Student attendance logs, IMPAQ International, LLC Page 6 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Research Questions Project Performance Indicators Data Source ▪ Has the percent of students (girls and boys) that regularly (80%) attend schools changed compared to the baseline? ▪ Have parents who are members of saving and internal lending communities used their savings on education costs? schools at least 80% of the time (boys and girls). a Number of group members that spend money from SILC on education costs a . Parent survey, PTA survey Output. 1.3.5. Increased Community Understanding of Benefits of Education ▪ Has the parents’ level of contribution to the school canteen changed? Number of months of community and/or government support for school canteens a . PTA survey. IR2.1 Improved Knowledge of Health and Hygiene Practices ▪ To what extent has students’ knowledge of health and hygiene practices changed? ▪ To what extent have students improved their hygiene-related practices? Number of students in target schools who pass a test of good health and hygiene practices by naming at least 4 situations in which they should wash their hands a . Percentage of schoolchildren who wash their hands at critical moments: before eating and after using the latrine a . Student survey, classroom observations. IR 2.3 Increased Knowledge of Nutrition ▪ To what extent has students’ knowledge of nutrition and dietary practices changed? ▪ What percent of children receiving a minimum acceptable diet has changed, if any, compared to baseline? Number of students in target schools who pass a test on nutrition and dietary practices by naming at least one food with iron and one with vitamin A and by naming one benefit of each a . Percentage of school-aged children (boys and girls) who receive a minimum acceptable diet (boys and girls). a Student survey. IMPAQ International, LLC Page 7 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Research Questions Project Performance Indicators Data Source IR 2.5 Increased Access to Preventative Health Interventions ▪ To what degree has students’ knowledge about vitamin A, iron and deworming medication changed since baseline? Percentage of students who have heard of vitamin A and iron, can cite a benefit of each, and know a food that contains each. Student survey. FR 4: Output1.4.4 & 2.7.4 Increased Engagement of Local Organizations and Community Groups ▪ To what extent are PTAs holding regular meetings? To what degree is this frequency different from baseline? Number of general assemblies and PTA council meetings held in the past school year. PTA survey. Source: BBII Terms of Reference (TOR) and Performance Monitoring Plan (PMP); a Key performance indicators. To complete the analysis of the key research questions, we also examined some non-key indicators as follows: • Number of libraries observed during the data collection nearby schools • Percentage of observed lessons in classrooms where teachers use learning materials in their teaching sessions. • Percentage of students participating in reading clubs • Percentage of schools observed with appropriate food storage conditions. • Percentage of parents reporting educating girls improves living standards 2.1.2 Quantitative Sampling Design At baseline, in light of budget and time constraints and with USDA’s approval, IMPAQ and CRS agreed to overlap the sample for the impact and performance evaluations and to limit the performance evaluation sample to the 22 control schools in the impact evaluation sample. The impact evaluation uses a randomized control trial (RCT) design in which 22 treatment schools (out of 44 schools) have been assigned to receive the BBII mentoring program. The performance sample excludes these treatment schools, so that the changes in outcomes examined in the performance evaluation can be associated only with the other BBII components and not the mentoring program. Since the performance evaluation focuses only on progress toward general outcomes such as literacy, nutrition, and hygiene, IMPAQ is not concerned with tracking individual students over time. Unlike the sampling strategy for the impact evaluation, for the performance evaluation we select a new student sample of the same size at all three data collection points (baseline, midline, and endline). Exhibit 3 shows the performance evaluation sample. IMPAQ International, LLC Page 8 Beoog Biiga II Midline Report July 2017 Exhibit 3. Performance Evaluation Sample Category Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Number of Individuals Selected per School Total Number of Individuals in All 22 Schools Girls 2 1 1 1 1 6 132 Boys 2 1 1 1 1 6 132 Parents 4 2 2 2 2 12 264 Teachers 1 1 1 1 1 5 110 PTA 1 1 22 As shown in Exhibit 3, for the midterm evaluation we randomly selected students in grades 2 to 6 5 in all 22 control schools from school lists shared by CRS. (See Appendix G for the full list of school names.) We included four second-graders (two girls and two boys) as well as one boy and one girl from each higher grade, for a total of six boys and six girls in each school. To ensure a large enough sample size, we also randomly sampled extra students (four boys and four girls for grade 2, and two girls and two boys for grades 3 to 6) to serve as replacements for students who might not be in school on the day of data collection. The sample also includes parents of each of the children, including the replacements. In addition, teachers of students in grades 2 to 6 in all 22 schools, as well as one PTA representative6 in each school, were included in sample. However, before the data collection started, CRS informed us about the possibility that some selected students would be unavailable for reasons such as school dropout or relocation to a new village. After contacting the school principals and checking the availability of selected students on our randomized list, we decided to re-randomize the problematic grades when none of the selected students from the main or the replacement lists were available. In some situations, we learned about this issue a day before the data collection. In these cases, in collaboration with CRS and school principals, we re-randomized students with a lottery process before the school visit. 2.1.3 Quantitative Data Sources To measure progress toward the key performance indicators, outlined in Section 1, at midline, we collected and analyzed data from six sources (included in Appendix F): (1) student survey, including the Annual Status of Education Report (ASER) reading assessment; (2) parent survey; (3) teacher survey; (4) PTA representative survey; (5) classroom observations; and (6) school district administrator survey. We used the baseline survey tools, which were translated and adapted to the Burkina Faso context, and updated them with minor changes after pilot testing before the midline data collection. Using the same instruments as at baseline provides us with consistent data to measure progress toward program objectives relative to the baseline values. 5 Based on the baseline field test, IMPAQ and CRS decided to exclude first-graders, since they were emotionally and cognitively unable to answer a survey. 6 In our sample, we planned to survey the PTA leaders as the board representatives so that they could respond to questions about both school involvement and managerial activities. IMPAQ International, LLC Page 9 Beoog Biiga II Midline Report July 2017 Surveys Students in grades 2 to 6 completed the same questionnaire. Questions covered reading habits, attendance, health, hygiene, and dietary practices. Whenever possible, we also observed hygiene practices of students at the school to assess to what extent they had adopted good handwashing practices.7 Students’ parents also completed a survey that included the same questions as the student questionnaire regarding their knowledge of nutrition and hygiene. The parent survey also had some questions regarding parents’ involvement with the school and attitudes toward girls’ education. All teachers completed the same teacher questionnaire, regardless of the grade level they taught. Teacher questions focused on teaching practices and attendance. In addition, one PTA leader from each school completed a questionnaire with questions related to meetings held by PTA members in the past year, the functioning of the school canteen, and the extent of community support for the school. Consistent with our baseline data collection, we collected data from district administrators or Chef de Circonscription d’Education de Base (CCEB) of the 22 sampled schools in Bam and Sanmatenga. In this survey, we collected data on the administrators’ knowledge about training teachers in classroom management methods and literacy instruction. ASER Reading Assessment At baseline, we developed, fielded, and used an adapted version of the ASER reading test8 to measure second-grade students’ reading levels. In collaboration with CRS staff, IMPAQ conducted an adaptation workshop and a pretest in 2015 to ensure that the test was culturally appropriate and consistent with Burkina Faso’s learning standards for each grade level in primary school. After the baseline data collection, CRS also used the same ASER test to collect monitoring data. Given the possibility of bias in reading outcomes — for example, students could have access to the test from older students or teachers could become aware of the test and start to prepare students for it — we updated the test content. We ensured that the updated test had the same level of complexity and difficulty as the one used at the baseline so we would be able to compare students’ reading skills between baseline and midline. We used four earlier versions previously calibrated to Burkina’s context at baseline and updated the midline ASER by borrowing some of their items rather than creating untested items. For example, at the sound level the appearance of the sounds was changed but we kept all key sounds. At the simple word level, we reshuffled some of the words and replaced the others with similar words from those validated versions. To further ensure the appropriateness and comparability of the updated test, at midline we conducted another one-day adaptation workshop; we also convened a group of local reading, curriculum, and assessment experts from the Ministry of Basic Education and Literacy (MENA), as well as the CRS educational team. In addition, we pretested the updated ASER in two schools in Ouagadougou whose characteristics were similar to those of the rural schools in the evaluation sample. The final version of the test included 11 levels (A – K), which roughly correspond to the reading standards for each grade level (see Appendix C). Exhibit 4 presents the structure of the ASER reading test, including the test’s levels and corresponding grades and reading skills. 7 Enumerators were instructed to survey students before their break or their lunch to have an opportunity to observe their handwashing practices. But only a few students ate or used the latrine after being surveyed, leaving us with few observations. 8 In 2005, the ASER Center pioneered a nationwide survey, composed of a reading and math test, to measure the achievements of children in primary school in rural India. Since inception, adapted versions of the tests have been implemented in Pakistan, Kenya, Mali, Senegal, Tanzania, Uganda, Ghana, and now Burkina Faso (Pratham 2015; USAID 2012). IMPAQ International, LLC Page 10 Beoog Biiga II Midline Report July 2017 Exhibit 4. ASER-Reading Test Structure Level Corresponding Grade Reading Skills Level 0 None None Level A Grade 1 (CP1) – Lower level Identify letters Level B Grade 1 (CP1) – Upper level Read simple sounds Level C Grade 2 (CP2) – Lower level Read complex sounds Level D Grade 2 (CP2) – Upper level Decode simple words (1-2 syllables) Level E Grade 3 (CE1) – Lower level Decode complex words (2-3 syllables) Level F Grade 3 (CE1) – Upper level Read simple sentences Level G Grade 4 (CE2) – Lower level Read complex sentences Level H Grade 4 (CE2) – Upper level Read simple stories Level I Grade 5 (CM1) – Lower level Answer reading comprehension questions on simple stories Level J Grade 5 (CM1) – Upper level Read complex stories Level K Grade 6 (CM2) Answer reading comprehension questions on complex stories Source: IMPAQ. Classroom Observations We used BBII literacy training materials to develop a classroom observation tool. An IMPAQ education expert, in collaboration with our fieldwork assistant, tested the tool in two schools in Bamako, Mali, and two schools in Ouagadougou, Burkina Faso. We worked closely with the CRS education program manager and specialist to finalize the instrument, fully adapting it to the local context. The final version of the classroom observation tool can be found in Appendix F. The tool was designed to observe instruction in any subject, including literacy, math, and human science, in grades 2 to 6. The instrument enabled us to observe whether trained teachers have been implementing the new teaching techniques and using the material kits provided by the program. 9 It also helped us to correlate the outcomes of classroom observation with the findings from the teacher survey and teacher key informant interviews. 2.2 Qualitative Approach Our qualitative study assesses both retrospectively and prospectively, the following principles: 10 (1) the relevance of all interventions, (2) the effectiveness of implementation strategies and activities, (3) the efficiency of the project, (4) the impact of the project, and (5) the likely sustainability of initiated actions. A description of each research domain is presented below in Exhibit 5. Exhibit 5. Qualitative Research Domain Descriptions 9 We initially planned to record videos in the classrooms so we could score teaching techniques and evaluate the pedagogical instructions as a back-up to our collected data. However, after practicing this approach in the pilot testing, we learned that video recording could reduce our efficiency and create distraction in class. 10 These five principles were developed by the Organisation for Economic Co-operation and Development. More information can be found at http://www.oecd.org/dac/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm. IMPAQ International, LLC Page 11 Beoog Biiga II Midline Report July 2017 Research Domain Description Relevance The extent to which the project interventions are meeting the needs of the project beneficiaries and are aligned with Burkina Faso’s and the U.S. government’s development goals, objectives, strategies; the extent to which the project considers economic, cultural, and political context; and the extent to which stakeholders are satisfied with their participation in the project. Effectiveness The extent to which the project is achieving its objectives, the interventions contribute to expected results or objectives, and the program coordinates and collaborates with other stakeholders and manages partners. Efficiency The extent to which project resources have led to the achieved results, the objectives have been achieved on time, and the project has responded to any internal or external factors that have hindered the efficient implementation of project activities. Perceived Impact The extent to which the project will achieve medium- to long-term effects, both intended and unintended, and the extent to which the effects are due to the project intervention or other factors. Sustainability The extent to which the project has planned for the continuation of project activities, developed local ownership of the project, and developed sustainability partnerships and the extent to which factors (and which ones) will likely influence achievement or non-achievement of project sustainability. Source: BBII Terms of Reference (TOR). 2.2.1 Qualitative Research Questions The IMPAQ team focused assessment of efficiency, project effectiveness, and perceived impact on the strategic objectives and results shown in Exhibit 6. The domain descriptions in Exhibit 5 and the key questions in Exhibit 6 guided the design of the qualitative methodology and the development of interview and focus group instruments. Data sources are defined in Section 2.2.2 below. Exhibit 6. Qualitative Research Questions Strategic Objectives and Results Key Questions Data Source SO1. Improved Literacy of School-Age Children ▪ What factors contributed to an increase in reading comprehension? What factors inhibited improvement? Teacher and school administrator interviews IR1.1. Improved Quality of Literacy Instruction ▪ Have teacher literacy instruction trainings been completed as per the project timeline and budget? ▪ To what extent are teachers implementing literacy teaching techniques acquired through the project? ▪ Are changes in classrooms starting to be evident (e.g., teachers see improved reading abilities of students)? If not, why not? What needs to change? ▪ What challenges do teachers face in using the new literacy teaching techniques? ▪ What aspects do they find most useful and why? Implementer, teacher, and school administrator interviews IMPAQ International, LLC Page 12 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Key Questions Data Source ▪ In what way has the quality of education improved as a result of the adoption of technical trainings for teachers? ▪ Where libraries are in place, to what degree do community members (including students and teachers) have access? Output. 1.1.1. More Consistent Teacher Attendance ▪ What are the greatest inhibiting factors to teachers’ attendance? ▪ What project interventions influenced the improvement of teacher attendance? Teacher and school administrator interviews Output. 1.1.2. Better Access to School Supplies & Materials ▪ To what extent have school supplies and materials been distributed as per the project timeline and budget? ▪ What materials have been supplied? ▪ Which school supplies do teachers find most useful and why? Which supplies and materials provided do students like and why? ▪ What other supplies would teachers and students prefer? ▪ Are material kits being used as intended? ▪ Do teachers and students need additional training to better use these materials? Implementer, teacher, and school administrator interviews; student focus groups Output. 1.1.3. Improved Literacy Instructional Materials ▪ To what extent have literacy instructional materials been distributed as per the project timeline and budget? ▪ Do the teachers consider literacy instructional materials to be an improvement over what they previously had? How? ▪ How are teachers using the materials provided? ▪ What, if any, other materials would they prefer? ▪ What do students like and dislike about using the literacy materials that have been provided? ▪ To what extent have reading groups been established? ▪ Who provides oversight of reading groups most often (teachers or older students)? ▪ Why do students join or not join reading groups? ▪ Have students or teachers noticed any change since students have joined reading groups? (e.g., personal confidence, better grades, more social cohesion among students, etc.) Implementer, teacher, and school administrator interviews; student focus groups Output. 1.1.4. Increased Skills and Knowledge of Teachers ▪ To what extent have teachers been trained as per the project timeline and budget? ▪ In what ways has the quality of teaching improved based on the tools and techniques used by teachers? ▪ What aspects of the trainings were not widely adopted and why? Implementer, teacher, and school administrator interviews; student focus groups IMPAQ International, LLC Page 13 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Key Questions Data Source ▪ How can the trainings have greater impact? ▪ What additional training topics would help the teachers be even more effective in literacy instruction? Output. 1.1.5. Increased Skills and Knowledge of School Administrators ▪ To what extent have school administrators been trained as per the project timeline and budget? ▪ To what extent do school administrators find the classroom observation technique useful? ▪ How is the technique received by teachers? ▪ Have the observations led to constructive feedback? If so, has the feedback been received by teachers and affected their teaching techniques? ▪ In what way has the quality of education improved based on techniques used by the administrators? Implementer, teacher, and school administrator interviews IR1.2. Improved Attentiveness Output 1.2.1. Reduced Short￾Term Hunger ▪ To what extent have school meals been distributed as per the project’s budget and timeline? ▪ How do students appreciate the ration size? Implementer interviews, student focus groups Output 1.2.1.1/1.3.1.1 Increased Access to Food (School Feeding) ▪ To what extent have take-home rations been distributed as per the project’s timeline and budget? ▪ How do students like the commodities provided for school meals? Implementer interviews, student focus groups IR1.3. Improved Student Attendance ▪ What are the greatest pull factors for students, regarding attendance? Student focus groups Output. 1.3.1 Increased Economic & Cultural Incentives (Or Decreased Disincentives) ▪ To what extent has the girls mentoring activity been implemented as per the project’s timeline and budget? ▪ Have the mentors been recruited and trained as per the project’s timeline and budget? ▪ Have the mentors conducted regular mentoring activities as planned? ▪ Have the SILC activities been implemented as per the project’s timeline and budget? Implementer interviews Output 1.3.4. Increased Student Enrollment ▪ Which factors have facilitated or have been obstacles to enrollment? Teacher interviews, student focus groups IMPAQ International, LLC Page 14 Beoog Biiga II Midline Report July 2017 Strategic Objectives and Results Key Questions Data Source Output. 1.3.5. Increased Community Understanding of Benefits of Education ▪ To what degree have the awareness-raising activities on the importance of education been completed as planned? Implementer interviews SO2. Increased Use of Health and Dietary Practices IR2.1 Improved Knowledge of Health and Hygiene Practices ▪ To what extent have students improved their hygiene￾related practices? ▪ To what extent has the project supplied handwashing stations to schools as planned? Implementer interviews, student focus groups IR 2.3 Increased Knowledge of Nutrition ▪ To what extent have PTA members, school management committee members, and food preparers been trained in good nutrition and dietary practices as planned? Implementer interviews IR2.5 Increased Access to Preventative Health Interventions ▪ To what extent has the project distributed micronutrients to students as planned? Implementer interviews Source: Terms of Reference (TOR) and Performance Monitoring Plan (PMP). 2.2.2 Qualitative Data Sources To address the research domains, IMPAQ conducted 45 – 60-minute interviews with individuals from six stakeholder groups: (1) USDA; (2) BBII implementers including CRS, OCADES, FAVL, and MENA; (3) county mayors; (4) teachers; (5) school administrators; and (6) students. Key informant interviews with these stakeholders probed the relevance of interventions, the effectiveness of implementation strategies and activities, the efficiency of the project, the impact of the project, and the likely sustainability of initiated actions. Five of the 22 control group schools in the performance evaluation sample (which is the same as the control group for the impact evaluation) were randomly selected for qualitative data collection. See Appendix G for a full list of selected schools. IMPAQ selected a small sample of teachers and students who completed the survey to participate in key informant interviews. In each of the five randomly selected schools, we conducted interviews with two teachers. We had at least one teacher representative from each grade and aimed for an even mix of male and female teachers. We conducted one focus group with five to eight students from one grade at each IMPAQ International, LLC Page 15 Beoog Biiga II Midline Report July 2017 school, for a total of five focus groups, one per grade. 11 We also interviewed the school principals at three of the five schools and two district administrators, one from each province. In addition to school-level interviews, IMPAQ interviewed key program stakeholders and implementers. We conducted one interview with a USDA representative; nine interviews with representatives of project implementers, including MENA; and two interviews with county mayors, one from each province. Exhibit 7 presents the number of interviews conducted per respondent type and the relevant research domains addressed by each group. Exhibit 7. Respondent Types and Relevant Research Domains Respondent Type Number of Interviews Research Domain(s) Project-Level Respondents USDA 1 Relevance, effectiveness, efficiency, perceived impact, sustainability Implementers 9 4 from CRS, 2 from OCADES, 1 from FAVL, 2 from MENA (1 per province) Relevance, effectiveness, efficiency, perceived impact, sustainability County Mayors 2 (1 per province) Relevance, effectiveness, perceived impact, sustainability School-Level Respondents Teachers 10 (2 per school) Effectiveness, efficiency, perceived impact, sustainability School Administrators* 5 2 school district administrators (1 per province) and 3 school principals Effectiveness, efficiency, perceived impact, sustainability Students 5 focus group discussions 1 per school, 35 students in total Effectiveness, perceived impact Source: IMPAQ. *School administrator refers to both district-level administrators/inspectors and school-level administrators/pedagogic advisors. When applicable, school district administrator is used to distinguish between the district-level administrator and the school director. Appendix F presents the qualitative instruments developed and used for each respondent group. 11 We conducted focus group discussions with students in order to obtain detailed information about personal and group perceptions and opinions. Compared to individual interviews, focus groups allowed us to speak to a greater number of students at each school, given time and resource limitations. IMPAQ International, LLC Page 16 Beoog Biiga II Midline Report July 2017 SECTION 3. FIELD WORK AND ANALYSIS According to U.S. Department of Health & Human Services, this study was exempt from Institutional Review Board (IRB) because: • There is no bio biomedical testing involved in this research, and • This research is being conducted in an established or commonly accepted educational setting, “involving normal education practices”12 . This section outlines the quantitative and qualitative fieldwork and subsequent data analysis for this evaluation. 3.1 Quantitative Fieldwork and Analysis 3.1.1 Quantitative Data Collection In collaboration with Action, Gouvernance, Intégration, Renforcement (AGIR), our data collection partner, we recruited and trained 16 enumerators to collect the midline data in May and June 2017. To enhance the efficiency of midterm data collection, AGIR was able to re-hire five enumerators of the 16 who collected baseline data. Enumerator training consisted of three days of theoretical indoor training, one day of pilot testing in two schools in Ouagadougou, and one day of debriefing and instrument refinement. Enumerators received training on the rationale behind each survey and learned how to collect data on paper and tablets. They used tablets to conduct the in-person surveys13 and electronically submitted the surveys periodically during the field work. With AGIR, we organized 14 enumerators into two teams of five individuals and one team of four. Each team had a team leader who was involved in the baseline data collection. The team leaders were already familiar with the project, the schools, and implementation of the ASER reading assessment. In addition to leadership tasks, they were mainly responsible for surveying second-graders and administering the ASER. Team leaders also contacted school principals to coordinate their arrival before visiting the school; ensured that randomly selected students were still enrolled and available to be surveyed; and, finally, asked principals to call on selected students’ parents (specifically mothers) to be present on a day of data collection to give their consent and be surveyed. The other two enumerators from the baseline data collection were responsible for collecting attendance data and surveying district administrators. Two IMPAQlocal consultants, one in each province, closely followed the teams of enumerators daily to oversee the quality of the data that enumerators collected and provide them with technical support. All enumerators regrouped with IMPAQ’s local consultants in their village several times during the data collection to debrief, submit daily data collection logs, submit electronic surveys, and review and plan for the next days of data collection. The team completed field work in 10 days. In addition, one of IMPAQ’s local consultant led the classroom observation activity for the midline evaluation after receiving training from IMPAQ’s classroom observation expert in Bamako, Mali. Classroom observation training consisted of three days of training in real classrooms in Bamako, one day of pilot testing in two schools in Ouagadougou, and a day of debriefing and instrument refinement. During 12 Exemption 45 CFR 46.101(b)(1). 13 There were only a few cases in which enumerators had to survey students on paper due to programming issues on the first day of data collection. That night, the team leaders transferred the information from paper to tablets. IMPAQ International, LLC Page 17 Beoog Biiga II Midline Report July 2017 the whole process of training, piloting, and debriefing, we worked closely with the CRS educational team to ensure that the tool was well adapted to the Burkina Faso context and to BBII teacher training components. 3.1.2 Quantitative Analysis After the field activities were completed we conducted a final review of the collected data to check: ▪ Data completeness, ▪ Duplicate entries, ▪ Final dispositioning of records, ▪ Skip pattern logic, and ▪ Data cleaning accuracy. We then compiled the survey responses into a master file for the analysis. In this report, we provide summary statistics (percentages and averages) at midline using data from individual or multiple survey items. We also outline subgroup analyses by grade, student gender, and province, highlighting emerging patterns. To measure the progress of BBII toward its core objectives, we conducted descriptive analyses comparing mean outcomes at baseline with those at midline by running t-tests and using p-values. 3.2 Qualitative Fieldwork and Analysis 3.2.1 Qualitative Data Collection With oversight from IMPAQ and logistical support from CRS, one of IMPAQ’s local consultants, an experienced qualitative researcher, led the qualitative data collection effort in Burkina Faso in May 2017. IMPAQ’s in-house qualitative expert conducted the USDA interview in Washington, DC, before the field qualitative data collection effort began. IMPAQ coordinated with CRS to identify appropriate individuals for interviews and worked closely with CRS to schedule and organize the data collection effort. In addition to supporting logistics and leading the in-field data collection effort, the local qualitative lead translated the local data collection guides into French and ensured that they were culturally appropriate. The qualitative lead moderated the interviews and focus group discussions, which were audio recorded, with consent, for note-taking and analysis purposes. Prior to each interview and focus group discussion, the qualitative lead asked each participant to read and sign an informed consent form. IMPAQ worked closely with the qualitative lead to obtain consent forms for those under 18 year old from guardians prior to the focus group discussion. The form informed participants (and their guardians, if applicable) of the following: ▪ Their participation and the information that they disclose will be kept private. ▪ Their name will not be used in any reports. The interviewers will be taking notes during the discussions about what was said and report aggregate responses and opinions. ▪ Their participation is voluntary and they may choose not to answer a question if they feel uncomfortable. ▪ With their permission, the entire session will be audio recorded for report writing and analysis purposes only. Only the evaluation team will have access to the audio recording. IMPAQ International, LLC Page 18 Beoog Biiga II Midline Report July 2017 Contact information for the study’s organizers was provided on the consent form and participants were encouraged to reach out to the organizer if they have any questions after the interview/focus group discussion. Further, the team adhered to the following data collection protocol throughout the project: • Interviews incorporated a degree of flexibility, and the evaluation team allowed additional questions in order to capture any information relevant to the research questions and domains. • The evaluation team followed a consistent data collection approach with each respondent and at each school while allowing for limited variation according to the cultural practices in each locality. • The evaluation team kept confidential all information and opinions expressed during individual interviews and focus groups. To the extent possible, only principal parties were present during the interviews. 3.2.2 Qualitative Analysis At the end of each data collection day, the qualitative lead documented the main points and key themes from each discussion and shared these insights with the IMPAQ team. These debriefings helped IMPAQ to: ▪ Identify what topics and issues needed further probing ▪ Determine how to adapt the guides in real time, if needed, to obtain more meaningful data ▪ Ensure that the research team shared a common understanding and interpretation of the main points and themes ▪ Establish quick-turnaround findings as warranted ▪ Build a strong framework for additional analyses to be conducted after the site visit After the data collection was complete, the team reviewed and analyzed the debriefing notes, supplemented by interview recordings and transcripts, to identify recurring patterns pertaining to the five research domains. Our analytical approach captured salient themes for each research domain and any important similarities and key differences that may inform the quantitative results. IMPAQ International, LLC Page 19 Beoog Biiga II Midline Report July 2017 SECTION 4. EVALUATION SAMPLES In this section, we provide summary statistics about students, parents, households, teachers, classrooms, and PTA representatives, and school district administrators. Since we drew a new sample at the midline, we also provide balance checks for key demographic characteristics pertaining to students, parents, households, and teachers to ensure the midline sample is observationally equivalent to the baseline sample. To measure the midline values for performance indicators, shown in Exhibit 2 in Section 2.1.1, and to measure progress toward the desired outcomes over time, we collected data from the 22 control schools in the impact evaluation sample14 for the performance evaluation. We followed the sampling strategy explained in Section 2 to randomly select students from the school list shared by CRS. However, as soon as the field operations started, as we expected, we found that some of the students on both the main lists and replacement lists were entirely unavailable for reasons such as relocation, grade repetition, and/or dropping out of school. In these situations, we followed our re￾randomization plan to ensure a large enough random sample. With our re-randomization strategy for students, we ended up with a random sample of 258 students, grades 2 to 6, and 262 parents. In each of the 22 schools, we also surveyed teachers of grades 2 to 6, as well as the PTA leader. In addition, we observed one course each in grades 1 to 5 in five randomly selected schools.15 All respondents, including students, parents, teachers, and PTA members, gave us their consent to proceed. 16 Exhibit 8 shows the distribution of sampled respondents by province. Exhibit 8. Sample Distribution by Province and Type of Respondent Province Type of Respondent Schools Students Parents Teachers PTA Leaders Bam 11 130 134 50 11 Sanmatenga 11 128 128 55 11 Total 22 258 262 105 22 Source: Surveys of students, parents, teachers, PTA members 4.1 Students To maintain a balanced boys-to-girls ratio that reflects the population of beneficiary students, in each school, we randomly selected two boys and two girls in grade 2 and one boy and one girl in grades 3 to 6. In total, we surveyed 258 students in grades 2 to 6: 125 boys and 133 girls. As Exhibit 9 shows, the 14 In April 2016, CRS informed us about their mistake in the control group list, in which two schools in the treatment group were swapped with control schools, so they were included in the baseline performance evaluation data collection. However, for the midline data collection, we collected data from the right list of control schools. Descriptive analysis in this report is based on the correct list of schools and the update baseline values. 15 We observed one lesson in a classroom of each grade (grades 1 through 5). We included first grade because it provides the base of learning in primary school. 16 In accordance the U.S. Department of Health and Human Services guidelines on Human Subjects Research (45 C.F.R. § 46), we asked all respondents for their consent to proceed with the survey. IMPAQ International, LLC Page 20 Beoog Biiga II Midline Report July 2017 proportion of girls to boys sampled in each grade is largely balanced (51.6 to 48.4 percent). Exhibit 47 in Appendix C also shows that the gender composition from baseline to midline has not changed. Exhibit 9. Sampled Student Gender Distribution by Grade Grade Male Female Total Percent Total Number of Observations Percent Total Number of Observations Grade 2 (CP2) 49% 43 51% 45 88 Grade 3 (CE1) 51% 21 49% 20 41 Grade 4 (CE2) 45% 20 55% 24 44 Grade 5 (CM1) 49% 21 51% 22 43 Grade 6 (CM2) 48% 20 52% 22 42 Total 48.4% 125 51.6% 133 100% Source: Student Survey; IMPAQ calculations. 4.2 Parents School principals were asked to call students’ parents — specifically mothers, who are usually the most informed about students — to be at school on the day of data collection to be surveyed and to give consent for their children to be surveyed. If siblings were included in the sample, we surveyed the parent separately for each sibling, to ensure that we collected data from parents for every student in our sample. This process enabled us to account for the possibility that children in one household could be treated differently, especially in the distribution of food. Almost all of the 258 surveyed parents were biological mothers (89 percent) or other female primary or secondary caregivers (10 percent). Exhibit 10 shows the differences between educational attainments of parents obtained from the parent’s questionnaire at baseline and midline. Parents at midline are less illiterate (9 percentage points), and more educated with primary education (7 percentage points) (statistically significant at the 1 percent level). We should compare literacy outcomes of children in Section 5 with caution since children are from families with a slightly higher level of educations at midline compared to baseline. Exhibit 10. Parents’ Educational Attainment Level of Education Baseline Midline Difference in Means Percent Percent (p-value test) None 95% 86% -9*** (0.00008) Primary education (Grades 1 to 6) 3% 10% 7*** (0.00016) Secondary education (Grade 7 and above) 2% 4% 2 (0.19087) Source: Parent Survey; N=343 (baseline) and N=262 (midline); IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 IMPAQ International, LLC Page 21 Beoog Biiga II Midline Report July 2017 4.3. Household Environment We also collected data from parents on household characteristics: • Household composition: household size, percentage of children under 16, and percentage of household members above 16 • Household access to basic services 4.3.1 Household Composition At midline, the average household size of about 21 household members was similar in both provinces. About half of the household members (11 people) were 16 years old or younger (Exhibit 11). The data indicate a large variation in reported household size, from 4 to 72. This finding suggests that respondents reported on an extended family when answering the questions about household size. Compared to baseline, household size is significantly bigger by 8 members on average (p < 0.01) (four more children under 16 and four more adults). Exhibit 11. Household Size Household Size Baseline Midline Difference in Means Percent Percent (p-value test) Household size 13 21 8*** (0.0000) Household members under 16 years old, including the respondent in the household 7 11 4*** (0.0000) Household members over 16 years old, including the respondent in the household 6 10 4*** (0.0000) Source: Parent Survey; N=343 (baseline) and N=262 (midline); IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 4.3.2 Household Access to Basic Services Exhibit 12 shows parents’ responses about their households’ access to basic services at baseline and midline. As shown in Exhibit 12, households at midline have better access to electricity, latrine, and metal rooves. However, accessibility to water for households at midline was very low (only 5 percent), compared to baseline (76 percent). Almost all households have access to electricity (96 percent); 67 percent had a latrine at home. However, some regional variation exists in the data. 72 percent of households in Bam had latrines versus 62 percent of households in Sanmatenga. Over half of the households (55 percent) had metal roofs: 49 percent in Bam and 62 percent in Sanmatenga. Exhibit 12. Household Access to Basic Services Indicator Baseline Midline Difference in Means Percent Percent (p-value test) Access to any source of electricity at home 58% 96% 38*** (0.0000) IMPAQ International, LLC Page 22 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Percent (p-value test) Access to any type of latrine at home 49% 67% 18*** (0.0000) Access to water at home for washing 76% 5% -71*** (0.0000) Households with different types of roofs Thatch Roof 19% 19% 0 (0.92131) Metal Roof 46% 55% 9** (0.03140) Combination of straw and metal roof 29% 23% -6* (0.07573) Roof made of some other material 5% 3% -2 (0.10174) Source: Parent Survey; N=341 - 35517 (baseline) and N=262 (midline); IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 4.4 Teachers In each of the 22 schools, we surveyed teachers in grades 2 to 6, for a total of 105 teachers. Only five teachers in three schools were absent for medical reasons at the time of the survey. The gender composition of teachers is similar across the two provinces, and between baseline and midline; 59 percent of teachers at midline and 50 percent at baseline were female. The average class size was 54 students at midline which was not significantly different from the average class size at baseline. Exhibit 13. Classroom Size Indicator Baseline Midline Difference in Means Mean Mean (p-value test) Average number of students per classroom, all grades 50 54 4 (0.13169) Average number of students in grade 2 classrooms 55 61 6 (0.38203) Average number of students in grade 3 classrooms 51 62 11* (0.08890) Average number of students in grade 4 classrooms 46 52 6 (0.28112) Average number of students in grade 5 classrooms 47 48 1 (0.79204) Average number of students in grade 6 classrooms 46 45 -1 (0.94560) Source: Teacher survey; N=101 (baseline) and N = 105 (midline); IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 17 Variation in the total number of observations is due to variation in the number of parents who answered the related question. IMPAQ International, LLC Page 23 Beoog Biiga II Midline Report July 2017 On average, teachers at midline were slightly less experienced (by 6 months) compared to teachers at baseline with 3.5 years of experience, marginally significant at the 10 percent level. As Exhibit 14 shows, teachers at midline were also less educated compared to teachers at baseline. Most teachers (83 percent) had only a Brevet d’Etudes du Premier Cycle (BEPC), which corresponds to a tenth-grade level of education. Also, more teachers in Bam (14 percent) than in Sanmatenga (7 percent) had baccalaureate18 degrees (not shown). Exhibit 14. Teachers’ Educational Attainment Level of Education Baseline Midline Difference in Means Percent Percent (p-value test) BEPC 68% 83% 15** (0.01484) Baccalaureate 19% 10% -9* (0.0908) Bachelor`s degree 5% 5% 0 (0.9501) Master 1 0% 1% 1 (0.32791) Other 8% 1% -7 (0.01432) Source: Teacher Survey; N=101 (baseline) and N = 105 (midline); IMPAQ calculations; None of the teachers had master 2 degrees neither at baseline nor at midline. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 As shown in Exhibit 15, 99 percent of teachers received teacher training while 95 percent of teachers received literacy training. Exhibit 15. Teacher Training Level of Education Baseline Midline Difference in Means Percent Percent (p-value test) Teachers that have received teacher training 96% 99% 3 (016234.) Teachers that have received literacy training 16% 95% 79 (0.0000) Source: Teacher survey; N=101 (baseline) and N = 105 (midline); IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 4.5 Classroom Observations We implemented a classroom observation instrument in five randomly selected schools in the performance evaluation sample19 . (See Appendix G for the full list of schools.) We observed 25 classes, five grades per school. 20 The average class size was 68 students per grade with a balanced proportion of girls to boys (49 to 51 percent). Almost all teachers (96 percent) taught in French. The only teacher who 18 Baccalauréat de l’Enseignement Secondaire Général corresponds to high school diploma at the end of grade 13. 19 Agreed between CRS and IMPAQ not to observe the classrooms during class break. 20 In one of the selected schools, Kougsable, the first-grade teacher was not available during the data collection. To keep the sample size consistent, we observed a first-grade class in a nearby school (Foulou) in the same CEB. IMPAQ International, LLC Page 24 Beoog Biiga II Midline Report July 2017 was observed using the local language was teaching first grade. Over 75 percent of the teachers were women (19 out 25). All the teachers had completed initial training, in either public or private teacher training schools. 4.6 Parent-Teacher Associations The IMPAQ team completed twenty-two surveys with PTA leaders. We received one survey from each school and 11 per province. Out of 22 surveyed PTA leaders, only three were women (14 percent). This gender composition is consistent with the baseline data, in which PTA leaders were predominantly men. 4.7 School District Administrators We surveyed 30 school district level staff (Chef de Circonscription d’Education de Base, CCEB), including 15 pedagogical advisors, and 15 school district administrators. CCEB staff were predominantly men, with only one female member, with an average of 4.5 years of experience. Exhibit 48 in Appendix C shows approximately 63 percent of the CCEB staff had some sort of college education. IMPAQ International, LLC Page 25 Beoog Biiga II Midline Report July 2017 SECTION 5. QUANTITATIVE OUTCOMES In this section, we analyze data from the student, parent, teacher, PTA, and school district administrator surveys and from classroom observations. We examined all the data by gender, grade, and province (if relevant) but highlight the differences here only when descriptive differences exceed 5 percentage points. Appendix C provides additional details. Self-reported data, especially those on culturally and socially sensitive topics such as food security, should be interpreted with caution due to social desirability bias. In analyzing performance data from student, parent, teacher, and PTA surveys, we compare mean outcomes at baseline and midline by using t-tests and p-values to highlight statistically significant differences. Such analysis can only suggest a correlation between the observed changes in outcomes and BBII interventions such as school feeding or teacher training. It cannot determine conclusively whether the interventions caused the changes. Other factors could have led to the observed changes over time, given that, at each data collection point (baseline and midline), we selected new samples of individuals to survey. There might be systematic differences in the two sets of samples that affected the outcomes. For example, improvements in literacy outcomes at midline relative to baseline could mean either that BBII teacher trainings were effective or that the students selected at midline came from better socioeconomic backgrounds. In addition, a simple difference in outcomes between baseline and midline could be caused by other general trends that affected all the schools. This section also presents classroom observation outcomes to correlate their results with the other qualitative and quantitative findings. These outcomes should be interpreted with caution because of the small sample21 of observations (25 classrooms). However, the outcomes presented can provide useful suggestions for further observations at endline. This section also presents midline outcomes from the school district administrator survey. Because the baseline sample was small, we did not conduct any descriptive analysis for these indicators. Exhibit 16 provides an overview of the baseline and midline levels of the key McGovern-Dole evaluation performance indicators described in detail in Section 2.1.1, as required by the terms of reference. The Data Source column shows which survey data we used to measure each indicator. As we did at baseline, we calculated the total numbers by applying the proportion to the total population of beneficiaries22 in the Bam and Sanmatenga provinces. This calculation included 201,852 students (105,894boys and 95958girls), 3, 993 teachers, 81 school district administrators, and 30,745 SILC members.23 Each of these indicators and other findings are discussed in detail further in this section. See Appendix B for detailed breakdown of this table. 21 Due to budget limitations, and with CRS and USDA approval, we kept the sample small at midline. 22 We used the percentages as basis for extrapolation to estimate the number of beneficiaries in each indicator. 23 This method is described in the CRS performance monitoring plan and was approved by USDA. IMPAQ International, LLC Page 26 Beoog Biiga II Midline Report July 2017 Exhibit 16. Overview of Baseline and Midline Levels of Key Performance Indicators Key Performance Indicators Data Source Baseline Midline Percentd Total Estimated Number 95% Confidenc e Interval Percentd Total Estimated Number 95% Confidence Interval 1. Percentage of students (boys and girls) who, by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade-level text. ASER reading test Boys: 12% 12,707 0 – 31% 16% 16,943 5 – 28% Girls: 17% 16,313 1 – 33% 32% 30,707 17 – 46% 2. Percentage of teachers who devote at least an average of 45 minutes a day to literacy instruction. Teacher Survey 55% 1,795 45 – 65% 66% 2,635 57 – 75% 3. Percentage of teachers who attend and teach school at least 90% of scheduled school days per year. District Administrator s 85% 2,775 NAa 88% 3,514 NAa 4. Percentage of school administrators and officials who demonstrate use of at least one new technique, skill, or tool as a result of USDA assistance. b District Administrator s 61% 49 NAa 97% 79 NAa 5. Number of preschool teachers bissongo caregivers) in target preschools who demonstrate use of at least one new teaching technique, skill, or tool. Pre- and post￾testing will be conducted before and after BBII training N/A N/A N/A N/A N/A N/A 6. Percentage of teachers/educators/ teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance. 24 Teacher questionnaire Student questionnaire 51% 1,677 42 – 61% 44% 1,757 34 – 53% 7. Percentage of students in target Teacher questionnaire 45% 74,400 35 - 54% 47% 94,870 37 - 56% 24 Teachers who declared using at least 5 of the 7 activities. IMPAQ International, LLC Page 27 Beoog Biiga II Midline Report July 2017 Key Performance Indicators Data Source Baseline Midline Percentd Total Estimated Number 95% Confidenc e Interval Percentd Total Estimated Number 95% Confidence Interval schools who are identified as attentive25 during class/instruction. 8. Percentage of students in target schools who indicate that they are hungry or very hungry during the school days. Student questionnaire Parent questionnaire 4% 8,075 2 - 7% 7% 14,130 4 - 11% 9. Percentage of students regularly (80%) attending USDA supported classrooms/schools (boys and girls). School registries Boys: 100% 105,894 100 - 100% 98% 103,776 96 - 100% Girls: 100% 95,958 99 - 100% 99% 94,998 98 - 100% 10. Percentage of group members that spend money from SILC on education costs. Parent/PTA questionnaire 48% 14,758 24-71% 63% 19,369 53-72% 11. Percentage of girl students who cite mentors as one of the top 3 reasons for their success or improvement in school. c Student questionnaire N/A N/A N/A N/A N/A N/A 12. Percentage of school-age children receiving a minimum acceptable diet (boys and girls). Student questionnaire Boys: 56% 59,301 49 - 64% 69% 73,067 61 - 77% Girls: 63% 60,454 56 - 70% 74% 71,009 67 - 82% 13. Percentage of school children that wash their hands at critical moments: before eating and after using the latrine. Student questionnaire 78% 157,445 74 - 83% 82% 165,519 77 - 87% 14. Percentage of students in target schools who achieve a passing score on a test of good health and hygiene practices by naming at least 4 Student questionnaire 5% 10,093 3 - 8% 4% 8,075 1 - 6% 25 We asked teachers on a scale of 1 to 10, where 1 is not attentive at all, to rate their students’ attentiveness. Passing attentiveness score defined as a score of 7 or more on a scale of 1 to 10. IMPAQ International, LLC Page 28 Beoog Biiga II Midline Report July 2017 Key Performance Indicators Data Source Baseline Midline Percentd Total Estimated Number 95% Confidenc e Interval Percentd Total Estimated Number 95% Confidence Interval situations in which they should wash their hands. 15. Percentage of students in target schools who achieve a passing scoreon a test on food nutrition and dietary practices by naming at least one food with iron and one with vitamin A and by naming one benefit of each. Student questionnaire 1% 2,019 0 - 1% 3% 6,056 1 - 5% 16. Number of months of community and/or government support for school canteens. PTA questionnaire 5 months N/A 4 - 7% 3 months N/A 2 – 5% Source: Surveys of students, parents, teachers, PTA members; IMPAQ calculations. a the small sample size of district administrators does not permit statistical inferences and a confidence interval is not meaningful. b In the BBII project, this indicator focuses on district-level administrators. c This indicator is irrelevant at the midline performance evaluation because the sample includes only the 22 control schools of the impact evaluation which has not received the mentoring intervention. d Percent columns show the percentages with respect to the surveyed respondents for each indicator at baseline and midline. These percentages are used as basis for extrapolation to estimate the total number. 5.1 School Outcomes School-level outcomes involve the presence of libraries and food storage for the school canteen. Both were determined by observation. As part of the BBII project, CRS and its partner FAVL established community libraries in the capital town of each county. We did not observe any of the schools to have a library nearby. This is in line with our qualitative findings described in Section 6—interviewed teachers and school administrators noted that libraries are often long distances from schools and therefore inaccessible to students and teachers. However, during interviews, teachers reported a desire to have access to books and resources through libraries and were disappointed with the location of the libraries. As observed, 21 out of 22 schools were equipped with food storage. In the school with no food storage, parents stored the food for the canteen at home. Storage spaces were mostly in good condition. All 21 had a metal roof and 20 had cement walls. The other building’s walls were made of mud. Two storage buildings had holes in the roof through which rain could penetrate. 5.2 Student Outcomes This section presents student outcomes, collected from the student survey, in the following six areas at midline in comparison with baseline: • Nutrition knowledge IMPAQ International, LLC Page 29 Beoog Biiga II Midline Report July 2017 • Hygiene knowledge and practices • Food security • Attendance • Literacy activities • Reading outcomes 5.2.1 Nutrition Knowledge As we did at baseline, to capture students’ nutrition knowledge, we asked children whether they had heard of vitamin A and iron. If they had heard of those nutrients, then we asked them to name benefits of vitamin A and iron, as well as foods that contain them. We also defined a passing score on a test of nutrition as the ability to name at least one food with iron, one food with vitamin A, one benefit of iron, and one benefit of vitamin A. Exhibit 17 shows that the proportion of students with the desired level of nutrition knowledge improved over time, although it is still low. The proportion of students who had heard of vitamin A and of iron increased 17 and 10 percentage points, respectively. The differences in means are statistically significant at the 1 percent level in both cases. In spite of these improvements, the proportion of students who could cite one benefit of vitamin A and one type of food that contains it has not substantially increased. However, the proportion of students who could cite one benefit of iron increased by 26 percentage points (from 32 percent to 58 percent) compared to baseline. The difference in means is statistically significant (p < 0.05). The proportion of students who achieved a passing score by naming benefits and foods for both nutrients increased by 2 percentage points at midline, with a statistically significant difference in means (p < 0.05), but this proportion still remains low. In interviews and focus group discussions, both teachers and students reported that dietary teachings about vitamins and minerals were useful, though it was unclear whether or how students apply these lessons to their everyday lives. Exhibit 17. Student Nutrition Knowledge Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students who have heard of vitamin A 32% 324 49% 258 17*** (0.0000) Students who can cite one benefit of vitamin Aa 41% 103 46% 126 5 (0.4273) Students who can cite a food containing vitamin Aa 23% 92 25% 107 2 (0.6940) Students who have heard of iron 9% 324 19% 258 10*** (0.0004) Students who can cite one benefit of irona 32% 28 58% 48 26** (0.0276) Students who can cite a food containing irona 29% 28 37% 48 8 (0.4354) IMPAQ International, LLC Page 30 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students who achieve a passing score on a test of food nutrition and dietary practices 1% 324 3% 258 2** (0.0220) Source: Student Survey; IMPAQ calculations. a Sample comprises only students who said they had heard of the nutrient. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.2.2 Hygiene Knowledge and Practices To measure students’ knowledge and practice of hygiene, we asked students to identify instances in which they should wash their hands (for example, before eating, after touching animals, before preparing food). We defined a threshold of hygiene knowledge as the ability to identify at least four situations in which people should wash their hands. We also asked students about their hand and body washing practices. The data show no significant changesfrom baseline to midline in children’s self-reported hygiene practices (Exhibit 18). At midline, almost all students (95 percent) reported washing their hands and more than half (63 percent) reported washing their bodies; these results are similar to baseline findings. Interestingly, in focus group discussions, children reported enjoying the handwashing practice. Teachers and school administrators at some schools reported that students’ hygiene practices have changed since the start of the project; handwashing has become a habit, they report. The lack of significant changes from baseline to midline could be explained by the fact that some schools lack reliable water sources and handwashing devices sometimes fall into disrepair, as reported by school-level interviewees, and explained in more detail in Section 6.2. Very few children achieved a passing score on the test of hygiene knowledge. The one percentage point decrease in means is not statistically significant. However, at midline almost all children (98 percent) reported their teachers taught them the importance of washing hands. These contradictory findings should be interpreted with caution due to social desirability bias (i.e., children did not want their teachers to look bad)26 . Exhibit 18. Student Hygiene Practices Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students who say that they washed their hands today 94% 324 95% 258 1 (0.5575) Students who say that they washed their body today 66% 324 63% 258 -3 (0.4722) 26 This indicator was added at the midline, therefore no comparison in mean of outcomes with baseline is available. IMPAQ International, LLC Page 31 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students who achieve a passing score on a test of good health and hygiene practices 5% 324 4% 258 -1 (0.4356) Source: Students Survey; IMPAQ calculations. Note: Excluded observations from baseline are students who did not answer these questions. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 We also asked children whether they washed their hands at critical moments, defined as before eating and after using the latrine. Then we compared these self-reported handwashing practices with students’ knowledge of the two critical moments. The green bars in Exhibit 19 show that the proportion of students who reported they washed their hands at critical moments slightly increased by four percentage points from baseline (78 percent), though the results are not statistically significant. Likewise, as shown by the red bars of Exhibit 19, knowledge about handwashing practices changed by only 4 percentage points from baseline to midline; the difference is not statistically significant. The inconsistency between children’s knowledge and their actual practices suggest a possibility of over-reporting by students on their hygiene practices. Therefore, these results should be interpreted with caution, because they derive from self￾reported data from young children. Also, as mentioned earlier in this section, the changes in outcomes can be due to external factors that are not necessarily associated with BBII. Exhibit 19. Students’ Knowledge of Handwashing versus Self-Reported Practices of Handwashing at Critical Moments Source: Student Survey; IMPAQ calculations. Note: for baseline, N=36, 414 (Knowledge) and N=129,104 (Self-reported Practices); for midline, N=43,035 (Knowledge) and N=135,725 (Self-reported Practices)27 . * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01, *shows the difference in means between self-reported practices and hygiene knowledge. 27 The calculations are based on the total number of responses to different options that were selected for all that applied. 22% 26% 78% 82% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Baseline Midline Knowledge Self-reported Practices IMPAQ International, LLC Page 32 Beoog Biiga II Midline Report July 2017 We also attempted to observe students’ handwashing practices at critical moments at school. However, almost 90 percent of students did not eat or use the latrine at the time of data collection. Out of the remaining 9 percent, 7 percent were observed washing their hands before eating and about 2 percent were observed washing their hands after using the latrine. As discussed in Section 7, this approach needs refinements at the endline to observe more students’ handwashing practices. 5.2.3 Food Security To measure food security among students, we looked at two factors: 1. Children’s food intake during the day (breakfast and lunch). We asked students whether they ate breakfast and lunch a day before data collection, where they ate those meals (at home or at the canteen), and whether they felt full after consuming each meal. 2. Minimum acceptable diet. We followed the UN Food and Agriculture Organization (2010) diversity index,recommended by the USDA Foreign Agricultural Service,to calculate the minimum acceptable diet in terms of diversity, first using student data and then using parent data for purposes of comparison. Using a list of 15 types of food, we defined a threshold for an acceptable diet as including at least five different foods per day. 28 The data show, as at baseline, that most children who ate breakfast and lunch felt full after eating. Exhibit 20 shows that the proportion of children who reported at midline that they were hungry during the school day, defined as whether they felt full after their breakfast using the previous day as reference, was as low as at baseline with a slight increase, but not statistically significant at any conventional level. Although this increase is not statistically significant, 29 it could be explained by fewer children (4 percentage points) eating before going to school (not statistically significant at any level) at midline. Students most commonly cited insufficient food (81 percent) as the reason why they could have eaten more after breakfast before going to school at midline. In addition, the proportion of students that had their lunch at the canteen increased to 80 percent at the midline, statistically significant at the one percent level. This outcome, however, should be interpreted with caution. It is because although 26 percent of surveyed students at baseline reported eating lunch at the canteen, the canteen component of the project had not yet started at that time. One potential explanation is that students might have been confused with the definition of the canteen about having one in their school at the baseline30 . During focus group discussions, students expressed much appreciation for the food they received through the canteens. They stated that the cooked food is tasty and that they often take some home to share with their siblings and parents. In fact, respondents described that, from their observations, the presence of canteens is directly associated with attendance; when canteens are providing food, students go to school. 28 IMPAQ removed spices from the list of 16 food groups in the index, because spices are not relevant for child nutrition. 29 The change that is not statistically significant could be attributed to chance. In better words, there’s always a chance that the differences we observe when measuring a sample of users is just the result of random noise. 30 Community (parents, school staff, etc.) provided “endogen canteen” for children at school which was feeding them during lunch before school canteens, provided by BBII, started functioning. IMPAQ International, LLC Page 33 Beoog Biiga II Midline Report July 2017 Exhibit 20. Students’ Food Intake Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Proportion of children that ate breakfast before going to school 90% 325 86% 258 -4 (0.1624) Children that felt full after the meal s/he ate before going to school. 96% 293 93% 207 -3 (0.1816) Proportion of students in target schools who indicate that they are hungry or very hungry during the school day. 4% 293 7% 223 3 (0.1816) Source: Student Survey; IMPAQ calculations Note: Excluded observations are students who did not answer these questions. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 Exhibit 21 shows that the proportion of children who received a minimum acceptable diet increased from baseline to midline, by 11 percentage points according to student reports and by 20 percentage points according to parents. Both findings are statistically significant, particularly the parent reports (p < 0.01). This increase in the diversity of household’s diet might be explained by different socio-economic backgrounds between baseline and midline samples or seasonality. There are differences between what parents and students reported and between boys and girls. However, the gaps have diminished over time. As shown in Exhibit 21, the proportion of boys receiving a minimum acceptable diet increased by 13 percentage points, which is slightly more than the increase for girls (11 percentage points); both changes are statistically significant (p < 0.05). See Appendix B for the breakdown of diverse foods received by girls and boys; those data show that girls ate more vegetables (of any type), while boys ate slightly more protein. Exhibit 21. Minimum Acceptable Diet Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students reported receiving a minimum acceptable diet 60% 324 72% 258 12*** (0.0029) Boys received minimum acceptable diet 56% 156 69% 125 13** (0.0336) Girls received minimum acceptable diet 63% 168 74% 133 11** (0.0361) Parents reported that students received a minimum acceptable diet 35% 343 56% 262 20*** (0.0000) Source: Student Survey; Parent Survey; IMPAQ calculations. Note: Missing observations represent respondents who did not answer these questions. IMPAQ International, LLC Page 34 Beoog Biiga II Midline Report July 2017 * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.2.4 Student Attendance To measure students’ attendance, IMPAQ and CRS agreed to select three representative months across the agricultural crop cycle and the academic school year: November, January, and April. Then, we collected attendance data for each child in the sample from teachers using school records. Overall, almost all students attended school at least 80 percent of the time (Exhibit 22). Compared to the baseline, the midline value showed a slight decrease in the proportion of students with regular attendance of 1 percentage point, a statistically significant finding at the 10 percent level. Boys’ attendance decreased by 2 percentage points at midline, somewhat higher than the decrease in girls’ attendance. However, it is important to note that the sample of student at midline for attendance data was smaller than the baseline attendance sample by nearly half.31 Moreover, our experience in the field showed that some teachers did not usually record attendance data and that, when they did, we could not ascertain how rigorously they reported, especially at baseline. Even with the best intent, it is possible that teachers neglected to record missing children. Therefore, the average attendance rate is likely overestimated. Exhibit 22. Student Attendance Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students attending class at least 80% of the time 100% 510a 99% 261 -1* (0.0814) Boys attending class at least 80% of the time 100% 219 98% 125 -2* (0.0628) Girls attending class at least 80% of the time 100% 204 99% 134 -1 (0.7648) Source: Student Attendance Data; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01; a See footnote 25 5.2.5 Literacy Activities Activities in Class We asked students about their reading and writing activities in class. We correlated the outcomes with teachers’ responses, discussed in detail in Section 5.4.2, to assess implementation of BBII teaching techniques in classrooms. Specifically, we asked students whether they had read alone or aloud, listened to other students reading, and worked on writing or words during class on the day before the survey. Exhibit 23 shows that the proportions of students who spent time reading alone and listening to reading in class increased by 7 and 8 percentage points, respectively; these changes are statistically significant. There were no statistically significant changes in time spent reading to someone else or doing word work 31 At baseline, the sample size for attendance did not exactly match the sample size of the performance evaluation because incomplete records were dropped from the performance evaluation data, and IMPAQ could not match the student data and attendance data. IMPAQ International, LLC Page 35 Beoog Biiga II Midline Report July 2017 in class. The only activity on which a smaller proportion of children reported spending time at midline than at baseline is writing; however, the decrease is not statistically significant. Exhibit 23. Student Reported Classroom Activities Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Spent time reading alone in class yesterday 63% 325 70% 245 7* (0.0638) Spent time reading to someone else in class yesterday 62% 325 63% 245 1 (0.7485) Spent time writing in class yesterday 71% 325 69% 245 -2 (0.6627) Spent time doing word work in class yesterday 43% 325 49% 245 6 (0.2191) Spent time listening to reading in class yesterday 72% 325 80% 245 8** (0.0376) Source: Student Survey; IMPAQ calculations. Note: Excluded observations from baseline are students who did not answer these questions. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 Activities Outside of Class We also looked at children’s literacy activities outside of their classrooms. We asked children whether they are part of a reading club, defined as a small group of children who read together outside of the school, as part of BBII program activities. In addition, we asked parents whether they take their children to the closest library. Exhibit 24 shows that the proportion of students who participated in a reading club decreased by 34 percentage points, from 60 percent at baseline to 26 at midline; the finding is statistically significant at the 1 percent level. However, there has been no statistically significant change in whether the clubs consist of whole school classes. These findings should be interpreted with caution since the reading club activity had not started when the baseline data collection was conducted (This activity started in second year of program32). There is a possibility that children at baseline confused the definition of a reading club with a reading group, a small group of children who read together in the classroom. The proportion of students visiting libraries with their parents remained very low, with no change from baseline to midline. Again, this is not surprising given the findings described above in Section 5.1. Exhibit 24. Student Reading Groups Indicator Baseline Midline Difference in Means 32 CRS monitoring data from 2017 showed that 682 reading clubs were registered across all project schools during the 2016-2017 school year. However, this does not mean that the reading clubs convened, only that they formed. IMPAQ International, LLC Page 36 Beoog Biiga II Midline Report July 2017 Percent Total Number of Observations Percent Total Number of Observations (p-value) Students who are in a reading club 60% 325 26% 245 -34*** (0.0000) Reading clubs that consist of the entire class 19% 195 17% 63 -2 (0.7893) Students who visit the library with their parents (reported by parents) 2% 343 2% 262 0 (0.6371) Source: Student and Parent Survey; IMPAQ calculations. Note: Excluded observations are students that did not remember what they had done the day before. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.2.6 Student Reading Outcomes We used the ASER literacy assessment to measure students’ grade-level reading competencies. We determined the thresholds for an acceptable reading level at each primary school grade according to the Burkina Faso curriculum guidelines and the calibration workshop that IMPAQ and CRS held in May 2017 (Exhibit 4 in Section 2.1.3). The outcome of the calibration is that level C in the ASER test, the ability to read complex sounds, is the minimum acceptable level at the end of grade 2. Exhibit 25 shows the distribution of the ASER literacy assessment results for baseline and midline. The minimum acceptable threshold, level C, is represented with a vertical orange line. Yet, at midline the majority of students are below the threshold for their grade level, showing that they have limited basic reading skills. Girls showed slightly higher reading proficiency scores than boys. (See Exhibit 51 in Appendix C). Exhibit 25. Distribution of Reading Skills for Second Grade Students IMPAQ International, LLC Page 37 Beoog Biiga II Midline Report July 2017 Source: Student Survey: IMPAQ calculations. N (baseline) = 40; N= (midline) = 87. Although the majority of students did not achieve grade level reading competencies, the proportion of second grade students with reading proficiency at grade level increased by 9 percentage points, but not statistically significant at any conventional levels (Exhibit 26). These outcomes are in line with CRS monitoring data in 2016 that showed improvements in children’s literacy outcomes. The improvement in reading ability at grade level was higher for girls (15 percentage points) compared to boys (4 percentage points). Lowering the threshold to Level B (ability to read simple sounds), the acceptable level at the beginning of grade 2, most students’ reading ability is still below their grade level however, it has improved over time (See Exhibit 52 in Appendix C). However, it is important to note that because of small sample sizes, the results only provide suggestive descriptive evidence for the changes in the indicators of interest and must be interpreted with caution. Exhibit 26. Reading Proficiency at Second Grade Level (Students who scored an ASER Level C (Read Complex Sounds) or higher) Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Second grade students demonstrating reading ability at grade level or above 15% 40 24% 87 9 (0.2458) Male students demonstrating reading ability at grade level or above 12% 16 16% 43 4 (0.7253) Female students demonstrating reading ability at grade level or above 17% 24 32% 44 15 (0.1811) Source: Student Survey; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 12 55 17 0 5 0 7 2 0 9 45 22 1 9 3 7 2 1 0 10 20 30 40 50 60 70 Level 0 Level A Level B Level C Level D Level E Level F Level G Level H Level I Level J Level K Percent Baseline Midline IMPAQ International, LLC Page 38 Beoog Biiga II Midline Report July 2017 These improvements are in line with school-level stakeholders’ perception of the project. All teachers and school administrators who participated in interviews believed that the project is improving literacy skills among children, due to the new teaching techniques. Teachers reported that they have observed improvements in students’ literacy skills and motivation to learn. 5.3 Parent Outcomes To look at children’s household environments, we surveyed their parents, specifically their mothers. Using this survey, we compared parent outcomes at baseline and midline in the following three areas: • Nutrition knowledge • Attitude towards girls’ education • School involvement 5.3.1 Nutrition knowledge A parent is a first child’s teacher. Parents play an important role in children’s knowledge and learning. To gain a deeper understanding of children’s nutrition knowledge, we asked parents the same nutrition questions that we asked students: whether parents had heard of vitamin A and iron and, if so, whether they could name benefits of and foods containing each nutrient. Exhibit 27 shows changes in parents’ nutrition knowledge over time. Overall, the proportion of parents who had heard of vitamin A and iron increased significantly by 10 and 7 percentage points, respectively, from baseline to midline. This increase in parents’ responses confirms the increase in proportion of children who heard of these nutrients. A significantly smaller proportion of parents were able to name a food containing vitamin A at midline than at baseline (a difference of 23 percentage points, p < 0.01). However, a larger proportion of parents could cite a food containing iron (a difference of 25 percentage points, p < 0.05). Also, there was a significant 20 percentage point decrease in the proportion of parents who named one benefit of iron. In general, the data show improvements in the percentages of parents and of students who have heard of iron and vitamin A. However, their knowledge of the benefits of these nutrients or the foods that contain them are still low. These findings are in line with their children’s low knowledge of nutrition, suggesting children did not have any other sources except teachers to learn about this knowledge. . Exhibit 27. Parent Nutrition Knowledge Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Parents who have heard of vitamin A 35% 343 45% 261 10** (0.0141) Parents who can cite one benefit of vitamin Aa 60% 120 55% 117 -5 (0.4117) Parents who can cite a food containing vitamin Aa 59% 120 36% 117 -23*** (0.0003) IMPAQ International, LLC Page 39 Beoog Biiga II Midline Report July 2017 Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Parents who have heard of iron 8% 343 15% 262 7*** (0.0026) Parents who can cite one benefit of irona 78% 23 58% 40 -20* (0.0991) Parents who can cite a food containing irona 28% 29 53% 40 25** (0.0390) Source: Parent Survey; IMPAQ calculations a Sample comprises only parents who had heard of the nutrient. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.3.2 Attitude towards Girls’ Education To measure parents’ attitude towards girls’ schooling, we asked parents whether they think schooling is good for girls. We also asked those with positive opinions to select from a list of reasons for supporting girls’ education. Exhibit 28 shows the changes in parents’ opinions about girls’ schooling. At midline, virtually all parents (100 percent) showed positive attitudes toward girls’ education; this represents a 21 percentage point increase from baseline, a statistically significant finding (p < 0.01). The exhibit also shows that the majority of parents mentioned that educating girls is good because it improves living standards, an increase of 14 percentage points over baseline (p < 0.01). The percentage of parents who said that girls’ schooling improves their health decreased by 6 percentage points, and the percentage of those who said that education allows girls to find better work decreased by 8 percentage points; both findings are significant (p < 0.01). Exhibit 28. Parents’ Opinions of Girls’ Schooling Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Parents who indicated that girls’ schooling is good 79% 343 100% 262 21*** (0.0000) Parents who indicated that girls’ schooling is bad 2% 343 0% 262 -2 (0.1195) Parents who did not know the answer 20% 343 0% 262 -20*** (0.0000) Source: Parent Survey; IMPAQ calculations * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 IMPAQ International, LLC Page 40 Beoog Biiga II Midline Report July 2017 Exhibit 29. Parents’ Reasons for Supporting Girls’ Schooling Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Parents who say educating girls improves living standards 54% 270 68% 261 14*** (0.0009) Parents who say educating girls improves their health 7% 270 1% 261 -6*** (0.0002) Parents who say educating girls improves the health of their future children 5% 270 4% 261 -1 (0.5984) Parents who say girls should be able to fulfill themselves 16% 270 18% 261 2 (0.6841) Parents who say educating girls allows them to find better work 14% 270 6% 261 -8*** (0.0009) Source: Parent Survey; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.3.3 Parents’ Engagement in School We asked parents about their participation in school as PTA members and in the village SILC. We also asked those who do belong to SILCs whether they used their savings to cover school expenses. Exhibit 30 shows that more parents said they were involved in their children’s school at midline than at baseline. The proportion of parents who were members of the PTA increased significantly, by 15 percentage points (p < 0.01). Exhibit 30 also shows a considerable increase in the percentage of parents who were SILC members; the 32 percentage point difference is significant at the 1 percent level. Of parents who were SILC members, a larger proportion at midline than at baseline reported that they used their savings to cover school expenses, but the change was not statistically significant. Of the parents who said that they used their savings for school expenses, the proportion of those who reported that the savings helped “a lot” with school fees decreased by 33 percentage points from baseline to midline. The proportion of those who said that the savings helped “some” increased by 38 percentage points. Both findings are statistically significant. A possibility of increase in education expenses or a decrease in parents’ purchasing power to afford school costs might be potential explanations to this shift in parents’ responses. We asked parents about their participation in school as a PTA member, as well as in SILC group to measure the extent to which the savings are used to cover school expenses. Exhibit 30 shows parents were more involved in their children’s schooling at midline. The proportion of parents who served as a member of a PTA more than doubled over time with statistically significant increase of 15 percentage points (p < 0.01). The exhibit also provides information about parents’ involvement in a savings group, SILC. From baseline to midline, there was a considerable increase in SILC membership from 9 percent to 41 percent, statistically significant at the 1 percent level. A larger proportion of parents reported that they used their savings to cover school expenses at midline, but the change was not statistically significant. However, at IMPAQ International, LLC Page 41 Beoog Biiga II Midline Report July 2017 midline, fewer parents reported that the savings helped a lot with school fees, while more of them thought that the savings only helped some. Exhibit 30. Parents’ Involvement in School and in Savings Cooperatives Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Parents who are members of the PTA 13% 343 28% 262 15*** (0.0000) Parents who are members of a savings group 9% 220 41% 262 32*** (0.0000) Parents who have used their savings for school expenses a 48% 21 63% 107 15 (0.2023) Parents indicating the savings helped “a lot” with school fees and tuition a 44% 9 12% 67 -33** (0.0116) Parents indicating the savings helped “some” with school fees and tuition a 11% 9 49% 67 38** (0.0309) Parents indicating the savings helped “a little” with school fees and tuition a 44% 9 39% 67 -6 (0.7492) Source: Parent Survey; IMPAQ calculations. a Sample comprises only parents who are members of a savings group. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 We asked parents whether they meet individually with teachers of their children to assess their involvement in school. We asked the same questions from teachers to triangulate their responses with parents’. Exhibit 31 shows the level of one-on-one meetings held between parents and teachers reported by both types of respondents. At midline, more parents (20 percentage points) reported they met individually with teachers in the past 12 months, statistically significant at the 1 percent level. Their responses were more or less consistent with what teachers reported. Also, more teachers (15 percentage points) reported they met one-one-one with parents at least 3 times in the past year at midline. However, this increase was lower when reported by parents (4 percentage points), but not statistically significant at any levels. The primary reason reported by both parents and teachersto meet with each other was “child’s performance” at both baseline and midline, with no statistically significant changes. Exhibit 31. Parents –Teacher Individual Meetings Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Teacher met individually with parents of students in the past 12 months a 74% 54c 83% 105 9 (0.1931) IMPAQ International, LLC Page 42 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Teacher met with parents of students about student performance at least 3 times in the past 12 months a 48% 40 62% 87 15 (0.1249) Parent or another adult in household met individually with teacher over the past 12 months b 45% 343 65% 262 20*** (0.0000) Parent or another adult in household met individually with teacher over the past 12 months b 28% 154 32% 170 4 (0.4189) Source: a Teacher & b Parent Survey; IMPAQ calculations; c Excluded observations from baseline are students who did not answer these questions * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.4 Teacher Outcomes From the teacher survey, we gathered baseline and midline data on teacher outcomes in the following three areas: • Nutrition Knowledge • Attendance • Classroom practices and teaching techniques In addition, this section also provides information on • Classroom observations, and • Students’ Attentiveness reported by teachers 5.4.1 Nutrition Knowledge To assess teachers’ nutrition knowledge, we asked them the same questions about vitamin A and iron that we asked students and parents. Exhibit 32 shows changes in teachers’ knowledge of nutrition from baseline to midline. Overall, since teachers seemed to be knowledgeable about nutrition at the baseline, changes were expected to be small. The data show that a larger proportion of teachers had heard of vitamin A at midline than at baseline, a 4 percentage point difference that is significant at the 10 percent level. However, their knowledge of vitamin A’s benefits or the foods containing this nutrient have not changed significantly. By contrast, although a slightly smaller proportion of teachers had heard of iron at midline among those teachers who had heard of iron, larger proportions could name a benefit and a food containing iron. The differences of 18 and 15 percentage points, respectively, are statistically significant (p < 0.01). This confirms that most teachers seem to have the knowledge necessary to increase students (and community) nutrition awareness, but may need support on how to teach it effectively. This may explain the reason why students’ nutrition knowledge (Section 5.2.1) is still significantly low despite a slight increase over time. IMPAQ International, LLC Page 43 Beoog Biiga II Midline Report July 2017 Exhibit 32. Teachers’ Nutrition Knowledge Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Teachers who have heard of vitamin A 95% 101 99% 105 4* (0.0888) Teachers who can cite one benefit of vitamin Aa 78% 96 78% 104 0 (0.9675) Teachers who can cite a food containing vitamin A a 71% 77 75% 104 4 (0.5929) Teachers who have heard of iron 98% 101 96% 105 -2 (0.4375) Teachers who can cite one benefit of irona 69% 87 87% 101 18*** (0.0023) Teachers who can cite a food containing irona 75% 69 90% 101 15*** (0.0097) Source: Teacher Survey; IMPAQ calculations. Note: Excluded observations from baseline are teachers who did not answer these questions. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 a Sample comprises only of teachers who had heard of the nutrient. 5.4.2 Attendance To measure the number of teachers who teach “regularly,” at least 90 percent of normal school days during the year, we collected attendance data for a sample of 110 teachers of grades 2 to 6 in the 22 targeted schools. The data were collected from school district administrators during the months of February, March, and April 201733 . Exhibit 33 compares the three-month attendance rate of teachers at baseline and midline. On average, a somewhat larger proportion of teachers attended and taught at least 90 percent of school days at midline (88 percent) than at baseline (85 percent), but the increase is not statistically significant. Teacher attendance data should be interpreted with caution. Attendance is a direct indicator of teacher performance that teachers and school principals can be reluctant to share, as our field experience shows. 34 In addition, we observed a high absence rate (25 percent) in March 2017, at midline. This high absence rate may have occurred due to a terrorist attack in the northern part of Burkina Faso during the reference period. This event could have led teachers to leave their schools or be afraid to report the following days. 33 We used this timeframe for collecting teacher’s attendance based on the performance monitoring plan (PMP), approved by USDA. Enumerators collected teacher attendance by asking school District Administrators to call principals. 34 At baseline, two school principals did not share the teacher attendance data. IMPAQ International, LLC Page 44 Beoog Biiga II Midline Report July 2017 Exhibit 33. Teacher’s Attendance over the Past Three Months Baseline Midline Difference in Means Indicator Percent Total Number of Observations Percent Total Number of Observations (p-value) Teachers who taught at least 90% of school days in February, March, and April 85% 95 88% 110 3 (0.3) Source: Teacher attendance data; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.4.3 Classroom Practices and Teaching Techniques Applying a variety of activities in class, including some that are teacher-centered, student-centered, and group-centered, is an effective way to teach children with diverse learning styles (Montgomery S. M. et al, 1998). To measure the extent to which teachers applied or planned to apply new teaching techniques and activities, we asked them about classroom practices they used or planned to use in a day or a week, including asking children to work in groups, to write, and to check each other’s work. Exhibit 34 outlines teachers’ responses about whether they had applied or planned to apply the seven techniques taught in BBII training on the day of data collection. It also shows changes from baseline to midline. The proportion of teachers who had already done each type of activity decreased significantly (p < 0.01) from baseline to midline with one exception: asking students to check their own work. The largest decrease (41 percentage points) was in asking students to write solutions on a slate to show to the class. By contrast, there were significant increases in the proportions of teachers who were planning to use several teaching techniques; however, pairing children with the same skill level was planned by a smaller proportion of teachers at midline (an 11 percentage point decrease). Findings on proportions of teachers who did not plan on using a particular activity are heterogeneous among the techniques. The activities with the largest increases in proportions of teachers who did not plan to use them were pairing children with same skill level (28 percentage points) and asking students to write on a slate for the class (25 percentage points). These findings are significant at the 1 percent level. Assuming that teachers’ schedules might be different from one day to another, we also asked teachers about the activities they used, planned to use, or did not plan to use during the week. The results, reported in Exhibit 50 in Appendix C, were more or less the same as the one-day findings shown in Exhibit 34. Most teachers reported that they were planning to use most of the teaching techniques. However, there were significant decreases from baseline to midline in the percentage of teachers who had done certain activities, including having students check each other’s work (a 23 percentage point decrease). IMPAQ International, LLC Page 45 Beoog Biiga II Midline Report July 2017 Exhibit 34. Utilization of Classroom Activities for the Day of Data Collection Utilization Baseline Midline Difference in Means Percent Total number of Observations Percent Total number of Observations p-value of the test Individual students check their own work and give themselves a mark or comments. Already used 23% 101 15% 105 -8 (0.1692) Planning to use 13% 101 27% 105 14** (0.0130) Not used and not planning to use 64% 101 58% 105 -6 (0.3591) Students check each other' s work Already used 46% 101 21% 105 -25*** (0.0002) Planning to use 22% 101 36% 105 14** (0.0229) Not used and not planning to use 33% 101 43% 105 10 (0.1333) Whole class checks the work of one student. Already used 43% 101 22% 105 -21*** (0.0014) Planning to use 25% 101 37% 105 12* (0.0551) Not used and not planning to use 33% 101 41% 105 8 (0.2203) Students write solutions on a slate and show to the teacher and class. Already used 64% 101 23% 105 -41*** (0.0000) Planning to use 19% 101 35% 105 16*** (0.0079) Not used and not planning to use 17% 101 42% 105 25*** (0.0001) Students of different skill levels are paired together. Already used 50% 101 31% 105 -19*** (0.0051) Planning to use 29% 101 33% 105 4 (0.4762) Not used and not planning to use 22% 101 36% 105 14** (0.0229) Students of the same skill level are paired together. Already used 27% 101 10% 105 -17*** (0.0012) Planning to use 22% 101 11% 105 -11** (0.0270) IMPAQ International, LLC Page 46 Beoog Biiga II Midline Report July 2017 Utilization Baseline Midline Difference in Means Percent Total number of Observations Percent Total number of Observations p-value of the test Have not and not planning to use 52% 101 80% 105 28*** (0.0000) Teacher asks group of 3 or more students to work together on a project and later provides feedback on group performance. Already used 39% 101 13% 105 -26*** (0.0000) Planning to use 21% 101 26% 105 5 (0.4060) Have not and not planning to use 41% 101 61% 105 20*** (0.0033) Source: Teacher survey; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 After examining individual classroom activities in the survey data (Exhibit 34), we determined a threshold to indicate thorough adoption of new teaching techniques: applying or planning to apply at least five of the seven different activities on the day of data collection. Exhibit 35 shows that, at midline, teachers were almost 9 percentage points less likely than at baseline to reach this threshold; however, this difference was not statistically significant. There are some potential explanations for this decrease. As mentioned in Section 4.4, teachers at midline were less educated compared to teachers at baseline. As discussed in Section 6.2, teachers with limited education found the training materials more difficult to understand, which may have led them to continue using traditional approaches—approaches they were more accustomed to using. . Another potential explanation could be the timing of training session. As explained in Section 6.2.2, some teachers reported that training sessions were held at the end of the school year, which did not give them enough time to prepare and apply the techniques in the same academic year. This decrease, however, should be interpreted with caution since the training component of the project had not yet started at baseline. However, more than half of the surveyed teachers (53 percent) at baseline reporting use of new techniques35 may suggest that teachers were already aware of these techniques even in the absence of the training. Exhibit 35. Teachers Using New Techniques or Tools Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Teachers who demonstrate use of new techniques or tools 53% 101 44% 105 -9 (0.2153) Source: Teacher Survey; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 35 CRS monitoring data in 2016 also showed more or less the same value (52 percent) as the baseline value of this indicator. IMPAQ International, LLC Page 47 Beoog Biiga II Midline Report July 2017 We assessed whether teachers were applying the BBII literacy instructional techniques to teach reading skills by asking teachers to provide examples, without enumerator prompting, of an activity or an instruction they used for teaching reading, such as phonological awareness, phonetics, vocabulary, and so on. Then, we calculated the average time that teachers reported they spent on all the literacy instruction per day. This indicator reflects both teachers’ understanding of and their practice of literacy instruction. Exhibit 36 shows that a larger proportion of teachers devoted at least 45 minutes a day to literacy instruction at midline than at baseline (11 percentage points), but the increase was not statistically significant. Exhibit 36. Time Spent by Teachers on BBII Literacy Instruction Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Proportion of teachers who devote at least 45 minutes a day to literacy instruction 55% 91 66% 103 11 (0.1160) Source: Teacher Survey; IMPAQ calculations. Note: Excluded observations from baseline are teachers who did not answer this question. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 In addition, to understand the extent to which teachers applied literacy practices without focusing on specific techniques, we asked teachers how frequently they have students read by themselves, read to others, listen to others’ reading, write, and play word games in class. Teachers were also asked how much time they spent on each activity per week. The results are shown in Exhibit 37. There were significant increases in the proportions of teachers who had students listen and read to others (10 and 7 percentage points, respectively) at least once a week. There was no change in teachers focusing on children’s writing. Teachers’ responses were more or less the same as students’, especially in practicing listening to others read aloud. Exhibit 37. Implementation of Literacy Activities Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Students read by themselves more than once a week 94% 92 90% 38 -4 (0.4397) Students read to others more than once a week 87% 83 94% 89 7* (0.0859) Students listen to others read aloud more than once a week 87% 77 98% 90 11*** (0.0071) Students write more than once a week 92% 99 92% 101 0 (0.9785) Students play word games more than once a week 83% 81 81% 91 -2 (0.8132) IMPAQ International, LLC Page 48 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) Average number of minutes spent per week on all literacy-related activities 60 91 62 103 2 (0.7435) Source: Teacher Survey; IMPAQ calculations. Note: Excluded observations from baseline are teachers who did not answer these questions. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.4.4 Classroom Observations To observe whether trained teachers have been implementing the new teaching techniques and using the material provided by the BBII program, we observed 2536 classrooms in the five randomly selected schools. We also observed teachers’ activities to see if they were implementing other instructional practices beyond BBII techniques. BBII Teaching Techniques As explained in Section 2.1.3, we used BBII literacy training materials to develop a classroom observation tool. BBII pedagogical instructions covers two literacy components: 1. Reading (lecture) skills: decoding (déchiffrage), defined as translating written words into the sounds and meanings of spoken words (often silently), or reading comprehension (compréhension). 2. Writing (écriture) skills: spelling (orthographe) or other conventions of written texts, such as grammar; written production. 37 Teachers are trained to apply decoding or reading comprehension to develop children’s reading skills. Writing and reading sessions are expected to be taught separately. Teachers usually cover a complete lesson in 6 sessions over two days. In addition, teacher training also encourages teachers to implement all the techniques by asking children to work in groups, especially for reading/decoding lessons. Reading groups were structured to let students choose a member who could read the text; pay attention; look up new words in a dictionary; answer comprehension questions; exercise syllabic awareness; and identify punctuation marks in collaboration with other group members. The observational data showed that across all five grades in five schools, teachers mostly focused on using decoding techniques (21 lessons out of 25), particularly in grades 1 to 3. In none of the observed classrooms did the teacher use any of the writing techniques. Reading comprehension techniques were 36 We observed one teaching lesson per grade in each of the schools. The observation took between 40 to 50 minutes to ensure that the whole lesson was taught completely by the teacher. 37 Written production enables students to develop and communicate their ideas, feelings, interests, and concerns. This form of communication requires development of skills and strategies that children gradually master over time through formal education. IMPAQ International, LLC Page 49 Beoog Biiga II Midline Report July 2017 observed in only four lessons, 38 one lesson in grade 4 and three in grade 5. These findings suggest that teachers were mostly comfortable using decoding techniques. Exhibit 38 shows the proportion of observed lessons in each grade in which teachers asked student to work in reading groups to implement BBII techniques. In 48 percent of observed lessons (12 lessons out of 25), students worked in groups focusing on reading skills. Teachers used reading groups mostly to practice decoding; reading groups were used less frequently in earlier grades. Exhibit 38. Observed BBII Techniques Using Reading Groups Grade Reading/Comprehension Reading/Decoding All lessons observed Grade 1 (CP1) 0% 20% 20% Grade 2 (CP2) 0% 20% 20% Grade 3 (CE1) 0% 60% 60% Grade 4 (CE2) 0% 100% 80 a % Grade 5 (CM1) 67% 50% 60% All Grades 50% 50% 48% Source: Classroom observation tool; IMPAQ`s Calculations. N= 25 observed lessons (5 lessons per grade= 5*5). In 5 observed lessons in 4th grade (1 reading comprehension and 4 decoding lessons), only 4 were observed to be done in groups. As shown in Exhibit 39, decoding and comprehension are each identified with a range of key elements that we observed in the classrooms. Exhibit 39. Observed Reading Lesson Elements Decoding Elements • Combination of letters and sounds • Reading aloud • Letters recognition • Respect of liaisons • Articulation of words • Respect of punctuation • Identification of breath groups • Morphemes and tool words • Fluency Comprehension Elements • Use of strategies to find answers in a text • Definition of new words • Comprehension of a whole sentence • Explicit teaching of comprehension (connectors, substitutes, punctuation, and so on) • Implicit comprehension and inference • Use of dictionary • Construction of hypothesis Source: Classroom observation tool. 38 In one of these four lessons, the teacher used comprehension techniques to remind children what was taught in the previous session. IMPAQ International, LLC Page 50 Beoog Biiga II Midline Report July 2017 In four observed lessons which applied reading comprehension instruction, using strategies to find answers in a text (for example, searching for familiar words) was the element observed most frequently, in three out of four lessons. Asking students to make hypothesis about the reading was not observed in any lessons. In one lesson, we observed that the teacher both taught comprehension explicitly by explaining the connectors’ chronological order and by asking questions that required implicit comprehension. Exhibit 40 shows the percentages of observed lessons in each grade in which teachers implemented key decoding elements (21 observed lessons). We observed variation in implementing the key decoding elements across grades. Teachers used a greater variety of decoding elements in the higher grades. As expected, practice with letter recognition and combination of letters and sounds were observed more frequently in lower grades. First- and second-graders did not read aloud in any of the observed lessons. Fluency was observed only in fifth grade; instruction in morphemes was not observed in any of the 21 lessons. Exhibit 40. Key Elements Observed Reading/Decoding Lessons by Grade Decoding Key Element Grade 1 (5 lessons) Grade 2 (5 lessons) Grade 3 (5 lessons) Grade 4 (4 lessons) Grade 5 (2 lessons) All grades (21 lessons) Combinations 100% 80% 80% 25% 50% 71% Reading aloud 0% 0% 80% 100% 100% 48% Letter recognition 80% 80% N/Aa N/Aa N/Aa 80% Respect of liaisons 0% 0% 20% 50% 50% 19% Articulation of words 0% 40% 40% 25% 0% 24% Respect of punctuation 0% 0% 40% 0% 50% 14% Identification of breath groups 0% 0% 0% 25% 50% 10% Morphemes and tool words 0% 0% 0% 0% 0% 0% Fluency 0% 0% 0% 0% 50% 5% Source: Classroom observation tool. IMPAQ’s Calculations; a This activity is not relevant to higher grades. These findings are in line with the qualitative data. Many teachers reported using decoding activities and reading group sessions during class useful. Teachers appreciated the reading group technique because they found that, during these sessions, students with stronger reading abilities mentor students with less￾developed skills. Another useful technique from teachers’ perspective is the inference technique, in which they invite students to answer questions after reading a text. This technique allows teachers to assess students’ comprehension and their creative and critical thinking abilities. For example, teachers can ask students to create a title for a body of text or to imagine the rest of the story. Teachers reported using this technique less than the reading group sessions. Teachers reported that they used the inference technique less frequently than others, such as reading group sessions, because some students do not have the skills necessary to understand the exercise, especially those in lower grades. Other Classroom Activities and Practices IMPAQ International, LLC Page 51 Beoog Biiga II Midline Report July 2017 To understand teacher classroom activities beyond BBII techniques, we observed other instructional practices that teachers used to develop various reading skills. While only four observed lessons out of 25 exclusively applied BBII reading comprehension techniques, the observations show that teachers integrated the comprehension elements into reading/decoding lessons, reinforcing the interconnection between the two reading skills (Exhibit 41). In 72 percent of the total observed lessons, teachers asked simple comprehension questions; in 60 percent, they asked detailed comprehension questions. However, generally teachers did not carry out any writing activities, either in groups or individually, including spelling or written production. In 36 percent of total observed lessons, teachers asked students to encode (spell) a word or sentence. However, they did not ask students to write on a free topic or improve their own written production. Exhibit 41. Other Observed Teacher Practices by Grade Activity Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 All a. Worked with students on recognition of letters or mastery of the complete alphabet 100% 80% 20% 0% 20% 44% b. Carried out activities on recognition of sounds of letters and letter combinations 100% 100% 40% 0% 0% 48% c. Had students decode an unknown word, sentence, or text, without repetition or memorization 60% 80% 80% 60% 20% 60% d. Had students encode a word, sentence, or text using knowledge of sounds and letters, regardless of spelling or other conventions of written texts 100% 40% 40% 0% 0% 36% e. Developed students’ lexical awareness or vocabulary: finding new words on a given subject, deducing new words’ meaning using their structure or the context, using tools such as dictionaries or vocabulary sheets, and so on 0% 20% 20% 40% 60% 28% f. Had students read a word, a sentence, or a text, aloud or not 60% 60% 60% 80% 80% 68% g. Had students read a sentence or a text with fluency and expression (respecting punctuation, not stopping at each word, reading for meaning) 0% 0% 80% 100% 100% 56% h. Asked students to answer simple comprehension question or to locate answers in a text 20% 60% 80% 100% 100% 72% i. Asked students to answer more subtle comprehension questions or questions of implicit comprehension involving, for example, causes and consequences, chronology, or riddles 0% 60% 60% 80% 100% 60% j. Asked students to make a hypothesis about a text 0% 0% 0% 0% 20% 4% k. Asked students to establish links from a text to their real life or to another text 0% 20% 0% 20% 0% 8% l. Asked students to write a word or a sentence, without copying or reciting but perhaps with guidance such as a pattern or “mad lib” 0% 0% 0% 0% 0% 0% IMPAQ International, LLC Page 52 Beoog Biiga II Midline Report July 2017 Activity Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 All m. Asked students to write on a free topic 0% 0% 0% 0% 0% 0% n. Asked students to deduce or practice rules of spelling, grammar, or conjugation 0% 0% 0% 20% 0% 4% o. Helped students to improve their own written production for style, coherence, spelling, punctuation, and so on 0% 0% 0% 0% 0% 0% Source: Classroom observations tool; IMPAQ’s calculations. Again, these findings are in line with what we heard during interviews with teachers. Teachers mentioned using these techniques infrequently. Teachers reported using these techniques infrequently for the following reasons: • Preparing lessons containing the new techniques can be time consuming and teachers may not have (or prioritize) the time needed to properly prepare for lessons. • Applying some of the new techniques in the classroom can be time intensive so teachers are reluctant to incorporate them into their lessons. • Some students, especially those in lower grades, do not have the skills required to fully comprehend and participate in some of the new techniques. • The training did not emphasize practice of the new techniques so some teachers feel ill-equipped to apply them in the classroom. • Some teachers did not fully understand the purpose and method of some of the techniques so have shied away from using them during lessons. School Supplies and Learning Materials We noted what school supplies and learning materials were used during the 25 observed lessons. In all 25 classrooms, teachers used blackboards. After blackboards, small slates (used in 60 percent of observed classrooms) and large slates (44 percent of classrooms) were the most commonly used learning materials, particularly in the earlier grades as expected39 . Again, interviews with teachers confirmed teachers’ preference in using blackboards and slates. With teachers focusing more on reading/decoding techniques (21 observed lessons out 25), we expected to see teachers using words learning materials more frequently. However, observations show otherwise. Reading books were used most often in fourth-grade and never in first- or second-grade lessons. Only second-grade classes (three out of five) used reading text posters or pictures. With teachers focusing on reading/decoding techniques (21 observed lessons out of 25), we expected to see frequent use of materials to facilitate learning words, but this was not the case. Learning materials such as letter cubes and sounds tables were not used frequently during the observed lessons (only once). We did not observe teachers using Bananagrams40 or word strips (étiquettes mobiles) 41 in any of the 25 classrooms. In general, classroom observation outcomes were more or less in line with the results of student and teacher surveys. Teachers focused on reading skills, especially decoding, and children’s reading skills seem to have improved. All three data sources (classroom observation, student surveys, and teacher surveys) showed less emphasis on writing skills. This area needs further research study to probe reasons that teachers are not emphasizing writing skills as much as reading skills. If this area of focus is added to the 39 Small and large slates were mostly distributed to first and second grades. 40 Bananagrams is a word game in which lettered tiles are used to spell words. 41 Translated as Étiquettes mobiles. IMPAQ International, LLC Page 53 Beoog Biiga II Midline Report July 2017 scope of work and budget, we can increase the number of classroom observations to find out whether training was not sufficient to encourage teaching of writing skills or whether teachers are not comfortable applying what they learned. 5.4.5 Attentiveness Together with CRS, IMPAQ decided how to best measure student attentiveness. IMPAQ and CRS noted that it is very hard to judge objectively the degree to which a given child is attentive. Some children are very restless when they are attentive, while others, on the contrary, are very still. Conversely, a very still student might be sleepy and an active one might only be distracted. Hence, even classroom observations may not be a good measure of attentiveness. Moreover, classroom observations are tricky because they must be completed in exactly the same circumstances in each classroom in order to get quality, comparable data. IMPAQ and CRS, therefore, decided that the best measure of a child’s attentiveness was the teacher’s own assessment of his or her students’ attentiveness. To measure this indicator, we collect data from teachers who are best able to judge the relative degree of the students’ attentiveness. We asked teachers on a scale of 1 to 10, where 1 is not attentive at all, to rate their students’ attentiveness. Exhibit 42 shows that according to teachers, students’ attentiveness have not changed over time. However, given the difficulty and subjectivity of measuring attentiveness, this outcome should be interpreted with caution. Exhibit 42. Students’ Attentiveness Reported by Teachers Indicator Baseline Midline Difference in Means Percen t Total Number of Observations Perce nt Total Number of Observations (p-value) Proportion of attentive students identified by teachers 45% 95 47% 105 2 (.8433) Source: Teacher survey; IMPAQ’s calculations. 5.5 PTA Outcomes To measure the level of community involvement with children’s schooling, we surveyed the PTA leader from each school. We asked these leaders questions related to PTA meetings held in the past year, the functioning of the school canteen, and the extent of community support for the school. We also asked them about the number of months in which the school canteen was covered by MENA, CRS, parents, and other responsible parties. It is important to note that the results presented in this section rely on a very small sample of PTA leaders; estimates for the indicators and their change over time have to be taken with caution. In general, the data do not suggest much change in the number of meetings held by PTA members over time. Exhibit 43 shows that there was no change in the average number of general assemblies or PTA council meetings, as expected. At midline, less schools (8 percentage points) held at least three general assembly meetings compared to baseline but not statistically significant at any conventional levels. A smaller proportion of PTA members attended general assemblies at midline than at baseline, but the decrease of 9 percentage points was not statistically significant. IMPAQ International, LLC Page 54 Beoog Biiga II Midline Report July 2017 Exhibit 43. Parent-Teacher Association Meetings Indicator Baseline Midline Difference in Means Mean/ Percent Total Number of Observations Mean/ Percent Total Number of Observations (p-value) Number of general assembly meetings held in past 6 months 3 21 3 22 0 (0.3481) Schools that held at least 3 general assembly meetings in past 6 months 71% 21 68% 22 8 (.8220) PTA representatives who attended a general assembly 100% 25 91% 22 -9 (0.1288) Number of PTA council meetings held past school year 4 21 4 21 0 (0.9452) Source: PTA Survey; IMPAQ Calculations * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 As shown in Exhibit 44, the proportion of PTA representatives who were involved in many school￾community activities decreased from baseline to midline. Having a meal with students at the school canteen decreased by 4 percentage points, participating in a school project by 1, and visiting a classroom by 12; however, the changes were not statistically significant. A marginally statistically significant decline (p < 0.10) is seen in the proportion of PTA representatives who served as cooks or storekeepers, a decrease of 25 percentage points. One potential explanation to this significant decrease is that PTA leaders might focus more on leadership responsibilities at midline, while these roles were filled by other PTA members compared to baseline. These results should be interpreted with caution given the small sample of PTA members. We recommend qualitative research at endline in this area to probe deeper into potential issues. The one area in which the proportion of PTA representatives involved in the school increased significantly (p < 0.10) is seeing a performance by the children, a difference of 29 percentage points. Exhibit 44. PTA Representative Involvement in the Past 12 Months Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) PTA representatives who have had a meal with students at school 4% 25 0% 22 -4 (.35) PTA representatives who have participated in school￾community projects such as cleaning 92% 25 91% 22 -1 (.9) PTA representatives who have helped the school as cook or storekeeper 52% 25 27% 22 -25* (.09) PTA representatives who have helped watch over a reading group 0% 25 9% 22 9 (.13) IMPAQ International, LLC Page 55 Beoog Biiga II Midline Report July 2017 Indicator Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value) PTA representatives who have visited a classroom 76% 25 64% 22 -12 (.37) PTA representatives who have attended a performance put on by students 44% 25 73% 22 29* (.05) PTA representatives who have helped in other ways 8% 25 5% 22 -3 (.64) Source: PTA Survey; IMPAQ Calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 Exhibit 45 shows PTA representatives’ reports on the total number of months in which the canteen was supported by MENA, CRS, parents, and other responsible parties. There is a significant decrease (by 2 months) at midline. This outcome should be interpreted with caution, because there is a possibility that canteen operations were misrepresented at baseline. PTA members over-reported at baseline the total number of months during which the government supported the canteen. Exhibit 45. Support for Canteens Indicator Baseline Midline Difference in Means Mean Total Number of Observations Mean Total Number of Observations (p-value) Number of months of community and/or government support for canteens 5 14 3 15 -2* (.06) Source: PTA Survey; IMPAQ Calculations. Excluded observations are the PTA members who did not know the answer. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 5.6 School District Administrator Outcomes In the BBII program, school district administrators have been responsible for training teachers on pedagogical practices. To gauge their level of confidence in and knowledge of teacher training programs, we asked district administrators about the importance of each literacy instruction subject they taught, the number of training sessions they held, and any supervision or observational activities they conducted. Due to the small sample at the baseline, the results presented below are merely descriptive; no statistical analysis (t-test of difference in means) can be conducted. Exhibit 46 shows which reading instructions district administrators said were most important. Compared to baseline, a larger proportion of administrators reported that phonemics (hearing, identifying, and manipulating individual sounds) is an important reading technique. Phonemics is considered to be the basis for learning phonics. By contrast, the proportions of administrators who mentioned phonics and fluency as important reading instructions decreased considerably. IMPAQ International, LLC Page 56 Beoog Biiga II Midline Report July 2017 Exhibit 46. Important Reading Instructions Reported by District Administrators Baseline Midline Technique Percent Total Number of Observations Percent Total Number of Observations Phonemic awareness 62% 13 73% 30 Phonics 100% 13 27% 30 Fluency 100% 13 67% 30 Vocabulary 62% 13 53% 30 Comprehension 85% 13 90% 30 Other techniques N/A* N/A * 13% 30 None of the Proposed Techniques N/A * N/A * 10% 30 *Option not available in baseline. Source: School district administrator survey; author’s calculations. At midline, almost all (29 out of 30) district administrators said that they held at least one teacher training on reading skills, in the previous school year. IMPAQ defined this indicator, demonstrating use of new techniques or tools, as school district mangers transferring skills to others through trainings. They reported an average of four training sessions with 122 teacher trainees. The number of district administrators using new techniques improved over time; only 8 out of 13 administrators surveyed held such training sessions at baseline. Furthermore, nearly all of the administrators (29) said they had observed, supervised, and/or monitored teachers in their classes during the previous academic year, conducting an average of 26 visits lasting an average of 85 minutes each. At baseline, only 9 out of 13 administrators reported that they had observed classrooms. At midline, administrators reported that they had seen 81 percent of teachers using at least one of the BBII literacy activities (writing, listening to others reading, reading to others, reading alone, and playing with words). This proportion is similar to what administrators reported about classroom observations at baseline. In interviews, teachers reported that they appreciate the teacher observations conducted by school district administrators, as they receive important technical support on teaching techniques and feedback on techniques that they have not yet mastered. Teachers also said that observations prompt them to focus on the quality of their teaching and follow the practices and standards they learned from the literacy trainings. IMPAQ International, LLC Page 57 Beoog Biiga II Midline Report July 2017 SECTION 6. QUALITATIVE FINDINGS Below is a summary of the main findings by the qualitative research domain. More detail regarding these results is elaborated in the narrative that follows. Summary of Main Qualitative Findings Relevance Overall, respondents reported satisfaction with the project, saying they that believed the project interventions are meeting the needs of beneficiaries and are aligned with Burkina Faso’s and the U.S. government’s development objectives. Respondents also believed that the project adequately considers economic, cultural, and political contexts. According to respondents, the primary strengths of the project design include its holistic approach to child development, its focus on evidence-based literacy techniques and provision of teaching resources, and its focus on community and government engagement. The primary weaknesses, according to stakeholders, include little ability to deal with lack of motivation on the part of teachers, lack of key equipment and resources for teacher monitoring and support, and the design of reading clubs. Effectiveness Respondents reported that immediate outcomes were reached in most dimensions of the project and that the interventions have contributed to expected results. Teachers reported that they have observed improvements in students’ literacy skills and motivation to learn. Teachers and students also reported that hygiene education has changed students’ behavior, as they are now washing their hands more frequently. Students appreciated the school meals and take-home rations received from the canteens, and teachers believed school attendance was closely tied to the presence of canteens. Though challenges exist, respondents said that CRS has effectively managed partners and that the program has coordinated and collaborated effectively with other stakeholders. Efficiency Respondents reported that the project resources have been sufficient to implement scheduled activities and, for the most part, objectives have been achieved on time. Some activities, including the provision of food in canteens and the provision of training materials, have been delayed due to slow administrative processes. Further, respondents noted that partners’ competing agendas and priorities sometimes exacerbate these delays. Despite challenges, respondents reported that CRS has adequately responded to internal and external factors that have hindered the efficient implementation of project activities. Perceived Impact Respondents believed that the project is on target to meet its medium- and long-term goals, particularly those related to students’ literacy outcomes, and that the effects are due to the project interventions. As for hygiene-related outcomes, respondents noticed positive changes at some schools related to students’ hygiene habits. Respondents believed that, if delivered properly, the educational activities related to hygiene and nutrition will strengthen students’ skills and knowledge. However, respondents described that limitations such as teachers’ lack of motivation to teach hygiene and nutrition knowledge and the lack of clean water sources and latrines may result in less significant outcomes. Respondents recommended several strategies to improve the impact of the project. Recommendations related to literacy included modifications to the teacher trainings to ensure that they are long enough to cover all content and allow sufficient time for practice, development of a motivational framework for teachers that focuses on incentives to implement new literacy techniques, and provision of sufficient materials and resources to enable teachers and students to implement these techniques. Recommendations related to health activities included engaging more teachers in health trainings to further promote the spread of information and ensuring that infrastructure, such as handwashing stations, are operational. IMPAQ International, LLC Page 58 Beoog Biiga II Midline Report July 2017 Summary of Main Qualitative Findings Sustainability Respondents believed that the engagement of communities and the government are key factors to sustainability and that CRS and its partners have successfully engaged these parties. There is still work to be done, however, to ensure project activities can be sustained after USDA funding ends. Respondents reported a need to organize a workshop with all relevant players to develop sustainability strategies and define how all parties can play a role in sustainability. 6.1 Relevance Through interviews with USDA, MENA, project implementers, and county mayors, the research team assessed the relevance of BBII interventions. Interview topics focused on stakeholders’ perspectives on the strengths and weaknesses of the project design and the extent to which the project considers economic, cultural, and political contexts. Stakeholders were also asked to share their perceptions on the extent to which the interventions are meeting the needs of beneficiaries and are aligned with Burkina Faso’s education and development strategies and with the U.S. government’s development objectives. Lastly, stakeholders shared their satisfaction with intervention activities. Below is a summary of the qualitative findings on the relevance of BBII. 6.1.1 Strengths and Weaknesses of Project Design Stakeholder respondents identified several strengths and weakness of BBII’s design. Below is a summary of their responses. Strengths of the Project Design Interviewees described several strengths that are helping BBII to meet its goals. Holistic approach of the project. Respondents believed that the project’s theory of change holistically and adequately captures factors, inputs, and activities that will lead to the intended long-term outcomes of the project. MENA representatives and project implementers said that the project design appropriately addresses the cyclical nature of student learning and health outcomes and, most importantly, involves the actors who are most vital to its success, such as teachers, PTA leaders, and community members. For example, understanding that students’ school performance will be negatively affected if they are hungry or sick due to hygiene-related diseases, the project encourages parents and community members to play an active role in encouraging their children to go to school, where they participate in literacy and health￾related education and receive supplemental food. Building on the BBI framework. Some project implementers thought that a strength of the BBII project is that the design relies on the findings and lessons learned from the evaluation of its predecessor, BBI. Building on best practices from BBI enabled CRS to hit the ground running and enhance its previous work to improve outcomes for children, respondents believed. Focus on literacy and teaching approach. All types of respondents reported that the most important strengths of the project design are the activities associated with improving students’ literacy abilities and the approach used to teach these literacy skills to students. Respondents believed that focusing on literacy abilities will have a positive impact not only on students’ ability to read and write but also on their ability to think critically across all disciplines. For example, as students improve their ability to decode and IMPAQ International, LLC Page 59 Beoog Biiga II Midline Report July 2017 understand text, they will be better able to comprehend disciplines like history, geography, science, and mathematics. Thus, focusing on student literacy is an effective way to improve overall learning outcomes, respondents explained. A related strength noted by teachers and school administrators is the approach the project uses to teach literacy, which respondents believe is more innovative and applicable than what is typically implemented by teachers. CRS and its partners use evidence-based techniques that come from the findings of education experts. The approach takes into account differences in students’ learning aptitudes. Further, the techniques promote the use of examples from students’ daily life to make reading and writing exercises relevant and tangible to students. Teachers found that these real-world examples actively engage students in learning, leading to improved outcomes. Alignment with education ministry strategic priorities. Another strength project implementers and MENA staff frequently mentioned was the project’s alignment with the strategic priorities of MENA. Respondents stated that the project effectively embraces MENA’s education objectives and actively encourages MENA’s involvement and participation. In partnership with CRS, MENA leads many field activities. For example, the training of teachers on the ground is conducted by MENA’s school district administrators. All types of respondents, including the USDA representative, believed MENA’s involvement in the implementation of teacher training — a key component of the project — is an important strength of the project’s design, as this partnership is important to sustainability. Teacher monitoring and support by school district administrators. MENA representatives mentioned that the BBII project does not limit its interventions to rolling out activities only; instead, subsequent follow-up is an integral part of the project design, particularly for the literacy component. In the design, school district administrators follow up and support teachers in the field as they implement the new literacy teaching techniques. Project implementers, MENA representatives, and teachers perceived this continued engagement of school district administrators as an important strength of the design, as it actively engages teachers in integrating the techniques, improves the quality of classroom management and instruction through quality monitoring, and provides hands-on support to teachers who experience challenges. Provision of learning materials. Respondents, particularly teachers, school administrators, and students, stated that the provision of training materials to teachers and learning materials to students is an aspect of the project that is valued by teachers and students alike. Respondents noted that such materials were greatly needed in classrooms, as they help facilitate teaching and learning. Weaknesses of the Project Design Respondents also identified weaknesses in the project design that may impede progress toward project goals. Those weaknesses relate to weak motivation of key actors, lack of key equipment and supplies for teacher monitoring and support, and the design of reading clubs. Weak motivation of key actors. During interviews, all types of respondents recognized the benefit of the project to communities and families and the project’s thoughtful design and approach to addressing relevant educational outcomes. However, respondents reported that some activities suffer from a lack of commitment from teachers. For example, some respondents noted that, unless teachers recognize the benefit of the new teaching techniques, they may not integrate the techniques into their daily lessons. Respondents believed that the project design did not adequately take into account teachers’ potential lack of motivation and fidelity to the model. IMPAQ International, LLC Page 60 Beoog Biiga II Midline Report July 2017 Lack of key equipment and supplies for teacher monitoring and support. Respondents reported that school district administrators sometimes lack the equipment they need to conduct scheduled school visits. For example, in some districts, mopeds dedicated to visits are frequently out of service, so that school district administrators must use their own means of transportation or postpone the trip. It is important to note that these supplies are provided by MENA and not directly via the BBII project. However, respondents described that resource limitations sometimes impact their ability to fulfill their BBII-related mission in the field. Further, respondents reported that teachers sometimes lack classroom materials, limiting their ability to implement new teaching techniques. For example, some teachers said they do not have enough books and teaching tools, such as large slates, to accommodate all of their students. Design of reading clubs. Teachers expressed that they have experienced challenges in finding volunteers willing and able to organize and facilitate reading clubs in the villages. Though the reading groups were intended to be led by teachers, secondary school students, or community mentors, teachers found that secondary school students do not have the skills necessary to facilitate these groups. Further, teachers reported that some parents do not allow their children to attend reading clubs, as they do not understand the importance of the activity and prefer that their children take care of domestic chores when not in formal school. Respondents believed that the design of the reading clubs did not anticipate the impact of these challenges on the clubs’ success or fully take into account the local context, including stakeholders’ priorities. 6.1.2 Alignment with Economic, Cultural, and Political Contexts Many respondents believed that the project effectively takes into account economic, cultural, and political contexts. For example, respondents explained that, for every BBII field activity, CRS field staff initiate talks with local communities to inform them of the activity and obtain their feedback and suggestions for an appropriate and effective roll-out of the activity in the local context. Furthermore, because the targeted areas continuously suffer from drought and food shortage, respondents said that the provision of food by CRS through school canteens is pertinent and important. 6.1.3 Alignment with Burkina Faso and U.S. Government Priorities When asked if the BBII project successfully aligns with Burkina Faso’s strategic goals related to education and development, project implementers and MENA representatives stated that the BBII project embraces the strategic priorities of Burkina Faso’s education ministry and found this alignment to be a strength of the project. The USDA representative believed the project closely aligns with USDA and U.S. government priorities. The literacy and school feeding components of the project are directly in line with priorities set by the Food for Education program at USDA. 6.1.4 Stakeholder Satisfaction CRS and its implementing partners reported high satisfaction with the BBII project, noting that it is well designed and implemented. Implementing partners reported satisfaction with CRS’ performance in implementing the project and managing stakeholders. All interviewed stakeholders emphasized that they hope the project continues, as they believe it has had a positive impact on children and communities, as discussed in more detail in Section 6.4 below. “The BBII project clearly aligns with [the] MENA strategic priority … to provide high￾quality, inclusive education of good quality for all.” –MENA Representative IMPAQ International, LLC Page 61 Beoog Biiga II Midline Report July 2017 Teachers reported mixed satisfaction with various aspects of the project. Teachers reported high satisfaction with the content of the literacy trainings and liked the design and content of the new teaching approaches and techniques. However, many teachers reported low satisfaction with the conditions of the literacy trainings, describing that the trainings were too short for the amount of content, the training spaces were inadequate (for example, the seats were for children, not comfortable for adults), and the per diem allowance, for many, was not enough to cover all expenses, particularly for “residents” as described in detail in Section 6.2.2. Teachers also reported mixed levels of satisfaction with the teacher observations. Several teachers described that the feedback given by the school district administrators during their visits was helpful and prompted them to prepare lessons and use techniques learned through the literacy trainings. However, some teachers also reported that these visits often disrupted their classroom agendas, primarily due to lack of communication about when the visits would take place. Often, teachers did not learn about visits until that day, when it was too late for teachers to ensure that their daily agenda was planned smoothly. Overall, teachers said that they appreciated the project’s immediate impact on their students’ behavior and literacy outcomes. As explained in Section 6.2 below, teachers described anecdotally positive changes they had seen in students’ literacy skills, changes that they believe will improve students’ performance in other disciplines. Students reported high satisfaction with the canteens, handwashing stations, and literary materials and tools. They appreciated and enjoyed the food received through the canteens and enjoyed using the handwashing stations and classroom materials. As described in the limitations section (Section 7.2), students did not discuss the new literacy techniques in detail. When we asked questions about teaching techniques, students were shy and unwilling to describe their experiences. Although students’ satisfaction was generally high, a handful of students from one focus group did report dissatisfaction with the micronutrients, saying that they experienced headaches and sometimes vomiting when taking them. 6.2 Effectiveness Through interviews with the stakeholder groups described in Section 2.2, the research team assessed the effectiveness of implementation strategies and activities. Interview topics focused on the extent to which the project is achieving its objectives, successes and challenges of program implementation, and the effectiveness of management arrangements and stakeholder collaboration. Below is a summary of the qualitative findings on project effectiveness. 6.2.1. Achievement of Objectives Respondents from CRS and its partners (MENA, OCADES, and FAVL) reported that the project has achieved its immediate goals in terms of implementation activities and is on track to achieve its longer-term objectives. Project implementers and school-level actors, including teachers and school administrators, reported observing positive changes, such as improvements in students’ literacy skills and behavior related to hygiene, such as handwashing. “I wanted to congratulate CRS for their work on the ground. They should work to ensure that municipalities understand their concepts and endorse the project. They can count on our engagement.” –County Mayor “I give my congratulations to CRS for its vision about the project and for all the support they brought us, and encourage CRS to continue in that way. They have as priority the future of kids … whom they will positively impact.” –MENA Representative IMPAQ International, LLC Page 62 Beoog Biiga II Midline Report July 2017 A summary of findings for activities associated with the two primary strategic outcomes—improved literacy and increased use of health and dietary practices—is presented below. Strategic Objective 1: Improved Literacy of School-Age Children All teachers and school administrators who participated in interviews believed that the project is meeting its goals in terms of improved literacy of children. Teachers maintained that the new training techniques have a beneficial effect on student literacy. Because the teaching approach joins reading and writing skills together, respondents reported that it is both efficient and effective: Students learn how to read and write at the same time, whereas, in the traditional practice, the two tasks were taught separately. Interestingly, however, classroom observations and the quantitative results suggest that teachers pay less attention to writing skills and are more focused on reading skills. They may do so because the ASER test focuses on reading abilities, so they are more concerned with improving students’ reading skills than writing abilities. Furthermore, as mentioned in Section 6.1.1, the techniques use a real-life approach, which teachers believe is beneficial for students. Teachers are trained to use common, real-world examples when teaching students how to pronounce and write letters and words. Teachers appreciate that the new training techniques focus on the process of student learning and encourage students to learn by engaging in the lessons through play and practice. Interestingly, the classroom observations showed otherwise; few teachers used real-life examples during lessons. Teacher respondents shared techniques they found to be the most useful in their classrooms; their responses align with the findings of the classroom observations. Many teachers reported using a decoding activity that consists of assisting students to recognize and read letters. Another widely applied technique that teachers found useful was reading group sessions during class. This activity consists of organizing students into small groups and asking them to take turns reading a text. Teachers appreciated this technique because they found that, during these sessions, students with stronger reading abilities mentor students with less-developed skills. Another useful technique from teachers’ perspective is the inference technique, in which they invite students to answer questions after reading a text. This technique allows teachers to assess students’ comprehension and their creative and critical thinking abilities. For example, teachers can ask students to create a title for a body of text or to imagine the rest of the story. Teachers reported using this technique less than the reading group sessions. Other techniques, including substitution, were mentioned less often than the techniques described above. Substitution is an exercise in which students are asked to find appropriate words to replace subjects or other words in a text. A technique that was not mentioned by any of the teachers was systematic screening, which consists of a weekly assessment of students’ reading ability. This technique, in principle, should enable teachers to identify students who need more practice and support and to create a plan for improvement. However, this technique was not applied by the teachers we interviewed. Classroom observations confirmed that these techniques were less commonly used by teachers. “After a reading, I do a dictation, and I do oral expression, asking them to build sentences with words. I usually invite them to copy some words after reading.” –Teacher in Bam Province “I use Bananagrams. I write a word and ask them to copy this word on their slates. After, I ask them to try building other words with these letters using the Bananagrams.” –Teacher in Bam Province IMPAQ International, LLC Page 63 Beoog Biiga II Midline Report July 2017 A sister activity to classroom reading group sessions is the reading clubs designed as part of the BBII project and implemented in targeted schools. In these reading clubs, groups ofstudents meet in the village after school hours to practice reading under the guidance of a volunteer facilitator. Teachers said that the reading clubs enable students with more advanced literacy skills to mentor students with less developed skills. Teachers reported that students who participate in a reading club that is facilitated by someone other than a teacher feel more comfortable participating and asking questions. Teachers also reported challenges with the reading clubs. Teachers explained that a lack of specific kinds of books to facilitate reading groups is a problem and that using the same books in reading clubs and classrooms does not motivate students, since the books teachers received through the project for classroom work were more like textbooks than reading books. Teachers noted that students would be more motivated to join reading clubs if there were different types of books to read, such asfictional novels and short stories. Further, teachers reported that most reading club members are not able to access the libraries due to long distances between villages and the capital town where libraries were established. Another limiting factor of reading clubs is the lack of quality facilitators. Teachers reported difficulty in finding facilitators from the communities because it is a volunteer activity; also, community members are often illiterate. When volunteers do accept the role, they often do not show up for the reading clubs, abandoning the students. Finally, respondents reported that some parents do not allow their children to attend reading clubs as they do not understand the importance of the activity and prefer that their children take care of domestic activities when not in formal school. Teachers said that classroom literacy activities are facilitated by valuable training materials. Teachers reported that they were particularly fond of the large slates and blackboards. Classroom observations confirmed that teachers use these materials more frequently than others. For example, large slates are generally used during reading group sessions, and their usefulness was emphasized by all teacher respondents. In fact, the slates are the most commonly used material, according to teachers. From teachers’ perspective, the large slates facilitate group work much better than the small slates that individual students use. The new large slates can display many words and sentences, and everyone in the group can see what is written. Additionally, teachers say that the larger slates are often a good substitute for chalkboards, as they are portable. Students who are toward the back of large classrooms sometimes struggle to see what the teacher has written on the chalkboard, so they have difficulty in decoding letters and words. In addition to the large slates, teachers noted that they appreciate the reading books that CRS distributes. Reading books contain texts, such as stories, and comprehension activities. Teachers reported that these materials are beneficial because students are able to use them to practice reading individually or in groups and because students are generally interested in using them. Teachers’ responses suggest that tools such as Bananagrams, letter cubes, 42 and dictionaries are less frequently used than the tools mentioned above. According to teachers, these tools require more time to use in the classroom than blackboards and slates. These findings are in line with classroom observations. Teachers reported, however, that lack of teaching resources, including the limited number of large slates and reading books, is a challenge. For example, teachers described classes of 60 students with only six reading books and classes of 120 students with only five large slates. Because teachers lack sufficient supplies, students cannot take full advantage of learning practices. Further, respondents reported that 42 Respondents reported that cubes have not yet been distributed to all schools. IMPAQ International, LLC Page 64 Beoog Biiga II Midline Report July 2017 reading books and other teaching materials, like dictionaries, letter cubes, and Bananagrams sets, are often stored in school administrators’ offices instead of being stored in classrooms where students can have easy access to them. As mentioned before, teachers also discussed the teacher observations conducted by school district administrators. Overall, teachers reported that, through this monitoring, they receive important technical support on teaching techniques and receive feedback on techniques that they have not yet mastered. Teachers also said that observations prompt them to focus on the quality of their teaching and follow the practices and standards they learned from the literacy trainings. Because teachers found the close monitoring of school district administrators to be helpful, some teachers regretted that school district administrators sometimes conduct their visits several months after the start of school year and even, in some cases, near the end of the school year. Further, in contrast to implementers’ suggestion to perform random observations, teachers suggested that they receive notification several days in advance of observation visits, as these visits often disrupt their classroom agendas. Again, it is important to note that school district administrator observations are overseen and funded by MENA; CRS and its partners work with MENA to reinforce these activities. Another constraint to the achievement of literacy objectives reported by respondents is the distance to libraries. School-level respondents stated there are no libraries in their communities. In the rare cases that libraries are present, they are located in the main town, not in the village itself, making it difficult to use these places for learning purposes, especially since some villages are 50 kilometers or more from the main town. Interviews with teachers revealed that libraries are not often visited by teachers or students for educational purposes unless the school is close to the library, which is uncommon. While it is important to note that CRS and FAVL intended for libraries to be located in the capital town and not close to all schools, many teachers reported a desire to have access to libraries and believed libraries would serve as a useful tool to teacher development and in classroom teaching. Conscious of this difficulty, FAVL has initiated mobile libraries and reading camps to increase access to library services. Mobile libraries consist of FAVL staff visiting schools with a package of books and other reading materials for students to browse on location. Reading camps are organized activities during school breaks that are dedicated to reading in a fun and playful way. Reading camps include activities such as competitions to reward outstanding readers. Teachers and FAVL staff reported that students enjoy the reading camps, which incite interest in reading. As one respondent noted, “Everyone wants to be valued in the village as the best reader.” During interviews, teachers and students asked that reading camps be continued. They emphasized that, even if FAVL could not continue to provide participants with breakfast as it has done in the past, students would attend the camps anyway. Furthermore, teachers reported that some students who are usually late for school are the first ones to arrive at reading camps. Teachers and school administrators reported having observed meaningful positive results as a result of the new techniques and materials implemented for this project. Teacher respondents emphasized that the project has improved students’ motivation and interest in reading and writing. Teachers said that decoding techniques have enabled younger students from lower grades to recognize letters of the alphabet and correctly spell basic words. For those in the higher grades, application of the techniques permitted students who previously were not able to read to start becoming more comfortable with reading and writing. From the teachers’ perspectives, students have become more enthusiastic about reading and writing. Students who struggled to read are now motivated to perform as well as the best readers in their class. IMPAQ International, LLC Page 65 Beoog Biiga II Midline Report July 2017 Strategic Objective 2: Increased Use of Health and Dietary Practices In this project, teachers have been taught health, hygiene, and dietary practices to transfer to students. Teacher respondents who attended the trainings reported that the trainings were useful. Some teachers recognized that they did not know how to wash their hands properly prior to the training and appreciated the opportunity to learn the technique and teach it to students. Children reported enjoying the handwashing practice; they liked going to the handwashing station during classroom breaks and before and after eating. Some schools reported that students’ health practices have changed since the start of the project. An example is the school of Foulou in Bam Provence, where respondents expressed that students have adopted handwashing as a reflex. Teachers reported that dietary teachings about vitamins and minerals are taught together with the handwashing practices. Both teachers and students found these teachings to be useful, though it was unclear from the interviews whether and how students apply these lessons to their everyday lives. Regarding the canteens, respondents reported that regular school meals and take-home rations are an important aspect of the project. Respondents described delays in commodities getting to schools; some canteens received foodstuffs late in the school year, limiting their ability to feed students as planned. Respondents reported that these delays were due to delays in government payment to food carriers. CRS addressed the issue, however, and worked with the government to allay future transportation payment delays. Regardless of the delays, students expressed much appreciation for the food they received through the canteens and did not mention delays. They stated that the cooked food is tasty and that they often take some home to share with their siblings and parents. Many types of respondents, including teachers, county mayors, and school administrators, reported limitations of the health and dietary activities that may be preventing the achievement of better results. The first limitation is related to teachers’ motivation to teach handwashing, hygiene, and nutrition knowledge. At each school, two people are trained to educate students about health and dietary practices: the school administrator and one teacher who has been designated as the “cleanliness lead” by the school. Because only two individuals per school are trained, respondents reported, sometimes teachers who were not trained do not want to participate in health and dietary activities. According to some respondents, these teachers are “boycotting” the activities because they did not attend the training and therefore did not receive per diem payments. As long as facilities are available at the school, respondents reported, students apply handwashing techniques and use latrines. But some schools lack water sources and latrines. In the same vein, the handwashing devices may fall into disrepair, and neither the school nor the PTA members replace them. Finally, the socioeconomic status of some parents limits students’ ability to practice positive health and dietary behaviors if they do not have clean water, soap, or appropriate latrines at home. However, respondents reported that students sometimes positively influence their parents’ behavior. For example, some students pass on information about the benefits of washing hands with soap to their parents and request that their parents buy soap and practice with them. Regarding dietary supplements, as mentioned in Section 6.1.4, a handful of interviewed students from one focus group reported that they stopped taking the micronutrients that teachers gave them because the supplements caused nausea or headaches. IMPAQ International, LLC Page 66 Beoog Biiga II Midline Report July 2017 6.2.2 Successes and Challenges of Program Implementation Respondents described successes and challenges in effectively implementing BBII. Below is a summary of their responses. Successes of Program Implementation Respondents emphasized that the involvement of communities in specific project activities is an important and resounding success of the project to date. Mentoring activities were seen as particularly successful by implementers. Mentoring consists of community women supporting girls by providing advice and serving as an advocate for them with their communities and families. Respondents reported that, because of the mentoring program, many girls who became pregnant remained in school. In addition to community involvement in mentoring activities, another success reported by stakeholders is communities’ engagement in SILC activities, in which community members are trained to set up financial cooperative groups. The funds that these groups raise enable them to better plan and pay for school￾related expenses. Project implementers reported that several parents have been able to buy school supplies and materials, such as books or a bag, for their children with money from SILC activities. According to respondents, an important factor that has helped the project reach success related to health and hygiene practices and attitudes is the dual implementation of BBII and another CRS project called Koom Yilma, which supports the provision of water sources and toilets in schools. Respondents including county mayors and MENA staff stated that the BBII schools that also participate in Koom Yilma have seen “significant positive results toward practice and attitudes of students,” as one MENA staff member put it. Challenges of Program Implementation The main challenge of the BBII project to date has been gaining sustained buy-in and engagement of the project’s central actors—the teachers. Respondents reported that, unless teachers fully recognize the benefits of the new teaching techniques, they are unlikely to use them in the classroom and instead will rely on the traditional approach. Some teachers reported that the new techniques are too time consuming in terms of preparation and application and that students are not yet comfortable with these practices. Further, stakeholders noted that some teachers do not consistently prepare for class and because the new teaching techniques require preparation, these teachers do not incorporate the techniques into their daily classroom lessons. Another challenge is related to teacher training. All interviewed teachers and some school district administrators said that the workload during the training is too large but the training is too short. In seven days, many topics are taught: literacy, hygiene, health and nutrition, community capacities, and reading groups, among others. Teachers reported that a week is not enough time to enable them to fully understand all the topics in order to adequately integrate them when they return to their schools. A related problem is the conditions in which training takes place. Some teachers reported that the training classrooms, which are the same classrooms used for the students, are cramped and the seats are too small for adults, making it difficult to concentrate. Additionally, teachers said that the trainings sometimes occur late in the school year; they would prefer to receive training at the beginning of the school year in order to implement the techniques immediately. A challenge reported by some school district administrators is that the concepts and techniques of the teacher training are too advanced for some teachers. Because teachers do not all have the same skills and “The workload for the training is too much. It doesn’t enable teachers to learn.” –School District Administrator IMPAQ International, LLC Page 67 Beoog Biiga II Midline Report July 2017 aptitudes, school district administrators described how trainers sometimes have difficulty ensuring that all teachers understand the topics discussed during the trainings. According to respondents, teachers are not the only group who need additional training and support; though parents receive some guidance, respondents reported that parents need additional training through the PTA and school management committees, as well as SILCs. For example, respondents said that students’ parents experience challenges with the use of performance tools distributed by teachers. The performance tools include canteen tracking files from the PTA management and performance records from teachers. Though parents receive training to read and use these tools to support their children, respondents explained that many parents still do not know how to interpret the tools. Respondents also reported that SILC leaders often struggle with management of group activities and SILC methodology and require further support to run the groups effectively. Further, respondents reported that some teachers have refused to participate in CRS trainings because they believe the per diem payment for training is inadequate to cover all of their expenses. For example, CRS determines whether teachers are “residents” or “non-residents” based on the distance between their homes and the training site. Teachers designated as “residents” do not receive per diem payments for accommodations. It is important to note that CRS’s per diem policy and rates align with the government’s policy and rates and that CRS chose these rates for sustainability purposes. However, according to some respondents, some teachers who are designated as “residents” live too far from the training site to return home each night and, therefore, are not able to attend the trainings due to financial constraints. It is not clear how many teachers have refused to participate in CRS trainings because of financial limitations. Teacher absenteeism may also present challenges to implementation of BBII activities. Although the quantitative data showed high teacher attendance, the quality of the attendance data is uncertain given the sensitive nature of teacher absenteeism at schools. Further investigation is needed to understand the scope of teacher absenteeism, as the field visits did not present a clear picture of the issue. Some school district administrators or school administrators do not keep records of absentees, and respondents reported varying levels of absenteeism. 6.2.3 Internal Collaboration and Management From respondents’ perspectives, collaboration and management of the project among internal stakeholders at CRS has been successful and effective. CRS holds regular meetings to coordinate activities, address issues as they arise, and manage day-to-day operations. Field issues are brought to top-level management through regular monitoring activities conducted by CRS’s Monitoring, Evaluation, Accountability, and Learning division. This division helps project partners to quality-check the work implemented on the ground and make sure the activities are reaching their targets. According to implementers, this mechanism has been a valuable tool to help implement the project. Respondents reported that all CRS staff are actively engaged in project implementation and monitoring and believe that, if staffing continues to remain stable, positive results will be achieved. IMPAQ International, LLC Page 68 Beoog Biiga II Midline Report July 2017 6.2.4 Collaboration with Project Stakeholders Respondents reported that collaboration among project stakeholders, including CRS, FAVL, OCADES, and MENA, has been positive. The main concern project implementers shared is related to overlapping agendas and lack of communication or miscommunication among stakeholders. Some project implementers described instances in which project activities were delayed because a partner was simultaneously rolling out other activities and was therefore unavailable. There is a need for the partners to have a mutual understanding about the agenda of activities to ensure that relevant partners are available to conduct activities. Respondents, including implementers, MENA, and county mayors, believed the program has been monitored effectively and appreciated the project’s participative approach, which involves all partners in monitoring activities. For example, local MENA representatives periodically travel to schools with county mayors to observe project activities. CRS gives county mayors funds for field monitoring, such as cash for gasoline. A few implementer respondents noted that the field reports could be improved by including qualitative data on county mayors’ and local MENA staffers’ perspectives on project progress, successes, and challenges, among other topics. 6.3 Efficiency Through interviews with USDA, MENA, implementers, teachers, and school administrators, the research team assessed the efficiency of the project. Interview topics focused on efficiency in the use of project resources, the timeliness of project activities, and the project’s responses to internal and external factors that may hinder implementation. Below is a summary of the qualitative findings on BBII efficiency. 6.3.1 Efficiency of Use of Project Resources The issue related to project resources that respondents most commonly expressed was delays in project implementation experienced by stakeholders in the field due to slow internal administrative processes. In some cases, field staff had to use their own means to perform activities and get reimbursed later because their host institution was unable to gather the funds in a reasonable time. Delays were particularly likely when the partner organization’s bank was different from CRS’s bank. Respondents reported that these delays were frustrating and hindered their ability to initiate some activities on time. Further, respondents from OCADES and FAVL emphasized that the limited size of their staffs sometimes affected their ability to roll out field activities and effectively monitor them. 6.3.2 Achievement of Project Timeline When asked whether the project has remained on schedule, implementers said that the project has reached its immediate results on schedule: number of trainings performed, number of schools reached, number of SILCs created, and number of libraries provided. Respondents from OCADES reported that they have completed their planned activities on time; in fact, they have initiated more SILCs than originally planned at this point in the project. The same holds for FAVL’s activities; respondents from FAVL reported that they have completed their activities as planned. However, FAVL experienced delays in library activities due to political instability in 2014. “CRS activities are conflicting with our regular tasks/activities. There is a need for better communication of their agenda.” –School Administrator IMPAQ International, LLC Page 69 Beoog Biiga II Midline Report July 2017 As described above, respondents reported that the project has experienced delays in the provision of some project activities and supplies, such as commodities for school canteens. Respondents reported that these delays were due to delays in payment to food carriers. Additionally, training materials and packages (books, letter cubes, Bananagrams sets, and so on) arrived at schools later than scheduled; some teachers did not receive materials until the end of the school year. Furthermore, some teachers did not receive literacy training until several months into the school year. Finally, some respondents reported that some school district administrators were deployed to the field for monitoring visits later in the school year than originally planned. Some teacher trainings by school district administrators were also delayed. 6.3.3 Response to Factors That Could Hinder Implementation Respondents reported that commodity management has been one of the most significant hindrances to efficiency throughout the project but that CRS and its partners have effectively managed the provision of project services and materials given the challenges described above. Respondents stated that CRS’s priority to engage government stakeholders, including MENA, has been beneficial not only to the roll￾out of project activities but also to project sustainability. In addition to the internal factors, respondents shared external factors that could challenge implementation moving forward. First, climate variation threatens food security in the region. From one year to another, populations could experience food shortages that would limit parents’ ability to provide foods in canteens. Second, because the northern region has been subjected to terrorist attacks, some targeted schools may be affected or may be excluded from the project for security reasons. Finally, county municipalities are under a new mandate to increase staff wages. This budget pressure may threatening county municipalities’ ability to take on more responsibility on projects such as BBII. 6.4 Perceived Impact Through interviews with all stakeholders described in Section 2.2, the research team assessed the perceived impact of the project on intended outcomes. Interview topics focused on stakeholders’ perception of the effect of the project on children’s education and health outcomes and the activities with the greatest and least medium- and long-term effects. Stakeholders were also asked to provide recommendations to improve the success of the project. Below is a summary of the qualitative findings on the perceived impact of BBII. Through interviews with all stakeholders described in Section 2.2, the research team assessed the perceived impact of the project on intended outcomes. Interview topics focused on stakeholders’ perception of the effect of the project on children’s education and health outcomes and the activities with the greatest and least medium- and long-term effects. Stakeholders were also asked to provide recommendations to improve the success of the project. Below is a summary of the qualitative findings on the perceived impact of BBII. “I feel CRS has done a really good job of getting the communities involved and having them be supportive of the commodities being in school, regardless of where they’re coming from.” –USDA Representative IMPAQ International, LLC Page 70 Beoog Biiga II Midline Report July 2017 6.4.1 Perceived Impact of Program Respondents of all types expected the new literacy approach to have an impact on student reading abilities if the teacher literacy trainings are sustained and if teachers apply the delivered techniques in their classrooms. Teachers believe the new framework of reading activities has stimulated students’ motivation to learn because being able to read correctly in front of others is perceived by students as a social recognition in the community. Reading group activities break down social barriers between girls and boys, respondents believed, as boys and girls are becoming more comfortable participating in group activities together. Teachers also stated that students are becoming more confident in speaking and reading. Finally, teachers explained that the new teaching methods improve students’ cognitive abilities and generate positive effects in other disciplines including mathematics, geography, and history. In the schools researchers visited, most of the interviewed teachers noticed variation in students’ attendance during the year; at the beginning of the school year, before canteens deliver food, many students miss school. But when canteens begin functioning, teachers said, students come to school more regularly. To respondents, it is obvious that the rise in attendance is due to the presence of canteens. As for hygiene-related outcomes, respondents including MENA staff and county mayors noticed positive changes in students’ hygiene habits during their field visits. In schools where water is available, students have changed their habits, developed positive attitudes toward hygiene practices, and adopted handwashing before and after meals. However, respondents believe there is still room for improvement. 6.4.2 Activities with the Greatest and Least Impact In general, respondents believed that the project has been more successful with literacy activities than with activities associated with hygiene, health, and nutrition. Many respondents, including teachers, school administrators, implementers, and county mayors, stated that it is difficult to determine which activities of the BBII project had the greatest impact, as each aspect of the project is important to reaching key outcomes. Overall, the respondents welcomed the new literacy techniques, recognizing them as one of the activities that is most likely to generate important and positive effects on students’ literacy and comprehension skills. Teachers, school administrators, and implementers believed that, if delivered properly, the educational activities related to hygiene, health, and nutrition will strengthen students’ skills and knowledge. As described in Section 6.2.1, respondents said that limitations such as teachers’ lack of motivation to teach handwashing, hygiene, and nutrition knowledge and the lack of clean water sources and latrines may result in less significant outcomes. All respondents believed that no activities are insignificant; every activity plays a role in the project and contributes to improvements in children’s education and health outcomes. “When I came into the class, only three pupils were able to read. But now, with the new techniques, most of them can read a text.” –Teacher in Bam Provence “We visited a school and were really impressed with the cleanliness. The school environment was so nice and clean and, as a surprise, the community was also a clean place. This is one of the most successful hygiene outcomes I’ve seen.” –MENA Representative IMPAQ International, LLC Page 71 Beoog Biiga II Midline Report July 2017 6.4.3 Respondents’ Recommended Strategies to Increase Impact Implementers, county mayors, teachers, and school administrators recommended ways to increase the impact of BBII activities based on issues met during implementation. Teachers and school administrators emphasized that the duration of the teacher literacy trainings was too short for the amount of information that was presented. A strategy recommended by teachers was to lengthen the literacy trainings, based on pilot testing, to appropriately reflect the amount of time needed to accomplish all training objectives outlined in the agenda. Teachers also suggested operating two separate sessions: one dedicated to literacy training and another dedicated to hygiene, nutrition, and health. Some teachers believed that putting both together is too cumbersome for participants. Also, respondents suggested that the training focus more on practice than lecture. One school district administrator suggested that techniques like “discussion” and working group sessions must be emphasized because many teachers have not mastered the literacy techniques yet. Respondents also suggested that the per diem payments for “resident” teachers be increased to cover all of their expenses. Because involvement of teachers is vital to success of the project, respondents also provided suggestions to improve teacher engagement based on their experience implementing BBII thus far. Respondents recommended that trainings include educating teachers about how their actions influence student success and how the new teaching techniques can yield improvements in the classroom. Teachers also recommended that trainers provide standardized sheets for lesson preparation as part of the training. Furthermore, some respondents suggested that CRS develop a motivational framework that provides incentives for teachers who use the new teaching techniques and disincentives for those who do not. For example, CRS could initiate a competition to reward the best students in reading as well as their teachers. Respondents believed that school district administrators should also be involved in reinforcing integration of the new teaching techniques in the classroom. To do so, administrators would need regular access to means of transportation and other relevant equipment. Finally, respondents suggested that MENA school district administrators promote encouraging teachers to engage in continuous training using frameworks like groupe d’animation pedagogique, in which schools are grouped based on location and teachers practice teaching techniques and share best practices with each other. CRS respondents also provided recommendations about the teacher observation process. In the traditional teacher observation approach, the evaluation of teachers by school district administrators consists of a process evaluation to observe whether teachers are following the recommended techniques or not. Respondents noted that this approach has limitations, because teachers may alter their behavior during visits in order to receive positive feedback. CRS respondents therefore suggested that school district administrators conduct observations at random without announcement. This procedure would require teachers to implement the new techniques consistently, resulting in better student outcomes. Additionally, respondents suggested that administrators should revise the observation methodology to focus on student literacy outcomes and promote real-time hands-on support for teachers. For example, respondents suggested that administrators ask a random sample of students to read a given text to assess their reading skills. This technique would allow administrators to observe patterns and work with teachers to develop an action plan to solve issues and improve student performance over time. To improve observation feedback, teacher respondents suggested that teachers receive a copy of the full feedback sheet that administrators prepare as part of their visit. Currently, teachers receive only a IMPAQ International, LLC Page 72 Beoog Biiga II Midline Report July 2017 summary of recommendations and advice; however, teachers suggested that the full review would help them better understand the feedback and action plan. Because materials and tools are important in facilitating teachers’ instruction, respondents recommended that teachers receive enough materials to support their large classrooms and that they receive them at the start of the school year. For example, teachers should receive a sufficient number of large slates to support small group work, ideally one large slate for every five to seven students. The most important training materials, respondents believed, were large slates and reading books. Respondents suggested that communities get more involved in the provision of training tools. For example, slates could be made with local materials. To increase access to libraries, respondents recommended that the mobile library concept be implemented more broadly, especially in remote areas with little access to permanent libraries. Respondents also suggested that schools and communities get more involved in providing books. Some suggested that, at the beginning of every year, schools could set aside funds to purchase students’ preferred books to grow their internal library. Others suggested that schools initiate book collection campaigns, encouraging older students and community members to bring books they no longer use. 6.5 Sustainability To assess the sustainability of project interventions, the research team asked all stakeholders, except students, to identify major factors that are likely to influence the achievement or non-achievement of project sustainability and to describe current sustainability activities implemented by CRS and its partners. Respondents were also asked to recommend strategies to sustain activities after funding ends. Below is a summary of the findings. 6.5.1 Sustainability Factors Project implementers and the USDA representative reported that one of the most important factors that will determine whether BBII activities will be sustained beyond USDA funding is the communities involved in implementation. Respondents reported that the dedication of communities, and specifically of PTAs, to continuing support of key project activities, such as canteens, mentoring, and SILC, will play a significant role in sustainability. For example, PTA members dedicate their own time to ensure that canteens function properly, serving as volunteer cooks, commodity watchers, and managers. Further, the communities supplement the government’s commodities to supply much of the food the canteens offer. Another important factor to ensure sustainability is engagement of local stakeholders, most notably MENA and local governments. Respondents said that, if local partners do not take responsibility for project activities after USDA funding ends, then the project will not continue. 6.5.2 Current Sustainability Activities Understanding the importance of stakeholder and community involvement, CRS has already begun to coordinate closely with MENA stakeholders and with municipalities, which are involved in all aspects of the project. For example, municipalities have taken over several activities including the provision of library managers’ wages, school materials, and other subventions, which the municipalities receive from the central state. Municipalities also sometimes take over light expenses, such as the expenses of school management committees. IMPAQ International, LLC Page 73 Beoog Biiga II Midline Report July 2017 CRS has also worked closely with local communities to involve them in implementing activities such as mentoring, SILCs, and canteens. The involvement of communities in these activities will be important for sustainability. Project implementers and county mayors stated that communities have fully endorsed and been actively engaged in project activities, including mentoring, which requires extensive community involvement. Furthermore, after OCADES provided training on SILCs, communities have continued to deliver the activities in their villages through prestataires de services privés or private service providers. These groups are village community members who are trained by OCADES to create SILCs. 6.5.3 Respondents’ Recommended Strategies to Sustain Activities Though it is too early to determine whether the project will be sustained after USDA funding ends, all types of respondents suggested that CRS and its partners continue to work closely with government stakeholders and communities on a sustainability plan. Throughout the interviews, respondents commonly referred to the project as “CRS’s project.” For this reason, respondents recommended organizing a workshop in order to emphasize the responsibility of each actor and obtain buy-in and ownership from stakeholders and communities. This workshop could serve as an opportunity to discuss sustainability strategies and how each stakeholder can play a role in sustainability. Respondents suggested that all stakeholders come to the table for this discussion, including community organizations and municipalities. IMPAQ International, LLC Page 74 Beoog Biiga II Midline Report July 2017 SECTION 7. CONCLUSION This report presents the midline results of the performance evaluation of the Beog Biiga II program, highlights changes in outcomes related to BBII core objectives, and informs potential additional questions to ask at endline in the provinces of Bam and Sanmatenga in Burkina Faso. To accurately reflect program performance over time, we followed the same performance evaluation methodology and sampling strategy used during the baseline to generate an appropriate comparison with the midline using USDA guidelines. For this midline performance evaluation report, we employed a variety of data collection methods including: 1) surveys of students, parents, teachers, PTA members, and school district administrators, 2) classroom observations, 3) focus group discussions, and 4) key informant interviews. We collected survey data from 258 primary school students grades two to six, 262 parents, 105 teachers, 22 PTA members, and 30 school district administrators; interview data from 27 stakeholders, including implementers, county mayors, teachers, and school administrators; and focus group data from 35 students grades two to six. We also collected attendance data for all the teachers and students in our sample. This section summarizes key findings in response to the main research questions, highlights study limitations, and provides recommendations for the evaluation and the overall project. 7.1 Key Findings 7.1.1 Students’ Outcomes • The proportion of students who achieved a passing score on a food nutrition test increased by 2 percentage points from baseline to midline (statistically significant at the 5 percent level) but remained low (3 percent). Surveyed data reported by teachers show that teachers are well￾informed on knowledge necessary to increase students’ nutrition awareness. However, low nutrition knowledge of students may suggest that teachers need support on how to teach it effectively. • There was a slight (4 percentage point) increase in the proportion of children who reported washing their hands at critical moments compared to baseline although the change was not statistically significant. The proportion of children who achieved a passing score on a test of good health and hygiene practices remained relatively low with no significant change at midline (4 percent). The lack of significant changes from baseline to midline could be explained by the fact that some schools lack reliable water sources and handwashing devices sometimes fall into disrepair, as reported by school-level interviewees. • At midline, the proportion of children who reported that they were hungry during the school day (after eating breakfast before going to school) remained low (7 percent) with only a slight increase (3 percentage points) from baseline. However, the change was not statistically significant. Although this increase is trivial, it could be explained by fewer children (4 percentage points) eating before going to school (not statistically significant at any level) at midline. Students most commonly cited insufficient food (81 percent) as the reason why they could have eaten more after breakfast before going to school at midline. • The percentage of children who consumed a minimum acceptable diet increased significantly from baseline to midline. Students reported an 11 percentage point increase, and parents reported a 20 percentage point increase (statistically significant at the 1 percent level). The increase in consumption of a minimum acceptable diet was slightly higher for boys than for girls. IMPAQ International, LLC Page 75 Beoog Biiga II Midline Report July 2017 This increase in the diversity of household’s diet might be affected by potential different socio￾economic backgrounds between baseline and midline samples or seasonality. • Almost all students attended school regularly (at least 80 percent of the time). However, there was a slight decrease in children’s regular attendance from baseline to midline (by 1 percentage point); the result was marginally statistically significant (Statistically significant at the 10 percent level) at conventional levels. Boys’ attendance decreased by 2 percentage points at midline, a slightly greater decrease than in girls’ attendance. The fact that student attendance data were collected from teachers who might not record attendance consistently could confound measurement of the attendance rate. • The proportion of second-graders with reading proficiency at grade level increased by 9 percentage points from baseline to midline. The improvement in reading proficiency was higher for girls (15 percentage points) than for boys (4 percentage points). However, the increase was not statistically significant at conventional levels for either group. 7.1.2 Parents and Parent-Teacher Associations’ (PTA) Outcomes • At midline, parents’ participation in community savings groups established as part of the BBII program increased by 32 percentage points compared to baseline (statistically significant at the 1 percent level). Although more parents reported at midline that they used their savings to cover school expenses, fewer parents said that the savings helped “a lot” with school expenses, with a significant 33 percentage point decrease compared to baseline. On the other hand, the proportion of those who said that the savings helped “some” increased by 38 percentage points. Both findings are statistically significant at the 5 percent level. A possibility of increase in education expenses or a decrease in parents’ purchasing power to afford school costs might be potential explanations to this shift in parents’ responses. • The total number of months in which the canteen was covered by MENA, CRS, parents, and other responsible parties decreased by 2 months. This finding is statistically significant at the 10 percent level at midline. This outcome should be interpreted with caution, because there is a possibility that canteen operations were misrepresented at baseline. 7.1.3 Teachers’ Outcomes • The proportion of teachers who attended at least 90 percent of school days did not change significantly from baseline to midline, increasing only slightly by 3 percentage points. • Teachers in target schools employed new teaching techniques at a lower rate: 44 percent at midline compared to 53 percent at baseline. This slight, statistically insignificant decrease should be interpreted with caution, because the training component of the project had not started at baseline. More than half of the surveyed teachers at baseline reporting using new techniques may suggest that teachers were already aware of these techniques even in the absence of the training. It is also important to note that teachers at midline were less educated compared to teachers at baseline. As discussed in Section 6.2, teachers with limited education found the training materials more difficult to understand, which may have led them to continue using traditional approaches— approaches they were more accustomed to using. • A larger proportion of teachers at midline than at baseline reported that they devoted at least 45 minutes a day to literacy instruction, specifically reading instruction (including, for example, phonological awareness, phonetics, and vocabulary). However, this increase (11 percentage points) was not statistically significant at any conventional levels. IMPAQ International, LLC Page 76 Beoog Biiga II Midline Report July 2017 • Teachers identified a slight increase (3 percentage points) in the proportion of students who were attentive during class from baseline to midline. This change was not statistically significant at conventional levels. However, given the difficulty and subjectivity of measuring attentiveness, this outcome should be interpreted with caution. 7.1.4 Qualitative Findings The qualitative study revealed that stakeholders are generally satisfied with the project and believe the project is having a positive impact on beneficiaries. While the quantitative results show mixed results, teachers reported that they have observed improvements in students’ literacy abilities and students’ motivation to learn these skills. Teachers and students also reported that hygiene education has changed students’ behavior as they are now washing their hands more frequently. As discussed, the quantitative data, however, did not show improvements in handwashing. Students appreciated the school meals and take home rations received from the canteens, and teachers believed school attendance was closely tied to the presence of canteens. Though challenges are present, respondents described that CRS has effectively managed partners and the program has coordinated and collaborated effectively with other stakeholders. According to respondents, the primary strengths of the project are its holistic approach to child development, its focus on evidence-based literacy techniques and provision of teaching resources, and its focus on community and government engagement. The primary weaknesses are lack of motivation by some key actors, lack of key equipment and resources for teacher monitoring and support, and the design of reading clubs. While it is too early to determine if the project will be sustained after USDA funding ends, all types of respondents suggested that CRS and its partners continue to work closely with government stakeholders and communities on a sustainability plan. 7.2 Limitations There are some limitations of the study that are worth noting: • An important limitation of the study is that it relied on self-reported data for a number of socially and culturally sensitive subjects, such as food consumption, hygiene practices, and student attendance. This self-reported data should be interpreted with caution and is particularly susceptible to social desirability bias. To help counter biases related to hygiene practices, we integrated observational data of children’s handwashing practices (at critical moments) at the school. However, this method has its limitations as many children did not eat or use the latrine during our school visit. • As described in Section 5, at each data collection point (baseline and midline), we selected new samples of individuals to survey. A limitation of this approach is that there might be systematic differences in the two sets of samples that could confound the observed changes in the outcomes of interest over time. Inspection of the data indicate that the baseline and endline samples were similar along many demographic covariates, except for household size, and accessibility to basic services, parents’ and teachers’ educational attainment, and literacy training received by teachers. • Another limitation is related to the small sample sizes for some indicators, in particular those related to the PTA leaders and survey administrators outcomes. It is important to note that the results presented in those sections rely on very small sample sizes which reduce the accuracy of IMPAQ International, LLC Page 77 Beoog Biiga II Midline Report July 2017 the indicators measured and the magnitude and direction of observed changes over time needs to be interpreted with caution. • Lastly, another limitation arises from the relatively small number of classroom observation due to budget limitations. Hence, these observations should be interpreted with caution without making a strong connection between students’ outcomes and instructional practices. In addition, we had some difficulties getting information from young students (grades 1-3) during focus group discussions as they were shy and hesitant to respond to many of our questions. 7.3 Recommendations We present the following recommendations to CRS based on both lessons learned from our experience in the field and our findings after analyzing the collected data. The recommendations are not intended to address all challenges identified through the study; instead they focus on the main drivers of project success. The recommendations are grouped by category. 7.3.1 Recommendations for the Project Project Management and Sustainability • Reevaluate how project activities are planned and communicated to ensure that relevant partners are available to conduct activities. Some project implementers noted that there have been instances in which project activities have been delayed because a partner is simultaneously rolling out other activities and is therefore unavailable. • Reevaluate staffing plan to ensure enough capacity to implement and monitor projects effectively and efficiently. Some implementers reported that the limited size of their staff sometimes impacted their ability to roll out field activities and effectively monitor their activities. When developing activity roll-out and monitoring plans, implementers should take into account the realities of staffing and resource limitations. • Provide additional training to parents on use of student performance tools and to SILC groups on management. Respondents described that students’ parents experience challenges with the use of performance tools distributed by teachersthrough a pilot activity at five of the BBII schools. Though these parents received training to read and use these tools to support their children, respondents explained that many of these parents still do not know how to interpret the tools. If CRS plans to sustain and/or expand this activity to other schools, they should consider enhancing the training to ensure that parents receive the support they need to effectively interpret and use the tools. Respondents also reported that SILC groups often struggle with management of group activities and require further support to run the groups effectively. Implementers should reevaluate their community sensitization and support structure to ensure that stakeholders are receiving the support and guidance they need to effectively initiate and manage project activities. • Reevaluate the sustainability strategy and develop a robust plan to engage all stakeholders. Respondents believed that the engagement of communities and the government are key factors to sustainability and that CRS and its partners have successfully engaged these parties. However, there is still work to be done to ensure that project activities can be sustained after USDA funding. CRS should continue engaging stakeholders and organize additional workshops with all relevant players to develop sustainability strategies and define how all parties can play a role in sustainability. IMPAQ International, LLC Page 78 Beoog Biiga II Midline Report July 2017 Literacy Activities • Explore methods to increase teacher motivation and engagement. Interview respondents reported that teacher motivation is lacking in many schools. In order to motivate teachers to implement new teaching techniques, stakeholders could consider developing a framework in which teachers are incentivized for using the new techniques and tools in their classrooms. For example, during observation visits, school district administrators should continue evaluating teachers’ use of the techniques using a standardized evaluation process and assessing the level of students’ literacy skills using a standardized test. Teachers who excel could be offered awards for their performance. Further, some respondents reported that some teachers have refused to participate in CRS trainings because they believe the per diem payment for training is inadequate to cover all of their expenses. Implementers could consider distributing a short survey to obtain feedback directly from teachers about factors that motivate or dissuade teachers from participating in trainings, and use this feedback to modify practices and policies. • Consult with teachers to reevaluate length, timing, and content of teacher training. Interview respondents reported that trainings are too dense, causing teachers to be overwhelmed by the amount of information. Implementers may want to consider pilot testing a different training approach, in which teachers participate in two or three mini-training sessions that focus on different topics or techniques so as not to burden teachers with information overload. Teachers desired more time to practice the literacy techniques with their peers during trainings, and felt the trainings were lecture heavy. Some teachers also reported that they received literacy training late in the school year, giving them little time to apply the new approaches in their classrooms. Further, interview respondents noted that some teachers have a difficult time understanding the training materials and therefore are not able to successfully implement the new techniques and tools in their classrooms. Implementers should obtain teachers’ input to reevaluate the length, timing, and content of teacher training to ensure that the trainings meet teachers’ needs. • Collaborate with MENA to ensure that teacher observations are fully meeting their objectives. For example, more real-time, hands-on support to teachers is needed through the use of more frequent student testing and summarized feedback. Teachers reported that they appreciated the feedback they received from administrators during observations but desired more detailed, data￾driven feedback. Implementers could consider implementing the ASER test more frequently and use the data to inform action plans with teachers that focus on ways to use the new teaching techniques and tools and support to teachers who face challenges. Implementers could consider having school-level administrators play a larger role in supporting teachers on a day-to-day basis. Further, the quantitative findings showed that teachers’ use of new techniques decreased by 9 percentage points over time, though the finding was not statistically significant. Implementers may want to use the observations as an opportunity to understand why teachers are not using the techniques, and how trainings and observations can be strengthened to better support teachers. Additionally, observation visits must occur early in the school year, as opposed to toward the end of the year, to give teachers adequate time to adjust based on the received feedback. • Reassess material distribution plan and further investigate why teachers are reluctant to use some tools. Teachers reported that they often did not have enough supplies to effectively use them in their classrooms. Implementers should ensure that teachers have more adequate quantities of materials to properly implement techniques. For example, teachers should receive one slate for every 5-7 students. Further, none of the teachers in classroom observations used Bananagrams, and word strips, and a very few of them used wood cubes, or phonemic tables as part of their lessons. In interviews, teachers reported only using large slates and blackboards. During trainings and observations by school district administrators, implementers may want to IMPAQ International, LLC Page 79 Beoog Biiga II Midline Report July 2017 explore if and why teachers may be reluctant to use some tools and how trainings and observations can be strengthened to support teachers’ use of these materials. • Investigate why teachers’ use of new teaching techniques has decreased over time. At midline, teachers were less likely than at baseline to apply or plan to apply at least five of the seven teaching techniques on the day of data collection. Although the difference is not statistically significant, CRS might want to investigate why this may be the case and adjust trainings and teacher support accordingly. • Reconsider design of reading clubs with a greater focus on local context. Interview respondents described several challenges with the reading club design, including the difficulty to find qualified volunteers to facilitate groups and the fact that many parents require that their children help with domestic activities rather than attend reading club meetings. Implementers should reassess how the reading club component can be better tailored to the culture and stakeholders’ priorities. • Exhort school administrators to keep consistent records of teacher attendance and monitor data collection. Although the quantitative data showed high teacher attendance, the quality of the attendance data is uncertain given the sensitive nature of teacher absenteeism at schools. Some school district administrators or school administrators do not keep a record of absentees, and respondents reported varying levels of absenteeism. Given the uncertainty of the attendance data, implementers should work with schools to develop more robust tracking systems and conduct spot checks to confirm that schools are collecting the data. Health and Dietary Activities • Work with schools to ensure that schools’ hand washing stations are in working order. Some respondents noted that hand washing stations are sometimes out of use and there is no effort made from the school nor the PTA to replace them. Implementers could develop a system whereby a designated teacher or the school-level administrator is responsible for managing the hand washing stations and reporting to CRS and its partners when it is out of order. • Further investigate why students’ hygiene and nutrition knowledge remains poor despite training. Very few children achieved a passing score on the test of hygiene knowledge and students’ level of nutrition knowledge remains very low. This area requires further probing by CRS to explore why student knowledge of good health and hygiene practices and nutrition are so low despite the training. A deeper analysis of why knowledge is so low would inform improvements to the program’s hygiene and health interventions, ideally improving students’ knowledge over time. Investigate health issues related to dietary supplements. A few interviewed students reported that they stopped taking the micronutrients that teachers gave them because it caused them to experience nausea and/or headaches. Because the intensity of this issue is unknown, implementers should consult with students and parents to better understand why students are reacting this way to the dietary supplements and adjust as needed. 7.3.2 Recommendations for the Evaluation • The enrollment list at participating schools should be updated on a more regular basis. After the randomization was complete and before midline data collection, we found some schools with unavailable students on the school lists due to relocations or because they dropped out of school. Collecting and quality-checking monitoring data more frequently (monthly or quarterly) would ensure that school lists are up to date. • Further research should be conducted to explore the reasons why some new teaching techniques, more specifically BBII writing techniques, are not being implemented by teachers. IMPAQ International, LLC Page 80 Beoog Biiga II Midline Report July 2017 Unlike what teachers mentioned in the qualitative section, we did not observed any teachers in any grades using BBII writing techniques, or reporting using them in the survey. Improvement in literacy outcomes should be measured by progress in both skills. It is worth further investigation through qualitative study at endline to explore why teachers are not applying writing skills as intently as reading techniques: whether they are not comfortable implementing them; training was not enough for them; and/or readings skills were the only component that been emphasized by the trainers. In sampling lessons to observe at endline, we will also be mindful to purposely observe writing lessons. Though we selected lessons to observe at midline at random, it could be that we coincidentally observed reading lessons only. • Observations of health-related activities should be considered. Since we experienced difficulty obtaining useful, detailed information from young students (grades 1-3) during focus groups, our observation of handwashing practices needs improvement, and we observed a drop in hygiene￾related practices among children and no change in knowledge of good health and hygiene practices, we may want to consider focusing our effort and resources on conducting additional observations of health-related activities, such as handwashing, canteens, and vitamin distribution, at select schools during the endline. Shifting our focus would allow us to obtain richer data on health-focused interventions that we can triangulate with survey, interview, and focus group data. 7.3.3 Recommendations for the Evaluation with Budget Implications • The scope of work and budget should be amended to include an additional subtest in the ASER reading assessment. BBII teaching techniques include both reading and writing skills together, and in order to assess children’s literacy outcomes we should consider both components together. Adding a subtest to ASER can provide additional data on writing while providing the same comparable data for the reading component. • Larger samples for classroom observations should be considered across treatment and control schools at the endline. We should enhance the size of classroom observations, to a) get a better understanding of how the new techniques are being implemented, b) study the correlation between teacher characteristics, instructional practices, and student’s literacy outcomes, and c) to be able to draw better conclusions from the data. If added to the scope of work and budget, we could study the connections between the qualitative findings, quantitative outcomes, and our observations in a more rigorous way. • Interviews with additional stakeholder groups should be considered. To broaden our understanding of the project’s impact on all BBII stakeholders, we may want to consider conducting interviews with stakeholders that we did not speak with during the midline, such as SILC members and PTAs. Quantitative data showed no changes in PTA activities including holding meetings or supporting canteens, and in some instances PTA support decreased over time. Obtaining data from these stakeholders will allow the team to provide a more comprehensive picture of intervention implementation and the perceptions, opinions, and experiences of all types of stakeholders. IMPAQ International, LLC Page 81 Beoog Biiga II Midline Report July 2017 REFERENCE LIST Abadzi, H. (2011). Reading fluency measurements in EFA FTI partner countries: Outcomes and improvement prospects (GPE Working Paper Series on Learning, No. 1). Washington, DC: World Bank. Catholic Relief Services. (2015). IMPAQ International, LLC Consulting Agreement and Statement of Work. Ouagadougou, Burkina Faso. Food and Agriculture Organization (2010). Guidelines for Measuring Household and Individual Dietary Diversity. Rome, Italy: United Nations. Montgomery, S. M., & Groat, L. N. (1998). Student learning styles and their implication for teaching (Vol. 10). Centre for Research on Learning and Teaching, University of Michigan. Pratham. (2015). Pratham Around the World. Retrieved from http://www.prathamusa.org/programs/pratham-around-world. Simons, K.A. (2012). Discussion Document #1: Multi-Country Assessments of Learning. Washington, DC: Center for Universal Education at the Bookings Institution. United States Department of Agriculture. (2014). Food for progress and McGovern-Dole indicators and definitions. Washington DC: Food Assistance Division, Office of Capacity Building, Foreign Agricultural Service. United States Department of Health and Human Services (USDHHS). Human Subject Regulations Decision Charts. Retrieved from http://www.hhs.gov/ohrp/policy/checklists/decisioncharts.html (2016, February 16). USAID. (2012). 2011 USAID education strategy reference materials. Washington, DC. Vagh, S.B. (2009). Evaluating the reliability and validity of the ASER testing tools. New Delhi: ASER Centre. Vanmeenen, G. (2010). Savings and Internal Lending Communities (SILC) - Voices from Africa. Nairobi: Catholic Relief Services. IMPAQ International, LLC Page 82 Beoog Biiga II Baseline Report February 2016 APPENDICES A. BBII Key Performance Indicators B. Additional detailed data breakdown C. Additional Tables and Complementary Outcomes D. ASER-Reading Test Instructions E. ASER-Reading Test Instructions F. Questionnaires G. Performance Evaluation Schools IMPAQ International, LLC Page 83 Beoog Biiga II Baseline Report February 2016 APPENDIX A. BBII KEY PERFORMANCE INDICATORS Strategic Objectives and Results Key Questions Data Collection Method Data Source SO1. Improved Literacy of School-Age Children What percentage of students (boys and girls) have increased their reading comprehension compared to baseline? What factors contributed to this? What factors inhibited this? Evaluation Student literacy assessment (ASER￾reading test) & Teacher interviews. IR1.1. Improved Quality of Literacy Instruction ▪ Have teachers’ literacy instruction trainings been completed as per the project timeline and budget? ▪ To what extent are teachers implementing literacy teaching techniques acquired through the project? ▪ Are changes in classrooms starting to be evident (e.g they see improved reading abilities of students)? If not, why not? What needs to change? ▪ How much time per day do teachers devote to literacy instruction? ▪ What challenges do teachers face in using the new literacy teaching techniques? ▪ What aspects do they find most useful and why? ▪ In what way has the quality of education improved as a result of the adoption of technical trainings for teachers? ▪ Have community library targets been met? How many exist? Where they are in place, to what degree do community members (including students and teacher) have access? CRS/ Monitoring & Evaluation CRS program data (training reports), Teacher surveys & Teacher key informant interviews. Classroom observations. Observations of libraries (as feasible) & FAVL staff interviews. Output. 1.1.1. More Consistent Teacher Attendance To what extent has teachers’ attendance improved in schools compared to the baseline? If so, why? What are the greatest inhibiting factors to teacher’s attendance? What project interventions influenced the improvement of teacher attendance? Evaluation Key informant interviews with school administrators & Teacher interviews. IMPAQ International, LLC Page 84 Beoog Biiga II Baseline Report February 2016 Strategic Objectives and Results Key Questions Data Collection Method Data Source Output. 1.1.2. Better Access to School Supplies & Materials ▪ To what extent have school supplies and materials been distributed as per the project timeline and budget? ▪ What materials have been supplied? ▪ Which school supplies do teachers find most useful and why? Which supplies and materials provided do students like and why? ▪ What other supplies would teachers and students prefer? ▪ Are material kits being used as intended? ▪ Do teachers/students need additional training to better use these materials? ▪ What has been the contribution to the education process of these materials? CRS/ Monitoring & Evaluation Distribution reports (CRS program data). Classroom observations. Teacher surveys, Teacher key informant interviews, & Student interviews. Output. 1.1.3. Improved Literacy Instructional Materials ▪ To what extent have literacy instructional materials been distributed as per the project timeline and budget? Do the teachers consider these materials to be an improvement over what they previously had? How? How are teachers using the materials provided? What, if any other materials would they prefer? What do students like and dislike about using the literacy materials that have been provided? ▪ To what extent have reading groups been established? What are the characteristics of a successful reading group? Are the reading groups segregated by gender, grade level, etc.? Who provides oversight of the reading group most often (teachers or older students)? Why do/ do not students join reading groups? Have students or teachers noticed any change since students have joined reading groups? (Personal confidence, better grades, more social cohesion among students, etc.) CRS/Monitoring & Evaluation Distribution reports (CRS program data). Classroom observations. Teacher key informant interviews & Student interviews. Output. 1.1.4. Increased Skills and Knowledge of Teachers To what extent have teachers been trained as per the project timeline and budget? What percent of teachers demonstrate use of new teaching techniques and knowledge? In what ways has the quality of teaching improved based on the tools and techniques used by teachers? What aspects of the trainings were not widely adopted and why? How can the trainings have greater impact? What CRS/Monitoring & Evaluation Training reports (CRS program data). IMPAQ International, LLC Page 85 Beoog Biiga II Baseline Report February 2016 Strategic Objectives and Results Key Questions Data Collection Method Data Source additional training topics would help the teachers be even more effective in literacy instruction? Teacher survey & Key informant interviews. Classroom observations. Output. 1.1.5. Increased Skills and Knowledge of School Administrators To what extent have school administrators been trained as per the project timeline and budget? What percent of school administrators demonstrate use of new techniques or tools? To what extent do school administrators find the classroom observation technique useful? How is the technique received by teachers? Have the observations led to constructive feedback? If so, has the feedback been received by teachers and affected their teaching techniques? In what way has the quality of education improved based on techniques used by the administrators? CRS/Monitoring & Evaluation CRS program data (training reports). School District Administrators interviews & Teacher interviews. IR1.2. Improved Attentiveness Output 1.2.1. Reduced Short￾Term Hunger To what extent have school meals been distributed as per the project’s budget and timeline? What percent of students in target schools indicate that they are hungry during the school days? How do students appreciate the ration size? What percent of students eat the entire school lunch at school and what percent bring a portion home for family members? Evaluation Distribution reports, student surveys & student interviews Output 1.2.1.1/1.3.1.1 Increased Access to Food (School Feeding) To what extent have Take Home Rations (THR) been distributed as per the project’s timeline and budget? How do students like the commodities provided for school meals? What percent of school level warehouses demonstrate appropriate storage of commodities? CRS/Monitoring & Evaluation Distribution reports (CRS program data). Student interviews & warehouse observations (as feasible). IMPAQ International, LLC Page 86 Beoog Biiga II Baseline Report February 2016 Strategic Objectives and Results Key Questions Data Collection Method Data Source IR1.3. Improved Student Attendance What is the current student attendance rate? Has the percent of students (girls and boys) that regularly (80%) attend schools changed compared to the baseline? If so, what are the greatest pull factors for students? Evaluation Student attendance logs & Student interviews. Output. 1.3.1 Increased Economic & Cultural Incentives (Or Decreased Disincentives) To what extent has the girls mentoring activity been implemented as per the project’s timeline and budget? Have the mentors been recruited and trained as per the project’s timeline and budget? Have the mentors conducted regular mentoring activities as planned? Have the SILC activities been implemented as per the project’s timeline and budget? CRS/Monitoring & Evaluation CRS program data (training reports, activity reports). OCADES key informant interviews. Output 1.3.4. Increased Student Enrollment To what extent has the enrollment of school‐age students (girls and boys) changed compared to the baseline? If so, how? Which factors have facilitated or have been obstacles towards enrollment? Evaluation Student registers & Teacher/school key informant interview. Output. 1.3.5. Increased Community Understanding of Benefits of Education To what degree have the awareness raising activities on the importance of education been completed as planned? Has the parents’ knowledge of the importance of education changed compared to baseline? Has the parents’ level of contribution to the school canteen changed? CRS/Monitoring & Evaluation CRS program data Parent survey. SO2. Increased Use of Health and Dietary Practices To what extent have the use of standard health practices and dietary diversity among students increased from baseline? Evaluation Student survey & Parent survey. IMPAQ International, LLC Page 87 Beoog Biiga II Baseline Report February 2016 Strategic Objectives and Results Key Questions Data Collection Method Data Source IR2.1 Improved Knowledge of Health and Hygiene Practices What percent of students have increased their knowledge of health and hygiene practices compared to baseline? To what extent have students improved their hygiene-related practices (what percent of school children wash their hands at critical moments)? To what extent has the project supplied hand washing stations to schools as planned? CRS/Monitoring & Evaluation CRS program data (distribution reports) Student surveys and observations (as feasible). IR 2.2 Increased Knowledge of Safe Food Prep and Storage Practices To what extent has the project completed trainings for food preparers as planned? Evaluation Training reports. IR 2.3 Increased Knowledge of Nutrition What percent of students have increased their knowledge of nutrition and dietary practices compared to baseline? What percent of children receiving a minimum acceptable diet has changed, if any compared to baseline? To what extent have PTA, COGES members, and food preparers been trained in good nutrition and dietary practices as planned? Evaluation Student survey & Parent survey. PTA/COGES/Food preparers training reports (including post-test results). IR2.5 Increased Access to Preventative Health Interventions To what extent has the project distributed micronutrients to students as planned? To what degree has students’ knowledge of Vitamin A, Iron and deworming medication changed since baseline? Evaluation Sample of student distribution records (school level), student surveys, CRS program data/distribution reports. IMPAQ International, LLC Page 88 Beoog Biiga II Baseline Report February 2016 Strategic Objectives and Results Key Questions Data Collection Method Data Source IR2.6 Increased Access to Requisite Food Prep and Storage Tools and Equipment To what extent has the project distributed food preparation and storage supplies as planned to preschools? Evaluation Distribution reports. FR 1: Output 1.4.1. & 2.7.1 Increased Capacity of Government institutions To what extent have local government officials been trained as planned? Evaluation Training reports. FR 2: Output 1.4.2.& 2.7.2 Improved Policy or Regulatory Framework To what extent have committees been formed and started drafting policy improvements in education related policies? Evaluation Committee member interview. FR 4: Output1.4.4 & 2.7.4 Increased Engagement of Local Organizations and Community Groups To what extent are PTA’s holding regular meetings? To what degree is this frequency different from baseline? Evaluation PTA survey. Source: Terms of Reference (TOR) IMPAQ International, LLC Page 89 Beoog Biiga II Baseline Report February 2016 APPENDIX B. BBII KEY PERFORMANCE INDICATORS Outcome Variable Mean Observations Indicator 1: Percent of students, who by the end of two grades of primary schooling, demonstrate that they can read and understand the meaning of grade level text Average reading level of second graders 3.1443 87 Proportion of boys that can read and understand the meaning of grade level text 0.16 43 Proportion of girls that can read and understand the meaning of grade level text 0.32 44 Indicator 2: Number of teachers who devote at least an average of 45 minutes a day to literacy instruction Average number of minutes teachers spent on phonemics 25.14 22 Average number of minutes teachers spent on phonics 19.20 20 Average number of minutes teachers spent on fluency 27.61 67 Average number of minutes teachers spent on vocabulary 16.46 24 Average number of minutes teachers spent on comprehension 19.75 60 Average number of minutes teachers spent on identification of key words in a sentence 19.79 28 Average number of minutes teachers spent on identification of key sounds 22.78 9 Average number of minutes teachers spent on ability to repeat words, sounds and syllables 18.35 40 Average number of minutes teachers spent on reading punctuation 19.69 26 Proportion of that teachers didn't know more than one techniques 0.61 105 Average number of minutes students read by themselves today 9 65 Average number of minutes students usually read by themselves during class 11.16 38 Average number of minutes students read to others today 11.16 38 Average number of minutes students usually read to others during class 16.75 89 Average number of minutes students listened to others read today 15.66 68 Average number of minutes students usually listen to others read during class 16.11 90 Average number of minutes students wrote today 14.20 66 Average number of minutes students usually write during class 21.99 101 Average number of minutes students did word play today 15.34 50 Average number of minutes students usually do word play during class 14.67 91 Indicator 3: Number of teachers in target schools who attend and teach school at least 90% of scheduled school days per year Proportion of teachers in target schools who attend and teach school at least 90% of scheduled school days per year .72 49 43 This value corresponds to between reading levels B (level 2) and C (level 3). IMPAQ International, LLC Page 90 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Indicator 4: Number of school administrators in target schools who demonstrate use of new techniques or tools as a result of USDA assistance Proportion of school administrators in targeted districts who demonstrate use of new techniques or tools as a result of USDA assistance .61 13 Indicator 5: Number of preschool teachers (bissongo caregivers) in target preschools who demonstrate use of at least one new teaching technique, skill or tool Proportion of preschool teachers (bissongo caregivers) in target preschools who demonstrate use of at least one new teaching technique, skill, or tool44 N/A N/A Indicator 6: Number of teachers/educators/teaching assistants in target schools who demonstrate use of new and quality teaching techniques or tools as a result of USDA assistance Proportion of teachers who have received literacy training in the last 12 months 0.95 105 Proportion of teachers who have had students check their own work this week 0.35 105 Proportion of teachers who are planning to have students check their own work this week 0.38 105 Proportion of teachers who have not and are not planning to have students check their own work this week 0.27 105 Proportion of teachers who have had students check each other’s work this week 0.44 105 Proportion of teachers who are planning to have students check each other’s work this week 0.43 105 Proportion of teachers who have not and are not planning to have students check each other’s work this week 0.13 105 Proportion of teachers who have had the whole class check the work of a student this week 0.48 105 Proportion of teachers who are planning to have the whole class check the work of a student this week 0.38 105 Proportion of teachers who have not and are not planning to have the whole class check the work of a student this week 0.14 105 Proportion of teachers who have checked students' work this week 0.49 105 Proportion of teachers who are planning to check students' work this week 0.33 105 Proportion of teachers who have not and are not planning to check students' work this week 0.18 105 Proportion of teachers who have paired students of different skill levels to work together this week 0.49 105 Proportion of teachers who are planning to pair students of different skill levels to work together this week 0.35 105 Proportion of teachers who have not and are not planning to pair students of different skill levels to work together this week 0.16 105 Proportion of teachers who have paired students of the same skill levels to work together this week 0.10 105 Proportion of teachers who are planning to pair students of the same skill levels to work together this week 0.15 105 Proportion of teachers who have not and are not planning to pair students of the same skill levels to work together this week 0.75 105 44 Will be collected with pre and post testing at the time of training provided by BBII IMPAQ International, LLC Page 91 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Proportion of teachers who have had students work together in groups this week 0.30 105 Proportion of teachers who are planning to have students work together in groups this week 0.41 105 Proportion of teachers who have not and are not planning have students work together in groups this week 0.30 105 Proportion of teachers using the phonemics technique 0.21 105 Proportion of teachers using the phonics technique 0.19 105 Proportion of teachers using the fluency technique 0.64 105 Proportion of teachers using the vocabulary technique 0.23 105 Proportion of teachers using the comprehension technique 0.57 105 Proportion of teachers using the identification of key words in a sentence technique 0.27 105 Proportion of teachers using the identification of key sounds technique 0.08 105 Proportion of teachers using the ability to repeat words, sounds and syllables technique 0.38 105 Proportion of teachers using the reading punctuation technique 0.25 105 Indicator 7: Number of students in target schools who are identified as attentive during class/instruction Average students' attentiveness on a scale of 1 (the lowest) to 10 (the highest), as reported by teachers 6.18 105 Proportion of students identified as 7 or above on the attentiveness scale, as reported by teachers 0.47 105 Indicator 8: Number of students in target schools who indicate that they are hungry or very hungry during the school day Proportion of children that felt full after the meal s/he ate before going to school 0.93 223 Proportion of students that could have eaten more before going to school 0.07 223 Proportion of students that cite lack of food as the reason for not having a meal before going to school 0.81 16 Proportion of students that mentioned there was no food that they liked before school 0.12 16 Proportion of students indicating another reason why they did not eat more before school 0.6 16 Proportion of children that felt full after eating lunch at home 0.94 150 Proportion of students that could have eaten more for lunch at home 0.06 150 Proportion of students that cite lack of food as the reason why they did not eat a meal for lunch at home 0.78 9 Proportion of students that mentioned there was no food that they liked for lunch at home 0.11 9 Proportion of students indicating another reason why they did not eat more for lunch at home 0.11 9 Proportion of children that felt full after eating lunch at the school canteen 0.88 207 Proportion of students that could have eaten more for lunch at the school canteen 0.12 207 Proportion of students that cite lack of food as the reason why they did not eat a meal for lunch at the school canteen 0.8 25 Proportion of students that mentioned there was no food that they liked for lunch at the school canteen 0.04 25 Proportion of students indicating another reason why they did not eat more for lunch at the school canteen 0.16 25 Proportion of children that felt full before going to bed 0.98 246 Proportion of children that could have eaten more before going to bed 0.02 246 IMPAQ International, LLC Page 92 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Proportion of children that did not eat a meal before going to bed for the lack of food 0.67 6 Proportion of children that mentioned there was no food they liked before going to bed 0.33 6 Proportion of students indicating another reason why they did not eat more before going to bed 0.00 6 Proportion of parents that indicated 1 or more of their children ate less because there was no food 0.34 355 Proportion of parents that indicated their children did not eat because there was no food 0.21 355 Proportion of parents that indicated 1 or more of their children went a full day without food 0.08 355 Indicator 9: Number of students regularly (80%) attending USDA supported classrooms/schools Proportion of students attending regularly (80%) 0.99 559 Proportion of boys attending regularly (80%) 1 238 Proportion of girls attending regularly (80%) 0.99 229 Indicator 10: Number of SILC group members that spend money from SILC on education costs Proportion of parents that are members of SILC 0.09 240 Proportion of parents that have used a portion of their savings accumulated in SILC for school expenses 0.50 24 Indicator 11: Number of girl students who cite mentors as one of the top 3 reasons for their success or improvement in school Proportion of girl students who cite mentors as one of the top 3 reasons for their success or improvement in school. 0 0 Indicator 12: Percent of school-age children receiving a minimum acceptable diet Proportion of boys that ate cereals (e.g. pasta, rice,noodles, etc) 0.97 125 Proportion of girls that ate cereals (e.g. pasta, rice,noodles, etc) 0.99 133 Proportion of boys that ate white roots like white potato 0.03 125 Proportion of girls that ate white roots like white potato 0.04 133 Proportion of boys that ate vitamin A rich vegetables (e.g. carrot, etc) 0.00 125 Proportion of girls that ate vitamin A rich vegetables (e.g. carrot, etc) 0.03 133 Proportion of boys that ate dark green leafy vegetables like spinach 0.62 125 Proportion of girls that ate dark green leafy vegetables like spinach 0.72 133 Proportion of boys that ate other vegetables (e.g. eggplant, tomato) 0.51 125 Proportion of girls that ate other vegetables (e.g. eggplant, tomato) 0.61 133 Proportion of boys that ate vitamin A rich fruit like mango & papaya 0.47 125 Proportion of girls that ate vitamin A rich fruit like mango & papaya 0.46 133 Proportion of boys that ate other fruit (e.g. watermelon, coconut) 0.26 125 Proportion of girls that ate other fruit (e.g. watermelon, coconut) 0.26 133 Proportion of boys that ate internal organs (e.g. liver, heart) 0.02 125 Proportion of girls that ate internal organs (e.g. liver, heart) 0.04 133 Proportion of boys that ate flesh meat (e.g. chicken, pork, beef) 0.27 125 Proportion of girls that ate flesh meat (e.g. chicken, pork, beef) 0.23 133 IMPAQ International, LLC Page 93 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Proportion of boys that ate any bird's eggs 0.06 125 Proportion of girls that ate any bird's eggs 0.03 133 Proportion of boys that ate seafood (e.g. shrimp, fish) 0.48 125 Proportion of girls that ate seafood (e.g. shrimp, fish) 0.47 133 Proportion of boys that ate nuts and seeds (e.g. lentils, beans) 0.76 125 Proportion of girls that ate nuts and seeds (e.g. lentils, beans) 0.80 133 Proportion of boys that ate dairy products (e.g. milk, cheese) 0.05 125 Proportion of girls that ate dairy products (e.g. milk, cheese) 0.06 133 Proportion of boys that ate oils and fats like butter 0.76 125 Proportion of girls that ate oils and fats like butter 0.79 133 Proportion of boys ate any sweets (e.g. sugar, honey) 0.38 125 Proportion of girls ate any sweets (e.g. sugar, honey) 0.37 133 On the scale of 1 to 15, FAO dietary diversity index for boys 5.65 125 On the scale of 1 to 15, FAO dietary diversity index for girls 5.9 133 Indicator 13: Percent of school children that wash their hands at critical moments Proportion of school children that wash their hands at critical moments 0.82 258 Proportion of children that washed their hands before eating 0.91 258 Proportion of children that did not wash their hands before eating 0.00 258 Proportion of children that did not wash their hands because they did not eat that day 0.09 258 Proportion of children that washed their hands before preparing food 0.38 258 Proportion of children that did not wash their hands before preparing food 0.07 258 Proportion of children that did not wash their hands because they did not prepare food that day 0.55 258 Proportion of children that washed their hands before giving food to others 0.30 258 Proportion of children that did not wash their hands before giving food to others 0.08 258 Proportion of children that did not wash their hands because they did not give food to anyone that day 0.62 258 Proportion of children that washed their hands when they were dirty 0.46 258 Proportion of children that did not wash their hands when they were dirty 0.25 258 Proportion of children that did not wash their hands because they were not dirty that day 0.29 258 Proportion of children that washed their hands after touching dirt 0.34 258 Proportion of children that did not wash their hands after touching dirt 0.26 258 Proportion of children that did not wash their hands because they did not touch dirt that day 0.40 258 Proportion of children that washed their hands after using the latrine 0.39 258 Proportion of children that did not wash their hands after using the latrine 0.12 258 Proportion of children that did not wash their hands because they did not use the latrine that day 0.49 258 IMPAQ International, LLC Page 94 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Indicator 14: Number of students in target schools who achieve a passing score on a test of good health and hygiene practices Proportion of students that believe they should wash their hands before eating 0.97 258 Proportion of students that believe they should wash their hands before preparing food 0.15 258 Proportion of students that believe they should wash their hands before feeding others 0.01 258 Proportion of students that believe they should wash their hands if they are dirty 0.08 258 Proportion of students that believe they should wash their hands after touching dirt 0.07 258 Proportion of students that believe they should wash their hands after using the latrine 0.28 258 Indicator 15: Number of students in target schools who achieve a passing score on a test of food nutrition and dietary practices Proportion of children that have heard of vitamin A before 0.49 258 Proportion of children that say vitamin A provides good vision 0.01 126 Proportion of children that say vitamin A protects the body against diseases 0.25 126 Proportion of children that say vitamin A helps kids to grow up healthy 0.23 126 Proportion of children that say vitamin A keeps the skin healthy 0.02 126 Proportion of children that specify something else vitamin A does 0.37 126 Proportion of children that were not able to specify what vitamin A does 0.25 126 Proportion of children that say liver has vitamin A 0.01 126 Proportion of children that say small whole fish has vitamin A 0.08 126 Proportion of children that say dairy products have vitamin A 0.08 126 Proportion of children that say egg yolks have vitamin A 0.06 126 Proportion of children that say carrot has vitamin A 0.01 126 Proportion of children that say sweet potato has vitamin A 0.00 126 Proportion of children that say squash has vitamin A 0.01 126 Proportion of children that say dark green leafy vegetables have vitamin A 0.10 126 Proportion of children that say red palm oil has vitamin A 0.00 126 Proportion of children that say papaya has vitamin A 0.02 126 Proportion of children that say mango has vitamin A 0.15 126 Proportion of children that say Nere flour (leaves of a local tree) has vitamin A 0.01 126 Proportion of children that say green pepper has vitamin A 0.00 126 Proportion of children that say melon has vitamin A 0.02 126 Proportion of children that specify another food having vitamin A 0.28 126 Proportion of children that were not able to specify a food containing vitamin A 0.51 126 Proportion of children that knew there is iron in the food they eat 0.19 258 Proportion of children that say iron provides good vision 0.06 48 Proportion of children that say iron protects the body against diseases 0.31 48 Proportion of children that say iron helps kids to grow up healthy 0.12 48 IMPAQ International, LLC Page 95 Beoog Biiga II Baseline Report February 2016 Outcome Variable Mean Observations Proportion of children that say helps the muscle function 0.06 48 Proportion of children that say iron helps the brain function 0.00 48 Proportion of children that say iron regulates the body temperature 0.00 48 Proportion of children that say iron prevents fatigue 0.00 48 Proportion of children that say iron is the oxygen carrier in the body 0.00 48 Proportion of children that say iron maintains healthy skin 0.00 48 Proportion of children that say iron helps hemoglobin formation 0.17 48 Proportion of children that specify something else iron does 0.21 48 Proportion of children that were not able to specify what iron does 0.25 48 Proportion of children that say fish has iron 0.08 48 Proportion of children that say the poultry family has iron 0.00 48 Proportion of children that say red meat has iron 0.15 48 Proportion of children that organ meat has iron 0.00 48 Proportion of children that say beans have iron 0.08 48 Proportion of children that say dry peas have iron 0.04 48 Proportion of children that say lentils has iron 0.00 48 Proportion of children that say cowpeas has iron 0.02 48 Proportion of children that say dark green leafy vegetables have iron 0.21 48 Proportion of children that say eggs have iron 0.00 48 Proportion of children that say potatoes have iron 0.02 48 Proportion of children that say tofu have iron 0.00 48 Proportion of children that say green beans have iron 0.00 48 Proportion of children that say nuts (sesame, cashew) have iron 0.00 48 Proportion of children that say green pepper have iron 0.00 48 Proportion of children that say watermelon have iron 0.00 48 Proportion of children that say tomatoes have iron 0.04 48 Proportion of children that specify another food having iron 0.27 48 Proportion of children that were not able to specify a food containing iron 0.44 48 Indicator 16: Number of months of community and/or government support for school canteens Number of months of community and or government support for school canteen 3 15 Proportion of parents that contribute wood every day that the canteen operates 0.43 14 Proportion of parents that contribute wood most of the days that the canteen operates 0.43 14 Proportion of parents that rarely contribute wood of the days that the canteen operates 0.14 14 Proportion of parents that never contribute wood of the days that the canteen operates 0 14 Proportion of parents that contribute utensils every day that the canteen operates 0.64 14 Proportion of parents that contribute utensils most of the days that the canteen operates 0.07 14 Proportion of parents that rarely contribute utensils every day that the canteen operates 0 14 IMPAQ International, LLC Page 96 Beoog Biiga II Baseline Report February 2016 Proportion of parents that never contribute utensils every day that the canteen operates 0.29 14 Proportion of parents that contribute to the storage of food every day the canteen operates 0 4 Proportion of parents that contribute to the storage of food most of the days the canteen operates 0.29 14 Proportion of parents that rarely contribute to the storage of food when the canteen operates 0.14 14 Proportion of parents that never contribute to the storage of food when the canteen operates 0.29 14 Source: Surveys of Student, Parent, Teacher, and PTA Outcome Variable Mean Observations Proportion of communities that pay women to cook for the canteen 0.23 22 Proportion of students that contribute food to the canteen every day it operates 0.5 14 Proportion of students that contribute food to the canteen most of the days it operates 0.36 14 Proportion of students that rarely contribute food to the canteen when it operates 0 14 Proportion of students that never contribute food to the canteen when it operates .14 14 IMPAQ International, LLC Page 97 Beoog Biiga II Baseline Report February 2016 APPENDIX C. ADDITIONAL TABLES AND COMPLEMENTARY OUTCOMES Exhibit 47. Gender Composition of Students Baseline Midline Difference in Means Percent Total Number of Observations Percent Total Number of Observations (p-value test) Proportion of Female Students 52% 343 52% 258 0 (0.82207) Source: Student survey; IMPAQ calculations. Exhibit 48. School District Staff’ Educational Attainment Source: School district administrator survey; IMPAQ calculation; N=30. 27% 13% 13% 37% 7% 3% 0% 5% 10% 15% 20% 25% 30% 35% 40% Percent of Counselors/Managers IMPAQ International, LLC Page 98 Beoog Biiga II Baseline Report February 2016 Exhibit 49. Minimum Acceptable Diet Reported by Parents Outcome Variable Percent Observations Household receiving a minimum acceptable diet The proportion of parents that offered their child cereals (e.g. pasta, rice, noodles, etc.) 98% 262 The proportion of parents that offered their child white roots like white potato 2% 262 The proportion of parents that offered their child vitamin A rich vegetables (e.g. carrot, etc.) 2% 262 The proportion of parents that offered their child dark green leafy vegetables like spinach 84% 262 The proportion of parents that offered their child other vegetables (e.g. eggplant, tomato) 64% 262 The proportion of parents that offered their child vitamin A rich fruit like mango & papaya 34% 262 The proportion of parents that offered their child other fruit (e.g. watermelon, coconut) 17% 262 The proportion of parents that offered their child internal organs (e.g. liver, heart) 5% 262 The proportion of parents that offered their child flesh meat (e.g. chicken, pork, beef) 16% 262 The proportion of parents that offered their child any bird's eggs 2% 262 The proportion of parents that offered their child seafood (e.g. shrimp, fish) 64% 262 The proportion of parents that offered their child nuts and seeds (e.g. lentils, beans) 47% 262 The proportion of parents that offered their child dairy products (e.g. milk, cheese) 5% 262 The proportion of parents that offered their child oils and fats like butter 48% 262 The proportion of parents that offered their child any sweets (e.g. sugar, honey) 25% 262 On the scale of 1 to 15, FAO dietary diversity index reported by Parents 5% 262 Source: Parent Survey; Author’s calculation Exhibit 50. Teacher Reported Utilization of Classroom Activities for this Week Baseline Midline Difference in Means Utilization Percent Observations Percent Observations P- value Individual student checks his or her own work and gives himself/herself a mark/comments Already used 27% 101 35% 105 8 (0.1890) Planning to use 20% 101 38% 105 18*** (0.0037) Have not and not planning to use 54% 101 27% 105 -27*** (0.0001) Students check each other' s work Already used 67% 101 44% 105 -23*** (0.0006) Planning to use 17% 101 43% 105 26*** (0.0000) IMPAQ International, LLC Page 99 Beoog Biiga II Baseline Report February 2016 Have not and not planning to use 16% 101 13% 105 -3 (0.6120) The whole class checks the work of a student Already used 62% 101 48% 105 -14** (0.0335) Planning to use 21% 101 38% 105 17*** (0.0064) Have not and not planning to use 17% 101 14% 105 -3 (0.6161) Students write solutions on a slate and show to the teacher and class Already used 77% 101 49% 105 -28*** (0.0000) Planning to use 18% 101 33% 105 15** (0.0108) Have not and not planning to use 5% 101 18% 105 13*** (0.0032) Students of different skill levels are paired together Already used 69% 101 49% 105 -20*** (0.0024) Planning to use 24% 101 35% 105 11* (0.0719) Have not and not planning to use 7% 101 16% 105 9** (0.0385) Students of the same skill level are paired together Already used 41% 101 10% 105 -31*** (0.0000) Planning to use 16% 101 15% 105 -1 (0.9054) Have not and not planning to use 44% 101 75% 105 31*** (0.0000) Teacher asks group (3 or more) of students to work together on a project and later provides feedback on group performance Already used 48% 101 30% 105 -18*** (0.0077) Planning to use 19% 101 41% 105 22*** (0.0005) Have not and not planning to use 34% 101 30% 105 -4 (0.5251) Source: Teacher Survey; IMPAQ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 IMPAQ International, LLC Page 100 Beoog Biiga II Baseline Report February 2016 Exhibit 51. ASER-Reading Test Results, Disaggregated by Sex (Grade 2) (Reading Level C: Read Complex Sounds) Source: Student Survey; authors’ calculations. N=16 (baseline); N=43(midline). Source: Student Survey; authors’ calculations. N=24 (baseline); N=44 (midline). 25 56 6 0 0 0 6 6 12 49 23 0 5 2 7 2 0 10 20 30 40 50 60 70 Level 0 Level A Level B Level C Level D Level E Level F Level G Level H Level I Level J Level K % Male Students Baseline Midline 4 54 25 0 8 0 8 0 0 7 41 20 2 14 5 7 2 2 0 10 20 30 40 50 60 70 Level 0 Level A Level B Level C Level D Level E Level F Level G Level H Level I Level J Level K % Female Students Baseline Midline IMPAQ International, LLC Page 101 Beoog Biiga II Baseline Report February 2016 Exhibit 52. Reading Proficiency at Second Grade Level (ASER Level B: Reading Simple Sounds) Indicator Baseline Midline Difference in Means Perce nt Observations Perce nt Observations (p- value) Second grade students demonstrating reading ability at grade level or above 32% 40 46% 87 14 (0.1550) Male students demonstrating reading ability at grade level or above 19% 16 40% 43 21 (0.1384) Female students demonstrating reading ability at grade level or above 42% 24 52% 44 10 (0.4106) Source: Student Survey; authors’ calculations. * P-value < 0.1, ** p-value < 0.05, *** p-value < 0.01 IMPAQ International, LLC Page 191 Beoog Biiga II Midline Report February 2016 APPENDIX G. PERFORMANCE EVALUATION SCHOOLS Redacted from public view.