PCI-TANZANIA END-LINE EVALUATION REPORT FOR FFE II PROJECT Submitted by Vincent Vyamana, Ph.D. Pascal Ntunda, M.P.H. Ms. Rehema Dulla Submitted on: May 26, 2017 Agreement Number: FFE-621-2013/033-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of Project Concern International or the United States Government. i ACRONYMS CBP Children’s Book Project DID Difference in Differences EGRA Early Grade Reading Assessment FFE Food For Education FGD Focus Group Discussion GoT Government of Tanzania HGSF Home Grown School Feeding KII Key Informant Interview MoEVT Ministry of Education and Vocational Training PCI Project Concern International RWHTs Rainwater Harvesting Tanks SDG Sustainable Development Goals SGSR School Graduation and Sustainability Readiness SMCs School Management Committees ToT Training of Trainers USAID United States Agency for International Development USDA United States Department of Agriculture VSAs Volunteer Student Aides WE Women Empowered WECs Ward Education Committees WFP World Food Program ii ACKNOWLEDGEMENTS The FFE II end-line evaluation would not have been possible without the hard work and dedication of many people. First and foremost, we would like to thank Mr. Michael Mulford, Former Project Concern International (PCI) Tanzania Country Director, Ms. Amy Cunningham, current PCI Tanzania Country Director, and Blastus Mwizarubi, Senior Program Manager, for their support during the evaluation. We are also thankful to Jennifer Simpson, PCI’s Monitoring, Learning and Evaluation Technical Advisor, and Camilius Kapela, PCI’s Monitoring, Learning and Evaluation Manager, for their technical assistance and guidance. Again, special thanks to all PCI staff who tirelessly implemented the program and fully supported the evaluation team. We would like to extend our sincere thanks to the Ministry of Education, Science and Technology (special mention to Ms. Sara Mlaki), the President’s Office Regional Administration and Local Government Authority (special mention to Dr. Emmanuel Gibai), the Mara Regional Commissioner (Dr. Charles Mlingwa), and all other government officials who were a part of all achievements and who fully participated in the implementation of the program and evaluation team support. We would also like to thank the schools’ management teams, community members, and local government for the support and enthusiasm they showed. We also thank the pupils who voluntarily participated in the assessment. The teachers involved deserve special commendation for their contributions to the end-line evaluation and for their involvement in the program’s many successes. Finally, but not least, this end-line evaluation report would not have been produced without data collected by the Data Vision Team; we truly appreciate their great effort and hard work during the entire period of data collection. Yours sincerely, Vincent G. Vyamana, Ph.D. Mr. Pascal M. Ntunda, M.P.H. Ms. Rehema Dulla May 26, 2017 iii TABLE OF CONTENTS ACRONYMS.................................................................................................................................................. i ACKNOWLEDGEMENTS.......................................................................................................................... ii TABLE OF CONTENTS ............................................................................................................................ iii LIST OF TABLES........................................................................................................................................ iv LIST OF APPENDICES.............................................................................................................................. iv EXECUTIVE SUMMARY............................................................................................................................ v 1.INTRODUCTION.................................................................................................................................... 1 1.1 Background .......................................................................................................................................... 1 1.2 Beneficiaries......................................................................................................................................... 1 1.3 Interventions and strategies.................................................................................................................. 1 1.4 Project Goal and Intermediate Results.................................................................................................. 2 1.5 Scope of Work ..................................................................................................................................... 2 2. METHODOLOGY AND APPROACH................................................................................................... 3 2.1 Evaluation design ................................................................................................................................. 3 2.2 Sampling............................................................................................................................................... 3 2.3 Data collection team recruitment and training....................................................................................... 5 2.4 Data collection tools............................................................................................................................. 5 2.5 Pilot testing the tools............................................................................................................................ 5 2.6 Data collection ..................................................................................................................................... 6 2.7 Data analysis......................................................................................................................................... 6 2.7.2 Estimation of the PCI-FFE II project impacts............................................................................... 7 2.8 Methodological limitations.................................................................................................................... 7 3. FINDINGS ............................................................................................................................................... 8 3.1 Relevance ............................................................................................................................................. 8 3.2 Performance and impacts ...................................................................................................................10 3.2.1 Overview.....................................................................................................................................10 3.2.2 Literacy of school age children.....................................................................................................14 3.2.3 School feeding and agriculture .....................................................................................................15 3.2.4 Impact estimation ........................................................................................................................16 3.3 Effectiveness......................................................................................................................................20 iv 3.4 Efficiency ...........................................................................................................................................24 3.5 Sustainability.......................................................................................................................................25 5. PROGRESS ON IMPLEMENTATION OF MID-LINE EVALUATION............................................27 6. CONCLUSIONS AND RECOMMENDATIONS .................................................................................27 6.1 Conclusions........................................................................................................................................27 6.2 Recommendations..............................................................................................................................27 7. LESSONS LEARNED............................................................................................................................28 REFERENCES ...........................................................................................................................................29 APPENDICES ............................................................................................................................................30 LIST OF TABLES Table 1: Summary of the sampled respondents for the PCI-FFE II project end-line evaluation...................... 4 Table 2: Key informants and participants of focus group discussions for the PCI-FFE II project end-line evaluation....................................................................................................................................................... 4 Table 3: Perceptions of parents and students on the usefulness of PCI-FFE II project interventions............. 8 Table 4: Project performance on school attendance and student performance in classes...............................10 Table 5: Project performance on improved quality of literacy instruction .....................................................12 Table 6: Project performance levels on improved literacy of school age children..........................................15 Table 7: Project performance levels on agriculture and children’s access to meals.........................................15 Table 8: Simple deference in difference EGRA performance........................................................................16 Table 9: Project performance levels for school feeding.................................................................................20 LIST OF FIGURES Figure 1: The proportions of students scoring at the Tanzanian benchmarks for three EGRA benchmark subtasks .......................................................................................................................................................14 Figure 2: Status of implementation of the mid-line evaluation of PCI FFE II project...................................27 LIST OF APPENDICES Appendix 1: Results Framework for the PCI-FFE II Project........................................................................30 Appendix 2: Detailed information for schools sampled and respondents......................................................31 Appendix 3: Progress on implementation of recommendations from PCI-FFE II project mid-line evaluation ....................................................................................................................................................................36 v EXECUTIVE SUMMARY Introduction PCI implemented the FFE project in the Bunda, Musoma Rural, and Butiama District Councils in Tanzania with the overall goal of assisting the Government of Tanzania in its efforts to improve the literacy of school age children. The FFE project endeavored to achieve this goal by improving the quality of literacy instruction, improving attentiveness, and improving student attendance. The project planned to promote teacher attendance and recognition, distribute school supplies and materials, distribute supplementary reading materials, establish libraries, train teachers and administrators, provide school meals, establish school gardens, form savings and lending groups, build and rehabilitate latrines, raise awareness about the importance of education, and establish partnership with farmer groups to supply goods to schools. The FFE also planned to increase the use of health and dietary services. Scope of Work The end-line evaluation focused on five key priorities: 1) Assess whether the project achieved the expected results as outlined in the results framework. The five evaluation criteria listed in the results framework are relevance, effectiveness, efficiency, impact, and sustainability of project interventions and strategies. 2) Measure project outcomes and assess attribution for results. 3) Assess the extent to which recommendations from the midterm evaluation were implemented. 4) Identify lessons learned and provide concrete recommendations for future strategies. 5) Assess the progress and effectiveness of using Volunteer Student Aides (VSAs) to change the knowledge and attitudes of students and parents/communities on the importance of education, use of the library, health and nutrition issues, etc. Evaluation design The end-line evaluation applied a quasi-experimental design, and all individuals were assessed across all variables. The comparison schools were randomly selected from a list of schools that did not receive the intervention and are different from those used as comparison during the baseline assessment. The control schools used in the baseline were contaminated during the end-line evaluation and could not be used for the end-line evaluation. Methodology Sampling The end-line evaluation included 800 pupils, 80 teachers, 280 parents and 40 head teachers from both comparison and treatment schools. A total of 327 key informant interviews and 60 focus group discussions (FGD) with Women Empowered (WE) group members, pupils, members of Zinduka, and Volunteer Student Aides (VSAs) were held. vi Data Collection Methods The data collection methods included four components: 1) Review of project documents and reports 2) Qualitative data collection using FGDs and key informant interviews 3) Administering EGRA to pupils from grades II and IV 4) Classroom observation of teachers at work Data Quality and Analysis Data from FGDs and KIs were transcribed and summarized around themes to identify commonalities and disparities. Quantitative data were coded and entered into Statistical Package for the Social Science (SPSS). Analysis involved the generation of descriptive statistics such as percentages, multiple responses and cross-tabulation. The Difference-in-differences (DID) method was used to estimate the project outcomes on EGRA scores. Findings Relevance The project is aligned with the constitution of the United Republic of Tanzania, 1977, which declares every person’s right to self-education, and that every citizen shall be free to pursue education in a field of his choice up to the highest level according to his merits and ability1 . Similarly, the broad project activities (school feeding initiatives, income generating activities, agriculture activities and health) align with Tanzania’s Law of the Child Act (2009), which requires a parent, guardian or any other person with custody of a child to maintain that child’s right to food, shelter, clothing, medical care, education and guidance, liberty and the right to play and leisure. Data analyses indicate that all interventions converged to improve school attendance (by 79% for boys and 80% for girls). Sustainability PCI has a strong focus on sustainability in the FFE project in terms of financial, technical and institutional aspects. The evaluation found coordination mechanisms already in place, such as regular communication among stakeholders to ensure ownership and institutional sustainability at schools, community and at the district level. To strengthen local capacity in managing the school feeding program, PCI made the following efforts to ensure program sustainability2 : • Facilitated trainings for various government officials to strengthen their capacity in leading trainings, providing support visits, mobilizing communities, and monitoring project activities. • Supported WE savings and lending groups to increase household income and support small businesses. 1 http://www.cssr.uct.ac.za/sites/cssr.uct.ac.za/files/Constitution%20Tanzania.pdf 2 PCI Tanzania FFE-621-2013/033-00 Summary Report: April 2016 to September 2016 vii • Trained Ward Executive Officers and other district officials on how to use the School Graduation and Sustainability Readiness (SGSR) tool. • Conducted sustainability workshops with district and regional officials to ensure close collaboration and build program ownership among government officials. Effectiveness Analyzed data show an improvement in literacy for schoolchildren, improved quality of literacy instructions, and improved attentiveness. The percentage of boys in target schools who, by the end of two years of school, demonstrated reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools increased by 24%, while girls increased their performance by 27%. Similarly, the percent of students in classrooms identified as inattentive by their teachers reduced by 6.4% from the baseline value, and the percent of parents in target communities who can name at least 3 benefits of primary education increased by 8.3% when compared to the baseline value. Additionally, the project completed all planned activities and achieved all targets related to the project objectives. Efficiency The project’s expected results, as stated in the FFE II Results Framework, have been achieved in a timely manner, notwithstanding the delay of latrine construction and Rain Water Harvesting Tanks (RWHTs) during FFE I for various reasons. The project requested a no cost extension to finalize the remaining latrines and RWHTs, which were accomplished during FFE II and assessed in this end-line evaluation. Overall, the project activities were found to have been completed in an efficient manner and within the agreed budget. Overall performance In comparison to baseline data, the project registered the following positive performances: 1) The proportion of girls and boys in the first and second quartiles of EGRA performance below the threshold changed from 91% to 33%, and 88% to 36% respectively. 2) The percentage of boys in target schools who, by the end of two years of school, demonstrated reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools increased by 24%, while girls increased their performance by 27%. 3) There was a significant increase in the proportion of girls and boys in target schools in the third and fourth quartiles of EGRA performance above the threshold from 9% to 67%, and 12% to 64% respectively. 4) The following table demonstrates project performances for output indicators against the targets: Indicator name Target Actual Achievements Number of boy students receiving school meals as a result of USDA assistance 45,110 61,842 137% Number of girl students receiving school meals as a result of USDA assistance 52,164 60,076 115% Number of school staff receiving school meals as a 1,675 1,836 109% viii result of USDA assistance Number of parents receiving school meals as a result of USDA assistance 422 422 100% Total quantity of commodities (listed in tons) provided for school meals as a result of USDA assistance 1,320 2,266 171% 5) Impact estimation: Findings showed desirable changes for all EGRA tasks for the treatment group (Table 8). It is interesting to note that the magnitude of the before-after changes were, in most cases, higher than 50% and sometimes as high as 2.8 fold. However, the changes observed in the treatment groups were accompanied by similar and almost equal changes in the comparison group. Thus, DID analysis revealed no net advantage in the treatment group over the comparison group for all EGRA tasks aside from phonemic awareness, for which DID revealed a net significant desirable change. Extent of Implementation of Recommendations from the Mid-line Evaluation Of the sixteen recommendations highlighted in the mid-line evaluation report, 86% were fully implemented, and all remaining activities are in progress. Conclusions The evaluation team concludes that the PCI FFE II project approach was effective in achieving the intended project outcomes. Although the project had no statistically significant impacts because of factors related to school selection and data analysis (to be discussed in the report), it recorded increases in several impact indicators. Recommendations 1) Proper planning for project sustainability between PCI and government authorities should be maintained to ensure gains are consolidated and continuity is maintained. 2) PCI should promote a monitoring and evaluation system that focuses on fewer sources of information and ensures better analysis and better dissemination of the information collected. 3) In addition to the immediate action of developing a sustainability plan with key stakeholders, adequate community mobilization in the subsequent projects needs to be integrated. 4) To increase community awareness on overall project progress, village leaders need to be supported to ensure that monitoring data from different portfolios of the school feeding project are properly communicated to different social groups in each village, including village leaders, men, women, youth and elders. 5) In the future, EGRA assessment should be designed in a way that permits direct linking of parents, household characteristics and other potential covariates to the students assessed. 6) To avoid the risk of intervention duplication in the same geographical area, PCI should work with the Regional Secretariat to develop and mainstream a project coordination mechanism. This task fits well into the on-going preparation of the exit strategy for the PCI FFE II Program. ix 7) As PCI finalizes its sustainability strategy, it should consider expanding in other districts that are underserved. 8) An influx of similar projects in the project area should be seen as an opportunity rather than threat. In this case, PCI needs to focus on strategic partnerships with a clear division of labor based on the comparative advantages of each partner. This arrangement can help avoid overlaps in specific intervention components, which makes for easier impact attribution. Lessons learned 1) Women’s savings and loans groups (implemented by PCI through WE) are effective in empowering women and enhances women’s ability to actively engage with schools to support their children and contribute food to the school feeding programs. 2) Climate change, unclear policy declarations regarding ‘free’ education, and inadequate community mobilization threaten the effective community engagement that is necessary for project sustainability. 3) The three year project life span was too short to sustain the interventions beyond the project. Additionally, no sustainability component was integrated in the original design framework, making it more difficult to create more lasting impacts. 4) Parents requested information on how much maize and beans the students consumed for school meals each year. Once the rations breakdown was explained, it is a relatively easy choice for parents to decide to contribute towards school meals. 5) Proper documentation and feedback on income and expenditures for school feeding programs increases transparency and trust between the school and teachers, the school committee and parents. 6) School health teachers can be trained to continue assessing the basic health and nutritional status of the students. 7) District and Regional officials’ involvement in school feeding activities is of great importance for project sustainability. 1 1. INTRODUCTION 1.1 Background PCI is an international non-governmental organization legally registered in Tanzania under the Companies Act and in compliance with the NGO Act of 2002 through certificate of compliance number 00001392. PCI’s mission is to empower people to enhance health, end hunger and overcome hardship. From 2011, PCI has worked in partnership with the Government of Tanzania (GoT) through the Ministry of Education, Science and Technology (MoEST), the President’s Office-Regional Administration and Local Government (PO-RALG), and the District Councils of Bunda, Musoma Rural and Butiama, as well as with local communities to implement a three-year Food for Education (FFE) program (2011-2013). The program is funded by the USDA McGovern-Dole International FFE and Child Nutrition Program. Based on the continued needs in the region and the success of the first program, the USDA awarded PCI with a follow-on grant to continue and expand the FFE program for an additional three years (2014- 2016). 1.2 Beneficiaries In collaboration with the Bunda, Musoma Rural, and Butiama District Councils, the project was initially implemented in 134 government primary schools in the Mara region. Due to increased enrollment rates, the Musoma District Council has since split 22 schools into A and B, making a total of 156 active schools. The project aimed to provide daily meals to a total of 91,560 pre￾primary and primary students, teachers and other school staff. The project also benefited parents, school administrators, government officials, extension agents, community members, nurses and health extension workers, reaching 106,449 direct beneficiaries and approximately 360,040 indirect beneficiaries, for a total of 466,489 beneficiaries over the life of the project. Direct beneficiaries included students who received school supplies and materials, improved quality of literacy instruction, improved nutrition, and improved health and hygiene practices. Teachers and administrators undertook a range of capacity building activities to improve their student education and school management skills, and community members were provided with the tools to ensure that processes and systems instituted under the project are sustained. 1.3 Interventions and strategies PCI employed innovative strategies to engage communities in project support, including PCI’s women economic empowerment methodology (WE), which is a savings-led, self-help group methodology, and is a means of improving the incomes of students’ caregivers in participating schools. PCI also utilized the Journey of Life methodology, a community mobilization methodology developed with the support of REPSSI (the Regional Psychosocial Support Initiative) that helps communities and caregivers support children in need through a series of interactive workshops in which children and adults identify the most pressing needs of children in their communities and develop and implement action plans to address them. Teachers at all participating schools applied new teaching techniques and tools in the classroom to increase literacy among students, oversee school feeding activities, utilize health clubs to promote health and hygiene practices, and encourage new agricultural techniques in school gardens. As of October 2015, a total of 103,014 (52,112 boys and 50,902 girls) students and 1,836 teachers in 156 schools benefitted from the provision of school meals and other project 2 activities through USDA assistance. Additionally, 1,193 teachers benefited from specific trainings on school feeding, health, agriculture, and/or education interventions. All 156 schools have established health clubs to promote health and nutritional issues, and almost all cultivated a school garden/farm at least once a year. A total of 1,191,808 tons of commodities were provided for school meals, and a total of 270 WE groups were formed, including 6,110 members. To date, a grand total of 69,547 books have been distributed across all FFE II schools. While visiting three comparison schools in February 2016, we observed varying levels of similar interventions from Education Quality Improvement Programme (EQUIP), a UK-based program focused on improving learning outcomes. 1.4 Project Goal and Intermediate Results The goal of the FFE II project was to improve literacy among school-aged children. This project aimed to achieve this goal through the following specific objectives: i) Improve the quality of literacy instruction through providing school supplies, establishing libraries, producing and distributing reading materials, and training teachers and school administrators ii) Improve attentiveness and reduce short-term hunger through the provision of school meals iii) Develop partnerships with farmer groups to supply food to schools iv) Establish school gardens v) Improve student attendance through better health and nutrition practices and increase community understanding of the benefits of education vi) Improve knowledge of health and hygiene practices through health screening and school health education vii) Increase knowledge of safe food preparation and storage practices through trainings viii)Increase access to food preparation and storage tools and equipment through the construction of energy-saving stoves ix) Increase access to clean water and sanitation services through the construction of new latrines and water systems 1.5 Scope of Work This evaluation’s purpose is to help inform the USDA McGovern-Dole FFE Program and relevant stakeholders on: a. Achievements of the expected results as outlined in the results framework and in line with the five evaluation criteria of relevance, effectiveness, efficiency, impact and sustainability of project interventions and strategies; b. Measurement and attribution of project outcomes; c. The extent of implementation of midterm evaluation recommendations; d. Assess progress/effectiveness of using VSAs in changing the knowledge and attitudes of students and parents/communities on the importance of education, use of library, health and nutrition issues; and, e. Identify lessons learned and provide concrete recommendations for future strategies. 3 2. METHODOLOGY AND APPROACH 2.1 Evaluation design The design for the end-line survey was pre-test post-test non-equivalent comparison group design with judgmental matching of the two groups as described by White and Bamberger (White and Bamberger, 2008). Treatment schools were selected by PCI during inception of the project and all received the same interventions. The comparison areas were selected judgmentally during the baseline assessment, but there has been no control on subsequent development projects entering the area. Treatment and control schools were selected randomly from each category. It is worth noting that comparison schools assessed at the end-line evaluation were different from those assessed at the baseline because the first set of the comparison schools were contaminated in the course of project implementation due to similar interventions by EQUIP. 2.2 Sampling Sampling for the final evaluation survey was accomplished in two stages – (i) selection of 30 schools from the list of 156 intervention schools as well as a sample of 10 from a list of comparison/control schools in non- project (Serengeti district) schools. (ii) Within each sampled school, the selection of respondents for the evaluation was as described below: Schools: A total of 40 schools were selected (see Appendix 2). Out of these 40, 30 were selected from the 156 PCI intervention schools in the project districts. The schools were stratified by student performance, as they were during the baseline survey (using pass rates from the 2014 results of Standard VII PSLE). The selection process ensured that low performing, medium performing, and high performing schools were equitably represented. As during the baseline, the sample of comparison schools could have been drawn from either the list of non-project schools in project councils or the list of schools in non-project councils in the Mara region. This was done to involve schools in one of the non-PCI project districts, as these were less likely to have been ‘polluted’ by the interventions of the FFE Phase II project. Schools in the Serengeti district were considered most applicable among those in the Mara region. Similarly, as it was during the baseline survey design, a propensity score matched pair sample procedure was used to select the ten comparison schools, from which Serengeti District schools were selected based on comparable characteristics to the selected intervention/treatment schools, including student performance. Pupils: Girls and boys were selected separately from each intervention and comparison school and from each selected stream in Standard 2 and Standard 4. Systematic Random Sampling was used to select the students. Ten pupils (5 girls and 5 boys) were selected from one stream in Standard 2 and from one stream in Standard 4, for a total of 20 pupils per school. Teachers: In each intervention and comparison school, the selected stream for Standard 2 and for Standard 4 was used for classroom observation. The respective teacher(s) for each student was thus observed while teaching and subsequently surveyed for the teacher interview. Parents: Each school requested a parent of one randomly selected student in each of Standards 1 to 6 to come to the school to be interviewed on the day of the survey (via a letter sent from the District ahead of time). Although many of the 2016 Standard 7 pupils had 4 completed their school exit exams and the Head Teacher from each school used records to identify and request a parent of one such ex-pupil. In total, 7 parents were selected in each sampled school. Members for FGDs: The composition of the proposed FGDs was presented to the Head Teacher in advance of the field work. The two FGD groups were: i) Children members of Zinduka Group3 , VSA members4 , and 40 VSA teams (each team composed of former students as described previously) ii) WE group members5 , and community members not in the village/Mtaa leadership or in school management committee. Each FGD comprised of not more than 12 participants and the respective FGD Guide. The guides were designed to facilitate effective and productive meetings, including identifying issues and/or concerns with respect to the evaluation framework. It comprised checklists, questions, or open ended questions focusing on those issues or concerns. To the extent possible, the genders were represented equally. Member(s) for Key Informant Interviews (KIIs): Key informants were selected on the basis of respective responsibility demanded by the final evaluation. The KIIs have been designed to facilitate effective and productive interviews/meetings, including identifying issues and/or concerns with respect to the evaluation framework. It comprised checklists, questions, or open ended questions focusing on those issues or concerns. Table 1 summarizes number of different categories of respondents sampled. Table 1: Summary of the sampled respondents for the PCI-FFE II project end-line evaluation Respondent group Sample size Interventions schools Comparison schools Total Standard 2 pupils 300 100 400 Standard 4 pupils 300 100 400 Teachers 60 20 80 Parents 178 60 238 Head teachers 30 10 40 Table 2: Key informants and participants of focus group discussions for the PCI-FFE II project end-line evaluation Category of participants Number of Key informants PCI staff 5 Representatives from the Ministry of Education 2 Representatives from Regional Secretariats 4 District authorities (Bunda TC, Bunda DC, Musoma DC, Butiama DC 16 3 Zinduka Group is a club which include students from class three to class seven 4 Volunteer Students Aides comprise of two ex-primary school leavers per school 5 WE GROW Group is a serving and credit group comprise of parents in the community mainly women and facilitated by PCI 5 Ward Education Coordinators 16 Village/street leaders 16 Members of school management committees 30 Teachers non-teaching school staff 238 Total key informants 327 Focus group discussions Pupils and members of Zinduka and VSAs 30 WE members and other parents 30 Total focus group discussions 60 2.3 Data collection team recruitment and training Data collectors with a minimum qualification of a degree in social sciences and previous experience of conducting or assisting in a similar assignment, especially in similar cultural and geographic communities were recruited. Seven data collectors and the external End-line Survey Coordinator also administered Version B of the EGRA that was prepared and pre-tested during the baseline survey. Prior to data collection, the data collectors were trained in data collection methodologies, data management processes and research ethics. For quality control, all completed reading assessments, questionnaires, observation forms, and checklists were reviewed daily by the supervisors to ensure complete and correct recording. The external Baseline Survey Coordinator also reviewed a sample of forms in the field each day during data collection as well as during data entry. 2.4 Data collection tools Early Grade Reading Assessment (EGRA) protocols used during the baseline survey were maintained to facilitate comparison between the intervention and non-intervention schools and for consistency among the schools being surveyed. The EGRA tool covered the following reading tasks: phonemic awareness, letter sound knowledge, invented word decoding, oral passage reading and reading comprehension. Other data collection tools adapted from the baseline survey included: interview guide for Head Teachers, school observation checklist, classroom teaching observation checklist, interview guide for teachers, interview guide for pupils, interview guide for parent, FGD protocol for adults and children and interview guide for school cook. 2.5 Pilot testing the tools Except for KIIs, all data collection tools were tested before the actual fieldwork. The main reasons for the pre-testing were to evaluate the questionnaire and to check whether it was applicable and suitable in the study, whether the questionnaire was understood well by the respondents; and to identify the time taken to administer one questionnaire. The pre-testing of the tools was conducted in two phases. In the first phase, EGRA tools and interview guide for pupils were tested using a total of 64 pupils in Standard 2 and Standard 4 in four schools were interviewed. In the second phase, the Head Teacher interview guide, school observation checklist, parent interview guide, designated teaching and non-teaching staff 6 interview guides and school management committee interview guides were pre-tested in 4 schools. Based on experiences from the pre-testing, questionnaires were discussed and revised by the evaluation team in collaboration with PCI technical team. 2.6 Data collection Collection of end-line evaluation data involved the following tasks: 1. Early Grade Reading Assessment (EGRA): This involved administering the EGRA one‐ on‐one with children in Standards 2 and 4. The EGRA tool covered phonemic awareness, letter sound knowledge, invented word decoding, oral passage reading and reading comprehension. 2. Student Interview: The same students assessed with EGRA were interviewed about their food consumption at school, feelings of hunger, health‐ related absences, and their knowledge related to health and hygiene. 3. Teacher Observation: Standard 2 and Standard 4 teachers were observed as they were teaching a reading lesson. The purpose was to identify teacher behaviors associated with positive student performance and to identify ways to support teachers to improve instruction in reading. 4. Teacher Interview: The same teachers observed teaching a reading lesson were also interviewed regarding their practices in literacy instruction, provision of school supplies and materials, training received, perceived student attentiveness and access to school meals. 5. School Administrator Interview: Head Teachers or other school administrators were interviewed at each school with the aim of investigating the use of new techniques/tools, training received and access to school meals. 6. School Checklist: Used for direct observations of school infrastructure and was conducted at each school to verify the existence of project-related supplies and infrastructure (latrines, food storage, library, etc.) 7. Parent Interviews: Parents of students in pre‐primary and Standards 1‐7 were interviewed to investigate their levels of awareness and understanding of the benefits of primary education, their participation in project‐supported groups and activities and their child’s attendance at school. 8. Secondary data: Project documents, progress reports, the baseline report and mid-line reports were reviewed to extract information regarding project design, progress and levels achievements. 2.7 Data analysis Qualitative data from FGDs and KIIs were transcribed and summarized around themes during discussions with community members in each village. The purpose was to identify 7 commonalities and disparities among different people, and where necessary, trying to understand the reasons for any disparity. Quantitative data were coded and entered into a computer spreadsheet in the Statistical Package for Social Science (SPSS). Analysis involved generation of descriptive statistics such as percentages, multiple responses and cross-tabulation. In addition, simple difference-in￾differences (DID) was conducted on parameters of interest including scores for different EGRA tasks. 2.7.2 Estimation of the PCI-FFE II project impacts As noted earlier, “Parallel Paths”, which is the basic assumption for validity for DID was not tested due to the lack of time series pre-treatment data (White, 2006; Lechner, 2011). Therefore, the DID findings need to be interpreted cautiously. The DID method used the following steps to estimate the project outcome: 1. We calculated the difference in the outcome (Y) between the before and after situations for the treatment schools (B-A) 2. We calculated the difference in outcome (Y) between the before and after situations for the comparison schools ( D-C) Then, we calculated the difference between the difference in outcomes for the treatment schools (B-A) and the difference for the comparison schools (D-C). OR, DID= (B-A)- (D-C). This ‘’difference-in-differences‘’ is our impact estimate. 2.8 Methodological limitations The approach used for end-line evaluation spent much of the time in the field, with comparatively little time devoted to follow up at the national level. Thus, national contextual issues were not a major focus of this end-line evaluation and were only addressed through secondary data, and/or KIIs. Due to unavoidable circumstances, at least two different evaluation teams, albeit uncoordinated, were engaged in this end-line evaluation in a relay fashion where the teams replaced one another in a succession. This arrangement has resulted in dilution or omission of personal experiences, especially of the first group; developed during design and field data collection. The baseline survey design was robust in that it applied propensity score matching technique to identify and select comparison schools. However, the end-line evaluation has not been able to build on that strength because the comparison schools were contaminated by EQUIP interventions during the course of project implementation. Due to the contamination, all of the comparison schools were replaced with uncontaminated comparison schools, even though they were not well-matched with the treatment schools, therefore the datasets collected do not have adequate pre-treatment explanatory variables (covariates) related to program participation and outcomes. This in turn limits the possibility to apply regression techniques to estimate the treatment impacts (e.g. Smith and Todd, 2005). Similarly, the “Parallel Paths” assumption that underlies DID technique for estimation of the treatment impacts has been applied but without validation. Authentication of the assumption 8 requires use of pre-end-line evaluation time series data to test for differences in the pre￾treatment trends of the treatment and comparison groups, for which there was no data. 3. FINDINGS Findings are described under the five evaluation criteria of relevance, efficiency, effectiveness, sustainability performance and impact. Analytical questions posed under each criterion are referred to in each subsection. 3.1 Relevance Relevance was assessed on the basis of three domains: 1) extent to which project interventions met the needs of targeted beneficiaries and responded to other emergent needs that arose out of the course of project implementation; 2) the extent to which the project interventions were aligned with relevant policies and priorities of other stakeholders; and 3) appropriateness of the project design. Findings on relevance are summarized around these domains. QN: To what extent did the intervention meet the needs of targeted beneficiaries and account for other emergent needs during the life of the project? Relevance of the different project interventions for each category of beneficiaries is reflected through retrospective linkage of the interventions to the perceived benefits self-reported by different groups of beneficiaries that were interviewed. The findings clearly show that more than 60% of all categories of beneficiaries interviewed either in groups (FGD) or individually recounted overwhelming lists of benefits associated with each and every PCI-FFE II project interventions: water supply, school latrines, school feeding, WE and farmer groups, Zinduka health clubs, HURU kits distribution, establishment of library and provision of library books; capacity building to parents, teachers and school administrators. Of paramount importance are the facts that almost all interventions converged toward improvement in school attendance (79% for boys and 80% for girls) which corresponds to necessary conditions for enhancing positive change in literacy performance. It is not surprising that during FGDs and KIIs, parents perceived improvements in performance of their children at schools and in the national standard 7 examination. For example, as indicated in Table 3, parent FGDs at Kambarage in Nyakatende Ward perceived that “the presence of the project has improved standard 7 performances in Bunda and Musoma.” These positive perceptions on the project from a range of beneficiaries provide significant indication that the project was in accordance with the needs of all categories of target beneficiaries and other emergent needs. The following are some of the quotes to confirm that the project was relevant to the needs of the beneficiates. Table 3: Perceptions of parents and students on the usefulness of PCI-FFE II project interventions 9 Project interventions Selected quotes on benefits School latrines Buzimbwe parents FGD in Butimba Ward reported, "...Another change that I have observed that I can attribute to the PCI project is the decrease in diarrhea diseases/worms....; our children don’t have worms anymore...." Provision of library books Mugane students FGD in Mugango ward reported, “...PCI project has provided us with a variety of books that help us to practice reading and acquiring additional knowledge; besides improving our reading skills, the different stories we read from these books have impart various knowledge that enable us to live well in our community...” Provision of food to students Chimati B students FGD reported, “...this food has enabled many children to come to school some children never wanted to come to school or sometimes they slept during classes because of hunger; now with the support from PCI pupils are more attentive in classes because they eat well and return to class.....we appreciate PCI and wish them peace, love and good progress..." QN: How aligned were project interventions with other stakeholder policies and priorities First, the broad project activities, i.e. school feeding initiative, income generating activities, agriculture activities and health align very well with Tanzania Law of Child Act (2009), which requires a parent, guardian or any other person having custody of a child maintain that child - in particular ensuring the child’s right to food, shelter, clothing, medical care, education and guidance; liberty and right to play and leisure. A parent with a stable livelihood (e.g. health, income and networks) is likely to fulfill her/his duties to abide to Tanzania Child Act. Furthermore, project focus on the quality of literacy instruction which is in line with major education programs in Tanzania at national level. These include Tanzania 21st Century Basic Education Programme (TZ21) funded by USAID, EQUIP in Tanzania funded by UKAID, Big Results Now and Literacy and Numeracy Education Support Programme (LANES). Secondly, the project fits with the most pressing needs in agriculture of providing capital and access to improved farming practices. These pressing needs include: improving the incomes of students’ caregivers in participating schools and improving agricultural practices that improve crop yields and support school feeding. However, the project needs to ensure that communities have access to relevant and updated marketing information. In addition to that, the project also aligns with Tanzania Development Vision 2025, which insists on high quality livelihoods whereby Tanzania will move from a low productivity agricultural economy to a semi￾industrialized economy to be led by modernized and highly productivity agricultural activities which are effectively integrated and buttressed by supportive industrial and service activities in 10 the rural and urban areas6 .The FFE also contributes to the “Free Education system’’ as announced in January 2016 by the President of Tanzania. The stated Policy required removal of school fees for children to attend schools to promote the enrollment of all school-aged children and reduce economic barriers for families to enroll their children in school. Thirdly, the FFE project intervention also aligns with Sustainable Development Goals (SDG) and supports Tanzania’s efforts to reach the following SDGs: end poverty in all its forms everywhere (SDG1), end hunger, achieve food security and improved nutrition, and promote sustainable agriculture (SDG2), ensure healthy lives and promote wellbeing for all at all ages (SDG3), ensure inclusive and equitable quality education and promote lifelong learning opportunities for all (SDG4), achieve gender equality and empower all women and girls (SDG5), and ensure availability and sustainable management of water and sanitation for all (SGD7). QN: How appropriate was the project design? The project design is full-bodied given that it covers three important domains (improve attentiveness, improve quality of literacy instructions, and improve student attendance) that determine learning performance outcomes. Similarly, the design has addressed all necessary intermediate causes of poor learning performance outcomes in Tanzania. Foundation causes (increase capacity of governance institution, improve policy and regulatory framework, increase government support and increase engagement of local organization and community) integrated in the project design are relevant by all standards. 3.2 Performance and impacts QN: What were the impacts of project activities on the target beneficiaries? 3.2.1 Overview Overall, the project has produced very positive results and has achieved its objectives around the four domains of change specified in the results framework: improved quality of literacy instruction, improved attentiveness and improved student attendance. These domains of change are further discussed in the light of the qualitative and quantitative evidences from the end-line evaluation collected. 1) PCI-FFE II project has improved school attendance and performance in classes. This outcome is assessed based on the quantitative findings presented in Table 4. Table 4: Project performance on school attendance and student performance in classes Indicator Achievement Status of change in Source of Treatment Comparison data 6 http://www.mof.go.tz/mofdocs/overarch/Vision2025.pdf 11 Baseline End-line Baseline End-line the treatment group Average number of days students report being absent from school due to illness during the previous month 2.28 NS 5.80 NS 2.37 NS 3.2 NS Student interviews Baseline report Percent of students in classrooms identified as inattentive by their teachers 14 NS 7.6 NS 18 NS 9.1 NS Teachers’ interview Baseline report Percent of students in target schools who indicate that they are “hungry” during the school day 45.0*** 36.0*** 76*** 54.5*** Student interview Baseline report Percent of students who report a decrease in health related absences 25 NS 56.05 NS 34 NS 59.4 NS Student interview Baseline report Percent of students in target schools who achieve a passing score on a test of good health and hygiene practices 85.2 NS 94.5 NS 89.5 NS 95.5 NS Student interview Baseline report Percent of parents in target communities who can name at least 3 benefits of primary education 85 NS 93.3 NS 84 NS 96.6 NS Parents interviews Baseline report Interpretation key Undesirable change but not significant Undesirable change and significant Slightly desirable change to no change Positive and/or significant change NS = Not significant; * = Significant at alpha = 0.05; ** = Significant at alpha = 0.01; *** = Significant at alpha = 0.001 Although quantitative data in Table 4 does not show any improvement in school attendance, there are strong qualitative evidence from a wide range of stakeholders, including students, to confirm that claim. The key weakness of the quantitative data reported is that they were collected at one point in time and covered a very short time span. To the contrary, qualitative data were collected through dynamic interactions with participants that enabled them to consider a relatively long historical time horizon. Increased school attendance was linked to water supply interventions, improved access books, improved nutrition through feeding intervention, improved school latrines and access to menstrual health facilities for girls. Examples of quotes from various FGDs are revealing: 12 Bwenda parents FGD in Rusoli Ward reported, “...we have received food which helped our children to improve their attendance to school and also we have received fruits for their health.....; "...first PCI has given food for our children and also they have given plates to them to reduce the burden to them instead of carrying plates with them all the times..."; "...Before PCI our children used to miss the school since a child when coming from home normally he/she is empty stomach so it happens that he fails to concentrate in the class...” The project has helped girls to attend classes even during their menstrual periods as a result of HURU kit distribution. This was confirmed in FGDs with girls and parents. It appeared in 102 out of 137 occasions in which successes emanating from HURU kits distribution were mentioned. For example Kiriba parents reported during an FGD that,“...on Huru it has helped our girls so much from missing school during their menstruation periods; through HURU plan we find our girls in class daily as opposed to the past when they used to miss school in 3 consecutive days during their menstrual period....” Improved student performance in classes is clearly exemplified by the findings on EGRA subtask scores that exceed the national benchmarks by 56% for Non-word Reading and by 42% for Reading Comprehension as indicated in Figure 1 and further discussed in sub-section 3.2.2. 2) The project has helped to empower women economically and socially through the WE groups who in turn use their income to provide for school requirements for their children. This fact came out very clearly in all women FGDs. For example, Kambarage parents FGD in Nyakatende Wards reported,“...these groups help us, our children continue going to school; when they need anything such as exercise books or school uniforms, and we get loans from these groups that help us to provide for our children...” 3) Teachers and school administrators have improved their literacy instruction approaches and skills that encourage student learning, as testified by parents and students encountered in FGDs and KIIs. Findings from the end-line evaluation showed that over 78.3% of teachers in the treatment schools use new teaching techniques other than blackboard and textbooks against the planned benchmark of 75% teachers, in comparison to only 10% recorded during the baseline. The proportion of teachers using new teaching methods (80%) in comparison schools was comparable to that of treatment schools (α = 0.025; p = 0.88). Lack of significant differences between the trends observed in the treatment and comparison schools is attributable to contamination of the comparison schools. Discussion with project staff and local government officials confirmed influx of similar education projects in the area that was designated as a control area. Table 5: Project performance on improved quality of literacy instruction Indicator Achievement Status of change Source of Treatment Comparison data 13 Baseline End-line Baseline End-line in the treatment group Percent of teachers in target schools who demonstrate use of new and quality teaching techniques or tools 10 NS 78.3 NS 15 NS 80.0 NS Class observations; Baseline report Percent of teachers in target schools who allow students to ask questions during class 17 NS 16.6 NS 21 NS 0.0 NS Class observations; Baseline report Percent of teachers in target schools who allow students to respond to questions raised in class by other students 12 NS 6.6 NS 17 NS 0.0 NS Percent of school administrators in target schools who demonstrate use of new and quality teaching techniques or tools 10 NS 76.6 NS 15 NS 80.0 NS Head teachers’ interviews; Baseline report Interpretation key Undesirable change but not significant Undesirable change and significant Slightly desirable change to no change Positive and significant change NS = Not significant; * = Significant at alpha = 0.05; ** = Significant at alpha = 0.01; *** = Significant at alpha = 0.001 4) The project has improved the health of students through increased access to improved latrines, water supply and nutritious food including fruits and vegetables served as part of school feeding intervention. Qualitative findings indicated that the PCI-FFE project had resulted in improvement of health of students through two pathways. The following examples of qualitative evidence support this proposition: Buzimbwe parents FGD in Butimba Ward reported, "...Another change that I have observed that I can attribute to the PCI project is the decrease in diarrheal diseases/worms....; our children don’t have worms anymore...." Kihuzu Pupils FGD reported, ‘’...A tank has been built for drinking water, most students used to 14 come from their homes with drinking water in their bottles others used to fetch water from unsafe source which could bring different problem such as cholera, PC1 has helped us so much...’’ Kihuzu Pupils FGD reported, ‘’...A tank has been built for drinking water, most students used to come from their homes with drinking water in their bottles others used to fetch water from unsafe source which could bring different problem such as cholera, PC1 has helped us so much...’’ ".... we are also happy because PCI has brought food for our children, when it reaches 10:00 am all children are happy, and they eat in the afternoon; when they come back home we see that their health is getting better.....” 3.2.2 Literacy of school age children The project envisages achieving the objective in “literacy of school children” at both outcomes and output levels. The outcome level focuses on overall reading performance in grade 2 and 4 for boys and girls, which must be equivalent to the predetermined national benchmark. The baseline report did not report any value for this variable as there was no national benchmark at that point in time. Therefore, performance in both treatment and comparison schools is hereby gauged against the national benchmarks and targets for the year 2016. Figure 1: The proportions of students scoring at the Tanzanian benchmarks for three EGRA benchmark subtasks Overall, the performance of the treatment group across all EGRA subtasks is better when compared to the comparison group. As indicated in Figure 1, the treatment group exceeded the Tanzania national benchmarks by 56% and 42%for Non-word Reading and Reading Comprehension, respectively. However, it scored lower than the national benchmark for Oral Reading Fluency by 15%. On the other hand, comparison group fell short of the national benchmarks for all EGRA subtasks with exception of Non-word Reading where the score 7.8% 17.8% 24.2% 7% 13% 14% 5% 21% 17% 1.3% 6.5% 12.1% 0% 5% 10% 15% 20% 25% 30% Non-word Reading Oral Reading Fluency Reading Comprehension Percentage of students EGRA Subtasks Treatment schools Comparison schools National target for 2016 Tanzania national EGRA 2016 15 exceeded the national benchmark by 40%. Similarly, review of project reports and monitoring data revealed significant improvement in the literacy of school children. The project performance exceeded the planned implementation targets by 160% and 168% for boys and girls respectively (Table 6). Table 6: Project performance levels on improved literacy of school age children Indicator Target Value at the baseline Value at the end￾line Achievement (%) Indicator: Percent of boys in target schools who, by the end of two years of school, demonstrate reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools 40% 12% 64% 160% Indicator: Percent of girls in target schools who, by the end of two years of school, demonstrate reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools 40% 9% 67% 168% Furthermore, it is appealing to note that these findings compare very well with illiterate rates of 35% for boys and 40% for girls recorded in the July-September 2016 PCI project performance report. This conformity indicates a high degree of reliability of the PCI-FFE II performance monitoring data. 3.2.3 School feeding and agriculture The activities employed to reduce short-term hunger and ensure pupils remains attentive in class included: provision of school meals, establishment of school gardens, and development of partnership with farmers group to supply food to school. School meals were provided to all pre￾primary and primary students and adults (teachers, cooks, guards) in all intervention schools. The FFE project analyzed data and reported that, a total of 674.49 metric tons of USDA-provided rice, pinto beans, and vegetable cooking oil were distributed to support school meals four times a week, while on the fifth day meals were provided by schools and the surrounding community. Table 7: Project performance levels on agriculture and children’s access to meals Indicator name Target Actual Achievements Number of boy students receiving school meals as a result of USDA assistance 45,110 61,842 137% Number of girl students receiving school meals as a result of USDA assistance 52,164 60,076 115% 16 Number of school staff receiving school meals as a result of USDA assistance 1,675 1,836 109% Number of parents receiving school meals) as a result of USDA assistance 422 422 100% Total quantity of commodities (tons) provided for school meals as a result of USDA assistance 1320 2,265.627 171% Under this outcome, the project supported the establishment of seasonal school gardens in all 156 intervention schools, distributed local varieties of nutritional vegetable seeds and agriculture teachers were trained on nutritious vegetables and improved agriculture practices, such as manure application and proper seed spacing. Regarding the establishment of school gardens, a total of 121,918 students, and 1,836 teachers were reported to benefit from the school garden interventions. In additional, FFE held a total of four farmer school days. These events aimed to provide commodity contributions, a portion of which was then used for cooking demonstrations to be tested by students for their preference: porridge with sugar, porridge without sugar, a boiled maize meal, boiled maize and beans meal. FFE also established a strong partnership with farmer groups to supply food to intervention schools. A total of 16 farmer groups consisting of 846 group members, including 343 members who also belong to WE savings and lending groups, contributed maize or sorghum inputs and received trainings on improved agriculture techniques. The aim of partnership was a part of the project sustainability techniques to create project ownership with the community members as well as farmers group. In total, 31 partnerships were developed with farmer groups to supply food to schools. To show the commitment between PCI and farmer groups, they signed an agreement to remit at least 20% of the farmer groups’ harvests to their associated beneficiary school. PCI collaborated with agriculture extension officers and local government officials, such as District officials and Ward and Village executive officers, to provide on-farm support and trainings as well as conduct monitoring visits. Project records show that to date 28,993 students in 29 schools are receiving food from farmers’ groups that is consumed on the fifth day of the week. 3.2.4 Impact estimation Findings showed changes in a desirable direction for all EGRA tasks for the treatment group (Table 8).It is interesting to note that the magnitudes of the before-after changes were in most cases higher than 50% and went as high as 2.8 fold in some cases. However, the changes observed in the treatment groups were accompanied with similar and almost equal changes in the comparison group. Thus, DID analysis revealed no net advantage of treatment over the comparison group for all EGRA tasks; except phonemic awareness for which DID revealed a net significant desirable change. Table 8: Simple deference in difference EGRA performance Variable Treatment group Comparison group f e r e n t 17 Value at baseline (A) Value at end￾line (B) Difference (B￾A) Value at baseline (C) Value at end￾line (D) Difference (D-C) a) EGRA score by Task 1. Phonemic Awareness 33% 39.32% 6.3% 35.0% 40.60% 5.6% 0.72% 2. Letter Sound Knowledge 12% 46.50% 34.5% 11.0% 51.66% 40.7% -6.16% 3. Inverted word decoding 22% 65.49% 43.5% 21.0% 64.08% 43.1% 0.41% 4a. Oral passage reading 25% 72.20% 47.2% 23.0% 69.34% 46.3% 0.86% 4b. Reading comprehension 11% 38.33% 27.3% 9% 38.20% 29% -1.87% b) Zero Scores by EGRA task 1. Phonemic Awareness 23.30% 14.00% -9.30% 22.50% 27.50% 5.00% -14.30% 2. Letter Sound Knowledge 20.00% 11.83% -8.17% 31.50% 15.00% -16.50% 8.33% 3. Inverted word decoding 42.70% 19.33% -23.37% 47.00% 13.50% -33.50% 10.13% 4a. Oral passage reading 42.70% 18.83% -23.87% 47.50% 10.00% -37.50% 13.63% 4b. Reading comprehension 48.80% 26.50% -22.30% 52.50% 22.00% -30.50% 8.20% c) Zero Scores by Task for girls 1. Phonemic Awareness 25.70% 7.50% -18.20% 26.50% 12.00% -14.50% -3.70% 2. Letter Sound Knowledge 22.00% 5.83% -16.17% 38.80% 6.00% -32.80% 16.63% 3. Inverted word decoding 45.30% 10.67% -34.63% 52.00% 5.50% -46.50% 11.87% 4a. Oral passage reading 45.30% 10.17% -35.13% 53.10% 4.50% -48.60% 13.47% 4b. Reading comprehension 52.70% 14.50% -38.20% 58.20% 9.00% -49.20% 11.00% Total EGRA Score 11.70% 0.83% -10.87% 16.30% 1.00% -15.30% 4.43% d) Zero Scores by Task for boys 1. Phonemic Awareness 20.80% 6.50% -14.30% 18.80% 15.5% -3.30% -11.00% 2. Letter Sound Knowledge 18.80% 6.00% -12.80% 24.80% 9.0% -15.80% 3.00% 3. Inverted word decoding 40.30% 8.67% -31.63% 42.60% 8.0% -34.60% 2.97% 4a. Oral passage reading 40.30% 8.67% -31.63% 42.60% 5.5% -37.10% 5.47% 4b. Reading comprehension 45.10% 12.00% -33.10% 47.50% 13.0% -34.50% 1.40% Interpretation key Significant undesirable change Undesirable change but not significant Desirable minor change to no change Significant desirable change The most plausible explanation for the obscured effects of the PCI interventions on EGRA tasks is the fact that the comparison group comprised schools that were contaminated due to influx of other education projects in the Serengeti district which were previously ear-marked during the baseline as the most appropriate comparison schools. 18 It was not possible to validate the “Parallel Paths”, the fundamental assumption underlying DID, because there were no time series data required for that analysis. According to Lechner (2011), for “Parallel Paths” assumption to be valid it requires that both treatment and comparison groups having the same historical trends for the variable of interest. Furthermore, application of regression based methods for estimation of the treatment effects from the cross-sectional survey of the treatment and comparison groups (White and Bamberger, 2008) was constrained as well. These methods work properly when the data set includes rich pre-treatment variables (covariates) related to program participation and outcomes, and when the dependent variable is measured in the same way for participants and non-participants (e.g. Smith and Todd, 2005). Conversely, some of the potential covariates, such as parent and household characteristics, collected from a sample of parents were rendered inefficient, because they lacked direct linkage to the interviewed students. This very challenge was highlighted in the baseline report for the same project. Thus, it is recommended that in the future, EGRA assessments be designed in a way that permits direct linking of parents, household characteristics and other covariates to the students being assessed. QN: What were the unintended impacts of the approaches used, either positive or negative? How did these impacts transpire and how could they have been maximized or mitigated? Positive unintended impacts for children and parents: i). Results from FGDs found that parents have become more responsible in following up at schools, assessing teaching processes and assessing progress in classes as they feel ownership due to the contribution of food and money to pay the cooks and guards. Initially this role was assumed to be the responsibility of the Ward Education Committee (WEC) and DEO. ii). FGDs with parents further revealed that moral conduct of their children were enhanced through the reading of story books that narrate lives of children as characters in the books. The project has already taken steps to enhance the sustainability of these impacts through sustainability planning workshops. Roles and responsibilities agreed upon in the sustainability workshops reinforce the roles of parents as key actors in inspecting schools. This in turn enhances downward accountability of teachers and school administrators. One negative unintended impact was impaired student attendance, due to inadequate water from the rain water harvesting systems. In essence, rainwater harvesting systems were installed to increase water supply; but the systems are not collecting adequate water in some schools, most likely due to inadequate rains or rainfall variability. As a result, students are forced to miss classes as they go to fetch water from distant sources. This is of particular concern as it jeopardizes the very impact intended by the project. Once students are involved in fetching water at school, they likely to resort back to absenteeism; and even when they stay in school, their performance is impaired as they use much of their time fetching water instead of studying. 19 QN: How can impacts be sustained through future programming and / or ongoing activities? The impacts recorded in the PCI-FFE projects are not guaranteed to be sustainable unless some salient sustainability issues are addressed before the final closure of the project. Prior to the end￾line evaluation, PCI had assessed school sustainability readiness using an innovative tool that explored key parameters to detect readiness and capability of a given school to move ahead with the project activities without PC1. The results showed that at least 40% of all schools reached were in danger of being unable to perpetuate project interventions beyond the project life. Given the interrelated challenges on unreliability of water supply from the rain water harvesting systems and reluctance of parents to contribute for school development in connection to the “Elimubure” presidential declaration (Fee Free Education), the 40% school readiness may not be realistic. In any case, sustainability the PCI-FFE II interventions cannot be guaranteed unless the project takes a proactive step to engage all stakeholders at all level to agree and develop realistic sustainability plan with commitments on the roles and responsibilities for each of the relevant actors. QN: How can future similar programs improve their planning processes to address internal and external challenges and constraints? The project arrangement with respect to food provision is that the project provides food for four out of five days of the week; and parents are required to provide food for the fifth day. In this regard, the major challenge narrated by PCI senior project management team and other staff, parents, students, teachers, village leaders and local government staff is the reluctance of parents to contribute food for the fifth meal on Friday. When asked how the challenge could be solved, the recurrent response was that parental awareness should be created and improved through meetings and seminars. A careful analysis of the situation revealed that the root cause of the problem was that community engagement processes during project inception did not cover the aspect of the parent contribution for the fifth meal. In fact, some parents and teachers claimed that during the first year of the project, PCI provided meals for all five days but later changed the process without adequate consultation with communities and stakeholders. To avoid such challenges in the future, PCI should invest more in community mobilization, stakeholder analysis and community action planning at the very beginning of the project. This will ensure clear roles and responsibilities pertaining to each stakeholder; and the action plans can be used to enforce accountability for each category of stakeholders. Another challenge was the “Elimubure” (free education) presidential declaration that was misinterpreted by many Tanzanians. Parents are using that declaration to refuse to contribute anything to support school operations. Specific to PCI project was the fact that smooth operation of the feeding program requires cash contributions from the parents, which is used to pay the cooks and guards. As a result, the cook occasionally boycott when they are not paid, and students miss their meals on those days. Similarly to the challenge of parents refusing to contribute food for their children, this too can be avoided in the future through appropriate community mobilization, stakeholder analysis and community action planning at the very 20 beginning of the project. 3.3 Effectiveness This section presents the status of achievement of the program's objectives and major factors influencing the achievement or non-achievement of the objectives. This section is discussed and presented under the expected results (guided by the evaluation questions) as outlined in the project results framework. QN: To what extent have the stated project results and purpose been achieved to-date? MGD 1.2.1: Reduce short term hunger The activities employed to reduce short-term hunger and ensure pupils remain attentive in class included the provision of school meals, establishment of school gardens, and development of partnerships with farmer groups to supply food to school. School meals were provided to all pre￾primary and primary students and adults (teachers, cooks, guards) in all intervention schools. The FFE-verified monitoring records and progress reports showed that at the time of the end-line assessment the project had worked in 156 primary schools, reaching 106,091 pre-primary and primary student beneficiaries, 1,675 teachers, 169 guards and 253 cooks. Under this objective, the project supported establishment of seasonal school gardens in all 156 intervention schools, distributed local varieties of nutritional vegetable seeds and provided training to the agriculture teachers on nutritious vegetables and improved agriculture practices, such as manure application and proper seed spacing. From the project records and extrapolation from the sample survey, it was established that 121,918 students and 1836 teachers from the 156 intervention schools have benefited from the school garden interventions. In addition, are views of progress reports that revealed that FFE held a total of four farmer school days to discuss food contributions for the fifth day meal, the day where USDA-provided commodities are not used. 7 The aims of these events were to motivate communities and other stakeholders to provide commodity contributions, and a portion of the contribution was then used for cooking demonstrations of meals to be tested by students for their preference: porridge with sugar, porridge without sugar, a boiled maize meal, boiled maize and beans meal. The evaluation team found all output indicators measuring this outcome had targets, hence enabling the team of evaluators to measure the achievements against the targets. As indicated in the indicator table below, the FFE indicates the highest output level of performance across all indicators. Of five indicators tracked, four indicators indicate over achievements and one indicator indicates perfect achievements. Table 9: Project performance levels for school feeding8 Indicator name Target Actual 7 Project Quarterly reports and Semi Annual progress report 8 Project Quarterly reports and Semi Annual progress report 21 Number of boy students receiving school meals (breakfast, snack, lunch) as a result of USDA assistance 45,110 61,842 Number of girl students receiving school meals (breakfast, snack, lunch) as a result of USDA assistance 52,164 60,076 Number of school staff receiving school meals (breakfast, snack, lunch) as a result of USDA assistance 1,675 1,836 Number of parents receiving school meals (breakfast, snack, lunch) as a result of USDA assistance 422 422 Total quantity of commodities (tons) provided for school meals as a result of USDA assistance 1320 2,265.63 Source: Project monitoring records and reports MGD 1.1.4 & 1.1.5: Increased skills and knowledge of teachers and school administrators Review of project reports showed that FFE trained teachers, school administrators and government officials in order to improve quality of literacy instruction in all intervention schools. In total 12 training sessions in educational methodologies were held to train 1,836 teachers, 184 school’s administrators and 24 government officials. Similarly, FFE conducted a literacy teaching training for 156 Head Teachers (31 female, 125 male) to strengthen capacity on supporting literacy teaching, leadership promotion activities, and improved support for library teachers. A Training of Trainers (ToT) was also facilitated for 27 district officials (9 female, 18 male) on mentoring to strengthen their professional teaching capacity9 . MGD: 1.3.1 Increased Economic and culture incentives or decreased disincentives The project summary report for April 2016 to September 2016 clearly indicated that FFE has supported 270 WE groups. These savings group provided 6,133 members (beneficiaries) with access to savings and the ability to take out loans to both increase household income and support small businesses. As of September 2016, the 270 Women Empowered groups had a collective savings of 639,073,630 Tanzania shillings (TSH) (approximately $292,081), had loaned out 723,820,856 TSH (approximately $330,814), and had a total of 46,330,991 TZS (approximately $21,175) in social funds (for emergencies, funerals, etc.)10. MGD 1.1.2: Better access to school supplies and materials The FFE project met its goals for this objective. In total FFE distributed 118,516 storybooks and 88 pre-primary level textbooks, along with maps, atlases, science equipment, toys and sports gear, dictionaries, calculators, Manila paper and other teaching materials to all 156 intervention schools. From these supplies and materials, a total of 1,160 teachers and 102,324, pupils benefitted from the established libraries.11 MGD 1.1.3: Improved literacy instructional materials 9 Project Quarterly reports and Semi Annual progress report 10Project Quarterly reports and Semi Annual progress report 11Project Quarterly reports and Semi Annual progress report 22 From project reports and monitoring records it was established that FFE project has introduced teachers’ book writing competitions to enhance book production and supplementary reading materials to improve literacy, and have introduced instructional materials across all intervention schools. PCI, in collaboration with the Children’s Book Partnership (CBP), prepared book writing competitions to encourage teachers to create and write their own supplementary reading materials. The top five books were selected as winners, and those five different teacher-created books were published, printed and distributed as supplementary books/material. All top winners were congratulated in a special recognition event prepared by PCI and CBP. Recognition events were also used by PCI to raise awareness on the importance of education among parents and community members who attended the respective events. In total, 156 schools received project￾produced books and supplementary reading materials. The project-produced books and supplementary reading materials benefited 1675 teacher and 106,091 students. Regarding the establishment of libraries, a majority of schools (95.5%) have a library and a total of 156 library teachers (102 male, 52 female) received training on library management. Schools with no library created space for libraries and began library operations. Based on the supportive supervision results, library system management has improved in the schools that were visited. Schools with no library space available used mobile libraries or corner library services instead. The display of teaching aids in classrooms and use of learning aids has also improved, resulting in gradual reduction of the number of illiterate pupils in lower classes, and making the teachers’ work simple. MDG 1.3.3 Improved school infrastructure Reports and records further revealed that to increase health and dietary practices, the project supported water, sanitation and hygiene activities through construction of improved ventilation pit latrines and rainwater harvesting systems at schools. To date, the project constructed a total of 36 latrines, 70 hand washing stations, and 20 rainwater harvest tanks (RWHTs) in the 40 selected schools12. MGD 1.3.5 Increased community understanding of benefits of education Review of project reports revealed that the FFE project had been engaged in raising awareness about the importance of education and child rights, the PCI used the Journey of Life (JoL) methodology to spread the messages in public meetings, school committee trainings, meetings with parent forums and with community leaders. In total, PCI held 3,445 forums and events to raise awareness on the importance of education. The community awareness forums and events benefited 156, parent-teacher associations, 35,178 people, and 11,786 girl students13. MGD 2.1 Improved knowledge of health and hygiene practices Review of project reports (Narrative and Semi Annual Progress Reports) showed that the project promoted health and dietary practices through school health clubs. To ensure that the health clubs continued to be active, PCI monitored the clubs at all 156 schools and held a school health screening exercise at 151 schools to strengthen the capacity of Head Teachers, health teachers, 12Semi Annual Progress Report 1313Project Quarterly reports and Semi Annual progress report 23 and community health workers on student health screenings, treating any minor cases, filling out national health cards, and referring any other cases to local health facilities. The overall effort reached 17,753 students. In additional to that, FFE coordinated health stakeholder meetings to enhance relationships among various players at district, ward, village and school levels to improve school health program initiatives, discuss challenges and recommend solutions.14 MGD 2.2 Increased knowledge of safe food preparation and storage practices To improve knowledge of safe food preparation and storage practices, PCI conducted a refresher training course to all 156 project schools. The training involved all Head Teachers, store teachers and WECs. During the training the participants had a chance to share their experiences in managing school feeding activities. A total of six training sessions were held on food preparation and storage practices. During these sessions, 2,288 school staff benefited from food preparation and storage practices training.15 MGD 1.4.4. & 2.7.4 Increased engagement of local organizations and community groups Review of documented project sustainability updates indicated that PCI continued to work in collaboration with stakeholders from various levels (village, school, ward, division, district and regional) to strengthen local capacity and continue handing over responsibilities to promote sustainability within these activities. Various trainings were held with local leaders and stakeholders to strengthen their capacity in school feeding, health, agriculture, education, and Women Empowered saving and lending group activities. PCI took a lead role in the following activities to strengthen local capacity and ensure program sustainability: 16 • Trainings to strengthen local capacity: PCI facilitated trainings for various government officials to strengthen their capacity in leading trainings, providing support visits, mobilizing communities, and monitoring activities following the project close out. • WE saving and lending groups: A total of 270 Women Empowered groups supported by FFE program, which provides 6,133 members with access to savings and the ability to take out loans to increase household income and support small businesses. • School feeding learning exchange visit: PCI participated in a learning exchange visit with World Food Program (WFP) in the Ikungi District schools to compare the Home Grown School Feeding (HGSF) program. • Sustainability workshops with district and regional officials: To continue ensuring close collaboration and to build program ownership among government officials, PCI facilitated a second round of sustainability workshops at each of the District Councils with district heads of departments and followed by a debrief meeting with regional heads of departments. QN: What chief factors were responsible for the achievement or failure of the objectives? The evaluation teams measured effectiveness on the basis of progress towards planned results 1414Project Quarterly reports and Semi Annual progress report 15Semi Annual progress Report and Quarterly report 16project sustainability updates Report 24 and key factors facilitating or hindering progress. Several factors were identified as major facilitators of the FFE project successes, namely the success of school meal, training teachers on best teaching and learning materials, training school administrators, building/rehabilitation of latrines, awareness raising on the important of education, and access to and use of the Supplementary Reading Materials in the classroom. Other factors that facilitated the project results included: teachers’ enthusiasm, engagement of the community in school interventions, teacher’s recognition events, presence of libraries in all targeted schools and school gardens. Similarly, the existence of strong relationship between farmer group members/parents and schools/students also facilitated the project success. The findings from KIIs with stakeholders and students indicate that in general all intervention schools found to have improved performance compared to the baseline survey results. The EGRA results indicate an improvement in literacy of school age children for boys (64%) and girls (67%) when compared to the baseline data. Similarly, percent of students in classroom identified as inattentive by their teachers was also found to have over/outperformed the targets. In addition, the analysis shows that 100% of FFE targets were achieved related to an increase in trained teachers, built/rehabilitated latrines, and number of students benefiting from establishing or strengthening school gardens, as per results framework objectives.. However, there were delays in completion of latrines that PCI had to request for no cost extension. The findings indicated that government authorities at all levels contributed toward much of the project performance. For instance, PCI has been including district council engineers in oversight for latrine rehabilitation, and district councils have also made some efforts to educate people on agricultural approaches to improve production to assist school feeding program. Based on the KIIs and FGDS results, the government authorities perceived PCI’s FFE project as having important contributions to raising the literacy level in their respective districts. 3.4 Efficiency This section details the ability of the FFE project to deliver the expected results on time in terms of its objectives. It also details the chief factors responsible for the achievement or failure of the objectives within the stated timeframe. It presents the results of the evaluation and produces a comparison with the baseline survey where possible. QN: To what extent were objectives achieved on time? QN: What chief factors were responsible for the achievement or failure of the objectives on time? PCI delivered its expected objectives within the specified timeframe successfully. However, the delivery of some of the outputs and activities were delayed for various reasons e.g. the latrine construction was not completed on time due to various reasons such as the presence of rocks, which reduced the pace of pit latrines excavation in some of the schools, performance of contractors and extended rainy seasons. To accomplish these deliverables, the project asked for a no cost extension to finalize remaining latrines and rain water harvesting tanks. Overall, the project activities were found to have been completed in an efficient manner and within the agreed budget. 25 3.5 Sustainability QN: What is the likelihood of the activities of the project to continue beyond the project life span? The findings indicate that some of the project activities will be sustained following the end of the project activities. Positive indication of project sustainability was found in terms of improved skills and commitment levels of teachers, Head Teachers, school committees and agriculture teachers. Some of the project outputs that are most likely to continue beyond the project life span include: good health/hygiene practices, school water and sanitation provision, library services, teaching and learning materials, VLSA activities, school management committee activities, and school demonstration gardens. Financial sustainability was found to be a major concern in most of the schools surveyed. The majority of the schools remain largely dependent on PCI interventions for provision of food. Additionally, the inputs PCI provides farmers in exchange for feeding pupils has not yielded the expected results due to climate change17. Unreliable rainfall has caused a progressive decline in crop yields, which affects the ability of farmers to contribute adequate amounts of food. Analysis of data from multiple respondents also suggests it is difficult for school feeding programs to continue following the end of the project. However, review of other similar interventions including a PCI intervention in the Kilimanjaro region showed that communities have been taking responsibility for school feeding for years. Thus, the failure of communities to take over responsibility for feeding their students in the evaluated project could be ascribed to the local government (village, district, regional secretariat) leadership in Mara Region rather than to project management. QN: To what extent were project efforts to promote sustainability successful (e.g., evidence of local Ownership, sustainable partnerships and collaborations, etc.) were achieved? (Assess the status of the sustainability plans FFE put in place) Although there was no written exit strategy or sustainability plan in place for the FFE II project, PCI has made efforts in collaboration with stakeholders from schools, communities and at the local authority levels to underscore the need for sustained daily school feeding. School collaboration with farmers’ groups/other entities to provide food or in kind commodities, coordination and communication between the different stakeholder’s e.g. community members, schools, municipal government and agencies and community engagement in school activities are some of the efforts made by FFE to promote sustainability. To promote sustainability, PCI also conducted joint sustainability workshops for Musoma, Bunda and Butiama districts during the second half of 2015 and PCI represented an important occasion for all the stakeholders to gather 17PCI FFE II Midterm Evaluation Information 26 and reflect on the sustainability of project activities. District officials, PCI staff and other relevant stakeholders discussed on possible strategies to secure sustainability of the interventions. Sustainability meetings called School Graduation and Sustainability Readiness (SGSR) were also held at the cluster level for all schools’ WECs, Head Teachers, and store teachers. Because the sustainability plan was still in progress during the end-line survey, the status of the sustainability plans the project put in place was not assessed. However, the following are the initiatives undertaken by PCI to promote project sustainability: • Facilitated trainings for various government officials to strengthen their capacity in leading trainings, providing support visits, mobilizing communities, and monitoring project activities. • Supported WE savings and lending groups to increase household income and support small businesses. • Conducted School feeding learning exchange visit with WFP to initiate the Home Grown School Feeding (HGSF) program. • PCI trained Ward Executive Officers and other district officials on how to use school Graduation and Sustainability Readiness (SGSR) tool. QN: What measures could have been taken to make these efforts more effective? (Recommendations for future plans, i.e. FFE III) a. The FFE III program needs to integrate strategies to enhance community resilience to the impact of climate change, specifically to enhance capability of farmers to contribute food to the feeding program. Practical measure includes: promotion of climate smart agricultural practices; diversification of enterprise especially incorporating improved beekeeping, fish farming and poultry. b. Develop sustainability plans with the communities and government officials that will ensure the continuity of the school feeding project intervention beyond the project life. c. The FFE III program needs to assess leadership capacity at various levels of the local government (village, district and regional secretariat) and integrate strategies to build leadership capacities. This is likely to enhance sustainability of the feeding program,. QN: To what extent were the sustainability milestones achieved? PCI has integrated community based structures and systems to enhance project sustainability. Good examples include: income generating activities, agricultural groups, coordination of sustainability meetings, and formulation of school committees which can support FFE project intervention at the respectively. The process to develop sustainability plan has already started. The project organized sustainability workshops where all key stakeholders agreed on their current and future roles and responsibilities. These will be finalized in a formal sustainability plan. 27 5. PROGRESS ON IMPLEMENTATION OF MID-LINE EVALUATION Several recommendations were made at the mid-line evaluation. The PCI-FFE II project was keen to implement the recommendations and the project developed an action plan that provides a framework for monitoring progress. The end-line evaluation team found significant progress on the implementation of recommendations from the mid-line evaluation18 (Figure 2; Appendix 4). Figure 2: Status of implementation of the mid-line evaluation of PCI FFE II project Of the 16recommendations highlighted in the mid-line evaluation report, 86% were fully implemented, and the remaining recommendations are in progress. This very encouraging pattern that suggests that the PCI project team is dedicated to effective project implementation. 6. CONCLUSIONS AND RECOMMENDATIONS 6.1 Conclusions The evaluation team concludes that the PCI FFE II project approach has been effective in achieving the intended project outcomes in accordance to the domains of change specified in the results framework, including improved quality of literacy instruction, improved attentiveness and improved student attendance. Interventions integrated all key ingredients of the primary education system and tackled many key causes of the problem. Although the project had no impacts, it recorded increases in several impact indicators. Nevertheless, before and after comparisons show tremendous improvement in all variables assessed. The most likely reason for the observed obscured net treatment effect is the fact that the comparison schools were contaminated as a result of influx of school interventions in Serengeti district that was earmarked during the baseline as the most appropriate comparison area. 6.2 Recommendations 1. Proper planning for project sustainability between PCI and government authorities should be maintained to ensure that gains are consolidated and continuity is maintained. 2. PCI should promote a monitoring and evaluation system that focuses on fewer sources 18 Implementation of the midline evaluation plan. Completed 86% In progress 14% 0% 28 of information and ensure better analysis and better dissemination of the information collected 3. In addition to developing a sustainability plan with key stakeholders as an immediate action, integration of adequate community mobilization in the subsequent projects needs to be taken on board in order to enhance sustainability. 4. To increase community awareness on overall progress of the project, village leaders need to be supported to ensure that monitoring data from different portfolios of the school feeding project are properly communicated to different social groups in each village including village leaders, men, women, youth and elders. 5. In the future, EGRA assessment should be designed in a way that permits direct linking of parents, household characteristics and other potential covariates to the students assessed. 6. To avoid the risk of duplication of interventions in the same geographical area PCI should work with the Regional Secretariat to develop and mainstream a project coordination mechanism. This task fits very well in the on-going preparation of the exit strategy for the PCI FFE II Program. 7. As PCI finalizes its sustainability strategy it should consider to expand in other districts that are underserved. 8. Influx of similar projects in the project area should be seen as an opportunity rather than threat. In this case, PCI needs to focus on strategic partnerships with clear division of labor based on comparative advantage for each partner. This arrangement can help to avoid overlaps in specific components of interventions, which makes easy attribution of impacts. 7. LESSONS LEARNED 1. Women’s savings and loan groups (WE Groups) are effective in empowering women thereby enhancing their ability to actively engage with schools to support their school children and for contributing food to feed the school children. 2. Climate change, unclear policy declarations regarding ‘free’ education and inadequate community mobilization threaten effective community engagement that is necessary for sustainability of the project. 3. The project life span of three years was too short to sustain the interventions beyond the project; as well as the fact that was no sustainability integrated within the project framework in the original design. 4. Parents needed to be informed on how much maize and beans students consumed for school meals each year. Once the rations breakdown was explained, it is a relatively easy choice for parents to decide on contributing towards school meals. 5. Proper documentation and feedback on income and expenditure for school feeding program increases transparency and trust between the school/teachers, school committee and parents. 6. It is possible to train school heath teachers in order to continue to assess the basic health and nutritional status of the children at school. 7. District and Regional officials’ involvement in school feeding activities plays an influential role and is of great importance for project sustainability. 29 REFERENCES Bamberger, M. (2006) Conducting Quality Impact Evaluations under Budget, Time and Data Constraints.IEG: World Bank, Washington D.C. [http://documents.worldbank.org/curated/en/255551468178129539/pdf/382670EN GLISH01evaluations01PUBLIC1.pdf].Downloaded April 27, 2017. LaLonde, R.J. (1986). Evaluating the econometric evaluations of training programs with experimental data. The American economic review 604 - 620. Lechner, M. (2011).The estimation of causal effects by difference-in-difference methods. Foundations and Trends in Econometrics4(3): 165 - 224. Pischke, J.S. (2005). Empirical Methods in Applied Economics: Lecture Notes [http://econ.lse.ac.uk/staff/spischke/ec524/evaluation3.pdf].Downloaded April 27, 2017. Ravallion, M. (2001). The mystery of the vanishing benefits: An introduction to impact evaluation. The World Bank economic review15(1): 115 - 140. RTI International (2016). Assistance to Basic Education: All Children Reading (ABE-ACR). Final findings report, Tanzania National; Early Grade Reading Assessment (EGRA). USAID/Tanzania Education Office, Dar es Salaam, Tanzania Washington DC: World Bank. [https://mpra.ub.uni￾muenchen.de/1111/1/MPRA_paper_1111.pdf].Downloaded April27, 2017. White, H. (2006) Impact Evaluation Experience of the Independent Evaluation Group of the World Bank, 30 APPENDICES Appendix 1: Results Framework for the PCI-FFE II Project 31 Appendix 2: Detailed information for schools sampled and respondents a) List of sampled schools Treatment group Comparison group District Name of school Number of students sampled District Name of school Number of students sampled Bunda DC Nyabulundu 20 Serengeti Mapinduzi B 20 Bunda DC Marambeka 20 Serengeti Morotonga 20 Bunda DC Kenkombyo 20 Serengeti Mugumu 20 Bunda DC Buzimbwe 20 Serengeti Tumaini 20 Bunda DC Mariwanda A 20 Serengeti Nichoka 20 Bunda DC Masahunga 20 Serengeti Nyamoko 20 Bunda DC Kasahunga 20 Serengeti Rung'abure 20 Bunda DC Kisorya 20 Serengeti Amani 20 Bunda DC Mchigondo 20 Serengeti Kisangura 20 Bunda DC Kiroreli 20 Serengeti Bwitengi 20 Total 200 Bunda TC Nyabehu 20 Bunda TC Nyantare 20 Butiama DC Kihuzu 20 Butiama DC Kirumi 20 Butiama DC Bumangi 20 Butiama DC Masurura 20 Musoma DC Mugane 20 Musoma DC Buraga 20 Musoma DC Chirorwe 20 Musoma DC Kwikerege 20 Musoma DC Karubugu 20 Musoma DC Butata A 20 Musoma DC Rusoli A 20 Musoma DC Chimati B 20 Musoma DC Bwasi A 20 Musoma DC Nyasaungu 20 Musoma DC Kiriba A 20 Musoma DC Bwenda A 20 Musoma DC Kambarage 20 Musoma DC Wanyere B 20 Total 600 32 b) Students sampled by grade Status of school District Number of schools Number of students sampled Grade2 Grade4 All grades Girls Boys Total Girls Boys Total Girls Boys Total Comparison Serengeti 10 49 51 100 50 50 100 99 101 200 Total for comparison schools 10 49 51 100 50 50 100 99 101 200 Treatment BundaDC 10 50 50 100 50 50 100 100 100 200 BundaTC 2 9 11 20 10 10 20 19 21 40 ButiamaDC 4 20 20 40 20 20 40 40 40 80 MusomaDC 14 73 67 140 70 70 140 143 137 280 Total for treatment schools 30 152 148 300 150 150 300 302 298 600 Total for all schools 40 201 199 400 200 200 400 401 399 800 c) Sample for other categories of respondents Status of school District No. of schools Number of teachers sampled Number of head teachers and other administrators sampled Number of parents sampled Male Female Total Male Female Total Male Female Total Comparison Serengeti 10 8 12 20 10 0 10 34 26 60 Total for comparison schools 10 8 12 20 10 - 10 34 26 60 Treatment BundaDC 10 8 12 20 10 0 10 33 25 58 BundaTC 2 1 3 4 2 0 2 6 6 12 ButiamaDC 4 2 6 8 3 1 4 13 10 23 MusomaDC 14 15 13 28 13 1 14 49 36 85 Total for treatment schools 30 26 34 60 28 2 30 101 77 178 Total for all schools 40 34 46 80 38 2 40 135 103 238 33 Appendix 3: Performance indicator tracking for PCI-FFE II end-line evaluation Performance Indicator Baseline Target Actual data Source of information for the actual data Number of social assistance beneficiaries participating in productive safety nets as a result of USDA assistance 73,000 106,449 133,082 Project reports Number of people trained in child health and nutrition as a result of USDA assistance 0 320 800 Project reports Number of public-private partnerships formed as a result of USDA assistance 0 26 31 Project reports Numbers of child health and nutrition policies, regulations and/or administrative procedures in each of the following stages of development as a result of USDA assistance 0 2 1 Project reports Percent of boys in target schools who, by the end of two years of school demonstrate reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools 12 40 12 End-line evaluation data Percent of girls in target schools who, by the end of two years of school demonstrate reading comprehension equivalent to their grade level as defined by national standards at USDA supported schools 9 40 9 End-line evaluation data Number of total individuals benefitting directly from USDA-funded interventions 73,000 106,449 133,082 Project reports Number of total individuals benefitting indirectly from USDA-funded interventions 288,000 360,040 532,328 Project reports Percent of teachers in target schools who demonstrate improved literacy instruction as identified by supervisors, mentors, or coaches 0 75 10 Project reports Number of students benefiting from the provision of school supplies 0 88,450 122,859 Project reports Number of teachers benefiting from the provision of school supplies 0 1,148 1,851 Project reports 34 Performance Indicator Baseline Target Actual data Source of information for the actual data Percent of teachers in target schools who demonstrate use of new and quality teaching techniques or tools 10 75 10 Project reports Percentage of teachers that allow students to ask questions during class 17 35 17 Project reports Percentage of teachers that allow students to respond to questions raised in class by other students 12 30 12 Project reports Percent of school administrators in targeted schools who demonstrate use of new techniques or tools 23 75 23 Project reports Percent of students in classrooms identified as inattentive by their teachers 14 10 14 End-line evaluation data Percent of students in target schools who indicate that they are "hungry" during the school day 45 25 45 Project reports Percent of students in target schools consuming daily meals at school 0 100 100 Project reports Percent of boy students regularly (80%) attending USDA supported classrooms/schools 0 90 79 End-line evaluation data Percent of girl students regularly (80%) attending USDA supported classrooms/schools 0 90 80 End-line evaluation data Percent of students who report a decrease in health-related absences 25 35 25 End-line evaluation data Average number of days per student that they were absent from school the previous month due to illness 2.28 2 2.28 End-line evaluation data Percent of schools in target communities that store food off the ground 51 100 100 Project reports Percent of students in target schools who achieve a passing score on a test of good health and hygiene practices 51 90 51 Project reports Percent of food preparers at target schools who achieve a passing score on a test of safe food preparation and storage 0 90 100 Project reports Average ratio of male students to latrines at target schools 0 65 70 Project reports Average ratio of female students to latrines at target schools 0 65 56 Project reports 35 Performance Indicator Baseline Target Actual data Source of information for the actual data Number of students benefiting from the building/rehabilitating of latrines 0 27000 22924 Project reports Number of students benefiting from the building/rehabilitating of wells and water stations/systems 0 25,000 14080 Project reports Number of SWASH committees formed / strengthened 103 134 Project reports Percent of target schools with improved food prep and storage equipment 77 100 Project reports Percent of parents in target communities who can name at least 3 benefits of primary education 85 156 85 Project reports Percent of Volunteer Student Aides with increased self-confidence 0 100 33 Project reports Number of parent-teacher associations contributing to their school as a result of USDA assistance 103 134 156 Project reports Number of events promoting advocacy on child protection and safety in target schools and communities 0 2,712 3581 Project reports 36 Appendix 3: Progress on implementation of recommendations from PCI-FFE II project mid-line evaluation Recommendation Actions Status of implementation and remarks 1. A strategy to promote reading practice at school, and/or community level: Storybooks distribution and trainings on library management should go hand in hand with training of teachers on use of supplementary reading materials in classroom; reading promotion strategies at school level such as a reading fair, inter-school reading competition, or prizes for the pupil reading more books in one school year; administering inter￾school storybooks lending in collaboration with Child Book Project (CBP) and under supervision of the Ward Education Coordinator to avoid reading same storybooks in one school; and promote exchange visits of poor performing schools visiting good performing schools. PCI was positive about the recommendation and gave it a top priority. To address this recommendation, PCI has done the following activities: • PCI distributed different categories of books to all 156 schools: 18,720 copies of storybooks which were a result of CBP’s teachers book writing competition; and 4,680 big books and 15,600 small story books • Organized one book writing completion event in collaboration with community members, local leaders and district officials. Recognition of 5 winners (teachers) is a motivation for teachers to keep writing and creating more books. The regular competition is planned annually • Refresher training for 156 library teachers • Refresher training to 156 Std 1 & 2 teachers • Reading competition planned and implemented annually at the ward level; schools are encouraged to conduct school level reading competition, of which 42 schools in Bunda and Butiama districts In progress Fully completed 37 Recommendation Actions Status of implementation and remarks have adopted reading competitions at classroom level 2. Rotating libraries system Developing a rotating libraries system operated at the ward level under WEC. It was suggested that each school be provided with different titles which can then rotate to another school at the end of a given period so students do not lose interests in reading as they run out of new titles. This links to the first recommendation; it has not been addressed as all schools have continued to receive the same book titles and hence have not required rotation. Not completed As recommendation 1 3. Strategy to avoid overlap with the emerging similar programmes in the area. A strategy should be agreed upon with district authorities and the UKAID – funded Education Quality Improvement Programme (EQUIP) in Tanzania to avoid overlapping of intervention areas between PCI and EQUIP in Mara region where the geographical coverage of the two organisations coincides. PCI conducted three initial meetings with EQUIP but has not had further development following transfer of the EQUIP Coordinator and other EQUIP staff in Mara region. EQUIP director in Dar. Discussions made with the RC and DC of Musoma and plan to meet the other DC’s and all the DEOs to work out a formal mechanism to minimize overlap among organizations while at the same time enhancing synergies. In progress 4. Strategy to ensure effectiveness and relevance of trainings PCI to work with local authorities to agree on PCI addressed the recommendation through the following activities: • Meetings and written agreement with 38 Recommendation Actions Status of implementation and remarks criteria for selection of relevant trainees for different trainings or sets of trainings to ensure that specific topics are administered to relevant trainees. This is to go hand in hand with effective training monitoring system with a data chain beginning from school and village administrations via community mobilizers, Ward Education Coordinator, District Education Officer. District Education Department, WECs; down to Head Teachers to ensure that teachers are assigned roles based on their abilities and capacities, and Head Teachers always send same teachers to attend trainings offered in consecutive events. • The project has created a database to keep track of teachers trained and other relevant training records. • The project shares records of trained teachers with respective DEO, WEC and school administration to ensure effective supervision of the trained teachers and tracking the extent the trained teachers apply skills and knowledge gained. • The project has taken a proactive step of sending back teachers who are not appropriate to a particular training. Fully completed 5. Fostering partnership and collaboration with CBP and other relevant projects/organizations This is linked to recommendations 1 and 3; PCI was encouraged to create partnership with CBP in order to avoid duplication of efforts, enhance synergies and cross-fertilization; and ensure PCI and CBP have initiated regular mutual exchange visits in order to learn from each other; PCI has already visited CBP-supported schools in Dar es Salaam and Dodoma regions. Good practices from CBP were documented and will be adopted in the USDA-supported schools in Mara region. PCI and CBP have now planned a joint refresher teachers training library teachers, Standard I and II teachers. The planned training has a Fully completed 39 Recommendation Actions Status of implementation and remarks effectiveness. particular focus on improving teacher’s ability to use libraries, story books, teaching aids, and other materials previously distributed to schools to promote the 3Rs including talking walls with pictures/ words/ letters, reading clubs, buddy reading, reading competitions. 6. Linking with other active projects in training teachers on early reading such as TZ21 to explore possibilities to acquire materials to supplement face-to-face trainings offered to teachers Explore the possibility of using materials developed by other projects active in the country (i.e. TZ21 Content for teachers training on early reading) to supplement face-to-face trainings with teachers. These materials could be provided during trainings and used by teachers when at school as a reference in case of doubts or if further training is needed. PCI implemented the recommendation as follows: • Linked with District Authorities and EQUIP-Tanzania to acquire current modules being used. • Plans to assess materials from TZ21 and other projects focused on early grade reading are underway. • PCI also has been working with CBP to encourage the development of local teaching materials among teachers in project schools which are shared as reference materials for teachers. • PCI organized a cross-visit to Right to Play organization project in Serengeti region to explore availability of relevant materials for teachers supplementary references under FFE II program • PCI met USAID Education Team Leader and the Director of Tusome Pamoja and Fully completed 40 Recommendation Actions Status of implementation and remarks agreed that PCI will help with printing and dissemination of the new pre-primary supplementary readers 7. Working with Head Teachers and WECs to establish and mainstream a system for coaches and mentors to support teachers to adopt new teaching materials and practices in classrooms Instead of one-off training, a system of technical persons including Head Teachers and WECs to whom teachers make consultation whenever they need any support regarding new concepts, and/or techniques. The PCI gave this a top priority similar to recommendation 1. Progress towards implementation is as follows: • PCI organised an exchange visits to other CBP-supported schools in Dodoma, Dar es Salaam and Shinyanga District Councils where the teacher mentoring system is being used • PCI in collaboration with the District authorities has established criteria and guidelines for selecting teacher mentors at each school; in total 156 Mentors have been identified and trained as ToT • PCI is working with the District authorities to establish a monitoring system to monitor and support performance of the mentoring system. Fully completed 8. Enhancing capacity of the school feeding committees and ensuring their effective supervision to guarantee the correct preparation and distribution of food to pupils PCI consider the recommendation as of great importance with respect to sustainability of the FFE II programme. The recommendation has been addressed as follows: • PCI organized a sustainability workshop Fully completed 41 Recommendation Actions Status of implementation and remarks Capacitating members of the committee on their role and importance to enhance their adherence to standards and norms regarding school feeding and rationing where roles for various stakeholders were discussed and agreed upon. • PCI in collaboration with teachers are closely monitoring and supervising preparation and distribution of food to pupils to ensure adherence to norms and standards. • PCI is working with Head Teachers and government officers at the district and ward levels to build their capacity to oversee the school feeding committees beyond project life span. 9. Working with health officers and dispensaries from the village, ward and district levels to raise awareness on healthy nutrition issues for infants and adults The purpose is to encourage intake of vitamins and minerals, which were very low during the mid-line evaluation. The typical ways to increase intake of vitamins and minerals is through promotion of consumption of fruits and vegetables in the short￾term. In a longer term, schools should be encouraged to grow fruit trees to boost intake of fruits among school children and helping schools to Progress on implementation: • Besides the well-known school vegetable gardening, PCI has already supported planting of fruit trees to 9 schools and this will expand in all schools. • District and ward health officers have been trained on how to conduct school health screening. • PCI has integrated health modules in training materials for School Health Clubs; they are now encouraging clubs to act out skits or sing songs promoting healthy nutrition, and health and hygiene practices in different community events. Fully completed 42 Recommendation Actions Status of implementation and remarks become agents of change in the communities. • PCI in collaboration with the District authorities is finalizing a guide for VSAs that integrates nutrition topics; these are supported to link with school health clubs and surrounding communities to promote behaviour change through peer education • PCI will also continue to promote vegetable gardens at all schools to teach students about traditional, nutritional vegetables. Additionally, PCI will continue to expand the tree planting pilot program, which includes fruit trees (avocado, mango, papaya, guava, and orange) to more schools in the next rainy season. • PCI has sensitized and supported WE groups to become promoters of healthy nutritional practices in their communities. 10.Conduct awareness activities for kids on food intake and good nutrition practices in order to develop a culture of healthy eating among school population This corresponds to recommendation 9 stated above As above Full completed 43 Recommendation Actions Status of implementation and remarks 11.Work with universities or other stakeholders to develop an awareness raising campaign at regional and national levels on the importance of school feeding practices in order to raise attendance and education performance The purpose is to communicate the importance of school feeding for healthy growth, better education outcomes from household, village, district; regional and even national levels. PCI considers this recommendation as an important component toward sustainability of school feeding intervention. Progress towards implementation of the recommendation: • In line with the current focus of the GoT on promoting the 3Rs in early grades PCI has already integrated the school feeding in the sustainability plan being prepared; stakeholders are in the process of agreeing on their roles and responsibilities to ensure continuation of the school feeding intervention beyond the project life • Worked with District Education Officers and Ward Education Coordinators to build their capacity for promotion and monitoring of school feeding interventions beyond the project life Full completed 12.Continue exploring the possibility of using Women Empowered (WE) groups as change agents in the communities. The idea is to capitalize on the strength and build capacity of the WE-GROW groups to effectively use them for awareness creation for different relevant social issues including integration of PCI fully supports this recommendation based on experiences with WE groups already acting as change agents in their communities. Thus PCI implemented the recommendation as follows: • PCI is working expand the number of farmer groups and linking them with WE groups and involve them in donating food to FFE-supported schools in return. To this Fully completed 44 Recommendation Actions Status of implementation and remarks Zinduka activities instead of exclusive focus on malaria and HIV. WE- as a link between schools and communities can be used as a hub for sustainability of project activities. effect, 13 WE groups have been added to the Farmer Groups • WE- groups are being supported to utilize their weekly social discussions to promote good health and hygiene practices, proper nutrition, and how to combat cholera, malaria, HIV/AIDS, etc. in communities. 13.Providing relevant advanced training to community mobilizers as per their sector of operation This is recommendation intends to build on the strength identified with respect to the fact that the community mobilizers were divided per sector. The recommendation intends to enhance effectiveness of the mobilizers hence elevating them to “advisors”. For example, those under education sectors, once appropriately trained, can monitor and advise Head Teachers and teachers on classroom management, use of materials and library practices. Mobilizers fall under health and education sectors; PCI has implemented the recommendation as follows: • Provided sector specific trainings to all community. • Monitoring roles have also been added to the responsibilities of mobilizers under their respective sectors • Plans are underway to link the mobilizers to ward officials to ensure collaboration among them to ensure sustainability; thus education community mobilizers are linked to WECs. • Besides sector specific trainings, community mobilizers and WECs will be trained in coaching and mentoring skills to ensure their effectiveness in performing their roles as advisors. Fully completed 45 Recommendation Actions Status of implementation and remarks 14.Exploring the possibility of using mobile phone data collection system for communication to and from schools The system was in place at the time of mid-line evaluation and it allows for mixed-mode data gathering (data, pictures and videos). Thus the point for this recommendation was to promote the systems as the major means for the schools to communicate difficulties encountered to PCI staff and requesting support. PCI was more than prepared to address this recommendation provided it has been using the mobile phone data collection program, Mobenzi, for school feeding activities and reporting: daily attendance and meal distribution, monthly enrolment figures, and monthly food stocks); “red flag” survey for any problems in regards to school feeding activities, which teachers can send into the system at any time for PCI to view. Thus the new response of PCI to address the recommendation is as follows: • Conducting Mobenzi refresher training to teachers to integrate more information in the existing reporting system. • The education sector has also developed checklist and integrated it in the “Mobenzi” where schools submit the required data on monthly. Fully completed 46 Recommendation Actions Status of implementation and remarks 15.Promoting local school exchange program The essence is Head Teachers and SMC chairpersons from poor performing schools visit high-performing schools to learn on how to perform better. The idea is to avoid the risk of each school having to pass through trial and error stage prior hitting to their breakthroughs. Progress on the implementation of the recommendation: • PCI is documenting and sharing success stories and lessons learned • Exchange visits to take poor performing schools to the best ones are underway On progress Good and poor performing schools already identified; exchanges visits not yet started 16.Regular contacts of School Management Committees with the District Authorities are commendable as it capacitate the SMCs in the execution of their functions. The essence of the recommendation is to encourage formalization of meetings and contacts between District Authorities and SMCs instead of the ad-hoc meetings between the two entities observed during the Mid-line evaluation PCI is supporting SMCs to comply with the new Tanzania education policies including strict regulations on school budget management for government funds. PCI is working with district, ward and village officials to ensure proper management of other school funds including contributions to school feeding. 840 School Committee members were trained in budget tracking, procurement, supervision of school funds and record keeping. Training themes included roles and responsibilities of school committees. Fully completed