EVALUATION Mid-Term Performance Evaluation for the Tusome Pamoja (Let’s Read Together) Activity in Tanzania November 2018 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by ME&A, Inc. Mid-Term Performance Evaluation of the Tusome Pamoja (Let’s Read Together) Activity This publication was produced at the request of the United States Agency for International Development Developed under Contract/Task Order Number: Contract No. AID-OAA-1-15-00024/AID-621-TO-17- 00005 It was prepared independently by: Megan Gavin, Ph.D., Team Leader Michelle Davis, Qualitative Analyst Immakulata Komba, Technical Specialist Brenda Pearson, Chief of Party Jacob Laden, Evaluation Advisor, Data for Development Godfrey Teli, Ph.D., Technical Specialist Gerald Usika, Technical Specialist Contractor: ME&A, Inc. USAID/Tanzania Data for Development (D4D) Project Plot 122 Mwai Kibaki Road, Mikocheni A, Dar es Salaam, Tanzania Email: bpearson@engl.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. ACKNOWLEDGEMENTS We want to acknowledge the support and collaboration of USAID staff, including but not limited to Christine Djondo, Education Office Director; Laura Kikuli, Tusome Pamoja Contracting Officer Representative; Kris Capella, Education Officer; and Monitoring, Evaluation, and Learning Point of Contact, Abbas Ansanzugwanko, who also participated in field work and accompanied us to Pemba. We also need to give special thanks to the efforts and information provided by RTI Tusome Pamoja Team, including but not limited to Alastair Rodd, Chief of Party; Carlton Aslett, Former Chief of Party (for Program Background and Design); Jana Scislowicz, Deputy Chief of Party; Juma Mwenga, Senior M&E Specialist; and Elizabeth (Bess) Long, Senior Reading Advisor, for their participation throughout the process, coordination, and support, and for providing information requested for the evaluation. ABSTRACT This is a mid-term evaluation of the Tusome Pamoja (TP) Activity, which utilized key informant interviews (KIIs), focus group discussions (FGDs), and brief quantitative surveys (QSs) to investigate: 1) progress towards improving reading, writing, and arithmetic [also referred to by the Government of Tanzania (GOT) as the “3Rs”]; 2) use of School Information System (SIS) to improve 3Rs; 3) internalization of capacity building and how to more effectively improve learning outcomes of the 3Rs; and 4) sustainability. The evaluation finds that quality reading materials have been designed and disseminated. The TP Activity and the GOT show commitment in working together to design and align materials. Capacity building activities have resulted in knowledge and attitude changes at the school and local government level, and some behavior change has occurred. Due to materials and training, gains appear to have been made in reading, teaching, and learning. Arithmetic has not yet been addressed because of funding limitations of the USAID Global Education Strategy at the time of this evaluation. Moving forward, SIS may support oversight of reading progress, but human and information and communications technology (ICT) challenges remain. Parents can also contribute to improving 3Rs and are motivated to help in schools; however, they require more training in order to understand how they can help their children master the 3Rs at home. CONTENTS EXECUTIVE SUMMARY................................................................................................................................................... i 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS.............................................................. 1 1.1 Evaluation Purpose .............................................................................................................................. 1 1.2 Evaluation Questions.......................................................................................................................... 1 2.0 PROJECT BACKGROUND............................................................................................................................. 2 3.0 EVALUATION METHODS AND LIMITATIONS...................................................................................... 3 3.1 Evaluation Methods............................................................................................................................. 3 3.2 Analysis................................................................................................................................................... 6 3.3 Limitations............................................................................................................................................. 6 3.4 Capacity Building and GOT Engagement During the TP Evaluation ....................................... 6 4.0 FINDINGS AND CONCLUSIONS ............................................................................................................... 7 4.1 EQ1: To what extent is the TP Activity making progress towards improving target beneficiaries 3Rs skills?....................................................................................................................... 1 4.2 EQ 2: How is the development of a SIS leading to improved learning for the 3Rs?........... 6 4.3 EQ 3: In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by Tusome Pamoja?................................................................................................................................................. 11 4.4 EQ4: How can TP’s activities be done moving foward in order to more efficiently improve learning outcomes for the 3Rs?........................................................................................................ 4 5.0 RECOMMENDATIONS.................................................................................................................................. 11 ANNEXES ......................................................................................................................................................................... 15 Annex I: Statement of Work ......................................................................................................................... 15 Annex II: Design................................................................................................................................................ 27 Annex III: Tools................................................................................................................................................. 51 Annex IV: Disclosure of any Conflicts of Interest.................................................................................... 76 Annex V: KIIs..................................................................................................................................................... 82 Annex VI: Systematic Literature Review.................................................................................................... 83 Annex VII: Consolidated Design ................................................................................................................... 93 LIST OF TABLES Table 1: FGDs and KIIs per Region............................................................................................................................... 5 Table 2: Stakeholder Population Served by Tusome Pamoja.................................................................................. 7 Table 3: EQ1 Findings and Conclusions....................................................................................................................... 5 Table 4: EQ2 Findings and Conclusions....................................................................................................................... 9 Table 5: Results to Date ................................................................................................................................................ 10 Table 6: EQ 3 Findings and Conclusions...................................................................................................................... 3 Table 7: DPLA Phase III - Percentage of Schools Meeting Student Performance Standards by Region and Percentage Gain ................................................................................................................................................ 5 Table 8: EQ4 Findings and Conclusions....................................................................................................................... 7 Table 9: EQ5 Findings and Conclusions..................................................................................................................... 11 Table 10: Recommendation 1....................................................................................................................................... 11 Table 11: Recommendation 2....................................................................................................................................... 12 Table 12: Recommendation 3....................................................................................................................................... 13 Table 13: Recommendation 4....................................................................................................................................... 13 Table 14: Recommendation 5....................................................................................................................................... 14 LIST OF FIGURES Figure 1: Results Framework .......................................................................................................................................... 3 Figure 2: Learners Reached with Reading Programs................................................................................................. 7 Figure 3: Books Provided with USG Assistance......................................................................................................... 1 Figure 4: Materials in schools during data collection ................................................................................................ 2 Figure 5: Oral Reading Fluency, 2016 ........................................................................................................................... 5 Figure 6: Comprehension 2016...................................................................................................................................... 5 Figure 7: Addition/Subtraction, 2016............................................................................................................................ 5 Figure 8: Missing Number, 2016 .................................................................................................................................... 5 Figure 9: Teachers Provided Professional Development on Reading Instruction............................................ 11 Figure 10: Education Administrators Provided with Professional Development............................................... 4 Figure 11: Materials Available in the Home (N=69).................................................................................................. 6 Figure 12: Challenges Mentioned by Participants....................................................................................................... 9 Figure 13: Recommendations from Participants to Ensure Sustainability .......................................................... 10 ACRONYMS Acronym Description 3Rs Reading, Writing, and Arithmetic ADS Automated Directives System BRN Big Results Now CDCS Country Development Cooperation Strategy CO Contracting Officer CoL Communities of Learning COP Chief of Party COR Contracting Officer Representative DCOP Deputy Chief of Party DEC Development Experience Clearinghouse DED District Executive Director DEO District Education Officer DFID Department for International Development DO Development Objective DP Development Partner DPLA Decentralized Periodic Learning Assessment EGRA Early Grade Reading Assessment EQ Evaluation Question EQUIP-T Education Quality Improvement Program Tanzania, DFID ET Evaluation Team FGD Focus Group Discussion FOI Fidelity of Information FY Fiscal Year GPE Global Partnership for Education GOT Government of Tanzania ICT Information and Communications Technology IP Implementing Partner IR Intermediate Result KII Key Informant Interview LANES Literacy and Numeracy Education Support LGA Local Government Authority LOE Level of Effort LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MEL Monitoring Evaluation and Learning MOEST Ministry of Education, Science, and Technology MOEVT Ministry of Education and Vocational Training NECTA National Examinations Council of Tanzania NGO Non-Governmental Organization NORC National Opinion Research Organization at the University of Chicago OCGS Office of the Chief Government Statistician PMP Performance Management Plan PORALG President’s Office Regional Administration and Local Government PTP Parent Teacher Partnership QA Quality Assurer viii Acronym Description REO Regional Education Officer RISE Research on Improving Systems of Education RTI Research Triangle Institute SEL Social Emotional Learning SOW Scope of Work STTA Short Term Technical Assistance SIDA Swedish International Development Corporation SIM Subscriber Identity Module SIS School Information System TA Technical Assistance TIE Tanzania Institute of Education ToT Training of Trainers TP Tusome Pamoja UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government WEO Ward Education Officer ZIE Zanzibar Institute of Education i EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of this performance evaluation is to analyze and take stock of the Tusome Pamoja (TP) Activity as it arrives in its third year of implementation. Specifically, the evaluation assesses the Activity through evidence-based findings triangulated across sources to produce 25 generalized conclusions and five actionable recommendations. The results are primarily for the United States Agency for International Development Tanzania (USAID/Tanzania) and RTI International (RTI) and may also be used by other stakeholders such as the Government of Tanzania (GOT) and other development partners (DPs) in country. PROJECT BACKGROUND The TP Activity is a five-year, $67 million activity awarded in December 2015 that started in January 2016. It is implemented by RTI and supports USAID/Tanzania’s Country Development Cooperation Strategy (CDCS), specifically Development Objective 1 (DO1): Tanzania’s advance toward middle-income status supported: Lifelong learning skills improved. The primary goal of TP is to improve lifelong learning skills through mastery of early grade reading, writing, and arithmetic (also referred to as the “3Rs”). TP’s activities are grouped into three results. This mid-term evaluation focused only on Results 1 and 2 as per the scope of work. These are: Quality of early grade basic skills instruction improved (Result 1), and Skills delivery and assessment systems of central, regional, and local government(s) strengthened (Result 2). See the full report for detailed project description. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS Evaluation Questions (EQs) • EQ1: To what extent is the TP Activity making progress towards improving target beneficiaries 3Rs skills? • EQ2: How is the development of a School Information System (SIS) leading to improved learning for the 3Rs? • EQ3: In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by TP? • EQ4: How can TP’s activities be done moving forward in order to more effectively improve learning outcomes for the 3Rs? • EQ5: What are the factors that have implications for sustainability of quality 3Rs instruction after TP has ended? Methods The Evaluation Team (ET) used a mixed-methods approach to ensure triangulation and, in turn, validity and reliability of findings. There are two primary instruments—the Key Informant Interview (KII) guide and Focus Group Discussion (FGD) protocol. The KII guide was used for Head Teachers (HTs), TP Activity staff/USAID, DPs, and the GOT, including the Ministry of Education, Science, and Technology (MOEST), President’s Office Regional Administration and Local Government (PORALG), and Ministry of Education and Vocational Training (MOEVT) in Zanzibar. The FGD protocol was used for teachers, parents, and Ward Education Officers (WEOs). In addition, the ET designed a brief survey, which included seven scale questions to be self-administered by parents as they were congregating for FGDs. These instruments can be found in Annex III. CAPACITY BUILDING DURING THE EVALUATION Data for Development promoted local capacity building during the TP evaluation. Collaboration and capacity building were built in from the beginning, with Implementing Partner (IP) staff and local USAID staff directly engaged in the evaluation design process, increasing knowledge of best practice in evaluation ii design. In addition, the evaluation itself included strong local participation, including three Tanzanian researchers, one local institution (Utafiti Associates), and one local Data for Development staff. Local team members participated in formal capacity building before the evaluation began by attending a two-day training on qualitative coding, data analysis, and use of Dedoose1 qualitative analysis software. Team members also received iterative, tailored capacity building during fieldwork, and were provided ongoing guidance from Data for Development and the team lead as they put the learning into practice and utilized Dedoose to code qualitative findings from the evaluation. FINDINGS AND CONCLUSIONS EQ1: To what extent is the TP Activity making progress towards improving target beneficiaries 3Rs skills? Materials: A large number of materials have been produced and disseminated; in some instances, they are the only resources available for reading instruction in the classrooms visited. Materials have quality content and visual aids and are durable. Teachers and authorities perceive them as a motivating factor for student attendance. • Conclusion 1. A large number of materials exist and are crucial to improve reading skills. • Conclusion 2. Materials are good quality in terms of design and content and they motivate children to come to school. Teaching Instructional Tool2 and Teacher’s Guides: The teaching instructional tool is a tool that TP developed; it accompanies the supplemental materials such as decodable readers that TP produced for all schools it serves. RTI is in the process of aligning the materials with the national text and curriculum. The teaching instructional tool assists teachers in teaching to very large classroom sizes, often over 100 students, in how to properly and easily use the decodables. • Conclusion 3. There is commitment by TP and interest by the GOT to work together to align TP with national needs and curriculum. Certification: Materials have been approved for use although they are currently in a process of validation and alignment. Standard 1 and 2 materials still need government certification by the Tanzania Institute of Education (TIE). • Conclusion 4. Some government entities such as TIE have been involved in the production of materials and they must continue to be involved. Reading and Writing (2Rs) + Arithmetic: Aligned with USAID’s strategy, the focus of TP has been on reading with some writing curriculum. The 2018 Decentralized Periodic Learning Assessment (DPLA) shows that between 2017 and 2018 there have been statistically significant improvements in some reading skills (lower order skills), while challenges remain with regard to higher order skills. The arithmetic component has not yet been implemented; however, there are opportunities to inform the national arithmetic curriculum in collaboration with other programs. • Conclusion 5. Gains appear to have been made with regard to teaching-learning in Reading/Writing but not yet in Arithmetic. • Conclusion 6. TP is a national example for Reading/Writing whereas other DPs [Department for 1 Dedoose is a free open source qualitative coding and analysis software. www.dedoose.com. 2 This report uses the term teaching instructional tool. In the case of Tanzania mainland, the term Teaching Guide is only used when it accompanies an official GOT textbook. The teaching instructional tool is the instructional tool that accompanies the supplemental material that TP provided. TP is currently assisting the TIE to edit the GOT’s official Teacher’s Guides which accompany the official GOT textbook. In the case of Zanzibar, the term teaching guide can be used to refer to the tool it gives to teachers to help them with teaching phonics. Given the sensitivity and room for confusion, the ET use the term teaching tool for both Tanzania mainland and Zanzibar when referring to the tool that is passed out to teachers with the supplemental materials and Teacher’s Guides when referring to the official guides that the GOT uses with their official textbook. iii International Development’s (DFID) Education Quality Improvement Program Tanzania (EQUIP T)] can be leveraged for Arithmetic. • Conclusions and technical recommendations to improve fidelity of implementation of the 3Rs using DPLA 2018 evidence can be found in EQ4. EQ2: How is the development of a SIS leading to improved learning for the 3Rs? Early Beginnings and Potential: As TP is still in the process of rolling out the SIS it is too early to tell how it will contribute to improved learning in the 3Rs. School administrators and local government authorities (LGAs) are hopeful about the SIS’ utility for management of schools and planning purposes. • Conclusion 7. SIS has the potential to provide information at the local level for decision-making and improving the 3Rs. Considerations/Potential Challenges: Potential challenges to the implementation of the SIS include access to the Internet and experience using tablet technology. Users asked for more training and support on use of the tablets. • Conclusion 8. There remain concerns in terms of Information and Communication Technology (ICT) access and human capacity, which require attention as SIS rolls out. National and Local Government Roles: It is not clear who on the local or regional level will maintain the SIS or who will provide ongoing training, maintenance, updates, etc. for its sustainability after TP ends. • Conclusion 9. There are at least two levels of support that require defining, strengthening, and aligning in order to effectively use, maintain, and sustain the SIS. EQ3: In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by TP? There is progress on using new teaching approaches such as participation and use of teaching aids, but less on using new teaching methods like phonics. Communities of Learning (CoLs) are considered useful school-level support for improving teaching and learning but their level of formality varies by school. Some HTs and local officials have doubts that the Training of Trainers (ToT) methodology results in teachers receiving all knowledge initially imparted by TP. • Conclusion 10. Attitude changes on how to teach the 3Rs signal possibilities of longer-term shifts in teaching, and initial use of some new approaches point towards teacher behavior change. • Conclusion 11. CoLs have a strong role in supporting ongoing internalization of approaches to teaching 3Rs and methods for teaching reading. • Conclusion 12. More support and tailored training are needed to adjust ideal approaches and methods to local realities. Head Teachers: HTs’ attitudes towards teaching have changed and they have used new approaches to oversee teaching and learning, including via classroom observation and facilitation of the CoL. Follow-up training would be useful to sustain and expand application of learning on leadership and CoL as well as skills-based training for use of the SIS. • Conclusion 13. HTs’ attitudes changed on oversight of reading, teaching, and learning. There is some HT behavior change in terms of follow-up and classroom oversight. • Conclusion 14. More training is needed on leadership skills. Further implementation of CoL and SIS offers opportunities to show leadership with regards to parent teacher engagement and performance management. Local Government: Local government shifted their approach to oversight of reading and feel more confident in their ability to track proper teaching of reading and provide support. While some local governments have taken strong ownership of the program, others have more limited interaction with TP. iv • Conclusion 15. Involvement of local government in implementation has resulted in initial shifts in capacity and approach to ensuring quality reading instruction at the local level. • Conclusion 16. There is variation at the local government level in terms of implementation of TP and ownership. Mtwara and Iringa show strongest fidelity of implementation. National Government: Some national level government actors are more involved than others. Stakeholders would like more joint planning from the beginning although, in real time, this may be difficult since there is very high government turnover and uncertainty in terms of government policy, ministry role, and individual staff tenure. • Conclusion 17. Government turnover means loss of knowledge and difficulty coordinating activities between TP and government officials. • Conclusion 18. There is emerging internalization of TP approaches at the national level. • Conclusion 19. There are opportunities to clarify roles in 3Rs’ sustainability. High-level officials believe sustainability lies at the district level but at the same time, mention possibilities of national level changes. EQ4: How can TP’s activities be done moving forward in order to more effectively improve learning outcomes for the 3Rs? DPLA 2018: Using results from the recent 2018 Phase III DPLA3 report the ET found that student performance remains a major stumbling block. This makes sense, as lower order skills first must be developed as the foundation on which to build other higher order skills. Detailed technical recommendations for improving phonics instruction can be found in the body of the report. • Conclusion 20. Student performance remains a major stumbling block across all regions. Fidelity of Implementation (FOI) recommendations (based on DPLA data) included in EQ4 have the potential to improve performance. • Conclusion 21. Performance is less problematic with low order skills such as letter sounds and decoding. FOI recommendations based on DPLA data have the potential to improve skills. EQ5: What are the factors that have implications for sustainability of quality 3Rs instruction after TP has ended? Local level ownership: Teachers reflect on the value of the CoLs to reinforce and clarify after trainings. Authorities have been involved in and feel a sense of ownership of TP. • Conclusion 22. The seeds for sustainability are planted on the ground and it will be achieved at the school/ward level via the CoLs and local entities. Technical sustainability: Teachers, HTs, and WEOs demand specialized trainings/coaching/mentoring to drill down on topics, such as phonics and reading instruction (see detailed topics in recommendations). Teachers, HTs, and WEOs demand expanding in arithmetic in Standards 1 and 2, and in Standards 3 and 4 in Mtwara and Zanzibar only as per the contract. • Conclusion 23. More coaching is required on specific topics at the school level directly from trainers with expertise in phonics and reading instruction. • Conclusion 24. There is a need for TP in spaces such as Arithmetic in Standards 1 and 2. National level ownership: Many high-quality materials have been produced for the four regions and 3 The DPLA is an approach that provides quality assurors with low-cost, rapid, district-level school monitoring data on school inputs, teacher practice, and student performance twice per school year. DPLA uses a sampling method known as Lot Quality Assurance Sampling (LQAS), which allows for district and regional-level decision making to be made from a small sample of schools per district. The approach was designed to improve upon existing school monitoring tools in Tanzania and to build the capacity of quality assurors to collect and analyze actionable monitoring data in order to be able to improve teacher and student performance in the early grades. v Zanzibar. However, the rest of the country does not have access to these materials. • Conclusion 25. While TP provides materials to four regions, the approach is parceled and ultimately the GOT will need to take ownership of the materials and approaches and collectively mobilize the DPs for scale up and expansion. RECOMMENDATIONS The following recommendations emerge from the above conclusions: • Recommendation 1: TP needs to strategically engage the central government actors in planning in order to move forward. There is interest on the part of the GOT to work with TP. Over the first two years, TP worked mainly with TIE and also worked at the local level. TP had worked with actors at the national level under Big Results Now (BRN). At this time, TP should also work more closely with other GOT institutions, namely PORALG and MOEST on the Tanzanian mainland and the MOEVT on Zanzibar at a technical level. • Recommendation 2: TP should ensure high FOI by strengthening the CoLs and injecting quality through the provision of videos, teaching modules on specific topics, etc. ultimately to model teacher behavior and change it to ensure that phonics methodologies are taught with high FOI and measured with statistically significant assessment like the DPLA to measure progress in order to adapt implementation to ensure high FOI. The CoLs have the potential to sustain implementation of TP methodologies. The CoLs need to be strengthened and expanded beyond WEOs and HTs as coaches) in schools and within wards to teachers as well. • Recommendation 3: TP should provide follow up through mentoring and coaching by WEOs and HTs. Joint visits between WEOs and a tutor should be considered in the future. In order to ensure sustainability, coaching needs to be institutionalized within the MOEST and in the WEO job description. This role should be played by the WEOs or someone else closer to the school. At the same time, tutors (from teacher training colleges) have more technical expertise. They could conduct joint visits with Quality Assurers (QAs) to provide guidance to teachers and WEOs to reinforce best practices from the trainings. It is important to consider the long-term sustainability and the role of other actors (including the GOT) for the provision of fuel for visits. • Recommendation 4: The DPLA should be aligned with the structures of the GOT, MOEST, and MOEVT to promote ongoing formative assessment and the provision of information for decision￾making. Findings suggest that the DPLA provides efficient useful information on the low performance of students across the regions. However, there is variation in skill and there have been marked gains. The TP Activity should work with the GOT, MOEST, and MOEVT to institutionalize DPLA for ongoing formative assessment and the provision of information for decision-making. • Recommendation 5: TP should complete the package of materials for Standards 1 and 2; arithmetic should be included through partnership. Specifically, with regard to arithmetic, TIE developed new official GOT textbooks and accompanying Teacher’s Guides for them in Quarter 3 of 2018. TIE has asked TP to edit the official Teaching Guides for Standards 1 and 2 arithmetic. In an initial review, the pedagogical approach to arithmetic in the official GOT teacher’s guides does not appear to be as student-centered as the approach to reading and writing. TP has supported integrating a student-centered (I do, you do, we do) approach into the TIE official arithmetic teacher guide and lessons. In addition, approaches from the above-mentioned DP supported materials from DIFID and EQUIP-T should be referenced and integrated into the math curriculum. 1 1.0 EVALUATION PURPOSE AND EVALUATION QUESTIONS 1.1 EVALUATION PURPOSE The purpose of the performance evaluation of the Tusome Pamoja (TP) Activity, which is now in its third year of implementation, is to analyze what is working or not working with Components 1 (Quality of early grade basic skills instruction improved) and 2 [Skills delivery of the Ministry of Education, Science, and Technology (MOEST) and Ministry of Education and Vocational Training (MOEVT) Strengthened]. Specifically, the evaluation will produce evidence-based findings and recommendations to inform future TP implementation and design of reading, writing, and arithmetic (also known as the “3Rs”) programming. The evaluation was conducted between July 22-August 12, 2018 by a team assembled by the Data for Development Activity implemented by ME&A, Inc. The team included: Megan Gavin, Ph.D, Team Leader; Michelle Davis, Qualitative Analyst; Immakulata Komba, Technical Specialist; Jacob Laden, Evaluation Advisor; Godfrey Teli, Ph.D, Technical Specialist; and Gerald Usika, Technical Specialist. The results of the evaluation will be primarily used by the United States Agency for International Development Tanzania (USAID/Tanzania). In addition, the results may be of interest to other stakeholders such as the Government of Tanzania (GOT) [including the MOEST and President Office Regional Administration and Local Government (PORALG)], development partners (DPs), such as the Department for International Development (DFID), Swedish International Development Corporation (SIDA), United Nations Children’s Fund (UNICEF), the Global Partnership for Education (GPE), Global Affairs Canada, and others. Specifically, the evaluation is intended to serve as an analytic deliverable, which brings stakeholders together (outside of the Joint Annual Reviews) to discuss the program and major achievements in Years 1 and 2. The evaluation serves as a platform for discussion on areas for improvement and potential adaptations to be made moving forward during the remainder of implementation (consolidated in Recommendations). Finally, the evaluation will also serve as an opportunity to discuss where stakeholders could potentially support and work together to scale up TP intervention(s). 1.2 EVALUATION QUESTIONS In keeping with the purpose of the evaluation, five evaluation questions (EQs) were developed: EQ1: To what extent is the TP making progress towards improving target beneficiaries 3Rs skills? EQ2: How is the development of a School Information System (SIS) leading to improved learning for the 3Rs? EQ3: In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by TP? EQ4: How can TP’s activities be done moving forward in order to more efficiently improve learning outcomes for the 3Rs? EQ5: What are the factors that have implications for sustainability of quality 3Rs instruction after TP has ended? With regard to EQ1, it is important to note that during consultations with USAID and the implementing partner (IP) it was determined that the third R was not being implemented in the first two years and would be rolled out in Fiscal Year (FY) 2019. 2 2.0 PROJECT BACKGROUND The TP Activity (AID-621-C-16-00003) is a $67 million, five-year activity awarded on December 31, 2015; implementation began in January 2016 and runs through January 2021. TP is being implemented by the RTI International (RTI) and supports USAID/Tanzania’s Country Development Cooperation Strategy (CDCS), specifically Development Objective 1 (DO1): Tanzania’s advance toward middle-income status supported: Lifelong learning skills improved. The primary goal of TP is to improve lifelong learning skills, defined as mastery of early grade reading, writing, and arithmetic (otherwise known as the 3Rs). A full results framework is included below. To measure achievement of this goal, two indicators are included in the Activity Monitoring Evaluation and Learning Plan. They are: Indicator 1: To increase the percentage of children who after two years of schooling can read and comprehend grade level text, decode simple sentences, and solve grade level arithmetic problems; and Indicator 2: To increase the percentage of children who after four years of schooling can read and comprehend grade level text, respond to simple writing prompts, and solve grade-level arithmetic problems. The TP Activity is organized around three intermediate results (IRs); these results align to support and achieve the overarching goal. As depicted in the detailed results framework on the next page, the three IRs are: • IR1: Quality of early grade basic skills instruction improved; • IR2: Skills delivery and assessment systems of central, regional, and local government(s) strengthened; and • IR3: Effective engagement in education of parents and community increased. (Note that Result 3 is not included in this evaluation as USAID has contracted a separate study to evaluate this result.) The TP Activity aims to reach 3,027 public primary schools from 34 districts with an estimated 1.4 million children directly benefiting over the five years of the Activity. In addition to direct beneficiaries, indirect beneficiaries from the spillover effects of the production of materials and development of methodologies/pedagogical approaches serve as a model for 3R reform throughout the country. TP works at the national, regional, district, and ward levels. Its purpose is to build the capacity of the Tanzania Institute of Education (TIE), MOEST, Zanzibar MOEVT, and PORALG. To do so, it: 1) works with key stakeholders on the official teacher’s guides and the development of student materials and on training in their use; 2) builds coaching and mentoring support networks; and 3) collects data for evidence￾based decision-making. Specifically, TP works in four regions of Tanzania Mainland (Morogoro, Iringa, Ruvuma, and Mtwara) and the two of the islands of Zanzibar: Unguja and Pemba. There has been one major modification to date, which introduced the SIS as part of the institutional strengthening. The modification also includes changes to implementation timeline, taking into consideration critical factors such as the time required for GOT approvals, institutional culture, and processes in order to garner national ownership, and external factors such as copyright and partner alignment. 3 Figure 1: Results Framework4 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODS To address the EQs, the evaluation team (ET) employed a mixed-methods approach, which analyzed project-related documentation, project monitoring data, and other documentation on education in Tanzania. Primary source data was collected by the ET through key informant interviews (KIIs), focus group discussions (FGDs), and an embedded mini-survey with various TP stakeholders involved in the program or who benefited from the program intervention. The use of multiple sources of data allowed the ET to triangulate information, ensuring that findings and conclusions are robust and that recommendations are sound. The ET’s goal was to generate not only an overall understanding about the 4 When TP was designed the term MOEVT was for both Tanzanian mainland and Zanzibar. Now only Zanzibar uses it MOEST is used for mainland; for this reason, both are used. Thus, all references to MOEVT should be read as MOEVT and MOEST. 4 project, its results, and effects but also a detailed assessment of its various components and the effectiveness of their approaches and implementing mechanisms. All primary and secondary data was disaggregated by appropriate demographics, including group, age, and gender/sex, whenever possible. Literature Review A systematic literature review was conducted. This included materials from the TP Activity such as quarterly reports, annual reports, the PMP, the contract Section C Scope of Work (SOW), the Request for Proposal (RFP) and other documents. In addition, gray literature (such as reports, case studies, best practice guides) was reviewed, including documents from DFID, GPE, the World Bank (WB), and Research on Improving Systems of Education (RISE). Academic literature was also reviewed. During the review process, each document was read, and a summary was produced. Then the documents were coded according to key words corresponding to the EQs. Annex VII contains the Systematic Literature Review table for reference. The literature review served both as a data source and as a source of input for the development of data collection instruments. Instruments There are two primary instruments: the KII guide and the FGD protocol. The KII guide was used with Head Teachers (HTs), local government authorities (LGAs), TP Activity staff, USAID staff, DPs, and the GOT (MOEST, PORALG, MOEVT, etc.). The FGD protocol was used with groups of teachers, parents, and Ward Education Officers (WEOs). In addition, a brief survey was designed (a half page document that included seven scale questions) to be self-administered by parents as they were congregating for FGDs. These instruments can be found in Annex III. Key Informant Interviews The ET worked purposively with the USAID/Tanzania Mission and RTI staff to identify the potential institutions which play a role in the national education system. RTI used a different approach in engaging each national stakeholder in implementing the 3Rs activities; that would mean the opinions from these groups would not necessarily be the same, hence all the key institutions were included in the sampling frame from which the national-level KIIs were selected. These institutions include: USAID/Tanzania Education Office, TIE, PORALG, MOEST, MOEVT, members from the education networks presented by UNICEF and DFID’s Education Quality Improvement Program Tanzania (EQUIP-T) as well as other relevant DPs. Because of the tight schedules of the key informants, a snowball sampling method was used to select one to two key informants from the aforementioned stakeholder groups. During the sampling process, RTI and/or USAID/Tanzania Mission staff were consulted by the ET to ensure the key informants would not be biased and would be relevant to interview. Based on the number of stakeholders, a total of 16 national-level KIIs were conducted in both mainland and Zanzibar. Focus Group Discussions Twenty-eight (28) FGDs were conducted with teachers, WEOs, college tutors/trainers, and parent representatives. The FGD protocol contained open-ended questions. The process included a note taker and translator as well as two technical team members. Recordings were also produced. Data was transcribed, cleaned, and entered to Dedoose for analysis. Each FGD contained an average of 10 participants. Brief Survey A brief written survey was given to parents prior to FGDs. The survey sought to gather insight on access to reading materials in the home. Data from the survey was entered in Excel and descriptive statistics were generated. A total of 69 surveys were completed by parents. 5 Classroom Checklist A 12-item classroom observation checklist was developed (see Annex III). Items included: decodable readers (y/n), assessment guides (y/n), big books (y/n), storybooks (y/n), teacher’s tool (y/n), other books (y/n), textbooks (y/n), materials on wall (y/n), pre-primary storybook (y/n), pre-primary teacher guide (y/n) and an open qualitative reflection: how do materials appear to be used? After interviewing the HT, the ET would ask if the HT would like to show them a Standard 1 and/or Standard 2 classroom (i.e., grade 1 and grade 2). In addition, there were questions asking about the quantity, availability, and use of books. This information was tabulated in Excel and included in this report. A total of 10-classroom checklists were completed. Fieldwork and Sampling The TP Activity is active in all government primary schools in four Mainland Regions, plus Zanzibar. In the Tanzania Mainland, the activity covers Morogoro, Iringa, Ruvuma, and Mtwara; in Zanzibar, the activity covers Unguja and Pemba. For the purpose of logistical feasibility this evaluation could not visit all regions in the Mainland and the two islands in Zanzibar. To achieve a geographic balance, the ET selected four field sites, including Morogoro, Iringa, and Mtwara in Mainland Tanzania, and one site in Pemba (North Region) in Zanzibar for field work, following discussion with the USAID/Tanzania Mission. Data was also collected at the national level in the Dodoma Capitol and Dar es Salaam. The evaluation used KIIs at the national level and FGDs, KIIs, observation, and mini-surveys at the local level (regional-school level). A purposive snowball sampling method was used for selecting regions, districts and schools from which the KIIs, FGDs, and mini-survey participants were drawn. The sample selection considered the urban and rural settings of the region as the main criteria for the selection of the schools. In each selected region, one rural and one urban district were selected. A total of eight districts and eight schools were selected, four urban and four rural. At the local level, when the ET attempted to reach specific stakeholders [i.e., WEOs, HTs, and Quality Assurers (QAs)], specific names and contacts were provided by the TP Activity; the ET selected a sample based on the schools and wards. The TP regional focal persons (typically from the regional education office) booked the appointments based on availability of the participants. Due to the large number (more than 10) of potential respondents to be invited in FGDs (WEOs, tutors, teachers and parents), the ET used randomized sampling to select 8-10 participants for the FGD and mini-survey. This approach was also discussed and agreed by the Mission and RTI before the field work. From the selected regions, districts and schools, a total of 32 KIIs and 28 FGDs were selected, as shown in the Table 1 below. Table 1: FGDs and KIIs per Region Focus Group Discussions Morogoro Iringa Mtwara Pemba/ Zanzibar Totals FGD1 – Teachers 2 2 2 2 8 28 FGDs total FGD2 – WEOs/Subject Advisors 2 2 2 2 8 FGD3 – Tutors/college trainers 1 1 1 1 4 FGD4 – Parents from the Parent Teacher Partnership (PTP) 2 2 2 2 8 Key Informant Interviews Head Teacher 2 2 2 2 8 32 KIIs total District Education Officer 2 2 2 2 8 District QAs 2 2 2 2 8 Regional Education Officers 1 1 1 1 4 TP trainers/technical staff 1 1 1 1 4 6 In addition to Table 1, Annex VII also shows a consolidated evaluation design which shows the evaluation questions, sources of data and methods of collection for each, and relevant methods of analysis. 3.2 ANALYSIS After qualitative data collection, notes were cleaned in a Word document (in English). After all fieldwork was completed, the ET met to design a code frame and define what fit within each code. The codes align to the EQs, for example, 3Rs, materials, teacher training, parental participation, SIS, sustainability, government involvement, among others (see the matrix at the end of this section). Dedoose (a free open source qualitative data software) was used to code qualitative data. First, the code frame was set up in Dedoose and then team members uploaded notes and attached “descriptors,” such as gender, location, and role to each set of notes. The team also practiced applying the codes in order to ensure consistency of application among team members. Once consistency was assured, team members coded all data. Descriptive analysis also took place, and as mentioned above, survey data analysis was conducted using Excel; results were tabulated (i.e., summed) and frequencies of counts were generated. 3.3 LIMITATIONS Selection bias. The list of KIIs was developed in coordination with the IP. In addition, some KIIs declined to be interviewed. Therefore, those who were willing to be interviewed may offer views that are different than those who declined or were not proposed to be interviewed. Recall bias. Since the majority of the interview questions asked participants to reflect on activities that took place between 2016 and 2017, recall bias may affect responses. They may either not remember or remember things more positively than they were. Halo bias. Similar to the above, this is the tendency for respondents to underreport socially undesirable answers and to alter their response for the interviewer. This is similar to an observer effect. The above limitations were largely mitigated via the approach described above, through using a mixed￾methods approach, which triangulates sources and methods. The ET also held a feedback session in the last week of data collection, which allowed findings to be validated with the TP Activity. Arithmetic. With regard to EQ1 it is important to note that during consultations with USAID and the IP it was determined that the third R was not being implemented in the first two years and would be rolled out in FY2019. 3.4 CAPACITY BUILDING AND GOT ENGAGEMENT DURING THE TP EVALUATION Data for Development engages and builds capacity with local institutions and firms throughout the evaluation process. Through the TP evaluation, the ET engaged regional and district authorities in the evaluation process, working with regional administrative secretaries (RAS), Regional Education Officers (REO), District Executive Officers (DEDs), and District Education Officers (DEOs). In Zanzibar, the team also worked with the Office of the Chief Government Statistician (OCGS), and the Second Vice President’s Office. Additionally, Data for Development worked to strengthen capacity of IP staff and local USAID staff by working with them throughout the evaluation design process. This engagement builds local IP and USAID staff knowledge of best practice in performance evaluation, ensures strong buy-in to the evaluation process, and increases their ability to consider how best practices in monitoring and evaluation (M&E) may be integrated into ongoing project implementation. 7 In addition to reinforcing IP capacity during the design, the evaluation itself included strong local participation: four Tanzanian researchers (of which one was Data for Development staff) and one local institution (Utafiti Associates). Local team members participated in formal capacity building before the evaluation began and also received iterative, ongoing tailored capacity building during fieldwork and analysis. Before data collection, four Tanzanian colleagues participated in a two-day training on qualitative coding, data analysis, and knowledge sharing in Dar es Salaam.5 This training aimed to improve the ET’s skills on qualitative data analysis concepts and on how to use Dedoose software to code and analyze data. This capacity building session also served as a Training of Trainers (ToT) so that these team members could go on to share their new knowledge with colleagues in their institutions and others in their professional settings. The pre/post training assessment showed significant improvement in participant knowledge on qualitative data coding and analysis skills, and on using the Dedoose software. Secondly, during data collection, Data for Development further strengthened capacity of the ET. Team members gained evaluation experience by participating in qualitative data collection, note taking, transcriptions and translation of recorded audios, qualitative data coding, and field logistics management. Several staff also participated in preparation of the debrief presentation and provided support for report writing. Additionally, to support ongoing professional development during the evaluation process, Data for Development and the Team Leader held twice weekly calls to review learning and promote best practices. Based on local team members’ pre-existing expertise and knowledge, the Team Leader provided tailored guidance to help each team member further strengthen their skills, particularly as related to USAID procedures and preferences. In addition, the Team Leader offered local team members the opportunity to solidify their knowledge of Dedoose by offering a short refresher course and then providing guidance as each team member began coding. 4.0 FINDINGS AND CONCLUSIONS The findings and conclusions below are based on evidence captured across five sites visited in three mainland regions (Mtwara, Iringa, and Morogoro) as well as North Pemba in Zanzibar. Prior to delving into each EQ, it is crucial to note the progress to date in terms of real numbers of districts, wards, schools, teachers, and pupils served in the four mainland regions and Zanzibar. Please see Table 2 below for all values and Figure 2 (right) on learners. Table 2: Stakeholder Population Served by Tusome Pamoja Region District Ward School Teachers Iringa 5 106 981 1,522 Morogoro 9 213 842 2,793 Mtwara 9 190 663 1,707 Ruvuma 8 173 765 1,819 Zanzibar 11 10 283 2,794 Figure 2: Learners Reached with Reading Programs 5 The training took place on July 19-20, 2018. 1 4.1 EQ1: TO WHAT EXTENT IS THE TP ACTIVITY MAKING PROGRESS TOWARDS IMPROVING TARGET BENEFICIARIES 3RS SKILLS? Given USAID’s Global Education Strategy 2011 that was active at the time of this evaluation and the TP Activity contract—both of which put heavy emphasis primarily on literacy (reading and writing)—findings presented below first relate to the quantity and quality of literacy materials. Then findings and conclusions related to the teacher’s instructional tool(s) for use of these materials and then official government certification of these materials is discussed. Lastly, the arithmetic (the “third R”) is examined and findings and conclusions are presented. In general terms great gains have been made with regard to literacy materials (quantity and quality) whereas (and given the parameters of the TP contract and USAID strategy) there is more work to be done in arithmetic. See more under section 3Rs + arithmetic. On November 14, 2018 USAID approved and released a new Education Policy which allows for more basic education funding to go towards arithmetic which will assist with expanding funding for the last of the 3Rs. Literacy Materials – Quantity With regard to EQ1 and the advancement to achieving the 3Rs, it is important to look closely at IR1: Quality of Early Grade Basic Skills Improved. Specifically, this includes IR1.1: Provide quality early grade reading instructional materials for reading, writing and arithmetic. Under this IR, over 1.5 million decodable readers have been produced. In addition, under IR1.2: Effective use of reading writing and arithmetic teaching and learning materials in the classroom by teachers increased, over 10,000 teachers have been trained, 8,000 of which were trained on using decodable readers (Quarter 2 report, FY 2018). Total books provided with United States Government (USG) assistance (a standard F indicator) in FY17 was 2,548,595 and in FY18 was 1,538,592, as shown in Figure 3. Figure 3: Books Provided with USG Assistance Figure 4, next page, presents data from the school checklist used during fieldwork. Results show that TP has provided many materials in the schools it serves. The ET wants to highlight that TP materials were not only designed and produced, but they were also delivered to schools. More so, these materials are often the only material available for reading in school. This is due to resource limitations (as confirmed by DEOs and REOs). Many times, even when government materials are reproduced, the additional costs (for example transportation) cause a bottleneck and materials do not actually arrive directly to schools. With regard to IR1.3 School leadership and management in support of reading, writing, and arithmetic strengthened, nearly 6,000 education authorities (i.e., WEOs, HTs, etc.) have been trained to support school-level mentoring/monitoring. In addition, under IR1.4, Quality of pre-primary reading and writing instruction improved, the TIE approved the introduction of the pre-primary storybooks for target schools and, as such, over 2,500 big books, over 12,750 small story books, and over 200 teacher’s tools have been produced (see EQ3 for more on internalization of training). 2 Materials – Quality In addition to the large quantity of materials (namely the decodables and storybooks (leveled readers), the materials have high-quality content and pictures/visuals and are made with durable materials. High-quality materials are important for improving the 3Rs and also for the first order effect of access and attracting children to school. As one REO noted during a KII, “The Tusome Pamoja project attracts more children to school because they like the stories and the pictures.” A WEO says, “Through the teaching material they issue, such as books, 3R teaching is easier since these books use easy methods and if the teacher uses them well, even slow learners will understand 3Rs if good follow up is made.” The quality was also confirmed by a DEO, who said, “The books are interesting because they have pictures and the pictures attract the children to read the stories. The stories are good because they relate to the traditions – for example you can see in the books the traditional clothes and the environment.” A teacher added, “The pupils like the materials.” In addition, parents reflected on the durability of the materials. A parent says, “The materials are durable. They have good quality covers. They do not tear easily.” Often, when discussing the materials, participants noted that there are a lot of materials, but there are also class sizes of 100 students and some stakeholders are concerned about the durability of the materials in the long-term given the large number of students using them. To increase durability, they have covered the books with paper to ensure they last. Figure 4: Materials in schools during data collection *note N= 10, all schools had the materials from TP, and it was the main source of reading Materials – Teacher’s Instructional Tool The teacher’s tool provides teachers with approaches for managing large classrooms, which as mentioned above contain upwards of 100 children per class in first and second standard. The tool provides teachers with a script they can rely on for classroom management and, in turn, increases the time dedicated to teaching reading during a given school day. While according to KIIs with USAID, the GOT evaluates the lack of autonomy given to the teacher when they use a script, arguably another way to see it is as a resource to help the teacher achieve real learning outcomes for students, especially given large class sizes. In an FGS, one teacher said, “we were provided with tools to help us in teaching directly in class and how we could support each other to fulfill our responsibilities in class.” TP states that they continue to meet with TIE to clarify and map out the formal Teacher’s Guide for Standard 1 and 2, arithmetic which are comprised of three teacher guides (each). Data collection and consultation with TP staff showed that the guides try to bridge the gap between TP materials and the national curriculum—although TP materials align to the national curriculum (the syllabus and the textbook) 3 the tool itself indicates the alignment for teachers more directly. This is crucial in terms of use of the materials [i.e., Fidelity of Implementation (FOI)] and is discussed more in EQ3. It is also important to ensure the alignment between the teacher’s tool and the materials with the national curriculum and working with the GOT, namely TIE, to certify the materials, which is further discussed in EQ5 with regard to sustainability Certification of Materials According to the KII with the TIE, they work effectively with TP to improve the 3Rs. TIE, as an institution, works in five areas: 1) development of curriculum, 2) development of materials and texts, 3) in service teacher training, 4) research, and 5) consultation to the government. As such, given its areas of intervention, TIE is TP’s counterpart and they have worked in close coordination throughout more than two years of implementation. Specifically, TP has worked with TIE to develop the capacity of their experts (curriculum specialists) to develop materials (for example in technical workshops to create the storybooks). In a KII, one curriculum specialist stated: “We work with the TP staff to write books, they support us in working sessions and TP has taught us to write storybooks. First we look at the curriculum and then develop the stories.” In so doing, TP is building the capacity of TIE (as noted by the Director) and producing deliverables. (See EQ3 for more on internalization). The specialists also participate in training as part of the TP’s Master Trainers who roll out training at the district level (with DEOs, WEOs, QAs). To date, they have largely been working on the 2Rs (reading and writing) and now they are turning their attention to capture the third R, arithmetic (see section below). They are also working together on the official Teachers’ Guides. With regard to the certification of materials, it is important to differentiate certification and approval for use and validation in the schools (further discussed below and in EQ5). This is noteworthy because when the materials are approved to be used, they can only be used for piloting purposes (for example, in TP schools) during a specific period of time. However, certification implies that the GOT has certified and allows the materials to be reproduced and used nationally (including by other DPs and funding sources). Currently, materials (including leveled readers and decodables) are approved and are in schools in the four regions and Zanzibar. They are being piloted until the middle of 2019, at which time TP will have insights and feedback from the end users (i.e., teachers and students) required for the “user validation.” Once users have validated the materials, TIE will complete expert validation and can then certify the materials. In this way, RTI and TP are in the process of learning from valuable feedback to iteratively improve materials and approaches. As with much of the Activity’s approach, they are also learning how to better serve their counterparts and how to better do development work, particularly with reading, e.g., improving the materials, and improving the approach to training. This has also been noted by other DPs, namely UNICEF and DFID. Now, as part of this learning, is shifting in their work especially with PORALG and MOEST primarily to include their voice and build ownership for the materials and methods in order to sustain the TP Activity. Photo 1: Teacher shows materials. 4 Progress in Reading and Writing (2Rs) + Arithmetic The quality materials and internalization of training (discussed in EQ3) may begin to influence student reading abilities. One WEO says, “Now a big number of students know how to read, write, and count. What led to this achievement were the frequent trainings and the techniques they were given that have helped them improve much on their ways of teaching.” There is significant qualitative feedback at the school and local level from teachers, HTs, WEOs, DEOs, and QAs. As a result, there has been student progress towards reading. The 2018 DPLA Phase III report, discussed below, partially backs these findings, showing some progress in key areas like letter recognition and decoding abilities. The same progress is not seen in writing. This is due to three factors: 1) the activity focused on the production of reading materials; 2) the training focused on the use of these materials; and 3) production of writing tends to be more demanding on students and tends to follow on or build upon closely after reading skills, for example one learns to read the letters a, b, c and then learns to write a, b, c. In addition to HTs and authorities (WEOs, DEOs, etc.) as noted in the quotes above, parents also confirm the progress made in reading and writing, although they do not discuss the progress in arithmetic (likely given the fact that less implementation has occurred in arithmetic to date). Parents (n=69) specifically identify their children’s ability to read (letters, words, and stories). Forty-nine (49) parents said that their children could read letters and words and 46 parents said their children could read a whole story. At both the local and the higher levels, further arithmetic support was asked for from TP. As one HT stated during a KII: “Once you have tasted the sweetness of the pedagogy in reading and writing you want it for the other subject areas like arithmetic.” TP has been asked and is in the process of responding to both PORALG and TIE requests for support in arithmetic. The plan has varied over the last quarters. To date, there are two work streams at play. The first with TIE is to develop a teacher tool to support the use of arithmetic materials (such as arithmetic flash cards, timelines, etc.) in Standard 1 and 2 as mentioned above under Teacher Instructional Tool (confirmed by a KII with TP). The second work stream is with PORALG. KIIs suggest that the TP-style cascade training model will be used to train district-level officials who will then train the use of the arithmetic materials at the school level. This is an efficient use of resources because PORALG already has trained the master trainers, and other DPs, namely DFID, already have materials (e.g., number lines and flash cards). USAID continues to explore funding possibilities with other DPs. In the case of Zanzibar, there were lower performance rates on assessments at the outset of the program, (see figures below from the 2016 Activity reading baseline, for example). These figures only contain the regions within the scope of this evaluation. In both reading fluency and comprehension, fewer than five percent of students reach the benchmarks. This was lower than Mainland. For the TP Activity, math results were equally concerning, with two percent of students in Zanzibar meeting benchmarks for addition/subtraction and zero percent meeting benchmarks for missing number subtasks (see the figures that follow). 5 Figure 5: Oral Reading Fluency, 2016 Figure 6: Comprehension 2016 Figure 7: Addition/Subtraction, 2016 Figure 8: Missing Number, 2016 Following the reading midline, which is being conducted this year, RTI will be able to know the extent to which reading has changed over the past two years following program intervention in the four TP regions in Mainland and Zanzibar. Although the recent DPLA 2018 showed low performance across regions (see detail in EQ4 and technical recommendations), progress has been made. See the text box below. DPLA 2017 Phase III Progress 2017-2018 - Iringa: Letter Knowledge (+13% pts), Invented words (+8% pts), Reading comprehension (+2% pts), and Dictation (2+% pts). - Mtwara: Letter Knowledge (+18% pts), Invented words (+5% pts), Reading comprehension (+1% pts), and Dictation (2+% pts). - Morogoro: Letter Knowledge (+3% pts), Invented words (-9% pts), Reading comprehension (+2% pts), and Dictation (2+% pts). - Zanzibar: Letter Knowledge (+29% pts), Invented words (+12% pts), Reading comprehension (+3% pts), and Dictation (5+% pts). Table 3: EQ1 Findings and Conclusions Evaluation Question 1 Findings Conclusions Materials – Quantity A large number of materials have been produced and delivered (story books, decodables, etc.). 1. A large number of materials exist and are Often the TP materials are the only resource for reading crucial to improve reading skills. instruction in classrooms and schools. 6 Evaluation Question 1 Findings Conclusions Materials - Quality Materials are engaging and attractive, they are colorful, and the stories are interesting. 2. Materials are good quality in terms of design and content and they motivate children to come to school. Teachers and authorities confirm materials motivate student attendance. Teacher’s Instructional Tool Teachers teach in the context of large class sizes, often over 100 students. 3. There is commitment by TP and interest by the GOT to work together to align the TP with the national needs and curriculum. The teacher guide is in the process of aligning the materials with the national text and curriculum.* Certification Materials have been approved for use and are currently in a process of validation and alignment. 4. Some government entities have been involved in the production of materials and need to continue to be involved. The Standard 1 and 2 materials still need to be certified by the government. TIE was involved in the workshops to create materials, for example storybooks. Reading and Writing (2Rs) + Arithmetic Reading has been the main focus, and this is according to USAID policy. 5. Gains appear to have been made with regard to teaching-learning in Reading/Writing and not in Arithmetic. 6. TP is a national example for Reading/ Writing whereas other DPs (EQUIP-T) can be leveraged for Arithmetic. Writing is folded into the reading (for example in the teacher guide) Arithmetic skills development is demanded by teachers and HTs. Examples exist in Tanzania from other DPs in arithmetic (for example EQUIP-T). TP (and its set of materials/methodologies) provides the foundation for reading/ writing. 4.2 EQ 2: HOW IS THE DEVELOPMENT OF A SIS LEADING TO IMPROVED LEARNING FOR THE 3RS? According to KIIs, SIS is a government initiative and started under the EQUIP-T activity funded by DFID. SIS was added to TP after initial design of the project and currently sits within IR2: Skills delivery and assessment system of PORALG, MOEST, and MOEVT improved. While it is still too early to know how TP implementation is leading to achievement of the 3Rs, this section will discuss potential contributions given the perceived utility of the system and potential challenges. SIS is designed to provide key recurring information and data points. DEOs and REOs confirmed this in KIIs. In addition to school profile data, SIS provides information on student attendance, end of grade assessments, student retention, and dropout, and can be used to generate reports for conversations between HT and teachers and parents at the WEO and school levels. SIS complements other pre-existing school data systems by providing local-level data and, in turn, reports for decision-making. Other data systems, like the worldwide Education Management Information System known as the Basic Education Information System in Tanzania, only provide large aggregate statistics at the national level and are not updated on a rolling basis like SIS (the SIS has access to data daily, weekly, monthly, quarterly, and annually). The HT collects the data at his or her own school and then can generate reports to share with the teachers and parents of children in the school. 7 For example, the HT can go daily and collect daily attendance from teachers in their classrooms. He or she can have reports on daily student and teacher attendance. Or, the HT can collect monthly examination data and will have monthly progress reports on student achievement. At this stage, HTs are learning how to use the tablets and collect the data. The next stage (described further below) is learning how to generate reports and share information; lastly, they will learn about planning and using the information to make changes to improve education (3R) achievement. Potential According to DPs on the national level (for example, UNICEF) “people are excited about the future use of the SIS. It has the potential to do a lot at the school level and fill in a gap in the system.” This attitude of positivity was also reiterated at the field level by stakeholders throughout the system including REOs, DEOs, WEOs, and HTs. They have expressed the potential for the SIS with the actors and this energy has been received and has ownership. For example, in one district, the DEO noted, “the SIS will be useful to help know the school progress and the student progress and to help us to plan with the information from the system.” He also could explain the system. He went on to say that “the SIS can generate information, daily, weekly, quarterly, and annually.” Participants in the study explained how SIS will relate to their work planning and how it has the potential to improve the 3Rs. For example, teacher and student attendance have an impact on learning to read. It was clear that in both Mainland and Pemba REOs and DEOs understood SIS and the purpose as two-fold: 1) to gather information; and 2) to plan and use information for decision-making. One REO said, “we will need to focus on the second phase and the role of participation and decision-making [based on the data].” He emphasized, “this includes parents, teachers, REOs, DEOs need to be strengthened so they can have regular meetings to discuss progress.” This will enable more analysis by WEOs and HTs on their particular situation and on how to improve the 3Rs in their ward/school. FGDs with WEOs and KIIs with HTs confirm other local government stakeholders’ sentiments. For example, in one WEO FGD, three participants in a row cited that SIS will help them get information quickly so they can follow up, will allow for follow-up on student results, and will increase teacher responsibility. A separate FGD with WEOs in a different region also shows a similar sentiment, with one participant saying, “The information collected by SIS system will help much in teaching and the improvement of 3Rs in schools. It will also promote accountability because WEOs will see the way each teacher executes his or her daily duties.” They continue that, “Also SIS helps us to present the challenges that we used to hesitate presenting, the information that seemed to be beyond our working boundaries and now such information will be sent directly.” HTs agree. One says that, “SIS will be useful because first, information about books should be entered, so that will help us to know the shortage of teachers, learning materials, infrastructure, also it will help all the teachers and school to be active in having all of the information and sharing this.” Concerns/Potential Challenges Despite the potential of the SIS, there were concerns regarding this information system depending on geography, human resources/capacity, infrastructure and school characteristics. Geographic differences posed challenges for implementation in some regions. For example, in Pemba, the tablets had not been delivered at time of data collection because the areas tend to be more remote. The concerns presented here are cross-cutting for mainland Tanzania and Zanzibar, although an implication is that they are even more of a concern for Pemba. Concerns with human capacity: Five KIIs with project staff show there are human capacity deficiencies. Not all HTs for example have past experience working with smart devices. Also, the information itself can be difficult to enter into the tablet platform, especially with over 100 children per class. Workload is a concern. 8 HTs also requested more training, not only on how to use devices but also on how to generate and use the reports and data that emerge from them. This is key and part of the next phase of SIS implementation. As indicated in the quote from the REO above, training is also required in terms of how to use the SIS for decision-making and conversations. Furthermore, one DEO noted “the HT was trained but when he leaves, we will need more training, because when that HT leaves there is no one left with the knowledge.” With regard to information and communication technology (ICT) access, one REO [in Pemba] noted that “it is crucial to think about how SIS will connect to the Internet in the interior.” In addition, although there may be Internet, as was the case on Mainland, it will also be key to assure the right subscriber identification module (SIM) cards and service providers are used. Related to these concerns is the roles and responsibilities at the local and national levels, and capacities to do so. For example, who within PORALG has the technical know-how to manage the SIS? Status of SIS SIS is just taking off in the project; materials were developed and over 4,000 tablets were procured for Tanzania and Zanzibar (although Pemba and Zanzibar had not yet received tablets). In the Tanzania mainland, one tablet is provided per school, two per district and two per ward. It is slightly different in Zanzibar given the different government structures and different terrain (more remote). At the time of data collection Phase 1, training had been completed with regards to using the tablets themselves but not on how to use information that comes out of reports to guide decision-making.6 After this first introductory training, stakeholders are given time to practice entering data about their school or wards. TP then selects individuals who perform best with the tablet and then use these individuals to help teach the others in other wards and schools. The second part of the training will focus on how to have the conversations about the data collected with the teachers and the parents; this is where the work comes in; it is also where it reaches into and relies upon R3 (see EQ4 below) which focuses on community engagement7. Therefore, SIS links the engagement of parents and community (by presenting them with data and results) with the achievement of the 3Rs. When parents have information, for example, attendance or performance, they can make better decisions on what to do to help their children learn. This is the crux of the initiative, not only to collect information but to use it for decision￾making by local stakeholders, especially HTs, teachers and parents. Local/ National Roles Right now (Quarter 3, FY18), there is political will and momentum which can be capitalized on and used to increase ownership of SIS by the local and national government. However, according to high-level KIIs, it is still unclear who will be able to provide long term technical expertise. According to one participant (KII DFID), EQUIP-T provides support in the form of a person providing technical assistance within PORALG’s ICT unit. But, these are EQUIP-T staff supporting PORALG, in the end, even according to this participant, more technical expertise for the management of the system is required. TP is adding value by 6 TP is investing in supporting the districts to identify the HTs who are about to retire and choosing others to train and to whom to provide tablets- therefore mitigating the loss of the capacity building and possibly the tablets. 7 R3 was agreed not to be included officially in this evaluation as it is the focus of another study conducted during the period the evaluation was being conducted. Photo 2: TP staff explains the SIS training. 9 expanding to the TP regions (in addition to the DFID-funded EQUIP-T regions) and also attempting to work with PORALG to build sustainability (technical and budgetary), (see EQ5). Furthermore, it is unclear who will be linking the SIS to 3Rs’ planning and decision-making; this profile is beyond ICT expertise at the local and national government levels. On the local level, similar to the central level, questions remain with regard to who will provide ongoing maintenance to the tablets, updates to the software, possible repairs or even replacement? Who will provide capacity building and reinforcement for new staff, in particular HT on the use of the tablets and in turn the use of reports for decision-making? As cited by the DEO above, who will train new HTs? Table 4: EQ2 Findings and Conclusions Evaluation Question 2 Findings Conclusions Potential Actors are positive and eager about the opportunity to collect and use data via the SIS. 7. SIS has the potential to provide information at the local level for decision￾During data collection relevant stakeholders knew about and making and improving the 3Rs. could describe the SIS. Participants are looking forward to being able to use and plan with the information. Government actors say the information will be useful for decision-making and allocating resources. Concerns Users do not have experience working smart devices or tablets and deficiencies persists. 8. There remain concerns in terms of ICT access and human capacity which require Users demand more training on tablets and on how to generate attention as SIS rolls out. reports to use information for 3Rs. Not all schools have access to the Internet or electricity or the correct SIM card. National and local government roles It is not clear if the SIS will be maintained by the GOT, if they have the capacity and political will at the national level. 9. Local and national government support requires further definition, strengthening, and alignment with existing information systems in order to effectively use, maintain, and sustain the SIS. It is also not clear at the local level who will maintain the SIS, i.e., train new staff, maintenance, updates, replacement and repair, etc. Table 5 below summarizes results to date. 10 Table 5: Results to Date Result 1 Quality of Early Grade Reading Basic Skills Instruction Improved Result 2 Skills Delivery and Assessment System of PORALG, MOEST, and MOEVT 2016-2017 2017-2018 2016-2017 2017-2018 - Leveled readers (10 book titles Tanzania, 10 titles Zanzibar) - 1.5 million grade 1, 2 decodables distributed, over 16,000 assessment guides - Finalization of teacher guide Zanzibar Quarter 2 FY18 - Primary baseline assessment report - Preprimary baseline assessment - Baseline pre-primary and primary, and community assessments disseminated -Teacher read aloud, big books (five titles Tanzania, five titles Zanzibar) - Grade 1 and 2 Kiswahili draft teacher guides - Community engagement baseline assessment - - Decodables (three titles Tanzania, three titles Zanzibar) - Printing 2,500+ big books, 12,600+ storybooks, and 200+ guides - Decentralized Periodic Learning Assessment (DPLA) - DPLA phase 2 complete, analysis and with Master trainer refresher Quarter 2 FY18 - 24 storybooks for pre-primary - Pre-primary storybooks (Mtwara), development of guide [produced and in 126 schools and teachers, HTs, WEOs and QAs trained] Quarter 2 FY18 - SIS long-term technical assistance (TA) mobilized SIS TA, procurement of tablets, officer set up - WEO mentoring professional development materials - SOW for assessment of teacher and HT - School Leadership training Quarter 2 FY18 - Road map for SIS national coverage - SIS Zanzibar procurement - Training on use of readers, 9,000 teachers, 700+ WEOs - on decodables, 12,000+ teachers, nearly 700 WEOs - Over 6,000 teachers, over 2000 section leaders, over 2500 HTs, and 645 WEOs trained - Draft developing culturally-relevant assessments - Research paper qualitative data analysis, proposal draft - - Standard 1 and 2 Arithmetic Teacher guide developed Quarter 2 FY18* - - Social Emotional Learning (SEL) report finalized Quarter 2 FY2018 - Research proposal on teacher practices Quarter 2 FY18 *Source FY 2017 annual report (i.e., first full year of activity), Quarter 2 FY 2018 report (most recent available to ET to date). 11 4.3 EQ 3: IN WHAT WAYS ARE GOVERNMENT OFFICIALS, SCHOOL ADMINISTRATORS, AND TEACHERS DEMONSTRATING THAT THEY HAVE INTERNALIZED THE CAPACITY BUILDING PROVIDED BY TUSOME PAMOJA? Teachers Training and capacity building have been provided via a ToT model, starting with key experts training local tutors from teaching institutes and cascading down through LGAs, HTs, and then teachers. The most recent data provided by RTI shows that they have directly trained, or reached via the cascade, more than 9,926 education administrators and officials and 22,297 teachers. Stakeholders underline that the initial training provided was not sufficient and there were still knowledge gaps that limited attitude and practice change. The desk review documents, including quarterly reports and field notes show that TP offered further follow-up ToTs, which helped address some, although not all, knowledge gaps with most stakeholders now more knowledgeable on the new approaches (i.e., participation) and less knowledgeable on new teaching methods (i.e., phonetics). During FGDs, teachers in all Mainland regions and Pemba were able to show new knowledge by citing many specific new approaches to teaching reading such as participation, I do-you do-we do, use of teaching aids, and songs. Teachers were able to cite some new reading methods such as teaching reading in three phases, phonemes/letter sounds, and similar, but with less frequency.8 One teacher showed knowledge by saying, “We should provide reading exercises, prepare enough teaching materials/aids in the class, involve students and use different teaching methodologies like songs, drama, role play, games….” Findings also show changes in teachers’ attitudes. Teachers say that they find that the new approaches like I do-you do-we do and use of teaching aids useful and that they want to keep utilizing these. Even though the shift from knowledge and attitude change to behavioral change in the classroom has begun, teachers themselves as well as HTs, QAs, and DEO/WEO cite that there is variance in teacher behavior change and real-time utilization and internalization of new approaches, methods and materials. As one QA puts it, “use of training [in the classroom] is 50/50.” Findings show that there is partial progress towards Sub IR1.2, Effective use of reading teaching and learning materials in the classroom by teachers increased. Although teachers can often clearly articulate how they should be using new methods and are trying to do so, triangulation of findings shows a more complex story on implementation. One DEO says, “Kids are participating more. The problem before was teachers were building a big wall, like a gap, between student and teacher, but now they are learning in a friendly way, that’s why they participate more compared to before.” All teacher FGDs show that teachers are making progress towards implementing improved teaching and learning around reading. Many teachers have started using new approaches like participation, group work, I do-you do-we do, songs, use of teaching aids like flip charts, etc. One teacher says, “We use the system of I do, we demonstrate [we do], then they do” and another states, “We used the knowledge, because we have Figure 9: Teachers Provided Professional Development on Reading Instruction 8 Teachers in Pemba cited specific methods with slightly more precision than those from mainland, it appears in mainland the first round of training was not adequate with teaching methods such as the sounds from letters. 2 been provided with the knowledge of using teaching aids and involving the students… if you look at our students, they are doing different[ly] compared to the beginning.” Findings also show that teachers have internalized training by beginning to develop their own teaching aids made from local materials, for example, “I will paint the picture on the topic [about] which I am teaching, for example when I want to teach the alphabet, I paint it...” These all represent major strides towards having a more participatory, engaging classroom. These changes align with findings from the mid-term DPLA completed by TP in summer 2018. DPLA assessed teachers on their approach to teaching reading on seven areas that encompass connecting with learners, using materials and teaching aids, and structure of a lesson.9 Findings show that as a general trend, teachers in Iringa, Morogoro, Mtwara, and Zanzibar improved in using best practices for teaching reading. In Iringa, 91 percent of teachers were using six of the seven best practices, a 23 percent increase from 2017. In Mtwara, 78 percent were using best practices which represents a 22 percent increase; in Morogoro, 71 percent were using best practices, a 9 percent increase. Zanzibar sits at 74 percent of teachers using six or seven best practices, a 26 percent increase. Despite these significant improvements in use of teaching approaches, challenges remain in training utilization on topics related to teaching methods. Teachers cite difficulty for example around the pronunciation of letter sounds. Although some respondents cite that teachers are using TP materials that utilize this method (decodable readers), teachers and oversight stakeholders say that there are challenges in correct pronunciation of sounds, which limits full implementation. The use of phonics methods themselves is cited as a challenge and more information on this is included in EQ4 below. In addition, QAs, teachers, and HTs, also say that teachers struggle to integrate government-supplied materials and TP methods and materials, in some cases teaching two separate lessons on reading. In these cases, teachers are teaching one lesson using what they refer to as “government methods and materials” and the other using TP approaches. All teacher focus groups held referenced challenges; and teachers in five focus groups directly requested more and longer trainings to address some of the knowledge and practice gaps. This finding is triangulated by nearly all HTs and LGAs also mentioning directly or insinuating indirectly that more training is required. New knowledge and attitude changes on how to teach the 3Rs signal possibilities of longer-term shifts in teaching and learning. Additionally, initial use of new approaches points towards likely teacher behavior change that aligns with the goals of IR 1 and Sub IR1.2. A teacher said, “Through the methods I learned from Tusome Pamoja, I used them to impart skills to the students, for example in the use of sounds, how to teach the students the sounds… the children were able to learn a lot of things by themselves.” Although the newer training approach mentioned above has been appreciated and helped with internalization, doubts emerged—unprompted—from teachers, HTs and LGAs in all Mainland regions visited by the ET (Iringa, Morogoro, and Mtwara) on the ToT cascade approach. The main concern is about whether the end users of much of the trainings—classroom teachers—get the same level of detailed instruction and training as those higher up the ToT chain. In seven interviews and focus groups with local stakeholders, participants wondered if teachers receive all of the knowledge that is imparted during the initial ToT sessions for trainers, and they see a risk that trainings for teachers may be shorter than intended or may not contain the level of depth necessary to fully transmit knowledge. Some stakeholders believe head teachers and WEOs may be too busy to provide the full training to teachers as ideal implementation of the cascade model would require. The challenges cited by local stakeholders are echoed in academic research and practitioner blogs on development practice; while the cascade/ToT model does allow for significant scale, there are risks about quality especially for those further down the cascade. This is because initial trainings may be too short, follow-up support could be limited, and those at the top of the cascade (master trainers) may receive 9 The seven areas are: framing the lesson, beginning with a warm-up activity, connecting the lesson to learners’ knowledge/experiences, assigning tasks, using teaching/learning materials, using student books, and summarizing lessons. 3 higher-quality training from national or international experts, where as those towards the bottom (teachers) receive their trainings from those with less expertise10 and that information is lost or mis￾interpreted in this transition.11 Findings show that local stakeholders are concerned by these exact issues. However, the ET also notes that TP has made course adjustments to address these challenges, such as reducing the number of trainers in the cascade and having an increasing role for professionals at the Ward level. HTs and LGAs believe that ongoing, tailored support for teachers is still needed from TP to help teachers better utilize methods in challenging environments. As cited throughout the document and in Recommendations 2 and 3 below, the ET believes that continuing to improve training approach and length and providing ongoing support and follow-up can address challenges that emerge due to the ToT approach. The Communities of Learning (CoLs) may be one way to provide this ongoing support. Based on KIIs and FGDs, CoLs fill some knowledge gaps related to approaches to teaching the 3Rs and methods such as phonics. The CoL was mentioned by school-level and government stakeholders in all regions visited as a critical change in approach to teaching and learning, as it creates a space where teachers can share their challenges in the classroom and discuss solutions as a group. One HT says, “In CoL each teacher comes up with the challenge when teaching specific subjects. We exchange knowledge. That helps us teachers to teach well” and a DEO in another region echoes, “We have regular CoL meetings. These CoL are able to solve the learning and teaching challenges.” Additionally, the CoL can fill knowledge gaps by allowing for ongoing sharing and learning regarding best practices. Since teachers may quickly gain confidence on different topics from the training, the CoL can become an opportunity for teachers who have understood specific new materials to coach colleagues. Given the above, it is possible that CoL could have a role in supporting ongoing internalization for teachers. The TP FY18 Quarter 2 report cites increased use of CoL’s as part of adaptive management, so the project has noted the importance of this activity for internalization. Head Teachers Findings point to some progress towards Sub IR1.3, School leadership and management strengthened for improved 3Rs implementation. All HTs (eight out of eight interviewed) demonstrated they retained knowledge from training. HTs can describe specific new teaching approaches to facilitate learning such as participation, talking walls, group work, and community/parent engagement. Some also described using or overseeing reading-specific methodologies like phonics or tracking words-per-minute. There are also noticeable changes in attitude and practice at the HT level, with all HT respondents from Mainland indicating changes in attitudes or practice and one out of two HTs from Pemba also indicating this. For example, there are shifts in attitude about HTs’ responsibility to oversee reading, teaching, and learning. One HT says, “Before [I thought] responsibility for teaching the 3Rs was only for teachers. But after training, I know now that I am also responsible for the 3Rs…I am able now to do monitoring in school because I am aware and I know what is required to be done.” Changes in behavior are also visible and findings indicate stronger school oversight and management (Sub IR1.3), although specific changes vary. All HTs could cite at least some changes that they have put in place as a result of TP training, including classroom oversight, supporting use of new materials, or providing wider guidance to teachers. In addition, HTs are supporting utilization of books; in all regions, except one (Mtwara), there was significant evidence that HTs were ensuring new TP materials provided under IR1.1, such as storybooks and decodables and were being utilized in classrooms.12 Other examples of behavior 10 Burns, Mary. Part 1: How the Cascade Approach Fails Teachers like Ismail. Global Partnership for Education Blog, World Bank. https://www.globalpartnership.org/blog/tale-two-teachers 11 Mpho, M. D. & Matseliso, M. L. (2012). Does the cascade model work for teacher training? Analysis of teachers’ experiences. In The International Journal of Educational Sciences, 4(3): 249-254 available at http://www.krepublishers.com/02-Journals/IJES/IJES￾04-0-000-12-Web/IJES-04-3-000-12-Abst-PDF/IJES-04-3-249-12-210-Dichaba-M-M/IJES-04-3-249-12-210-Dichaba-M-M-Tt.pdf 12 This issue of materials not being used, but rather preserved for future use, was brought up frequently in Mtwara. QA in the region cited that they are providing ongoing guidance to schools to utilize materials. 4 change include taking leadership roles in facilitating collaboration spaces like PTPs13 and CoLs14 that are key platforms for supporting teaching and learning. Teachers confirmed high levels of HT engagement during the recent DPLA assessment, where 95-96 percent of teachers in Iringa, Morogoro, and Mtwara say that the HT provides regular support and feedback at least once per month. The above findings show there are observable changes in attitudes by HTs on how to oversee reading, learning, and teaching, and that some behavior change is taking place regarding follow-up and supporting teachers and students. In addition, though there was little formal mention of leadership skills, there is progress towards Sub IR1.3 as HTs show potentially strong leadership skills by facilitation of the CoL and PTP, and those skills may further emerge with ongoing leadership-specific training planned for this year. Further implementation of the CoL and the upcoming launch of SIS offer more opportunities to show leadership. Local Government Triangulation of findings from KIIs with DEOs and QAs and FGDs with WEOs/subject advisors and tutors show that LGAs have increased knowledge on reading and demonstrate behavior change by shifting their approach to oversight and teacher support, showing progress towards Sub-IR2.2, Coaching and mentoring of teachers in the use of reading and writing materials and lessons improved. Two representatives of QAs in all regions and Pemba cited that they have a better idea of what teaching and learning practices they should be looking for during visits. A QA explained the knowledge gained has influenced the way he does his job, saying, “The training was useful for the inspectorate, we go to schools and apply it by seeing if teachers are using the books and the content on phonics, administration…the way I work as an inspector has changed a lot since I received the training.” There is also evidence that stakeholders have shifted their mentality about teaching and learning. For example, government oversight stakeholders in multiple regions insist that their job title is now “Quality Assurer” as opposed to their former title, “Inspector,” which reflects an attitude change. This change in name and the affiliated explanations provided by QAs point to a shift from an audit model of government school inspection towards a collaborative model based on improvement and support. In Iringa, findings show that the TP project has been completing joint quality assurance monitoring visits that include TP staff and government QAs. After the joint visit, a short debrief or discussion takes place on what each person saw regarding teaching and learning of reading. QAs considered these visits a useful exercise in knowledge exchange. Although all government actors were aware of TP and have participated in activities and training, the level of ownership and behavior change varies by region. Iringa and Mtwara have strong government coordination with TP and show strong ownership. Stakeholders in Iringa, for example, complete the joint monitoring visits with QAs and can cite significant school-level changes. One DEO from a high ownership Figure 10: Education Administrators Provided with Professional Development 13 This is happening in all regions with the exception of Morogoro. HTs cite they have taken strong roles in facilitating and promoting the PTPs (which are discussed in more detail in EQ4). 14 Though some HTs were initially resistant to the idea of CoL, HTs cite that over the longer term they have become critical tools for teacher support. 2 region says, “After receiving this training, I felt inside that I have changed. I learned a lot of things that I was able to present to the teachers.” In Mtwara, government stakeholders have taken such strong ownership of the project that they maintain that TP simply provides technical assistance while they themselves implement. A government stakeholder in Mtwara says, “How can you separate me and project…The project has come to support key functions of education, and these are core functions of education department.” In this same region, WEOs show strong knowledge on how TP approaches and methods apply to their jobs. Comparatively, local government stakeholders (LGAs) in Morogoro are less directly involved with the project and had a harder time explaining their involvement or citing changes at the school level like PTP. In Pemba, some aspects of collaboration seem strong but for longer-term sustainability, more government coordination is needed to ensure internalization, including aligning plans, processes, and materials with Zanzibar Institute of Education (ZIE) and MOEVT. Interestingly, these findings on government involvement in implementation are mirrored by results of the DPLA assessment conducted by RTI in August 2018. Regions with higher FOI and better relationships between TP and local government stakeholders show more gains. For example, lower-order skills mentioned in EQ1, such as student letter knowledge, improved by 13 percent in Iringa and 18 percent in Mtwara, compared to 3 percent in Morogoro.15 Along these same lines, as discussed above regarding use of best practices when teaching reading, there were 22 percent and 23 percent increases in Mtwara and Iringa respectively, as compared to a 9 percent increase in Morogoro.16 Findings point towards the idea that involvement of local government in implementation has resulted in initial shifts in capacity and practice to ensure quality reading instruction, pointing towards progress on IR2.2 with strong progress in regions with better government ownership. In some regions, attitude and behavior change is ingrained enough that changes could carry past the project because key oversight professionals like HTs and QAs are changing their approaches to oversight of teaching and learning. National Government Findings comparing all 16 high-level KIIs government representatives of TIE, PORALG, and MOEST/MOEVT, as well as four KIIs with REOs, show that some national-level government stakeholders are more involved with TP than others. Findings also show that national-level actors have wanted to be more directly involved in planning. TP had engaged in this sort of direct coordination with the government in the past, but recent changes in the make-up of government ministries may have required engaging with the new government stakeholders more directly in collaborative planning. Also, all government officials are now based in Dodoma making it more difficult to plan and collaborate without involving extensive travel. Current government stakeholders feel that the program is prepared by TP and brought to them just for approval, as opposed to a more collaborative planning approach. One official said, “TP was organized before and brought to us…It could be better if they would have collaborated with us in the planning stages…” However, it is important to note that TP was designed under Big Results Now (BRN) and there was a change of government after elections. Stakeholders cite that challenges that come up during implementation could have been resolved if they were involved from the beginning during the planning process. However, planning with national stakeholders is challenging since there has been high turn-over and significant instability in institutions responsible for primary education. Therefore, while many stakeholders cite that they would like to be involved more deeply in planning, there may have been 15 Zanzibar results are not fully comparable as TP MTE focused on Pemba only whereas DPLA shows results for both islands of Zanzibar together. For reference, Zanzibar showed significant improvement from a very low baseline, having a 29 percent increase that brought student letter sound knowledge to 34 percent. 16 Zanzibar showed a 26 percent increase, but again this number encompasses both islands of Zanzibar and may not be representative of Pemba, where qualitative fieldwork took place. 3 practical barriers to doing this. Government turnover may mean difficulty coordinating, but even so further direct, early coordination with all national stakeholders has been requested. Many national stakeholders acknowledge TP’s contribution in terms of the quality of teaching and materials available in schools, and state assessment instruments developed by the program are high quality. Some stakeholders expressed the opinion that guidance from TP has been internalized; findings show the stakeholders interviewed from TIE suggest that they have learned how to develop materials like decodable readers and leveled storybooks, which will positively impact their ability to devise these materials moving forward. Additionally, TP influenced the government to adopt key aspects of the program and internalize them: for example, in terms of assessments, National Examination Council of Tanzania (NECTA) adopted the system of assessing Standard 1 and 2 students on their 3Rs progress. This is a new assessment that was not implemented previously and was piloted in 2017 in five zones across the country.17 One national government stakeholder said, “USAID advised that assessment should be streamlined within the government. NECTA adopted this and they are doing it countrywide in collaboration with USAID, but later will be owned by the government.” This shows there is some emerging internalization of TP approaches at the national level. Despite evidence that there are some changes at the national level, findings also show that key stakeholders, including REOs and high-level decision-makers, cite that success of TP ultimately lies with local stakeholders as district councils are more informed and involved in the implementation than at the national government, which is more focused on broad policy matters. Thus, there are opportunities to clarify roles in terms of 3R sustainability long term. High-level government officials believe the sustainability of TP depends on the implementation agents at the district and lower levels (Districts Councils, DED, etc.) but at the same time, mention possibilities of national level changes. The project therefore has an opportunity to capitalize on both of these perspectives and promote both local and national-level internalization. Table 6: EQ 3 Findings and Conclusions Evaluation Question 3 Findings Conclusions Teachers Training has changed teachers’ knowledge and attitudes on teaching and learning, although teachers seem more confident on teaching approaches and less knowledgeable on methods. 10. Attitude changes on how to teach the 3Rs signal possibilities of longer-term shifts in teaching, and initial use of some new approaches point towards teacher behavior change. 11. CoLs have a strong role in supporting ongoing internalization. 12. More support and tailored training is needed to adjust ideal approaches and methods to local realities. There is progress on using new teaching approaches, e.g., participation and use of teaching aids. There is some progress— but less—on using new teaching methods like phonics. The CoL could have a role in supporting ongoing internalization for teachers. In some cases, teachers are utilizing TP and government materials separately rather than as complements. While the cascade/ToT model does allow for significant scale, there are risks about quality especially for those further down the cascade. Head Teachers Perspectives on approach—and HT role in overseeing instruction—have changed. 13. HT attitudes changed on oversight of reading, teaching, and learning. There is some behavior change in terms of follow-up. 14. More training is needed on leadership skills. Further implementation of CoL and SIS offer opportunities to show leadership. HTs have started to better oversee teaching and learning, including via classroom observation and facilitation of the CoL. In many regions, schools are actively using TP materials. In others, schools still use materials cautiously in order to preserve them. 17 The full report according to NECTA is yet to be published. 4 Evaluation Question 3 Findings Conclusions Local Government LGAs have increased knowledge on reading and demonstrate behavior change by shifting their approach to oversight and teacher support. 15. Involvement of local government in implementation has resulted in initial shifts in capacity and approach to ensuring quality reading instruction at the local level. 16. There is variation at the local government level in terms of implementation of TP and ownership. Joint monitoring visits with TP alongside QAs/WEOs are useful for improving government QA skills. Some local governments have taken strong ownership of the program, but others have more limited interaction with TP. QAs insist their jobs title is now quality assurers as opposed to the previous name, inspectors. National Government Some national-level government actors are more involved than others. Factors outside of TP’s control influence in the level of involvement. 17. Government turnover means loss of knowledge and difficulty coordinating. 18. There is emerging internalization of TP approaches at the national level. 19. There opportunities to clarify roles in 3R sustainability. High-level officials believe sustainability lies at the district level but at the same time, mention possibilities of national level changes. TP had engaged in direct coordination with the government in the past, but recent changes in the make-up of government ministries mean TP may have to reengage with the new government stakeholders directly in collaborative planning. National stakeholders feel local level involvement is key. Some institutions have been influenced by TP. TIE feels they can reproduce similar content based on training and internalized the ability to write the stories and the NECTA assessment approach. 4.4 EQ4: HOW CAN TP’S ACTIVITIES BE DONE MOVING FOWARD IN ORDER TO MORE EFFICIENTLY IMPROVE LEARNING OUTCOMES FOR THE 3RS? Note that EQ4 is inherently asking for recommendations, i.e., what to do (either differently as originally noted in the SOW or moving forward, the current question). To avoid overlap with the recommendation section at the end of this report, broader suggestions on how to move forward are consolidated there. This EQ4 section focuses specifically on technical fidelity of implementation recommendations based on data from the DPLA.18 DPLA is an approach that provides QAs with low-cost, rapid, district-level school monitoring data on school inputs, teacher practice, and student performance twice per school year. DPLA uses a sampling method known as Lot Quality Assurance Sampling (LQAS), which allows for district and regional-level decision-making to be made from a small sample of schools per district. The approach was designed to improve upon existing school monitoring tools in Tanzania and to build the capacity of QAs to collect and analyze actionable monitoring data in order to be able to improve teacher and student performance in the early grades.19 DPLA focuses on grade 2 reading. Data was collected across a large range of indicators broadly falling into three categories: 1) School Inputs, 2) Teacher Practice, and 3) Student Performance. More specifically, school inputs included teacher attendance, pupil attendance, teacher materials, student materials, and pupil exercise books. Teacher practice included pupil engagement, time on task, teaching reading, and teacher preparedness. Student performance included four skills as measured by a group administered literacy 18 The ET’s conclusions on reading performance based on the DPLA report might be more conclusive after RTI's Midline findings are released for EGRA later this year. 19 DPLA draft provided by RTI 10/25/18. 5 assessment (GALA): letter sounds, invented words, reading comprehension, and sentence dictation.20 Performance There are time lags between the provision of materials and the training of educators and changes in student performance. Research suggests measurable change in student performance takes 10 years.21 Gains have been made in the provision of a large number of high-quality materials and internalization by student participants, however, performance remains low across all regions as shown in Table 7 below according to DPLA Phase III October 25, 2018 data on percentage of schools meeting student performance standards by region. As can be seen below, only one district had more than 50 percent of schools able to meet the standard for letter knowledge (Iringa with 73 percent). Only 34 percent of the schools in Zanzibar met the threshold. The greatest percentage of schools for a district with reading comprehension was only 7 percent, in other words, only 7 percent of schools in Ruvuma met the threshold for reading comprehension (note Ruvuma was not sampled in this performance evaluation). Furthermore, only one district had more than 50 percent of schools able to meet the standard for invented words (again Iringa). However, there is variation by skill type and also improvement (as noted in EQ1). Low Order Skills While performance remains low, there is variation by skill type and this (much like time lag) is important to understand in terms of process of literacy development. Lower order skills must be developed prior to developing higher order skills.22 There is relevant, visible improvement in lower order skills in the last year including increases of up to 30 percent in letter recognition in Zanzibar and Ruvuma, and double￾digit increase in Iringa and Mtwara. Decoding (represented by the invented words category) also shows double-digit progress in Ruvuma and Zanzibar, with some gains in Iringa and Mtwara. As seen above, there is low performance in the higher order skills, for example reading comprehension, which shows limited changes and, as it stands, no district had more than 25 percent of schools meeting the standard for reading comprehension. Only 1 percent of schools in Mtwara met the standard. Furthermore, no district had more than 25 percent of schools meeting the standard for dictation. Table 7: DPLA Phase III - Percentage of Schools Meeting Student Performance Standards by Region and Percentage Gain Performance Standard Year Iringa Mtwara Ruvuma Morogoro Zanzibar Letter Knowledge 2018 73 39 45 50 34 +13 +18 +30 +3 +29 Invented Words 2018 51 20 32 39 14 +8 +5 +16 -9 +12 Reading Comprehension 2018 2 5 7 4 3 +2 +1 +6 +2 +3 Dictation 2018 24 8 8 8 6 +21 +2 +7 +2 +5 One explanation for low student performance comes from the continuing issues with attendance. While improvements were made this year in terms of both teacher attendance and the availability of teacher materials (across nearly all regions), the percentage of schools with at least 90 percent of teachers in 20 DPLA draft provided by RTI 10/25/18. 21 See for example Hanusheck 1990. 22 See the work of Catherine Snow, Paton Tabors, Mariella Paez, 2003, among others. 6 attendance at the start of the school day was still under 60 percent overall. Pupil attendance and pupil material availability standards remained even lower. Ultimately, if teachers and pupils are not in school and they do not have the standard materials required for teaching and learning, it is difficult to expect large improvements in student reading ability.23 Similarly, there are few resources for reading in the home. In the small survey the ET conducted with parents, for the majority, the most common material to read in the home was either a religious book or the cellphone. Far fewer had newspapers, magazines, or books. TP story books were also noted to be at home by more than half (38) of PTP parents (see Figure 11). A curricular specialist at TIE also recommended that it is not simply orienting the parents but orienting them in the TP approach on how to help children read. This is crucial in terms of the link between school and home (as well as between IR1 and 3 of the TP). Findings show that some teachers are struggling to implement phonics methods in classes and feel they need more support and training on how to implement the methodology. One teacher stated, “I am expected to stand and pronounce a sound which I cannot even remember how the tutor pronounced [during training].” Based on all available information from DPLA findings, qualitative field work, and literature,24 the ET suggests some specific drill down topics for phonics which may help to improve FOI. The most important reason for DPLA is to use data and evidence-based findings to improve FOI. In the end, it is to better serve the children and teachers in schools with phonics techniques that can create real changes, not only in letter recognition and decoding but also in reading comprehension and dictation. First, guidelines are presented to address remaining challenges with lower order skills. Then, guidelines are presented to maintain and strengthen implementation. Finally, guidance is provided to address higher order skill deficiencies. See phonics recommendations in textbox below. Special attention should be placed on increasing vocabulary and oral reading fluency in order to get more children to read with comprehension. These are found in an order of importance to reinforce lower order skills as the building blocks for higher order skills. Teachers from field work reflect that the phonics approach and learning phonemes is still a difficulty, for example some state: “for me it is difficult recognizing and teaching the sounds of the alphabet.” The textbox on the next page provides guidance on the technical approach to phonics. Please refer to the USAID-funded Global Reading Network for resources on teaching phonics, especially the Early Grade Reading Assessment (EGRA) Toolkit found on https://www.globalreadingnetwork.net/resources/early￾grade-reading-assessment-egra-toolkit-second-edition or the Education Links website with literacy resources just released with USAID’s new education policy found on https://www.edu￾links.org/topics/reading-and-literacy. See specifically page 19 of the EGRA Toolkit referenced above Figure 11: Materials Available in the Home (N=69) 23 Draft of DPLA 2018. 24 https://my.vanderbilt.edu/specialeducationinduction/files/2011/09/1-Literacy-teaching-guide-phonics.pdf 7 regarding the five components of reading and their description.25 The textbox below contains additional reporting and field reflections on DPLA. DPLA is a sizable and noteworthy initiative under the Activity and is worth presenting as it relates to participation with stakeholders. Furthermore, it is important to note that according to the DPLA (2018): 1) follow-up support and planned visits have been jointly developed with the QAs; 2) QAs provide school-level feedback; 3) WEOs are able to follow up; and 4) DPLA has gained acceptance. This was documented and validated by the ET. Now is the time to capitalize on the momentum of DPLA (see Recommendation 4 in the Recommendations section of this report). Table 8: EQ4 Findings and Conclusions Evaluation Question 4 Findings Conclusions Performance Only one district had more than 50 percent of schools able to meet the standard for letter knowledge. 20. Student performance remains a major stumbling block across all regions and implementation of FOI recommendations based on DPLA data has the potential to improve performance. The greatest percentage for a district with reading comprehension was only 7 percent. Only one district had more than 50 percent of schools able to meet the standard for invented words. Low order skills No district had more than 25 percent of schools meeting the standard for reading comprehension. 21. Performance is less problematic with low order skills (such as letter sounds and decoding) and implementation of FOI No district had more than 25 percent of the schools meeting the standard for dictation. 25 Five components are generally accepted as necessary to master the process of reading: phonological awareness, phonics (method of instruction that helps teach sound–symbol relationships), vocabulary, fluency, and comprehension (Armbruster,Lehr, & Osborn, 2003; Vaughn & Linan-Thompson, 2004). The skills within each component are not sufficient on their own to produce successful reading, but they build on one another and work together to reach the ultimate goal of reading comprehension. (Page 19 of the EGRA Toolkit, https://www.globalreadingnetwork.net/resources/early-grade-reading-assessment-egra-toolkit-second-edition) DPLA Additional Reflections Q2 2018 TP report: According to the Quarter 2 2018 Report, DPLA has received government buy-in. In addition, the Quarter 2 2018 Report confirms that data has been collected by QAs and that school assessment results are presented to schools and communities. In the field: The KII with PORALG confirmed government buy-in for DPLA and several QAs confirmed that they administer DPLA. “We listen to the sounds, dictation; we share results with the school and REO.” Stakeholders in Mtwara spoke up without solicitation about DPLA more than other region. The ET did not hear about DPLA from HTs or other actors more widely. In addition, QAs had positive views on DPLA as evidenced in the quotes below: “DPLA is very good because before we could only assess a few schools due to a shortage of funds but with DPLA we have better assessment” – KII Inspector “DPLA allows follow up of schools for evaluating how the three Rs are doing.” – KII Inspector “TP brought a very good approach for assessment, the DPLA, it is for pupils in the second grade. We assess many aspects relevant to reading.” – KII Inspector 8 Evaluation Question 4 Findings Conclusions All districts had more than 25 percent of the schools meeting the standard for letter knowledge. recommendations based on DPLA data has the potential to improve skills. All districts on mainland had 20 percent or more of the schools meeting the standard for invented words. 4.5 EQ5: WHAT ARE THE FACTORS THAT HAVE IMPLICATIONS FOR SUSTAINABILITY OF QUALITY 3RS INSTRUCTION AFTER TP HAS ENDED? The sustainability of a project relates to the political will and ability to be maintained at the national level as well as the commitment and capabilities to be maintained at the local level. Furthermore, the FOI or the technical quality under which the project is maintained is key to examine as this ensures what is sustained is actually the TP Activity. Local Sustainability As seen in EQ3, CoLs are crucial for sustainability at the teacher and the ward level. In addition, LGAs feel empowered. For example, one REO said, “The good thing is that from the beginning I was involved. I was involved in the planning and the training.” He went on, “it is a good thing that the local authorities were involved, including the selection of who should be involved.” He concluded, “It is good to collaborate; this should continue in the future.” It is possible that the local actions provide an example for the national level; in field work it was confirmed that invitations for training come from the districts, not the project. However, there are resource needs at the local level, especially as pertains to the transport of materials and transport of staff for mentoring and coaching. One REO (in Pemba, Zanzibar) shared, “I use a 15-year￾old motorcycle and I pay for the gas/petrol and the maintenance. But it still is difficult to make follow up visits.” In other words, he uses his own resources. Despite the limitations, there is a creative and optimistic attitude at the local level. With regard to mentoring, the WEOs make two school visits a month the QAs make one visit every two years. These visits are sometimes conducted as joint visits with TP staff, as mentioned in EQ3. The joint visits cover the ministry responsibilities and the TP responsibilities and are used to reinforce training. According to KIIs with eight QAs, they conduct classroom observations and assess the methodologies. This helps ensure the FOI. See the Recommendations section for a discussion of how providing evidence for the results from these expenditures will be crucial for making a convincing argument to invest in coaching/mentoring in the future, as it is crucial to improve learning. Teachers say coaching will be useful. It is crucial to differentiate on the issue of local sustainability between Mainland and Zanzibar. It is also necessary to be careful with causality in the sense that TP is operating within the context of pre-existing differences between Zanzibar and the Mainland as there are historical, political, and geographical differences, which contribute to how and why the education systems have developed differently and, in turn, how TP has been implemented within the context of those systems. In the case of Zanzibar, there are typically lower performance rates on assessments as seen in EQ1. Given the geographic location of Zanzibar (as an island), operating within a different political administration than the Mainland requires civil servants (such as DEOs and REOs) to use their own resources and figure out how to accomplish the task at hand. It is this type of beneficiaries who are especially motivated to maintain TP. In some cases, WEOs can creatively come up with fuel payments, but lack of fuel often remains an impediment to being able to visit schools. In addition to building WEOs capacity for site visits, addressing the issue of fuel (either by government or other stakeholders) will need to be considered for the sustainability of the important site visits. 9 Technical – Sustainability TP has worked to construct a sustainable base. At the same time, there needs to be quality assurance of the technical attributes of the activity (for example, mentoring, videos, focused CoLs). KIIs and FGDs with local stakeholders mentioned the following challenges: children are not attending school, there are insufficient materials, there is not enough training for teachers and other education actors, there are not enough classrooms or schools, there are not enough teachers, there are large class sizes, there is a lack of nutritional programs, and there is a the lack of funds to monitor schools. Large class size is a challenge (11 KIIs) and relates to the findings in EQ1 with regard to the quantity and quality of materials provided, as well as the scripts found in the teachers’ instructional tool for guiding instruction in large class sizes. Similarly, due to the large class sizes, although a lot of materials have been provided, materials may still be needed for the future (as they may deteriorate or as class sizes continue to grow) (11 KIIs). Related to large class size and materials is the role of training. Findings show that there is not enough training/coaching/mentoring (6 KIIs) to date (this includes going more in depth on topics) and that teachers demand expansion into arithmetic (mentioned for all of Standard 1 and 2 per the TP contract). Also, training teachers on classroom management techniques is required. Lastly, there is a challenge faced without funds to monitor schools. Although Figure 12 notes all challenges explained by KIIs, challenges such as infrastructure or contracting teachers is outside of the scope of TP. With regard to training HTs, teachers requested more coaching/mentoring. For example, one HT said, “more time is required so everyone can understand.” Also, as described under EQ1, since participants like the reading and writing methodologies, they also want arithmetic methodologies. Furthermore, according to one TP specialist in the field, “there are deficiencies in the teachers’ methodologies with arithmetic, they do not know how to teach arithmetic, for example, even when they have sticks, stones, and their own fingers teachers do not use these to teach arithmetic.” The TP Activity has already made adjustments to improve the technical sustainability. Originally, the activity worked with universities, then they trained districts and then rolled out wards and schools (i.e., the cascade approach). This approach tended to be theoretical, and less practical. Therefore, for increasing technical sustainability the TP activity made course corrections as mentioned in EQ3. In particular, they have extended the training to five days. It includes one day on the principles of adult learning, two days on technical aspects such as phonics and the teacher instructional tool, one day on CoLs, and one day on HT and WEO supervision. This was noted during field work. Figure 12: Challenges Mentioned by Participants *note N= 27 KIIs who identified challenges 10 The new approach is considered highly participatory, including engaging small work groups to refine skills. From there, the HTs and Academic Officers go back and teach teachers and others how to use the instructional tool and the decodables in the CoLs. As explained in more detail in the Recommendations section, more materials (13 KIIs) and more arithmetic (seven KIIs) are recommended. This again includes printing and distributing materials in arithmetic. TP funds can be used to support arithmetic training and funds can be leveraged from other DPs for materials. Other recommendations included resources for transportation, mentoring, parental involvement, and government involvement. National Sustainability At the national level, the country is divided into regions. Each donor is working in one set of regions (i.e., TP works in four regions and Zanzibar), whereas EQUIP-T works in nine regions, UNICEF in four regions (but not all schools in some regions), and the government, with funds from the GPE Literacy and Numeracy Education Support (LANES), works in the remaining regions. The TIE Director notes, “I am not comfortable with the way the regions are divided, there are 27 regions and the result is that we are dividing the country into pieces.” DPs use different approaches given their funding mechanisms, contractual agreements, and technical perspectives and expertise. There is demand for TP throughout the country, as noted in a KII with USAID. The TP Activity aims to respond to the governments’ needs (and has demonstrated its commitment to working with the government, for example, including arithmetic when the contract allows it, and pre￾primary as a small pilot to test the intervention which are not typically in USAID’s early grade reading programs). To do so, the Activity is in the process of creating the unified approach with TIE using the teacher instructional tool (as described in EQ1). USAID’s new Education Policy allows for greater flexibility with regards to numeracy and other things that can be considered in the future. By creating a large number of materials and the tool, TP has created materials that are representative of the Tanzanian context and could be easily utilized beyond the TP pilot regions. These materials and tools can be used as a base to build the dialogue with the government for sustainability. DPs (such as DFID, the donor funding EQUIP-T), government and the project itself agree that it is valuable that the program focus on building government capacity, collaborating and supporting the government, and learning from implementation for government ownership. This is currently happening with the way the official teachers’ guides are being developed and validated in partnership with the TIE for each standard and for each subject. As indicated in EQ3, at the national level there is need for more ownership and internalization for sustainability. Figure 13: Recommendations from Participants to Ensure Sustainability *note N= 21 KIIs who provided recommendations 11 Table 9: EQ5 Findings and Conclusions Evaluation Question 5 Findings Conclusions Local Level Teachers reflect on the value of the CoLs to reinforce and clarify after trainings. 22. The seeds for sustainability are planted on the ground and will be achieved at the school/ward level via the CoLs and local entities. Authorities have been involved in and feel a sense of ownership for the TP Activity. Technical – Sustainability Teachers, HTs, and authorities request specialized trainings/coaching/mentoring to drill down on topics. 23. More coaching is required on specific topics at the school level directly from people with expertise. Teachers, HT, and authorities request expanding in arithmetic. 24. There is a need for TP in spaces such as arithmetic in Standards 1 and 2. National Level Many high-quality materials have been produced for the four regions and Zanzibar. 25. While TP provides materials to four regions, the approach is parceled and ultimately the GOT will need to take ownership for the materials and collectively mobilize the DPs for scale up and expansion. The rest of the country does not have access to these materials. According to high-level KIIs, GOT ultimately will need to be the owner of the materials. 5.0 RECOMMENDATIONS This section presents the recommendations for the TP Activity. As seen below the recommendations are supported by the conclusions, which are based on the evidence and findings. Largely, the recommendations apply to Mainland Tanzania and Zanzibar with the exception of SIS, which requires a more detailed and developed approach. Table 10: Recommendation 1 Recommendation 1: The TP needs to strategically engage the central government actors in planning in order to move forward. Supported by the following conclusions 3. There is commitment by TP and interest by the GOT to work together to align the TP with national needs and curriculum. 4. Some government entities have been involved in the production of materials and need to continue to be involved. 9. Local and national support that require defining, strengthening, and aligning in order to effectively use, maintain, and sustain the SIS. 17. Government turnover means loss of knowledge and difficulty coordinating. 18. There is emerging internalization of TP approaches at the national level. 19. There opportunities to clarify roles in 3R sustainability. High-level officials believe sustainability lies at the district level but at the same time, mention possibilities. 12 Recommendation 1: The TP needs to strategically engage the central government actors in planning in order to move forward. 25. While TP provides materials to four regions, the approach is parceled and ultimately the GOT will need to take ownership for the materials and collectively mobilize the DPs for scale up and expansion. There is interest on the part of the GOT to work with TP. Over the first two years, TP worked with TIE and also worked at the local level. Therefore, TP is willing and trying to work with the GOT. At this time the TP should work closely with other GOT institutions, namely PORALG (and MOEVT). Specifically, to do this the TP can 1) engage a government strategy consultant and 2) design a government engagement plan with indicators. The consultant would serve an intermediary role between the Activity and the GOT. While the TP has PMP indicators (aligned to results), there are no indicators that measure “government internalization and sustainability;” the strategic plan would outline these targets and be the SOW for the consultant. Furthermore, as part of the engagement with PORALG, a clear plan needs to be developed for sustainability and national-level technical capacity related to SIS. This requires planning in terms of programing, ongoing maintenance, and budgetary support. As part of this plan, roles and responsibilities need to be defined, including the role of the stakeholders at the local level for ongoing repairs and updates of tablets as well as training support to staff (including REOs, DEOs, and HTs). Special attention (planning and budgeting) should be paid to rolling out SIS in Zanzibar given the distance (rural and remoteness) which inhibits effective uptake of SIS. Table 11: Recommendation 2 Recommendation 2: TP should take measures to ensure FOI by strengthening the CoLs and reinforce quality of teaching practice through the provision of follow-up teaching modules on specific topics. Supported by the following conclusions 5. Gains appear to have been made with regard to teaching-learning in Reading/Writing and not yet in Arithmetic. 8. There remain concerns (SIS) in terms of ICT and human capacity which require attention as SIS rolls out. 10. Attitude changes on how to teach the 3Rs signal possibilities of longer-term shifts in teaching, and initial use of some new approaches point towards teacher change. 11. CoLs have a strong role in supporting ongoing internalization. 12. More support and tailored training is needed to adjust ideal approaches and methods to local realities. 22. The seeds for sustainability are planted on the ground and it will be achieved at the school/ward level via the CoLs and local entities. The CoLs have the potential to sustain implementation of TP methodologies. The CoLs need to be strengthened and expanded beyond teachers (and WEOs) in schools and wards. If the CoLs continue to engage only school-level actors with some input from WEOs, there may not be sufficient technical inputs for quality implementation. Innovative ideas must be offered to CoLs over the long-term to ensure ongoing learning. Sources of information for the CoLs could be 20 to 30-minute videos by Tanzanians available on DVD or Internet or sounds on MP3 players. In addition, TP could produce quarterly newsletters with tips (or ideas and guidance) on implementation (see above). 13 Table 12: Recommendation 3 Recommendation 3: The TP should provide follow up through mentoring and coaching. Joint visits between a QA officer and a tutor should be considered. Supported by the following conclusions 5. Gains appear to have been made with regard to teaching-learning in Reading/Writing and not yet in Arithmetic. 8. There remain concerns (SIS) in terms of ICT and human capacity which require attention as SIS rolls out. 10. Attitude changes on how to teach the 3Rs signal possibilities of longer-term shifts in teaching, and initial use of some new approaches point towards teacher change. 11. CoLs have a strong role in supporting ongoing internalization. 12. More support and tailored training is needed to adjust ideal approaches and methods to local realities. 23. More coaching is required on specific topics at the school level directly from people with expertise. Technical strengthening through mentoring and coaching should be provided at the local level. Specific topics need to be determined based on consultation with local stakeholders. During this evaluation, needs emerged regarding: 1) training on how to best utilize DPLA; 2) further support on SIS (this may be addressed by currently planned training); 3) training on specific teaching methods related to phonics such as letter sound pronunciation; 4) ongoing support to follow-on trainings already offered on how to align TP and government materials; and 5) new training for parents on how they can support their children in learning to read even if they have low literacy levels, and how they can offer this support in a way that is aligned with the new reading methods used in schools. In order to ensure sustainability, the coaching needs to be institutionalized within the MOEST. This role should be played by the QAs. At the same time, tutors (from teachers’ colleges) have more technical expertise. They can conduct joint visits with QAs to provide guidance to teachers and WEOs to reinforce best practices from the trainings. Table 13: Recommendation 4 Recommendation 4: The DPLA should be aligned with the structures of the GOT and MOEST to promote ongoing assessment and the provision of information for decision￾making. Supported by the following conclusions 20. Student performance remains a major stumbling block across all regions and implementation of FOI recommendations has the potential to improve. 21. Performance is less problematic with low order skills (such as letter sounds and decoding) and implementation of FOI recommendations has the potential to improve. Findings suggest that the DPLA provides efficient useful information on the low performance of students across the regions. However, there is variation in skill and there have been marked gains. The TP Activity should work with the GOT and MOEST to institutionalize DPLA for ongoing assessment and the provision of information for decision-making. 14 Table 14: Recommendation 5 Recommendation 5: The TP should complete the package of materials for Standard 1 and 2 and complementary materials with arithmetic should be included through partnership. Supported by the following conclusions 1. A large number of materials exist and are crucial to improve reading skills. 2. Materials are good quality in terms of design and content and they motivate children to come to school. 6. TP is a national example for Reading/Writing whereas other DPs (EQUIP-T, for example) can be leveraged for Arithmetic. 24. There is a need for TP in spaces such as arithmetic in Standards 1 and 2. Specifically, with regard to arithmetic, TIE developed new textbooks and accompanying teacher’s guides in Quarter 3 2018. TIE has asked TP to edit the official teaching guides for Standard 1 and Standard 2. In an initial review, the pedagogical approach to arithmetic does not appear to be student centered, as the approach is in reading and writing. Therefore, TP has supported integrating a student-centered (I do-you do-we do) approach into the arithmetic teacher guide and lessons. In addition, the above-mentioned DP￾supported materials should be referenced and integrated. 15 ANNEXES ANNEX I: STATEMENT OF WORK STATEMENT OF WORK Mid-term Performance Evaluation of TP, “Let’s Read Together” PURPOSE OF THE EVALUATION Why the evaluation is being conducted: USAID/Tanzania worked closely with the Government of Tanzania (GoT) with inputs and collaboration from donor partners such as DFID, Swedish SIDA, UNICEF, Global Partnership of Education, and the Canadian High Commission to design TP, Let’s Read Together. TP, USAID/Tanzania’s signature early grade reading and writing activity, ultimate goal is to improve student outcomes in reading and writing. The activity is valued at $67 million over a 5-year period in 4 regions, plus Zanzibar and fully aligns with the GoT’s 3Rs reform, which focuses on reading, writing, and arithmetic. As TP is in its second year of implementation, it is an opportune time to take stock and analyze what is working or not working with components 1 (quality of early grade basic skills instruction improved) and 2 (skills delivery of MoEVT Strengthened), especially in light of the many education policy changes that have happened within the GoT, Ministry of Education, Science and Technology (MOEST), President’s Office Regional Administration and Local Government (PORALG), and primary schools. *Component 3 will not be included as RTI is conducting a Community Engagement Annual Monitoring survey which measures the actions and behavior change of key program stakeholders in 176 schools, notably parents, teachers, and heads of schools. A baseline, midline, and endline will be conducted. In addition, a two-phase study on the facilitation of a community education mobilization and action planning process and the development of parent teacher partnerships is being conducted by the Rapid Feedback Monitoring, Evaluation, Research and Learning (RF MERL) Consortium funded and contracted by USAID. Who will use the results of the evaluation: The results will primarily be used by USAID/Tanzania, RTI and the GoT (MOEST, PORALG). Development partners such as DFID, Swedish SIDA, UNICEF, Global Partnership of Education, and the Canadian High Commission will also receive the evaluation and could use it if they find the recommendations relevant to their work. How will they use it: Mid-term evaluation results will be used as a tool to bring USAID/Tanzania, RTI, MOEST, PORALG, and development partners (DPs) together outside of the Joint Annual Reviews, which have been happening each year, to discuss what the program has achieved in its first two+ years; changes or adaptations that could be made to the project before it ends; and ways for MOEST, PORALG, and other DPs to work together to scale up the program or specific components of TP nationally. SUMMARY INFORMATION Option 1: For strategies, projects, or activities with one implementing partner Strategy/Project/Activity Name Tusome Pamoja Implementer Research Triangle Institute Cooperative Agreement/Contract # AID-621-C-16-00003 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $67 million 16 Life of Strategy, Project, or Activity January 2016-January 2021 Active Geographic Regions 4 Regions, plus Zanzibar-Tanzania Mainland: Morogoro, Iringa, Ruvuma, Mtwara; and Zanzibar: Unguja and Pemba Development Objective(s) (DOs) DO 1: Tanzania’s advance toward middle income status supported: Lifelong learning skills improved USAID Office Education BACKGROUND Description of the Problem, Development Hypothesis(es), and Theory of Change Instructions: Include details on: --The specific problem or opportunity the strategy/project/activity to be evaluated was designed to address; --The development hypothesis(es) often expressed as an if/then statement26; --The theory of change that underlies the design (including a list of the intended results and critical assumptions); -- Results Frameworks: Include here or as an annex the graphic of the Mission’s Results Framework and the Project’s Logical Framework (if applicable) highlighting the elements to be evaluated. If the evaluation is at the Activity level then include the Activity’s Logical Framework (and linkages to the project-level). In all cases, account for changes (if applicable) since the original design. Summary Strategy/Project/Activity/Intervention to be Evaluated The Tusome Pamoja program’s primary goal is to increase the percentage of children who: a) after two years of schooling, can read and comprehend grade level text and solve grade-level arithmetic problems, and; b) after four years of schooling, can read and comprehend grade level text, respond to simple writing prompts, and solve grade-level arithmetic problems. Expected Impact: Approximately 3,027 total public primary schools will benefit, from 34 districts with an estimated 1.4 million children directly benefiting over the five years of the program. No private primary schools will be beneficiaries of Tusome Pamoja. In addition to direct beneficiaries, Tusome Pamoja materials and methodologies will serve as models for 3Rs reform in other regions of the country. Project Overview: Tusome Pamoja will work at the national, district, and ward levels—and to some degree the regional level—to build the capacity of the Tanzania Institute of Education (TIE), Zanzibar Institute of Education (ZIE), MoEST, Z/MoEVT, and PORALG on several policy and institutional issues. This means (1) working with +key stakeholders on teacher guides, student materials, and the training to use them; (2) building coaching and mentoring support; and (3) collecting data for evidence-based decision making and change. Quality of Early Grade Basic Skills Instruction Improved Collaborate with TIE and ZIE to develop appropriate instructional materials for the 3Rs. Work with governmental partners to conduct a comprehensive review of existing frameworks, curricula, standards, and student and teacher materials. 26 If the design document does not contain an implicit development hypothesis, consult with the DO Team to develop the development hypothesis. 17 Review research that has been done on existing teacher practices in Tanzania and develop a training manual that supports teachers in adapting existing practices into new, research-based practices. Training sessions will focus on teaching content and instructional strategies that cut across the reading, writing, and arithmetic lessons. Collaborate with DEOs and the Chief Inspector at the regional and district levels to develop a model for regular contact time between school principals and inspectors who provide constructive feedback on teacher strengths and areas of improvement in lesson planning and teaching. Work with the TIE, the ZIE, and pre-primary education units in the respective Ministries to develop reading and writing teaching and learning materials for the pre-primary level preceding Standard 1. Build the capacity of Ward Education Officers (WEOs) and Training Resource Center (TRC) tutors as key school support resources able to provide consistent, frequent, and targeted instructional support. The project may use technology – such as tablet-based applications for student assessment and cellphone SMS. Skills Delivery System of MOEST and MoEVT Strengthened Build the capacity of counter-parts at national, regional, district and local levels, through every phase of data collection and, when possible, data analysis and its implications for policy. Provide training workshops and follow up support to ensure that managers and supervisors are fully equipped especially for the task of mentoring and coaching teachers. Classroom observation tools and coaching tools including SMS messages will be compiled into a quality assurance package for 3Rs. Reinforce the capacity of MoEST, PO-RALG, and the Z/MoEVT to monitor and support school-level change in order to sustain the ambitious objectives of the 3Rs. Effective Engagement of Parents and Community in Education Increased Build the capacity of parents and the schools’ communities to assist children in practicing and developing 3Rs in the home. Help parents and communities discuss and address issues of school safety; understand the importance of including children with disabilities in schools; encourage girls to go to school and stay in school; and create welcoming and child-friendly environments in and around the school Modifications: The introduction of the school information system (SIS) as part of the institutional strengthening of system assessment component. Further modifications were made to the timeline to take into consideration critical factors, such as GOT approvals, institutional processes to arrive at national ownership, and external factors such as copyright, and, in some cases, development partner alignment. Geographic Location: Tanzania Mainland: Morogoro, Iringa, Ruvuma, Mtwara; and Zanzibar: Unguja and Pemba Partners: Ministry of Education, Science, and Technology (MOEST), the President’s Office for Regional Administration and Local Government (PO-RALG), and Zanzibar Ministry of Education and Vocational Training (Z/MOEVT) Summary of the project/Activity Monitoring, Evaluation, and Learning (MEL) Plan Instructions: Specify what relevant documents will be available to the evaluators. In particular, identify the existence and availability of relevant performance information sources, such as performance monitoring indicators and/or previous evaluation reports. In addition, identify any other documents or sources of information from outside of USAID that would be useful to the evaluation team (e.g., government or international data). If this section is long it may also be included in an annex. 18 USAID/Tanzania will provide Data for Development the following documents from the Tusome Pamoja project: • Quarterly progress reports • Financial reports • Annual progress and financial reports • Success stories • Work plan • Mobilization plan • Performance Management and Evaluation Plan • GPE Lanes project description • DFID Funded EQUIP Project description • DPLA • Other documents as requested RTI will provide Data for Development a package of all teaching and learning materials which they developed. The package will include the following for Component 1: • Standard 1 and 2 Decodable readers • Zanzibar Standard 1 and 2 Kiswahili Teaching tool • Mainland Tanzania Standard 1 & 2 Kiswahili Teaching tool • Pre-primary story books (Mtwara only) • Standard 3 & 4 Fiction and non-fiction readers/reading to learn • Ward Education Coordinator Quality Assurance guide • School leadership training materials and training plan • Pre-primary student learning materials and teacher guide • Quality assurance package with teacher observation and coaching tools • Other documents as requested RTI will provide Data for Development reports, assessments, materials and information on Component 2: • Ward Education Coordinator and Teacher Resource Center • Tutor Training materials and training plan • Student Reading and writing assessments • Training plan for administration of reading assessments • Results of reading and writing assessments • Capacity assessment of centralized and decentralized education delivery • Performance Monitoring and Evaluation Indicators: • Number and percent of children who, after two years of schooling, can read and comprehend grade-level text, encode simple sentences, and solve grade-level arithmetic problems • Number and percent of children who, after four years of schooling, can read and comprehend • grade-level text, respond to simple writing prompts, and solve grade-level arithmetic problems • Number and percent of learning environments with the minimum required characteristics (books, trained teachers who arrive on time, absence of violent behavior) for successful 3Rs learning • Number of evidence-based policies developed to support the 3Rs reform • Number and percent of early grade teachers implementing Tusome Pamoja 3Rs interventions successfully • Number and percent of regional education offices capable of coaching with 3Rs teachers and assessing student learning in the 3Rs 19 • Number and percent of parents and community members engaging with 3Rs students to improve mastery of 3Rs; and • Number and percent of communities holding schools and local governments accountable for quality 3Rs instruction All data will be disaggregated by sex, age, and geographic location. In addition, baseline and midline data will be available to Data for Development for the measurement of all progress toward all targets. EVALUATION QUESTIONS Instructions: Include 1–5 specific questions focused on key program areas and/or performance and directly linked to the purpose of the evaluation and its expected use. Sub-questions or narrative text may be included to elaborate on the main question, but not to add new areas of inquiry. NOTE: Not every aspect of a program, project, or activity needs to be, or should be, the focus of the evaluation. Rather, the evaluation should examine specific aspects of the program, project, or activity where there are questions unanswered by performance monitoring or other data. Guidelines: Questions should be precise. Vague terms that can be defined or applied in a variety of ways (such as “relevance,” “effectiveness,” etc.) should be defined clearly. If any specific terminology or standards are included in the evaluation questions indicate the source or definitions. Questions should be researchable. Questions should have an answer that can be obtained through the use of social science methods and tools (qualitative and quantitative) rather than relying on the evaluators’ judgments. Questions should integrate gender. Questions should identify when sex-disaggregated data are expected. Where appropriate, the evaluation questions can include a separate question aimed at evaluating the gender-specific effects of the activity or project. [See the How-To Note on Engendering Evaluation] Questions should be presented in order of priority, or the priority of questions should otherwise be identified. A request for recommendations is not an evaluation question. If you want the evaluators to provide recommendations, describe what aspects of the program, project, or activity you want recommendations to address in a separate paragraph or following the questions. Evaluation Questions: General Program Questions • To what extent is Tusome Pamoja making progress towards improving target beneficiaries' 3Rs skills? • How is the development of a School Information System leading to improved learning outcomes for the 3Rs? • In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by Tusome Pamoja? • How can Tusome Pamoja's activities be done differently in order to more efficiently improve learning outcomes for the 3Rs? • What are the factors that have implications for the sustainability of quality 3Rs instruction after Tusome Pamoja has ended? EVALUATION DESIGN AND METHODOLOGY Mixed-methods mid-term performance evaluation will enable evidence-based answers to all evaluation questions. The following methods could be used: 20 • Initial stakeholder analysis • Document review • Cost benefit analysis • Focus group discussions • One on one interviews with key stakeholders • Case studies • Suggested interviews with: Permanent Secretary MOEST, Deputy Permanent Secretary PO-RALG, Commissioner of Education, MOEST, Director of Policy and Planning MOEST, Acting Director of Basic Education PORLAG, Staff at Tanzania Institute of Education, Staff at Zanzibar Institute of Education, Regional Education Officers (REOs), Regional Administrative Secretaries, District Education officers, Ward Education officers, teachers, head of school, teachers who were trained by TP, quality assurance inspectors (formerly known as inspectors), and the PTPs (UWAWAs), and CEMS (community engagement officers), • Suggested interviews with staff from: GPE Lanes, DFID, EQUIP-T, UNICEF team working on reading programs, Swedish SIDA, Canadian High Commission, and other stakeholders working on early grade reading Protocol and IRB Requirements: A letter should be written to the Permanent Secretary MOEST and Deputy Permanent Secretary PO￾RALG to inform them about this mid-term performance evaluation. USAID/Tanzania suggests involving MOEST and PORALG officials in the actual performance evaluation exercise IRB in Zanzibar, COSTEC and NBS—all take time a long time for approvals so Data for Development needs to plan in advance The following simple design matrix can be included as a summary of evaluation design and methods, and to supplement the narrative section above, but should not replace the narrative. Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. [Insert Evaluation question] [Documents (including. performance monitoring data, previous evaluations, etc.), national statistics, project staff, stakeholders, expert knowledge, beneficiaries…] [Key informant interviews, questionnaires or surveys, focus group discussions, direct observation, desk review…] [To be determined by evaluation team] [Requested level of disaggregation—gender, ethnicity, location (district, province), etc.…] 2. [Insert Evaluation question] ditto ditto ditto 3. [Insert Evaluation question] ditto ditto ditto Notes: (*) It is acceptable to include data sources that do not need to be collected but may be analyzed by the evaluation team. In planning for and preparing the Evaluation SOW it is a good practice to examine available data sources especially performance monitoring data. 21 DELIVERABLES AND REPORTING REQUIREMENTS Evaluation Work plan: Within 2 weeks of the agreement meeting with R4D and USAID/Tanzania’s Education team, a draft work plan for the evaluation shall be completed by the lead evaluator and presented to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. Evaluation Design: Within 3 weeks of approval of the work plan, the evaluation team must submit to the Agreement Officer’s Representative/Contracting Officer’s Representative (AOR/COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. [If applicable add a requirement to include a conflict of interest mitigation plan based on the Disclosure of Conflict of Interests submitted with the awardee’s proposal]. USAID offices and relevant stakeholders are asked to take up to 10 business days to review and consolidate comments through the AOR/COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 days. In-briefing / inception report: Within [ # days] of arrival in [specify location], the evaluation team will have an in-briefing with the [insert offices/audience] for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), if necessary. [The in-briefing could take place after the evaluation team has had the chance to conduct a desk review or examine secondary data.] Mid-term Briefing and Interim Meetings: The evaluation team is expected to hold a mid-term briefing with [specify USAID offices and/or staff] on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. Final Exit Briefing: The evaluation team is expected to hold a final exit briefing prior to leaving the country to discuss the status of data collection and preliminary findings. This presentation will be scheduled as agreed upon during the in-briefing. [Specify guidelines of the presentation, e.g., who should be included, such as implementing partner staff or other stakeholders; preferred medium (joint or separate briefings); and expected maximum length] Final Presentation: The evaluation team is expected to hold a final presentation in person/by virtual conferencing software to discuss the summary of findings and recommendations to USAID. This presentation will be scheduled as agreed upon during the in-briefing. [Specify guidelines of the presentation, e.g., who should be included, such as implementing partner staff or other stakeholders; preferred medium (joint or separate briefings);expected maximum length; and timing (before or after the final report)]. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, [insert office/s]will have [number] business days in which to review and comment on the initial draft, after which point the AOR/COR will submit the consolidated comments to the evaluation team. 22 The evaluation team will then be asked to submit a revised final draft report [number] business days hence, and again the [insert office/s]will review and send comments on this final draft report within [number] business days of its submission. [A good practice is for the evaluation team to share an early draft or detailed outline that includes main findings and bullets before finalizing the draft evaluation report] Final Evaluation Report: The evaluation team will be asked to take no more than [number] business days to respond/incorporate the final comments from the [insert office/s]. The evaluation team leader will then submit the final report to the AOR/COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. Evaluation deliverables: • Initial stakeholder analysis • Scope of Work • Design/inception report • In briefing with USAID before beginning field work • Debriefing with USAID after conclusion of field work and before the analysis • Stakeholder findings workshop • Draft evaluation report • Final report—will adhere to PPL’s guidance on evaluation report requirements. The evaluation team shall incorporate USAID’s comments and submit final report to USAID in electronic format as well as printed and bound copes. R4D will submit the final evaluation report to the Development Experience Clearinghouse within 90 days of approval. EVALUATION TEAM COMPOSITION Evaluation Team composition: The evaluation team will be composed of four individuals: team leader, 2 senior technical experts, and a logistician. Team Leader/Senior Evaluation Specialist: S/he will be responsible for coordinating the activities of the evaluation team, and have the authority to make budgetary and programmatic decisions regarding the evaluation. S/he will serve as the main point of contact between USAID and the contractor’s headquarters. The Team Leader will approve the final evaluation design, oversee the development of evaluation instruments, integrate the findings of different team members and coordinate the preparation of the final reports. The Team Leader should have at least ten years of experience in the administration of multi￾faceted education projects in developing countries - preferably in East Africa. S/he should have experience in managing multi-disciplinary teams and developing and conducting qualitative evaluations, the ability to conceptualize and structure evaluation activities and write clearly and concisely. An MA or PhD in education administration, planning, economics of education or similar field is required. Education Specialist: The composition of the two subject matter specialists will depend on the final evaluation questions, but one should be a methodologist (evaluation specialist) and the other a sectoral specialist (Education, early grade reading). The Education Specialist will hold an advanced degree in Educational Research and/or Statistics and have extensive experience in evaluating educational programs. S/he will have at least 8 years of experience working with educational projects in developing countries, preferably in Africa. 23 Tanzanian Education Context Specialist(s): The local consultant shall have experience and knowledge about the education context in Tanzania, particularly on early grade reading and numeracy, working with the President’s Office, Regional Administration and Local Government (PORALG), Ministry of Education, Science and Technology (MOEST), and the district level regional education offices. Logistics Coordinator: S/he will serve as the main logistical coordinator for the performance evaluation, and work with local partners to plan travel, data collection, interviews and assessment activities as required. S/he will work with the Team Leader to review, edit, and format the final report of the evaluation, prepare it for production, supervise the production and distribute it to USAID/Tanzania. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. The [insert name] will participate on the evaluation team in [describe role]. See Guidance for USAID Staff Participation on External Evaluations for language. The COR of the Evaluation may observe [insert all or some] of the data collection efforts. EVALUATION SCHEDULE Sample Format: Illustrative Schedule Timing (Anticipated Months or Duration) Proposed Activities Important Considerations/Constraints - Preparation of the work plan and evaluation design - - USAID review of the work plan and evaluation design Take into account availability in the Mission or Washington OU - Travel [optional: evaluation design] and preparations for data collection Take into account visa requirements (if an expatriate team is being mobilized) - In-Briefing - - Data Collection Take into account the number of sites, methods, sectors, etc. - Data Analysis Take into account the number of sites, methods, sectors, etc. - Report writing Take into account the number of sites, methods, sectors, etc. - USAID review of Draft Report Take into account availability in the Mission or Washington OU - Incorporate USAID comments and prepare Final Report - 24 Sample Table: Estimated LOE in days by activity for a team of four Task LOE for Expat Team Lead LOE for Expat [subject matter] Specialist LOE for Local [subject matter] Specialist LOE for Local [subject matter] Specialist Total LOE in days Document review/desk review/work planning (evaluation design remote or in￾country) - - - - - Preparations for travel and organizing data collection (contracting translators, vehicles, etc.). - - - - - In-brief, Evaluation Design (including meetings with USAID) - - - - - Preparations for data collection (scheduling) - - - - - Data collection days by method by site - - - - - Data analysis - - - - - Briefing - - - - - Draft final report and debrief to USAID [include time for translation if necessary] - - - - - Final report - - - - - Totals - - - - - Sample Table: Estimated LOE in days by position for a team of four Position Preparation Travel to/from Country In-Country Data Collection Finalization of Report Total LOE in days Expat Team Leader - - - - - Expat Specialist - - - - - Local Specialist - - - - - Local Specialist - - - - - 25 Position Preparation Travel to/from Country In-Country Data Collection Finalization of Report Total LOE in days Totals - - - - - FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: The Evaluation SOW; Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; All sources of information, properly identified and listed; and Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. Any “statements of difference” regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team. Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.27 27 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 26 Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. Evaluation methodology should be explained in detail and sources of information properly identified. Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether Select those that are applicable and included: in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. LIST OF ANNEXES Instructions: Include annexes to the SOW that will help the evaluation team design an effective proposal. This includes primary USAID guidance documents, publically available reports and data on the strategy/project/activity to be evaluated, and prior evaluation, etc. 27 ANNEX II: DESIGN . USAID/Tanzania Data for Development Project EVALUATION PLAN AND WORK PLAN PERFORMANCE EVALUATION FOR TUSOME PAMOJA ACTIVITY (Let’s Read Together) Contract No. AID-OAA-1-15-00024/AID-621-TO-17-00005 May 21, 2018 This document was produced for review by the United States Agency for International Development Tanzania Mission (USAID/Tanzania). EVALUATION PLAN and WORK PLAN: PERFORMANCE EVALUATION OF USAID TANZANIA’S TUSOME PAMOJA ACTIVITY (Let’s Read Together) EVALUATION PLAN AND WORK PLAN PERFORMANCE EVALUATION FOR TUSOME PAMOJA ACTIVITY (Let’s Read Together) USAID/Tanzania Data for Development Project Submission Date: May 21, 2018 Contract Number: AID-OAA-1-15-00024/AID-621-TO-17-00005 Activity Start Date and End Date: COR Name: Angela Mwaikambo Prepared by: Jacob Laden, Evaluation Advisor Data for Development, NORC at the University of Chicago Ritu Nayyar-Stone, Principal Research Scientist, NORC at the University of Chicago Submitted by: ME&A (Mendez England & Associates) 4300 Montgomery Ave. Suite 103 Bethesda, MD 20814 Tel: 0742 788 466 xxix LIST OF ACRONYMS Acronym Description CO Contracting Officer COR Contracting Officer Representative DEC Development Experience Clearinghouse DO Development Objective FGD Focus Group Discussions GoT Government of Tanzania IP Implementing Partner KII Key Informant Interviews LGA Local Government Authority LOE Level of Effort MOEST Ministry of Education, Science and Technology NGO Non-Governmental Organization NORC National Opinion Research Center at the University of Chicago PMP Performance Management Plan PORALG President’s Office-Regional Administration and Local Government SOW Statement of Work STTA Short Term Technical Assistance TIE Tanzania Institute of Education USAID United States Agency for International Development USG United States Government WEO Ward Education Coordinators Officer ZIE Zanzibar Institute of Education Z/MoEVT Zanzibar Ministry of Education and Vocational Training EVALUATION PLAN and WORK PLAN: PERFORMANCE EVALUATION OF USAID TANZANIA’S TUSOME PAMOJA ACTIVITY (Let’s Read Together) EVALUATION PLAN and WORK PLAN: PERFORMANCE EVALUATION OF USAID TANZANIA’S TUSOME PAMOJA ACTIVITY (Let’s Read Together) OVERVIEW AND OBJECTIVES As part of the Contract/Task Order Number AID-OAA-1-15-00024/AID-621-TO-17-00005 Data for Development (D4D) Activity, USAID has asked Data for Development to design and budget for a Performance Evaluation of USAID Tanzania’s Tusome Pamoja Activity (Let’s Read Together) implemented by RTI International (Research Triangle Institute) with funding from USAID from January 2016-January 2021. ACTIVITY BACKGROUND The Tusome Pamoja program’s primary goal is to increase the percentage of children who: a) after two years of schooling, can read and comprehend grade level text and solve grade-level arithmetic problems, and; b) after four years of schooling, can read and comprehend grade level text, respond to simple writing prompts, and solve grade-level arithmetic problems. Expected Impact: Approximately 3,027 total public primary schools will benefit, from 34 districts with an estimated 1.4 million children directly benefiting over the five years of the program. No private primary schools will be beneficiaries of Tusome Pamoja. In addition to direct beneficiaries, Tusome Pamoja materials and methodologies will serve as models for Reading, Writing and Arithmetic (3Rs) reform in other regions of the country. Project Overview: Tusome Pamoja will work at the national, district, and ward levels—and to some degree the regional level—to build the capacity of the Tanzania Institute of Education (TIE), Zanzibar Institute of Education (ZIE), the Ministry of Education, Science and Technology (MoEST), the Zanzibar Ministry of Education and Vocational Training (Z/MoEVT), and the President’s Office-Regional Administration and Local Government (PORALG) on several policy and institutional issues. This means (1) working withkey stakeholders on teacher guides, student materials, and the training to use them; (2) building coaching and mentoring support; and (3) collecting data for evidence-based decision making and change. Quality of Early Grade Basic Skills Instruction Improved Collaborate with TIE and ZIE to develop appropriate instructional materials for the 3Rs. Work with governmental partners to conduct a comprehensive review of existing frameworks, curricula, standards, and student and teacher materials. Review research that has been done on existing teacher practices in Tanzania and develop a training manual that supports teachers in adapting existing practices into new, research-based practices. Training sessions will focus on teaching content and instructional strategies that cut across the reading, writing, and arithmetic lessons. 31 Collaborate with District Education Officers (DEOs) and the Quality Assurors at the regional and district levels to develop a model for regular contact time between school principals and inspectors who provide constructive feedback on teacher strengths and areas of improvement in lesson planning and teaching. Work with the TIE, the ZIE, and pre-primary education units in the respective Ministries to develop reading and writing teaching and learning materials for the pre-primary level preceding Standard 1. Build the capacity of Ward Education Officers (WEOs) and Training Resource Center (TRC) tutors as key school support resources able to provide consistent, frequent, and targeted instructional support. The project may use technology – such as tablet-based applications for student assessment and cellphone SMS. Skills Delivery System of MOEST and MoEVT Strengthened Build the capacity of counter-parts at national, regional, district and local levels, through every phase of data collection and, when possible, data analysis and its implications for policy. Provide training workshops and follow up support to ensure that managers and supervisors are fully equipped especially for the task of mentoring and coaching teachers. Classroom observation tools and coaching tools including SMS messages will be compiled into a quality assurance package for 3Rs. Reinforce the capacity of MoEST, PO-RALG, and the Z/MoEVT to monitor and support school￾level change in order to sustain the ambitious objectives of the 3Rs. Effective Engagement of Parents and Community in Education Increased 28 Build the capacity of parents and the schools’ communities to assist children in practicing and developing 3Rs in the home. Help parents and communities discuss and address issues of school safety; understand the importance of including children with disabilities in schools; encourage girls to go to school and stay in school; and create welcoming and child-friendly environments in and around the school Modifications: The introduction of the School Information System (SIS) as part of the institutional strengthening of system assessment component. Further modifications were made to the timeline to take into consideration critical factors, such as Government of Tanzania (GOT) approvals, institutional processes to arrive at national ownership, and external factors such as copyright, and, in some cases, development partner alignment. 28 Although this component is outside of the scope of this evaluation the design is a holistic approach where in the connection of student reading to parent involvement still vital role; during the evaluation we will be reaching parent representatives of the PTP in order to get parent perspectives on student reading and practice in the 3Rs. 32 Geographic Location: Tanzania Mainland: Morogoro, Iringa, Ruvuma, Mtwara; and Zanzibar: Unguja and Pemba Partners: Ministry of Education, Science, and Technology (MOEST), the President’s Office-- Regional Administration and Local Government (PO-RALG), and Zanzibar Ministry of Education and Vocational Training (Z/MOEVT) II. SUMMARY INFORMATION Figure 1: Summary information on Tusome Pamoja under this mixed methods evaluation Strategy/Project/Activity Name Tusome Pamoja Implementer Research Triangle Institute Cooperative Agreement/Contract # AID-621-C-16-00003 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $67 million Life of Strategy, Project, or Activity January 2016-January 2021 Active Geographic Regions 4 Regions, plus Zanzibar-Tanzania Mainland: Morogoro, Iringa, Ruvuma, Mtwara; and Zanzibar: Unguja and Pemba Development Objective(s) (DOs) DO 1: Tanzania’s advance toward middle income status supported: Lifelong learning skills improved USAID Office Education Office 33 Figure 2: Theory of Change Figure 3: Interventions at Grade Level by Region Region Pre￾Primary Standard 1 Standard 2 Standard 3 Standard 4 Zanzibar X X X X X Iringa - X X - - Morogoro - X X - - Mtwara X X X X X Ruvuma - X X - - 34 EVALUATION DESIGN AND METHODOLOGY I. PURPOSE OF THE EVALUATION Why the evaluation is being conducted: USAID/Tanzania worked closely with the Government of Tanzania (GoT) with inputs and collaboration from Arithmetic such as DFID, Swedish SIDA, UNICEF, the Global Partnership of Education (GPE), and the Canadian High Commission to design Tusome Pamoja, Let’s Read Together. Tusome Pamoja, USAID/Tanzania’s flagship early grade reading and writing activity, ultimate goal is to improve student outcomes in reading and writing at the local, regional and national levels. The activity is valued at $67 million over a Five-year period in four regions, plus Zanzibar and fully aligns with the GoT’s 3Rs reform, which focuses on reading, writing, and arithmetic. As Tusome Pamoja is in its second year of implementation, it is an opportune time to take stock and analyze what is working or not working with components 1 (quality of early grade basic skills instruction improved) and 2 (skills delivery and assessment system of MOEST, PORAG and MoEVT Strengthened), especially in light of the many education policy changes that have happened within the GoT, MOEST, PORALG, and primary schools. *Component 3 will not be included as RTI is conducting a Community Engagement Annual Monitoring survey which measures the actions and behavior change of key program stakeholders in 176 schools, notably parents, teachers, and heads of schools. A baseline, midline, and endline will be conducted. In addition, a two phase study on the facilitation of a community education mobilization and action planning process and the development of parent teacher partnerships is being conducted by the Rapid Feedback Monitoring, Evaluation, Research and Learning (RF MERL) Consortium. Who will use the results of the evaluation: The results will primarily be used by USAID/Tanzania, and the GoT (MOEST, PORALG). DPssuch as DFID, Swedish SIDA, UNICEF, GPE, and the Canadian High Commission will also benefit from the results. How will they use it: Mid-term evaluation results will be used as a tool to bring USAID/Tanzania, MOEST, PORALG, and DPs together outside of the Joint Education Sector Reviews (JESR), which have been happening each year, to discuss what the program has achieved in its first two years; changes or adaptations that could be made to the activity before it ends; and ways for MOEST, PORALG, and other DPs to work together to scale up the program or specific components of Tusome Pamoja nationally. The team proposes to conduct the evaluation using a participatory approach, engaging the USAID Mission, RTI, project beneficiaries, and other stakeholders, through various phases of the evaluation. This includes working collaboratively to: Identify appropriate questions keeping in mind users and uses of the evaluation for Mission, IP, and GoT decision making; Identify pertainent documentation for desk review, including provision of existing data from quarterly and annual reports; 35 Plan field work and review sample frame for data collection; this includes assisting the team in identifying participants for Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs); Review questions for KIIs, FGDs, and the participant questionnaire that solicit responses that address target research objectives; Select appropriate data collection methods and analysis to answer evaluation questions and to best meet the decision making needs of the users of the evaluation; Participate in a participatory workshop to review findings, conclusions, and recommendations to ensure feasibility and utilization; and Serve as feedback providers for reports and other deliverables. At the same time, the evaluation team will remain independent and will take steps to maximize the quality of the information and minimize the impact of various potential sources of bias on the evaluation. Accordingly, IP staff will not be involved directly in data collection activities for the purpose of maintaining objectivity and for insuring respondent/beneficiary confidentiality as they provide feedback. The evaluation will take into consideration the local context and project implementation results by analyzing the achievements of targeted results revealed in existing reporting documents, considering the opinions and recommendations elicited during the KIIs and FGDs, and undertaking quantitative analysis of the results from secondary data provided by RTI. The evaluation team will use these different sources of data to triangulate findings and answer the main research questions outlined in the scope of work (SOW) document provided by USAID. Available data and primary source collection will be disaggregated by appropriate demographics including age and gender/sex, as well as by region and occupation whenever possible. The evaluation design process started in May 4th 2018. The 3-4 week data collection period will begin July 23rd –August 17th and close with a workshop session in Dar Salaam with USAID staff on July 20th to review preliminary findings and discuss conclusions and recommendations. Final submission of the report is planned by October 15th after USAID draft review. More details are provided in Figure 7. USAID/Tanzania will provide DATA FOR DEVELOPMENT the following documents from the Tusome Pamoja project: Quarterly progress reports Financial reports Annual progress and financial reports Success stories Work plan 36 Mobilization plan Performance Management and Evaluation Plan GPE Lanes project description DFID Funded EQUIP Project description Other documents as requested RTI will provide D4D a package of all teaching and learning materials which they developed. The package will include the following for Component 1: Standard 1 and 2 decodable readers Zanzibar Standard 1 and 2 Kiswahili Teaching tool Mainland Tanzania Standard 1 & 2 Kiswahili Teaching tool Pre-primary story books (Mtwara only) Standard 3 & 4 fiction and non-fiction readers/reading to learn Ward Education Officer Quality Assurance Guide School leadership training materials and training plan Pre-primary student learning materials and teacher guide Quality assurance package with teacher observation and coaching tools Other documents as requested RTI will provide D4D reports, assessments, materials and information on Component 2: Ward Education Officer and Teacher Resource Center Tutor Training materials and training plan Student reading and writing assessments Training plan for administration of reading assessments Results of reading and writing assessments Capacity assessment of centralized and decentralized education delivery 37 Performance Monitoring and Evaluation Indicators: Number and percent of children who, after two years of schooling, can read and comprehend grade-level text, encode simple sentences, and solve grade-level arithmetic problems Number and percent of children who, after four years of schooling, can read and comprehend grade-level text, respond to simple writing prompts, and solve grade-level arithmetic problems Number and percent of learning environments with the minimum required characteristics (books, trained teachers who arrive on time, absence of violent behavior) for successful 3Rs learning Number of evidence-based policies developed to support the 3Rs reform Number and percent of early grade teachers implementing Tusome Pamoja 3Rs interventions successfully Number and percent of regional education offices capable of coaching with 3Rs teachers and assessing student learning in the 3Rs Number and percent of parents and community members engaging with 3Rs students to improve mastery of 3Rs; and Number and percent of communities holding schools and local governments accountable for quality 3Rs instruction All data will be disaggregated by sex, age, and geographic location. In addition, baseline and midline data will be available to DATA FOR DEVELOPMENT for the measurement of all progress toward all targets. Evaluation Questions: General Program Questions To what extent is Tusome Pamoja making progress towards improving target beneficiaries' 3Rs skills? How is the development of a School Information System leading to improved learning outcomes for the 3Rs? In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by Tusome Pamoja? How can Tusome Pamoja's activities be done differently in order to more efficiently improve learning outcomes for the 3Rs? What are the factors that have implications for the sustainability of quality 3Rs instruction after Tusome Pamoja has ended? 38 Mission Suggested Data Collection Suggested interviews or FGDs with: Permanent Secretary MOEST, Deputy Permanent Secretary PO-RALG, Commissioner of Education, MOEST, Director of Policy and Planning MOEST, Acting Director of Basic Education PORLAG, Staff at Tanzania Institute of Education, Staff at Zanzibar Institute of Education, Regional Education Officers (REOs), Regional Administrative Secretaries, District Education Officers, Ward Education Officers, teachers, heads of school, teachers who were trained by TP, quality assurors (formerly known as inspectors), and the PTPs (UWAWAs), and CEMS (community engagement officers), Suggested interviews with staff from Development Partners: GPE Lanes, DFID EQUIP, UNICEF team working on reading programs, Swedish SIDA, Canadian High Commission, and other stakeholders working on early grade reading Protocol and IRB Requirements: A letter should be written to the Permanent Secretary MOEST and Deputy Permanent Secretary PO-RALG to inform them about this mid-term performance evaluation. USAID/Tanzania suggests involving MOEST and PORALG officials in the actual performance evaluation exercise The Institutional Review Board (IRB) in Zanzibar, the Office of the Chief Statistician (OCGS) and the Tanzania National Bureau of Statistics (NBS)—all take time a long time for approvals so D4D needs to plan in advance III. EVALUATION APPROACH The evaluation will include the following methods: Structured desk review of materials related to Tusome Pamoja, such as the Statement of Work (SOW), Performance Management Plan (PMP), teacher training materials, and other materials recovered from RTI; Up to 50 KIIs will be conducted to capture qualitative perspectives from teachers, administrators, GOT and program staff. Using a retrospective training evaluation approach, the KIIs for teachers and other training participants will also include quantitative scalar29 questions on learning and application of learning to provide answers directly corresponding to the evaluation question three on the extent they’ve internalized capacity building and other interventions. The evaluation team will tabulate the results to these closed-ended questions and produce descriptive statistics that will supplement the qualitative findings. The evaluation team will attempt to interview the following groups: Head of schools/administrators 29 Scalar questions are asked using two questions, one asking respondents to indicate the direction of their attitude (i.e. satisfied or dissatisfied) and another question asking respondents to indicate the strength or intensity of their attitude. 39 District Education Officer-DEO Regional Education Officers- REOs GOT Ministries and Instritutions - MOEST, MOEVT, PORALG, TIE and ZIE Other DPs in the education space (GPE Lanes, DFID EQUIP, UNICEF team working on reading programs, Swedish SIDA, Canadian High Commission) Master Trainers TP trainers/technical staff TP Senior Program Staff (Regional Chiefs, Senior Reading Specialist, Chief of PartyCOP, Monitoring and Evaluation (M&E) Specialist). USAID Staff in Dar es Salaam Up to 28 FGDs will be conducted with teachers, Ward Education Officers (WECs), tutors/college trainers (regional level), and Parent Representatives of the PTPs-Parent Teacher Partnerships (across grade levels, Standard 1-4). Four FGD protocols will be developed, one for each of the aforementioned groups. The evaluation team will invite up to 15 participants for each FGD, keeping in mind refusals and no-shows. Data for Development will aim at having 10 participants per FGD and will require a minimum of 4 participants to conduct the FGD; if fewer than 4 participants participate we will change to having a group interview or KII instead, following the same discussion guide as the FGD. The FGD will be conducted by Swahili-speaking facilitators who are knowledgeable of the activity content and Tanzanian education context; facilitators will be well trained in conducting FGDs to solicit responses from the participants by asking neutral probes and without introducing their own biases. The evaluation team will provide facilitators with training (e.g., role playing) and materials to ensure that they understand the project, FGD guide and can moderate the discussion to obtain maximum response and discussion. The methodology used by the team in conducting FGDs/KIIs is outlined in the figures below. Methodology for Conducting FGDs Each FGD will include a maximum of 15 participants who will engage in an open discussion structured around predetermined questions (included in a discussion guide) led by a moderator. The moderator will be assisted by a note taker, and all FGDs will be recorded with informed consent obtained from all participants prior to the start of the discussion. The note taker will record key words, expressions, silences, and non-verbal language of the participants. Prior to the discussion, each participant will fill out a brief questionnaire containing 10-12 closed-ended, quantitative questions that will supplement the qualitative data. 40 The discussion guide will include 9-12 questions for a 90 minute focus group, starting with broad questions and moving into narrower or key questions. The following types of questions will be used: opening, introductory, transition, key, and ending. The questions will be neutrally worded and neutral probes will be used. Reporting of the FGD will include a general summary of participants’ responses to each question – highlighting the range of responses and experiences. This will be supported by quotes from participants. Methodology for Conducting KIIs Each KII will include one participant who will engage in a semi-structured discussion around predetermined questions (included in an interview guide) with an interviewer. The interviewer will be assisted by a note taker, and all KIIs will be recorded with informed consent obtained from the participant prior to the start of the discussion. The note taker will also record key words, expressions, silences, and non-verbal language of the participant. The interview guide will include 15-20 open-ended questions, starting with broad questions and moving into narrower or key questions. The questions will be neutrally worded and neutral probes will be used. The interview guide will also include 10-12 closed-ended, quantitative questions that the interviewer will pose to each participant to supplement the open-ended, qualitative data. Reporting of the KII will include quotes from the interviewee. The following figure illustrates our approach to answering each evaluation question. 41 Figure 4: Evaluation Design Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 1. To what extent is Tusome Pamoja making progress towards improving target beneficiaries' 3Rs skills? 1. performance monitoring data, previous baseline and midline evaluations, and other research documents such as the most significant change 2. Basic education statistics, Results from EGMA (Arithmetic) and EGRA (reading and writing) assessment tools- baseline data if conducted (3Rs). 3. Teachers, administrators and teacher trainers; 4. project staff, quality assurers, and GOT stakeholders, 5. Parent teacher committee members 1. Document review; 2. Review of existing data; 3. Key informant interviews, focus group discussions with closed ended fact-based questions for teachers Qualitative analysis of FGD and KII discussions; 2. Descriptive quantitative analysis of available monitoring data and national EGRA and EGMA data. 3. Descriptive quantitative analysis of closed-ended questions on FGDs and KIIs 2. How is the development of a School Information System leading to improved learning outcomes for the 3Rs? Ward education officers/Subject Advisors, heads of schools/administrators, program staff and teachers’ perspectives on monitoring tools and the use of the data. Tablets use and teacher performance assessments. Interviews and FGDs with Ward Education Officers (WECs), head of schools/administrators, program staff and teachers and committee members Qualitative analysis of FGD and KII discussions; 2. Review of monitoring data 42 Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 3.In what ways are government officials, school administrators, and teachers demonstrating that they have internalized the capacity building provided by Tusome Pamoja? Ed leadership participants: WECs, DEOs, and other govt officials, school administrators; 3Rs training participants: 1. TP trainers, 2. Master trainers (trained by TP), 3. College/resource center tutors and 4. school teachers (trained by tutors for reading and arithmetic pedagogy skills) and use of training skills on the job/classroom. Interviews and FGDs with 3Rs training participants 1. TP trainers, 2. Master trainers (trained by TP), 3. College/resource center tutors and 4. WECs and school teachers School leadership: Dist Ed officers (DEO), Ward ed officers, school administrators and committee members. Qualitative analysis of FGD and KII discussions; 2. Descriptive quantitative analysis of closed￾ended questions in FGDs and KIIs 4.How can Tusome Pamoja's activities be done differently in order to more efficiently improve learning outcomes for the 3Rs? Best practices suggested from other DPs, program staff and GOT Expert viewpoints on the team. School leadership: Dist Ed officers (DEO), Ward ed officers, school administrators and committee members. Interviews with DPs, KII with TP and govt officials and committee members Interviews and FGDs with 3Rs and leadership training participants 1.Qualitative analysis of FGD and KII discussions (DPs, TP staff); 2. Review of monitoring data; 3. Descriptive quantitative analysis of closed-ended questions on observations, FGDs and KIIs 3Rs training participants: 1. TP trainers, 2. Master trainers (trained by TP), 3. College/resource center tutors and 4. school teachers (trained by tutors for reading and arithmetic pedagogy skills) and use of training skills on the job/classroom. 43 Questions Suggested Data Sources (*) Suggested Data Collection Methods Data Analysis Methods 5. What are the factors that have implications for the sustainability of quality 3Rs instruction after Tusome Pamoja has ended? Institutionalization of TP components Commitments of TP and will the GOT or Districts or schools be able to acrry on wth this KIIs with GOT officials MOEST and PORALG, EOs (Reg/Distr), school administrators. Qualitative analysis of FGD and KII discussions; 2. Review of monitoring data; 3. Review of documents on context IV. TARGET AREAS AND SAMPLING Four field sites including Morogoro, Iringa and Mtwara in Mainland Tanzania and one site in Pemba (North) in Zanzibar have been selected for field work. Morogoro will serve as a pilot site. Following inputs from the mission and IPs, two districts per region and one school from each district with exception of a pilot in Morogoro will be selected. A total of 25 FGDs will be conducted with 250-375 participants (assuming 10-15 participants per FGD). An estimated 42-49 KIIs will be conducted (38 in catchment areas). 44 Figure 5: Sampling Frame Dar es Salaam HQ Dodoma Morogoro (Pilot) Iringa Mtwara Pemba/ Zanzibar (North) Total Comments Focus Group Discussions FGD1- Teachers - - 2 2 2 2 8 28 FGDs total; 280-420 participants FGD2- WEOs (and Subject Advisors in Zan) - - 2 2 2 2 8 FGD3-Tutors/college trainers (regional) - - 1 1 1 1 4 FGD4- Parent Reprepresentatives from the Parent Teacher Partnership - - 2 2 2 2 8 Key Informant Interviews Head of schools/administrators - - 2 2 2 2 8 43-50 KIIs total District Education Officer￾DEO - - 1 1 1 1 4 District Quality Assurers (formerly education inspectors) - - 1 1 1 1 4 Regional Education Officers￾REOs - - 1 1 1 1 4 GOT Ministries and Instritutions- MOEST, MOEVT, PORALG, PORALG/SD, TIE and ZIE 2-3 PORALG 2-3 - - - 1 4-6 DPs (GPE Lanes, DFID EQUIP, UNICEF team working on reading programs, Swedish SIDA, Canadian High Commission) 2-3 EQUIP 1 - - - - 4-6 Master Trainers 2-3 - - - - - 2-3 45 Dar es Salaam HQ Dodoma Morogoro (Pilot) Iringa Mtwara Pemba/ Zanzibar (North) Total Comments TP trainers/technical staff (reading and arithmetic specialists and school leadership specialist) 2-3 - - - - - 2-3 USAID Staff in Dar es Salaam (Christine Djondo and Laura Kikuli) 2 - - - 2 TP Senior Program Staff (Regional Chiefs, Sr reading specialist, COP, M&E Specialist) 2-3 1 1 1 1 6-7 V. REQUIRED MISSION AND IMPLEMENTER INPUTS The evaluation team will need an accurate list with contact information (name, title, institutional affiliation, telephone number, e-mail, physical address, and gender) for technical and leadership staff within RTI. In addition the IP would be requested to provide assistance and outreach to Mentioned stakeholders in the Permanent Secretary MOEST, Deputy Permanent Secretary PO￾RALG, Commissioner of Education, MOEST, Director of Policy and Planning MOEST, Acting Director of Basic Education PORLAG, Staff at Tanzania Institute of Education, Staff at Zanzibar Institute of Education, Regional Education Officers (REOs), Regional Administrative Secretaries, District Education Officers, Ward Education Officers, teachers, heads of school, teachers who were trained by TP, quality assurors(formerly known as inspectors), and the PTPs (UWAWAs), and CEMS (community engagement officers) whom the team should organize KIIs or FGDs with The evaluation team will also need USAID and RTI assistance to encourage participation in data collection efforts, to help increase response rates so that the evaluation team has as complete information as possible to evaluate the effectiveness of the project. An introduction letter from USAID will be needed to encourage participation with the logistics for FGDs and to ensure IP participation in KIIs. Furthermore, additional documentation will be requested by the team to complete the checklist of documents for review mentioned in the scope of work. A working Microsoft Excel checklist of all documents received to date has been compiled with notation on missing or partial documentation. The team will continuously update this checklist based on documents received. Mission and IP will be regularly asked to participate in meeting with the team, in the design and scoping phase and updated on key deliverables. Quick review and turn around in providing feedback on intermediate deliverables will help the team meet its tight timeline for completion. 46 EVALUATION LIMITATIONS It’s important to identify here some limitations inherent to the design of this evaluation: Data availability and data quality: While the implementer and evaluation team will collect and generate primary data, some administrative data that will inform the evaluation may be difficult to obtain or be of questionable quality. We know from previous and ongoing data quality assessments that there may be issues related to the reliability and integrity of monitoring and secondary source data. Early Grade Reading Assessment (EGRA) and Early Grade Arithmetic (EGMA) data vital to answering question 1 on the achievent of students in the the 3Rs may not be available in time for this evaluation and may be outside of our control in including it in the findings, conclusions, and recommendations. We are currently coordinating with RTI to see if it will be possible to include these data for the assessment; this may have an effect on timeline if it is to be included. Selection bias: As some key informants may decline to be interviewed, there is a possibility of selection bias, i.e. those respondents who choose to be interviewed might differ from those who do not in terms of their attitudes and perceptions, affiliation with government/non-government structures, and socio-demographic characteristics and experience. In addition, the purposive nature of the site selection process introduces additional selection bias. Recall bias: Since a number of questions raised during the interviews will address issues that took place 2014-2016, recall bias may affect the responses provided. Halo bias: There is a known tendency among respondents to under-report socially undesirable answers and alter their responses to approximate what they perceive as the social norm (halo bias). The extent to which respondents will be prepared to reveal their true opinions may also vary for some questions that call upon the respondents to assess the attitudes and perceptions of their colleagues or people on whom they depend upon for the provision of services. To mitigate this limitation, the Evaluation Team will outline confidentiality and anonymity statements to all who participate in KIIs, FGDs with road association or community members. The Evaluation Team will also conduct the interviews in as neutral a setting as possible where respondents feel comfortable. The community consumer FGDs that target both men and women will also be conducted separately for each gender. EVALUATION TIMELINE AND DELIVERABLES The Evaluation Team’s anticipated work schedule is provided below. Data for Development will have weekly meetings with the evaluation team for the duration of the work, supervising and managing the process and ensuring smooth progress of the evaluation. The schedule is designed to provide USAID with preliminary findings at the end of fieldwork in late August, and a first draft of the evaluation by mid Sept. The evaluation is anticipated to be complete by mid-October. 47 Figure 6: Timeline and Deliverables Timing (Anticipated Dates) Proposed Activities (SOW) April 18, 2018 April 19-May 11th Discussion of SOW with Mission Preparation of the work plan and evaluation design May 21-30 USAID review of the work plan and evaluation design May 4th-June 5 Preparation and submission of instruments/tools and final work plan and evaluation design May 30 COR and CO approval of all members of the team June 4th-25th IRB approvals, protocol development, piloting, Kiswahili translation, finalization of instruments, and preparations for data collection ( will include an NBS and OCGS approval) July 19th STTA travel and in-brief preparation July 20th Team mobilization/in-briefing July 23rd-August 17th Data collection (4 weeks of collection with 1 day between each site back in Dar for preliminary analysis of findings and Dar interviews) August 13th Participatory findings, conclusions and recommendations meeting and STTA outbriefing August 10th– 30th August Data analysis (with concurrent work on analysis during the collection period) August 30th Preliminary Findings Conclusions and Recommendations Matrix September 1-20th Report writing September 20th Submit Draft Report September 20-Oct 4th USAID review of draft report Oct 4th-15th Incorporate USAID comments and submit final report Oct 15th Final report submitted Oct 15th-20th Utilization and action planning, after action review session. Dissemination to PO and Education team for their dissemention to IPs and public dissemination via the DEC. 48 Timing (Anticipated Dates) Proposed Activities (SOW) Oct 30th Upload to DEC Note: This schedule is predicated on timely approval of USAID/Tanzania for the evaluation team proposed in this document, as well as the timely feedback by USAID/Tanzania of deliverables including the instruments and the draft evaluation report. PROPOSED STAFFING Data for Development has selected an exceptionally qualified team to conduct the performance evaluation of Tusome Pamoja. The team consists of 2 expat STTA team members who will travel to Tanzania, as well as 2 local STTA staff, who will serve as researchers and subject matter experts for the evaluation during all phases of instrument development, data collection, and analysis and report writing. The NORC HQ team includes the Evaluation Team Lead & Education Expert, Megan Gavin, and Qualitative Analyst, Michelle Davis. The two local subject Matter Experts include Godfrey Telli, PhD and Immaculata Mdemu who are education experts here in Tanzania. Data for Development staff includes Gerald Usika (Survey and Data Collection Specialist), who will provide technical support in evaluation, will report to national and regional authorities and will serve in the data collection process. Jacob Laden (Evaluation Advisor) will advise the evaluation design and analysis and will provide coordination and management support for field work. Data for Development staff will support all logistics and data-collection efforts in the four project sites and support the team in overall design, facilitating client and IP communication, designing the quantitative survey, sampling, and conducting the data analysis to be used in the team’s evaluation report. Data for Development will also facilitate the review of interim findings, and draft and final reports. In addition, NORC and ME&A HQ will provide operational and technical support, as well as editing and branding on final deliverables. To meet the tight time constraints of the evaluation, the following is a detail of the level of effort (LOE) for the primary evaluation team members and extended team. 49 Figure 7: Detail of Estimated LOE per Team Member *Total LOE: 259 days Task Team Lead & Education Expert (Megan Gavin) Qualitativ e Analyst (Michelle Davis) Local Educatio n SME STTA (Godfre y Tilly) Local Educatio n SME STTA (Immac ulata Mdemu) Evaluatio n Advisor & (Jacob Laden) Data for Developm ent Survey Specialist (Gerald Usika) Qualitative data collector Translator 1 (TBD) Trans 2 (TBD) Document review/desk review/work planning (evaluation design remote or in￾country) 2 days 2 days 2 days 2 days 1 day 2 days - - - Preparations for travel and organizing data collection (contracting translators, vehicles, etc.). 1 day 1 day 1 day 1 day 2 days 1 day - - - Instrument Developmen t, Evaluation Design and clearnaces (including meetings with USAID) 1 day 1 day 1 day 1 day 2 days 1 day - - - Preparations for data collection (scheduling) 3 days 3 days 2 days 2 days 1 day 2 days - - - Data collection days by method by site 20 days 20 days 20 days 20 days 1 day 20 days 20 days 20 days 20 days 50 Task Team Lead & Education Expert (Megan Gavin) Qualitativ e Analyst (Michelle Davis) Local Educatio n SME STTA (Godfre y Tilly) Local Educatio n SME STTA (Immac ulata Mdemu) Evaluatio n Advisor & (Jacob Laden) Data for Developm ent Survey Specialist (Gerald Usika) Qualitative data collector Translator 1 (TBD) Trans 2 (TBD) Data analysis 8 days 8 days 7 days 7 days 2 days 7 days - - - Briefing 1 day 1 day - - 1 day - - - - Draft report and debrief to USAID 6 days 6 days - - 2 days - - - - Final report after USAID comments 1 days 1 days - - 2 days - - - - Totals 43 days 43 days 33 days 33 days 14 days 33 days 20 days 20 days 20 days 51 ANNEX III: TOOLS Tusome Pamoja Midterm Evaluation School Observation Checklist 1 Jina la Shule/School: - 2 Namba ya usajili/Code: - 3 Mahali Shule ilipo/Location: - 4 Darasa/Madarasa unayofanyia uchunguzi/Grade: - Ndio/Yes or Hapana/No Idadi/ How many 5 Decodable Readers - - 6 Muongozo wa kufanyia tathmini/Assessment Guides - - 7 Vitabu vikubwa/Big Books - - 8 Vitabu vya hadithi/Story Books - - 9 Kiongozi cha mwalimu/Teacher Guides - - 10 Zana nyingine (taja)/Other materials (list) - - 11 Vitabu/Textbooks - - 12 Zana zilizochorwa ukutani/Materials on walls - - Kama inawezekana /If applicable Darasa la awali/Pre-primary - - 14 Vitabu vya hadithi/Story Books - - 15 Kiongozi cha mwalimu/Teacher Guides - - 16. Chunguza (wapi vifaa vinahifadhiwa? Je, inaonekana kuwa vinatumika? Je! Hutumiwa na wote au watoto wengine pekee)? /Qualitative Reflections (where are the materials stored? does it appear they are in use? Are the in use by all or only some of the children)? 52 Tusome Pamoja Midterm Evaluation Mini Survey with Parents 1 Jina la kwanza tu/First name (only): - 2 Jinsi/Sex: - 3 Mahali shule ilipo/Location: - 4 Darasa la mtoto au watoto/Grade of child(dren): - Ikiwa mtoto yupo darasa la 2, fikiria mtoto huyu. Ikiwa mtoto hayupo darasa la 2, fikiria mtoto ambaye ni karibu na darasa la 2, ama darasa la 1 au darasa la 3, ni vyema ikiwa darasa la 3 kabla ya la 1. Ikiwa sio darasa la 1 au la 3, kisha uchague darasa lingine. / If child in grade 2, reflect on this child. If child not in grade 2, reflect on child which is closest to grade 2, either grade 1 or grade 3, preference grade 3 before grade 1. If not grade 1 or 3, then choose another grade. 5 Mtoto wako anaweza kusoma/Can your child read? Ndiyo/Yes au Hapana/No 6 Mtoto wako anaweza kuandika/Can your child write Ndiyo/Yes au Hapana/No 7 Mtoto wako anaweza kuhesabu/Can your child do arithmetic? Ndiyo/Yes au Hapana/No 8 Mtoto wako anaweza kusoma barua/Can your child read letters? Ndiyo/Yes au Hapana/No 9 Mtoto wako anaweza kusoma maneno/Can your child read words? Ndiyo/Yes au Hapana/No 10 Mtoto wako anaweza kusoma sentensi/Can your child read sentences? Ndiyo/Yes au Hapana/No 11 Mtoto wako anaweza kusoma aya/Can your child read paragraphs? Ndiyo/Yes au Hapana/No 12 Mtoto wako anaweza kusoma kitabu kizima/Can your child read a whole book? Ndiyo/Yes au Hapana/No 13 Je, Mtoto wako anaweza kusoma nini (chaguo Zaidi ya moja)/What does your child read (all that apply)? Gazeti/Newspaper - Jarida/Magazine - Simu ya mkononi/Cellphone - Kitabu cha nukuu/Notebook - Kitabu cha dini/Religious book - Kitabu cha hadithi/Storybook - Ubao wa kusomea/Blackboard - Kingine/Others - 14 Ni nyenzo zipi za kusoma ulizonazo nyumbani (chagua Zaidi ya moja)/What reading material do you have at home (check all that apply)? Gazeti/Newspaper - Jarida/Magazine - Simu ya mkononi/Cellphone - Kitabu cha dini/Religious book - Kitabu cha hadithi/Storybook - Kingine/Others - 15 Huwa unasoma pamoja na mtoto wako/Do you read with your child? Ndiyo/Yes au Hapana/No 53 1 Jina la kwanza tu/First name (only): - 16 Kama ndio, ni kwa muda gani kwa wiki/If Yes, how much time a week? Masaa 30-1/30-1 hour - Lisaa 1-2/1-2 hours - Zaidi ya masaa 2/More than 2 hours - 17 Huwa unamsaidia mtoto wako kwenye kazi za darasani nyumbani/Do you help with homework? Ndiyo/Yes au Hapana/No 18 Je, Unaamini kujua kusoma ni muhimu/Do you believe reading is important? Ndiyo/Yes au Hapana/No 19. kwanini/ Why? (Andika/write response) 54 Key Informant Interview – Head of School Name: Date: School (Code): Location: Interviewer: Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague__________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 45 minutes. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this interview will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 55 1. Let’s start by talking about how we were approaching the 3Rs a few years back, three years ago. What was the state of the learner’s level of the 3R? a. What were the major challenges? b. Were teachers prepared to teach these subjects? c. Were there adequate materials? d. Were the materials useful? e. What was the quality of the materials? 2. Did you or your school participate in any activity related to 3R in the past two years? a. If yes, what was the activity and who delivered it? ** Use this question as a reference to lead into future questions on capacity building, materials, and SIS 3. What is your specific role in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 4. Did you or your school participate in any capacity building activities in the past two years (related to the 3Rs)? a. If yes, what were the CB activity(s) about and who provided it/ them? b. Was the training general or was it directly related to your role? 5. Have you used the knowledge gained in the CB activities? a. If yes, please explain the ways you have executed to ensure that the knowledge gained in the CB is integrated in your day to day activities? 6. Describe any follow up support provided by TP to ensure that you or anyone else is using the CB provided? 7. What can be done or designed differently to ensure that [teachers] can internalize CB provided by TP? 8. Has there been any shift in the usefulness or availability of materials since TP started? Please reflect on any changes or improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. a. What are the materials like? b. Are there enough materials? c. What do teachers think of the materials? d. Do the materials align well with the capacity building activities we discussed? e. Are there any different materials needed to meet the 3Rs goals? 9. Are you familiar with any School Information Systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. 10. Are you aware of the SIS implemented by the TP? If yes, what is it about and how does it work? 11. How does SIS differ from other School Information Systems? 12. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 56 13. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 14. How do parents and teachers convene to discuss school matters? What is the name of the platform? Please explain the formation of the platform - when did it start? How does it work? Who are the members? How are they selected? How frequently do they meet? Who are the leaders? What measures are taken to ensure quality 3R instruction? How was the situation before the formation of the platform? What is the plan to ensure the platform sustains without TP support? 15. To what extent do you believe that the activities we have been discussing, implemented by TP, will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the main lessons learned? b. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 16. What are the key success factors to ensure that TP achieve its objectives? 17. Please mention the key factors that hinder the achievement of the 3Rs long term objective of improving lifelong learning skills. What can be done to address these challenges? Thank you. If there is more you would like to share, please feel free to do so. 57 Key Informant Interview- Education Authority (REOs, DEOs, GOT- MOEST, MOEVT, PORALG, TIE, ZIE) Name: Date: Role: Location: Interviewer: Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague__________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 45 minutes. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this interview will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 58 1. Let’s start by talking about how we were approaching the 3Rs a few years back, three years ago. What was the state of the earner’s level of the 3R? How was your office/ministry/etc. approaching these topics? What were schools doing? a. What were the major challenges? b. Were there adequate materials of quality? 2. Did you or your government office participate in any activity related to 3R in the past two years? 3. If yes, what was the activity and who delivered it? 4. Please explain the work of TP to support system actors at National, Regional and District Level to manage sustainable 3Rs implementation by Districts, Wards and Schools. ** use this question as a reference for all further questions on activities 5. What is your specific role in improving 3R skills? 6. Did you or your government office participate in any capacity building activities in the past two years? 7. If yes, what were the CB activity(s) about and who provided it/them? 8. Have you used the knowledge gained in the CB activities? a. If yes, please explain the ways you have executed to ensure that the knowledge gained in the CB is internalized in your day to day activities? 9. Is there appropriate support and guidance from TP when your team members go on to train others? 10. Are the materials you all use to train others useful? Why or why not? 11. What can be done or designed differently to ensure that [teachers] can internalize CB provided by TP? 12. Are you familiar with any School Information Systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. 13. Are you aware of the SIS implemented by the TP? If yes, what is it about and how does it work? 14. How does SIS differ from other School Information Systems? 15. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 16. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 17. How do parents and teachers convene to discuss school matters? What is the name of the platform? Please explain the formation of the platform - when did it start? How does it work? Who are the members? How are they selected? How frequently do they meet? Who are the 59 leaders? What measures are taken to ensure quality 3R instruction? How was the situation before the formation of the platform? What is the plan to ensure the platform sustains without TP support? 18. What can TP do differently? What is the planned commitment to carry over the TP support after the TP has ended? 19. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the main lessons learned? b. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 20. How did you use the knowledge gained in the activities described above to improve learners’ mastery of the 3Rs? 21. What have been the perceived and actual impact(s) of the 3R activities? a. What was the state of the learner’s level of the 3R prior to the implementation of the activities the past two years? b. Were there adequate materials? c. Were the materials useful? d. What was the quality of the materials? e. Please reflect on the improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. How has it improved? 22. Please mention the key factors that hinder the achievement of the 3Rs long term objective of improving lifelong learning skills. What can be done to address these challenges? Thank you. If there is more you would like to share, please feel free to do so. 60 Key Informant Interview – Donor (GPE Lanes, DFID EQUIP, UNICEF, Swedish SIDA, Canadian High Commission) Name: Date: Role: Organization: Interviewer: Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague__________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 45 minutes. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this interview will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 61 1. Did you or your organization participate in any activity related to 3R in the past two years? 2. If yes, what was the activity and who was it delivered to? 3. How does your work/project articulate with TP? What is your organization’s role in improving 3R skills in Tanzania? 4. Please reflect on the improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. How has it improved? 5. Are you familiar with any School Information Systems? How does this differ from other school information systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. 6. Are you aware of the SIS implemented by the TP? 7. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 8. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 9. Are you aware of any capacity building activities that TP provided related to the 3Rs? What do you think of the model? 10. Does your institution ever work with parents and communities around the 3Rs? How do parents and teachers convene to discuss school matters? Please provide any details on work with parents and communities around the 3Rs or challenges to this work. 11. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the strengths and weaknesses of the model? b. In your opinion, what can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 12. What have been the perceived and actual impact(s) of the 3R activities implanted by TP? 13. Please mention the key factors that hinder the achievement of the 3Rs long term objective of improving lifelong learning skills. What can be done to address these challenges? Thank you. If there is more you would like to share, please feel free to do so. 62 Key Informant Interview – TP (TP Senior Staff, COP, M&E Specialist, USAID Tanzania COR) Name: Date: Role: Interviewer: Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague__________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 45 minutes. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this interview will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 63 What is your role in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 1. Please explain the work of TP to support system actors at National, Regional and District Level to manage sustainable 3Rs implementation by Districts, Wards and Schools. 2. Did you help design activities around capacity building, or help implement those activities? If yes, what is your feeling and perception about these capacity building activities? What worked well and what didn’t work as well as you would have liked? How did you all adapt capacity building over time to improve the offer? 3. Do you believe the reinforcement of CB provided will sustain the ambitious objectives of the 3Rs? If yes, how? 4. Describe any follow up support provided by TP to ensure that you or anyone else is using the CB provided. 5. What can be done or designed differently to ensure that [teachers] can internalize CB provided by TP? 6. How did TP start including activities on School Information Systems? When and why did this become part of the project? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. 7. How does SIS work and why is it different from other School Information Systems? What are the pros and cons of SIS to these other systems? 8. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 9. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 10. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. Have there been any moments where you have felt like you started to see the impact of your work materializing in real changes? Was there a specific moment or event that made you feel this? b. What are the main lessons learned thus far? c. Was anything harder or more challenging than initially foreseen? What has worked to address these difficulties? d. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? Are there any activities that we are missing, need to prioritize more, or should do less of? If 11. What can TP do differently? What is the planned commitment to carry over the TP support after the TP has ended? Thank you. If there is more you would like to share, please feel free to do so. 64 Focus Group Discussion Guide- Teachers School: Location: Date: Number of participants (males, females): Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague __________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja project. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 1 hour. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this FGD will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 65 1. Did you participate in any activity related to 3R in the past two years? If yes, what was the activity and who delivered it? 2. What is your role in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 3. Did you participate in any capacity building activities in the past two years? If yes, what were the CB activity(s) about and who provided it/them? 4. Have you used the knowledge gained in the CB activities? How? 5. Describe any follow up support provided by TP to ensure that you or anyone else is using the CB provided. 6. What can be done or designed differently to ensure that [teachers] can internalize and always use CB provided by TP? 7. Has there been any shift in the usefulness or availability of materials since TP started? Please reflect on any changes or improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. a. What are the materials like? b. Are there enough materials? c. What do teachers think of the materials? d. Do the materials align well with the capacity building activities we discussed? e. Are there any different materials needed to meet the 3Rs goals? 8. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the main lessons learned? b. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 9. Are you familiar with any School Information Systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. 10. Are you aware of the SIS implemented by the TP? If yes, what is it about and how does it work? 11. How does SIS differ from other School Information Systems? 12. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 13. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 14. Is there any way/platform at which teachers and parents convene together to discuss school matters? What is the name? Please explain, the formation of that platform, when has this started? How does it work, who are the members, how are they selected, how frequent do they meet, who are the leaders.? Roles to ensure quality 3Rs instruction, how was the situation before the formation? What is the plan to ensure the platform sustain without TP support? 66 15. How did you use the knowledge gained in the activities described above to improve learners’ mastery of the 3Rs? 16. Please describe the key factors that hinder achievement of the 3R long term objectives of improving lifelong learning skills. What can be done to address those challenges TP? Thank you. If there is more you would like to share, please feel free to do so. 67 Focus Group Discussion Guide - WEOs Location: Date: Number of participants (males, females): Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague __________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja project. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 1 hour. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this FGD will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 68 1. Did you participate in any activity related to 3R in the past two years? If yes, what was the activity and who delivered it? 2. What is your role in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 3. Did you participate in any capacity building activities in the past two years? If yes, what were the CB activity(s) about and who provided it/ them? 4. Have you used the knowledge gained in the CB activities? How? Have you gone on to provide training or capacity building to others? 5. Describe any follow up support provided by TP to ensure that you or anyone else is using the CB provided. Have you felt supported as you have gone on to train other teachers? What was this experience like? 6. What can be done or designed differently to ensure that [teachers] can internalize and always use CB provided by TP? 7. Has there been any shift in the usefulness or availability of materials since TP started? Please reflect on any changes or improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. a. What are the materials like? b. Are there enough materials? c. What do teachers think of the materials? d. Do the materials align well with the capacity building activities we discussed? e. Are there any different materials needed to meet the 3Rs goals? 8. Are you familiar with any School Information Systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. a. Are you aware of the SIS implemented by the TP? If yes, what is it about and how does it work? b. How does SIS differ from other School Information Systems? 9. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 10. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 11. Is there any way/platform at which teachers and parents convene together to discuss school matters? What is the name? Please explain, the formation of that platform, when has this started? How does it work, who are the members, how are they selected, how frequent do they meet, who are the leaders.? Roles to ensure quality 3Rs instruction, how was the situation before the formation? What is the plan to ensure the platform sustain without TP support? 12. How did you use the knowledge gained in the activities described above to improve teacher’s ability to ensure learners mastery of the 3Rs? 69 13. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the main lessons learned? b. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 14. Please describe the key factors that hinder achievement of the 3R long term objectives of improving lifelong learning skills. What can be done to address those challenges TP? Thank you. If there is more you would like to share, please feel free to do so. 70 Focus Group Discussion Guide- Tutors Location: Date: Number of participants (males, females): Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague __________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja project. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 1 hour. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this FGD will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 71 1. Did your group participate in any activity related to 3R in the past two years? If yes, what was the activity and who delivered it? 2. What is your role in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 3. Did your group participate in any capacity building activities in the past two years? If yes, what were the CB activity(s) about and who provided it/ them? 4. Have you used the knowledge gained in the CB activities? How? Have you gone on to provide training or capacity building to others? 5. Describe any follow up support provided by TP to ensure that you or anyone else is using the CB provided. Have you felt supported as you have gone on to train other teachers? What was this experience like? 6. What can be done or designed differently to ensure that [teachers] can internalize and always use CB provided by TP? 7. Has there been any shift in the usefulness or availability of materials since TP started? Please reflect on any changes or improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. a. What are the materials like? b. Are there enough materials? c. What do teachers think of the materials? d. Do the materials align well with the capacity building activities we discussed? e. Are there any different materials needed to meet the 3Rs goals? 8. Are you familiar with any School Information Systems? If yes, describe it briefly and its role in improving the learning outcomes of the 3Rs. a. Are you aware of the SIS implemented by the TP? If yes, what is it about and how does it work? b. How does SIS differ from other School Information Systems? 9. What do you believe is the role of SIS in improving the learning outcomes of 3Rs? Please explain the suitability of the data to be collected by SIS for improved learning among faculty and admins of the schools. How it would help to improve coaching, capacity building, follow-up, and other activities? 10. Can you explain the ways SIS can be implemented or designed differently to ensure it meets its goal of improving the learning outcomes of the 3Rs? 11. Is there any way/platform at which teachers and parents convene together to discuss school matters? What is the name? Please explain, the formation of that platform, when has this started? How does it work, who are the members, how are they selected, how frequent do they meet, who are the leaders.? Roles to ensure quality 3Rs instruction, how was the situation before the formation? What is the plan to ensure the platform sustain without TP support? 12. How did you use the knowledge gained in the activities described above to improve teacher’s ability to ensure learners mastery of the 3Rs? 72 13. To what extent do you believe that the activities implemented by TP will lead to improvement of target beneficiaries’ [children’s] 3Rs? a. What are the main lessons learned? b. What can be done differently to ensure that TP achieves the objective of improving the quality of 3Rs skills instruction? 14. Please describe the key factors that hinder achievement of the 3R long term objectives of improving lifelong learning skills. What can be done to address those challenges TP? Thank you. If there is more you would like to share, please feel free to do so. 73 Focus Group Discussion Guide - Parents School (code): Location: Date: Number of participants (males, females): Introduction and Consent Interviewer: To start the interview, please read the following script: Hello and thank you for agreeing to speak with us. My name is_______________ (interview name) and this is my colleague __________________. We work with the Data for Development project, a USAID￾funded platform that seeks to improve the quality and use of data in decision-making in Tanzania. We are undertaking an evaluation to understand what is working or not working with Tusome Pamoja project. The project and the evaluation are funded by USAID. In the context of this evaluation, we would like to interview you about the implementation of this project. This discussion will last approximately 1 hour. Your participation in this interview is voluntary and you can choose not to answer a question and skip it. Or stop the interview at any moment without providing a reason. Your perspective is very important to help USAID improve its programs. The information we will be collecting through this FGD will be kept safe by our team. Your responses will be kept anonymous and not linked to your name-each person interviewed will be given a unique identification number. Your identity will be kept confidential and will not be shared outside of the evaluation team. Other information that could identify you (e.g., position, community, district) will be excluded from report and other documents produced by our team and shared with USAID. If you have any questions you may ask them now or later, even after the survey has started. If you wish to ask questions later, you may contact any of the following: [Gerald Usika at 0756180413, gusika@engl.com] This proposal has been reviewed and approved by NBS, which is a committee whose task it is to make sure that research participants are protected from harm. Do you agree to participate in this interview today? Yes No 74 1. Is there any way/platform at which teachers and parents convene together to discuss school matters? What is the name? Please explain, the formation of that platform, when has this started? How does it work, who are the members, how are they selected, how frequent do they meet, who are the leaders? Roles to ensure quality 3Rs instruction, how was the situation before the formation? What is the plan to ensure the platform sustain without TP support? 2. How do your kids feel about reading? Do they enjoy reading? Why or why not? If you have older children who have gone through school, how can you describe their experience with reading, are your younger children having a different experience with reading? 3. What is your role as parents and community members in implementing the improvement of the 3Rs skills? Please explain how you perform this role. 4. Did parents or community members participate in any capacity building activities in the past two years to support in this role? 5. If yes, what were the CB activity(s) about and who provided it/them? 6. Have you used the knowledge gained in the CB activities? a. If yes, please explain the ways you have executed to ensure that the knowledge gained in the CB is internalized in your day to day activities? b. How many times have you used what you learned? 7. What have been the perceived and actual impact(s) of our new approach to reading and the 3Rs? a. What was the state of the learners’ level of the 3R prior to the implementation of the activities the past two years? b. Were there adequate materials? c. Were the materials useful? d. What was the quality of the materials? e. Please reflect on the improvement (availability, usefulness, and quality) of the materials related to 3R after the TP intervention. How has it improved? 8. Please describe the key factors that hinder achievement of the 3R long term objectives of improving lifelong learning skills. What can be done to address those challenges TP? Thank you. If there is more you would like to share, please feel free to do so. 75 ANNEX IV: SOURCES Tusome Pamoja Contract, 2015 RFP Tusome Pamoja, 2014 Tusome Pamoja, PMP, 2017 Tusome Pamoja, Annual Report, 2017 Tusome Pamoja, Annual Work Plan, 2017 Tusome Pamoja, Quarterly Reports (various 2016-2017) BRN Education, NKRA Lab Report (2013) BRN National Key Result Area: 2013/14- 2015/16 Assistance to Basic Education: All Children Reading (ABE-ACR): Final Findings Report, Tanzania National Early Grade Reading Assessment (EGRA), 2016 National Baseline Assessment for the 3Rs (Reading, Writing, and Arithmetic) Using EGRA, EGMA, and SSME in Tanzania (2014) Ministry of Education Science and Technology and President’s Office, Education Program for Results, 2017 EQUIP- Tanzania Impact Evaluation, Midline, 2016 GPE Literacy and Numeracy Education Support Programe (Lanes) program document, 2013 NBER working paper: Improving Educational Outcomes in Development Countries: Lessons from Rigorous Impact Evaluations (Ganimian, A & Murnane, R, 2014). Peer Effects and Textbooks in African primary education (Frolich, M, and Mihaelowa, K, 2000). RISE Output in Education: Assessment of BRN Implementation and Ideas for Future Reform (2017) Rise Is the System Tuned to Deliver? Evidence from the Competence Based Curriculum Reforms for Basic Education in Tanzania (2017). RTI Early Reading- Igniting Education for All (2011) USAID Education Strategy (2011-2016) USAID Tanzania CDCS World Bank Facing Forward (2018) World Bank Global Data Set on Edu Quality (2018) World Bank Training Teachers (2018). 76 ANNEX IV: DISCLOSURE OF ANY CONFLICTS OF INTEREST Name Megan Gavin Title TL Organization NORC Evaluation Position? X Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome Pamoja I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. - I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 06/31/2018 77 Name Michelle Davis Title Qualitative Analyst Organization Data for Development, NORC Representative Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome Pamoja, RTI I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Michelle Davis, Date 06/31/2018 78 Name Jacob Laden Title Evaluation Advisor Organization Data for Development, NORC Representative Evaluation Position? Team Leader X Team member Evaluation Award Number (contract or other instrument) AID-OAA-1-15-00024/AID-621-TO-17-00005 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome Pamoja I have real or potential conflicts of interest to disclose. Yes X No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Jacob Laden Date 06/31/2018 79 80 81 82 Annex V: KIIs Title Organization Date Director Tanzania Institute for Education (TIE) 24 July 2018 Curriculum Developer TIE 24 July 2018 Curriculum Developer TIE 24 July 2018 Commissioner of Education Ministry of Education, Science and Technology (MOEST) 24 July 2018 Quality Assurance Officer MOEST [joined KII w Commissioner] 24 July 2018 Education Specialist UNICEF 24 July 2018 Education Specialist UNICEF 24 July 2018 Chief of Party (COP) RTI 26 July 2018 Technical Coordinator RTI 26 July 2018 Sr. M&E Specialist RTI 26 July 2018 Gender & Equity Specialist RTI 26 July 2018 Sr. Reading Advisor RTI 06 August 2018 COR USAID 27 July 2018 Education Officer USAID 27 July 2018 M&E/Education USAID 27 July 2018 National Team Leader EQUIP 09 August 2018 Director ZIE 28 July 2018 Director MEOVT 26 July 2018 Education Officer PO RALG 08 August 2018 83 ANNEX VI: SYSTEMATIC LITERATURE REVIEW Title Author Summary Key Words LANES Program GPE Tanzania has been part of the GPE Grant ($94.8) since 2012. The Literacy and Numeracy Education Support (LANES) program is in response to challenges identified by sub sectors: PEDP, ANFED, FEDP, which have resulted fro the expansion of primary education over the past 15 years. The focus is on: (1) improving teacher incentives, (2) enhancing student assessment, (3) strengthening parent and citizen engagement for accountability. In addition, there is the Big Results Now (BRN) government initiative to fast track achievements. LANES aligns to 4 out of 5 of GPE’s 2012-2015 Strategic Objectives. The program is national and focused on reading, writing and numeracy for in/out of school children (w a focus on marginalized and hard to reach regions), day care (2-4 years) and 9-13 year old non formal. The components include: improve mastery of skills in (a) pre/ primary; in (b) non-formal, (c) promote ECD; (d) institutionalization; and (e) capacity strengthening for education delivery. Context, DPs Improving Educational Outcomes NBER This is a meta review of 223 rigorous impact evaluations of education interventions from 56 low and middle-income countries. Four lessons emerge: 1) Reducing the costs of going to school and expanding schooling options increase attendance and attainment, but do not consistently increase student achievement. 2) Providing information about school quality, developmentally appropriate parenting practices, and the economic returns to schooling affects the actions of parents and the achievement of children and adolescents. 3) More or better resources improve student achievement only if they result in changes in children’s daily experiences at school. 4) Well-designed incentives increase teacher effort and student achievement from very low levels, but low-skilled teachers need specific guidance to reach minimally acceptable levels of instruction. Specifically, with regard to Tanzania the authors cite a 2006 study by Grigorenko et al. which found that providing access to deworming drugs to children in grades 2-6 improved performance on cognitive tests. Background, SIS, Materials, Resources Peer Effects and Textbooks in Africa Frolich and Michaelowa Textbooks could be a cheap and efficient input to primary school education in Africa. In this paper, we examine the effects of textbooks on student outcomes and separate between direct effects and externalities. Using the rich data set provided by the “Program on the Analysis of Education Systems” (PASEC) for five Francophone, sub-Saharan African countries, this paper goes beyond the estimation of direct effects of textbooks on students' learning and focuses on peer effects resulting from textbooks owned by students' classmates. Using nonparametric estimation methods, we separate the direct effect of textbooks from their peer effect. The latter clearly dominates but depends upon the initial level of textbook availability. Background, Africa, Materials 84 Title Author Summary Key Words RISE powerpoint RISE RISE had the following research agenda: system diagnostic, curriculum reforms, implementation report, estimates of impact. It provides a taxonomy of BRN: 1) pressure to perform, 2) teacher motivation, 3) back to basics, 3Rs, 4) school management/finance. Regarding information communities not informed on rank vs districts yes, and head teachers access website, private schools informed (see slides for stats). Government surpassed targets in Student Teacher Enrichment Program (STEP) and 3R training, few received the toolkit and toolkit training. 3Rs, SIS RISE Assessment RISE Between October 2016 and February 2017, the RISE Tanzania Country Research Team conducted a survey of 74 District Education Officers (DEOs), 638 Head Teachers (HTs), and 430 School Management Committee and Board of Governors (BOG) Chairs across seven regions in Tanzania. District Education Officers were surveyed in person while other respondents were surveyed over the phone. The aim of the survey was to generate evidence on the degree and fidelity of implementation of key components of the Big Results Now in Education (BRNEd) reform initiated in 2014. Survey findings point to a very mixed picture on the quality of implementation across the components of BRN. On a positive note, the timely dispersal of capitation grant funding and the delivery of instructional materials is high. Our survey results suggest that knowledge of school performance is poor amongst some of the key actors in the education system. Design, BRN, Materials, SIS RISE Evidence RISE The document provides an overview of the CBE in Tanzania. The major strength of Competence-Based Education (CBE) as documented in the literature is its ability to produce graduates with employability skills who could generate knowledge, think creatively and solve social and economic problems in the society. Competence-based learning is a “system of instruction, assessment, grading and academic reporting that are based on students demonstrating that they have learned the knowledge and skills that they are expected to learn as they progress through their education” And requires: a) A shift to a dynamic orientation to the identification of the skills that students need to master; b) a change in the form of instruction, from teacher-centered to student-centered learning; and c) a change in assessment and grading, from content-based testing to the assessment of demonstrated skills. The document also details the list of actors. Background, Materials, 3Rs, Teacher Training, Instruction 85 Title Author Summary Key Words Competency powerpoint RISE The ppt is a good source to describe the overall research associated w the Competency Based Curriculum reform. The ppt captures the evidence to date and concludes: that the curriculum document intentions differ from the actual instructional practices, and that CBC reform has not translated into substantive changes. Specifically: 1) Teachers’ lack of knowledge and skills for effective implementation of CBC; 2) Teachers’ lack of proper understanding of the objectives of CBC. 3) Teachers’ failure to implement CBCs in classroom teaching (lesson plans, engaging students and assessing). 4) TTCs studies show that although tutors seem to understand CBC their actual instructional practices contradict their knowledge. The objectives of the RISE research are: 1) To determine accountability relationships between and among the actors and how these hinder or facilitate the implementation of CBC; and 2) To explore the coherence of accountability relationships between key actors in delivery of CBC reforms. CBC, Background, Materials EQUIP EQUIP Provides an overview of the EQUIP-T project funded by the UK. The purpose of the project is to improve the learning outcomes at the basic level, especially for girls. Specifically: improve teacher performance, strengthen school leadership/ management, enhancing district planning/ management, strengthen community participation/ accountability, and strengthening learning/ dissemination. The EQUIP-T works in 7 (+2) regions, 51 (+12) LGAs. In 4,500 schools, w 49,000 teachers, 2.3 million children; for a total of 50m EU over 4 years, plus 30m EU over 2 years (inclusion, gender and construction). DPs, Context, Background GPE GPE The ppt describes much of what is covered in the document, just in ppt form. The notes indicate that it is a national program aimed at improving the 3Rs for in and out of school children and youth. It derived from the sub-sector plans from: PEDP, ANFED and FEDP. The principles: build on and improve existing initiatives, enhance pre-primary and primary teacher capacity, devise cost effective profession development, use a cascade approach (region, LGA, ward, school), requires a firm commitment of wards and schools, and incentives to schools. The ppt also outlines four challenges, including to the rollout of the SB-CPD; and five lessons learned. DPs, Context, Background RISE PowerPoint RISE This is not a very helpful ppt. There is one slide on learning outcomes that indicates: invest in teachers, rationalize school management, measure outcomes, and use public rankings. There is one table on education reform. The research agenda is also presented: deployment of teachers, school management, measuring outcomes, and public ranking of schools. n/a 86 Title Author Summary Key Words RISE Overview RISE The Research on Improving Systems of Education (RISE) Program is a multi-country research project, co-funded by the United Kingdom’s Department for International Development (DFID) and Australia’s Department for Foreign Affairs and Trade (DFAT). While many countries have been successful at increasing primary and secondary school enrollments, learning levels remain strikingly low. While stand-alone programs and interventions such as school feeding and remedial education can have positive impacts on learning outcomes, questions still exist around how to generate large positive impacts on learning at scale. Tanzania is one of four inaugural countries competitively selected for the RISE Program that also include India, Vietnam, and Pakistan. A total of six countries will be studied over the course of the program. Evidence, Background RISE Overview (continued) RISE A systems framework to examine the accountability relationships - delegation, resources, information, and motivation. They examine the extent to which these initiatives succeeded in practice (and how) as well as how key elements help to build and sustain a coalition to support effective and durable reforms. In studying the impacts of these initiatives on learning outcomes in Tanzania’s primary and secondary schools, we propose to employ multi- and mixed method research approaches—combining qualitative and quantitative strategies. They list tentative research questions. They also identify progress to date including: an education conference, a Joint Education Sector Review, and a strategic meeting with stakeholders (2016) and their plan for the future: annual review and planning and design exercise. Evidence, Background RISE- Notes RISE Well done bi-weekly notes, not particularly useful for us. n/a RISE USAID TP Intro TP/USAID This is a great source for the evaluation team because it provides an overview of the project. It includes the coverage and the amount $68mil over five years; four regions and Zanzibar and the technical scope. The primary goal is to improve lifelong learning skills defined as EGR, writing and arithmetic (the 3Rs). To do so the project includes: quality of early grade basic skills improved, skills delivery and assessment systems throughout the system strengthened, and effective engagement of parent and community. The project has an emphasis on each level of governance: national, regional, district, ward, community, and school. TP, Background, 3Rs, SIS, levels of intervention 87 Title Author Summary Key Words Lightening Education For All RTI This document provides a summary of the most important aspects of early grade reading globally. USAID global education policy aligns with the recommendations, and Tanzania country strategy's education portion aligns with the global strategy. Topics touched on include: challenges to ensuring early grade reading and the results of assessment globally; complexities of teaching reading in multi-lingual countries and differences in reading learning curve among different languages; keys to effectively teaching reading including what school systems, teachers, and families need; how to use EGRA type assessments to measure; and five key recommendations. The five key recommendations are: train teachers on how to teach reading, maximize instructional time in the classroom, put appropriate books into the hands of children and mobilize communities to use them, implement appropriate language policies and promote mother-tongue instruction, and testing to measure reading skills. EGR Education NKRA Big Results Now￾Report and Roadmap Tanzania Development Vision Team: Education Lab The Education Lab is made up of professionals from 31 government and NGO organizations who looked to determine the biggest challenges and quickly implementable solutions in primary education. Over the last 10 years Tanzania experienced increased primary enrollment but decreased quality (based on pass rates) and low equity of education. The lab identified 18 major problems that can be grouped into five main reasons for low quality: lack of accountability, availability of teaching and learning materials, low support for struggling students, poor school management, and no national standard assessment for the 3R's. They determine which of the 18 issues can be solved most quickly, with at least several from each challenge category and suggest four macro focus': 1. Create performance transparency: create school rakings based on improvement and overall scores on test results with rankings at granular level for ministry and then grouped into green, yellow, or red and published nationally; conduct national 3R assessments) 2. Motivate through incentive: reward the 4000 move improved schools and recognize the top 200 performers) 3. Provide support where needed most: distribute school improvement toolkits on roles and responsibilities, practical guidance on key issues, metrics to track performance etc. and do a ToT for head teachers to roll out the toolkit; train teachers in 3R teaching skills; ToT to implement Student Teacher Enrichment Program-STEP to support low performing students; consider reducing number of school subjects for more focus time on 3Rs; ensure on time delivery of books and materials and reduce book sharing - five students currently share one book- by implementing a tax on the communications industry which will help cover costs to provide each student with their own books/reduce sharing; continue to construct basic facilities 4. Improve teacher conditions: recognize teachers through non￾monetary incentives and ensure all claims are resolved. - 88 Title Author Summary Key Words USAID Education Strategy: Opportunity Through Learning 2011-2017 USAID The education strategy outlines three main goals: 1) improve reading skills in primary grades; 2) improved ability of tertiary and workforce development programs; and 3) increased equitable access to education in crisis and conflict zones. Tusome Pamoje related to Result 1 which is broken down into three subIRs: 1.1 Improved reading instruction through improving teacher effectiveness, increasing availability and use of reading materials, and strengthening classroom and school management. 1.2 Reading delivery systems improved via clear standards and benchmarks, develop reading promotion programs, improve testing systems, decrease teacher absenteeism, increase availability of textbooks, etc. and gender lens 1.3 Great engagement and Accountability by communities such as school management committees, utilizing education data for local decision making, engaging the private sector to provide materials, etc. - Facing Schooling Learnin Forward: for g in Africa World Bank General overview of best practices when schooling for learning in Sub-Saharan Africa with unique cross analysis of challenged faced as a country from 1990 - 2015 (ie conflict, natural disaster etc.) and progress in primary education (delayed, emerging, emerged, established). Shows how countries compare to others with similar experiences. More detailed review of best practices then presented on universilizing basic quality education, ensuring effective management/support of teachers, increasing education financing, closing institutional capacity gaps. - Training Teachers on the Job: What works and how to measure it. Policy Research Working Paper 7834, background paper to the World Dev Report World Bank￾Africa Region- Anna Popova, David Evans, Violeta Arancibia The paper shows that in-service training initiatives are not well reported on and not streamlined in part because there were no standard instruments to measure success. The authors create a new instrument called In-Service Teacher Training Survey Instrument (ITTSI) to look at what specific elements of teacher training programs can improve student learning in low and middle-income countries. They complete a mixed methods analysis of in-service teacher training programs using ITTSI and find important impacts on student learning due to provision of materials alongside training; provision of story books; providing training with a dual focus on pedagogy and a secondary focus on subject taught (but programs that focus only on pedagogy do not show these results); on engaging teachers on designing content for training; by holding training in a university or teacher center (as opposed to a hotel or government building); and by using participatory methods. The inclusion of follow-up mentoring visits to specifically review the material taught (not just monitor) was also important. - 89 Title Author Summary Key Words CDCS USAID The Tanzania CDCS has a goal of Tanzania's socio-economic transformation towards middle income status by 2025 advanced and three DOs: 1) Women and Youth empowered; 2) inclusive broad-based economic growth sustained; and 3) democratic governance improved. Tusome Pamoja fits in DO1, IR1.3 with cross cutting references to 1.1 on gender and 1.2, especially on stunting/nutrition. DO1 IR's are: IRs1.1- gender equality; 1.2-Health Status Improved; 1.3 Lifelong Learning Skills improved. The project structure aligns directly with sub-IRs under 1.3 that mirror the global USAID Ed strategy, which are: 1.3.1 is basic skills instruction improved, 1.3.2 Basic Skills Delivery System strengthened, 1.3.3, Parental and community engagement enhanced. Reading IR 1.3, relevant info is available on the 2014 Status of Tanzania's early education and reading programs. For example, there was no reading area in the curricula, no reading standards existed, and teacher training did not include skills in teaching reading. Background, Context TP Contract USAID The purpose of this Contract is to support the Government of Tanzania (GOT) in improving lifelong learning skills, defined as mastery of early grade reading, writing, and arithmetic. Technical assistance will cover: 1) improving quality of basic skills instruction at the early primary level; 2) strengthening Ministry of Education and Vocational Training (MOEVT) skills delivery systems; and 3) increasing community and parental engagement in early primary education. These three components align with GOT’s ten-year Education Development Sector Plan (EDSP) and other reforms underway, such as Big Results Now (BRN), Performance4Results (P4R), and Global Partnership for Education Literacy and Numeracy Support (GPE LANES) program on 3R’s (reading, writing, and arithmetic) reform. 67 USD Million over five years, signed 12/31/ 2015. Background, all topics of Activity TP SOW TP Proposal PMEP USAID RTI RTI No difference with Contract and vice versa, included SOW in Contract - difference states between 60-70 million USAID. Important information: Subs: MISKE Witte and Associates Inc., Plan International USA, Inc., Room to Read, and DataVision; Key Personnel: COP: Alex Alubisia, Sr. Reading Specialist, Sr. Finance Specialist: Daud Kweba, Sr. M&E Specialist: Daud Kweba. Helpful Results framework w three results and the intermediate results. 15 Indicators, rare way to present in the table without numbers. Mostly output indicators, i.e., No. of… Not a very robust Learning Agenda. I would say this is a must read for the Team. Background, M&E Annual WP RTI (TP/USAID) Strange that this was revised between May and January. Has annual outputs per Result and Intermediate results. This would be a useful reference to see how the Activity met these planned outputs. Also see how it aligns w the Annual Report. [there should be another WP] Background 90 Title Author Summary Key Words Annual Report RTI This document should be complemented with the most recent QR. Which should be Jan- March 2018. Program has the goal to achieve age-appropriate, curriculum-defined levels of reading and writing in Standards 2 and 4 for at least 75 percent of classrooms in the target areas over a five-year period. In first year, it set-up of Parent Teacher partnerships (PTP) to provide a classroom-by-classroom structure for school/community collaboration, and the provision of Standard 1 and 2 readers and associated training of teachers, head of school and decentralized managers from January 2017 onwards. Background, all project Annual Report (continued) RTI In support of the Government of Tanzania, the program has developed and supplied early-grade levelled classroom supplementary readers (10 titles for Mainland Tanzania and 10 titles for Zanzibar), teacher read alouds (big books) for classroom instruction (five 5 titles for Mainland Tanzania and 5 titles for Zanzibar), decodable readers for classroom teaching and learning (3 titles of sequenced short stories for Mainland Tanzania and 3 titles for Zanzibar), and 24 storybooks supporting preliteracy. 2 In 2017, 623,056 levelled readers, 54,980 teacher read alouds, and 314,342 decodable readers (Zanzibar only) have been distributed to date. A further 1,400,000 decodable readers (Mainland Tanzania), and more than 4,000 storybooks will have been distributed before the end of calendar year 2017. Background, all project Jan-April 2016 Quarterly Report - Most activities were start up focused including devising the PMEP. The target numbers in each district were revised. Started on some activities on 1.1 (initial identification of materials, etc.). Challenges identified regarding the differences between working in Mainland and Zanzibar. - May-June 30 quarterly report - Progress on 1.1 USAID Tusome Pamoja has developed 40 classroom readers (20 Standard 1 and 20 Standard 2) and 10 teacher story books (“read alouds”), made progress on the 2017 TLMs. Baseline assessment tools developed and ready to be used in September 2016. LQAS developed for in-school six monthly reading assessments; Challenges: Goals for timeline of material development were lofty; difference between Mainland and Zanzibar; overlap of component three on community work with GEP Lanes program and UNICEF; pre-primary requires more support than initially considered; there is little empirical research on parental engagement. Considerations/LL: may need to review CMEP for mainland versus Zanzibar; high demand for decentralized data; need to open Pemba sub-office under Zanzibar local office. Important: GOT asks the project to provide support to the SIS system. This is added to the project activities under component two IR 2.4 - 91 Title Author Summary Key Words Jan 1 to March 31 2017 Quarterly Report - Progress: Comp 1: 6 decodable readers for Standards 1 and 2 (three for each), and provision of supplementary student readers (25 titles for TZ and 25 for Zanzibar), 14k teachers participated in 12 teacher continuous professional development (CPD) modules, 24 storybooks aligned to pre-primary curriculum, pre-primary teacher manual, communities of learning manual support in school based CPO framework. This included the provision of 639,390 Standard 1 and 2 supplementary student readers for use in the classroom, training of all Standard 1 and Standard 2 teachers (8,230) in the use of student readers and introduction of key reading components, 2,975 Heads of School (HOS) and 747 school supervisors (Ward Education Officers (WEO) and Subject Advisors (SA)) in support of teachers, 1,295 education officers at district/council, region and national level in support to program implementation and the commencement of school￾based School Committee (SC) training (3,025) and Parent Teacher Partnership (PTP) set-up. Comp 2: Baseline complete; community engagement tools developed; DPLA; working on Quality Assurors/Inspector tools, Institutional Capacity Assessment for national implementation of the School Information System (SIS) and the subsequent National SIS Roadmap, Comp 3: Parent teacher committees set up, Challenges: The Classroom Readers approach of providing 10 titles per 10 children has been positive for teaching and learning, but has resulted in up to 16 children crowded around one-book and The limitation on book/title numbers means there are insufficient books for all children to take books home – a critical strategy for improving reading/learning ability; Heads of Schools cannot participate in every Teacher Continuous Professional Development (CPD) training; For CPD to work, it is critical for Ward Education Officers (WEO) to visit the school to support the Head of School and Teachers. Currently, there are no motorbikes or funds for WEO travel; There is only minimal evidence of school reports being acted upon. Upward reporting with no dialogue is undermining monitoring systems; Parents’ literacy is often a key factor to non-engagement with those unable or with limited reading skills often not feeling welcome in the school and Lack of reading materials in the home is seen as a key obstacle to parents’ greater involvement in their children’s reading Considerations/LL: No resource for WEO mentoring/ monitoring at school level; Greater collaboration is necessary between TIE, MOEST and PO-RALG to ensure that the implementation responsible Ministry is fully aware and included in the curriculum developments of the sector Changes: additional funding allows Mtwara to be added to pre￾primary. - 92 Title Author Summary Key Words Oct 1 to Dec 31 2017 - Key Progress: Comp 1: Standard 1 & 2 decodable readers: distribution of 1,531,150 readers to four Mainland target regions and 16,321 decoding and reading and writing assessment guides, Training of 61,77 teachers, 2,216 academic /Section Leader teachers, 2,558 heads of school (HOS), and 645 ward education officers (WEO), Printing and distribution of storybooks: 2,556 big and 12,780 small storybooks, and 219 teacher implementation guides, School leadership & ward coordination: 2,561 head of Schools, 2,044 academic teachers, 651 WEO (Mainland), 261 HOS, 282 section leaders, 5 subject advisors (Zanzibar) trained, Comp 2: Baseline findings disseminated to national stakeholders, Decentralized Periodic Learning Assessment (DPLA) data collection phase II completed; data analysis and report writing under development, Local Government Authority (LGA) monitoring facility: Commencement of LGA joint monitoring facility across target areas. LL: Decentralized Periodic Learning Assessment (DPLA) has received overwhelming government commitment and support, which has contributed to its smooth implementation, and hence more immediate feedback to schools, The Local Government Authority (LGA) and ward education coordinator facility for school visits and mentoring will not work without full buy-in from the decentralized system; Positive collaborations between village leaders and the school management harmonize and simplify community interventions, Remoteness of some schools affects implementation of Decentralized Periodic Learning Assessment (DPLA); some schools are not reachable during the wet season - 93 ANNEX VII: CONSOLIDATED DESIGN Consolidated Design30 Question Source Methods Analysis 1. To what extent is TP making progress towards improving target beneficiaries’ 3Rs skills? -Parent Survey [N=69], Classroom Checklist [N=10] - HT, GOT, And TP/ USAID -Parents, Teachers, and WEOs -Survey, School Checklist - KIIs (Qualitative) - FGDs (Qualitative) -Quantitative analysis in Excel -Qualitative analysis 2. How is the development of a SIS leading to improved learning outcomes for the 3Rs? -WEOs -HTs -Literature - FGDs - KIIs -Project Reports -Qualitative Analysis -Coding Documents 3. In what ways are government officials, HTs, and teachers demonstrating they have internalized capacity building provided by TP? -GOT Authorities, HT -Teachers -KIIs -FGDs -Coding with Dedoose 4. How can TP activities be done differently in order to more efficiently improve learning outcomes for 3Rs? -Teachers -HTs -GOT -TP Activity (Trainers, Staff) -Literature - FGDs - KIIs -Checklist, Survey -Coding/Analysis with Dedoose -Quantitative in Excel -Literature Review – Academic Documents 5. What are the factors that have implications for the sustainability of the quality of the 3Rs instruction after TP has ended? *In addition to those above… -DPs -GOT - KIIs -Coding/Analysis with Dedoose -Literature Review - Donor Documents and GOT Plans 30 Some differences in the original design and approach or number of targeted interviews and FGDS can be noted here. For a full explanation of the methodology and original evaluation design see Annex II (evaluation design). 94 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523