December 2013 This publication was produced for the USAID Time to Learn Project. It was prepared by Zachariah Falconer￾Stout, Jonathan Jones, and Lyn Messner, Time to Learn/EnCompass, LLC. U.S. Agency for International Development Time to Learn Project: Year Two Performance Review Report Time to Learn Project Time to Learn is funded by the U.S. Agency for International Development (USAID) in Zambia under contract number AID 611-C-12-00002, funded March 1, 2012. Time to Learn is implemented by Education Development Center, Inc., in collaboration with Camfed (the Campaign for Female Education), EnCompass LLC, and FAWEZA (the Forum for African Women Educationalists in Zambia). The project assists the Ministry of Education, Science, Vocational Training and Early Education through a 5-year national program to provide an equitable standard of education service for vulnerable learners, improve reading skills, and implement practical strategies to strengthen school quality and promote community engagement in community schools. Cover photo: By Zachariah Falconer-Stout, group discussion session with Parent Community School Committee members in Muchinga Province. USAID TIME TO LEARN PROJECT YEAR TWO PERFORMANCE REVIEW REPORT December 20, 2013 Contract Number AID-611-C-12-00002 DISCLAIMER This evaluation is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of Education Development Center and do not necessarily reflect the views of USAID or the United States Government. CONTENTS Acronyms ...........................................................................................................................................................................1 Performance Review Team..........................................................................................................................................2 Executive Summary ........................................................................................................................................................3 1.1. Review Purpose and Questions ..................................................................................................................3 1.2. Project Background.........................................................................................................................................3 1.3. Design, Methods, and Limitations .............................................................................................................4 1.4. Findings................................................................................................................................................................4 1.5. Conclusions ........................................................................................................................................................5 1.6. Recommendations...........................................................................................................................................6 2. Performance Review Purpose & Questions ...........................................................................................8 2.1. Performance Review Purpose......................................................................................................................8 2.2. Performance Review Questions..................................................................................................................9 3. Project Background...................................................................................................................................... 10 3.1. Project Context............................................................................................................................................... 10 3.2. Project Description ....................................................................................................................................... 11 4. Performance Review Design, Methods & Limitations..................................................................... 16 4.1. Performance Review Design ..................................................................................................................... 16 4.2. Limitations ....................................................................................................................................................... 19 5. Findings............................................................................................................................................................. 20 5.1. Understanding and Perceptions of TTL’s Intended Process and Results................................. 20 5.2. Implementation Process Related to the Literacy Instruction Teacher Training Cascade... 22 5.3. Implementation Results Related to the Literacy Instruction Teacher Training Cascade.... 23 5.4. Implementation Process Related to PCSC Management and Capacity Building.................. 25 5.5. Implementation Results Related to PCSC Management and Capacity Building................... 27 5.6. Implementation Process Related to Enhanced MESVTEE Monitoring of and Support to Community Schools ..................................................................................................................................... 28 5.7. Implementation Results Related to Enhanced MESVTEE Monitoring of and Support to Community Schools ..................................................................................................................................... 31 5.8. Implementation Process Related to Developing and Disseminating TLM.............................. 32 5.9. Implementation Results Related to Developing and Disseminating TLM............................... 34 5.10. Changes in Pupils’ Classroom Experiences.......................................................................................... 34 6. Conclusions and Recommendations ..................................................................................................... 36 6.1. Conclusions ..................................................................................................................................................... 36 6.2. Recommendations........................................................................................................................................ 39 1 ACRONYMS FY fiscal year IR intermediate result MESVTEE Ministry of Education, Science, Vocational Training, and Early Education OGCS Operational Guidelines for Community Schools OVC orphans and vulnerable children PCSC Parent Community School Committee TLM teaching and learning material TTL USAID Time to Learn project USAID U.S. Agency for International Development ZIC Zonal In-service Coordinator 2 PERFORMANCE REVIEW TEAM Zachariah Falconer-Stout, Research and Evaluation Specialist, Time to Learn/EnCompass, LLC Jonathan Jones, Senior Evaluation Specialist, Time to Learn/EnCompass, LLC Lyn A. Messner, Senior Research and Evaluation Advisor, Time to Learn/EnCompass, LLC Muchinga Province Data Collection Team Paul Daka, Provincial Outreach Coordinator, Time to Learn/Education Development Center Zachariah Falconer-Stout, Evaluation Specialist, Time to Learn/EnCompass, LLC Boniface Lisuba, Central Counterpart, Ministry of Education, Science, Vocational Training, and Early Education Tesho Pardon, Provincial Counterpart, Ministry of Education, Science, Vocational Training, and Early Education Lusaka Province Data Collection Team Godfrey Chitalu, Provincial Outreach Coordinator, Time to Learn/Education Development Center Nathan Kayombo, Intern, Time to Learn, Lusaka Provincial Office Charity Kombe, Central Counterpart, Ministry of Education, Science, Vocational Training, and Early Education Sylvester Zimba, Provincial Counterpart, Ministry of Education, Science, Vocational Training, and Early Education Southern Province Data Collection Team Zachariah Falconer-Stout, Evaluation Specialist, Time to Learn/EnCompass, LLC Kenneth Jinaina, Central Counterpart, Ministry of Education, Science, Vocational Training, and Early Education Nathan Kayombo, Intern, Time to Learn, Lusaka Provincial Office Peter Mufwinda, Provincial Outreach Coordinator, Time to Learn/Education Development Center Regina Siamusiye, Provincial Counterpart, Ministry of Education, Science, Vocational Training, and Early Education 3 EXECUTIVE SUMMARY 1.1. Review Purpose and Questions The Time to Learn project (TTL) is conducting internal performance reviews in Project Years 2 and 4 to understand project implementation and why the project has been effective and the ways in which it can be improved. The performance reviews augment performance monitoring data (ongoing) and impact evaluation activities (Project Years 1, 3 and 5) by providing intermediate data for learning and adapting. This report covers the first of the two performance reviews and provides an early opportunity for TTL staff, partners, and stakeholders to learn from project experiences thus far by providing evidence on what is occurring, why, and what changes are needed to address challenges and build on successes. This first performance review sought to answer three questions:  What should be continued and what should be done differently so that implementation proceeds as intended?  What should be continued and what should be done differently so that implementation contributes to desired results?  How are pupils experiencing literacy lessons? 1.2. Project Background Part of USAID/Zambia’s education portfolio, TTL is a 5-year (2012-2017), USAID-funded project that collaborates with the Ministry of Education, Science, Vocational Training, and Early Education (MESVTEE). The project seeks to improve reading among 500,000 primary grade learners by 2017 in all community schools in six of Zambia’s 10 provinces, and increase equitable education services for orphans and vulnerable children in secondary schools in these provinces. It is anticipated that by 2016, as a result of TTL’s interventions, community school teachers in TTL-supported schools will have improved skills in reading instruction; community school learners will achieve higher scores on standardized reading tests; communities will advocate more effectively for high-quality reading instruction and social services and support; and the MESVTEE will be better positioned to manage the quality of instruction in community schools. This performance review examined the following activities related to functioning of community schools:  Literacy Instruction Training Cascade for Teachers  Parent Community School Committee Management and Capacity Building  Enhanced MESVTEE Monitoring of and Support to Community Schools  Development and dissemination of teaching and learning material 4 1.3. Design, Methods, and Limitations To leverage learning, this performance review utilized a purposive sampling strategy designed to provide depth and capture the spectrum of stakeholder perspectives, rather than representative breadth. Data collection was carried out by 15 data collectors from the MESVTEE, TTL, and the Examinations Council of Zambia from September 23 to October 4, 2013 in the three sample provinces, and from October 14 to November 8, 2013 for central-level stakeholders. In total, 20 schools were visited across three provinces. These schools were selected to capture all three language groups with which TTL works, as well as a mixture of rural and urban areas, and provinces where stakeholders perceive the intervention as comparatively more successful, those considered average, and those facing challenges. The review also collected data from Zonal, District, and Provincial MESVTEE Officials and Provincial TTL staff in all zones, districts, and provinces in the sample, as well as Central MESVTEE Officials and Lusaka- and U.S.-based TTL staff. The performance review used the following methods:  Document review  Semi-structured interviews with teachers, Head teachers, traditional leaders, MESVTEE Officials (national and provincial) and TTL staff  Group discussion sessions with Parent Community School Committees (PCSCs), pupils, and MESVTEE Officials (district and zonal)  Observations of classrooms and schools. Tools were piloted during a Data Collectors Training held in September 2013. This internal performance review occurred one year after TTL activities were initiated in the three sample provinces. The short implementation timeframe means care must be taken to distinguish between incomplete implementation and inappropriate strategies. In addition, several activities were recently launched and ongoing during the data collection period. Because the sample was purposive and not representative, findings cannot be generalized to the entire TTL intervention area. 1.4. Findings Key findings from the performance review are as follows: Stakeholders demonstrated understanding of and appreciation for TTL’s role and purpose and many stated that they perceived improvement in pupil performance and attitudes. Stakeholders identified some cross-cutting constraints: small numbers of teachers, lack of teacher salaries, and inadequate allowances for TTL activities. Training content was seen as valuable and contributing to knowledge gain in the area of literacy instruction, but more follow-up in the form of additional training and monitoring was 5 desired. Ideas for how to achieve this additional training time varied, but there was a general wish for training to be conducted during school holidays in order to focus on the training and not be pulled away by their school-related responsibilities. Some training participants would like more recognition for participating in trainings. Teachers and Head teachers reported that TTL trainings helped them improve literacy instruction, but generally only the more basic teaching practices were observed in literacy lessons. Many stakeholders cited teacher motivation as an enabling factor for improving literacy lessons or community schools more generally, and specifically stated improved attitudes since TTL began. Community school management practices were reported as beginning to somewhat improve. PCSC members perceived training content as valuable and appropriate, and asked for more training and follow-up. Some concern was raised that training messages may not be reaching community members, and many respondents felt that not enough monitoring is being done. The process of developing and delivering the PCSC training content was valued by those involved. Parents, teachers and Head teachers reported that PCSCs are mobilizing other community members, participating actively in school governance, and increasing collaboration with schools. Change is slower for broader parental involvement, where perceptions were inconsistent. TTL-MESVTEE collaboration across project activities was viewed by the MESVTEE as building MESVTEE capacity at lower levels, but not yet at the central level; central level Officials, nevertheless, felt highly involved in project activities. There was strong appreciation by TTL and the MESVTEE for what was perceived as wide collaboration between each other. MESVTEE Officials would like more collaboration with TTL, in particular with regards to improving coordination of activities and schedules, and to ensure that the MESVTEE is seen in the leadership role. The MESVTEE also desired stronger collaboration among the USAID literacy projects. MESVTEE attitudes towards community schools and literacy pedagogy are seen to be improving, but perceptions of actual changes in MESVTEE management of community schools are inconsistent. The collaborative development process used for teaching and learning material (TLM) was appreciated, and the distribution process was seen to build ownership. In both development and distribution, human resources were a challenge leading to some bottlenecks. End-users found TLM practical and easy to use and there is a strong desire for more, especially for pupils. Limited data on TLM usage indicated materials are being used. 1.5. Conclusions Overall, TTL’s purpose and collaborative approach to implementation are well understood, highly appreciated by its stakeholders, and are starting to build country ownership. Many stakeholders would like to further improve collaboration in the year ahead. The MESVTEE highly valued TTL’s collaborative approach to TLM development and dissemination, training cascade 6 implementation, and development of PCSC training content. It is commendable that the project was able to generate such strong sentiment after only one year of implementation, and notable for its alignment with USAID Forward and PEPFAR principles of country ownership and “true partnership.” The MESVTEE would like to build on this by having a closer, deeper collaborative relationship with TTL and seeing more coordination between TTL and the other USAID literacy partners so to achieve “one MESVTEE literacy project” and assume more leadership thus improving the chances for project sustainability. Stakeholders highly valued teacher and PCSC training and the resulting changes they observed. However, they would like to receive recognition for training, more training time, and follow-up to training to ensure that the cascades are followed through. TLM received were highly valued by teachers and Head teachers, and were found practical and easy to use. But some TLM are not reaching target schools. Teachers expressed a desire for more materials for pupils. TTL’s activities appear to be contributing to desired results in its first year of implementation. The results include improvements in school management practices, parental and traditional leader involvement, teachers’ attitudes, PCSC engagement with schools, and MESVTEE attitudes towards community schools and literacy pedagogy. Actual changes in MESVTEE community school management were inconsistent. TTL-MESVTEE collaboration has been mutually beneficial, progress has been made, and the MESVTEE is fully on board with the project. The findings show an overwhelming sentiment that, as a result of TTL, PCSCs are becoming stronger and to a lesser extent, that parents are becoming more involved, although the depth of improvement varied. Teachers’ classroom instruction is showing that some basic good practices are being adopted, but the more complicated pedagogical skills promoted by TTL and the MESVTEE do not seem to be occurring on a broad basis. Teachers and Head teachers are moving to phonics-based instruction and modeling literacy, which can be considered first order skills. Nonetheless, practices that contributed to reading comprehension are not fully understood by all or it is too early in the project for change to occur at that level. For example, reading aloud was a dedicated training topic, but emerged as one of the weakest areas during classroom observations. Financial and human resources constraints among stakeholders are hindering additional changes. The small number of teachers and lack of payment for teachers in some community schools, and low or absent travel allowances for MESVTEE Officials and teachers to participate in TTL trainings are seen as preventing more changes and starting to cloud relationships with TTL. There are no crosscutting similarities or differences at the provincial level, however findings seem to indicate that rural and to a lesser extent peri-urban areas are seeing less success than urban areas. 1.6. Recommendations Training Cascades  TTL should engage with PCSCs, teachers and MESVTEE Officials to determine how TTL 7 training duration and timing can be improved to mitigate conflicting priorities, increase active participation and follow through, and improve knowledge absorption.  TTL should engage in discussions with teachers, PCSCs and the MESVTEE to determine the best way to meet the need for certificates or other forms of recognition for participation in TTL trainings in order to improve motivation.  TTL should develop a strategy with the MESVTEE for joint monitoring and follow-up on the teacher and PCSC training cascades to ensure that the training messages and content are consistent all the way down the cascade.  TTL should articulate its strategy and rationale related to travel allowances to all relevant stakeholders participating in the project.  TTL and the MESVTEE should include a feedback mechanism, such as written evaluations or pre- and post-knowledge tests at the end of each training, in order to elicit immediate input on content and logistics for each individual training as well as to improve the training cascades overall to increase participation and retention. Relationship with the MESVTEE  The MESVTEE should schedule regular meetings with all USAID literacy partners and USAID to enhance collaboration and coordination and reduce competition so the MESVTEE can work with “one literacy” project for Zambia. TTL and the MESVTEE should jointly consider whether there are existing structures at the MESVTEE that could assume this responsibility so as to prevent further strain on personnel. TLM  TTL should clearly articulate to its stakeholders what TTL-sponsored TLM are being developed to ensure transparent distribution and stakeholder understanding of what they should be getting so they can demand the TLM if they don’t get them.  TTL and the MESVTEE should establish procedures for TLM distribution, ensure they are followed, and document distribution reconciliation systematically, as well as identify funding for the MESVTEE to disseminate TLM to the schools in order to make the printing process more efficient. TTL and the MESVTEE should ensure that the end user has received the TLM, and there is back up documentation to substantiate receipt. 8 2. PERFORMANCE REVIEW PURPOSE & QUESTIONS 2.1. Performance Review Purpose The U.S. Agency for International Development (USAID) Time to Learn project (TTL) is conducting continual assessments over the life of the project through internal impact evaluations and performance reviews to provide the project with lessons learned and areas for improvement (see Exhibit 1: TTL Impact Evaluations and Performance Reviews). These evaluations assess TTL on three of the project’s five intermediate results (IRs) that contribute to improved literacy (indicated by the red boxes in Exhibit 2: TTL Results Framework in the context of USAID’s Results Framework). Combined, these evaluations and reviews reflect a multilevel and sequential mixed-method approach that enable TTL to assess its interventions at different points over the life of the project, and ultimately provide a holistic understanding of the project’s results over time. The impact evaluations will determine if TTL has achieved improved reading among grade 2 learners (USAID/Zambia IR 3.1) through a cross-sectional design utilizing the Early-Grade Reading Assessment protocol. Impact evaluation findings will be supplemented by assessment of change in teaching practice to determine if teachers are utilizing the techniques promoted by TTL through a longitudinal design utilizing the Standards-Based Classroom Observation Protocol in Education. 1 Performance reviews are being conducted in Project Years 2 and 4, in between the impact evaluation activities, to understand project implementation and why the project has been effective and the ways in which it can be improved. The performance reviews augment performance monitoring data and the impact evaluations. 1 The unit of analysis for teaching practice is the school, as TTL’s interventions seek to build the capacity of the systems that support teaching practice, i.e. school managers and teacher trainers in select geographic areas, and not of specific teachers. EXHIBIT 1: TTL IMPACT EVALUATIONS AND PERFORMANCE REVIEWS 9 This report covers the first of the two performance reviews and provides an early opportunity for TTL staff, partners, and stakeholders to learn from project experiences thus far by providing evidence on what is occurring, why, and what changes are needed to address challenges and build on successes. 2.2. Performance Review Questions The following three key questions were developed jointly with TTL stakeholders:  What should be continued and what should be done differently so that implementation proceeds as intended?  What should be continued and what should be done differently so that implementation contributes to desired results?  How are pupils experiencing literacy lessons? Questions 1 and 2 focus on TTL activities involving key determinants of effective education: community school teachers and Head teachers; involved parents/active communities; Ministry of Education, Science, Vocational Training, and Early Education (MESVTEE) leadership and management; and teaching and learning material (TLM). Question 3 relates to learners. A list of evaluation questions with sub-questions can be found in Annex 1. EXHIBIT 2: TTL RESULTS FRAMEWORK IN THE CONTEXT OF USAID’S RESULTS FRAMEWORK 10 3. PROJECT BACKGROUND 3.1. Project Context Zambia declared free basic primary education in the early 2000’s and male and female enrollment in early grades has increased steadily since, much of which is attributed to community schools (TTL Performance Management Plan 2013). Created by communities, community schools are typically grades 1 to 7, 2 managed by the community through a Parent Community School Committee (PCSC), which has the main responsibility for supporting the school. As the HIV epidemic swept through Zambia, community schools absorbed orphans and vulnerable children (OVC) who were unable or not attending government schools, including learners with disabilities, displaced persons, school-age children who have dropped out of or never attended formal school, children living in geographically isolated areas, and street and working children. Zambia counts approximately 3,000 registered community schools whose pupils comprise at least 18% of Zambia’s primary school pupils. 3 In 2011, the Government of the Republic of Zambia passed the Education Act, which included community schools as an official type of school along with government, grant-aided, and private schools. In order to integrate community schools better into the Zambian education system the government committed to improving MESVTEE supervision and support to community schools. Teachers in community schools are usually not government employees and many have no formal teacher training (often grade 12 education). Registered community schools are eligible to receive government assistance in the form of continuing professional development/in-service training, small grants, books or other materials, and seconded trained government teachers. According to TTL’s 2013 Baseline Study Report, very little MESVTEE support went to community schools and support given was often irregular and varied greatly across districts. The USAID Education Strategy 2011-2015 highlights evidence that early grade reading ability is a key determinant of retention and success in future grades, making literacy levels a particular concern. TTL’s 2013 Baseline Study Report showed that the majority of learners in community schools in the six TTL provinces were unable to sound any letters correctly (68%), unable to decode any nonsense words (90%), and unable to read any words from a passage (94%). Almost no child in the random sample was able to complete the reading-comprehension sub-task. Learners were slightly more successful at listening comprehension (64% of responses correct), 2 Grade 7 is the median highest grade of community schools in TTL’s intervention area, but there is great variation. See: TTL Baseline Study Report (2013). 3 TTL bases this number on the total number of schools in 10 provinces, but this is to be confirmed based on the 2012 verified community school data currently being processed by MESVTEE. 11 EXHIBIT 3: MAP OF PROVINCES WHERE TTL WORKS orientation to print (55% of responses correct), and English vocabulary (14% of responses correct) sub-tasks. 3.2. Project Description Part of USAID/Zambia’s education portfolio, TTL is a 5-year (2012-2017) USAID-funded project that collaborates with the MESVTEE to improve reading among 500,000 primary grade learners by 2017 in all community schools in six of Zambia’s 10 provinces (see Exhibit 3: Map of provinces where TTL works), and increase equitable education services for OVC in secondary schools in these provinces. It is anticipated that by 2016, as a result of TTL’s interventions, community school teachers in TTL-supported schools will have improved skills in reading instruction; community school learners will achieve higher scores on standardized reading tests; communities will advocate more effectively for high-quality reading instruction and social services and support; and the MESVTEE will be better positioned to manage the quality of instruction in community schools. TTL’s interventions aim to inform and inspire policy dialogue at the Central MESVTEE level, creating a favorable environment for effective implementation of MESVTEE policy for integration of community schools, and providing a wide range of MESVTEE actors with an opportunity to understand how to sustain and generalize or scale up these interventions. This performance review examined activities related to functioning of community schools, and those falling under three of the project’s five intermediate results (indicated by the red boxes in Exhibit 2: TTL Results Framework in the context of USAID’s Results Framework). These activities are:  Literacy Instruction Training Cascade for Teachers  Parent Community School Committee Management and Capacity Building  Enhanced MESVTEE Monitoring of and Support to Community Schools  Development and dissemination of teaching and learning material 12 3.2.1. Literacy Instruction Training Cascade for Teachers TTL developed and supports a cascade of trainings (the modules for which are designed in collaboration with the MESVTEE) that aims to build capacity from the Central MESVTEE down through the provinces, districts, and zones to teachers in community schools. The project planned to conduct two types of teacher trainings in FY2013: Quick Start Literacy Program and Monthly Trainings. The 2-day introduction/overview Quick Start trainings began in January 2013 and covered classroom literacy instruction basics for teachers. The cascade is structured as follows (and illustrated in Exhibit 4: Quick Start Teacher Training Cascade): TTL conducts a central-level training of trainers for Provincial MESVTEE Officials who in turn conduct a training of trainers in their provinces for District MESVTEE Officials, who then conduct a training of trainers in their districts for Zonal In-service Coordinators (ZICs), who in turn conduct a training of trainers in their zones for Head teachers, who then train teachers at their schools. The Monthly Trainings provided in-depth information on specific aspects of literacy instruction introduced in Quick Start Literacy Trainings. The Monthly Trainings were designed at the Central level, but as demonstrated in Exhibit 5: Monthly Training Cascade, featured a shorter cascade that began in zones where Head teachers were trained in 1 day. Head teachers subsequently trained other teachers in their schools through 2 half-day Teacher Learning Circles. At the time of data collection, the third zonal Monthly Training on alphabet sounds was in progress. The previous two Monthly Trainings were on reading, and writing, respectively. The fiscal year (FY) 2013 target and actual numbers of MESVTEE Officials, Head teachers MESVTEE and TTL train Provincial Officials Provincial Officers train District Officials District Officials train ZICs ZICs train Head teachers Head teachers train teachers EXHIBIT 4: QUICK START TEACHER TRAINING CASCADE EXHIBIT 5: MONTHLY TRAINING CASCADE 13 and teachers trained through the literacy instruction training cascade reported by TTL are provided in Exhibit 6: FY2013 Literacy Instruction Training Cascade Targets and Results. Starting in August 2013, TTL supported the MESVTEE to develop Education Leadership and Management guidelines for Head teachers in community schools. Following a TTL-facilitated national material design workshop, District MESVTEE Officials began the roll-out of training for Head teachers in late September 2013 to develop Head teachers’ capacity to better manage resources, information and records; conduct and supervise school-based assessments; assess effective teaching; and provide psychosocial counseling, environment, health and hygiene education, school improvement planning, and monitoring and evaluation of school performance. EXHIBIT 6: FY2013 LITERACY INSTRUCTION TRAINING CASCADE TARGETS AND RESULTS Cascade Level FY2013 Target FY2013 Actual MESVTEE Officials (central, provincial, district) 98 119 Zonal In-service Coordinators 461 471 Head teachers 1,611 1,665 Community school teachers 3,222 3,221 Total 5,392 5,476 Because the Education Leadership and Management guidelines training started the month before data collection, the review did not intend to include it. However, some Head teachers mentioned it spontaneously so it is represented in the findings to help inform overall project strategy. 3.2.2. Parent Community School Committee Management and Capacity Building The 2007 MESVTEE Operational Guidelines for Community Schools (OGCS) states that community schools must have PCSCs comprised of parents, teachers and prominent community members elected by communities. PCSC composition ranges from 6 to 13 members who serve for two years after which new members are elected. Teachers are accountable to PCSCs, and this accountability is viewed as a major strength of community schools. TTL trains PCSCs to increase their capacity to: manage and administer community schools, develop and implement school improvement plans, engage parents in the educational process, mobilize resources and community and private sector support, advocate/champion the cause and issues of community schools to local representatives, monitor teacher performance and results, and monitor and track students. TTL aimed to train 1,600 PCSC members in FY2013 and reported having trained 1,814 members. 14 Two primary activities were conducted to increase parents’ and communities’ capacity to support community schools: Community Literacy Mobilization trainings in March 2013, and review and orientation on the new draft OGCS in August and September 2013. The Community Literacy Mobilization training was held in two districts in each of TTL’s six provinces for two PCSC members from each school, who were supposed to train other community members. The OGCS orientation was conducted for all community schools in the TTL-intervention area with one PCSC member and one Head teacher from each school participating. Both trainings were designed and implemented through a similar process. A collaborative, national-level meeting was held with Central, Provincial, and District MESVTEE Officials, civil society, and other stakeholders who drafted the training content in conjunction with TTL. The representatives from each school then participated in a 2-day, district-level training and were expected, in turn, to train other community members. According to TTL staff, the OGCS orientation and consultation sought to improve teamwork by inviting both the PCSC Chairperson or any member of the committee, and the Head teacher. At the time of data collection, only the Community Literacy Mobilization training had been completed; the other training occurred concurrently with data collection. For this reason, this review sought to examine only the Community Literacy Mobilization training, but many respondents spontaneously cited the other training and in some cases, it was unclear to which training respondents were referring. 3.2.3. Enhanced MESVTEE Monitoring of and Support to Community Schools TTL has not articulated specific “capacity-building” activities in its FY2013 workplan, as it aims to work through MESVTEE structures and systems and reinforce its capacity to train, manage, plan, monitor, and evaluate community school progress toward improved education standards. This includes setting annual targets for increased financial support to community schools, developing monitoring and evaluation systems and instruments, and incorporating community schools into MESVTEE monitoring and supervision routines. For example, TTL supports the Community School Steering Committee, under the direction of the Directorate of Planning and Information, which TTL reported met to discuss the development of a National Policy for Community Schools, the continued development of the OGCS, and the organization of a Community School Forum/Symposium. 3.2.4. Develop and Disseminate Teaching and Learning Materials TTL aims to develop low-cost, easily replicable text books and instructional resources to improve reading instruction in community schools, such as a reading/learning kit for pupils and instructional resources for teachers and teachers’ manuals with scripted follow-up lessons. TTL also has produced and distributed a range of management materials, including attendance logs, enrolment forms, and continuous assessment booklets, in support of the Educational Leadership and Management activity described above. TTL intends to work with the MESVTEE Directorate of Planning and Information to ensure regular increases in TLM and textbooks provided to 15 community schools over the life of the project. In the last quarter of FY2013, TTL reported completing a range of teaching, learning, and classroom management materials and a distribution plan to move the materials from Lusaka to the provinces. For FY2013, TTL aimed to produce and distribute 4,000 teacher materials and 84,000 pupil materials. At the end of FY2013, TTL reported having produced and distributed 800 MESVTEE Basic Education syllabi, 7,000 Teaching Guides (teacher training modules), and prepared an additional 442,400 pupil materials for distribution in FY2014 (Story Cards and short story books in local language). 16 EXHIBIT 7: SAMPLING CRITERIA FOR PROVINCES Goal: Potential to yield rich data that capture the breadth of experience in TTL’s first year  Mix of urban and rural areas  Inclusion of all three TTL language groups (iCiBemba, CiTonga, and CiNyanja)  Representation of those provinces where TTL interventions are perceived as being comparatively more successful, those considered average, and those facing challenges 4. PERFORMANCE REVIEW DESIGN, METHODS & LIMITATIONS 4.1. Performance Review Design This performance review, designed as a learning tool for TTL and its stakeholders, was grounded in systems thinking and appreciative inquiry, and applied mixed-methods, utilization-focused, participatory approaches. This internal review focused on progress to date in achieving desired changes in behavior, and the factors perceived to have facilitated or hindered those changes. This performance review did not aim to measure project impact, rather to add context to the impact evaluations. It used a purposeful sample of cross-sectional data to assess current progress and issues. Stakeholders were actively engaged in the performance review design process. TTL facilitated a half-day focusing session on August 9, 2013 with 18 representatives from the MESVTEE (central and provincial); Zambia Open Community Schools; community schools; PCSCs; and the University of Zambia, as well as TTL staff. USAID/Zambia, the Examinations Council of Zambia, and the Curriculum Development Center were also consulted in preparation for the session. The focusing session used appreciative inquiry and participatory methods to generate the following:  Performance review questions  A purposeful sampling strategy that can cost-effectively yield the richest information for TTL at this point in time  A list of key informants  Composition of data collection teams. 4.1.1. Sampling To leverage learning, this performance review focused on depth, rather than representative breadth. By focusing on only three provinces and small number of schools (5) in each province, the review team was able to gather data across all stakeholder groups throughout the TTL system. Using criteria determined at the focusing session (see Exhibit 7: Sampling Criteria for Provinces), Lusaka, Muchinga, and Southern Provinces were selected for data collection. Selection of two sample districts per sample province was pre￾determined in order to evaluate the Community Literacy Mobilization training for PCSCs, which TTL 17 conducted in only two districts per province in FY2013. TTL Provincial Outreach Coordinators selected five schools within each province to represent the two districts and five zones per province, representing a mix of urban, rural, and peri-urban areas, and for accessibility. The review also collected data from Zonal, District, and Provincial MESVTEE Officials and Provincial TTL staff in all zones, districts, and provinces in the sample, as well as Central MESVTEE Officials and Lusaka- and U.S.-based TTL staff. 4.1.2. Methods The performance review employed document review, semi-structured interviews with teachers, Head teachers, traditional leaders, MESVTEE Officials (national and provincial) and TTL staff; group discussion sessions with PCSCs, pupils, and MESVTEE Officials (district and zonal); and observations of classrooms and schools. Methods used are described below, and Annex 2 contains all semi-structured interview guides, questions used in facilitated group discussion sessions, and the school observation and classroom observation protocols. Annex 3 provides a rationale for using and presenting qualitative data. Semi-structured stakeholder interviews with TTL partners, including training participants and individuals who have led or designed TTL trainings: Head teachers, teachers, Provincial and Central MESVTEE Officials, TTL staff, and traditional leaders. Interviews focused on TLM, training materials, the training cascade, successes and vision for community schools, capacity building activities (MESVTEE, parents, communities), involvement in and knowledge of the project, and wishes for TTL. Facilitated group discussions with PCSCs and District and Zonal MESVTEE Officials focused on participants’ understanding of TTL, and participation in and perceptions of TTL trainings and capacity-building activities. Pupil group discussion sessions were held to elicit their perceptions of the literacy lesson observed by the evaluators (see classroom observations below). School observations used a checklist to document TTL teaching and learning material at the school, where they were stored, and if they appeared open or used or both, and basic school demographic information. Photographs were taken of the visitors’ log to identify the number of visits by MESVTEE Officials over the past year. Classroom observations captured teaching practices to see if teachers are using the literacy techniques promoted by the trainings. These observations used a protocol developed in conjunction with TTL technical specialists specifically for this evaluation and designed to be grade appropriate. The protocol included 29 criteria grouped into eight literacy domains that corresponded to TTL training content delivered to date. Document review to assess the extent to which outputs are on target and to contextualize the findings. The team reviewed TTL project documents and reports listed in Annex 4. Data collection was carried out by 15 data collectors from the MESVTEE, TTL, and the Examinations Council of Zambia, all of whom were trained by TTL September 10-13, 2013 in 18 Lusaka, Zambia. The 4-day training aimed to ensure quality data collection by orienting participants to the review protocols and key qualitative methods underpinning the tools and providing opportunities to practice and reflect on the methods, including piloting the tools to determine feasibility and utility in answering the performance review questions. Data collection occurred from September 23 to October 4, 2013 in the three sample provinces, and from October 14 to November 8, 2013 for central-level stakeholders. Exhibit 8: Performance Review Sample details the sample by province, stakeholder group, and data collection method. To ensure achievement of the sampling target of five schools, the review team over-sampled by selecting seven schools per province resulting in a total of 20 schools visited. EXHIBIT 8: PERFORMANCE REVIEW SAMPLE Data Collection Method Lusaka Province Muchinga Province Southern Province Central Level TOTAL Semi-structured Individual Interview Teacher 5 1 7 - 13 Head Teacher 6 5 5 - 16 Traditional Leader 2 2 - - 4 Central MESVTEE Official - - - 4 4 Provincial MESVTEE Official 2 3 2 - 7 TTL staff 1 1 1 6 9 Group Discussion Session Central MESVTEE Official - - - 1 1 District MESVTEE Official 2 2 2 - 6 Zonal MESVTEE Official 3 2 4 - 9 PCSC 7 6 7 - 20 Pupil 7 - 6 - 13 School Observations 4 6 6 - 16 Classroom Observations 7 5 6 - 18 19 4.1.3. Data Analysis Qualitative data from individual interviews and facilitated group discussion sessions were transcribed and coded in Dedoose, a cross-platform application for qualitative and quantitative analysis. To ensure confidentiality, interview and group discussion data were coded and referred to only by stakeholder group and province, except for Provincial MESVTEE Officials, where the province is not included to ensure confidentiality. School and classroom observation data were entered into Excel for descriptive statistical and ranking analysis. Data from all sources were triangulated to verify emerging themes. Data were analyzed by source and disaggregated by province, area (i.e., urban, peri-urban, rural) and respondent sex. Except where indicated in the report, data analysis didn’t show significant themes or differences by province or area; no differences were found by sex. The findings, conclusions and recommendations presented here represent the full view of all EnCompass performance review team members. 4.2. Limitations This internal performance review occurred 1 year after TTL activities were initiated in the three sample provinces. The short implementation timeframe means care must be taken to distinguish between incomplete implementation and inappropriate strategies. Similarly, distribution of TLM was limited to teaching guidelines and material in Project Year 2 so data are based on a small selection of teacher-support materials. In addition, several activities were recently launched and ongoing during the period of data collection. Many respondents spontaneously cited trainings in progress, and in some cases, it was unclear to which training respondents were referring or if respondents were able to discern or recall differences between TTL trainings and trainings conducted by other projects or organizations. The sample was designed to capture the broadest possible range of perspectives and contexts to enhance learning, but findings cannot be generalized to the entire TTL intervention area. Efforts were made to capture pupils’ perceptions of literacy classes through participatory group discussions with the aim of triangulating with the observation and interview data, but pupils’ young age resulted in data with insufficient depth to contribute meaningfully to the overall analysis. Achieving sustained reading improvements in community schools is contingent in part on the MESVTEE taking leadership of the design, implementation, and monitoring and evaluation of education interventions. To this end, the MESVTEE is the primary TTL partner and all TTL research and evaluation activities are undertaken in the context of building MESVTEE capacity in these areas. Engagement of MESVTEE Officials, who are not seasoned qualitative evaluators, builds their capacity over the long-term, but could affect data quality (integrity and precision) in the short term. 20 5. FINDINGS The findings presented below in bold are organized by the TTL activities assessed by this performance review (Exhibit 9: TTL Framework) and two key performance review questions:  What should be continued and what should be done differently so that implementation proceeds as intended?  What should be continued and what should be done differently so that implementation contributes to desired results? EXHIBIT 9: TTL FRAMEWORK The Finding section starts with cross-cutting findings on the TTL implementation process and results, and ends with a finding related to TTL’s ultimate goal of changes in pupils’ classroom experiences The quotes presented below exemplify key themes emerging from the analysis. 5.1. Understanding and Perceptions of TTL’s Intended Process and Results Finding 1: TTL’s role and purpose are well-understood and appreciated by its stakeholders. In response to direct questions about their role in or understanding of TTL, interview and group discussion respondents demonstrated an accurate awareness of the TTL project. The majority of 21 respondents who provided this information (predominantly PCSC members followed by Zonal and District MESVTEE Officials) understood TTL’s role as increasing reading and writing in early grades with a focus on community schools. Many respondents understood that TTL is a USAID￾funded project working with the MESVTEE, teachers, parents, and community members with a focus on capacity building and training for these groups. A few respondents across all stakeholder groups spoke about teacher training as a component of the TTL project. The only exception was that the Head teacher and PCSC group discussion participants in one rural school reported never having heard of TTL. I don’t really know much but what I know is that it has come to encourage parents to be role models to children about learning to read and is encouraging children to develop a reading culture. PCSC group discussion, Southern Province Time to Learn is an initiative to sensitize parents on how to develop their community through uniting and carrying out projects in developing their community school. Traditional Leader respondent, Muchinga Province Provincial, Zonal and District MESVTEE Officials, teachers and Head teachers in Lusaka and Muchinga Provinces (all but one in Lusaka Province) spontaneously expressed deep appreciation and thanks for TTL. Maybe just to thank the project, for thinking of community schools and coming up with a project to help them. The government is overwhelmed, so to have a partner, for me, I am very happy and thankful just to have a partner. Just to encourage them where they may be discouraged in certain areas where the MESVTEE may say, “no we can’t do this, no we can’t do that.” To forge ahead. They’ve been a good partner. Provincial MESVTEE respondent Finding 2: Respondents identified the following as key constraints to achieving literacy: inadequate or absent allowances for TTL activities, small numbers of teachers, and lack of teacher salaries. MESVTEE Officials at all levels (with a plurality from Lusaka) felt that their involvement in the project was hindered by a lack of a transportation allowance. TTL staff and teachers in Lusaka Province said that the allowances provided to participate in TTL trainings were insufficient to cover their expenses, particularly transportation costs including standard transportation allowances regardless of travel distance. We have some people coming from very far away and does it make sense to be giving them the same allowance as those coming from right here? Zonal MESVTEE respondent, Southern Province The allowances should be increased to adequately cater for us. Teacher, Lusaka Province Some stakeholders, primarily PCSCs, traditional leaders, Head teachers and teachers from rural and peri-urban areas mentioned “not enough teachers” as a constraint to exacting more changes that would improve reading and writing in community schools, while MESVTEE Officials 22 (provincial, district, central) mentioned “unpaid teachers.” Some teachers and Head teachers, especially in Muchinga Province, said that that PCSCs need to provide more financial support to schools. I’m only having problems because, the fact that I’m alone here. So more changes would have been done here, but the changes that I stated as a result of these trainings, I’m not teaching effectively because I was not able to plan lessons. Head teacher respondent, Muchinga Province One of the factors that can hinder a change is teacher retention. These teachers are paid by the community, so if the community is not involved and they don’t pay the teacher, the teacher can stop at any time, even though they have received this beautiful training. Provincial MESVTEE respondent 5.2. Implementation Process Related to the Literacy Instruction Teacher Training Cascade A large number of teachers and Head teachers reported participating in a Quick Start Literacy Training and at least one Monthly Training at the zonal level or at their school as part of a Teacher Learning Circle. According to teachers interviewed, of those schools with more than one teacher and, therefore, appropriate for a Teacher Learning Circle, the majority had held one. A few teachers, from schools not included above, stated that they had not participated in a TTL training. At least one school reported receiving only part of the intended teacher training program, and another remarked that the Head teacher had not conducted the Teacher Learning Circle after participating in the zonal training. Teachers at two different schools reported receiving some TTL teacher training, but the data were unclear as to which trainings and what process the cascade went through. Accounts from TTL staff, teachers, and MESVTEE Officials indicated the cascade moving at varying speeds across provinces, districts, and zones. Finding 3: Training content was seen as valuable and contributing to knowledge gain in the area of literacy instruction, but more follow-up was needed. Teachers and Head teachers overwhelmingly expressed appreciation for the TTL training content and stated in interviews that they found it useful. MESVTEE Officials expressed appreciation, as well. Some in Muchinga and Southern Provinces specifically stated that the content was simple to understand. MESVTEE Officials in Lusaka and Muchinga Provinces, and teachers from Southern and Lusaka Provinces said that they especially appreciated the focus on phonics. The presentation of the lessons was being done in a simple manner, and quite understandable. Head teacher respondent, Muchinga Province When we attended workshop there was emphasis on sounds and this has helped reading outcomes. The success is attributed to such trainings. Head teacher respondent, Southern Province 23 Provincial, District, and Zonal MESVTEE Officials mentioned the importance of monitoring the training cascade’s implementation for message consistency across cascade levels, and to verify that the desired teacher behavior change is occurring after trainings. Most of these respondents saw a need for more follow-up than is currently being done. This view was supported by TTL respondents. I feel teachers get recruited every year. In the community schools, they’ve mentioned that is a challenge. Maybe they’re just there quarterly … It’s not good assuming that just because it has gone through the cascade it is being done. Central MESVTEE respondent Finding 4: Some training participants would like more recognition for participating in trainings. Some teachers and MESVTEE Officials at district and zonal levels (all but one from Lusaka Province) suggested providing certificates of training to participants. One TTL staff member expressed a similar sentiment and the topic of certificates also emerged from a PCSC group discussion in Lusaka Province. Some of these respondents suggested that a certification or points system be attached to the training program whereby trainees receive certificates after attending a certain number of sessions. Receiving certificates after training would motivate the teachers. Head teacher respondent, Southern Province Finding 5: MESVTEE Officials and teachers expressed a desire for more training time, ideas for how to achieve this varied. MESVTEE Officials (with a plurality from Lusaka) suggested that more trainings should be conducted, including follow-up training to account for attrition among those who have been trained. A few TTL staff, MESVTEE Officials (district and zonal from Lusaka and Muchinga Provinces) and teachers (Southern and Lusaka Provinces) recommended less frequent, but longer in duration trainings. MESVTEE Officials and teachers recommended allotting a longer period for trainings. TTL staff, Provincial MESVTEE Officials, and teachers suggested conducting TTL trainings when school is closed for holidays so they can focus on the training and not be pulled away by their school-related responsibilities. The first thing is, realistically, the trainings have been too short. Because when you talk about teaching literacy, it’s not something people can absorb in 2 days; you need at least 5 days. Zonal MESVTEE group discussion, Muchinga Province 5.3. Implementation Results Related to the Literacy Instruction Teacher Training Cascade Finding 6: Teachers and Head teachers reported that TTL trainings helped them improve literacy instruction, but only a few teaching practices were observed. 24 The majority of Head teachers and teachers interviewed said that TTL teacher trainings helped improve literacy instruction. MESVTEE Officials also reported having seen some improvements in teacher practice. This perception was most common among respondents in Lusaka Province, and least common among respondents in Muchinga Province. The improvements mentioned spontaneously most often by these three stakeholder groups (mostly teachers and Head teachers) were moving to phonics-based instruction and improved planning for literacy lessons. Specific forms of improved planning included adherence to the literacy curriculum and syllabus, and increasing the amount of classroom time devoted to literacy, with many specifically citing the “hour per day” curriculum guideline. The second most mentioned changes of classroom practice by teachers and Head teachers were modeling literacy by reading to pupils and encouraging them to read and write, and improving cross￾cutting pedagogical skills such as classroom management and using new methodologies, for example, group work. I had emphasized to say you have to teach literacy every day, it has to be on the timetable. But looking back, they used to teach children once a week, maybe twice a week, whereas now, maybe it’s on a daily basis. Head teacher respondent, Southern Province I have learnt how to make lesson plans and schemes of work. I plan my lessons well [and] I follow the one hour literacy lessons. Teacher respondent, Lusaka Province Nevertheless, when asked for specific examples, several of the teachers and Head teachers who stated they had improved literacy instruction could not point to a single specific change in classroom practice that they had implemented since the trainings began. Classroom observation of 18 literacy lessons supported both the perception of limited change, as well as the difficulty many teachers demonstrated in pinpointing specific changes. Of the eight literacy domains observed, the domains where teachers performed best were letter sounds, English vocabulary, and word decoding, where about half of the criteria TTL and MESVTEE literacy approaches promote were observed. The first and last of these correspond to phonics/phonemic awareness where, as sated above, teachers also perceived themselves to be doing relatively better. By comparison, fewer teachers were observed undertaking tasks related to reading comprehension, oral passage reading, listening comprehension, and orienting pupils to print. 1 LITERACY LESSON IN MUCHINGA PROVINCE 25 Finding 7: Community school management practices are reported as beginning to improve somewhat. Head teachers and MESVTEE Officials (at district, zonal, and provincial levels) primarily from Lusaka Province, spontaneously mentioned a perceived improvement in community school management. The specific forms of management mentioned were widely varied. The most cited were: Head teachers observing and evaluating their teachers, improved community engagement, and improved record keeping and budgeting. They are being responsible. They are keeping records. The managers are keeping school records, which you never used to see. You would go into these community schools and there wouldn’t be a single record. Provincial MESVTEE respondent Finding 8: Many stakeholders cited teacher motivation as an enabling factor for improving literacy lessons or community schools more generally, and specifically stated improved attitudes since TTL began. When asked in interviews and group discussions what has made successes in literacy lessons and community schools possible, a large number of respondents representing all stakeholder groups (primarily in more rural areas of Lusaka and Muchinga Provinces) said, “teacher motivation.” A section of teachers, PCSCs, and zonal and district MESVTEE Officials attributed this increased teacher motivation to TTL activity writ large, and some teachers said they have been encouraged by a perceived improvement among their pupils since the start of TTL. In general, teachers (mostly from Lusaka Province) said that dedication or love of the profession helped them make changes or achieve success, and fewer said that teachers are trying hard to implement changes or renew their teaching efforts more generally. My contribution was that I was a full time teacher, giving of myself to these workshops. Head teacher respondent, Muchinga Province Without TTL the children would still be at home despite the infrastructure being in place. We see children being more diligent in lesson attendance and teachers being more serious. Because of these interventions parents can proudly send their children to school. PCSC group discussion, Muchinga Province 5.4. Implementation Process Related to PCSC Management and Capacity Building Finding 9: PCSC members perceived training content as valuable and appropriate, and asked for more training and follow-up. During group discussions, PCSC members remarked favorably about the topics covered in the 26 trainings they attended, and what they learned. After the training, PCSC members reported a better understanding of their role in the success of community schools and felt a greater appreciation for the need to be more involved in their children’s schooling, overall. During group discussions, PCSC members in rural and peri-urban areas asked for more training. The need for more PCSC trainings was echoed by teachers and Head teachers. What we learnt was that we as parents have a mandate to help our children value education, more especially opening up to the children so that learning starts [at] home as opposed to sending children to entirely depend on teachers. PCSC group discussion, Southern Province Finding 10: Some concern was raised that training messages may not be reaching community members, and not enough monitoring is being done. PCSC group discussion participants expressed concern that the member trained was not doing enough to train others. Among these groups, some discussed the need for more follow up/monitoring of the training cascade. This sentiment was echoed by Provincial MESVTEE Officials who expressed concern that MESVTEE Officials weren’t doing enough to monitor training outcomes, and TTL staff who expressed concern about the lack of monitoring, in general. There must be regular visits by facilitators to evaluate and follow up what was delivered during training. PCSC group discussion, Muchinga Province PCSC group discussion participants from Lusaka and Southern Provinces suggested expanding PCSC trainings to include other community stakeholders. Teachers from Southern and Muchinga Provinces, and MESVTEE Officials from Muchinga and Lusaka Provinces made similar remarks, although their opinion about who should be included was mixed, and ranged from traditional leaders to more community schools. Hold open air sensitizations so that everyone in the community benefits because currently it is limited to parents at the school (small group). PCSC group discussion, Lusaka Province Finding 11: The process of developing and delivering the PCSC training content was valued by those involved. TTL staff and MESVTEE Officials in Muchinga and Southern Provinces who were involved in developing and delivering the PCSC training content stated that they valued the process and the role they played in terms of becoming more engaged in communities, and having ownership over the process. No respondents conveyed reservations about their role in the development of training content. I liked the way it was done in terms of we were given a chance to write the training manual, and we used the same training manual which we got ourselves. It was not a training manual which was produced by other people and then we got it; it was a manual that we produced ourselves and then we used it. Provincial MESVTEE respondent 27 5.5. Implementation Results Related to PCSC Management and Capacity Building A number of Head teachers, teachers, PCSC members, and traditional leaders spoke of improvements in parental and community involvement and attributed those improvements directly to TTL PCSC trainings. Finding 12: PCSCs are perceived to be mobilizing other community members, participating actively in school governance, and increasing collaboration with schools. Parents, teachers and Head teachers reported that PCSCs are mobilizing other community members by attempting to influence their attitudes towards education, urging them to play a greater role in their children’s education, or soliciting material contributions to the school. Many of these respondents expressed that this mobilization was a part of the TTL PCSC training program and, in some cases, said that those who had attended TTL trainings in turn passed the information to the other parents during meetings, or through informal or personal conversations. Teachers, Head teachers, TTL staff, traditional leaders, and PCSC members also mentioned that PCSCs are actively participating in school governance or convening more frequently. Almost all stakeholder groups mentioned that community collaboration – specifically within PCSCs, and between PCSCs and the school – had improved, or that the amount of conflict had reduced. The strategy of linking the community and the school under TTL seems to be more effective than the linking which was there before. Because it completes bringing the parents into the school […] So the net is cast widely in the way the community has been involved in the new strategy of community school linkage, of partnership. Provincial MESVTEE respondent Last year the committee members were not there. So as for this year now they are in place, so they are able to cooperate and join with us and cooperate with the teachers and say ‘what can we do? We can do this and this and this, so as to improve our community school.’ Teacher respondent, Southern Province Respondents across all stakeholder groups in Lusaka and Southern Provinces said that they perceived PCSCs to have a better understanding of their role. A limited number of Head teachers, teachers, traditional leaders, and TTL staff remarked on the positive role of traditional leaders. Forms of support were diverse, including using their influence to encourage parents on the importance of education, participating in PCSC meetings, and providing land for the schools. Several respondents reported on improved support from parents in general, but did not specify any particular form of support. Finding 13: Parents’ active involvement with community schools and support for their children’s education was reported as having increased in some schools. 28 In response to direct questions about changes in parental involvement in the last year, almost all Head teachers and half of the teachers interviewed, as well as some PCSC group discussion participants, traditional leaders, and TTL staff responded that PCSCs or individual parents are monitoring community schools by checking registers and pupil progress (frequently their own children but sometimes of the broader student body), and observing classes. They also said that household members are checking homework, reading to children, or telling stories and folktales, and that parents are providing support to community schools in the form of infrastructure, labor, and teacher salaries or in-kind payment (most commonly agricultural produce). Parents, teachers, Head teachers, MESVTEE Officials (central, provincial and district) and TTL staff reported a perceived improvement in parents’ attitudes toward education in general, and their children’s education in particular. This perception was most common in Southern Province. For example, many community-level respondents (teachers, Head teachers, parents, and traditional leaders) said that parents are taking an active role to reduce absenteeism, and early or forced marriage by sending their children to school. This year the parents have been more committed in helping…than the other years. The involvement of the TTL training has enlightened them of what they need to do. Teacher respondent, Southern Province I think it’s that TTL’s activity in these schools has really rejuvenated performance in these schools. I think the parents in particular are appreciating now the importance of the school, and their participation. Central MESVTEE respondent Although many parents, teachers and Head teachers perceived parental involvement as improving, several others perceived no changes in parental involvement in the past year, and an additional smaller group said that while there is improvement the pace of change remains slow; teachers were more likely than Head teachers to report no improvement. Most of the TTL staff interviewed echoed the sentiment that there had been no or only slow change in parental involvement. A few Head teachers and PCSC group discussion participants expressed the sentiment that changes in parental involvement were slow or non-existent. Parents come on open days to see the work of their children, though not most of them. Teacher respondent, Lusaka Province Unlike [in other] places, in peri-urban I have not seen much improvement because the parents are too busy to be sensitized and to sensitize others. Zonal MESVTEE group discussion, Southern Province 5.6. Implementation Process Related to Enhanced MESVTEE Monitoring of and Support to Community Schools 29 Finding 14: TTL-MESVTEE collaboration across project activities is viewed as building MESVTEE capacity at lower levels, but not the central level. TTL staff and MESVTEE Officials from zonal, district, and provincial levels primarily in Lusaka and Southern Provinces, mentioned the following activities (in order of most often mentioned) when asked what capacity building activities TTL had conducted in the past year: teacher training, OGCS orientation and Community Literacy Mobilization trainings for parents and communities, baseline study, educational leadership and management training, and MESVTEE participation in developing training and teaching/learning materials. TTL staff and Central and Provincial MESVTEE Officials stated that TTL-MESVTEE collaboration is a form of capacity building; this was the only form of capacity building that Central MESVTEE Officials mentioned. TTL staff added the Community School Steering Committee as a form of capacity building. Central MESVTEE Officials did not perceive TTL as conducting activities to build their capacity directly, but many felt that TTL is building the capacity of other stakeholders. Some of these respondents stated that the MESVTEE has learned to serve community schools better by being involved in the project. All but one Central MESVTEE Official listed a number of ways that they felt they are contributing expertise to TTL, listing specific TTL activities they have directly supported. Central MESVTEE Officials expressed their appreciation for their ongoing collaboration with TTL, and they and TTL staff spoke about the importance of collaboration to the success of the project, especially with respect to implementing the training cascade. They are very good programs, they are actually very good. I’ve never felt like that for any other organization. Like other, you tell, this organization has come to spend money around. But that is not the case with TTL, they are actually programs that are intended to help the Zambian child. Provincial Ministry respondent Finding 15: More collaboration with TTL is wanted from the MESVTEE. As illustrated in Exhibit 10: MESVTEE Capacity Building, MESVTEE Officials requested deeper collaboration with TTL, especially in terms of being informed of project activities so they can coordinate better internally; no MESVTEE Official requested less collaboration. 2 ZONAL MESVTEE OFFICIALS COMPLETE APPRECIATIVE INTERVIEWS AS PART OF DATA COLLECTION IN MUCHINGA PROVINCE 30 EXHIBIT 10: MESVTEE CAPACITY BUILDING My biggest wish, wow, it would be something which I have already said: at implementation level, there is need to improve, in terms of collaboration with the ministry, but my biggest wish is that at implementation level, that we will collaborate to see, what is planned, that it is also seen in monitoring, together with our partner. Provincial MESVTEE respondent TTL staff and Central MESVTEE Officials both stated a need for the MESVTEE to play an increasing leadership role in the project to ensure that the MESVTEE is driving implementation and increased project sustainability. I’ve heard reports where the officer, the [TTL] Provincial Outreach Coordinator, goes out to do the monitoring, and then of course the vehicle is TTL and the officer is TTL and then it is not seen as the Ministry. I understand that they have their mandates, and that we sometimes move a bit slowly, but I think we can work something out so that the ministry is the face of these activities. Central MESVTEE respondent Finding 16: The MESVTEE desired stronger collaboration among USAID literacy projects. Central MESVTEE Officials remarked on the need for better collaboration and coordination between TTL, Room to Read and Read to Succeed. One respondent said, “now, it’s starting to get better, they’re improving,” but all remarked that more needs to be done, and recognized that the TTL Chief of Party has been working to improve collaboration. These respondents remarked that TTL and Read to Succeed are doing almost the same activities with the only difference being the former works with community schools and the latter with government schools. This was perceived as “all the same work” and “duplicative.” One specific example given was producing books and literacy materials. These respondents perceived that these projects, especially TTL and Read to Succeed, were working in isolation, “each on its own, each in its own province, or even where projects overlap.” These respondents remarked that schools attach activities to individual projects because they 31 have different titles and they would like to see the projects work together as “one ministry program,” which would maximize resources and be owned by the MESVTEE, rather than being TTL or Read to Succeed activities. Come together so that we can do one scope and sequence and then work together. So it’s important that if we are coming for the cause of the child, the Zambian child, I think it is important for us to sit down and not duplicate the same works and over drain the same human resources from the MESVTEE.” Central MESVTEE respondent 5.7. Implementation Results Related to Enhanced MESVTEE Monitoring of and Support to Community Schools Finding 17: MESVTEE attitudes towards community schools and literacy pedagogy are seen to be improving. TTL staff, MESVTEE Officials (at all levels), and to a lesser extent Head teachers and PCSCs pointed to positive changes in attitudes by the MESVTEE. Subsections of these respondents stated that the MESVTEE is changing its attitude towards community schools, increasingly viewing these schools as its responsibility, understanding the need to integrate these schools into all the MESVTEE’s work, and demonstrating a more positive attitude toward literacy instruction and methodology. Before TTL came, I had very limited knowledge of community schools, but I think now, I get to know them better. Central MESVTEE respondent One other achievement is at the ministry level – it’s not only us, but in combination with Read to Succeed and Step-up – is that everybody has the same understanding about how we should go about literacy instruction… So we have common understanding of what literacy should be like in primary school, in early grades. And that’s also an achievement. TTL respondent In general, these respondents perceived these changes in attitudes as critical precursors to achieving the long-term desired results, and that it is a good first step for the first year and consistent with the level of effort put into capacity-building activities. I think it has broadened my scope in my knowledge for community schools. But whether we…here have done enough to make conscious the other top officials, I don’t know if we’ve done enough. I think that has remained as business as usual for us here. Central MESVTEE respondent I can’t say it has worked well, but it’s taking root and going in a positive direction. TTL respondent Finding 18: Perceptions of actual changes in MESVTEE management of community schools are inconsistent. 32 Perception of changes – and the current situation itself – in MESVTEE support to community schools was widely varied, ranging from “things are improving,” to isolated instances of, “it has gotten worse.” Community-level stakeholders (parents, teachers, Head teachers, and traditional leaders) described current MESVTEE support, providing a “snapshot in time” of how the MESVTEE is currently perceived by community schools. These respondents generally had difficulty pinpointing the extent to which this has changed over the project period. Specific forms of support mentioned by Head teachers and teachers included: providing grants to community schools in the form of TLM and Free Basic Education materials; monitoring community schools; including community schools in training; providing furniture, infrastructure, teachers; and advising on how to do something or an effort to systematically improve something at district level. We are given some few text books and story books. Teacher respondent, Lusaka Province Their support is mostly in monitoring form. They [MESVTEE Officials] come and advise us. They support us morally and financially. Teacher respondent, Southern Province At the same time, several parents, teachers, Head teachers, and traditional leaders from rural and peri-urban areas said that this support did not represent progress and perceived that there had been no change in MESVTEE support to community schools. A few said that support had decreased of late. This perception is substantiated by a partial review of visitors’ logs at schools where data collection occurred. The visitors’ logs showed only two documented visits by MESVTEE officials over the past year: one by a District MESVTEE Official in Muchinga who delivered desks and one by an MESVTEE Official regarding a Ministry of Health hygiene program. However, many of the photos of visitors’ logs were of insufficient clarity or legibility to determine the affiliation of visitors. Additionally, some schools did not have logs, thus preventing a full review of all schools visited. The zonal and district officials used to visit in the past but have stopped. Traditional leader respondent, Muchinga Province 5.8. Implementation Process Related to Developing and Disseminating TLM Many teachers and Head teachers interviewed stated that they have received or seen at least one of the following TLM: training materials (teacher training, OGCS, Educational Leadership and Management, Community Literacy Mobilization), Zambia Basic Education Syllabi, and newsprint or other basic education materials (e.g., pupil notebooks, chalk). No teacher or Head teacher reported, spontaneously, having received training packet annexes containing MESVTEE schedule for introducing letter sounds and flash cards that teachers can cut out. Several Head teachers and teachers interviewed, representing all three provinces but typically those from more rural areas, said they had not received any materials their school should have received according to TTL’s distribution plans, although poor data collector follow-up may have failed to uncover a number of instances where respondents misunderstood what is meant by TLM. 33 School observations were applied inconsistently, but those data show that 9 of the 19 schools where data were collected received some kind of TLM, although some of these appeared not to receive some of the TLM they were supposed to, and 5 of 19 did not receive any TLM, or were at least missing TLM they should have received. Finding 19: Understanding of what “teaching and learning materials” comprised varied widely. In response to questions on TLM, teachers and Head teachers gave different responses from TTL staff, and TTL field staff perceptions differed from TTL Lusaka-based staff. TTL field staff, teachers and Head teachers considered only learner materials as TLM, while TTL Lusaka-based staff considered all materials that aid in the teaching and learning process TLM: training guides containing pedagogical material, and resources that help teachers plan for class. Some respondents stated that they had not received TLM, then detailed training packets with literacy￾specific information and syllabi that they had. Nothing so far [was] given to us except for syllabi, some writing paper but completely no teaching and learning materials. Only monthly training modules given to the school but these have no bearing on pupils, who need teaching and learning materials. Teacher respondent, Lusaka Province Finding 20: TLM development process was appreciated, but human resources were a challenge. TTL and Central MESVTEE Officials spoke positively about MESVTEE involvement in TLM development. TTL staff mentioned the important role MESVTEE Officials played to help TTL build on existing materials and ensure that new content aligned with the needs of community schools. MESVTEE Officials expressed appreciation for TTL meaningfully including them in TLM development. Both stakeholder groups also perceived some challenges in the collaboration. TTL staff spoke of challenges associated with ensuring that the government provided the requisite human resources to develop the materials due to competing priorities and what Central MESVTEE Officials described as “lack of financial motivation” from TTL. … we have a challenge. The project did not seem to be ready to support the motivation of the people who are illustrating the books. We have our own illustrators here, but he has his own outside work, and he needs to be motivated financially. Central MESVTEE respondent Finding 21: TLM distribution process was seen to build ownership, but experienced some bottlenecks. TTL staff and Central MESVTEE Officials said that a key enabler for TLM distribution was MESVTEE involvement and collaboration with TTL at different levels of the process. These respondents said that this process allowed different stakeholders to feel ownership of the 34 materials and process. At the same time, respondents said that the distribution process was hampered by two delays: printing process and reaching teachers through the cascade. Reasons cited for these delays included no standardized procedure for the distribution process, and a lack of financial resources to disseminate materials to the schools. At times it has been the, sometimes some districts, at the zonal level, if the materials are taken to the zone, the zone is just quiet about it. At times the zone will keep the materials that they are not able to distribute themselves. TTL respondent Finding 22: End-users found TLM practical and easy to use and there is a strong desire for more, especially for pupils. Teachers and Head teachers who had received TTL materials found them easy to follow and useful in their work, especially in providing guidance on teaching reading. Specific materials reported as useful were: the pamphlet, training packet, and syllabi. Most teachers and Head teachers did not suggest any content improvements. Teachers, MESVTEE Officials, and PCSCs overwhelmingly expressed a desire for more TLM, particularly those intended for use by pupils, books, and teaching guides. Many simply noted that more TLM were needed in schools and did not specify what kind. Because most of my teaching I base on the syllabi, and the syllabi at least helps me to use, to go about lessons correctly. Head teacher respondent, Muchinga Province We need more materials at school and some reading material for parents to teach their children in. PCSC group discussion, Southern Province 5.9. Implementation Results Related to Developing and Disseminating TLM Finding 23: Data on TLM usage, although limited, indicated materials are being used. When asked about TLM usefulness, very few respondents described how they have used TTL￾sponsored TLM but the teachers and Head teachers who did, said that they used the materials for lesson planning and found them beneficial. Limited observation data indicated that the TLM received by schools are being used. Of the 11 schools where the evaluation team physically viewed TLM, nine of the packets were assessed by the observer as “opened and appear used” indicating that, at the least, these materials had not sat idly on shelves or in storerooms. 5.10. Changes in Pupils’ Classroom Experiences Finding 24: Stakeholders perceived improvement in pupil performance and attitudes. 35 Teachers, Head teachers, parents, and District MESVTEE Officials (mostly from Muchinga Province) perceived improved pupil performance in recent months. Parents and teachers also said that they have seen improved pupil attendance, classroom participation, and more positive attitudes towards literacy or education. Almost all respondents who mentioned improvement in attendance were in Muchinga Province, and those who mentioned improved attitudes were predominantly in Lusaka Province. At first, we had a very few number of pupils, because the teaching staff was too low. So, upon my coming into teaching, when the parents observed that at least their children are learning, are changing, because some others began writing and reading…And me, I am changing a lot because of the trainings I am receiving from organizations, like TTL in particular…I think there are more learners because...the parents have observed change. So, the number of them has increased from less to a big number. Head teacher respondent, Muchinga Province 36 6. CONCLUSIONS AND RECOMMENDATIONS The following conclusions are drawn from the findings presented in the previous section and illustrated Exhibit 11: TTL Framework: Summary of Findings, and are organized into three sections corresponding to the performance review questions. EXHIBIT 11: TTL FRAMEWORK: SUMMARY OF FINDINGS 6.1. Conclusions 6.1.1. What should be continued and what should be done differently so that implementation proceeds as intended? TTL’s purpose and collaborative approach to implementation are well-understood, highly appreciated by its stakeholders, and are starting to build country ownership, but many stakeholders would like more collaboration. TTL’s stakeholders demonstrated a clear understanding of what TTL is, what it does, and for what reason. Respondents expressed great gratitude for TTL’s work and focus on Zambia’s community schools. The MESVTEE highly valued TTL’s collaborative approach to TLM development and dissemination, training cascade implementation, and development of PCSC training content. 37 This collaboration was seen by the MESVTEE to be building its capacity at provincial, district and zonal levels. The TLM distribution process, in particular, has built ownership within the MESVTEE, but the MESVTEE has human resource constraints, which have caused some bottlenecks in the process. It is commendable that the project was able to generate such strong sentiment after only one year of implementation. USAID Forward and PEPFAR are committed to the principles of country ownership and “true partnership” with countries. Partnerships that develop partners’ capabilities to direct their own development, strengthen gaps, pave the way for new approaches based upon common directions, and achieve long-term, sustainable development. TTL has established a foundation to achieve this in the remaining three years, but there is room for growth. The MESVTEE would like to build on this moving forward by having a closer, deeper collaborative relationship with TTL and seeing more cooperation between TTL and the other USAID literacy partners so to achieve “one MESVTEE literacy project” and assume more leadership going forward, improving the chances for project sustainability. Teacher and PCSC training are highly valued by stakeholders and changes are being observed, but recognition and more training time are desired, and follow-up is needed to ensure that the cascades are followed through. The findings show that the teacher and PCSC trainings delivered were highly valued and viewed as contributing to changes in literacy instruction and involvement by parents and other community members in their community schools. PCSCs, MESVTEE Officials and teachers requested more training time either by having more frequent trainings or longer trainings. They also suggested holding the trainings during school holidays so participants can focus on the training and not be pulled away by their school￾related responsibilities. There is also a strong desire by all involved for TTL to provide certificates of training to recognize stakeholders’ participation and completion of TTL training courses. The PCSC and teacher training cascades are mostly being followed, but the extent to which they are being fully extended to the end target groups – parents, other community members, teachers – varied. More monitoring and follow-up by MESVTEE Officials and TTL staff was requested to identify where the cascade is successful and where it is falling short, as well as to verify that training messages are not being distorted over the course of the cascade. In most cases it appears that those trained are training others, but the extent to which the training messages and content are fully reaching community members is unclear. This lack of monitoring in between trainings has created a desire by the MESVTEE and TTL for more information to make decisions regarding the overall strategy of the cascade. TLM received are highly valued by teachers and Head teachers, and more materials for pupils are desired, but some TLM are not reaching target schools. The MESVTEE, with TTL support, is developing and distributing TLM, which are reaching some, but not all targeted schools. The TLM that are reaching schools and being used by teachers, were found to be practical and easy to use, and there is a desire for more TLM, especially for pupils. There is insufficient clarity across the cascade about what constitutes TLM, making it difficult to interpret data on the extent to which this activity is achieving desired results. 38 6.1.2. What should be continued and what should be done differently so that implementation contributes to desired results? TTL’s interventions are strengthening PCSCs and, in some cases, parental involvement. The findings show an overwhelming sentiment that, as a result of TTL, PCSCs are becoming stronger and to a lesser extent, that parents are becoming more involved, although the depth of improvement varied. Community school teachers’ accountability to PCSCs is considered a major strength of community schools so active PCSCs are key to a successful community school system. These findings were drawn from a broad cross-section of community members. That is, not only parents and PCSCs perceived themselves to be more active, but also teachers and Head teachers, groups that would be less expected to have a positive bias. Teachers were less likely to report that there had been improvement than Head teachers, and many TTL staff stated that PCSCs had not yet changed. The fact that Head teachers’ role includes working with the PCSC may put them in a better position to observe early changes. However, as data were collected only from districts where the Community Literacy Mobilization training was implemented, comparison with districts without training was not feasible. Therefore, it was not possible to know whether training or other factors may have influenced parental involvement. Changes by MESVTEE and PCSCs are emerging and may be influencing pupil performance and attitudes. TTL’s activities appear to be contributing to desired results in its first year of implementation. These include improvements in school management practices, parental and traditional leader involvement, teachers’ attitudes, PCSC engagement with schools, and MESVTEE attitudes towards community schools and literacy pedagogy. Actual changes in MESVTEE community school management were inconsistent. TTL did not established standalone MESVTEE “capacity-building activities,” but it intended to build capacity by working hand-in-hand with the MESVTEE each step of the way. This type of capacity building is hard to quantify and respondents may not perceive themselves to be benefitting from it. This made it difficult to pinpoint specific gains in MESVTEE capacity after the first year, but findings indicate that TTL-MESVTEE collaboration has been mutually beneficial, progress has been made, and the MESVTEE is fully on board with the project. Although this evaluation cannot attribute changes in pupil learning to the TTL project, the perception of improvement can foment positive attitudes towards TTL interventions, and motivate stakeholders to take risks and adopt changes. For example, as parents and teachers witness or perceive improvement that they attribute to new methodologies, they may be motivated to make further changes and investments in their school. First order skills are beginning to be applied in the classroom, but reading out loud is not happening, yet. Teachers’ classroom instruction is showing some basic good-practices starting to occur, but the more complicated pedagogical skills promoted by TTL and the MESVTEE do not seem to be occurring on a broad basis at present. The findings show that teachers and Head teachers are moving to phonics-based instruction and modeling literacy, which can be considered first order skills. However, practices that contributed to reading comprehension are not being fully understood by all, or it’s too early in the project for change to occur at that level. 39 For example, reading aloud was a dedicated training topic, but emerged as one of the weakest areas during classroom observations. Teacher practice does not change quickly, and the project started the teacher training cascade this year, so all indications are that the project is moving in the right direction in terms of teacher practices. Financial and human resources constraints among stakeholders are hindering more changes. The small number of teachers and lack of payment for teachers in some community schools, and low or absent travel allowances for MESVTEE Officials and teachers to participate in TTL trainings are seen as preventing more changes and starting to cloud relationships with TTL. There are no crosscutting similarities or differences at the provincial level, however findings seem to indicate that rural and to a lesser extent peri-urban areas are seeing less success than urban areas. As noted throughout the findings, there were some similarities and differences between provinces. There is, however, no overarching theme or construct that ties these similarities and differences together at the provincial level. The analysis did reveal some cross-cutting differences between rural, peri-urban, and urban areas that seem to indicate that rural, and to a lesser extent peri-urban, areas reported less success than urban areas. For example one rural school reported that it had never heard of TTL, and the perception of not receiving materials per TTL’s distribution plans was more prevalent among Head teachers and teachers in rural schools. With respect to training, PCSC members from rural and peri-urban areas asked for more training (members in urban areas did not). In terms of results, PCSCs, traditional leaders, Head teachers, and teachers from rural and peri-urban areas mentioned “not enough teachers” as a constraint to exacting more changes that would improve reading and writing in community schools. Teachers and Head teachers in rural areas were also more likely to mention teacher motivation as an enabling factor. Parents, teachers, Head teachers, and traditional leaders from rural and peri-urban areas reported not seeing progress with regards to MESVTEE support to community schools. 6.1.3. How are pupils experiencing literacy lessons? As indicated in the Limitations section, the review team conducted group discussions with pupils to glean their perceptions of literacy lessons and differences in learners’ perceptions from those of teachers, Head teachers and other respondents. However, due to pupils’ young age the group discussion data were of inadequate depth to contribute meaningfully to the overall analysis and the only Finding related to this was related to perceptions of pupil performance by teachers, Head teachers, parents, and District MESVTEE Officials. Therefore, there are insufficient data for the review team to draw conclusions. The Year 3 mid-line impact evaluation will provide concrete data on pupil performance. 6.2. Recommendations 6.2.1. Training Cascades  TTL should engage with PCSCs, teachers and MESVTEE Officials to determine how TTL 40 training duration and timing can be improved to mitigate conflicting priorities, increase active participation and follow through, and improve knowledge absorption.  TTL should engage in discussions with teachers, PCSCs and the MESVTEE to determine the best way to meet the need for certificates or other forms of recognition for participation in TTL trainings in order to improve motivation. This recognition could be after completing a series of trainings, required number of hours, or other criteria jointly determined with stakeholders, or a points system whereby participation in training leads to some sort of certification process.  TTL should develop a strategy with the MESVTEE for joint monitoring and follow-up on the teacher and PCSC training cascades to ensure that the training messages and content are consistent all the way down the cascade. While it is not feasible to monitor the entire change process, a good practice would be to monitor implementation periodically at randomly selected schools and training events and improve output tracking.  TTL should articulate its strategy and rationale related to travel allowances to all relevant stakeholders participating in the project. Coordination to align this policy with other USAID/Zambia education projects may be useful to mitigate confusion.  TTL and the MESVTEE should include a feedback mechanism, such as written evaluations or pre- and post-knowledge tests at the end of each training, in order to elicit immediate input on content and logistics for each individual training, as well as to improve the training cascades overall to increase participation and retention. 6.2.2. Relationship with the MESVTEE  The MESVTEE should schedule regular meetings with all USAID literacy partners and USAID to enhance collaboration and coordination and reduce competition so the MESVTEE can work with “one literacy” project for Zambia. TTL and the MESVTEE should jointly consider whether there are existing structures at the MESVTEE that could assume this responsibility so as to prevent further strain on personnel. 6.2.3. TLM  TTL should clearly articulate to its stakeholders what TTL-sponsored TLM are being developed to ensure transparent distribution and stakeholder understanding of what they should be getting so they can demand the TLM if they don’t get them. This will also contribute to more accurate monitoring and evaluation of TLM development, distribution, use, and utility.  TTL and the MESVTEE should establish procedures for TLM distribution, ensure they are followed, and document distribution reconciliation systematically, as well as identify funding for the MESVTEE to disseminate TLM to the schools in order to make the printing process more efficient. TTL and the MESVTEE should ensure that the end user has received the TLM, 41 and that there is back up documentation to substantiate receipt. The numbers reconciled at the level of the school (end users) must match the number acquired at the central level and dispatched. It should be documented where there are discrepancies. 42 ANNEX 1. EVALUATION QUESTIONS Determinants Key Evaluation Questions Sub-questions 1. Trained Teachers 1.1 To what extent is implementation proceeding as intended? What should be continued and what should be done differently? 1.1.1 To what extent is the teacher training cascade being implemented as intended and contributing to desired outcomes? 1.1.2 What has enabled or constrained implementation of the training cascade? 1.2. To what extent is implementation contributing to desired results? What should be continued and what should be done differently? 1.2.1 To what extent are teachers applying the knowledge and skills in which they were trained? 1.2.2 What has enabled or constrained teachers' application of new knowledge and skills in literacy? 2. Involved Parents / Active Communities 2. 1. To what extent is implementation proceeding as intended? What should be continued and what should be done differently? 2.1.1 To what extent is the PCSC training program being implemented and contributing to outcomes as intended? 2.1.2 What factors have enabled or constrained the cascade of the literacy mobilization and operational guidelines for community schools training? 2.2. To what extent is implementation contributing to desired results? What should be continued and what should be done differently? 2.2.1 To what extent are the PCSC trainings perceived to be changing or mobilizing community support for schools? 2.2.2 What factors have enabled or constrained community support for schools? 3. Ministry Leadership / Management 3.1. To what extent is implementation proceeding as intended? What should be continued and what should be done differently? 3.1.1 To what extent is capacity building for MESVTEE Officials being implemented and utilized as intended and perceived to be contributing to project outcomes? 3.1.2 What factors have enabled or constrained 43 Determinants Key Evaluation Questions Sub-questions implementation of the capacity-building activities? 3.2. To what extent is implementation contributing to desired results? What should be continued and what should be done differently? 3.2.1 How do MESVTEE Officials perceive capacity-building activities to be increasing their ability to support and manage community schools? 3.2.2 What factors have enabled or constrained MESVTEE Officials' ability to support and manage community schools? 4. Materials 4.1. To what extent is implementation proceeding as intended? What should be continued and what should be done differently? 4.1.1 To what extent are TTL-sponsored teaching and learning materials getting to classrooms? 4.1.2 What factors have enabled or constrained delivery of TTL-sponsored teaching and learning materials to classrooms? 4.2. To what extent is implementation contributing to desired results? What should be continued and what should be done differently? 4.2.1 To what extent are TTL-sponsored teaching and learning materials being used in the classroom? 4.2.2 What factors have enabled or constrained TTL-sponsored teaching and learning materials usage? 5. Learners 5.1. How are pupils experiencing literacy lessons?' 44 ANNEX 2. DATA COLLECTION INSTRUMENTS Interview Guides ........................................................................................................................................................... 45 Interview Guide: Head Teachers ........................................................................................................................ 45 Interview Guide: Ministry Curriculum Development Center ................................................................... 48 Interview Guide: MESVTEE Provincial Officers.............................................................................................. 49 Interview Guide: Provincial Education Officers............................................................................................. 51 Interview Guide: Teachers .................................................................................................................................... 51 Interview Guide: Traditional Leaders and other non-PCSC Community Members ........................ 54 Interview Guide: TTL Staff..................................................................................................................................... 55 Facilitated Group Discussion Guides .................................................................................................................... 58 Facilitated Group Discussion: MESVTEE Central and District Officers ................................................. 58 Facilitated Group Discussion: MESVTEE Zonal In-service Coordinators............................................. 58 Facilitated Group Discussion: Parent Community School Committees .............................................. 59 Facilitated Group Discussion: Pupils................................................................................................................. 59 Observation Protocols................................................................................................................................................ 60 Classroom Observation Protocol....................................................................................................................... 60 School Observation Protocol .............................................................................................................................. 63 45 Interview Guides Introduction and informed consent statement Thank you very much for setting aside time to talk with me/us today. EnCompass LLC is the evaluation partner for the USAID-funded Time to Learn project (TTL), which collaborates with MESVTEE to improve reading in community schools and increase equitable education services for orphans and other vulnerable children. The project provides training and capacity building for Ministry staff, head teachers, teachers, and PCSC representatives. We are conducting an evaluation to assess the implementation of the TTL program. We want to look at what progress has been made and what can be learned from experiences so far. For this evaluation we are doing interviews and group discussions in three provinces: Lusaka, Muchinga, and Southern. This evaluation is not an evaluation of the Ministry, school, teachers, or PCSC. We would like your perspective on the TTL program. I also want to tell you that I am not directly involved in any funding decisions for TTL. I am here simply to listen to your opinion about the TTL project. Before we begin this interview, I want to let you know that any information we discuss during this interview will not be connected to any specific person. If we use a quote from you, you will not be connected to that quote. Also, it is important for you to know that you are free not to respond to any of our questions or stop the interview at any time. The interview will take about one hour. If you don’t mind, I would like to record this conversation, solely for the purposes of having a backup of what you say in case I am not able to write everything down. Is that alright? (If not, take written notes only.) We would also like to take pictures for our report; you will not be identified in any photos. Is that alright? (If not, take no pictures.) Before we begin, do you have any questions about this interview? Interview Guide: Head Teachers Background 1. In what ways have you been involved with the TTL project? 46 Successes and Vision 2. I would like you to think of a time when you felt especially successful in your work and made a difference in pupil reading and comprehension. PROBE for examples of changes in behaviors by head teachers, teachers, pupils, Ministry officials and others vis-à-vis reading instruction in community schools. a. Tell me why you think you had this success? b. What role did you specifically play? c. What role did others play? 3. I would like you to now dream about the future for a minute. Imagine that it is now the year 2017 and we are meeting because your community school is a huge success in the area of reading. a. What changes will have been made in your school? b. What made these changes possible c. What are you as a head teacher doing differently than you are today? d. What about the other teachers at this school? Ministry Leadership and Management Strengthened 4. What kinds of support are Ministry personnel (at central, provincial, district, and zonal levels) providing to you or your school? a. What kinds of support have you most appreciated? b. What could be better? c. What other projects are contributing to your success? 5. What types of management or supervision is the Ministry doing at your school? 6. Over the last year, what changes have you experienced in the Ministry’s support and management at your community school? PROBE for assessment of reading standards, development and use of observation and testing protocols based on reading standards. Training Cascade 7. Which TTL trainings have you participated in over the past year? PROBE for head teachers receiving (from ZIC) and subsequently providing teachers the Quick Start Training and monthly trainings through Teacher Learning Circles (Alphabet sounds, Writing I, Writing II, Reading, and Read Alouds) 8. Who delivered the trainings? 9. What did you appreciate most about these trainings? 47 PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification). 10. How could the training be improved? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification. 11. Now that you have been trained by TTL, what training have you delivered? 12. Have you led Teacher Learning Circles? If so, what have you learned from leading the Teacher Learning Circles? 13. What additional training would be useful to you in designing and delivering reading lessons, and to help you support your teachers? 14. What has changed as a result of these trainings? PROBE for changes in teachers’ knowledge and skills and behavior in the classroom (e.g., teachers are teaching and reading in local languages, children reading more, collaboration amongst teachers in reading lesson planning) a. What else has helped you make these changes? b. What has prevented you from making even more changes? 15. What changes have you seen amongst your pupils during reading lessons over the past year? PROBE for: behavior, attitudes. Capacity of Parents and other Community Members 16. What are parents, traditional leaders, and other community members doing to support reading or education more generally? PROBE for: engagement with each other, their children, community schools and the MESVTEE. 17. How are parents and other community members involved in monitoring the quality of instruction and school performance? 18. How does the PCSC support your community school? a. What is different now, compared to last year? b. Why do you think the PCSC’s support has changed? c. How could the PCSC be even more supportive? d. What do you think helps the PCSC support the school? e. What prevents PCSC from being even more supportive? Teaching and Learning Materials 19. Which TTL teaching and learning materials are in your community school? 48 PROBE for: training packets, syllabi, newsprint, draft story cards, and annexes to training packets containing MESVTEE schedule for introducing letter sounds and flash cards that teachers can cut out. 20. Which TTL teaching and learning materials do you find most useful? Why? 21. Which TTL teaching and learning materials do you find least useful? Why? Concluding Questions 22. If you were granted three wishes to improve the TTL project to achieve your vision, what would your three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 23. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 24. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: Ministry Curriculum Development Center Background 1. In what ways have you been involved with the TTL project? Teaching and Learning Materials 2. Which TTL teaching and learning materials have been developed and disseminated to community schools? How many materials and to which schools? 3. What role has the Curriculum Development Center of Zambia played in the development of these materials? 4. What do you think has helped the development of these materials? a. What can be improved in how these materials are developed? 5. What about how the materials are disseminated. What has worked well? a. How could the dissemination process be improved? 6. To what extent do you think these materials are meeting needs of teachers and pupils to improve literacy? Training Materials 7. Which TTL training curricula have been developed and implemented? a. How many trainings 49 b. To whom? 8. What role has the Curriculum Development Center of Zambia played in the development of these training curricula? 9. What has supported the development of these curricula? a. What about the development of these curricula can be improved? 10. To what extent is the training being “cascaded” from Provincial officers to District officers to Zonal officers to PCSCs to community schools? a. What has supported the training cascade? What has made it successful so far? b. What can be improved about the training cascade? Concluding Questions 11. If you were granted three wishes to improve the TTL project’s efforts around teaching and learning materials, what would your three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 12. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? Any other comments/insights/questions you would like to share? 13. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: MESVTEE Provincial Officers Background 14. In what ways have you been involved with the TTL project? Successes and Vision 15. I would like you to think of a time when you felt especially successful in your work and made a difference in pupil reading and comprehension. PROBE for examples of changes in behaviors by head teachers, teachers, pupils, Ministry officials and others vis-à-vis reading instruction in community schools. a. Tell me why you think you had this success? b. What role did you specifically play? c. What role did others play? 16. I would like you to now dream about the future for a minute. Imagine that it is now the year 2017 and we are meeting because the community schools in your province are a huge success in the area of reading. 50 d. What changes will have been in community schools? e. What made these changes possible? f. What are MESVTEE officers doing differently? g. What about head teachers and teachers? Capacity Building 17. What TTL-sponsored capacity-building activities have you participated in over the past year? 18. What did you appreciate most about these activities? 19. In what ways could these activities be improved to help you support and manage community schools better? 20. What additional activities would be useful to help you support and manage community schools better? Training Cascade 21. Which TTL trainings have you participated in over the past year? What training have you delivered? 22. What did you appreciate most about these trainings? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification). 23. What is being done differently as a result of these trainings? PROBE for “cascading” the training down to district and zonal levels. a. What else has helped these changes happen? b. What has prevented further change? 24. In what ways could the training be improved? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification). 25. What additional trainings would be useful to you? Concluding Questions 26. If you were granted three wishes to improve the TTL project to achieve what would your three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 27. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 28. Do you have any questions for me? 51 THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: Provincial Education Officers TTL Project in Province: 29. In what ways have you been involved with the TTL project? 30. What have you appreciated most about TTL activities in your Province? Probe for: training cascade, capacity building, teaching and learning materials, support to PCSCs, etc. 31. In what ways could these activities be improved to strengthen the community schools in your Province? 32. What is being done differently in the community schools in your Province as a result of TTL? 33. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 34. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: Teachers Background 35. Use a maximum of 3 minutes to ask any questions you may have as a result of the class observation. These should be clarifying questions about why or why not certain behaviors were done. 36. In what ways have you been involved with the TTL project? Successes and Vision 37. I would like you to think of a time when you felt especially successful in your work and made a difference in pupil reading and comprehension. PROBE for examples of changes in behaviors by head teachers, teachers, pupils, Ministry officials and others vis-à-vis reading instruction in community schools. a. Tell me why you think you had this success? b. What role did you specifically play? c. What role did others play? 52 38. I would like you to now dream about the future for a minute. Imagine that it is now the year 2017 and we are meeting because your community school is a huge success in the area of reading. a. What changes will have been made in your school? b. What made these changes possible? c. What are you as a teacher doing differently than you are today? d. What about the head teacher at this school? Ministry Leadership and Management Strengthened 39. What kinds of support are Ministry personnel (at central, provincial, district, and zonal levels) providing to you or your school? a. What kinds of support have you most appreciated? b. What could be better? c. What other projects are contributing to your success? 40. What types of management or supervision is the Ministry doing at your school? 41. Over the last year, what changes have you experienced in the Ministry’s support and management at your community school? PROBE for assessment of reading standards, development and use of observation and testing protocols based on reading standards. Training Cascade 42. Which TTL trainings have you participated in over the past year? PROBE for head teachers providing teachers the Quick Start Training and monthly trainings through Teacher Learning Circles (Alphabet sounds, Writing I, Writing II, Reading, and Read Alouds). 43. Who delivered the trainings? 44. What did you appreciate most about these trainings? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification). 45. How could these trainings be improved? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification). 46. What additional training would be useful to you in designing and delivering reading lessons? 47. What have you done differently as a result of these trainings? 53 PROBE for changes in teachers’ knowledge and skills and in the classroom (e.g., teachers are teaching and reading in local languages, children reading more, collaboration amongst teachers in reading lesson planning, teachers reading out loud in class) a. What else has helped you make these changes? b. What has prevented you from making even more changes? 48. What changes have you seen amongst your pupils during reading lessons over the past year? PROBE for: behavior, attitudes. Capacity of Parents and other Community Members 49. What are parents, traditional leaders, and other community members doing to support reading or education more generally? PROBE for: engagement with each other, their children, community schools and the MESVTEE. 50. How are parents and other community members involved in monitoring the quality of instruction and school performance? 51. How does the PCSC support your community school? a. What is different now, compared to last year? b. Why do you think the PCSC’s support has changed? c. How could the PCSC be even more supportive? d. What do you think helps the PCSC support the school? e. What prevents PCSC from being even more supportive? Teaching and Learning Materials 52. Which TTL teaching and learning materials are in your community school? PROBE for: training packets, syllabi, newsprint, draft story cards, and annexes to training packets containing MESVTEE schedule for introducing letter sounds and flash cards that teachers can cut out. 53. Which TTL teaching and learning materials do you find most useful? Why? 54. Which TTL teaching and learning materials do you find least useful? Why? Concluding Questions 55. If you were granted three wishes to improve the TTL project what would your three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 56. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 54 57. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: Traditional Leaders and other non-PCSC Community Members Background 58. What do you know about the TTL project? Are you involved with the project in any way? a. If so, how? Successes and Vision 59. I would like you to think of a time when you felt the community was especially successful with pupil reading and comprehension. PROBE for examples of changes in behaviors by head teachers, teachers, pupils, Ministry officials and others vis-à-vis reading instruction in community schools. a. Tell me why you think you had this success? b. What role did you specifically play? c. What role did others play? 60. I would like you to now dream about the future for a minute. Imagine that it is now the year 2017 and we are meeting because your community school is a huge success in the area of reading. a. What changes will have been made in your school? b. What made these changes possible c. What are teachers and head teachers doing differently? Ministry Leadership and Management Strengthened 61. What kinds of support are Ministry personnel (at central, provincial, district, and zonal levels) providing to you or your school? a. What kinds of support have you most appreciated? b. What could be better? 62. What types of management or supervision is the Ministry doing at your school? 63. Over the last year, what changes have you experienced in the Ministry’s support and management at your community school? Capacity of Parents and other Community Members 55 64. What are parents, traditional leaders, and other community members doing to support reading or education more generally? PROBE for: engagement with each other, their children, community schools and the MESVTEE. 65. In what ways are parents and other community members involved in monitoring the quality of instruction and school performance? 66. In the past year, what differences have you seen in the way parents, traditional leaders, and other community members are supporting reading or education more generally? a. Why do you think these differences happened? b. How could the community be even more supportive? c. What do you think helps the community support the school? d. What prevents the community from being even more supportive? 67. What differences have you seen, in the past year, in the ways in which your PCSC engages with and supports the community school? a. Why do you think these differences happened? b. How could the PCSC be even more supportive? c. What do you think helps the PCSC support the school? d. What prevents the PCSC from being even more supportive? Teaching and Learning Materials 68. Which TTL teaching and learning materials are in your community school? 69. Which TTL teaching and learning materials do teachers and students find most useful? Why? 70. Which TTL teaching and learning materials do teachers and students find least useful? Why? Concluding Questions 71. If you were granted three wishes to improve the TTL project what would your three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 72. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 73. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. Interview Guide: TTL Staff Background 74. What is your role on the TTL project? 56 Successes and Vision 75. I would like you to think of a time when you felt especially successful in your work on the TTL project and made a difference in pupil reading and comprehension. a. Tell me why you think you had this success? b. What role did you specifically play? c. What role did others play? 76. I would like you to now dream about the future for a minute. Imagine that it is now the year 2017 and we are meeting because community schools in Zambia are a huge success in the area of literacy. a. What changes will have been made in community schools? b. What made these changes possible? c. What is are Ministry officials doing differently from today? d. What about head teachers, teachers, and PCSCs? Ministry Leadership and Management Strengthened 77. What capacity-building activities have been implemented with MESVTEE officials to date? a. What has worked well in the delivery of these capacity-building activities? What could be improved? b. In what ways have these capacity-building activities increased Ministry officials’ ability to support and manage community schools? c. How can capacity-building activities be improved to increase Ministry officials’ ability to support and manage community schools? Training Cascade 78. Which TTL trainings have been implemented over the past year? Who delivered the trainings? PROBE for head teachers training teachers Quick Start Training, monthly training and Teacher Learning Circles (Alphabet sounds, Writing I, Writing II, Reading, Read Alouds, Alphabet Sounds). 79. What about the trainings went especially well? PROBE for usefulness and practicality of information provided, duration of training (too long, too short), timing of training (season, school year, weekend), mode of delivery, certification, materials (letter cards at rear of Alphabet Sounds, language schedule, Story Cards, Handwriting Practice Sheets, Newsprint Paper for Writing, Syllabus) 57 80. What about the trainings could be better? 81. What do you think is the most appreciated aspect of the trainings? Capacity of Parents and other Community Members 82. In the past year, what changes have you seen in the ways in which parents, traditional leaders, and other community members are supporting literacy or education more generally? PROBE for: engagement with each other, their children, community schools and the MESVTEE. 83. In what ways are parents and other community members involved in monitoring the quality of instruction and school performance? 84. What differences have you seen, in the past year, in the ways in which PCSCs support the community school? 85. What do you think helps the PCSC support the school? What would you like to see more of? 86. What prevents the PCSC from being even more supportive? Teaching and Learning Materials 87. Which TTL teaching and learning materials have been developed and disseminated to community schools? How many materials and to which schools? 88. What has enabled the development and dissemination of these materials? 89. What has hindered the development and dissemination of these materials? Concluding Questions 90. If you were granted three wishes to improve the TTL project, what would they three wishes be? PROBE for improvements to specific aspects of the current project (actions, materials). 91. Is there anything else that you want to tell me but didn’t because I didn’t ask the right question? 92. Do you have any questions for me? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. 58 Facilitated Group Discussion Guides Facilitated Group Discussion: MESVTEE Central and District Officers We will not use any names in reporting, all quotes in the evaluation report will be attributed to a general stakeholder group (e.g., PCSC), not by individual. You are free to not participate in any activity and may leave the group.  What do you know about TTL and its capacity-building activities?  What changes have you made as a result of the capacity-building activities?  What have you appreciated most about these activities?  What enables you to support and manage community schools?  What challenges do you face in supporting and managing community schools?  What is your top wish related to TTL capacity-building activities? Handout: Three Wishes for TTL Capacity-Building Activities Reflect on what we have discussed regarding what you appreciate about TTL capacity-building activities, what has been accomplished so far as a result of these activities, and what enables or hinders your efforts to support and manage community schools. If you were granted three wishes for improving TTL’s capacity-building activities with MESVTEE officers, what would they be? (1) (2) (3) Facilitated Group Discussion: MESVTEE Zonal In-service Coordinators We will not use any names in reporting, all quotes will be attributed to a general stakeholder group (e.g., Ministry Officials), not by individual. You are free not to participate in any activity and may leave the group at any time.  What do you know about the TTL project?  In thinking about the TTL trainings you have led or participated in, what was one training that was an exceptional experience for you and the teachers you trained?  What are your three wishes for TTL trainings?  What do you feel has enabled or supported the implementation of TTL trainings, either as a facilitator or as a participant?  What challenges have you faced in the implementation of TTL trainings, either as a facilitator or as a participant? Handout: Exceptional TTL Training 1. Reflect on the TTL training you have led to date. Pick one training that was an exceptional experience for you and the teachers you trained. 59 2. Tell the story about this one, exceptional training to your partner. Be sure to include the following points:  Why do you think this training was exceptional?  What aspects of the training made it special to you? For example - was it the content, participants, length of time, timing of the training, methodologies used?  What was the biggest “take-away” from that training?  If you were granted three wishes to make such exceptional trainings happen more often what would they be? The following is for taking notes on your partner’s story; the notes will be used by the evaluation team so please write clearly and take complete notes. Notes on your partner’s story: Three wishes for TTL trainings: Best quote that came out of the interview: Facilitated Group Discussion: Parent Community School Committees We will not use any names in reporting, all quotes in the evaluation report with be attributed to a general stakeholder group (e.g., PCSC), not by individual. You are free not to participate in any activity and may leave the group at any time.  What do you know about the TTL project?  Think about the TTL literacy mobilization trainings in which you have participated: o How many trainings have you participated in? o Who facilitated the trainings? o What trainings have you led for other community members? o What have you appreciated most about the trainings? o What would make these trainings more relevant and useful for you? o Based on these trainings, what changes have been made by you or others in the community?  Probe for parent/household-based reading activities, support for children’s reading, PCSCs mobilizing parents and communities around reading and advocating to the MESVTEE on behalf of their schools, and parents’ involvement in monitoring quality of instruction and school performance.  What changes would you like to see more of? Facilitated Group Discussion: Pupils If you do not want to participate, you do not have to. No one will be angry with you. If you want to participate in the activity, but do not want to answer a question, you do not have to. There 60 are no right or wrong answers and everything you say will be kept private. No one but us will know what you have said. We want to have fun. Do you want to participate in this activity?  What did you like about the story you just heard? o PROBE for reading out loud in class, teacher showing the book to pupils, teacher placing finder on text as s/he reads  What did you like about reading out loud?  What did you like about practicing letter sounds and sounding out words by letter?  What is your favorite thing to read?  Who do you practice reading with at home?  What did you like about the writing you just did? Who can write their name? Can you show me?  What language do you speak at home? How do you like it when the teacher reads in [language used in class]? Observation Protocols Classroom Observation Protocol Background Information Province_______________________________________________________________________ District________________________________________________________________________ Zone_________________________________________________________________________ School________________________________________________________________________ Teacher’s sex  Male  Female Subject (Reading or Writing)__________________________________________ Grade level of Class observed  Grade 1  Grade 2  Other___ (if multi-grade) Name of Evaluator/Observer___________________________________________ Date of Observation ______ /______ /______ dd / mm / yyyy Time of Observation (note the time the lesson started) Beginning Time ____:____ Ending Time ____:____ (Use 24 hour time) 61 No. Question Yes No Comments / Additional observations I. ORIENTATION TO PRINT (top to bottom, left to right, return) 1. Does the teacher use his/her finger/pointer to show which direction to read and how to move to the next line? 2. Does the teacher watch learners’ fingers as they track across the page? 3. Does the teacher use terms such as “top of the page” “bottom of the page” “next line”? 4. Does the teacher show the learners what these terms mean by using a pointer or fingers? II. LETTER-SOUND KNOWLEDGE (letter sounds production & recognition) 5. Does the teacher teach or revise letter sounds? 6. Does the teacher demonstrate or require students to demonstrate phonemic awareness (letter sounds) to build and decode (take apart) words? III. WORD DECODING (nonsense words decoding, syllables, coding and decoding) 7. Does the teacher teach or revise syllables? 8. Does the teacher teach coding and decoding skills? (building words from syllables and taking apart words using syllables) 9. Does the teacher sound out the letters in words? 10. Does the teacher ask learners to sound out the letters in words? IV. ORAL PASSAGE READING (fluency) 11. Does the teacher offer learners an opportunity to read aloud to the class/teacher? 12. Does the teacher model fluent reading by reading aloud to the class? 13. Does the teacher provide (grade level) text for learners to read? 14. Do learners read in class? 62 No. Question Yes No Comments / Additional observations V. READING COMPREHENSION (comprehension) 15. Does the teacher ask questions about material that s/he has read to the class or that learners have read themselves? 16. Does the teacher ask learners to predict what will happen as they are reading (reading individually or together as a class)? 17. Does the teacher introduce new vocabulary found in classroom reading material? 18. Does the teacher talk about the meaning of the words? VI. LISTENING COMPREHENSION (comprehension) 19. Does the teacher check learners’ comprehension while reading aloud? 20. Does the teacher ask prediction questions? 21. Does the teacher encourage dialogue (not question and answer) between learners and/or between teacher and learner as a means of teaching listening skills? VII. ENGLISH VOCABULARY (vocabulary, listening comprehension, English language instruction) 22. Does the teacher teach English language? 23. Does the teacher introduce new English vocabulary to learners? 24. Does the teacher write English vocabulary so that learners can see, read and/or recognize the words? VIII. WRITING (handwriting/letter formation, drawing, original writing, copying) 25. Does the teacher practice strokes with the children? e.g., /// ||| \\\ OOO 26. Does the teacher explicitly teach learners how to form letters by writing, in their notebooks or on the board, the curves and lines in a letter? 27. Does the teacher give learners the opportunity to draw as a part of the lesson? 28. Does the teacher ask the learners to write their own thoughts and ideas, without telling them what to write? 29. Does the teacher ask learners to copy words and letters from the board? 63 School Observation Protocol School Information MESVTEE Support and Teaching and Learning Materials 1. Photo taken of school’s visitor’s log for the past year (from January 2013)?  Yes  No If no, why not? Province____________________________________________________________ District_____________________________________________________________ Zone ________________________________________________________________ School Name__________________________________________________________ Type of location  Urban  Rural  Peri-urban Grade level of Class observed  Grade 1  Grade 2  Other___ (if multi-grade) Number of students ___ Number of female students ___ Number of male students ___ Number of teachers ___ Number of female teachers ___ Number of male teachers ___ Does the school receive grants or other support from groups other than TTL or the Government of the Republic of Zambia?  Yes  No If yes, approximate amount _______________ (Consider discussing during head teacher interview how this support is supporting or hindering TTL interventions) Name of Evaluator/Observer____________________________________________ Date of Observation ______ /______ /______ dd / mm / yyyy 64 2. Requested head teacher to view TTL Teaching and Learning Materials received by the school (e.g., training packets, syllabi, draft story cards, and annexes to training packets containing MESVTEE schedule for introducing letter sounds, and flash cards that teachers can cut out)?  Yes  No If yes, list materials viewed and condition:  Opened and appear used  Opened and appear unused  Unopened  Unopened and inaccessible to teachers and students 3. Where are the materials kept? 4. If there are no TTL Teaching and Learning Materials in the school, why not? 5. Other relevant observations. 65 ANNEX 3. USING AND PRESENTING QUALITATIVE DATA Interview and group data collection capture perspectives of stakeholders in their own words. It is also useful to inform program interventions in order to understand what key stakeholders think is going well, where they see challenges, what changes they perceive to be occurring as a result of the intervention, and how they view the intervention overall. Qualitative data adds important context to help program stakeholders learn what needs to be continued or changed about the intervention thus increasing chances that the intervention will contribute to desired results. Qualitative data can also add important context to any results emerging from an intervention. If a change is observed, is it due to the design of the intervention? The manner of implementation? Sound management? Or a combination of these (and other factors)? These questions can be answered by capturing others’ perspective though qualitative data. From a program perspective, it is often as useful to understand the ‘how’ and ‘why’ questions as it is the binary question of whether a program has caused a result or not. Interview and group data collection are often conducted using semi-structured data collection tools. Important topics and a general question outline are typically prepared in advance. However, the evaluator also has leeway to word the questions and decide on sequence based on the flow of the interview/group discussion. The evaluator is also free to pursue topics that emerge during the data collection and are deemed important. When analyzing qualitative data captured through semi-structured methods it is incumbent upon the data analysts to identify emerging patterns and themes from the data. Presenting findings using proportions or percentages of respondents is typically not useful if the data are derived from semi-structured or open ended methods. For example, if data are presented as follows: “24 of 80 respondents felt that the training content was relevant,” the reader might wrongly interpret the finding to mean that the majority of respondents felt that the training content was not relevant. However, it could be the case that the majority of respondents were not asked a specific question on the relevance of training content during data collection. Alternatively, respondents might have provided such information spontaneously when answering a semi-structured question. Indeed, spontaneous responses to a semi-structured question are just as important, if not more important, than responses to structured questions, because the respondent is empowered to answer a given question however they choose. Due to the flexible style of qualitative research, qualitative data analysis depends on the experience and integrity of the data analysts to present the data in a manner that is fair, accurate and useful. Baseline data are frequently used in evaluations that seek to assess change over a period of time, but the necessity of baseline data should be determined by the evaluation questions. Because this evaluation seeks to understand project implementation from the perspective of stakeholders, baseline data is neither relevant nor needed as opinions on project implementation are minimal prior to the launch of a project’s activity. 66 ANNEX 4. DOCUMENTS REVIEWED Education Development Center, Time to Learn Project: Five Year and Annual Work Plan, September 30, 2012 Education Development Center, Time to Learn Quarterly Report, July 1 – September 30, 2013, FY2013 Quarter 4 Ministry of Education, Operational Guidelines for Community Schools, 2007 The Time to Learn Project: Baseline Study Report, June 30, 2013 Time to Learn 2013 Annual Portfolio Review, PowerPoint Presentation Time to Learn Performance Monitoring Plan: Five-Year Plan + Annual Plans, July 30, 2013 Time to Learn Project Plot No. #203B, Off Kudu Road Kabulonga, Lusaka Zambia