Contract No: 72067418D00001, Order Number: 72067419F00015 EVALUATION REPORT SUMMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA May 14, 2021 Contract No: 72067418D00001, Order Number: 72067419F00015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT EVALUATION REPORT SUMMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA May 14, 2021 Photo: Reading Support Project Lesson Plan for Setswana Home Language for Grade 2 Photo Credit: Khulisa Management Services (2020) AUTHORS Jennifer Bisgard, Project Director Margaret Roper, Senior Project Manager Benita Williams, Senior Evaluator Daleen Botha, Data Quality Analyst Leticia Taimo, Evaluation Coordinator Heather Dixon, Evaluator David Ndou, Evaluator Carole Metekoua, Statistician Mosangoaneng Leteane, Emergent Evaluator CONTACT DETAILS Jennifer Bisgard 26 7 th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: jbisgard@khulisa.com Web Address: www.khulisa.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contract No: 72067418D00001, Order Number: 72067419F00015 ACKNOWLEDGEMENTS Khulisa Management Services (Pty) Ltd would like to thank the United States Agency for International Development (USAID) for supporting the development of this report through the funding mechanism which has enabled the evaluation to take place and through their ongoing engagement with the evaluation team. Khulisa specifically thanks USAID for their contribution and support in the design and delivery of the evaluation, and their input into the content and review of all reports. Khulisa would also like to thank the Department of Basic Education (DBE) for their input into the evaluation design and the North West Provincial Education Department (PED) for their input and continuing endorsement. Khulisa also thanks the Foundation for Professional Development (FPD) and their Consortium partners for their collaboration, support, and input into various evaluation processes. Finally, Khulisa appreciates the time and effort by all who participated in the evaluation, without which this evaluation would not be possible, including the coaches, teachers, and SMTs that we interviewed in the North West Province of South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |i ABSTRACT Khulisa Management Services (Pty) Ltd conducted a Summative Evaluation of the USAID￾funded Reading Support Project (RSP) in 2020. The evaluation questions: 1. To what extent did the FPD Consortium implement the intended intervention design with fidelity? 2. Is there evidence of RSP uptake by teachers in class? 3. Is there evidence of changed practices based on the SMT support? 4. Was there any major divergence from the assumptions in the Theory of Change? 5. How could the management and implementation structures be improved? 6. How could the M&E systems be improved? 7. What are the unique challenges of delivery at scale? 8. Is the RSP time and cost efficient? 9. What is the actual vs. anticipated unit cost? 10. How does the RSP unit costs compared to the EGRS and how are they different? 11. What time commitment is required from other stakeholders to implement at scale? 12. Which efficiencies could have been realized? 13. Which economies of scale could be realized in Year 2 and beyond? 14. To what extent did COVID-19 affect implementation and how did the program respond and adapt? The design was mixed methods. Findings include: - Teacher training was delivered with fair fidelity. The coaching and School Management Team (SMT) interventions had little fidelity. - No substantial difference between the Coaching, and Learning and Teaching Support Material (LTSM) intervention was found. - The SMT intervention, as implemented, did not contribute to desired change. - Uptake by teachers across interventions was similar. - COVID-19 affected implementation and fidelity of the RSP. - Lessons can help scale future early grade reading initiatives. - The intervention was complex, poorly managed and not time or cost efficient. Key words: South Africa; Education; Early Grade Reading; Implementation Evaluation; Teacher Training; and Coaching Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |ii CONTENTS EXECUTIVE SUMMARY XIII SECTION 1: INTRODUCTION 1 SYNOPSIS OF TASK, PURPOSE, AND AUDIENCE 1 SECTION 2: BACKGROUND 4 EGRS I AND RSP 4 THE EVALUATION 6 DESIGN EVALUATION 7 FORMATIVE AND SUMMATIVE IMPLEMENTATION EVALUATION 8 SECTION 3: METHODOLOGY 10 METHODOLOGICAL LIMITATIONS AND CHALLENGES 11 BIAS AND CONFLICT OF INTEREST 13 SECTION 4: ABOUT THE INTERVENTION 14 DESCRIPTION OF THE INTERVENTION 14 1. IS THE RSP THEORY OF CHANGE LIKELY TO LEAD TO THE ANTICIPATED RESULTS? (QUESTION 1) 16 2. TO WHAT EXTENT DID THE FPD CONSORTIUM IMPLEMENT THE INTENDED INTERVENTION DESIGN WITH FIDELITY? (QUESTION 2) 17 2.1. DOSAGE AND COVERAGE 18 2.2. ADHERENCE AND QUALITY 23 2.3. PARTICIPANT RESPONSIVENESS 34 2.4. PROGRAM DIFFERENTIATION 38 3. WERE THERE ANY MAJOR DIVERGENCES FROM THE ASSUMPTIONS IN THE THEORY OF CHANGE? (QUESTION 5) 40 4. TO WHAT EXTENT DID COVID-19 AFFECT THE IMPLEMENTATION OF THE RSP AND HOW DID THE PROGRAM RESPOND AND ADAPT? (QUESTION 15) 42 4.1COVID-19 RESPONSE IN SOUTH AFRICA 42 4.2HOW DID THE RSP DESIGN CHANGE TO ADAPT TO COVID-19? 45 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |iii 4.3HOW DID TEACHERS ADAPT AND RESPOND TO TEACHING PRIOR TO, DURING AND AFTER SCHOOL CLOSURE? 50 4.4HOW DID SCHOOLS RESPOND TO COVID-19 AND ARE THERE DIFFERENCES ACROSS THE INTERVENTION TYPES? 54 SECTION 5: HOW COULD THE INTERVENTION BE IMPROVED? 57 5. HOW CAN THE MANAGEMENT AND IMPLEMENTATION STRUCTURES BE IMPROVED? (QUESTION 6) 57 5.1 THE EFFECT OF MANAGEMENT ON IMPLEMENTATION 58 5.2 MANAGEMENT IMPROVEMENTS NOTED IN 2020 59 5.3 AREAS THAT REQUIRED ATTENTION (2020) 60 6. HOW COULD THE M&E SYSTEMS BE IMPROVED? (QUESTION 7) 62 6.1 WERE THE M&E SYSTEMS IMPROVED IN 2020? 63 SECTION 6: INTERVENTION FINDINGS 68 7. WHAT EVIDENCE IS THERE OF RSP UPTAKE BY TEACHERS IN CLASS? (QUESTION 3)68 7.1TEACHER PRACTICE IN IMPLEMENTING RSP METHODOLOGIES 69 7.2CLASSROOM MANAGEMENT 75 7.3LEARNERS’ BOOKS 76 7.4 LTSM AVAILABILITY AND USE 81 7.5 CLASSROOM LIBRARIES 84 7.6DISTRICT OFFICIALS’ VIEW OF RSP UPTAKE IN THE CLASSROOM 85 8. IS THERE EVIDENCE OF CHANGED PRACTICES BASED ON SMT SUPPORT? (QUESTION 4) 86 8.1SMT CHANGE IN BEHAVIOR AS A RESULT OF SMT INTERVENTION 87 8.2 SMT VIEW OF COACH SUPPORTED TEACHER UPTAKE 89 8.3 TEACHER PERCEPTIONS OF SMT SUPPORT 90 9. WHICH EFFICIENCIES COULD HAVE BEEN REALIZED? (QUESTION 13) 92 10. WHICH ECONOMIES OF SCALE COULD BE REALIZED? (QUESTION 14) 94 11. WHAT IS THE ACTUAL VERSUS ANTICIPATED UNIT COST? (QUESTION 10) 95 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |iv 12. HOW DOES THE RSP UNIT COST COMPARE TO EGRS I UNIT COSTS AND HOW ARE THEY DIFFERENT? (QUESTION 11) 97 SECTION 7: LESSONS FOR THE FUTURE 100 13. WHAT ARE THE UNIQUE CHALLENGES OF DELIVERY AT SCALE? (QUESTION 8) 100 13.1 RSP TEACHER TRAINING 100 13.2 RSP COACHING 101 13.3 LTSM 103 13.4 EMBEDDING TEACHER PRACTICE 103 13.5 DISTRICT OWNERSHIP OF RSP 104 13.6 COLLECTING QUALITY DATA FOR ON-TIME DECISION MAKING AND MANAGEMENT 104 14. WHAT TIME COMMITMENT IS REQUIRED TO IMPLEMENT AT SCALE? (QUESTION 12) 105 15. IS THE RSP COST EFFICIENT? (QUESTION 9) 107 SECTION 8: DISCUSSION 108 SCALING READING PROGRAMS 108 SCALING RSP 110 LESSONS FROM THE RSP FOR PED REPLICATION 113 RSP IMPROVEMENT SUGGESTIONS: TEACHERS 117 RSP IMPROVEMENT SUGGESTIONS: SMT 119 SECTION 9: CONCLUSION 121 SECTION 10: RECOMMENDATIONS 122 REFERENCES 127 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |v ANNEXURES 1. SCOPE OF WORK AND AMENDMENT 2. PROFILES OF EVALUATORS 3. CONFLICT OF INTEREST STATEMENTS 4. EXECUTIVE SUMMARY OF FORMATIVE EVALUATION 2019 5. METHODOLOGY AND STUDY PLAN 5.1 DATA SOURCES AND RESPONDENTS 5.2 EVALUATION TOOLS 5.3 COACHING CRITERIA 6. FIDELITY TABLES AND INDICATORS 7. EVIDENCE OF TEACHER UPTAKE AND LTSM IN CLASSROOMS 8. COACHING CASE STUDIES 9. RSP THEORY OF CHANGE ASSUMPTION RATINGS 10. ANALYSIS OF LEARNER WORKBOOKS (2020) 11. LIST OF RSP MATERIALS 12. STATEMENT OF DIFFERENCE Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |vi TABLES Table 1: Frequency of school based workshops (2020) 22 Table 2: RSP online coach training 25 Table 3: RSP Term 4 coach training 25 Table 4: RSP and UNICEF coach training comparison 27 Table 5: Coaches communicated with SMT and Coaching teachers in 2020 27 Table 6: Coaches that conducted SBWs during coach shadowing 28 Table 7: Rating of coaches as excellent, average or as needing more development 29 Table 8: Adherence to observation stages during coach shadowing by all coaches 30 Table 9: The difference between SBW and Group Feedback 32 Table 10: Themes from SMT members on SMT PLCs / Workshops 34 Table 11: Perception of teacher training usefulness by intervention group 35 Table 12: Teachers view of coaches 36 Table 13: Teacher perception of improvement of RSP between 2019 and 2020 39 Table 14: Teaching response to COVID-19 by intervention and grade 51 Table 15: Focus of learning during last week (Term 4) 52 Table 16: Teachers ability to teach all learning areas in the last week (Term 4) 53 Table 17: Teachers indicating a difficult curriculum practice given COVID-19 53 Table 18: Revised Custom Indicators (2020) 64 Table 19: Summary of Data Quality Results 66 Table 20: LTSM Distribution in Sample Schools 66 Table 21: Number of Lessons Observed by District, Grade and Language 69 Table 22: Days with at least one full written sentence in learner exercise book 79 Table 23: RSP Lesson Plans used 81 Table 24: Lesson Plans used by grade 82 Table 25: Reported tablet use 87 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |vii Table 26: RSP Cost per unit cost versus anticipated unit cost 96 Table 27: RSP Cost per unit for full program 96 Table 28: Differences in the design elements of EGRS I and RSP 97 Table 29: Cost per unit for EGRS I and 2018 RSP Revised Design 98 Table 30: RSP Coach Salaries compared to Benchmark 98 Table 31: RSP LTSM costs compared to Benchmark 98 Table 32: RSP accommodation, venue and catering costs compared to Benchmark 99 Table 33: Coach to school to teacher ratios from 2019 to 2020 102 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |viii FIGURES Figure 1: RSP Treatment and comparison groups 4 Figure 2: RSP Evaluation process 9 Figure 3: RSP Theory of Change 15 Figure 4: Four RSP Budgets 15 Figure 5: Percent of teachers who attended 1 to 7 training sessions in 2019 and 2020 19 Figure 6: Teachers received 0 to 7 individual coaching sessions (5 term target = 6.25) 20 Figure 7: Number of teachers who attended 0 (none) to 6 SBW in 2020 (n=796) 21 Figure 8 Analysis of coach WhatsApp chats 48 Figure 9: Pages in DBE workbook with writing at least one full sentence 78 Figure 10: Days with at least one writing exercise completed in exercise books 78 Figure 11: Pages in DBE workbook with writing of at least one paragraph 79 Figure 12: Days with writing of at least one paragraph in learner exercise book 79 Figure 13: Average pages with at least one full sentence across intervention groups 80 Figure 14: Average pages with written work per language 81 Figure 15: RSP materials and their availability in the classroom 83 Figure 16: Teacher Management and Storage File 83 Figure 17: Classes with correctly completed book progress or management chart 84 Figure 18: Teacher perception of SMT support 91 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |ix ACRONYMS AND ABBREVIATIONS ATP Amended Teaching Plans BWE Benita Williams Evaluation CAPS Curriculum Assessment Policy Statements CA/SA Curriculum Advisor / Subject Advisor CPD Continuing Professional Development DBE Department of Basic Education DDD Data Driven District DQA Data quality assessment EFAL English First Additional Language EGRS Early Grade Reading Study EMIS Education Management Information System FPD Foundation for Professional Development GR Grade HL Home Language HOD Head of Department IDIQ Indefinite Delivery Indefinite Quantity JIT Just in Time KII Key Informant Interview LOLT Language of Learning and Teaching LTSM Learning and Teaching Support Material M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning Molteno Molteno Language Institute OUP Oxford University Press of South Africa Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |x PERFORM Practical Education Research For Optimal Reading and Management PERFORMANCE Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate PED Provincial Education Department PLC Professional Learning Community PMT Project Management Team PPE Personal Protective Equipment PoE Portfolio of Evidence PSRIP Primary School Reading Improvement Program R South African Rand (ZAR) RCT Randomized Control Trial RSP Reading Support Project SAIDE South African Institute for Distance Education SBW School based workshop SMT School Management Team SOP Standard Operating Procedure SoW Scope of Work ToC Theory of Change USAID United States Agency for International Development VSO Voluntary Services Overseas WITS University of the Witwatersrand WSoE Witwatersrand School of Education Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xi The following icons are used for navigation in the report: Group 1: Learning and Teaching Support Materials (LTSM) and Teacher Training (referred to as the LTSM Intervention) Group 2: Coaching, LTSM and Teacher Training (referred to as the Coaching Intervention) Group 3: a subset of Group 2 schools which received Coaching and LTSM, as well as a School Management Team (SMT) program (referred to as the SMT Intervention) Teacher Training COVID-19 Monitoring Data / M&E / Theory of Change Fidelity Teacher Classroom Uptake Classroom Libraries Management Structures Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xii Scale Time and Cost Meets standards or target met Does not meet standards or target not met Partial / somewhat meets standards No evidence or assessment due to implementation change Divergence from design, target or assumption Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xiii EXECUTIVE SUMMARY The Reading Support Project (RSP) was an educational program implemented in 252 schools1 in two educational districts in the North West Province of South Africa (Dr. Kenneth Kaunda and Ngaka Modiri Molema) from January 1, 2019 to September 30, 2020. The United States Agency for International Development (USAID) provided a no-cost extension up to December 31, 2020. RSP focused on improving the pedagogical practices of Grade 1 to 3 teachers to teach reading in Setswana Home Language (HL) and English First Additional Language (EFAL). The RSP was originally working in two other districts (started in 2016). The RSP was revised to scale the Department of Basic Education’s (DBE) Early Grade Reading Study (EGRS I2 ), which was a randomized control trial (RCT) with 230 schools from 2015 – 2017. In 2019, the Foundation for Professional Development (FPD) Consortium started implementing the RSP in two educational districts in the North West Province (the same as the EGRS: Dr. Kenneth Kaunda and Ngaka Modiri Molema). Since the RSP was set up as a randomized control trial, the schools were randomly allocated to three groups, each receiving a different combination of structured pedagogy interventions, which include: Group 1: 121 schools provided Learning and Teaching Support Materials (LTSM) and Teacher Training referred to as the LTSM Intervention. The LTSM was a set of lesson plans aligned with Curriculum and Assessment Policy Statement (CAPS) and integrated with an extensive package of LTSM. Teachers were trained quarterly just before the start of each school term. LTSM packages were provided to 263 schools. Group 2: 140 schools received Coaching from 14 coaches as well as LTSM and Teacher Training. This is referred to as the Coaching Intervention. The coaching design specified 20 individual coaching sessions over two years, and 15 school based workshops (SBW), which was intended as a forum for group coaching. Group 3 (a subset of Group 2): 65 schools received Coaching, LTSM, Teacher Training and a School Management Team (SMT) training program referred to as the SMT Intervention. The SMT training involved Head of Departments (HOD) and principals attending five leadership development training sessions with the requirement of compiling a portfolio of evidence. In addition, one hundred of the RSP schools also received classroom libraries through the DBE. 1 The original target was 263 schools. However the target was amended to 255 schools as some schools opted out of the intervention. Analysis indicates that in October 2019 RSP was implemented in 256 schools, and in 2020 in 251 schools. 2 Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xiv Concurrent with the educator focused activities, the RSP provided professional development to Curriculum advisors responsible for support to Foundation Phase teachers in EFAL and Setswana Home Language. They participated in coach training, monitoring of teacher training, and the participation in a reference group that vetted all LSTM before they were finalized. RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning. Overall, RSP was designed to improve the reading skills of learners in the Foundation Phase (Grades 1 to 3). This US $5.5 million project3 was managed in conjunction with the South African DBE. The Foundation for Professional Development (FPD) implemented the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), South African Institute for Distance Education (SAIDE) and one international partner, Voluntary Services Overseas (VSO). VSO and SAIDE both exited the consortium in 2019. For the period 2019 - 2020, USAID commissioned Khulisa Management Services, to conduct a Design and Implementation Evaluation of the Reading Support Project (Award Number: AID-674-A-16-00007) to the United States Agency for International Development (USAID) under Indefinite Delivery Indefinite Quantity (IDIQ) Contract Number: 72067418D00001, Order Number: 72067419F00015. The task order has three phases: • The Design Evaluation determined the extent to which the RSP revisions incorporated EGRS I elements and could be comparable, differences between the design of the RSP and the EGRS I, and whether the RSP design could support the EGRS I impact goals. • The Formative Evaluation provided an account of how the RSP was adapted to include EGRS, whether the adapted design was likely to lead to impact, if the dosage and coverage of services was sufficient to support pathways to impact, and whether there was fidelity to the intended design in the actual implementation. • The Summative Evaluation provides an overall assessment and accountability of the FPD Consortium in terms of effectively and efficiently carrying out the RSP program. As a primarily qualitative evaluation, the Summative Evaluation aims to enhance understanding of, and illuminate, the complexities associated with scaling-up the RSP. Khulisa is in the process (2020-2021) of collecting data for an impact evaluation. There are fifteen Summative Evaluation questions. The methodology comprised a mixed methods approach involving collection, collation, triangulation, and valuing of qualitative and 3 A budget of $5,505,295 was approved. Including the amount of $2,144,670 that was spent on the original RSP prior to its redesign, the total USAID expenditure was $7,649,965 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xv quantitative primary and secondary data. The evaluation scope did not include an LTSM materials review, as a separate larger literacy review was planned outside the scope of this assignment. The findings in this report should be read with an understanding of the disruptions caused by COVID-19 in 2020, which was the second year of RSP implementation. South Africa imposed a strict lockdown in March 2020. Schools were closed and all gatherings were prohibited. Primary schools reopened in a phased manner from August 2020, with the youngest learners returning last. While some learners were given school work during Term 2 (typically April to June) most had no activities during this period. To maintain social distancing, schools deployed a platooning model which meant that learners attended either every other day or every other week. The school year was extended until mid-December 2020. ABOUT THE INTERVENTION Is the RSP theory of change likely to lead to the anticipated results? (Evaluation Question 1) This question is not answered in this report, since it was extensively answered in the design evaluation report (refer to https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf). To what extent did the FPD Consortium implement the intended intervention design with fidelity? (Evaluation Question 2) The teacher training intervention was delivered with fair fidelity in 2019, but less so in 2020. The RSP training program reached 1,925 teachers (it targeted 1,530) and also reached the intended number of schools. However, only 47 percent of teachers attended at least six out of seven training sessions. The RSP delivered the intended number of training sessions in 2019, but not in 2020. Training in Term 2 of 2020 was cancelled because of COVID-19 disruptions in schools. A cluster training approach with fewer teachers and shorter sessions were delivered in Terms 3 and 4 of 2020. There was little fidelity to the design of the coaching intervention. Teachers only received an average of three visits over the two-year period, which is significantly lower than the planned ten coaching sessions per annum. SBWs data in 2019 was incomplete. In 2020 only 194 sessions were recorded across the 135 coaching schools, averaging 1.4 SBWs, far below the target of 15 SBWs per school, per year. The SMT intervention was implemented with poor fidelity. The SMT intervention reached the planned number of SMT members, and almost all received the tablets with pre-loaded content. However, only 16 percent of the SMT members attended all five training sessions, and only 31 percent completed all four assignments in their Portfolio of Evidence (PoE). Only about a third of the schools received a support visit in 2019. Overall, the teachers were responsive to the coaching and LTSM. The coach training appears to have added value to coaching knowledge and practice. Teachers involved in both EGRS I and RSP indicated that the coaching quality was similar. The evaluation was unable to determine if there were changes between the baseline and endline on indicators that serve as a proxy for teacher classroom practice. For example, the Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xvi workbook analysis showed that the amount and quality of work completed by learners declined between baseline and endline, but this is likely because of the COVID-19 disruptions in schools in 2020. The evaluation was also not able to determine if a minimum standard for classroom practice were met at endline, because this standard is not clearly set out in the CAPS, it was also not communicated by the RSP. To what extent did COVID-19 affect the implementation of the RSP and how did the program respond and adapt? (Evaluation Question 15) The formative evaluation showed that the RSP implementation fidelity, of especially the coaching intervention, was constrained in 2019. The RSP had little opportunity to improve on it in 2020, due to severe disruptions related to COVID-19. RSP was severely affected by COVID-19. Lockdown restrictions meant that coaches could not go into schools, affecting both their role and function. FPD encouraged coaches to provide teacher support through WhatsApp groups during the lockdown period. Analysis of the WhatsApp groups indicates that coaches primarily used the platform for logistics. However, two coaches provided coaching and psychosocial support, illustrating that coaching and psychosocial support was possible online. The FPD Consortium moved coach and head coach training, and the Reference Group meetings with Provincial Education Department (PED) and District officials online. Distribution of the LTSM relied on school principals collecting materials during school closures due to COVID-19. Schools in the SMT intervention were better equipped to plan and distribute LTSM than the other intervention schools. Only a third of teachers (39%, 121 out of 311) across all interventions reported sending work home for learners during school closures. Most schools did not implement remote learning. The District circulated the DBE Amended Teaching Plans (ATPs) to teachers when schooling reopened. Coaches supported the teachers to integrate the LTSM into the ATPs and classroom implementation. Overall, the findings indicate that despite planned adaptations, implementation was affected and FPD Consortium adaptations did not sufficiently overcome the COVID-19 disruptions to maintain teaching and learning during the lockdown. Were there any major divergence from the assumptions in the Theory of Change (ToC)? (Evaluation Question 5) The ToC assumptions were documented in the Design Evaluation. In general, the evaluation team found that the assumptions documented in the Design Evaluation ToC, were plausible and proven. A number of the assumptions seemed plausible but remained untested, and yet others were not met. The Formative and Summative Evaluation found implementation remained generally partial or insufficient, and therefore, many of the assumptions related to activity dosage and reach were not met. They could plausibly have been met, but weren’t in the case of the RSP, simply because not all training or coaching were offered in 2019 and 2020. For example, neither the assumption “teachers attend all training sessions” nor the assumption “teachers schedule time for coaches and lesson observation takes place” were fully met. The assumption “all teachers receive all LTSM” were also not fully met in 2019. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xvii There were other reasons beyond implementation fidelity issues why some assumptions were not met: The assumption “Coaches have the necessary competency and skills to coach” was not met for at least one training session in 2019, because they were unable to execute dry runs. The assumption “Adequate catch up sessions are implemented for those who missed training” were not met because coaches did not really understand what catchup training entailed, and reported sharing any group feedback and/or School Based Workshops at coaching schools as catchup. The assumption “SMT receive the necessary support from international volunteers” were not met because VSO international withdrew from RSP. Some assumptions could not be tested by the evaluation team, because the evaluation was not designed to do so, or unforeseen data constraints limited what was possible. For example, the assumption “It is possible to implement the strategies in the lesson plans with large classes” wasn’t tested because the team did not observe implementation of lesson strategies at endline. The assumption “Teachers show improvement in lesson coverage, pacing, curriculum coverage” remained untested because the comparison of 2019 and 2020 learner workbooks was problematic due to the significant disruption in schooling in 2020. HOW THE INTERVENTION COULD BE IMPROVED How could the management and implementation structures be improved? (Evaluation Question 6) The Formative Evaluation identified the following management and implementation structures that needed improvement: (1) ensure the proper dosage and coverage is delivered, (2) track implementation and flag issues as they arise, (3) ensure the quality of delivery and (4) facilitate positive responses to the RSP by school management and teachers. Recommendations were made to improve: LTSM delivery and quality, roles and responsibilities, reception of RSP in schools (and with the districts and province), and SBWs. In response, FPD developed an RSP Improvement Plan for 2020. The plan focused on strengthening 17 areas to increase implementation fidelity, improve monitoring and reporting, and increase accountability and program quality. In 2020, improvements were noted in working relationship between DBE and the RSP Consortium members. The DBE and FPD directed and focused approach to include the Curriculum/ Subject Advisors (CA/SAs) in 2020 improved District sense of ownership. The CA/SAs were observed helping with RSP implementation and with training teachers. A reported success of the implementation and management of the project was the positive and accepting reception of RSP at schools in 2020. However, program implementation was still weak in 2020 according to key informants. Particularly in the coaching intervention and targeting teachers for consistent attendance at the training was problematic. Areas identified for further improvement include consortium management structures and accountability, managing absenteeism of teachers at teacher training and coach visits, and integrating the SMT intervention. How could the M&E systems be improved? (Evaluation Question 7) In 2019, the Formative Evaluation found that the RSP monitoring and evaluation (M&E) systems needed to be improved. Recommendations included improving and adjusting indicator design, tool design, data collection and management, and reporting. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xviii The Formative Evaluation recommended that the Monitoring, Evaluation, and Learning (MEL) plan reflect the final iteration of the RSP design. Together, these improvements were necessary to produce reliable data for tracking implementation in 2020. Although improvements are noted in the MEL Plan for 2020, data collection and data management practices remained weak. The M&E system did not accurately track implementation fidelity nor was it used for adaptive program management. The systems provided some degree of accountability but were unintegrated and poorly administered. Which efficiencies could have been realized? (Evaluation Question 13) It is possible that greater efficiencies could have been realized. The budget process for RSP was not ideal since the initial budget was awarded for a different design. There were problems with the budget for the revised design, as some costs were not budgeted for and later added. A long period of implementation lapsed before the final budget was approved in March 2020. This negatively impacted FPD and USAID’s ability to monitor expenses against budget, resulting in inefficiencies. Which economies of scale could be realized? (Evaluation Question 14) Economies of scale were anticipated, but not realized as follows: 1. Bulk printing: the materials first were constantly being adapted in 2019 and then needed to be further adapted due to COVID-19. Therefore, no economies of scale for printing, and some print runs were not fully utilized or provided, as they needed further adaptation (e.g., to accommodate the change in the curriculum). 2. Per teacher support: since most primary schools run their timetable the same for all three grades, for example, EFAL taught at the same time for Grades 1, 2, and 3. Monitoring data show that coaches observed only two classes a day. 3. Support: due to compliance requirements and duplication of effort by consortium members, the analysis shows two support personnel for each person involved in direct service delivery. The scale of the program may have reduced economies of scale in some cases. Assembling the correct skills, content knowledge and experience to reach more than 1,500 teachers in 252 schools required a large consortium. In turn, this meant duplication of some management and support functions and inefficiencies. INTERVENTION FINDINGS Is there evidence of RSP uptake by teachers in class? (Evaluation Question 3) Teacher uptake and implementation of the RSP strategies and practices varied widely in the sample of schools considered by the evaluation. The evaluation team checked for evidence of teaching strategies to decipher text, opportunities to read, strategies to encourage text comprehension, examples of writing and evidence of vocabulary development. However, there are no benchmarks to determine if “highest uptake” was “adequate” to promote improved early grade reading. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xix The evaluation observed differences when analyzing the uptake in HL and EFAL, and when comparing the uptake by teachers in different grades. There were slight differences in the uptake of teachers in different intervention groups - Teachers in schools where the SMT intervention ran, had slightly lower uptake than those in the other intervention groups. . There was no observable or reported differences between teachers in different grades, different intervention groups or different districts in how useful they found the RSP LTSM. Availability and display of RSP materials in the classroom were observed in 51 percent of coaching classes, 55 percent of SMT and 62 percent of LTSM classrooms. Overall, 78 percent of teachers found some of the LTSM useful, and 21 percent of teachers found all the LTSM useful. Eighty-one percent of teachers found the scripted lesson plans the most useful HL resource. For HL and EFAL, the themed vocabulary flashcards were used almost every day by the SMT and LTSM teachers. The least used resource was the curriculum tracker: 17 percent of teachers used it every day, and 27 percent using it once a week. Classroom management across interventions was generally of a high standard. Analysis of 2020 DBE learner workbooks indicates that learners from LTSM schools on average completed more pages with an exercise and writing exercises (an average of 14.25 pages and 11.54 pages respectively) compared to the other intervention groups. There was no statistically significant difference between the groups. This was lower than what was observed in 2019 (on average a minimum of 25 pages and 17 pages completed all intervention groups). Nonetheless, all groups had a similar average of 25 days spent on any exercise and writing exercises since the beginning of 2020. The DBE workbooks analysis for Term 4 showed that there was more writing in Setswana HL than EFAL. There was less writing in both languages in the same Term 4 period for 2020 compared to 2019. On average, all three groups had completed between 15 and 19 pages with writing at least one full sentence in DBE workbooks. Learners used their exercise books, with many showing more than 20 pages with at least one daily writing exercise. A third of 214 classes (29%) received the classroom library intervention, and were observed in 91 percent of 63 these classrooms. English books were available in 64 percent of classes, and Setswana books in 89 percent of classrooms. The management tracking chart/reading cards, although visible, were unused. Most district officials (CAs/SAs) indicated that there was teacher uptake of the RSP strategies and there had been an improvement in implementation since 2019. One official was unsure. Anecdotal evidence was provided by the officials of changed practices. SMT members reported changing their behavior, and noticing teachers change their behaviors in classrooms. In 2020, 59 of 64 (92%) SMT members from SMT schools reported that their teachers had shifted their practice as a result of coach visits. Sixty-one (95%) of SMT members would recommend using coaches in schools. Is there evidence of changed practices based on SMT support? (Evaluation Question 4) The SMT intervention was not implemented with the expected fidelity, with only 21 (16%) SMT members receiving the full intervention. Evidence from the classroom observations showed little to no difference between the uptake of the RSP by teachers in the schools that Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xx had the SMT intervention to those in the coaching or LTSM interventions. When it came to strategies to facilitate meaning in text, SMT teachers performed not only below the other groups but worse than the 2019 level. Teachers reported feeling supported by their SMT. Teachers in the SMT group reported receiving more support (95%) compared to the other groups, but only marginally (95% of Is the RSP cost efficient? (Evaluation Question 9) An analysis of the cost data shows that budget management and cost controls were not optimal. An analysis of the budgeted time shows duplication in roles between different consortium partners, and a high ratio of support personnel to personnel responsible for direct service delivery. These factors, together with the poor fidelity of implementation, allow the evaluation team to conclude that the RSP was not as cost-efficient as it could have been. What is the actual versus anticipated unit cost? (Evaluation Question 10) The total actual cost of the RSP was R 79,077,638. A budget of $5,505,295 was approved. Including the amount of $2,144,670 that was spent on the original RSP prior to its redesign, the total USAID expenditure was $7,649,965. The actual unit costs were R313,800 per school, R51,685 per teacher, and R1,396 per learner. For the two-year implementation period, RSP coach salaries came to R18,222 per teacher, the package of materials (excluding development costs) was R8,190. Accommodation, venue and catering costs for four sessions of two-day residential training plus four session of single-day training over two years was budgeted at R6,084 per teacher. These costs are unlikely to accurately reflect the cost of implementing an Early Grade Reading study on a larger scale. How does the RSP unit cost compare to EGRS I unit costs and how are they different? (Evaluation Question 11) The EGRS I and the RSP worked in the same districts, and therefore it was expected that some unit costs may be comparable, for example, the costs of accommodation and coach salaries. However, there are significant differences between the two programs, which complicates a direct comparison of RSP and EGRS I unit costs. The programs differed in terms of their 1) implementation period, 2) scope, 3) intervention components, 4) dosage, and 5) implementation fidelity. A rough comparison of the total unit costs shows that RSP resulted in a lower per teacher cost than EGRS I. However, when considering the total cost and output, RSP per teacher cost is higher than the cost benchmarks researched by Venter and Sharif (2020) for the DBE. They found that the cost for early grade reading (EGR) training and coaching of a teacher for a period of two years, come to R 32,578 (p.40). The higher total cost per teacher of RSP in comparison to the Venter and Sharif (2020) cost benchmarks is due to higher overhead expenses. LESSONS FOR THE FUTURE What are the unique challenges of delivery at scale? (Evaluation Question 8) The RSP was an attempt to replicate the EGRS I on a larger scale in the same context. The design of the RSP is plausible even though there were some challenges such as finding and managing a large team of experienced coaches and assembling a consortium of organizations with relevant experience. It was entirely possible for the RSP to deliver the Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |xxi training to a larger number of teachers than in the EGRS I, and the RSP showed that it is possible to target HL and EFAL in all three grades at the same time, thus scaling the content focus too. A higher number of individual coaching visits could have materialized, if there were better alignment between the initial FPD offering, and the expectations from the DBE. In this instance, the coaching was not effectively implemented, and there was not a real opportunity to test the implications of having a coach to teacher ratio of 1:58 (minimum 1:45; maximum 1:76), instead of the norm of 1:30 (Hofmeyer, 2019). What time commitment is required to implement at scale? (Evaluation Question 12) RSP budgeted for 17 coaches and head coaches as well as 58 other full-time staff (12,438 person-days spread over 30 months). This excluded the LTSM freelancers who were paid on outputs and DBE and Provincial Education Department staff. For each direct implementer (e.g., coach, facilitator) there were approximately two support personnel providing support in project management, logistics, administration, information technology, finance, human resources and monitoring and evaluation. The time commitment from government officials includes time from the provincial department of education and the DBE plus the roughly the equivalent of one month per Curriculum Advisor. Had the program been implemented as planned, each teacher would have spent about 48 hours per annum in training, and those involved in the coaching intervention, would have spent an additional 18 hours in coaching activities. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |1 SECTION 1: INTRODUCTION SYNOPSIS OF TASK, PURPOSE, AND AUDIENCE This report, produced by Khulisa Management Services (Pty) Ltd. (Khulisa), is submitted under the Design and Implementation Evaluation of the Reading Support Project (Award Number: AID-674-A-16-00007) to the United States Agency for International Development (USAID) under Indefinite Delivery Indefinite Quantity (IDIQ) Contract Number: 72067418D00001, Order Number: 72067419F00015 (Scope of Work in Annex 1). For the period 2019 - 2020, USAID commissioned Khulisa to conduct a Design and Implementation Evaluation of the Reading Support Project (RSP). The purpose of the Design Evaluation4 was to ensure that the design of the RSP, as it evolved, was well understood prior to conducting an Implementation Evaluation. The purpose of the Implementation Evaluation (first the Formative Evaluation, followed by this Summative Evaluation) was to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. This is important for a number of reasons cited below. Firstly, it is critical towards accurately attributing any impact to the program. If successful, the program can be repeated in another context as the implementation elements will be known. If unsuccessful, the evaluation will provide some insight into whether this was due to design failure or to implementation failure, or a combination thereof. Second, the Formative Evaluation assisted the implementers of the RSP to understand any gaps in implementation or identify pockets of success. The data was used in a formative manner to give the program the best possible chance to lead to impact. The Summative Evaluation was carried out by the same independent evaluation team who conducted the Design and Formative Evaluations (profiles of the team are included in Annex 2 and conflict of interest statements are included in Annex 3). The team consisted of evaluation specialists from Khulisa Management Services and Benita Williams Evaluation (BWE). The Scope of Work (SoW) for this evaluation, in Annex 1, highlights the importance of the evaluation for learning and accountability, with USAID, the South African Department of Basic Education (DBE), and the RSP Implementers (i.e., the FPD Consortium) being the main target audiences. The North West Provincial Education Department (PED) is another key audience. As expected, the FPD Consortium used the 2019 Formative Implementation Evaluation results to inform programming in 2020. The FPD Consortium developed improvement plans based on the Formative Evaluation Findings. It is expected that the DBE and other implementation organizations and service providers will use the results and lessons learned from the Summative Implementation Evaluation to inform their approaches 4 The RSP Design Evaluation Report is available on the USAID Development Experience Clearinghouse (DEC) https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |2 to improving the quality of reading, and that USAID and other donors will use the evaluation results to guide future investments in early grade literacy interventions and the impact evaluation in South Africa. On March 27, 2020 South Africa went into a national lockdown due to COVID-19 (formally known as ‘2019 novel coronavirus' or '2019-nCoV). Due to COVID-19, South Africa imposed a strict lockdown in March 2020. Schools were closed and all gatherings were forbidden. From May 2020, a gradual and phased easing of the lockdown restrictions began. From 1 June, the national restrictions were lowered to level 3 and to level 2 on 17 August 2020 and learners returned to school in a phased manner thereafter. While some learners were given assignments during Term 2 (typically April to June) most had no activities during this period. Schools returned using a platooning model which meant that learners did not attend every day. The school year was extended until mid-December 2020. As a result of the lockdown, RSP interventions could not proceed as per the design. Therefore, an amendment (included in Annex 1) was provided to include a question on the effects of COVID-19 on the implementation of the RSP and how the program responded and adapted. These findings will inform and guide DBE, USAID and other donors and implementing organizations on how the program, teachers and schools responded and provide insights into how early grade reading teaching and learning was affected. The answers to the evaluation questions are grouped according to themes below: About the Intervention Is the RSP theory of change likely to lead to the anticipated results? (Question 1) a) What is the impact theory of change? b) What is the process theory of change? c) What are the critical assumptions in the theory of change? d) How does the design differ from the EGRS? e) Is the design uniform or customized for different schools (functionality, level of engagement)? f) Does the design make provision for exit and sustainability? To what extent did the FPD consortium implement the intended intervention design with fidelity? (Question 2) a) Did the Training / Coaching / SMT Training / District Training / Volunteer support happen as planned (i.e., the anticipated number of contact sessions, the specified duration, targeted number of participants, quality of input, within the planned time frame). Were there any major divergence from the assumptions in the theory of change? (Question 5) To what extent did COVID-19 affect implementation and how did the program respond and adapt? (Question 15) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |3 How the intervention could be improved How could the management and implementation structures be improved? (Question 6) How could the M&E systems be improved? (Question 7) Which efficiencies could have been realized? (Question 13) Which economies of scale could be realized in Year 2 and beyond? (Question 14) Intervention Findings Is there evidence of RSP uptake by teachers in class? (Question 3) a) Is there evidence of RSP practices being implemented in class by teachers? b) Are the RSP LTSM and classroom libraries i. Available for teachers and learners to use in the classrooms? ii. Is there evidence that the RSP LTSM and classroom libraries are being used as intended? iii. Is there a working system for students to check out books from the classroom libraries? iv. Is the quality of the materials durable enough for the classroom environment? Is there evidence of changed practices based on the SMT support? (Question 4) Is the RSP cost efficient? (Question 9) What is the actual versus anticipated unit cost? (Question 10) How does the RSP unit costs compared to the EGRS unit costs and how are they different? (Question 11) Lessons for the future What are the unique challenges of delivery at scale with regards to (Question 8) a) RSP Teacher Training (material development, logistics, (Quality control) b) RSP Coaching (recruitment, selection, training, and supervision of coaches, scheduling and conducting coaching sessions) c) RSP Materials (development, printing, distribution, use) d) Involving the Provincial / District officials Costs What time commitment is required from other stakeholders to implement at scale? (Question 12) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |4 SECTION 2: BACKGROUND EGRS I AND RSP To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the EGRS I in two districts in the North West province of South Africa (districts of Ngaka Modiri Molema and Dr. Kenneth Kaunda). The EGRS I evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The EGRS I targeted one Grade per year (Grade 1 in 2015, Grade 2 in 2016 and Grade 3 in 2017). The interventions ended in 2017 and EGRS impact results (Taylor et al, 2017) showed substantial impacts on learner results through training, coaching, and provision of LTSM. Figure 1: RSP Treatment and comparison groups Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |5 In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS I, specifically to scale the coaching intervention, which showed the most impact. In response to this appeal, USAID requested that the FPD Consortium modify their existing early grade reading project to include the selected intervention design elements from EGRS I. This was documented in a RSP Design Evaluation that preceded the RSP Formative and Summative Evaluation reports. The revised RSP comprised groups of schools receiving interventions to improve the quality of teaching and learning in the classroom and, in turn, improve early grade reading. However, the RSP scaled the EGRS I in terms of the number of schools involved, the number of Grades (Grades 1 to 3 simultaneously) and the number of subjects (Setswana Home Language/HL and EFAL. The schools were divided into three groups, each receiving a different combination of interventions, which include5 : Group 1: schools receiving LTSM and Teacher Training (referred to as the LTSM Intervention) Group 2: schools receiving Coaching, LTSM and Teacher Training (referred to as the Coaching Intervention) Group 3 (a subset of Group 2): schools receiving Coaching, LTSM, Teacher Training and a School Management Team training program referred to as the SMT Intervention RSP specifically aimed to improve subject matter knowledge, promote more effective pedagogic practices, improve in-class time management, increase effective use of LTSM, and foster a school environment that supports teachers’ ability to implement the full curriculum and facilitate successful teaching and learning. In addition, a number of RSP classrooms received classroom libraries through a joint program of the North West PED and the University of the Witwatersrand School of Education (WSoE), delivered via the University of the Witwatersrand (WITS). Another key change was that the original RSP aimed to improve knowledge of subject advisors but the competency of the subject advisors in Dr. Kenneth Kaunda and Ngaka Modiri Molema was high and the consortium could not come up with relevant training content. USAID and the DBE then agreed that the Subject Advisors would function as partners not trainees. Subsequently, they were involved in oversight and in the Reference group. In order to implement the RSP, a Project Management Team (PMT) was established, which comprised of members from USAID, DBE and the FPD consortium that met on a quarterly basis to discuss the RSP implementation. In 2020, Khulisa was included as a member of the PMT in order to provide updates on the evaluation process and to better coordinate evaluation efforts with project implementation activities. 5 As per final RSP list of project schools provided to Khulisa and last updated in October 2019. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |6 THE EVALUATION To support the Government of South Africa (GoSA), USAID is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve the reading skills of primary grade learners. Khulisa was awarded the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) IDIQ contract. The contract aims to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from the PERFORM interventions. Under the PERFORMANCE contract, USAID, along with its partner, the DBE, commissioned a Design Evaluation, an Implementation Evaluation and finally, a Summative Evaluation of the RSP to evaluate the implementation of RSP as it scales-up components of the EGRS I. The Implementation Evaluation provided a careful account of how the RSP was adapted to include EGRS I, and made judgements on whether the adapted design was likely to lead to impact, whether the dosage and coverage of services was sufficient to support pathways to impact, and whether there was fidelity to the intended design in the actual implementation of the program. The results of the 2019 evaluation were formative (i.e., used to improve the program in 2020) and the 2020 are summative (i.e., compared to the 2019 results). The objectives for the three (Design, Formative and Summative) evaluations included: 1. Identify any risks to the successful implementation of the RSP. 2. Provide early findings to strengthen the implementation of the RSP. 3. Ascertain whether the allocated time and resources were sufficient to successfully implement the RSP. 4. Determine whether the FPD Consortium is implementing its activities as planned. 5. Determine whether impact-level change can realistically be expected with the RSP theory of change, its design and within the timeframe of the RSP implementation. Summative Implementation Evaluation Questions: 1. To what extent did the FPD Consortium implement the intended intervention design with fidelity? 2. Is there evidence of RSP uptake by teachers in class? 3. Is there evidence of changed practices based on the SMT support? 4. Was there any major divergence from the assumptions in the Theory of Change? 5. How could the management and implementation structures be improved? 6. How could the M&E systems be improved? 7. What are the unique challenges of delivery at scale? 8. Is the RSP cost efficient? 9. What is the actual vs. anticipated unit cost? 10. How does the RSP unit costs compared to the EGRS unit costs and how are they different? 11. What time commitment is required from other stakeholders to implement at scale? 12. Which efficiencies could have been realized? 13. Which economies of scale could be realized in Year 2 and beyond? 14. To what extent did COVID-19 affect the implementation of the RSP and how did the program respond and adapt? EVALUATION QUESTIONS Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |7 6. Explore whether and how the RSP theory of change could be strengthened to produce the intended outcomes. 7. Validate the data collected through the RSP Monitoring and Evaluation (M&E) system. 8. Determine whether teacher training and coaching is leading to any early changes in teaching methods (for example, Group Guided Reading, phonics, use of chapters in lesson plans to document implementation). 9. Ascertain what dosage of support teachers are receiving from coaches (i.e., Number of visits, time spent per visit). 10. Determine whether coaches are supporting teachers and reporting on their coaching visits. 11. Determine the contribution of the School Management Team intervention in generating change in school management (amended, refer to Annex 1). 12. Determine whether the School Management Team (SMT) support can be linked to changed support practices in schools. 13. In 2020, the COVID-19 pandemic resulted in the closure of schools and a change in learning and teaching methods. As a result, an additional focus was included on COVID-19 to determine how COVID-19 affected the implementation of the RSP and how did the program respond and adapt (Amended SoW in Annex 1). DESIGN EVALUATION According to the South African National Evaluation Policy Framework (NEPF, 2011), the purpose of a Design Evaluation, during implementation, is to “…assess the quality and implementability of the intervention design in practice.” A Design Evaluation is important towards understanding both how an intervention works and whether it is likely to achieve its outcomes and eventual impact. The original RSP Theory of Change (ToC) was revised to incorporate the intervention design elements from EGRS I. In the context of the RSP, the purpose of the Design Evaluation was to determine the extent to which the revisions of RSP incorporated these elements and was therefore possibly comparable, determine the differences between the design of the RSP and the EGRS I, and whether the RSP design could support the impact goals of the EGRS I. The final Design Evaluation report was submitted on 22 November 2019. The Design Evaluation reviewed relevant project documents and literature (termed a mini literature review), summarized findings from interviews with key stakeholders and from a design workshop led by an expert panel and hosted at the DBE, surfaced the RSP ToC and compared it to the EGRS ToC. The Design Evaluation findings specified that the RSP and EGRS have similar understandings of reading acquisition and how teachers’ capacity to teach effectively can be enhanced. In fact, they draw on the same reading acquisition theory. Yet there are differences between these programs both in the theory and the action (i.e., activities), including length and frequency of training, and the type and weighting of support provided. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |8 While the RSP ToC is based on evidence about reading acquisition and effective teacher development strategies, the quantum of desired change is not specified, neither is teachers’ experience, urban or rural schools differentiated, nor the inclusion of previously participating EGRS teachers considered. The success of the RSP ToC depends on the uptake of the lesson plans, materials, and classroom libraries. The program assumes good quality training, materials are delivered, and that teachers have an incentive to implement. The quality, nature, and dosage of coaching may influence the degree to which the RSP is able to replicate or exceed the EGRS. Ultimately, the success of the RSP ToC depended on the uptake of RSP lesson plans, LTSM, and classroom libraries in the classroom. The program assumed that good quality training and lesson materials are delivered6 , and that teachers would have an incentive to try these out in class. For the schools that received the coaching intervention, the quality of the coaches, the substance of the individual coaching sessions, frequency and dosage of coaching may influence the degree to which RSP is able to replicate or exceed the success of the EGRS I. FORMATIVE AND SUMMATIVE IMPLEMENTATION EVALUATION An Implementation Evaluation, sometimes called a “process” or “performance” evaluation, is defined in the South African National Evaluation Policy Framework (NEPF, 2011) as follows: “[An Implementation Evaluation] aims to evaluate whether an intervention’s operational mechanisms support achievement or not and understand why. Looks at activities, outputs, and outcomes, use of resources and the causal links. It builds on existing monitoring systems, and is applied during program operation to improve the efficiency and efficacy of operational processes. It also assesses the quality of the indicators.” USAID’s requirements for a Performance Evaluation are that it could be timed in such a way that the results are made available prior to key decision-making points, such as developing new strategies, designing new projects, and making decisions about new procurements. Khulisa conducted the Formative Evaluation in 2019 (the Executive Summary is included in Annex 4). The key findings included: The dosage, adherence, quality, participant responsiveness, and program differentiation (i.e., program fidelity) varied from being on target to under-performance. There was no major divergence from the ToC assumptions. However, some assumptions were not being met due to issues with implementation. For example, 6 A review of LTSM was not commissioned as part of the Evaluation (Design, Implementation or Summative). Therefore, although a pedagogy expert conducted a cursory review of the materials, the evaluation team cannot comment on whether this assumption has been met. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |9 fewer teachers attended training than expected, some teachers were not receiving all LTSM, and the dosage of coaching support was not sufficiently high to support behavior change. Teachers in general were receptive and responsive to the materials. However, they faced implementation challenges. Effective coaches were contributing to improved delivery. The Formative Evaluation was followed by this Summative Evaluation. Data was collected in 2020 and the report drafted in 2021. The findings of the Summative Evaluation are presented in this report. Under a separate task order, Khulisa is in the process (2020-2021) of collecting data for the impact evaluation. Figure 2: RSP Evaluation process Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |10 SECTION 3: METHODOLOGY The approved methodology comprises a mixed methods approach involving the collection, collation, triangulation, and valuing of both qualitative and quantitative primary evaluation data. See Annex 5 for the USAID-approved Methodology and Study Plan, Annex 5.1 for the Summative Evaluation data sources and respondents, Annex 5.2 for the 2020 evaluation tools7 and Annex 5.3 for the evaluation criteria for the coaching case studies. As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling-up EGRS I. In order to meet this aim, key implementation data was collected from 60 sampled schools8 in 2019 and the same schools in 2020 (although only 599 schools during 2020): Group 1: 20 schools LTSM and Teacher Training (referred to as the LTSM Intervention) Group 2: 2010 schools receiving Coaching, LTSM and Teacher Training (referred to as the Coaching Intervention) Group 3 (a subset of Group 2): 20 schools receiving Coaching, LTSM, Teacher Training and a SMT training program referred to as the SMT Intervention In addition, three coach case studies (see Annex 8) provide a deeper dive into the practices, profiles, and performance of three RSP coaches. The purpose of the three coach case studies is to illustrate an excellent, average and poor performing coach. In addition to the school level data, other data collection methods included, review of project documentation, interviews with district officials, the DBE, FPD Consortium members and other key stakeholders. In 2020, following a request from USAID, the evaluation questions and objectives were revised to include questions related to COVID-19 and remove questions related to VSO. As a result, the methodology was adapted to include data sources that would be helpful to understand how the program adapted its implementation due to the COVID-19 pandemic. This included adding questions to the existing tools, analyzing WhatsApp chats that coaches 7 The 2019 Formative Evaluation tools are included in the Formative Evaluation Report available on the USAID DEC. 8 For school-level sampling, the actual sampling strategy was defined in consultation with the DBE and USAID, to draw a sample of 60 schools. In total, 16 schools were selected from Dr. Kenneth Kaunda District and 44 from Ngaka Modiri Molema district to reflect the distribution of schools per district. The full detailed explanation on the sampling can be found in Annex 5 Methodology and Study Plan. 9 One school chose to leave the RSP as it was already receiving the DBE Primary School Reading Improvement Program (PSRIP) intervention. 10 One school left the RSP at the end of 2019, therefore data was collected from 20 schools in 2019 but only 19 in 2020. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |11 had with teachers and HODs and conducting school functionality assessments in each of the sampled schools. Multiple sources of data were collated and compared. Khulisa used thematic analysis to analyze group discussion data and interview data, and summary statistics to analyze the quantitative data collected through the evaluation. The data was valued by the evaluation team in collaboration with experts11 in 2019 (investigator triangulation). The data was collected from a multitude of sources including USAID interviews; DBE and PED interviews; FPD Consortium interviews; coaches and head coach interviews; Teacher surveys and classroom observations; school Principals and SMT member interviews; and data checks from FPD M&E data, school logbooks, asset registers, and workbook analysis. Table 1 in Annex 5.1 summarizes all the data sources, respondents and sample for the implementation evaluation. METHODOLOGICAL LIMITATIONS AND CHALLENGES There are several limitations in the methodology, which stem from a few factors: Khulisa proposed including an LTSM materials review in the evaluation but this was removed from the evaluation scope of work (SoW) as the DBE and USAID indicated it would be addressed as part of an ongoing separate literacy materials review. A LTSM EFAL material review was conducted in 2020 by Rebeca Martinez, a USAID Early Grade Reading expert. The DBE is planning a review of the Setswana materials developed by RSP and other programs. A basic underlying assumption in the RSP ToC is that the LTSM are of good quality and are sufficiently well organized for teachers to implement. The evaluation team is unable to test this assumption given the removal of the expert materials review from the scope of the evaluation. So that the team was able to understand the content of the materials, the evaluation team asked the pedagogy expert to conduct a cursory review of the EFAL materials and this expert, in collaboration with a Setswana junior evaluator from BWE, conducted a basic review of the Setswana materials. The Summative Evaluation was based on a randomly selected sample of 5912 schools. Only the FPD Consortium had any data at the population level (i.e., monitoring data). For this Summative Evaluation, the evaluation team returned to the same schools. Schools were selected for the Formative Evaluation using random selection protocols. This enhances the generalizability of the findings. However, given that the results are a sample, not population results, the evaluation team used FPD’s monitoring data, to conduct calculations of dosage and coverage. This data was assessed during the fieldwork data quality assessment (DQA) showing that the monitoring data was reasonably accurate. 11 A pedagogy expert, a coaching expert and an international reading expert reviewed the Theory of Change assumptions (including conducting a cursory review of LTSM) in 2019 during the Design Evaluation. 12 One school that was included as part of the original sample for the Formative Evaluation left the RSP as it already was part of the PSRIP. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |12 Throughout the evaluation the final list of schools receiving interventions was not well defined. The DBE and FPD Consortium presented the evaluation team with slightly different lists of schools. The evaluators used the target values of 263 as the base number of schools in the calculations presented in the 2019 Formative Evaluation report and 255 for the 2020 data analysis for the Summative Evaluation. The FPD documents were inconsistent but it appears that actual delivery happened in 251 schools in 2020. The original number of RSP schools changed over time as documented by the FPD list of 15 removed schools. Some of the reasons given for the attrition of schools included having a language of learning and teaching (LOLT) other than Setswana, being multi-grade schools or being part of other interventions. The final RSP Monitoring, Evaluation and Learning (MEL) Plan and indicators was only approved by USAID in March 2020. Working with different versions of the RSP MEL Plan meant that the evaluation team had to make many assumptions on indicators and actual data that would be collected over the duration of the program. The findings in this evaluation are thus indicative of how the project was implemented rather than comprehensive. Nonetheless, triangulation of data across a number of sources aids in providing insights to inform lessons learned and recommendations. Data was collected during this evaluation from teachers and principals on the number of days lost due to COVID-19. However, there are too many discrepancies across the data sets to determine actual days lost at teacher, school or district level. The COVID-19 pandemic closures affected school attendance, teaching and learning, and meant that RSP could not adhere to the planned RSP design after the first term in 2020. Consequently, the RSP implementation had to adapt to these school closures. Therefore, the fidelity of the model as described in the Design Evaluation cannot be evaluated as planned for the full period of 2020. A number of factors affected the summative evaluation data collection conducted in schools, which included: Teachers followed the revised DBE provided Amended Annual Teaching Plans (ATPs) which trimmed the curriculum. This influenced the curriculum coverage and teachers were at different points of the curriculum meaning that the evaluation classroom observations were not always comparable. Also, teachers were at different points with regards to the lesson plans, which reduced the comparability of the data. As a result, the evaluators observed if the teacher was using a RSP lesson plan or not. Schools were operating on a rotational basis, which meant that learners took their DBE workbooks and/or learner exercise books home. The evaluation design required collecting DBE workbooks and learner exercise books from the same learner. In some instances, DBE workbooks or learner workbooks for the same learner were not available for the assessments. To remedy this issue, the evaluation team collected available workbooks or learner exercise books from other learners, when there was one missing. The data collection (with schools and including classroom observations) were conducted during the end of school year assessment period, meaning that the evaluation team was not able to observe a ‘typical’ early grade reading lesson in some schools. This contradicted the DBE directive suspending formal assessments as part of the ATPs, but followed the other DBE directive to assess learning gaps. In a few cases, the teachers accommodated the evaluation team, by delaying the assessment and demonstrating a lesson following a RSP lesson plan. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION |13 Overall, the evaluation was hampered by a lack of valuing criteria or benchmarks, for example, how many sentences should a learner be writing in their exercise book in Grade 1 versus Grade 3? The number of sentences are documented in the exercise book analysis but the evaluators were unable to specify if those sentences matched a benchmark. This limitation was a constraint throughout the observation data. BIAS AND CONFLICT OF INTEREST The evaluation team was able to work freely and without interference and there were no known or unresolvable differences of opinion or conflicts of interest either within the evaluation team or between the evaluation team, the DBE and USAID. For the qualitative data collected through the evaluation, the evaluation team attempted to ensure that a range of stakeholders with different perspectives were interviewed, that the team provided a space for alternate views, and that the data was triangulated in an open and transparent manner (both data triangulation and investigator triangulation). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 14 SECTION 4: ABOUT THE INTERVENTION DESCRIPTION OF THE INTERVENTION13 The Reading Support Project (RSP) was an educational program designed to be implemented in 263 schools14 in two educational districts in the North West Province of South Africa (Dr. Kenneth Kaunda and Ngaka Modiri Molema) from January 1, 2019 to September 30, 2020. USAID provided a no-cost extension up to December 31, 2020. RSP focused on improving the pedagogical practices of Grade 1 to 3 teachers in how they teach reading in Setswana Home Language (HL) and English First Additional Language (EFAL) with the ultimate outcome to improve the reading skills of Foundation Phase learners. The RSP was revised as an attempt to scale the DBE’s Early Grade Reading Study (EGRS I 15), which was a randomized control trial (RCT) with 230 Quintile 1 to 3 schools in the same two districts from 2015 - 2017. The findings from the EGRS sustainability study indicated that the results were sustainable after project exit. Critical to the continued use of materials was ongoing support to teachers. Therefore, building capacity across the system is important to embed support beyond the project. The RSP specifically aimed to improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning in early grade literacy. RSP had six focus areas. Since it was set up as a RCT, not all schools benefitted from all six focus areas: 1. The professional development of Curriculum Advisors in the two participating districts 2. The development of the leadership capacity of principals/deputies and HODs to promote a culture of reading in their schools (in 65 schools) 3. The provision of Setswana HL and EFAL Learning and Teaching Support Material (LTSM) packages to 254 schools 4. Quarterly training for teachers (referred to as ‘just in time’ training) on the implementations of the LTSM to meet the Foundation Phase CAPS and lesson plans 5. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers 6. Through the DBE, classroom libraries are provided to 100 schools 13 The Design Evaluation provides more information on the EGRS I and RSP interventions. 14 The analysis presented in Annex 6 indicates that in October 2019 FPD was implementing the RSP in 255 schools and in 2020 implementing in 251 schools. 15 The Early Grade Reading Study ran between 2015 and 2017 in North West Province. Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 15 The Foundation for Professional Development (FPD) implemented the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), South African Institute for Distance Education (SAIDE) and one international partner, Voluntary Services Overseas (VSO). VSO and SAIDE both exited the consortium in 2019. The ToC (refer to Figure 3 below) assumptions were documented in the Design Evaluation. USAID approved the revised RSP MEL Plan in May 2020 which did not change the assumptions. Figure 3: RSP Theory of Change This US $7.65 million project was adapted and managed in conjunction with the South African DBE. After the RSP was originally awarded, its design was adjusted. There were numerous budgets during the lifespan of the RSP and line-item adjustments and allocations varied. Figure 4: Four RSP Budgets Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 16 The 2016 Original RSP Design commenced in November 2016 with an approved budget (referred to as Budget 0) and continued as planned until July 31, 2018. The RSP was then redesigned to resemble EGRS I more closely. The 2018 RSP Revised Design and Budget 1 was approved by USAID on September 10, 2018. However, in May 2019, the design parameters were refined and documented in the May 2019 RSP Revised Design Specification. Budget 2 for the May 2019 RSP Revised Design was only approved by USAID in March 2020. In the FPD close-out report, the budget was adjusted to R82,806,753. At R79,077,638, the RSP actual expenditure was R679,829 less than the approved Budget 2. 1. IS THE RSP THEORY OF CHANGE LIKELY TO LEAD TO THE ANTICIPATED RESULTS? (QUESTION 1) The Design Evaluation16 indicated that there were differences between the EGRS I and the RSP, including length and frequency of training, and the type and weighting of support provided. For example, the length of training for the coaching interventions differ, as does the coaching to teacher ratio, the amount and type of support to SMTs, and the Curriculum Advisor (CA) intervention was not part of the EGRS I. a) What is the impact theory of change? b) What is the process theory of change? c) What are the critical assumptions in the theory of change? d) How does the design differ from the EGRS? e) Is the design uniform or customized for different schools (functionality, level of engagement)? f) Does the design make provision for exit and sustainability? A number of critical elements were highlighted in the Design Evaluation, which, if implemented, could contribute to the sustainability of the results. These included firstly, that teacher training contributes to the professional development of teachers through alignment to continuing professional development (CPD) and achievement of points; secondly that the coaches provide afternoon workshops as part of DBE professional learning communities (PLCs); and thirdly that the project has systemic impact. System impact requires working at, and involving, multiple levels (e.g., schools, district, province, national) and by building relationships and strengthening linkages between role-players (e.g., teachers, Head of Department (HOD), Principal, coach, CA). 16 Available at https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 17 Ultimately, the Design Evaluation indicated that the success of the RSP design ToC, depended on the uptake of the lesson plans, LTSM, and classroom libraries17 in the classroom. The program assumes that good quality of training and lesson materials are delivered and that teachers will have an incentive to try these out in class. For the schools in the coaching intervention the quality of the coaches, the nature and dosage of coaching may influence the degree to which it is able to replicate or exceed the success of the EGRS I. 2. TO WHAT EXTENT DID THE FPD CONSORTIUM IMPLEMENT THE INTENDED INTERVENTION DESIGN WITH FIDELITY? (QUESTION 2) Implementation fidelity is usually assessed with criteria (Buckley et al., 2017) that includes: 2.1 Dosage (the frequency of program delivery) and coverage (number of target participants reached). 2.2 Adherence (whether program components were delivered as prescribed) and quality of delivery (how well the program material was implemented). 17 The Classroom Libraries are provided through a different Service Provider (University of the Witwatersrand School of Education) to the FPD Consortium. However, they are part of the overarching Theory of Change of the RSP program and are, as such, included in this Design Evaluation. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: − RSP monitoring data 2019 and 2020 − RSP Standard Operating Procedures. − RSP Design Document 2018 − Design Evaluation (reflecting against ToC). − KIIs (District and Provincial staff, FPD Consortium, teachers, coaches, and head coaches). − Classroom observations in 119 Grade 1 and 120 Grade 3 classrooms (Grade not clear in three observations). − Teacher training observation 2019 and 2020 − Coach training observation 2019 and 2020. − Coaching case studies with 14 coaches 2019 and 2020. − Monthly plans for the month of coach observations submitted by each coach 2019 and 2020. − School Data Quality Assessment in 58 schools. − Classroom library assessments in 58 schools. − Classroom materials assessment in 57 schools − Learner workbook assessments with 118 Grade 1 and 120 Grade 3 learner workbooks. − Teacher surveys with 105 Grade 1, 103 Grade 2, and 96 Grade 3 teachers. − FPD MEL plan May 2020 − FPD Improvement plan for 2020 − Data quality assessment (DQA) 2020 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 18 2.3 Participant responsiveness (how well the instruction was received or perceived). 2.4 Program differentiation (the degree to which the program was differentiated according to type of intervention actually delivered). Dane and Schneider (1998, cited in Century et al., 2010) recommend that these criteria will assist with understanding program integrity. We use the RSP monitoring data as a base, triangulate these data with other sources of information, and comment on the quality of the RSP monitoring data. The USAID Indicator Summary Table and RSP Custom Indicators, and related data referred to are provided in Annex 6. 2.1. DOSAGE AND COVERAGE Dosage is a measure of the amount of time spent on program components (for example, on teacher training, coaching, and others), or the amount of intervention received by program participants, while coverage is a measure of whether all the targeted participants are reached with an intervention (Carroll et al., 2007; Century et al., 2010). This section disaggregates dosage and coverage by RSP component and compares implementation between 2019 and 2020. PARTICIPATING SCHOOLS Number of participating schools FPD reached 251 schools in 2020. The 255 target was downscaled from the original 298 schools design specification as per the design evaluation report. Fewer schools were eligible to participate in the RSP than originally specified as they did not meet the selection criteria. Selection criteria included: • LOLT at the school is Setswana • The school is not a multi-grade school • The school does not participate in another program, such as the Primary School Reading Improvement Program (PSRIP) Annex 6.1 shows the number of schools reached indicator and the number of participating schools in 2019 and 2020. The number of schools reached was not affected by COVID-19. TEACHER TRAINING SESSIONS Number of teacher training sessions Training sessions for 2019 and Term 1 of 2020 were delivered at the intended dosage level and in the planned form of residential workshops. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 19 Training in Term 2 of 2020 did not take place due to the COVID-19 pandemic. In Terms 3 and 4 of 2020, the training approach was changed to shorter training sessions with fewer teachers in multiple clusters. FPD delivered the required number of training sessions prior to the COVID-19 disruptions. However, in Term 2 of 2020, none of the planned training was delivered due to COVID-19, and in Terms 3 and 4 2020 the training approach was changed and shorter training sessions were delivered. Detailed data provided in Annex 6.2. Number of teachers participating in training sessions by Grade, School and Term In 2019 and in 2020, the program met the target for the number of teachers reached (Target = 1,530). More teachers (1,029) attended all training sessions in 2020 than in 2019 (826). In 2019, only 44 percent of teachers participated in all four training sessions. In 2020, 67 percent of teachers participated in all three training sessions. For the full implementation period, only 47 percent of the teachers attended at least six out of seven training sessions. A master list of participating Foundation Phase teachers was not well-maintained during implementation. The teachers who never attended training have not been included in this calculation. See Annex 6.3. Fidelity of teacher attendance at training Across the terms, the number of teachers that participated in the program were mostly constant. The exception was Term 2 of 2019 where a number of non-project teachers attended the training. This was caused by the list of participating schools not being finalized and no pre-populated list of teachers. Over the full project implementation period, 1,925 teachers were reached. Between 2019 and 2020, 226 new teachers were added (12% out of 1,925), and 478 of the 2019 teachers (25% of 1,925) did not return to training in 2020. It is not clear whether the turnover was a result of teachers moving between grades and between schools, or if it is because of fluctuating engagement with the program. In 2019, 826 (44% of 1,882 teachers) of teachers attended all four training sessions, and in 2020, 1,059 (67% of 1,580) attended all three training sessions, demonstrating an increase in fidelity. Altogether, 547 of 1,925 (28%) of teachers attended all seven training sessions over the two years. 5% 11% 13% 13% 10% 20% 28% 1 2 3 4 5 6 7 Percentage of educators Number of teacher training sessions Figure 5: Percent of teachers who attended 1 to 7 training sessions in 2019 and 2020 N = 1,925 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 20 An increase in fidelity of teacher attendance is seen between 2019 and 2020, with some turnover between teachers during the full implementation period. A little over a quarter of teachers attended all training sessions held over the two years. See Annex 6.4 and 6.5. COACHING INTERVENTION Number of participating teachers in individual coaching sessions In 2020, 91 more teachers were reached than in 2019, even though teacher support was only implemented as planned in 2020 Terms 1 and 4 due to COVID￾19. The coverage of coaching support to teachers was inadequate in 2019: 668 teachers were supported which was lower than the 810 target. The percentage of teachers who received the full dosage of individual coaching (6.25 visits) for the five terms analyzed (Terms 1, 2, 3 of 2019 and Terms 1 and 4 of 2020) was less than 20 percent for both Grades 1 and 2. Although COVID-19 interrupted the ability to host coaching sessions in Terms 2 and 3 of 2020, more teachers were reached than in 2019. The full dosage of support aimed for was however only received by less than a fifth of teachers. See Annex 6.6. Number of individual coaching sessions The percentage of individual coaching sessions conducted in Term 1 and Term 4 of 2020 (44% of target) is higher than sessions conducted in 2019 (28%). An average of three coaching sessions per teacher were conducted which is substantially lower than the ten in the target. The number of coaching sessions was below the revised design specification target for 2019 and 2020 (not accounting for COVID-19 disruptions). ‘Virtual coaching’18 took place in Terms 2 and 3 of 2020. FPD did not track these coaching engagements in REDCap. 18 Virtual coaching refers to the move from in-person coaching establishing WhatsApp groups with the teachers. FPD developed a “Teacher Engagement Strategy, August 2020” to guide coaches on providing ‘virtual coaching’. 14% 12% 12% 15% 17% 12% 8% 8% 13% 15% 12% 17% 14% 14% 10% 6% 0 1 2 3 4 5 6 7 and more Percentage of teachers Number of individual coaching sessions Grade 1 (n=379) Grade 2 (n=355) Figure 6: Teachers received 0 to 7 individual coaching sessions (5 term target = 6.25) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 21 When the target is adapted based on cancellation of training due to COVID-19 and unavailability of data for one term in 2019, 88 percent of the adapted number of lesson observations (1,782 of 2,025) were reported by coaches in 2020. 19 See more detail on teacher participation in coaching sessions in Annex 6.5 and 6.6. The FPD monitoring data on individual coaching sessions (also referred to as lesson observations) in 2019 was incomplete and likely underreported. According to the FPD monitoring data, teachers received on average only one individual coaching visit in 2019, and fewer than 30 percent of the expected individual coaching sessions were delivered. In 2020 teachers received on average nearly two individual coaching sessions. The figures reported by FPD in the indicator reporting table submitted to USAID on a quarterly basis, also indicated a very low number of individual coaching sessions. See Annex 6.6. Data collected via the teacher survey confirms that coaching visits did take place. In 2019, 199 of 200 (99.5%) teachers in 1) coaching or 2) coaching and SMT schools reported receiving visits from their coach. In 2020, of 105 coaching intervention teachers, the majority (94.6%) reported receiving a visit from their coach, and 104 of coaching and SMT intervention reported receiving coaching visits (97.2%). However, the data does not document the extent of the coverage and dosage of the coaching per teacher. In 2020, due to COVID-19, schools were closed, which meant that coaches could not provide coaching support visits to teachers. As a result, FPD developed a teacher engagement strategy intended to formalize a process of coach-teacher engagement using WhatsApp. However, as reported previously, only two coaches used this platform effectively. Number of school based workshops (SBW) Very few SBW were facilitated by coaches. Only 10 percent of planned sessions were held in 2020. 75 percent of teachers attended at least one SBW in 2020. No teachers received the full dosage of 15 SBWs in 2019 or the adapted dosage of 7.5 SBWs in 2020. Five teachers, on average, attended a SBW in 2020. Altogether, 20 schools in 2019 and 21 schools in 2020 out of the evaluation sample of 59 schools reported at least one SBW being held. Over the five months of 2020 when SBWs could be held (Table 1), there was an increased frequency once schools resumed in October 2020. Thus, they had more SBW in Term 4 of 2020 than in Term 1 of 2020. 19 The target was not adapted by the RSP implementers to take into account the effects of COVID-19 on implementation. Figure 7: Number of teachers who attended 0 (none) to 6 SBW in 2020 (n=796) 196 334 216 40 5 5 0 1 2 3 4 5 Number of teachers Number of SBW attended Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 22 Table 1: Frequency of school based workshops (2020) SBW Frequency in 2020 Feb Mar April- Aug Sept Oct Nov Number of SBW held with Foundation Phase teachers attending 38 32 N/A due to COVID-19 lockdown 10 66 48 Percentage of schools where a SBWs was held 28% 24% 7% 48% 36% Detailed data is provided in Annex 6.9. SMT INTERVENTION SMT intervention training attendance Only 21 SMT members (16%) attended all five training sessions (also referred to SMT PLCs), submitted all four items for the Portfolio of Evidence (PoE) and received a tablet. Six were Principals, three Deputy Principals and 12 Department Heads. For five schools, both the Principal and Deputy Principal completed all activities and received tablets. Number of SMT training sessions held Five SMT training sessions (PLCs) and distribution of tablets were delivered at the intended dosage level. Details of data is provided in Annex 6.10 and Annex 6.11. Number of schools with participating SMT members The target for schools participating in the SMT training (PLC) sessions were nearly met with only one school not reached. Educators from two coaching-only schools attended one PLC. See detailed data in Annex 6.12. Number of SMT trainings sessions and portfolios of evidence (PoE) submitted 134 SMT participants attended the SMT training sessions, achieving the target. Only 34 (25%) SMT members attended all five training sessions, while 42 (31%) completed all four items for their PoE. Two thirds of the SMT members completed at least one assignment. See detailed data in Annex 6.13. Number of tablets distributed to SMT members and support visits More than 80% of SMT members received tablets with pre-loaded content. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 23 Only a third of the schools received an SMT support visit from a VSO volunteer in 2019. VSO withdrew in 2019. The SMT mentor appointed in 2020 to replace VSO could not conduct site visits as planned due to COVID-19. The evaluators could not verify that the tablets were used as intended. The SMT Intervention data shows that the number of sessions held and number of schools reached were in accordance with the intended targets. Almost a third of SMT members completed all required items for their PoE and two thirds managed to complete at least one assignment. LTSM LTSM distribution to teachers and schools In 2019, 70% (1,071 or 1,530) and 73% (1,117 of 1,530) in Terms 1 and 4 of 2020 of teachers received LTSM. LTSM was distributed via teacher training sessions. In Terms 2 and 3 of 2020, LTSM was collected by schools and not distributed at teacher training sessions. RSP reported that almost all schools collected LTSM (97% or 1,481 of 1530). More schools received the LTSM if they collected it than if it was distributed at training. COVID-19 did not negatively affect LTSM distribution. See detailed data in Annex 6.14. ENGAGEMENT OF DISTRICT OFFICIALS Participation and engagement of district officials in project activities FPD did not meet the revised target of 12 Curriculum / Subject Advisors (CA/SAs) attending the Reference Group and teacher training sessions. However, there were only eight Advisors in the Foundation Phase in the two districts. Attendance rates fluctuated between 50 and 63 percent. Further detail of the participant and engagement of district officials in the project activities is discussed later in the report. See detailed data in Annex 7.15 2.2. ADHERENCE AND QUALITY Adherence refers to whether "…a program service or intervention is being delivered as it was designed or written” and quality is defined as “…the manner in which a teacher, volunteer, or staff member delivers a program” (Mihalic, 2004, cited in Carroll et al., 2007, p.2). Adherence, together with quality, also refers to how well the program material was implemented. (Buckley et al., 2017). “If the content of an intervention is delivered badly, then this may affect the degree to which full implementation is realized. In studies evaluating fidelity the provision of extensive training, materials, and support to those delivering an intervention is an implicit acknowledgement that effort is required to optimize the quality of the delivery of the intervention being evaluated.” (Carroll et al., 2007, p.6) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 24 Quality of the teacher training provided by coaches Teacher training observations in November 2020 (Term 4) are summarized as follows: Coaches were prepared to conduct the training and used standardized agendas, presentations and registers. The HL and EFAL LTSM was ready for each teacher at the start of each of the session. The style of training was designed to be participatory, however the coaches tended to speak in a lecture mode and ask teachers questions. Some of the coaches engaged the teachers more in the discussions, or did an activity to engage teachers. The coaches tended to demonstrate activities to model how teachers would implement the tasks in the classroom. Teachers were engaged in the training but not excited by it, the lesson plans or LTSM. Coaches’ ability differed on how they encouraged teacher responsiveness during training: for example, one coach asked teachers to sing one of the HL songs when energy was low. Quality and use of LTSM by teachers Increased involvement and collaboration with the DBE and contribution of CA/SAs in LTSM development was reported to have improved the LTSM quality in 2020 compared to 2019. However, due to COVID-19 and its effect in schools, teachers still found integrating the LTSM and ATPs difficult to implement. Overall, 78 percent of teachers found some of the LTSM useful, while 21 percent found all LTSM useful. Teachers found the HL scripted lesson plans the most useful material. Refer to Annex 7 for usefulness and frequency of use in the classroom of LTSM materials. This is also discussed further in Section 6 on teacher uptake. Quality and adherence of coaches: Professional development 16 coaches attended training sessions, meeting the target. Coach training in Term 2 of 2020 could not take place face-to-face due to COVID-19. Instead, virtual sessions took place from July 13 to 16, 2020. No monitoring data related to this training was provided to the evaluators. Altogether, 22 people were employed as coaches in the two-year period. Three coaches resigned during Term 1 of 2019 and another three at the end of 2019. Each training session had 16 coaches attending (see Annex 6.8). Ten (63%) coaches attended all seven training sessions. Key findings from the coach training observations include: In 2020, a key shift in the coach training was its delivery method through an online platform in Terms 2 and 3 due to COVID-19. Only Term 1 and Term 4 training occurred face to face. The online delivery method shifted the training from being three full days (2 days training plus one day dry run), to training over four days (2 days per language): Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 25 Table 2: RSP online coach training Day 1 Day 2 Day 3 Day 4 Setswana HL 3 hours x 2 sessions Setswana HL 3 hours x 2 sessions plus an hour reflection EFAL 2.5 hours x 2 sessions EFAL 2.5 hours x 2 sessions plus an hour reflection Total Setswana HL = 13 hours Total EFAL = 11 hours The RSP online delivery method from the UNICEF coach training was the first experiment with online delivery. A report was produced and lessons provided to the FPD Consortium to inform the RSP planning for online training sessions. In general, facilitators of the coach training were of good quality, with clear content knowledge, good facilitation skills and reflective pedagogy throughout training was used. The evaluators observed both 2019 and 2020 Term 4 coach training. Table 3: RSP Term 4 coach training Day 1 Day 2 Day 3 Setswana HL 3 hours EFAL 3 hours Setswana HL 3 hours x 2 sessions Coach dry runs for EFAL and HL teacher training were run in parallel in 2020 (in 2019, the coaches refused to do the dry runs as they had grievances about the structure and planning of the training) The following are highlights from both observations: • The Term 4 in 2020 training format allowed group activities and sharing coach reflections on their RSP experience and the COVID-19 pandemic. • Contrary to what was observed in 2019 during the Formative Evaluation, coaches in 2020 appeared familiar with the training process and were ready to provide training to teachers. The preparation of coaches as trainers for teacher training was adequate, with participation by CA/SAs, dry runs carried out and feedback provided to coaches. There were some differences noted in terms of training delivery in EFAL and in HL: • In 2020, there were differences in HL and EFAL dry run structures. • The EFAL sessions were more structured with rubrics provided by OUP and filled in by OUP and CA/SAs, which allowed for in-depth feedback to coaches. • The HL sessions were less structured as there were no rubrics provided for the feedback process and this was done rather ad-hoc and left open for the CA/SAs and Molteno facilitators to provide feedback. • In the August 2020 PMT minutes, FPD confirmed that coaches said that the training facilitated by Molteno did not prepare them adequately in terms of implementing Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 26 COVID-19 protocols, compared to the OUP EFAL training. Molteno was requested by the PMT to re-run the training, but the evaluation team was unable to verify if it took place and assess the training quality to compare to the first delivery. Strengths and areas of improvement for coach training: Strengths: • Co-developing and facilitating sessions by OUP and Molteno. • Involving CA/SAs in delivering training sessions created synergy between the work of CA/SAs and the RSP coaches. • Including practical activities (including dry runs on the last day of face to face training) allowed for an engaging training format rather than lecture style. • Sharing and discussing materials (not only LTSM, but ATPs, risk management tools, baseline assessments, etc.) was effective. • Communicating training dates, materials and links (where appropriate) Areas that required improvement: • Content and delivery of EFAL and HL should be standardized. • While the facilitators provided some opportunities for coaches to reflect on their practice, the use of reflective pedagogy was not used inconsistently. • Sometimes it was difficult to get reciprocal engagement from coaches, even after facilitator tried hard (this was especially the case during Term 4 2020 training, where coaches appeared despondent according to the Khulisa evaluator). This was also noted in some of the online sessions, where the same group of coaches appeared to actively engage and respond during sessions. The UNICEF coaching course20 successfully complemented the RSP training. RSP coaches reported that they valued the training and how it added to their leadership skills. The training focused on coaching, classroom culture and literacy, and pedagogy. In contrast, the RSP training focused more on how to implement the RSP LTSM in the classroom, how to train teachers and how to apply coaching in RSP schools. Table 4 below provides a summary of both trainings. For the UNICEF training, there was overall good participation from the coaches especially in the interactive afternoon sessions where coaches reflected on the work that they were doing while offline. 20 In 2020, all RSP coaches attended the UNICEF-funded coaching course, implemented by Molteno and Class Act. The course had four sessions comprised of 12 modules spread across four sessions (one per quarter). The course focus was on coaching, classroom culture and literacy, knowledge and pedagogy. The aim was for this training to complement the RSP coach training. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 27 Adherence and quality of coaching provided to teachers Evidence from the evaluation shows that teachers were receiving coaching visits when schools were operating. However, the extent of the coaching (dosage and coverage) was limited. In 2020, due to COVID-19 school closures, FPD adapted coaching support provided to teachers, and limited coach-teacher interaction occurred via WhatsApp. In terms of adherence and quality, teachers reported that coaches communicated their planned visits with teachers and with sufficient lead time. Table 5: Coaches communicated with SMT and Coaching teachers in 2020 Coach communicated planned visit Coach communicated sufficiently in advance Large / very large extent Large / very large extent Coaching 93% (98 out of 105) 94% (99 out of 105) SMT 96% (100 out of 104) 92% (96 out of 104) In addition, 61 of 64 (95%) SMT members from the 59 sampled schools would recommend using coaches in schools as they noticed their teachers shifting their practices. Table 4: RSP and UNICEF coach training comparison UNICEF training RSP training Delivery Face to face session in 2019, Online in 2020 Face to face and online (during Terms 2 and 3 2020) Number of training sessions 4 Sessions in total (comprised of 12 modules) One session termly, prior to the start of every school term Structure Face to face (Session 1): 28 hours Online: 4 days with one 4 hour training daily Face to face: 3 days (Setswana HL 9 hours and EFAL 3 hours plus dry runs) Online: 2 days per language (Setswana HL 13 hours and EFAL 11 hours) Content Coaching, classroom culture and literacy and pedagogy How teachers can implement the RSP LTSM (including revised ATPs in Term 4 2020), how to train teachers and how to apply coaching in the RSP schools. However, in August 2020, the focus of the training was in Leadership and Management for the RSP. After reviewing the content, the evaluators felt there was some duplication with the content delivered through the UNICEF coach training. However, this training focused on how to apply leadership and management skills and their role as a coach in the RSP. Other Coaches provided with 10GB of data to attend each session Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 28 Key challenges21 to both adherence and quality of the coaching intervention: • Coaches’ time, feedback and activities varied in schools. • No evidence of coaches planning what they would cover and how they would facilitate SBWs sessions in 2019. • Monitoring data indicates coaches planned and facilitated SBWs in 2020. • Not all coaches conducted SBWs during coach shadowing in 2019 and 2020 (see Table 6). Observations in 2020 showed coaches’ plan and facilitate SBWs, as well as how they provided feedback to teachers varied (such as time spent, content, and approach). Table 6: Coaches that conducted SBWs during coach shadowing No SBW held 1 SBW out of 2 Schools visited (50% of expected) 1 SBW out of 1 School visited (100% of expected) 2 SBWs out of 2 schools visited (100% of expected) 4 coaches 5 coaches 2 coaches 3 Coaches Data note: While the RSP Design Specification assumes that a SBW would be held each day, this is based on the assumption that a different school is visited each day. Some coaches supported larger schools, and needed to visit these schools over 2 days to cover all their teachers – in which case, one SBW was held with all the teachers of that school. Adherence and quality as observed during coach shadowing In 2019, evaluators shadowed all 14 coaches for four days. In 2020, evaluators shadowed all 14 coaches, and the two head coaches for two days each. Khulisa also interviewed all 14 coaches and the two head coaches. Fourteen detailed coach case studies were provided in the 2019 Formative Evaluation Report. To ensure privacy, all coaches were provided with a number in 2019 that remained consistent in 2020, thus Coach 1 in 2019, is also Coach 1 in 2020. Coach 6 and one Head Coach left RSP. Coach 14 was promoted to a head coach, and Coach 15 is the second head coach. Coach 16 and Coach 17 are two new coaches hired in 2020. In 2020, each coach was compared to the 2019 case study, the rubric updated to illustrate changes and thereafter three case studies (see Annex 8) were developed documenting: An excellent coach: Coach 1 An average to good coach: Coach 5 A weak coach: Coach 7 Although these are actual coaches, they are illustrative of the three coach types. Coaches were evaluated against whether they performed the duties required in each observation, but the evaluation team were particularly looking for a shift from an authoritative, administrative focused coaching style to a reflective and relationship-driven coaching style. The evaluation team noted that the coaches’ job descriptions22 focused 21 Sources: interviews, observations and coach shadowing 22 RSP Coach and Head Coach Job Descriptions, 2018 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 29 heavily on the technical, compliance and administrative elements, but less on promoting coaching practice and developing teachers. The evaluators used criteria sourced from best practice coaching literature for the coach assessments. The RSP model of coaching, exists between ‘relationship-driven’, emphasizing relationship building to ease teacher resistance and a lesser focus on teacher￾accountability, and ‘teacher-centered’, relying on improving teacher’s ability to teach and their technical expertise23. This further highlights the need to clarify the role of the subject advisor (holding teachers accountable) and coaches (support and resource provision). The RSP coach training primarily focused on the content and methodology and how to use the RSP LTSM and how coaches should train teachers. UNICEF training: Module 1, Module 4, Module 7 and Module 10 were on coaching practice. These modules focused on the role of coaches and the models of coaching, and the style of coaching expected. The modules also emphasized the importance of relationship building and “defining the nature of the relationship as a support and critical friend, not as a monitor”. Coaches were advised to use WhatsApp groups as a supportive mechanism. Eight of the modules of the UNICEF training also included suggestions for coaches on School-Based/ Needs-based Workshop content. The ratings in Table 7 are based on the overview of the data collected on each coach and the debrief discussion with the evaluators who shadowed coaches. Coach 1, 3, 8 and 9 were the most effective coaches. Coaches 1 and 9 were the two coaches who demonstrated effective coaching using WhatsApp. Table 7: Rating of coaches as excellent, average or as needing more development Annex 5.3, highlights key evaluation elements, and what were considered by the evaluators to be the most important variables that affected the coaching practice. This matrix was developed based on RSP program expectations of the coach and the literature on coaching in education. These skills include content expertise; whereby coaches should be experts in literacy instruction and pedagogy, as well as interpersonal skills; such as relationship building skills, communication skills, empathy, being flexible, and trustworthy, and finally organizational skills, such as planning24 . 23 Sweeny and Harris, 2017, as cited in Hofmeyer, J., 2019 24 Quintero, 2019; Knight 2009; Toll 2014; as cited in Hofmeyer, 2019 Excellent Coaches Average to good coaches Weak coaches Coach 1 Coach 3 Coach 8 Coach 9 Coach 2 Coach 5 Coach 10 Coach 13 Coach 16 Coach 17 Coach 4 Coach 7 Coach 11 Coach 12 Date note: Case studies of the coaches in bold are provided in Annex 8 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 30 An excellent coach ticks all the boxes in Annex 5.3, and was likely to have had a good balance of all the coaching skills (technical and administrative abilities, relationships and organizational skills), and had positive interactions with the teachers. The commitment to the development of the teacher was evident, and who not only performed all their duties but, in some areas, exceeded expectations. For example, the evaluator who shadowed Coach 8 noted that they “engaged with teachers and learners”. Coach 8 was noted as providing clear strategies to teachers to support them and that they were “not only a check box coach, but they follow through.” An average to good coach is a coach who has some strengths. Perhaps they were coaches that were very supportive to teachers, and provided strong feedback – but gave teachers direct feedback, rather than encouraging teachers to reflect on their own practice. These coaches were less likely to be productive, or they might have been administratively weak. These coaches needed additional support. This could have been provided by head coach feedback on their performance, or additional training to help them refocus their attention, and develop them to be excellent coaches. Weak coaches were coaches that were compliant to a bare minimum. These coaches stuck to the observation tool only, following procedure but without focus on teacher growth or development. Some evaluators comments included: “This coach wanted to get things done and out of the way” (on Coach 7). Coach 4 was noted as being “poor, they stuck to the observation tool and did nothing else, the coach was not developmental towards the teachers.” Coach 11 and 12 actually disrupted fieldwork by the evaluation team when the evaluation teams went back to schools and disrupted the learners in the process. Areas of strength and areas requiring further development Coaches were observed during lessons and for SBWs (if they occurred). The criteria for observation and coach assessment findings are presented in Table 8 below. Further detail on the criteria is included in Annex 5.3. Table 8: Adherence to observation stages during coach shadowing by all coaches Stages of a lesson observation/coach shadowing Fully Partial to Incomplete Planning and Preparation 11 coaches 3 coaches Pre-observation discussion 9 coaches 4 coaches 1 coach Classroom observation 7 coaches 7 coaches Post-observation discussion/Feedback 4 coaches 10 coaches Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 31 Stages of a lesson observation/coach shadowing Fully Partial to Incomplete School-based Workshops 3 Coaches 6 coaches 5 coaches Data note: This refers to the number of coaches that meet the criteria. For more information on the criteria refer to Annex 5.3 Table 8 illustrates that most coaches (11 out of 14) are fully competent and compliant with the aspects of planning and preparation before they arrive at schools. Most coaches (9 out of 14) are also compliant with what they are expected to do before the lesson begins. Coaches are equally split between an average performance or fulfilling partial requirements (7 out of 14), and fulfilling all the requirements expected during classroom observations (7 out of 14). However, the major distinction between coaches then comes when their feedback process is observed, with most coaches (10 out of 14) only completing partial elements of the feedback process and with varying quality. This is an area for further development across coaches. SBWs were a second distinguishing element: five coaches (out of 14) did not conduct an SBW during shadowing. Only Coach 8 fully adhered to all observation stages and could be deemed ‘fully compliant’. Differentiation between coaching practices As in 2019, coaching practice differed between individual coaches. Interviews with RSP training partners revealed that all coaches improved, albeit to varying degrees. In particular, they noted improvement in terms of their dry runs and teacher training on the RSP materials. Some differences in terms of how the coaching intervention was delivered was observed: 1) Dosage and lesson observation times FPD measured coach performance based on ranked measurement against the number of lesson observations conducted per day. Number of lessons observed, or number of teachers supported was also an indicator to determine whether the required dosage of the coaching intervention was met. However, coaches did not observe lessons for equal lengths of time. Some coaches stayed for the entire lesson to ensure that they observed all lesson components. Other coaches stayed for shorter periods of time, but were then able to see more teachers in a school on one day. Lesson observations ranged from ten minutes (one teacher with one language) to 110 minutes (one teacher one language). An average lesson observation took 50 minutes, seven out of 50 lesson observations took place for less than 30 minutes. While coaches might have observed a teacher for longer in total, they would then record both languages observed (despite the variance in time spent) as two separate lesson observations. This also meant that some coachers supported more teachers, but for a shorter amount of time. A timetabling issue also affected this, as some schools scheduled Grade 1 EFAL, Grade 2 and Grade 3 EFAL concurrently, followed by HL (or vice versa). However, unless the coach asked teachers to change the timetable, a maximum of two lessons could be observed. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 32 2) School Based Workshops/Feedback Some coaches did not provide individual feedback at the end of a lesson, but instead provided group feedback at the end of the school day. This is appropriate in some circumstances as teachers may learn from their colleagues and benefit as a collective from the coaches’ feedback. This approach depended on the relationship between coach and teacher as some teachers might have felt embarrassed by open discussions. Table 9: The difference between SBW and Group Feedback SBW Group Feedback Coaches identify key and common needs among teachers, and then provided guidance Coach provides feedback to all teachers on lessons observed Workshop methodology Discussion, with questions While this approach has its benefits and its disadvantages, group feedback differs from a SBW. If this were the case, the evaluation team determined that a SBW had not occurred as this session was the role of feedback. Some coaches provided individual feedback to each teacher after the lesson, but then had a group discussion with all of the teachers which was more general and was a space for teachers to ask general questions. In these instances, the evaluation team determined that while this was less of a ‘workshop’ as designed, this did fulfill the role of a SBW as feedback had already happened. Coaches were expected to do SBWs every day, which in some cases did not make sense to the coaches. Some coaches were able to visit their smaller schools more than once a month, and therefore did not see the need to conduct a SBW after every visit – and so some would conduct observations in the first week, and then use those observations to inform the SBW that they would conduct the second week. Similarly, coaches have to visit their bigger schools multiple times over consecutive days in order to visit all their teachers at those schools, and in those cases did not feel that it made sense to hold a SBW at the same school for multiple days in a row, despite this being the expectation. 3) Coaches’ level of agency/initiative/adaption/discretion The above responsiveness to the situation and the level of adaption to conduct a SBW highlighted the need for differentiation in coaching practice. Differentiation relates to the level of discretion or adaption that coaches could exercise. An adaptive coach, who is responsive and flexible, embodies a more ideal coaching methodology. However, their flexible practice may result in them not meeting the fidelity criteria. Without knowing how they have adapted the program results in a challenge to measure coaches’ level of agency. Therefore, although the level of fidelity may not have been upheld, the adaptive coaches may have provided a quality service to the teacher and school. This was challenging for the evaluators to compare across coaches. This was an overall observation and was seen in various examples, but it is difficult to quantify. This was seen in the following ways: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 33 Firstly, some coaches indicated that they could never deviate from their monthly plan even if they encountered challenges. For example; when shadowed, Coach 2 (an average to good coach) knew in advance that the teachers in the school on the second day were unavailable for shadowing as they were writing assessments. When asked by the evaluator why the coach did not change their plan to support other teachers, the coach said “because they were on my plan for that day, and I cannot deviate from my plan”. The coach and the evaluator then conducted the interview, and left the school. Another coach indicated that they cannot change their schools on their plan if crises happen; such as flooding/ unavailability of teachers. This not only raises the question of the role of the coach during assessment time; as other coaches were also observed during assessments, but indicates the perceived lack of schedule flexibility. The cost-effective, time-efficient and responsive course would have been to adjust the coach’s schedule to see other available teachers. Secondly, coaches were meant to communicate their schedule in advance to the teachers. Two coaches indicated to the evaluator that they often will tell a school in advance, but will not tell the teachers. They explained that some teachers might be absent if they expect a coach visit, or might over-prepare and the lesson observation will be unnatural. These coaches choose to ‘surprise’ the teachers, and tell them to always be ready. Thirdly, coaches are meant to provide individual feedback to teachers after the lesson observation. However, in some instances, coaches recognize that that might not be the most conducive way to offer feedback. Coach 3 (an excellent coach) was also observed providing feedback in a group setting and said to the teachers; “You must rely on each other and learn from each other’s strengths, when I’m not here, you will have each other.” “Ranking my teachers [has changed a lot from 2019] because now when I am doing my plan, I’ll focus on which teacher needs support, because I cannot support the teacher whilst there’s no need. Now, I will check on my report that, [if] [teacher’s name] is not doing good in reading. So, I will just put [her] in the forefront that I’m going to assist” (Coach 3) Coach 17 (an average to good coach) was observed providing feedback to teachers in a group setting. The teachers had been quite weak overall, and a new teacher had been defensive at being observed. The coach then provided general feedback to all the teachers, without naming names. The coach said to the evaluator; “If I told her she was wrong, she would not have listened, but she can hear the feedback with her colleagues and take it to heart and learn from it.” These were examples of coach adaption and flexibility observed during coach shadowing. While these adaptions make sense and illustrate a sensitive and responsive approach, at a larger scale and to a larger extent, these kinds of adaptation might impact program fidelity. Quality of the SMT intervention In terms of quality of the intervention, 33 (60%) of 55 SMT members who attended any PLCs or workshops self-reported that some workshops more useful than others, compared to 22 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 34 (40%) who did not find any workshop more useful than others or attended only one workshop. Altogether only nine out of 64 SMT members did not attend any workshops, and therefore could not respond as to their usefulness and quality. When asked what the best part of the PLCs or workshops was, 40 SMT members mentioned a number of elements as can be seen in the table below. Table 10: Themes from SMT members on SMT PLCs / Workshops Best part of the SMT PLCs / Workshops - Themes Number of responses Methodologies for teaching reading/literacy and supporting teachers (e.g., group guided reading) 11 Leadership and role as principal/SMT member 10 Group discussions and sharing with others 8 Management and lesson planning (e.g., how to use Data Driven Dashboard) 6 Workshop facilitation and content of the workshop 5 Data note: SMT members mentioned more than one theme. N=40 responses from 64 SMT members from SMT schools who completed the survey SMT members commented: “The Presentation about the history of reading i.e., poor performance of reading in the country and how RSP would improve- I was not aware of this until it was presented. It made me change my focus. Facilitators were friendly and good.” (SMT member) “When we interact with other teachers and share the problems we encounter at school and how to overcome them.” (SMT member) “It was very educational and I was motivated to do the work and guide my learners.” (SMT member) 2.3. PARTICIPANT RESPONSIVENESS Participant responsiveness is defined as “How far participants respond to, or are engaged by, an intervention” (Carroll et al., 2007, p.3), or how well the instruction was received or perceived by participants (Buckley et al., 2017). Overall, the teachers were responsive to the coaching and LTSM (refer to next section for more on teacher uptake in the classroom). The coach training appears to have added value to the coaches’ knowledge and practice of coaching. However, teachers who were involved in both RSP and EGRS I mainly indicated the coach quality was similar. The relationship Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 35 between the coaches and head coaches improved in 2020, compared to 2019. Both were more responsive to each other’s needs and development. Responsiveness of teachers to teacher training Teachers’ overall rating of relevancy and quality of training slightly declined between Terms 1 and 4 in 2020. The only intervention group to report a slight increase (of 1%) was the coaching group (refer to Table 11). Table 11: Perception of teacher training usefulness by intervention group How valuable did you find the Term 1 and Term 4 training workshops? Term 1 (2020) Term 4 (2020) Response ratings of Valuable/Extremely Valuable Coaching Teachers 84 out of 95 (88%) 82 out of 92 (89%) SMT Teachers 91 out of 99 (92%) 94 out of 102 (80%) LTSM Teachers 76 out of 81 (94%) 67 out of 86 (78%) Total 251 out of 275 (91%) 243 out of 280 (87%) Grade Teachers 1 82 out of 93 (88%) 77 out of 93 (83%) Grade 2 Teachers 81 out of 89 (91%) 78 out of 86 (91%) Grade 3 Teachers 88 out of 97 (91%) 87 out of 101 (86%) Total 251 out of 275 (91%) 243 out of 280 (87%) Data note: Data was unavailable by venue or facilitator A few teachers (7 out of 275 for Term 1 and 11 of 280 for Term 4) indicated that the teacher training sessions was “not at all valuable”. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 36 Teacher responsiveness to coaches Overall, more than 85 percent of teachers in both coaching and SMT interventions felt that coaches had assisted them as shown in Table 12. Table 12: Teachers view of coaches Coaching Intervention SMT Intervention Teachers felt their coaches had helped them to grow to a large or very large extent 93%, 98 of 105 98%, 102 of 104 Teachers gained helpful resources from their coach to a large or very large extent 94%, 98 of 105 94%, 98 of 104 The teacher felt coaches overcame barriers (to a large or very large extent) to teaching and learning 88%, 92 of 105 95%, 99 of 104 Teachers reported that coaches set action plans after each feedback session 84%, 88 of 105 91%, 95 of 104 Coaches helped them to implement the revised ATPs 88%, 93 of 105 84%, 98 of 104 Responsiveness of coaches and head coaches In 2020, when the coaches were asked about the type of support that they received during the RSP implementation, the coaches noted that the two head coaches had increased support and presence. “What has changed in the last year is that the head coaches are more visible at schools.” Coaches had to move their communication to virtual platforms with the head coaches due to COVID-19. The coaches reported that the head coaches provided implementation support by sending out resources and instructions, such as activities that coaches could send out to teachers such as topic of the week. However, the analysis of the WhatsApp group chats does not indicate these resources were then shared with teachers. The coaches also noted that head coaches were more visible in 2020. They stated that the head coaches’ visits to the schools during lesson observations assisted them with the implementation of the project. Several coaches reported that one head coach provided feedback from the lesson observation and SBWs. “The head coach comes to schools that I am visiting, they give their inputs during feedback sessions and also during SBW. They also come to trainings and provide assistance there, they also provide support by providing materials if I need.” (Coach 4, a weak coach) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 37 One head coach was reported to have increased emotional and psychological support to coaches during 2020. This head coach was reported to be available to coaches at any time to deal with work related or personal issues. However, one of the implementing partners felt this head coach was disorganized and a poor role model. The other head coach was reported to visit schools but did not provide feedback to the coaches on their performance. Therefore, the coaches did not feel their developmental needs were being met. However, they felt supported by the program overall. Head coaches were seen by the coaches to provide administrative support to the coaches and provide them with resources when needed. One coach reported that although the head coaches are visible and provided support to implement project activities, they felt they did not provide the support they needed throughout the project. This coach felt that “the only support I know is administrative because the main priority that makes us communicate more is the [monthly] report.” (Coach 5, an average coach) The head coaches stated the coaches were better equipped to carry out their roles and responsibilities in 2020. The coaches were each provided with a car, the coach trainings developed coaches’ skills and knowledge, and coaches were better able to train teachers. The FPD provincial office administration improved and this supported overall program implementation. For example, there was an improvement in sending out event invites to teachers for trainings and to provincial representatives for reference meetings. Respondent reported that the improvement in coordination of sending invitations to schools for teacher trainings increased teacher attendance and increased engagement with the targeted teacher population, assisting coaches to provide support to the correct teachers. Both head coaches stated that delivery of materials to teachers in 2020 (due to school principals collecting materials or disseminating them during the teacher training) was one of the biggest successes and improvements, given how challenging this was in 2019: “Biggest improvement can be seen with the distribution of materials, they are delivering on time and if someone says that they need material then it is dispatched to them quickly.” (Head Coach) The head coaches also stated that another improvement was better project management and implementation in 2020, assisting the coaches to improve their facilitation of the teacher trainings. The head coaches mentioned that the coaches were more comfortable with presenting and delivering the training and that was due to having the coaches carry out dry runs during the coaches’ trainings. Responsiveness of SMT members Project documentation reports engagement of SMT members in the PLCs and that training materials such as manuals and presentations were well received and uploaded to tablets. SMT members indicated that they had changed their practice in schools (83%, 55 out of 66). However, as seen in the dosage/coverage section, only a limited number of SMT members attended all PLC workshops. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 38 2.4. PROGRAM DIFFERENTIATION Program differentiation is defined as “The degree of contrast between treatment and control strategies and/or activities [emphasis added]” (Buckley et al., 2017, p.5). This Formative Evaluation examined program differentiation specifically in terms of the degree to which the program was differentiated within each of the treatment and comparison conditions. Ideally, instead of a “one size fits all” approach, the program would be implemented differently (for example, light touch versus more intensive support) according to (i) status of reading in the school, (ii) rural versus urban schools, (iii) previous EGRS schools versus new schools, and so forth. Therefore, ideally the RSP should have applied a differentiated implementation approach. For example, in 2019, of the 102 surveyed teachers who reported receiving RSP coaching support, 89 percent reported that they also received support from coaches affiliated with the EGRS. Given the large numbers of teachers who had previously interacted with EGRS coaches, RSP interventions could have differentiated across teachers who had had prior exposure to coaching. For example, previous EGRS schools receive a lighter touch intervention than those who were previously in a control condition in the EGRS I. Similarly, use of baseline data on reading collected prior to RSP implementation could have allowed differentiated support according to the reading level of learners in the schools. Baseline of learner reading performance was taken as part of the EGRS sustainability evaluation25. Although this baseline included RSP schools, FPD, the Coaches and schools were not given baseline results to use as a diagnostic tool for differentiation. The summative analysis of delivery differentiation between the RSP interventions indicate that without high levels of SMT support, the two coaching conditions are very similar. Without proper dosage and quality of coaching, it is important to explore the extent to which these interventions are differentiated in practice. In 2020, of the 309 teachers who responded to the survey, 88 percent (272 of 309) indicated they had participated in the RSP in 2019 (37 teachers were new to the project: 19 in the coaching schools, 10 in SMT and 8 in LTSM schools). The monitoring data indicates that 75 percent of teachers participated in the RSP in 2019 and in 2020. However, not all teachers completed the survey. 25 The baseline data was collected under PERFORMANCE Task Order 1 72067418D00001, Order Number: 72067418F00010 in 2018. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 39 Table 13: Teacher perception of improvement of RSP between 2019 and 2020 How was your experience with RSP in 2020, compared to 2019? (n=270) Coaching SMT LTSM TOTAL Worse than 2019 14 (16%) 5 (5%) 9 (11%) 28 (10%) About the same as 2019 41 (46%) 49 (51%) 39 (47%) 129 (48%) Better than 2019 35 (39%) 43 (44%) 35 (42%) 113 (42%) Total 90 97 83 270 More Grade 1 teachers (47%, 41 out of 88) indicated the program had improved since 2019 than those that indicated it had stayed the same (36%, 32 out of 88) whereas the majority of Grade 2 and 3 teachers indicated the RSP had stayed the same (54%; 44 out of 82 and 53%, 53 out of 100). However, 17 percent (15 out of 88) of Grade 1 teachers indicated the RSP was worse in 2020 than in 2019. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 40 3. WERE THERE ANY MAJOR DIVERGENCES FROM THE ASSUMPTIONS IN THE THEORY OF CHANGE? (QUESTION 5) The ToC (refer to Figure 3 above) assumptions were documented in the Design Evaluation. USAID approved the revised RSP MEL Plan in May 2020 which did not change the assumptions. In some areas in 2019, the assumptions were not being met. This included in the areas of coach training (coaches were not able to execute dry runs during the last coach training), teacher training (fewer teachers attended than expected), LTSM delivery (not all teachers are receiving LTSM and some teachers are not receiving all LTSM), and dosage of Coaching support. Given the centrality of the Coaching intervention to the RSP, the 2019 Formative Evaluation reviewed data against the coaching assumptions outlined in the Design Report. Indications were that the all coaches were monitoring and checking teacher implementation against the lesson plans and documenting their observations. Strong coaching practices included: adequate planning and preparing for visits to the schools, and engaging appropriately with the school principal, SMT, and the teachers. Weaker coaches were compliant only with reporting but lacked coaching skills and experience. Without the SBWs, the coaching dosage was too low to support differential impact. The unmet dosage assumptions were noted as the greatest threat to the RSP ToC. In general, the assumptions identified in the Design Evaluation ToC, are plausible and proven. However, both the Formative and Summative Evaluation found implementation remained generally partial or insufficient. Further detail is provided in Annex 9. The following assumptions was found not to hold in 2020: “Sequencing HL and EFAL training is optimal, and complements teachers’ learning”: does not hold as not plausible because the HL and EFAL training sessions were alternated as first and second session during training and this did not affect teacher engagement, participation or learning experience. The major divergences from the ToC were as a result of COVID-19, or a change in design due to implementation factors. The following divergences were noted in 2020: To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: - Design Evaluation Report - Formative Evaluation Report - Findings presented in the previous questions of this Summative Evaluation Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 41 a) The evaluation was not able to ascertain if it is possible to implement the strategies in the lesson plans with large classes26 . b) Due to COVID-19, coaches were not able to run “Adequate catch up sessions are implemented for those who missed training.” Evaluators note that implementation varied by coach. In addition, ‘catch up’ was not clearly defined in the RSP Standard Operating Procedures (SOP). c) Elements related to the SMT elements did not hold as VSO withdraw from RSP such as “SMT receive the necessary support from international volunteers”: and “SMT receive supervisory visits from volunteers and maintain a positive working relationship”. There are 19 instances where the assumptions are plausible and hold, but are unproven. This is either due to a) lack of data, b) evidence required is beyond the scope of this Summative Evaluation (as it requires a sustainability study or impact evaluation), or c) there is insufficient evidence for confirmation of the assumption. Refer to Annex 9 for further detail of the ratings of the ToC activity assumptions. 26 According to the OECD, smaller size classes are often seen as beneficial because they allow teachers to focus more on the needs of individual students, and reduce the amount of class time needed to deal with disruptions. According to the DBE Norms and Standards: a primary school classroom should not have more than 35 learners per educator. The Annual Performance Plan (2018) for the North West Education Department, however, indicates that 45 percent of learners are in classes with more than 45 learners. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 42 4. TO WHAT EXTENT DID COVID-19 AFFECT THE IMPLEMENTATION OF THE RSP AND HOW DID THE PROGRAM RESPOND AND ADAPT? (QUESTION 15) 4.1 COVID-19 RESPONSE IN SOUTH AFRICA On March 23, 2020 the South African president announced a national lockdown in an effort to control the spread of COVID-19 in South Africa. On March 27, 2020 South Africa went into a national Level 5 Lockdown27 . What began as a three-week lockdown period transformed into an ongoing lockdown with various levels of restrictions (such as curfews, social distancing and restrictions on the number of people who can gather or be in a workplace) for the remainder of 2020. Schools were closed from Wednesday, March 18, and started a phased re￾opening from June 1, 2020: the exit grades (Grades 7 and 12) returned first. The initial plan was for the Foundation Phase of schooling to return to school on July 6, but this was postponed to a later date. Schools were again closed from July 27 to August 24, when they reopened for all grades. The DBE issued Guidelines for Development of the School Timetables – Reopening of Schools COVID-19 (2020)28 , which explored the different models’ schools could follow when they reopened to ensure that all schools adhered to the social distancing regulations. The models entailed: 1) platooning (using limited classroom space in turns on a rotational basis); 2) alternating days of attendance by Grades per week, or 3) implementing bi-weekly 27 South Africa has five levels and level 5 is most restrictive. See this summary of the levels https://www.mict.org.za/wp-content/uploads/2020/04/COVID19-Presentation.pdf 28 http://section27.org.za/wp-content/uploads/2020/05/guidelines-for-timetabling.pdf To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: - Review of literature on COVID-19 in South Africa - Review of key project documentation - Classroom observations - School Functionality assessments - Teacher survey - SMT survey - Interviews with 6 district / provincial officials Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 43 rotational attendance of learners. In South Africa, the education impact of the lockdown and school closures were profound. South Africa has approximately 13.3 million learners and 440,000 teachers in 29,749 education institutions29 . By the end of Term 2 (August 7, 2020) Van der Berg et al estimated that at least four million children could have missed more than half (57%) of the number of school days scheduled up to that point30 . This, coupled with the social distancing guidelines, meant that children probably attended only half the available school days from August to December 2020. The staggered re-opening of grades in 2020 resulted in a range of school days being lost. According to Hoadley (2020, pg. 4), “…Grades 5 and 8 and Grade 4 and 9 suffering the greatest proportion of days lost from the pre-COVID calendar (42% and 39% respectively). The actual number of school days ranged from 118 days to 168 days”. For Grades R, 1, 2, and 3 (as well as Grades 6, 10 and 11), Hoadley estimates that 68 percent of the pre-COVID school calendar days (204 days) were available for instruction during 2020. Based on the Government Gazette, school dates on August 11, 2020, Mohohlwane et al (2020, pg. 23) indicated that for Grades 1, 2 and 3, 82 days (40%) of schooling were lost in 2020 due to COVID-19. The NIDS-CRAM Wave 2 data (2020) estimated that learner attendance rates by grade prior to school closures in July for Grade 1 was 18 percent; Grade 2 14 percent and Grade 3 25 percent (Mohohlwane et al, 2020). Additional time losses were incurred due to social instability, timetable modelling, closures due to COVID-19 infections, teacher shortages and absenteeism, learner absenteeism and the cessation of instruction for 2020 before the scheduled end of term on December 15, 2020 (Hoadley, 2020). DBE monitoring data in 611 schools (Mohohlwane et al, 2020) indicated that seven percent of primary schools had at least one COVID-19 case amongst teachers, and this would have resulted in additional closures. During the school closures, learning relied on the parent or caregiver’s ability to provide support to children. This would have varied depending on the caregiver’s education levels, work requirements and availability of educational resources. This, according to Hoadley (2020, pg. 9), “will create disparities that map on to existing social and educational inequalities.” According to Gustafson and Nuga (2020, cited in Hoadley 2020 pg. 8), learning-adjusted measures of time lost require inflating actual days lost by 25 percent. Hoadley points to the need to determine the actual number of days lost, if losses over time are linear, and the effect of losses on different groups. 29 SA Education Stats, 2016. https://www.education.gov.za/Portals/0/Documents/Publications/Education%20Statistic%20SA%2020 16.pdf?ver=2018-11-01-095102-947 30 Van der Berg, S & Spaull, N. (2020). Counting the Cost: COVID-19 school closures in South Africa & its impacts on children. Research on Socioeconomic Policy (RESEP). Stellenbosch University. Stellenbosch. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 44 For the North West Province, the following data is available: The National Income Dynamics Study – Coronavirus Rapid Mobile Survey (NIDS-CRAM) Wave 2 (2020) data estimates for the North West Province that 92 percent of learners attended for “open” grades (Grade 6, 7, 11 and 12) and 27 percent attended for the “not yet open” grades (Grades 1 to 5, and 8 to 10) In general, primary schools in the North West adopted a platooning model of “the rotating model where some of the learners come to school for two days and some of the learners come for three days in a week and vice versa.” (RSP PMT minutes, September 3, 2020) Nine RSP schools were closed due to COVID-19 during 2020 due to positive incidents of teacher infections and due to principals being quarantined, as reported in the RSP PMT minutes of September 3, 2020 DBE monitoring data31 provided to the evaluators indicated that one school principal, one deputy principal and five Foundation Phase teachers passed away from COVID-19 in the Ngaka Modiri Molema District. These losses affected seven Primary Schools. No staff passed away in the Dr. Kenneth Kaunda District. Based on the available published research presented above, the evaluation estimates that out of the 2020 planned schooling days (204 days), between 82 days (Mohohlwane et al. indicated 48% of 204 lost) and 138 days (Hoadley indicated 68% of 204 days lost) were lost. In response, the DBE undertook four strategies to address curriculum recovery (Hoadley, 2020): 1. Adjusted the curriculum to reduce the content requirements 2. Rationalized and suspended subjects, for example the North West PED suspended the teaching in Life Skills in the Foundation Phase (Grade 1, 2 and 3) of schooling (PMT minutes, September 3, 2020) 3. Changed assessment procedures, for example in Grade 1, 2 and 3 no formal assessments were undertaken (DBE national Assessment Circular 02 of 2020, 09/07/2020) 4. Encouraged remote learning through using online platforms or sending materials for the child to complete at home. The COVID-19 pandemic closures affected school attendance, teaching and learning, and meant that RSP could not adhere to the planned RSP design after Term 1 of 2020. Consequently, the RSP implementation had to adapt to these school closures. Therefore, the fidelity of the model as described in the Design Evaluation cannot be evaluated as planned for the full period of 2020. 31 Data provided by the DBE via e-mail on February 22, 2021. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 45 The summative evaluation therefore examines the following three implementation scenarios for 2020: These scenarios are discussed throughout the report. 4.2 HOW DID THE RSP DESIGN CHANGE TO ADAPT TO COVID-19? The May 26, 2020 RSP PMT minutes, which was the first meeting held after the lockdown announcement and initial three-week lockdown, reflect that three meetings (March 25, April 14 and April 24, 2020) were held to discuss the COVID-19 response on the RSP between USAID, DBE and the FPD Consortium. The RSP COVID-19 strategy was approved and adopted at the May PMT meeting. This strategy included following DBE protocols and the procurement and use of personal protective equipment (PPE). The expectation expressed during the PMT meetings was that schools would re-open in a phased approach on June 1, 2020. The focus for the RSP was to deliver the LTSM to schools when possible. In addition, based on a recommendation from the DBE and while waiting for the final new school calendar, FPD requested a no-cost extension from USAID until the end of the 2020 academic year (December 2020). The RSP workplan was therefore amended, the main activity, as noted in the minutes that required re-organization was the on-site coaching support and re-scheduling of training. The coach training was moved to an on-line platform (this included the RSP coach, head coach and the UNICEF32 training sessions). The evaluation team attended some online trainings, and reflections on these are provided in Section 4.2. The effectiveness of providing virtual coach training is reviewed later in the report. The coaching on-site support was changed to coaches setting up WhatsApp groups (see RSP Teacher Engagement Strategy, August 2020). 32 The UNICEF coach training was not part of the RSP. This was provided as additional training to coaches. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 46 The June 30, 2020 PMT minutes noted that all teacher training for Term 2 and 3 were suspended due to school closures as a result of the COVID-19 pandemic. Some teachers were however engaged in WhatsApp platforms, although access to data due to connectivity and cost reportedly affected teacher engagement in this platform. The evaluation team analyzed data from teacher WhatsApp groups with their coaches and findings are presented later in this report. LTSM Big Books were unavailable during the South African lockdown due to delays in the printing, packaging and delivery, which were ultimately delivered in Term 4. However, the LTSM Big Books were not part of the initial design. The PMT minutes further noted that provincial approval had been sourced for coaches to enter schools. Coaches with co-morbidities were a concern and were unlikely to enter schools once they re-opened. This constrained RSP capacity to provide coaching, and DBE requested FPD to account for the number of coaching days lost33 . Although six coaches reported co-morbidities and three were over the age of sixty, the coaches continued to provide coaching services and 13 of the 1434 went into the schools once permission had been granted by North West PED (Coach data and June 2020 PMT Minutes). The coach shadowing process conducted as part of the evaluation confirmed that all coaches went back to provide support to teachers from October 1, 2020 – November 30, 2020. The DBE amended ATPs were also factored into the workplan and that the content of the RSP lesson plans were to be amended and aligned to the DBE Curriculum and Subject Advisor (CA/SA) plans. The reference group material development for Term 4 focused on aligning the content of the RSP LTSM into the revised ATPs. The evaluation team verified this by participating in the reference group meeting and reviewing the ATPs after they were finalized. Through coach training observations, the evaluation team confirmed that coaches were trained on the revised ATPs during coach training. In turn, during the coach shadowing/fieldwork process and teacher training observations, it was verified that coaches then trained teachers on how to use the revised ATPs during their school visits and later during Term 4 training. Coaches focused their Term 4 support to teachers (including SBW content) on the revised ATPs. The August 27, 2020 minutes indicated that the North West PED stated they would only allow face to face training to begin at the end of September 2020 and therefore training could not occur the week before the start of term as per the RSP design. The September 3, 2020 minutes noted that “RSP team has tried to adapt and align learning materials to work that is done by the department through the trimmed curriculum.” These materials were usually distributed at the teacher training, however as this could not be undertaken, an alternative strategy was implemented. Schools arranged to collect material from a central point. The fourth and final session of the SMT PLC was moved to the end of 2020. The October 13, 2020 minutes indicated that teacher training had resumed. Terms 2 and 3 training was combined to maximize face-to-face contact between coaches and teachers, and to reduce teacher time out of the classroom. The minutes of the PMT for November and 33 While this request was documented in the minutes, there was no record of FPD responding to this request. 34 One coach was on sick leave for six weeks (went into surgery in early October and returned to work on November 17, 2020), but the head coach filled in during the absence. Once the coach returned to work, a Khulisa evaluator conducted the shadowing process with the coach. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 47 December 2020 noted the implementation of the amended workplan and activities for the close-out of the project as USAID had granted the no-cost extension. The Summative Evaluation team conducted fieldwork in November 2020. The findings indicate the effect of the amended implementation plans on the three intervention models. Overall, the findings below indicate that despite the planned adaptations, implementation of the RSP was affected and the responses did not sufficiently overcome the COVID-19 disruptions to maintain teaching and learning during lockdown. Changes to the coaching intervention The COVID-19 pandemic and its resultant lockdown had a number of effects on coaches as they could not go into schools to carry out their planned activities. This meant that the coaches had to adapt their activities to the situation on the ground. Coaches reported that the lockdown affected their role and function as coaches. As part of building strong working relations with the teachers and fostering cohesion within the coaches’ cluster of schools, coaches were encouraged to use virtual and social platforms such as WhatsApp when schools were closed, and prior to being able to meet with teachers face-to-face. WhatsApp became the primary mode of communication as the teachers and coaches all used the platform. The coaches reported that they had created WhatsApp groups for all their teachers. When interviewed, some coaches reported that they used WhatsApp to send out any questions from teachers, to send out activities and other education materials that could assist in implementing the project. Coaches also reported that they would send teachers reminders and questions on the core methodology like the group guided reading: “I have WhatsApp groups with my coaching teachers and my LTSM schools teachers. I sent them a message every day.” (Coach 9) The coaches reported that the although the WhatsApp groups were very useful to them supporting teachers during lockdown, they experienced a number of issues such as teacher responsiveness and the interpretation of the RSP materials, “We didn’t manage to meet with the teachers; some of them are unable to interpret the contents of the lesson plan without the assistance of the coach, especially in terms of assessment.” (Coach 7) Coaches reported that they would call teachers to try and offer more support and guidance, but the coaches reported it was difficult to coach over the phone. Coaches also reported that although they had adapted the implementation of the project in 2020 to rely on WhatsApp for communication, teachers were often unresponsive. Reportedly the cost of data became an issue: “The interaction was there, but not that much. Mobile data is a challenge. Some teachers said they didn't download what I sent them because they don't have enough data.” (Coach 9) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 48 Figure 8 Analysis of coach WhatsApp chats Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 49 Analysis of the available WhatsApp groups35 (refer to Figure 8), indicates that except for two coaches, the platform was used as a bulletin board and logistics platform. Two coaches, Coach 1 and 9, however used the WhatsApp platform not only as a communication channel, but to provide coaching and psychosocial support. The head coaches also used the platform as a communication channel. The two coaches’ WhatsApp group chats illustrate that a considerable amount of personal coaching and psychosocial support is possible online. A platform such as WhatsApp could have provided greater guidance to teachers on how to provide learning opportunities to learners during school closures. The coaches further reported that their roles were adapted in September 2020 when schools opened and they were allowed back into schools. Some coaches reported that they offered teachers emotional support over WhatsApp or during the training. One coach reported that: “…during teacher trainings in September, COVID-19 had left them traumatized and it was evident in trainings.” (Coach 4) However, Coach 4 did not expand on how the trauma was evident or how it may have affected the teachers’ receptiveness to the training. When they were allowed into schools, coaches faced a number of challenges that affected their planned activities. These ranged from school rotation systems to learners to attending school. Coaches reported that COVID-19 reduced the number of SBWs (SBW) they could provide. “It also affected my target for the SBWs because sometimes when you go to schools there are no teachers. And also the lesson observations. …..schools of mine that I visited last week, I couldn't observe because there were no kids. I need to observe the teachers practicing with the learners. But there were no learners.” (Coach 9) Changes to the LTSM intervention The following changes were noted from the PMT Minutes and key informant interviews (KIIs) with the implementing consortium: The material distribution for Terms 2 and 3 relied on principals or schools arranging to collect the LTSM. For Term 2, FPD developed a LTSM Dispatch Plan that included COVID-19 protocols when LTSM was being collected (FPD document, week ending July 10, 2020). The effectiveness of this strategy is discussed in later in the report). The printing of some of the LTSM materials, particularly the Big Books, was delayed as printers were closed during the national lockdown (FPD Consortium interview, 2021). 35 Coaches were requested to extract WhatsApp media from their phones and send them to two Khulisa representatives. Nine coaches, and one head coach responded. Fourteen group chats were analysed. Three of the fourteen groups had commencement dates in 2019, the rest were opened from May 2020 onwards. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 50 The Reference Group meeting with Provincial and District officials was held online rather than in person. FPD enabled the CA/SAs to participate online through providing data prior to each meeting. Changes to the SMT intervention SMT training (also referred to as SMT PLCs) was held in 2019 to February 2020. COVID-19 did not affect the training for SMTs. However, no support visits were conducted to the schools due to the withdrawal of VSO and changes to the SMT mentor SoW. The SMT training resumed in November 2020 with the final consolidation session. Is there a difference in the schools which had SMT support to those that did not? Of the 49 teachers in the SMT intervention group, 19 (39%) reported during COVID-19 school closure that schools planned that teachers send learning materials home to learners. In contrast, 18 out of 32 teachers (56%) of LTSM schools and 13 out of 40 (33%) of coaching teachers reported schools sending materials home to learners. Actual distribution of materials by teachers to learners during school closure was however, reported by slightly more SMT teachers (46%, 49 out of 107) than coaching (36%, 40 out of 111) and LTSM schools (35%, 32 out of 93). This indicates that the SMT intervention potentially had a slightly stronger response to ensuring teaching and learning continued during the lockdown, according to the self-reported data. The teachers’ initiative to send materials to children during the lockdown period was the same across the coaching and SMT interventions (57%), while LTSM teachers reported a much lower initiative to send materials (31%). 4.3 HOW DID TEACHERS ADAPT AND RESPOND TO TEACHING PRIOR TO, DURING AND AFTER SCHOOL CLOSURE? During school closure A higher percentage (42%) of teachers in the Ngaka Modiri Molema district than Dr. Kenneth Kaunda district (33%) reported sending materials home with learners during the COVID-19 school closure. The percentage varied across the interventions and grades (Table 14). All interventions and grades had a low level of teacher performance in continuing to provide learning opportunities to learners. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 51 Table 14: Teaching response to COVID-19 by intervention and grade Question Responses Coaching SMT LTSM Grade 1 Grade 2 Grade 3 During school closure did you send any materials home with your learners? Answer: Yes (121 out of 311) Yes 40 49 32 31 50 40 No 71 58 61 71 49 70 Total 111 107 93 102 99 110 Was this (sending materials home to learners) that the school planned with all teachers or you decided to send materials? Answer: I decided to send the materials to my own learners (more than one option was provided) Yes 23 28 10 18 28 15 N= 40 49 32 31 50 40 During school closures, did you communicate with your learners through their parent? Answer: Yes (n=76 out of 311) Yes 21 39 16 24 24 28 No 90 68 77 78 75 82 Total 111 107 93 102 99 220 Only a third of the teachers (39% or 121 out of 311) reported that they sent work home for the learners during lockdown. Of those, half (61 of 121 teachers) reported that it was a school wide effort. Only a quarter (76 or 311 teachers or 24%) said they communicated with their learners through the parents. Overall, of the 76 teachers who reported communicating with parents, 66 percent communicated via WhatsApp (50 out of 76) followed by telephone calls (32%, 24 out of 76) and SMS (21%, 16 out of 76). No teachers reported communicating using emails. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 52 Communication was primarily about encouraging learners to read (96%, 73 out of 76), sending learning materials (21%, 18 out of 76) and asking them about their child’s well-being (21%, 16 out of 76) 36 . Term 4 Teaching and Learning Almost half the respondent teachers (48%, 150 of 311) returned to school on August 24, 2020, however 39 percent (121 of 311) reported returning prior to then and 13 percent (40 out of 311) returned after this date37 . The focus of the teachers upon returning to school in Term 4 was primarily on Setswana HL (Table 15). Table 15: Focus of learning during last week (Term 4) In the last week, which learning area did you manage to mostly focus on? (N=124, single response option) Coaching SMT LTSM Grade 1 Grade 2 Grade 3 Home Language 33 35 24 29 35 28 EFAL 4 5 7 3 5 8 Mathematics 9 2 5 2 8 7 Total 46 42 36 33 48 43 The DBE designed ATPs were disseminated to all teachers by the district and province. The RSP consortium mapped their lessons plans to the ATPs. This was shared with the teachers through coaches during Term 3 and 4 teacher training sessions. The coaches focused on supporting teachers to understand and implement the lessons plans and ATPs. Teachers focused more on phonics because of the emphasis on assessment and trimmed curriculum. In Term 4, at the time of conducting the evaluation, the priority in the Foundation Phase was on learner assessments. The ATPs specified the number of hours teacher were required to focus on HL, EFAL and Mathematics. For HL and EFAL, the lesson plans prioritized the use of reading books. As indicated in Table 15, the priority at the start of the term was on HL across all grades. The majority – sixty percent – of teachers (187 out of 311) indicated that they had not managed to teach all the learning areas in the previous week at the start of Term 4. A slightly higher percentage – forty percent – of teachers (46 out of 111 teachers) in the coaching intervention teachers managed to teach all the learning areas (Table 16). 36 Respondents could select more than one option 37 Percentages are rounded up in the narrative Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 53 Table 16: Teachers ability to teach all learning areas in the last week (Term 4) In the last week, did you manage to teach all the learning areas to your class? (N=311) Coaching SMT LTSM Grade 1 Grade 2 Grade 3 Yes 46 42 36 33 48 43 No 65 65 57 69 51 67 Total 111 107 93 102 99 110 Overall, (Table 17), across all interventions teachers indicated that it was difficult (71%, 222 out of 311) to implement the curriculum given COVID-19. However, a lower percentage of teachers in the coaching intervention reported this as a challenge (65%, 72 out of 111). Table 17: Teachers indicating a difficult curriculum practice given COVID-19 Was there a curriculum practice that was difficult to implement given COVID-19? (n=311) Coaching SMT LTSM TOTAL Yes 72 (65%) 79 (74%) 71 (76%) 222 (71%) No 39 26 22 89 Total 111 107 93 311 Teachers’ opinions of learner attendance in Term 4 indicates that 82 percent (255 out of 311) think that the maximum number of learners were back at school, whereas 18 percent (56 out of 311) indicated that not all learners returned to class. There was fairly consistent reporting across Grade 1 to 3 of learners not returning to class (84%; 82% and 89% respectively, n=311). However, across districts the teachers in the Dr. Kenneth Kaunda reported a higher frequency of learners not attending class (22%, 25 out of 115) than Ngaka Modiri Molema teachers (16%, 31 out of 196). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 54 4.4 HOW DID SCHOOLS RESPOND TO COVID-19 AND ARE THERE DIFFERENCES ACROSS THE INTERVENTION TYPES? Principals and SMT survey respondents indicated the following challenges in managing teaching and learning during the COVID-19 pandemic and schools closures (the open-ended responses were coded by theme): • Teacher absenteeism • Psychosocial issues such as. anxiety, depression, fear and a low morale • Lack of parental involvement or not being able to communicate with parents • Inadequate communication channels with parents • Learner drop-out as some learners did not return to school • Vandalism and theft of school property The following challenges during Term 4 were noted by the SMT: • Weekly rotation meant that the syllabus/curriculum could not be covered • PPE made it difficult for learners to communicate • Insufficient human resources to manage the additional smaller classes • Infrastructure limitations (including toilets, classrooms) to adhere to health protocols • Learner absenteeism • Comorbidities among the school staff • Non-compliance with COVID-19 regulations for example, not wearing mask The SMT responses to the survey indicated that the main strategy to deal with these challenges was to communicate to teachers about COVID-19 and provide PPEs to teachers upon returning to school. Most SMT respondents (182 of the 204 or 89%) reported that they put in place strategies to support teachers with coping with the pandemic. Twelve SMT members indicated they did not put any strategies in place. The most frequently cited responses were providing information to teachers on how to prevent infection, safety at the schools, and information on COVID-19 – this was provided either through SMT run-workshops or WhatsApp messages. School safety and following health guidelines were implemented in schools. Teachers were encouraged to stay home during school closure, and support was provided for teachers who were either at risk of infection due to comorbidities or if they were tested positive for COVID-19. Briefings were provided to teachers on the return to school and platooning or rotation of teachers. Counselling was provided to teachers when necessary, and the district support services were brought in when required from education and health departments. A few SMT respondents (6) reported communicating with parents and ensuring learners had access to LTSM or orientating parents and learners. One response noted that the home environment affected learner’s ability to engage with learning material at home and therefore did not reach out to learners. On the other hand, one respondent indicated that they encouraged learners to read aloud to their peers. Only one respondent noted making available ‘easy and accessible worksheets for learners’ (SMT member). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 55 One Principal indicated sharing the ATPs and sitting with teachers “to discuss the way forward”, and another held a meeting to discuss how to teach during the period and putting strategies in place to cope. Only one SMT member indicated teaching teachers how to teach online. Another SMT member encouraged teachers to “go the extra mile by attending during weekends to catch up lost time”, while another indicated that “team work was established and subjects shared to minimize contact and respect COVID-19 regulations”. The majority of schools did not implement remote learning. Only 17 percent (35 out of 204) respondents in the SMT survey indicated that the school put in place remote learning strategies during school closures. These included the following38: • Nine reported sending WhatsApp messages to parents with activities for their children, one indicated sending a letter to parents, and seven reported sending activities to parents but did not indicate how they sent them • Four reported sending activities to be done in the workbooks or worksheets • Two indicated they encouraged learners to read or listen to the radio • One indicated sharing a home activity timetable with parents • Nine responses referred to the approach of teaching upon school reopening, rather than during the school closure period. The following quotes by SMT members illustrate three approaches used to engage parents in their child’s learning: Learner activities with dates for each activity. And supervision from parents was monitored by checking the date at which parents assisted their kids and appended their signature.” (Principal, Female) “Learners were given work to do at home. Learners whose parents opted to teach their children at home were given exercises and other assessment tasks. When done they would to submit at school.” (Principal, Female) “Messages were sent to parents to come and collect the worksheet so as to ensure that learners are continually learning. But this process did not succeed as most parents were not interested.” (Principal, Female) The school functionality observations indicate that schools put in place a number of protocols and measures in response to COVID-19: Learners and teachers were monitored on entry and exit in adherence with COVID-19 protocols. This was the case in all (100%) SMT and LTSM schools, with most coaching schools (95%) monitoring their learners (95%). In all (100%) Coaching and LTSM school teachers were monitored on entry/exit, while this happened in 90 percent of SMT schools. All schools had hand sanitizers available for use. In fact, 50 schools (86%) had the sanitizer available for use in two or three areas: 95 percent of schools had it available at the 38 Coded analysis of qualitative responses was conducted Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 56 reception area, 91 percent had it in the principal’s office, 81 percent had in the classrooms and 10 percent had it available near toilets. According to observations, all schools were providing food to learners in Foundation Phase (Grade R-Grade 4). Most schools provide breakfast (92%) and lunch (95%), while all schools provide a mid-morning snack. Most (90%) of schools provide a variety of food groups including (carbohydrates, fruits, vegetables and protein). Only five schools (9%) had food preparation areas that were inadequate and six schools (10%) were not clean. The classroom observations indicate that 204 (95%) of classrooms followed relevant COVID-19 precautions. Ten (5%) classrooms from four different schools did not. • Out of 54 teachers observed, only four teachers were not wearing masks. Two of these schools were coaching schools. However, 12 teachers were not wearing masks properly, mostly from coaching schools. • Most (85% of 204) observations reported all learners were wearing masks, but of these 50 percent of observations indicated children were wearing masks properly. In addition, in 204 (95%) classrooms learners were wearing cloth masks and in only six percent of 204 classrooms there were one or more learners wearing disposable masks. • In 79 percent of 204 classrooms that followed COVID-19 regulations, desks were spaced out and 93 percent of learners were not sharing desks. • Observations showed that in 87 percent of observations, learners would sanitize or wash hands before entering the classroom and in 63 percent of classrooms learners were encouraged not to share materials. • In terms of maintaining social distancing, observations showed that there was only 50 percent compliance when learners were doing group work, but 78 percent compliance from teachers when interacting with learners. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 57 SECTION 5: HOW COULD THE INTERVENTION BE IMPROVED? 5. HOW CAN THE MANAGEMENT AND IMPLEMENTATION STRUCTURES BE IMPROVED? (QUESTION 6) The Formative Evaluation in 2019 identified the following management and implementation structures that needed improvement: (1) ensure the proper dosage and coverage is delivered, (2) track implementation and flag issues as they arise, (3) ensure quality of delivery and (4) facilitate positive response to the RSP by school management and teachers. The Formative Evaluation Report detailed how management and implementation structures can be improved across: LTSM delivery and quality, roles and responsibilities of different stakeholders, reception of RSP in schools (and with the districts and province), and SBWs. In response, FPD developed an RSP Improvement Plan for 2020 to address the concerns raised in the Formative Report. The plan focused on strengthening 17 areas to increase fidelity of implementation, improve the monitoring and reporting, increase accountability and improve the quality of the program. Within the plan, responsibilities for the improvements were defined across the Consortium partners and stakeholders (for example, UNICEF provided additional coach training). To answer this question the evaluation team triangulated information from a variety of sources including: - Interviews with coaches. - Interviews with head coaches - Interviews with 6 district and provincial officials - Interviews FPD, Molteno OUP and SMT mentor - RSP Improvement Plan 2019 and 2020 - RSP minutes of meetings - Data Quality Assessment Data (2020) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 58 5.1 THE EFFECT OF MANAGEMENT ON IMPLEMENTATION This Summative Evaluation shows that despite attempts to align the RSP design with EGRS I, there were many challenges which prevented it from successfully implementing the project with fidelity. Therefore, the RSP did not adhere to the ToC in terms of dosage, coverage or delivery of change at the schools in 2019. There was improvement in Term 1 in 2020 after the formative evaluation results were reviewed but this success was usurped by the COVID - 19 pandemic. During a forced COVID-related hiatus in Term 2 of 2020, the project unsuccessfully pivoted to an online model. When the RSP was up and running again in Terms 3 and 4 of 2020, the RSP had to deviate from the implementation plans and dosage, to help the teachers catch up and complete the curriculum. Delivering a technically challenging project at this scale, in a context of great uncertainty, required an implementation team with deep experience, a strong shared vision and the ability to innovate and adapt quickly. The management and implementation structures of the FPD consortium were constrained in various ways: The RSP had to pivot from the original design. This required changes in thinking about the project, and development of a new shared vision. The original RSP design focused on improving school management as the means to improve learner reading performance in another district. USAID and the DBE asked FPD to pivot in year two of their cooperative agreement, to use the remaining 3 years of their funding to experiment with scaling the EGRS I. FPD was reluctant to shift their initial plan to include the EGRS I elements, and one of the consortium partners, SAIDE, left during the negotiations with USAID. While USAID authorized the FPD consortium to begin work on the EGRS I scale-up, the negotiations took more than a year to finalize. It took even longer to pivot, and receive USAID approval for the revised MEL plan which was only approved in May 2020. One consortium member, VSO, was built into the modified RSP but struggled with visas for international volunteers, and eventually also withdrew from the FPD consortium. Some of the relevant experience to implement this kind of initiative at scale was scarce and had to be developed during the course of implementation. The consortium was led by the Foundation for Professional Development (FPD) which is a well-known USAID contractor in the health sector. This was one of FPD’s forays outside of the health sector into education. Despite their long track record in implementing projects at scale, some of the critical human resources deployed on this assignment required significant on-the-job training. A very specific blend of content knowledge, skills and contextual knowledge were required for implementation. For example, FPD could not find fourteen coaches with the exact right blend of skills and experience and had to develop the human resources as the project was implemented. A crisis like COVID-related school closures required the human resources to quickly adapt. Responding to such a crisis would be challenging to most organizations, however, the FPD consortium was constrained by the limited experience of some of project managers and their coaches, but also by inadequate monitoring and support from those responsible for their development. FPD institutional leadership appears to have left the RSP project leadership to manage on their own, and the RSP project could have drawn more on the organizational Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 59 strengths of FPD – for example, the monitoring systems for the RSP was found to be lacking despite the deep experience that FPD has in this area. Some of the relevant experience to implement this kind of initiative at scale was scarce and had to be assembled through a consortium with multiple layers of authority. It involved organizations with divergent organizational cultures. The consortium included three other organizations, two of which are prominent and well￾respected in education. Interviews with multiple sources indicate that the consortium did not function optimally. The consortium members had different technical expertise, but also differing organizational cultures. Molteno and OUP, with their expertise in HL and EFAL learning and teaching, excelled technically but were unable to innovate as they wanted to, because of the larger FPD project bureaucracy. Interview data show that the RSP management was unable to bridge differences in organizational culture to resolve issues efficiently. Such a consortium is expected to be somewhat unwieldly because it requires some duplications in project management and financial management roles. However, there are multiple examples of the bureaucracy within the FPD team being unnecessarily unwieldly, and contributing to inefficiencies: Project staff would pass messages or important communiques through intermediaries. For example, the budget negotiations were conducted with project staff who passed the message to the FPD finance staff, who then missed important nuances. Likewise, the coaches were not present at the formative evaluation report feedback session or part of the improvement plan session, leading to messages only being partially passed on to the them. A consequence of this ‘broken telephone’ was that the coaches did not really understand the revised RSP theory of change or implement the improvement plan adequately. In conclusion, there were too many mitigating factors to conclude that the RSP was, in fact, an example of scaling EGRS I. It provides lessons that the DBE can use to scale up future reading improvement projects. This is further discussed in the scaling up section of this report. 5.2 MANAGEMENT IMPROVEMENTS NOTED IN 2020 The FPD Consortium partners interviewed noted that the 2019 Formative Evaluation had assisted in improving the management and implementation of the project for 2020. However, due to COVID-19 further adaptations had to be made (refer to Question 1 of this report). The RSP Consortium partners and DBE officials stated that there was an improvement in the working relationship between them. In 2019 little collaboration between the district officials and the RSP was noted. However, in 2020 the FPD Consortium partners indicated they were able to work closely with DBE. This was reiterated by the Subject Advisors. “What I have noticed is that the subject advisors have been involved more this year in the project. The DBE and FPD are working better together.” (Subject Advisor) The DBE and FPD directed and focused approach to include the CA/SAs in 2020 was effective in terms of ownership by the District DBE: the CA/SAs were seen to be helping with Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 60 the implementation of the project at schools and with the training of teachers as there was an increase in their participation compared to 2019. In turn one of the national DBE representatives stated buy-in from the district was evident and the district more committed to carrying out the RSP in 2020. A success of the implementation and management of the project was the positive and accepting reception of the program at schools especially in 2020. A FPD Consortium partner noted that buy-in was evident at school level, as the RSP project was well-received by the teachers. “…At the moment, you can see that teachers are happy. And when we receive their report […] it was very positive” (FPD Consortium partner) The Coaches further reported that teachers are stating that the RSP program was helping teachers improve their teaching practices. 5.3 AREAS THAT REQUIRED ATTENTION (2020) There were a number of weak management and implementation structures and processes which required further attention during 2020. Overall RSP implementation Implementation of the program was still seen to be weak by DBE national and two of the three consortium partners, particularly in the coaching intervention and targeting teachers for consistent attendance at the training. A national DBE respondent indicated that some of the coaches selected for the project lacked the required skills and knowledge. This was corroborated by the Subject Advisors. “It [the coaches’ dry runs of teacher training] was an eye opener. The trainers were a little shocked that not all the coaches had an in-depth understanding and some coaches were presenting the wrong content” (EFAL Subject Advisor) Alongside these issues with some of the coaches, teacher catch up trainings were problematic. One of the DBE respondents stated that the lack of clear processes as to how teachers that missed a particular training will catch up and be supported in the RSP design, led to confusion of how these catch up training should be implemented. As evident in the FPD and DQA monitoring data not all schools had an SBW, and attendance was low. In addition, particularly in 2019, this was noted to be due to the way the RSP sent out invitations for trainings as several instances were reported where teachers did not attend due to an invitation mix-up (DBE respondent). Although the sending of invitations improved in 2020, individual teachers were not tracked or follow-up conducted by FPD with school managers to ensure that the teachers attended regularly. Consortium management structure and accountability The RSP structure was designed to be efficient on paper but the structure had many layers or levels to it. While one of the FPD Consortium partners stated that the consortium was set up in an effective way, another FPD Consortium partner disagreed and stated that while this was the case on paper, in practice this was not always evident. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 61 Implementation of the project was affected by team members in the consortium not carrying out their allocated tasks efficiently or not done at all, which at times resulted in delays or inefficiencies. A FPD Consortium partner indicated that when assessing the structure and individual members, capacity gaps in skills became evident. While skills development trainings were offered to the head coaches and the coaches, this did not seem to be sufficient according to the same FPD Consortium partner. Managing absenteeism The absenteeism of teachers and learners from school affected fidelity of implementation particularly in Term 4 of 2020. Even when coaches were present and had confirmed school visits with teachers, teachers would often not show up on the agreed day, impacting on support delivery. “We were not able to carry out what we had planned for 2020” (Coach 11) In cases where learners were absent, teachers reported low parental support in trying to assist the learners while at home to ensure they remained up-to-date. In addition, teachers found it difficult upon return to school to bring learners up to date with the curriculum. The management and implementation structures were not setup to address absenteeism. There was varying success by coaches to address this situation at the schools or in their planning. If the SMT intervention was fully integrated into the RSP design and implementation, the assumption is that there would have been stronger SMT support to ensure teachers attended the training or been present on days of planned coach visits. Even without SMT support, the coaches and FPD provincial and national staff could have engaged with school Principals to gain their support for the same teachers to attend the training or coach visits. Managing the SMT intervention The SMT intervention was insufficiently incorporated into the full RSP implementation practice and therefore with the withdrawal of the partner the intervention was delayed. The provincial and district department officials stated that the SMT component of the RSP could have been implemented more efficiently. The district officials reported that in 2020 the SMT members had been trained on their involvement in the RSP intervention and that they received resources. The officials suggested that there could have been more training to ensure the SMT members were equipped to provide better support to the teachers in the absence of coaches and Subject Advisors. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 62 6. HOW COULD THE M&E SYSTEMS BE IMPROVED? (QUESTION 7) In 2019, the Formative Evaluation found that the RSP M&E systems needed to be improved. It recommended adjustments to indicator design, tool design, data collection and management, and reporting. The Formative Evaluation recommended that the Monitoring, Evaluation, and Learning (MEL) plan needed to reflect the final iteration of the RSP design. Together, these improvements were deemed necessary to produce reliable data for tracking implementation in 2020. 2019 Recommendations from Formative Evaluation included: Data Management System and processes – The evaluation recommended that a relational database be developed to resolve discrepancies in the number of units (like teachers or schools). It also recommended that the data management processes be strengthened through documenting the data flow, data quality control processes, version control processes, access control processes and the process of archiving hard copy data. The evaluation suggested that the RSP Standard Operating Procedures (SOPs) should be amended to improve the quality of data and delineate responsibilities for quality control at various levels. RSP MEL Plan - The evaluation recommended that the indicators, indicator specifications and targets in the RSP MEL Plan be expanded and updated to correspond with the May 39 The name of the data file is in brackets. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: FPD monitoring data: - School master list (Revised list of schools for FPD 28 Oct 2019)39 - Subject advisor attendance of Reference groups and teacher training (Subject Advisors Info) - Subject advisor attendance of coach training sessions (Attendance list provided) - Subject advisor tablets (SA Asset Loan Forms) - Teacher attendance of training in 2020 (Teacher Data_2020_Updated Jan2021) - Teacher attendance of training in 2019 and 2020 (USAID Template_Teacher Training_FPD) - Teacher support, summary of individual coaching sessions in 2020 (Khulisa Dataset_Updated) - Teacher support, 2020 SBW sessions per month (Teacher Data_2020_Updated Jan2021) - LTSM Term 1 and 4 of 2020 (2020 Term 1 and Term 4 LTSM) - LTSM term 2 and 3 of 2020 (LTSM_Consolidated_Khulisa) - Coach training (Coach Training_2019_Updated 2020) - SMT training attendance, tablets and POE (SMT PLC Attendance_PoE_Updated 2020) Khulisa datasets: - School DQA tool Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 63 2019 RSP design specification. The formative evaluation noted a disjuncture between the indicators and the program activities, and suggested that additional indicators be developed to track how many of the participants received the full dosage of RSP support. FPD Reporting - The evaluation found some inconsistencies in the reported figures and instances of contradictory reporting against targets. The evaluation team recommended that USAID and FPD work to identify and resolve such reporting issues. FPD Improvement Plan 2020: In response to the formative evaluation, the FPD developed an improvement plan and updated the M&E Plan. The improvement plan addressed the above M&E issues by indicating the following actions would be taken during 2020: a) “Urgently finalizing the Monitoring and Evaluation Plan based on the activity description as submitted for the current modification. M&E plan must meet USAID and DBE requirements. The focus was on strengthening the monitoring of the coaching intervention at teacher level”. (FPD Improvement Plan 2020) b) “Improved quality and integrity of data overall, including a review of all indicators and data sources. FPD prioritized the training of coaches on REDCap. Using REDCap, data will be collected and uploaded in real-time, and go through several verification processes before final approval”. (FPD Improvement Plan 2020) The RSP MEL Plan was amended and approved by USAID in May 2020. Although improvements are noted in the MEL Plan for 2020, data collection practices remained weak. Accurately tracking the implementation fidelity of a large￾scale reading intervention projects provides useful information for the identification of implementation challenges, and for adaptive management. To realize the potential of using monitoring data for adaptive management, an integrated monitoring system with real time, or near-real time, insights and reports are required. 6.1 WERE THE M&E SYSTEMS IMPROVED IN 2020? The 2020 Summative Evaluation team reviewed the updated MEL Plan 2020, monitoring data provided by FPD from REDCap, Excel Spreadsheets, tablet issue forms and some attendance lists. The investigation shows that some improvements were made in response to several 2019 Formative Evaluation recommendations. Does the RSP MEL Plan reflect the updated design? The MEL plan was updated to adequately reflect the revised design. Indicators to track important RSP components such as SBWs were added. Indicators and targets (where stated) were aligned with the design specification. The number of participating schools, teachers, SMT members, coaches and Curriculum / Subject Advisors were specifically tracked in reach indicators. The revised plan included both output and Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 64 outcome indicators, but no impact indicators. The SMT indicators were improved to better reflect the intended results of the SMT intervention. The revised MEL Plan included new indicators to track how many of the participants received the full dosage of RSP support. These were: (1) Percentage of teachers who attended the full training program, (2) Number of lesson observations conducted annually per teacher and (3) Average number of SBWs per school. One remaining area of improvement related to the tracking of the delivery of LTSM to teachers. No indicator was defined to track the number of teachers who received all LTSM, and no target was set. Do the indicators match the design specifications targets and USAID indicators? The revised indicators aligned well with the design specifications, and no inconsistencies were found. However, eleven RSP custom indicators did not specify any targets. Targets were documented as N/R (not reported) or TBD (to be determined) or N/A (not applicable). The evaluation found that the updated MEL Plan comprehensively described data sources and tools. The MEL plan adequately documented the purpose, responsibilities, frequency, due dates as well as the levels of reporting for all data sources. The SMT indicators were improved when compared to the 2019 MEL Plan. The revised indicators aligned better with the RSP SMT activities and intended results. Although most indicators are focused on reach, the revised MEL Plan includes indicators to track how many of the participants received the full dosage of RSP support (Table 18). The unit of analysis for indicators was better aligned with data sources, addressing one of the 2019 concerns. Although in some cases, the indicator denominator was not collected for example, indicators related to classrooms, rather than teachers or schools. Table 18: Revised Custom Indicators (2020) Custom Indicators Data Provided (1) Percentage of teachers who attended the full training program The updated MEL plan does not include targets for these 3 indicators. Targets are documented as N/R – not reported or TBD – (2) Number of lesson observations to be determined or N/A – not applicable conducted annually per teacher Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 65 Custom Indicators Data Provided (3) Average number of SBWs per school. No indicator is included to track how many teachers received all LTSM Does the MEL Plan for 2020 address the concerns raised in the Formative Evaluation Report? The updated MEL Plan of 2020 addressed the concerns raised in the formative evaluation of 2019. The plan was more closely aligned to the May 2019 RSP design specification. The plan included the important program components and revised indicators. The improvements are discussed below. Data Management Systems and Processes FPD responded to the data management concerns outlined in the 2019 Formative Evaluation Report, by committing to the implementation of the REDCap data management system. All new monitoring data was to be captured in REDCap, and some of the back￾dated data was also to be entered into REDCap. The completeness and consistency of the 2020 data sets were improved, however some challenges remained. FPD still maintained some data in Excel Spreadsheets. In some instances, the REDCap dataset was more complete, and in some instances the Excel dataset was more complete. Discrepancies between the two data sets were common. • Better consistency, completeness and data quality Data extracted from the REDCap system as well as the data captured in Excel were more consistent and complete than the data received in 2019. A few inconsistencies were found in the Excel data sets, for example, Education Management Information System (EMIS) number and school names did not match for a few records. • Data quality remained a constraint The evaluation team compared the FPD RSP monitoring data to the data collected through the DQAs in the 59 sample schools. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 66 Table 19: Summary of Data Quality Results Type of Data Errors Found in Data Quality Assessment Teacher training attendance 22% of records Term 1 LTSM 42% of records Term 2 LTSM 14% of records Term 3 LTSM 10% of records SBW 32% of records Limited discrepancy in training attendance data (Term 1 2020 teacher training): o The data in both datasets corresponded in 46 (78%) of the 59 schools visited o There was over-reporting in six schools and under-reporting in seven schools40 . Table 20: LTSM Distribution in Sample Schools LTSM Term 1 HL LTSM Term 1 EFAL LTSM Term 2 Term 3 Number of schools that received MORE LTSM than number of teachers at school 4 3 4 4 Number of schools that received LESS LTSM than number of teachers at school 21 22 4 1 Note: Terms 1 and 4 = 1,117 (73%) LTSM delivered during training and Terms 2 and 3 = 1,481 (97%) delivered directly by coaches to schools. • LTSM Term 1 2020: o All principals in the 59 schools reported that their teachers received the HL and EFAL LTSM for their schools. o There were no discrepancies between the datasets in 34 (58%) of the 59 schools. But in 25 (42%) schools there were discrepancies: • LTSM Terms 2 and 3: o In Terms 2 and 3 no face-to-face training was possible due to COVID-19. The HL and EFAL LTSM materials were delivered to the schools by the coaches to ensure the teachers have access. As the HL and EFAL LTSM were sent to the 40 DQA data was collected through conducting interviews with principals or SMT members. Where possible, data reported was verified through checking school logbooks. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 67 schools simultaneously, the exact same data discrepancies and similarities can be seen for the EFAL LTSM in Terms 2 and 3. o There were no discrepancies in the datasets in 51 (86%) out of the 59 schools. o In Term 3 far less discrepancies between HL and EFAL LTSM distribution were reported between the two datasets. No discrepancies between the FPD and DQA data were reported for 53 (90%) out of the 59 schools. SBWs were not accurately monitored in the system and the data was not used to ensure that the RSP would take corrective actions. For example, the Principals of the 59 sample schools were asked to indicate whether or not the coach held SBWs between January and November 2020. Twenty-one (36%) out of 59 schools indicated on the DQA that SBWs were held. However, the monitoring data indicates that 19 (32%) schools actually participated in SBWs during Term 1. However, there were discrepancies in records between the monitoring data and the DQA data in 19 schools (32%) – these were not necessarily the same 19 schools that attended the SBWs in Term 1. • Keeping records in REDCap and Excel The REDCap system was implemented at the start of 2020. For the summative evaluation, some but not all monitoring data were extracted from this relational database. For certain program components, for example, teacher training attendance, the REDCap dataset was not comprehensive and complete enough for use in the analysis. For example, coaches did not capture all teachers’ details and attendance of training sessions. In some instances, the FPD team entered data in the REDCap and in Excel. For example, the teacher training attendance data were captured in an Excel workbook with separate Excel sheets per coach as well as on REDCap. FPD indicated that the Excel data provided a more accurate and comprehensive view of teacher participation in training sessions. In the case of the individual support for teachers, data were extracted from REDCap and summarized for 2020. • 2019 data not in REDCap The 2019 data were neither uploaded nor captured in the REDCap system. For this analysis, the 2019 and 2020 data were merged to establish dosage across the full year of implementation. Merging data sets required a unique identifier which was, for the most part, included in data sets and nearly 100 percent complete for the 2020 data. • Missing data The 2019 Term 4 coaching data was not analyzed in the 2019 Formative Evaluation and was unavailable in electronic format. Computer hardware failure reportedly caused the loss of data (FPD interview). • Master lists In 2020, FPD maintained a school master list which was used to track when schools ceased to participate in the program. A similarly structured teacher master list was not successfully compiled. The list of teachers used in this analysis includes only those teachers who participated at least once in a program activity. Teachers who never participated were excluded in the calculations of program reach. In the absence of a master list of teachers, assumptions were made about the number of teachers per school to set targets. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 68 SECTION 6: INTERVENTION FINDINGS 7. WHAT EVIDENCE IS THERE OF RSP UPTAKE BY TEACHERS IN CLASS? (QUESTION 3) Evidence of uptake of RSP in classrooms can be determined primarily by examining the following practices: 7.1 Teacher practice in implementing RSP methodologies 7.2 Classroom management 7.3 Analysis of learner workbooks 7.4 Teacher use of RSP LTSM 7.5 Use of classroom libraries 7.6 District Officials’ view of RSP uptake in the Classroom The purpose of classroom observations was to determine teacher practice, use of and availability of LTSM and the extent to which there is evidence of RSP uptake in the classroom. The analysis of this observation data also seeks to determine differences between the LTSM, SMT and Coaching interventions. This evaluation question compares results from 2019 and 2020. Although, in some instances data is not directly comparable as the instruments were updated from 2019 based on lessons learned, and in order to answer additional questions such as those relating to the COVID-19 pandemic, fidelity and scale. It is important to note that some of the teachers were observed during assessment periods, and so their lesson content would have been different. Classroom observations were done (refer to Table 21) of 104 Grade 1 and 110 Grade three classrooms, from 59 schools, of which 65 were in coaching, 78 in SMT and 71 LTSM schools To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: - Classroom observations - Classroom library assessments - Assessments of learner workbooks - Teacher survey of 311 teachers - Interviews with 6 district / provincial officials Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 69 Table 21: Number of Lessons Observed by District, Grade and Language District Number of lessons observed Grade 1 Grade 3 Language HL EFAL HL EFAL Dr Kenneth Kaunda 48 11 11 13 13 Ngaka Modiri Molema 166 41 41 41 43 Total 214 52 52 54 56 However, without clear benchmarks or learner assessment data, the evaluators were unable to determine if the uptake of the RSP methodologies or practices, or use of LTSM in the classrooms was sufficient or adequate to bring about improvements in the learners’ early grade reading skills. The ratings therefore only relate to improvements between 2019 and 2020 data where possible. 7.1 TEACHER PRACTICE IN IMPLEMENTING RSP METHODOLOGIES The evaluators measured RSP uptake through a change in teacher practice and an improvement in the teaching of reading through the following criteria: a) Strategies utilized by teachers to help learners decipher text, b) Opportunities for reading and types of reading methodologies utilized in the classroom, c) Strategies to facilitate comprehension of text, d) The extent to which learners are writing, and e) Vocabulary development. a) Teaching strategies to decipher text The RSP methodology provides teachers with various strategies to help learners that are reading aloud and are struggling to decipher texts. These were compared across intervention groups, in both 2019 to 2020. These strategies were observed only when learners were given opportunities to read aloud. Learners were given an opportunity to read aloud in 150 of 214 (70%) lesson observations. Of these, reading aloud in 63 percent (41 out of 65) of coaching classrooms, in 77 percent (60 out of 78) of SMT classrooms and in 69 percent (49 out of 71) LTSM classrooms. See Annex 7 Table 1 for more detailed data. There were minimal differences between intervention groups: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 70 In the coaching schools, there was a slight improvement in strategies modelled by teachers to help a learner determine words (‘sounding out a word’, ‘guessing a word’). While there was a decrease in the SMT intervention school teachers’ practice, they outperformed coaching school teachers for both strategies in both 2019 and 2020. There was little variation or difference observed between these strategies when comparing HL and EFAL; except for the strategy of skipping a word and returning to it using context. This occurred in 15 percent of the observations in HL, and only six percent of the observations in EFAL. This is expected, since learners have more HL contextual vocabulary. This is a key strategy to help learners place words in context which contributes to their reading comprehension levels. b) Opportunities to read Another key measure of whether RSP methodology has been implemented is in actual learner reading. The classroom observations recorded the items listed in Annex 7 Table 2. All intervention groups performed in a similar range, although items were observed more frequently in the SMT intervention than in the coaching and LTSM groups. Items were only recorded if they were noted applicable to the actual lesson being observed (hence the denominator changes per item observed). While the SMT intervention had more observations of reading, the Coaching and LTSM intervention groups in shared reading, the overall results are low; with shared reading being observed in less than half of all observations across groups in 2020. Even less frequently observed were instances of learners being split into groups for reading; with this being observed in less than 40 percent of all observations across groups, and only being observed in 24 percent of LTSM intervention schools. This might be due to COVID-19 restrictions and social distancing. Evaluators observed coaches advising teachers on how shared reading could be managed, particularly as platooning means smaller classroom numbers. One coach suggested conducting group guided reading where learners were seated, instead of having learners move to a carpet. Learners could also be seated in a socially distant manner, due to reduced class size. Another teacher was observed conducting group guided reading after sanitizing all learners’ hands, and plastic covered books, before and after the guided reading activity. In over 60 percent of observations, learners were given an opportunity to read aloud, across intervention groups. This included class reading aloud following the teacher or individual guided reading practice. Although between the three groups, this was observed least among the coaching group (64% of observations). c) Strategies to encourage text comprehension While learners’ ability to decipher text is key, a significant aspect of the ability to read is interpreting and understanding texts – “read for meaning”. See Annex 7 Table 3 for the various strategies utilized in classrooms when extended texts (such as sentences or paragraphs) were part of the lesson plan and lesson observation. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 71 Whether or not teachers asked learners questions after reading was based on observations where reading aloud by learners occurred, and was evaluated as per the lesson plan. Extended texts were part of the lesson observation for 40 of the 65 coaching classroom observations (62%), 44 of the 78 SMT observations (56%) and 52 of the 71 LTSM observations (73%). One strategy to facilitate comprehension in learners is to ask learners questions about the text after reading it, which was observed in 86 percent of all observations (79 of 92) where reading aloud took place. This was observed most frequently in the coaching intervention observations in 89 percent (25 out of 28) of the time. Across all items observed, there was an increase from 2019 to 2020 for the coaching intervention. The LTSM intervention improved from 2019 across items observed, except for learners answering open ended questions which was observed in 65 percent of classrooms in 2019, and then 64 percent (33 out of 52) of classrooms in 2020, although this difference is marginal. Across all categories, either the coaching or the LTSM interventions performed better than the SMT intervention group. The coaching intervention performed better than the LTSM and SMT intervention in five items: teacher asking learners questions after reading, learners encouraged to discuss illustration, teachers’ use of illustrations to aid understanding and learners answering open ended questions. The LTSM intervention performed better than the SMT and Coaching in three items: learners encouraged to retell, act, summarize story, learners identify main theme of the text and learners answer literal, evaluative, appreciative and critical questions. The SMT intervention performed the poorest out of the three intervention groups across seven of the eight observed items. The SMT intervention also decreased from 2019 to 2020 in six of the eight items observed. The most substantial decreases (between 2019 and 2020 data) were observed in the following items: • “Learners encouraged to retell, act, summarize story” decreased from 57 percent (44 out of 77) in 2019 to 36 percent (16 out of 44) in 2020; • “Learners answer open ended questions” decreased from 73 percent (56 out of 77) to 55 percent (24 out of 44); • “Learners answer predictive/inferential questions” decreased from 62 percent (48 out of 77) to 16 percent (7 out of 44); and • “Learners answer literal, evaluative, appreciative and critical questions” decreased from 69 percent (53 out of 77) to 30 percent (13 out of 44). d) Writing In the Foundation Phase classes, children need to be taught handwriting which includes: how to hold a pencil or pen, form the letters correctly and efficiently with correct direction and spacing. In Grade 1, children learn how to write print and by the end of Grade 3, they should have been taught to write in cursive script. In addition to the mechanics of writing, it is also expected that children are able to formulate their own ideas in writing. This type of writing is Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 72 about the content and the organization of this content, and not about the technical aspects of forming letters. Teachers are expected to take corrective action if learners are holding their pens or pencils incorrectly. Taking corrective action was noted by fieldworkers only if learners were expected to complete written tasks during the lesson observed. Learners were expected to complete written tasks in 77 percent (50 out of 65) of the coaching observations, 73 percent (57 out of 78) of SMT observations and 80 percent (57 out of 71) LTSM observations. See Annex 7 Table 4 for more detail. There was an increase across all intervention groups between 2019 and 2020 of teachers correcting handwriting (See Annex 7 Table 4). The largest increase from 2019 to 2020 in observations of this corrective action was seen in the LTSM group, where teachers took corrective action in 17 percent of observations in 2019, but 70 percent of observations in 2020. Checking writing practice and corrective action was observed more often in the LTSM group when compared to the coaching and the SMT groups. It must be noted however, that ‘non-observation’ in this instance does not automatically translate to poor teaching practice. This type of correction may have occurred earlier than Term 4, and all learners may be using their writing implements correctly. In order to be competent readers and writers, it is important that learners progress beyond copying from the teacher writing on the board, and begin writing their own words and sentences. Across intervention groups, learners were less frequently observed drawing patterns in preparation for cursive writing when compared to 2019. See Annex 7 Table 5 to see the progression of the development of the skill of writing from learners drawing patterns, to learners writing full sentences. The largest decrease was seen in the LTSM group; which decreased from 14 percent (6 out of 42) to five percent (3 out of 57) in 2020. However, the LTSM group increased from 29 percent (12 out of 42) of observations where learners were writing their own vowels, letters or syllables to 46 percent (26 out of 57) showing greater progression to more advanced tasks. The LTSM group performed higher in terms of the teachers’ expectations of learners and writing full words. The LTSM group had the lowest number of observations where learners were not expected to write at all (39%, 22 out of 57), whereas in 66 percent of observations (33 out of 50) in the coaching intervention, teachers did not expect learners to write at all. The above, where in 66 percent of observations in the coaching intervention, learners were not writing at all, is a high percentage considering that handwriting ‘cuts across the curriculum’ and should be practiced four days a week41 in Grade 3. The coaching group also did not have any instances of observations where learners were writing ten or more words, 41 As per the RSP lesson plans for Term 4 reviewed by the evaluators Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 73 whereas this occurred in both the SMT and the LTSM observations (35%; 9 out of 26, and 6%; 2 out of 35 respectively). Across the three intervention groups, most learners in the classrooms observed fell into the category of writing three to ten words (See Annex 7 Table 6). Without clear benchmarks for each grade and HL and EFAL, it is unclear if this is sufficient for learners in Term 4. Due to the RSP intervention, learners should be writing. For example, while 54 percent of learners in SMT schools (27 out of 50) are expected to write sentences, this means that the remaining 46 percent are (23 out of 50) not expected to write at all 42 . Those writing one to two, three to five and five or more sentences make up that 54 percent (See Annex 7 Table 7). While the LTSM group has the most learners writing one to two sentences (16 out of 35), (the lowest category), holistically, most of the LTSM learners were engaged in some sentence writing. While the SMT intervention had more learners than the LTSM group writing at a higher volume (three to five sentences) (14 out of 26), they had the highest number of learners (54%, 31 out of 57) who were not engaged in any writing at all. What this reflects, is that even though overall the LTSM observation groups were “writing the least”, with 46 percent of learners writing one to two sentences (16 out of 35), and 43 percent of learners writing three to five sentences (15 out of 35), more learners are writing than in the other interventions. Further, there is no real difference between the SMT intervention (where 11 percent of learners are writing five or more sentences, 3 out of 26) compared to the LTSM intervention (where 11 percent of learners are writing five or more sentences, 4 out of 35). However, in the SMT and LTSM groups, learners were observed writing five or more sentences - over double the amount of observations in 2019. SMT observations of five or more sentences went down from 25 percent (10 out of 40), and LTSM interventions went down from 30 percent (11 out of 37). The coaching group was the only intervention group to gain in this regard, where learners writing five or more sentences in 2019 was only observed in eight percent of observations (4 out of 48), compared to 15 percent (4 out of 27) of observations in 2020. However, this is still lower than the observations of 2019 in the SMT and LTSM groups. Writing sentences is expected for both Grade 1 and Grade 3. Grade 3 learners are expected to be writing more full sentences, whereas Grade 1 learners are still learning new words and beginning to construct sentences. In the lessons observed, it was noted whether teachers provided all or most learners with writing activities such as writing words or sentences. This was observed in 164 lessons (50 in the coaching group, 57 in the SMT group and 57 in the LTSM group). In these lessons, the evaluators compared the extent to which learners had to write across intervention 42 This inverse cannot be applied to 2019 data, due to multiple categories (none, less than a quarter of the class, about half the class, etc.) of how many learners were expected to write in the classroom. Whereas 2020, the tool asked whether most/all learners were expected to write. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 74 groups, language and grade. For Grade 1 learners one compared the extent to which learners had to write words (See Annex 7 Table 8), while for Grade 3 learners one compared the extent to which learners had to write sentences (See Annex 7 Table 9). When comparing the Grade 1 lessons where teachers provided writing activities in 2020 (total of 27 lessons), most learners having to write more than ten words was only observed once. This was for a HL lesson in a SMT school (out of 5 HL lessons in the SMT group). In comparison, in Grade 3, learners across intervention groups were observed most frequently writing between three and five sentences. There was an increase in writing in Grade 3 EFAL compared to the Grade 1 data. This would be expected as in Grade 4 learners are typically expected to transition to English as the LOLT. The fact that none of the learners are writing one to two sentences in HL for the LTSM intervention in 2020 is a success, as most learners are writing three to five sentences (See Annex 7 Table 8). Grade 3 learners are also expected to learn how to write in cursive. The extent to which this is observed is compared for 2019 and 2020 across EFAL and HL and intervention groups in Annex 7 Table 10. The skill of writing in cursive is taught more in HL than EFAL for all intervention groups. Without a benchmark for clarity of the expectation, the evaluators are unable to determine a value judgement of sufficiency of observation. The LTSM group performed better than the coaching and the SMT interventions in how much cursive is taught overall, with overall observations of 28 percent compared to 12 percent in coaching and 16 percent in SMT. e) Evidence of vocabulary development Evidence of vocabulary development (such as learning new words or meanings) and the testing of spelling decreased from 2019 to 2020 for coaching and SMT interventions (Annex 7 Table 11). Evidence of testing spelling increased for the LTSM group from 2019 to 2020 (Annex 7 Table 11). However, evidence of vocabulary development decreased in LTSM observations. Vocabulary and spelling development were observed less in 2020, compared to 2019 for SMT and LTSM intervention groups. This decrease may be as a result of COVID-19 and the loss in teaching time. Some teachers indicated to the coaches (noted during coach shadowing) that they had struggled with the platooning model, as learners could not remember the content from the last day they had been at school, and had forgotten what they had learned earlier in the year, and therefore had to repeat lessons often. It is possible that teachers were then prioritizing phonics instead of spelling or vocabulary development. While there were no major differences in the development of vocabulary between Grade 1 and Grade 3 learners, this observed item recorded noticeable differences between EFAL and HL. Grade 3 learners should be doing more vocabulary and spelling development than Grade 1 learners. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 75 However, the differences between Grades was minimal. In 78 percent of HL classrooms there was vocabulary development in HL, but only in 69 percent of EFAL classes. There was evidence of spelling development in 68 percent of classrooms observed, but only in 55 percent of EFAL classrooms. 7.2 CLASSROOM MANAGEMENT Classroom management across interventions was generally of a high standard. This relates to learner discipline and whether learners are engaged in their work or are disruptive, how clean and orderly classes are, how much teaching time is wasted or slowed down, and finally how responsive teachers were to the COVID-19 pandemic, through their compliance to COVID-19 regulations. Due to the pandemic, some observed items were reprioritized, for example, compliance with COVID-19 protocols was observed. But for hygiene reasons; some items observed in 2019, were no longer a priority in 2020. For example, use of reading mats stopped due to social distancing requirements. In some classrooms, posters had not been displayed or they had been damaged due to classroom fumigation. The results of the classroom observation data for the question, “Are RSP posters on display”, did not request fieldworkers to provide an explanation. Therefore, the reason for why teachers might not have displayed the items, cannot be determined if this was due to a lack of RSP uptake, or whether this was due to classroom fumigation43 . Classroom environment The majority of classrooms were clean and teachers could move around comfortably and walk easily between all the desks. The SMT intervention had the most classrooms which were clean (99%), followed by LTSM (96%) and coaching (91%). The teacher could move around easily in the classrooms in 94 percent of LTSM classrooms and in 92 percent of SMT and coaching classrooms. Classroom organization In a well-managed classroom, the teacher would ideally use classroom monitors to hand out books or stationary, would have familiarized herself with the lesson plan beforehand and the class would be arranged in a way that minimizes disruptions. Time wasting practices were observed to either happen “Not at all”, “Hardly at all”, “To some extent” or “To a large extent”. Across intervention groups, these time wasting practices occurred “Not at all” most of the time: • Handing out of workbooks or other materials did not disrupt lessons in only 28 (59 out of 214) percent of observations, this caused a disruption “To a large extent” in five percent (11 out of 214) observations. 43 Observed during coach shadowing, classroom observations and informal discussions with teachers. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 76 • The teacher did not need to consult her notes in 43 percent (92 out of 214) of observations, and the teacher consulting her notes happened “To a large extent” in one percent (3 out of 214) of observations. • Classroom furniture did not need to be rearranged in 53 percent (113 of 214) of observations, and occurred “To a large extent” in one percent (2 out of 214) observations. • The lesson was not disrupted by learners having incorrect stationary in 45 percent (96 out of 214) of classrooms, in only one lesson did this happen “to a large extent”. These time-wasting practices were also observed least among the LTSM intervention group, when compared to the coaching and the SMT intervention groups. In 68 percent of all observations (146 out of 214), there were no outside interruptions to teaching in the classroom; and when there were interruptions, they tended to be only one or two times per lesson. LTSM schools had the least outside interruptions, with 75 percent (53 out of 71) of classes uninterrupted, followed by coaching schools where 71 percent (46 out of 68) of classes were uninterrupted, while 60 percent (47 out of 78) of SMT classes were uninterrupted by outside disturbances. Learner management The evaluators observed the teacher’s management of the learners and their written tasks. This was observed through the extent to which the teacher differentiated support; for example, learners are given independent work, and how the teacher supervises that all learners are engaged and doing work, and when the learners have completed their tasks, where stronger learners might be given additional tasks. The teacher should be attuned to their learners, so that they are all engaged in learning and do not distract other learners. In general teachers provided independent tasks to learners across all intervention groups, with coaching teachers performing marginally higher. Teachers from LTSM schools performed better in managing their classrooms through providing independent work and tasks, supervising learners and keeping learners engaged with written and reading work (Annex 7 Table 12). 7.3 LEARNERS’ BOOKS The EGRS Wave 3 Report (2018, p.25) notes that one of the most important national interventions over the last few years has been the provision of the “DBE Workbooks”, which are a hybrid between a textbook and an exercise book, with numerous exercises for learners to independently complete. The lesson plans provided through both the EGRS and RSP incorporate the DBE Workbooks into the daily lessons referring to specific page numbers for exercises to complete. In addition, learners complete work in their own exercise books. One would expect that RSP teachers engage in better practices or at least have a better knowledge of good instructional practices in the teaching of reading compared with teachers who have never been exposed to the intervention. Therefore, analyzing data from work completed in learner workbook and exercise books yields suggestive evidence of changes in practice and as a result, confirm teacher uptake of the RSP. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 77 In particular, a workbook and exercise book analysis is indicative of children being provided with opportunities to write on a regular basis, since writing is essential and supports the acquisition of reading. However, the CAPS for Foundation Phase requirements are unclear on the writing benchmark. It is not articulated in CAPS that Grade 3 learners should be doing more writing than Grade 1 learners. This section considers whether teachers’ use of the RSP lesson plans and thus more structured planning of time for learning, is accompanied by increased completion of work in learners’ DBE workbooks and exercise books. Teachers were asked to choose one learner whose workbooks and exercise books could be evaluated by fieldworkers for Grades 1 and 3 (from the classrooms that were observed) and EFAL and HL per Grade. For the analysis, fieldworkers looked at DBE workbooks from Term 4 (or whatever term books the teacher was using to teach), and all learner exercise books for the learner for the year. Therefore, the results presented in this section are an indication of learner’s writing, but with some caveats: • For the RSP, lesson plans integrated the use of DBE workbooks in specific tasks, while most of the opportunities to write were provided in the learner exercise books or loose pieces of paper. • Findings may have been influenced by the COVID-19 pandemic school closures and the revised ATPs that do not require completing DBE workbooks as with the curriculum prior to COVID-19. In 2020, only 27 (45%) of 59 schools appeared to be using solely the Term 4 workbooks for teaching (110 of the DBE workbooks analyzed in 27 schools were only Term 4 workbook). In 33 schools, both Term 3 and Term 4 books were analyzed. This may provide an indication of the extent to which teachers need to catch up the curriculum given the COVID-19 pandemic effects on teaching and learning. DBE workbook assessment data for 2020 show, on average, learners from LTSM schools completed more pages in the DBE workbooks with any exercise and with writing exercises (an average of 14.25 pages and 11.54 pages respectively) compared to the other intervention groups. But there was no statistically significant difference between the groups (p = 0.295 and 0.236 respectively). In addition, this was still lower than what was observed in 2019 (on average a minimum of 25 pages and 17 pages completed respectively for all intervention groups). Nonetheless, all groups had similar average days spent on any exercise and with writing exercises since the beginning of 2020: on average more than 25 days were completed across all groups, but the coaching group had the most pages completed on average (29.47 and 26.76 pages respectively). The classroom observation data presented earlier in this report is aligned with the similarities across intervention groups with respect to the extent to which ‘most learners’ and ‘all learners’ were engaged with written work during the lesson – with all intervention groups being very similar. When analyzing (Figure 9) the number of pages completed in DBE workbooks across all three intervention groups, more pages involving exercises comprising the writing of at least Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 78 one full sentence, the data shows that on average all three groups had mostly between 15 and 19 pages completed. Figure 9: Pages in DBE workbook with writing at least one full sentence The evidence further (Figure 10) shows that learners are using their exercise books, with many using it more than 20 days in any activities/exercises and, in particular, with at least one writing exercise. Figure 10: Days with at least one writing exercise completed in exercise books When compared across both grades, there was a statistically significant difference in the writing of at least one full sentence (p <0.001). Learners in Grade 1 mostly spent zero to nine days writing one full sentence, while Grade 3 learners mostly spent between five and twenty or more days writing one full sentence. In fact, 50 percent of Grade 3 learner workbooks analyzed had completed more than 15 days of work that involved writing one full sentence. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 79 Table 22: Days with at least one full written sentence in learner exercise book Number of days None 1 - 4 5 - 9 10 - 14 15 - 19 20+ Grade 1 11 (11%) 39 (38%) 36 (35%) 7 (7%) 2 (2%) 9 (9%) Grade 3 1 (1%) 5 (5%) 16 (15%) 32 (29%) 22 (20%) 33 (30%) Total 12 44 52 39 24 42 213 Note: 2 learner workbook assessments did not contain information on the grade and therefore were removed from the analysis Both the DBE workbook analysis and the learner exercise book analysis showed that learners had not been provided with many opportunities to write full paragraphs, as can be seen from the visuals below. Over half of the DBE workbooks did not have any pages completed with at least one written paragraph. Figure 11: Pages in DBE workbook with writing of at least one paragraph Likewise, over 45 percent of learner exercise books did not have any days where learners wrote at least one paragraph. Figure 12: Days with writing of at least one paragraph in learner exercise book Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 80 In 2019, the DBE workbook assessments showed statistically significant (p = 0.028) difference between the intervention groups in terms of the amount of written material observed (see Figure 13 below). In particular, SMT schools had on average more pages complete with a writing activity involving at least one full sentence. However, in 2020 there was a large decrease in number of pages completed for the same type of exercise (see Figure 13 below). Interestingly, the LTSM group had, on average, completed more pages compared to the other groups. However, the number of pages in the DBE workbooks involving writing of at least one full sentence was not significantly different (p = 0.180) across the groups. Figure 13: Average pages with at least one full sentence across intervention groups In 2019, in the analysis of the DBE workbooks for Term 4 showed that there was more writing in Setswana HL than EFAL. The difference between both languages decreased in the same Term 4 period for 2020, as can be seen in the Figure 14 below. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 81 Figure 14: Average pages with written work per language The findings presented in this section appear to be aligned with findings from the EGRS I, where the results relating to coverage of work in DBE workbook suggest less work in DBE workbooks in intervention schools compared with control schools (EGRS I, 2019, p.55). Similar to the EGRS I, teachers exposed to the RSP may be using exercise books more than DBE workbooks. This may be the case especially since the RSP lesson plans provide guidance on how to use the DBE workbook for specific tasks related to their lessons, rather than completing it in full and sequentially. For the full analysis and visuals of the workbook data see Annex 10. 7.4 LTSM AVAILABILITY AND USE All teachers that were part of the RSP received a package of LTSM and were trained on how to use them. The package included scripted lesson plans, posters, flashcards, graded readers, among others. For the full list of LTSM provided to teachers see Annex 11. Overall, in 95 percent of 211 lessons observed, the teachers had access to and were using the RSP Lesson Plans. Table 23: RSP Lesson Plans used Coaching SMT LTSM RSP Lesson Plans used in Observed Lesson 31 39 33 Total Lessons Observed 33 39 36 Percentage of teachers using Lesson Plans 94% 100% 92% Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 82 Table 24: Lesson Plans used by grade RSP Lesson Plans used in Observed Lesson Total Lessons Observed Percentage of teachers using RSP Lesson Plans Gr 1 HL 45 50 90% Gr 3 HL 53 55 96% Total HL 98 105 93% Gr 1 EFAL 48 52 92% Gr 3 EFAL 55 56 98% Total EFAL 103 108 95% There was no large difference reported between teachers in the different intervention groups and how useful they found the LTSM. There were no observable differences between teachers in different grades, or different districts in how useful they found the RSP LTSM. Three quarters of teachers surveyed (78%) found some of the LTSM useful, and 21 percent of teachers found all the LTSM useful. • Most teachers (81%) said the scripted lesson plans were the most useful HL resource, followed by 11 percent of teachers who found the Theme Vocabulary Flashcards (HL) the most useful teaching resource. • Half of the teachers (54%) felt the posters for listening and speaking the most useful resource overall, followed by 20 percent of teachers who found the curriculum tracker the most useful. • Equal numbers of teachers (11%) felt the handwriting poster was the most useful teaching resource, and the classroom library to be the most useful teaching resource. For both HL and EFAL the themed vocabulary flashcards were used more frequently by the SMT and LTSM teachers than the coaching teachers, and these were reported as being used every day. Further analysis is included in Annex 7, Figure 2. LTSM use was based on self-reporting in the teacher survey. Annex 7, Figure 13 sets out the details of all LTSM and frequency of daily use. However, the most teachers reported that they used their lessons plans every day across interventions, with 79 percent of teachers reporting they used HL Lesson plans every day, and 78 percent of teachers reporting they used EFAL lesson plans every day. SMT teachers self-reported more lesson plan usage than other interventions. Other resources were used less, with the least used resource being the curriculum tracker with only 17 percent of teachers using it every day, and 27 percent of teachers reporting that they used it once a week. RSP LTSM were observed as available44 in the classroom and readily available for the teacher to use in 51 percent of coaching classes, 55 percent of SMT classes and 56 percent of LTSM classrooms. EFAL and HL materials were generally visible on classroom walls (upwards of 60 percent of all interventions), as seen in the figures in Annex 7. 44 This excluded the Lesson Plans. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 83 Figure 15: RSP materials and their availability in the classroom Teacher management and storage files were poorly used, with teachers either having them not at all, or with the files being there but not containing any resources. Files did not contain any resources in 61 percent of LTSM classrooms, 55 percent of coaching classrooms and 53 percent of SMT classrooms. Figure 16: Teacher Management and Storage File Graded HL readers were available in • 59% of coaching classrooms • 67% of SMT classrooms • 71% of LTSM classrooms EFAL Big Books and Graded EFAL readers (New Heights) were available in Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 84 • 47% and 49% of coaching classrooms • 26% and 56% of SMT classrooms • 39% and 49% of LTSM classrooms respectively. This illustrates room for further dissemination of EFAL resources. Availability of alphabet freezes and flashcards were also assessed, refer to Annex 7 for further details. 7.5 CLASSROOM LIBRARIES Sixty-three of the 214 classes observed (29%) had received the classroom library intervention. Overall, in 91 percent of these classrooms, there was a reading corner/box library or book collection equivalent present in the classroom. It is important for learners to have access to a range of different texts to read so that they can become confident and independent readers. However, the availability of story books was considered “poor” where no story books were available in 25 percent of coaching classes assessed (4 out of 16), 12 percent of SMT classrooms (3 out of 25), and only five percent (1 out of 22) LTSM classes. This meant that learners had a “classroom library” that consisted only of picture books. In 56 percent of classrooms, more than 20 story books were available. This was observed more in SMT classrooms where 60 percent of the classes had more than 20 story books available, but only in 55 percent of LTSM classes and in 50 percent of coaching classes. English books were available in 64 percent of classes, and Setswana books were available in 89 percent of classrooms. In two classrooms, books were available in other languages as well. Books were assessed to determine whether they were “well-preserved”, “adequately displayed/stored” or “tatty, and poorly displayed/stored”. In 18 percent of classrooms books were considered to be tatty and poorly displayed or stored. While this suggests that the books are well-used by learners, this does also suggest that the readers are not durable. During some coaching observations, coaches had suggested to teachers to use clear plastic tape along the spines of the books to increase durability. However, in the majority of classrooms, books were found to be either well-preserved (35%) or adequately displayed and stored (48%). While the provision of adequate reading material is paramount, the classroom library intervention goes beyond the provision of books and includes a book management chart that is to be filled out, and the use of reading cards to track the reading of the learners. In 64 percent of classrooms this management chart was available, observed least in the SMT classrooms (60%), followed by the LTSM classrooms (64%) and then observed most in the coaching classrooms (68%). However, the progress chart was not filled out correctly in 52 percent of total observations, indicating that while available, it is not necessarily used in many classrooms. Figure 17: Classes with correctly completed book progress or management chart Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 85 The classrooms that filled in the chart the most were LTSM classrooms where 68 percent of classrooms had filled the chart in. However, in 56 percent of SMT classrooms, and 75 percent of coaching classrooms, the chart had not been filled in. Reading cards appeared to be utilized even less, with reading cards visible and available in only 29 percent of all classrooms. Although, in 83 percent of all classrooms with classroom libraries, it seemed that books were made available for regular use by learners in the classroom. This was seen most in SMT classrooms (88%), followed by coaching classrooms (81%) and then LTSM classrooms (77%). Overall this indicates that, even if the full classroom library intervention is not being implemented, the books still appear to be utilized. 7.6 DISTRICT OFFICIALS’ VIEW OF RSP UPTAKE IN THE CLASSROOM Five of the six district officials interviewed reported that there was evidence of RSP practices being implemented in class by teachers. The sixth district official indicated that they were unsure. Two CA/SAs said they could see that the lesson plans were being used and that the teachers were filling them in and were prepared for their lessons. Two CA/SAs referenced the reading charts of the classroom libraries, and that they were being used. Three CA/SAs referred to the difference in the learner’s books and the writing activities being done by the learners. “Yes, definitely, I can see that (evidence of RSP practices in classrooms) in school visits, and it is amazing… When you take a learner’s book…You can really see the strategies that the teachers are using in relation to a school that’s not on RSP. And if you think you know [name of another reading intervention]; it is the same, and the schools are, that is on [name of another reading intervention], they are not doing that, and it is the same method. [This might be because] the way in which the RSP lesson plans are written, is much more, easier for the teachers to understand…It is much more user friendly, and it guides you…Even with the overview for the week, it is assisting the teachers.” (Subject Advisor, 2020) Five of the six district officials indicated that there had been an improvement of the implementation in the RSP in the classrooms when compared to 2019, despite a shorter teaching year. The officials attributed this to the teachers being more confident with the materials, their knowledge had grown, and that the teachers were now more on board with the RSP as they now saw the value of the program. Two CA/SAs specifically noted that the program should have been extended and that if it were, even more improvement would be seen. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 86 8. IS THERE EVIDENCE OF CHANGED PRACTICES BASED ON SMT SUPPORT? (QUESTION 4) While in 2019 SMT classrooms provided more opportunities for language and literacy development than the other groups, in 2020 this was not the case. In fact, SMT classrooms were recorded to perform worse than the other intervention groups. Nonetheless, qualitative evidence from the teacher surveys indicates that SMT intervention members and teachers valued the SMT training intervention as they reported that improved HODs support for teachers in implementing RSP the classrooms. Despite perceptions of SMT intervention focusing mostly on leadership skills instead of how to better support early grade reading in schools, a review of the training materials indicates that the PLCs focused on the latter. In fact, the training was focused on capacitating principals and SMT members with “knowledge, skills and attitudes relevant to school management and literacy teaching in order to plan better and devise strategies to support the cultivation of a reading culture in their schools.” (RSP SMT report, 2020) As mentioned previously, the SMT intervention was not implemented with the expected fidelity, with only 21 (16%) SMT members receiving the full intervention. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: - SMT survey - Teacher survey - KIIs with FPD staff, DBE officials and SMT mentor - FPD monitoring data - Project Documentation Main Themes for SMT PLCs / workshops: “Cultivating Leadership SMT members for Reading Literacy Improvement”, divided into 5 sessions: Workshop 1: What is the reading challenge at your school? (May 2019) Workshop 2: What SMT Leadership is required for Literacy Improvement at your school? (July 2019) Workshop 3: What is your Leadership SMTs Vision/Mission for your schools’ Reading Literacy improvement? (September 2019) Workshop 4: How are you resolving the Reading Literacy Challenge at your school through SMT Leadership? (February 2020) Workshop 5: Consolidative activities (November 2020) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 87 It thus is not surprising that evidence from the classroom observations in Section 4.3 showed that there is little to no difference between the uptake of the RSP by teachers in the schools that had the SMT intervention, and when it came to strategies to facilitate meaning in text they in fact had performed not only worse than the other groups but worse than 2019 performance. Despite the above, some SMT members reported changing their behavior, noticing teachers change their behaviors in classrooms, and teachers report feeling supported by their SMT. More detail on this is provided in this section. 8.1 SMT CHANGE IN BEHAVIOR AS A RESULT OF SMT INTERVENTION According to the RSP ToC, SMT members, through participating in PLCs and receiving tablets which would give them access to both online and offline LTSM resources (consisting of graded readers and other supplementary reading materials, links to online resources and good practice videos), then they were expected to improve their competencies (school leadership) and in turn lead to more effective school leadership and management. The evaluation collected self-reported data on the use of tablets and on changes in leadership/management in schools. According to the Principal/SMT survey, 29 (45%) of 64 survey respondents in the 59 sampled schools reported receiving tablets. In turn, nine respondents said they did not receive any tablets, two said this question was not applicable to them and 24 did not respond to this question. The monitoring data in turn confirms that 111 (81%) of 137 SMT members received tablets, as discussed in Section 4.2. When asked how they used the tablets, 29 SMT members mentioned the following: Table 25: Reported tablet use Reported tablet use Number Teaching and learning (Lesson planning, assessments) 12 To enter data on the Data Driven Districts (DDD) Dashboard 5 To access Reading materials and videos provided by RSP 4 For communications (Calls/SMS/e-mail) 3 To do research online 3 To complete the requisite RSP Portfolio of Evidence (PoE) 2 Never used / Nothing 2 For taking photos 2 Data note: SMT members reported using the tablet for more than one purpose In response to whether the SMT is doing anything differently as a result of attending the training, in 2019 SMT members reported three key changes: 1) Improved teaching and learning: training assisted the HODs to improve their work in the classrooms. Training encouraged the use of reporting to South Africa-School Administration and Management System (SA-SAMS) on a weekly basis. RSP meetings are being held in the schools and language policies being streamlined Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 88 accordingly. Learners taking library books home to read with parents. Educators are encouraged to use LTSM (9 of 20 schools). 2) Management and Leadership Skills: improved planning and monitoring of school finances, accountability has increased (sessions are held with teachers). Positive change among educators when they experience positive leadership. SMT members started developing policies due to the training presentations (5 of 20 schools). 3) Enhanced support structures: SMT members conduct monitoring visits to supervise and support educators. Educators receive feedback on anything related to assessments, through the moderation of their work, feedback, and suggestions for improvement (4 of 20 schools). Similarly, in 2020 40 of 55 SMT members (73%) in the 59 schools reported changed practices as a result of participating in the intervention. Examples include: • Using strategies to support educators and to teach reading (17 of 40 SMT members) • Improving monitoring and management of teachers and reading practices (14 of 40 SMT members) • Prioritizing reading in schools (7 of 40 SMT members) • Using more reading resources and establishing school libraries, reading clubs and reading corner in classrooms (3 of 40 SMT members) “I now sit with the HOD and discuss what is expected in RSP and that makes it easy to implement. Previously any [one] of us could implement anything without us agreeing with each other and it caused confusion. Now we discuss challenges in reading and we mitigate. We have introduced more reading in the school due to RSP [and] learners read even at assembly to motivate others.” (SMT member, 2020) A key avenue of support to teachers in schools is the establishment of PLCs in schools. Establishing PLCs in schools serves two major goals: (1) improved teacher practice which leads to (2) improved learner achievement (DBE, 2015). In line with the DBE’s Integrated Strategic Planning Framework for Teacher Education and Development 2011-2025, PLCs should be needs-driven and should stimulate collaborative learning and present teachers and principals with an opportunity to improve their level of competence by sharing inclusive practices. However, overall only 113 (76%) of 140 SMT members mentioned having established a PLC in their schools, with 51 (79%) of 64 SMT members from the SMT schools reporting not establishing a PLC. When asked if they could change one thing about the RSP Coaching, what it would be and why, 113 (56%) of 202 SMT members across all 59 schools did not see any need for change (they identified the program as functioning positively and influencing both teachers and learners in a positive manner). The remaining survey respondents provided recommendations that have been summarized in the recommendations section of this report. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 89 8.2 SMT VIEW OF COACH SUPPORTED TEACHER UPTAKE SMT members were asked if they felt that Foundation Phase teaching had changed as a result of the teacher’s interactions with their coach. In 2020, 59 of 64 (92%) SMT members from SMT schools reported that their teachers had shifted their practice as a result of coach visits. In fact, 61 (95%) would recommend using coaches in schools. Both in 2019 and 2020 SMT members felt that changes revolved mostly around teachers being better able to implement learning strategies and noticed enhanced learning as a result. This was the case in 14 of 20 schools in 2019 and with 58 of 64 (90%) SMT responses in 2020. Examples from the qualitative responses in the SMT survey include: • Coaches have helped encourage teacher confidence. “Coach is continuously supporting them [the teachers], she visits in classes and discusses feedback with them and that has boosted confidence of teachers in delivering lessons is now great they cope very well. Subject knowledge-teachers are more empowered than before because of coaching.” (SMT member, 2020) “We have realized a change. Our teachers are a bit confident. We are impressed about the Foundation Phase learners’ performance. That is one of the highlights, even the vocabulary has improved.” (SMT member, 2019) • Educators have been empowered to teach reading “The approach to lessons, especially reading, has improved. Learners now read with comprehension than before RSP.” (SMT member, 2020) “The materials, LTSM are very easy to understand. Teachers enjoy using materials supplied by RSP.” (SMT member, 2020) “Teaching methodologies, the teachers are now well equipped with methods of how to teach. They now value the importance of reading for understanding.” (SMT member, 2020) “The educators have developed the love for using teaching and learning aids. Learners actively participate in lessons and are free.” (SMT member, 2020) • Learners’ reading skills have improved, as a result of improved teaching practices “The approach to lessons, especially reading, has improved. Learners now read with comprehension [compared to] before RSP. Subject results are also an indicators of RSP being a good initiative.” (SMT member, 2020) “…Learners can tell stories using posters…Learners are given opportunity to choose reading material.” (SMT member, 2020) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 90 • All components of language are covered during lesson presentations. “Teachers are able to plan lessons in such a way that they can teach all the structure of language and there is improvement in phonics and sounds. Teacher planning has really improved due to RSP lesson plans.” (SMT member, 2020) “We implement all the learning strategies to enhance learning.” (SMT member, 2019) • Improved time management, example applied in shared reading and Group Guided Reading. “Yes, it has changed. We used to help the learners to read but without following any structure. We now follow a proper structure, such as time management, especially when learners are reading books. For example, we apply time management when learners are doing group guided and shared reading.” (SMT member, 2019) While in 2019 SMT members from three schools specifically referred to negative changes to the teaching and learning as a result of the teacher-coach interaction, no negative changes were reported in 2020. Similarly, in 2019, SMT members from one school felt that although teachers were doing their work better, changes had not occurred, as the program is similar to EGRS I. “RSP is demanding that we ignore other subjects. Mathematics and Life Skills are suffering. After school we need to stay behind to prepare for the next day and forget other subjects.” (SMT member, 2019) “Although RSP involves a lot of administration and it negatively affects the performance of learners on subjects that are not in RSP. The time allocations per lesson are unreasonable, limiting, and restrictive. When I attended training, the coaches said the lesson plan starts from the day the school opened. The tracker contradicts this. The teachers find it difficult to follow the lesson plans. The program is good but has its disadvantages.” (SMT member, 2019) 8.3 TEACHER PERCEPTIONS OF SMT SUPPORT From the perspectives of 290 teachers in the sample (of 311) in 2020, teachers felt they received support from their SMT and that it was generally good across all interventions. Teachers in the SMT group reported receiving more support (95%) compared to the other groups, but only marginally (95% of teachers in the SMT group compared to 93% in the Coaching group and 91% in the LTSM group). Despite the long gap in training for SMT members in 202045 , there is evidence of SMT support as reported by the teachers (refer to Figure 18). This was also the case in data collected in 2019. In particular, teachers in the SMT group reported receiving more support 45 Prior to the school observations, SMT members had engaged in their last PLC in February 2020. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 91 from their SMT members (to a large extent and to a very large extent) as can be seen from the visual below. Figure 18: Teacher perception of SMT support Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 92 9. WHICH EFFICIENCIES COULD HAVE BEEN REALIZED? (QUESTION 13) Data sources: 1) FPD monitoring data 2) Cost data as per approved RSP budgets and actual expenditure figures as per management accounts on January 31, 2021 3) Key informant interviews 4) Cost benchmarks researched for the DBE by Venter and Sharif, 2020 5) Data Driven Districts (DDD) enrolment for Term 4, 2020 https://dbedashboard.co.za 6) RSP Close-out Report, March 31, 2021 The final RSP close-out audit had not yet been concluded by April 30, 2021. It is possible that greater efficiencies could have been realized. The budget process for RSP was not ideal since the initial budget was awarded for a different design. There were problems with the budget for the revised design, as some costs were not budgeted for and later added. A long period of implementation lapsed before the final budget was approved in March 2020. This negatively impacted FPD and USAID’s ability to monitor expenses against budget, resulting in inefficiencies. The DBE’s initial estimates of the anticipated number of schools, teachers and learners were significantly higher than what materialized. Such over-estimation of the number of participants introduced some inefficiencies and wasteful expenditure, for example some interview data show that more LTSM may have been printed than needed. FPD and USAID’s management of the budget and expenditure was not optimal. A substantial error was made in the calculation of the salary line item in Budget 1. The audit fees were significantly under-budgeted in Budget 2. Budget periods did not align in the different versions of the budget. Costs were not accurately allocated to a set of consistent budget line items. These limitations imply that managers were less able to track which items “The struggle throughout implementation has been always trying to keep FPD compliant with the expenditure regulations, requirements within the Cooperative Agreement, especially during the second part of implementation when the Project Accountant was switched. Had RSP been switched to a Contract when it was modified (instead of sticking to a Cooperative Agreement), USAID could have played a better oversight role… One aspect that I could add is the limited role the Chief of Party seems to have had in making substantive input into the budget and expenditures throughout the implementation period.” (USAID Key Informant) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 93 are over- or under-budget as the project progressed. Better budget and expenditure monitoring may have resulted in greater efficiencies. The COVID-19 disruption in schools affected FPD’s ability to conduct on-site support in schools in 2020. The variable cost items such as accommodation and travel were not realized. On a budget of R79,757,468 only a saving of 0.86 percent or R679,829 were realized. A no-cost extension was granted to FPD to allow the program activities to conclude in December 2020, causing salaries, consulting fees and fringe line items to be significantly overspent. FPD missed an opportunity to reallocate some of the savings of the project to enable teachers to participate more actively in the coaching WhatsApp groups provided during COVID-19 related school closures. It is calculated that RSP project saving of R679,829 could have been allocated to provide 1,530 teachers with a 500 MB bundle of data for a period of three months (1,530 x R150 x 3 = R688,500) for roughly the same amount. Removing one of the hurdles of internet access, may have facilitated better use of coaches’ time in the period when access to schools were limited. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 94 10. WHICH ECONOMIES OF SCALE COULD BE REALIZED? (QUESTION 14) Data sources: 1) FPD monitoring data 2) Cost data as per approved RSP budgets and actual expenditure figures as per management accounts on January 31, 2021 3) Key informant interviews 4) Cost benchmarks researched for the DBE by Venter and Sharif, 2020 5) Data Driven Districts (DDD) enrolment for Term 4, 2020 https://dbedashboard.co.za 6) RSP Close-out Report, March 31, 2021 The final RSP close-out audit had not been concluded by April 30, 2021. At the start of the RSP, it was expected that some economies of scale would be possible. For example, printing LTSM in bulk may result in lowered printing costs. However, the LTSM in the RSP were adjusted and redesigned during implementation and there was pressure on the printing timelines. This meant that instead of running the printing for the full two-year program as a single print job, multiple smaller print jobs were accommodated. The costs for LTSM in the RSP was, therefore, higher than the Venter and Sharif (2020) cost benchmarks, but it may be possible that future replications of the project may benefit from the scaling printing. It was expected that coaches supporting more teachers per school over a period of two years would lead to efficiencies in the per-teacher cost of support. When the budget figures are considered in isolation from the actual implementation performance, it does seem to be the case that the per-teacher cost of the RSP was lower than the per school teacher cost of the EGRS I, which was implemented in the same context. However, RSP monitoring data show that individual coaching and also group coaching in afternoon workshops did not materialize as planned – which raises questions about the RSP assumptions about what is feasible for a coach to deliver (and a teacher to absorb) over the period of implementation. Interestingly, the analysis presented in this report suggests that the scale of the program may have reduced economies of scale in some instances. There were duplication of effort and inefficiencies because a large consortium was required to assemble the correct skills, content knowledge and experience to reach more than 1,500 teachers in 252 schools. The budget made provision for approximately two support personnel for every one person involved in direct service delivery. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 95 11. WHAT IS THE ACTUAL VERSUS ANTICIPATED UNIT COST? (QUESTION 10) Data sources: 1) FPD monitoring data 2) Cost data as per approved RSP budgets and actual expenditure figures as per management accounts on January 31, 2021 3) Key informant interviews 4) Cost benchmarks researched for the DBE by Venter and Sharif, 2020 5) Data Driven Districts (DDD) enrolment for Term 4, 2020 https://dbedashboard.co.za 6) RSP Close-out Report, March 31, 2021 The final RSP close-out audit had not yet been concluded by April 30, 2021. The total actual cost of the RSP was R79,077,638. A budget of $5,505,295 was approved. Including the amount of $2,144,670 that was spent on the original RSP prior to its redesign, the total USAID expenditure was $7,649,965. The actual unit costs were: • R313,800 per school, which is 18 percent higher than the anticipated cost of R265,361 • R51,685 per teacher, which is 17 percent higher than the anticipated cost of R44,227 • R1,396 per learner, which is 24 percent higher than the anticipated cost of R1,129 When considering the total cost and output, RSP is more expensive than the cost benchmarks proposed by Venter and Sharif (2020). They found that the cost for EGR training and coaching of a teacher for a period of two years, come to R32,578 (p.40). The RSP cost comes to R51,685, but this also includes an SMT training component which made up roughly 15 percent of the total RSP costs. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 96 Table 26: RSP Cost per unit cost versus anticipated unit cost Unit Planned Reach46 Actual Unit Cost based on originally planned reach Actual Reach 47 Actual Unit Cost based on actual reach Amount Difference = E-C % Difference =(E-C)/E *100 (A) (B) (C) (D) (E) (F) (G) Number of schools 298 R265,361 252 R313,800 R48,439 18% Number of teachers 1 788 R44,227 1,53048 R51,685 R7,458 17% Number of learners 70 000 R1,130 56,63849 R1,396 R266 24% Total Actual Expenditure over 27 months (Aug 1, 2018 to Dec 31, 2020) R79 077 638 Source: RSP Design specification, Specifications for the Reading Support Project,15 April 2019, RSP Budget vs Actual costs, Monitoring data provided by FPD – 29 Jan 2021, DDD enrollment data The higher total cost per teacher of RSP in comparison to the Venter and Sharif (2020) cost benchmarks is due to higher overhead expenses. RSP budgeted less than the benchmark for coach salaries or accommodation, venue and catering. RSP did provide a significantly more expensive LTSM, but the costs of the SMT component in RSP, and other overhead costs account for the relatively higher unit cost of RSP. Table 27: RSP Cost per unit for full program Item RSP Venter & Sharif (2020) Benchmark Coach cost per teacher R18,222 R20,330 LTSM cost per teacher R8,19050 R3,111 Accommodation, venue and catering costs for a program that delivers four sessions of two-day residential training plus four session of single-day training over two years51 R6,084 R13,372 Source: RSP Design specification, Specifications for the Reading Support Project, April 15, 2019, RSP Budget vs Actual costs, Monitoring data provided by FPD – 29 Jan 2021, DDD Term 4 2020 enrollment data 46 Original specification of 2018 RSP revision 47 FPD monitoring data, RSP Close-out Report, March 31, 2021 48 MEL plan of 2020 49 Data Source: DDD - Enrolment T4 of 2020 50 Note the RSP included time for OUP to host reference group meetings and to adjust or redesign LTSM for printing. The Venter and Sharif (2020) cost benchmarks provided for printing costs alone. 51 The substantial difference may be because RSP budgeted for one night accommodation for each two-day training session. It is likely that Venter and Sharif (2020) allowed for two nights’ accommodation for each two-day training session. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 97 12. HOW DOES THE RSP UNIT COST COMPARE TO EGRS I UNIT COSTS AND HOW ARE THEY DIFFERENT? (QUESTION 11) Data sources: 1) FPD monitoring data 2) Cost data as per approved RSP budgets and actual expenditure figures as per management accounts on January 31, 2021 3) Key informant interviews 4) Cost benchmarks researched for the DBE by Venter and Sharif, 2020 5) Data Driven Districts (DDD) enrolment for Term 4, 2020 https://dbedashboard.co.za 6) RSP Close-out Report, March 31, 2021 The final RSP close-out audit had not yet been concluded by April 30, 2021. The EGRS I and the RSP worked in the same districts, and therefore it was expected that some unit costs may be comparable, for example, the costs of accommodation and coach salaries. However, there are significant differences between the two programs, which complicates a direct comparison of RSP and EGRS I unit costs. The programs differed in terms of their 1) implementation period, 2) scope, 3) intervention components, 4) dosage, and 5) implementation fidelity. Table 28: Differences in the design elements of EGRS I and RSP Design Element EGRS I 2016 RSP - Planned 2018 RSP Revised Design - Planned May 2019 RSP revision - Planned May 2019 RSP revision - Actuals Design Differences − 2 training sessions − More individual coaching − Less group coaching Not Available − 4 training sessions per annum − Fewer individual coaching sessions − More group coaching sessions Includes residential training not budgeted Target HL only Intervention in 1 grade only HL and EFAL In all 3 grades simultaneously A crude comparison of the total unit costs shows that RSP resulted in a lower per teacher cost than EGRS I. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 98 Table 29: Cost per unit for EGRS I and 2018 RSP Revised Design Cost per unit EGRS Actual Reach EGRS: Total Expenditure (2015, 2016, 2017) RSP Actual Reach at end of program 2018 RSP Revised Design: Total Expenditure (Aug 1, 2018 - Dec 31, 2020) Number of schools 147 R95,777 252 R313,800 Number of teachers 184 R76,518 1,53052 R51,685 Number of learners 10,966 R ,283 50,469 R1,396 Total expenditure R14,079,317 R79,077,638 The evaluation did not compare the unit costs of the RSP to the EGRS I, due to differences in budget assumptions. The evaluation did conduct a comparison or the RSP costs with cost benchmarks researched for the DBE by Venter and Sharif (2020). When comparing the budgeted cost of the RSP for coach salaries, LTSM, accommodation, venue and catering it is found that the RSP budgeted less than the cost benchmarks researched by the DBE for coach salaries, venue and accommodation, but more for the LTSM. Table 30: RSP Coach Salaries compared to Benchmark Coach salaries and fringe (Budget values) RSP Total cost R14,760,000 RSP Cost per teacher in coaching condition (810)53 R18,222 Lower unit cost of RSP Venter & Sharif (2020) Benchmark cost per teacher54 R20,330 Table 31: RSP LTSM costs compared to Benchmark LTSM (Budget cost in sub￾agreements) RSP Total cost R12,531,266 RSP Cost per teacher (1 530) R8,190 Higher unit cost of RSP Venter & Sharif (2020) Benchmark cost per teacher R3,111 52 RSP MEL Plan May 2020. 53 Assumes a minimum of 60 teachers per coach, and a dosage of 9 to 13 one hour visits over two years, and 9 to 13 afternoon workshops over two years 54 Assumes a maximum of 50 teachers per coach, and a dosage of 2.5 hours contact time for each teacher per month Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 99 Table 32: RSP accommodation, venue and catering costs compared to Benchmark Accommodation, Venue & Catering (Budget values) RSP Cost per teacher (4 nights’ accommodation, 12 days venue and catering) R6,08455 Lower unit cost of RSP Venter & Sharif (2020) Benchmark cost per teacher (12 days residential training) R13,372 55 RSP budgeted 4 days accommodation at R885.10, and 12 days venue and catering at R212. Costs for 12 days of training over two years = R6,084.40 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 100 SECTION 7: LESSONS FOR THE FUTURE 13. WHAT ARE THE UNIQUE CHALLENGES OF DELIVERY AT SCALE? (QUESTION 8) 13.1 RSP TEACHER TRAINING One of the most critical, and difficult, aspects of delivering training to teachers at scale is that there is often insufficient time and/or budget to provide direct training from master trainers to teachers. The Design Evaluation documented that many early grade reading programs turn to cascade training models. The RSP adopted this cascade training approach whereby master trainers’ cascade training to coaches, who then train teachers – referred to as Just in Time (JIT) training. The literature indicates that while cascade training is likely to be more cost-efficient, it is likely that efficiency gains may be offset against effectiveness losses. This is a necessary strategy when implementing at a larger scale, but it may result in variable quality of training. One of the risks identified in other JIT projects is that JIT training programs tend to focus mainly on content, while pedagogy and coherence between reading methods is typically not adequately covered given the time constraints. The results from the coach and teacher training observations, as well as the classroom observation data, support this concern. Important concepts such as Group Guided Reading were covered in the training presentation to teachers, but there was insufficient time to allow for modelling, practice, and information sharing between teachers. Classroom observations found that teachers struggled to implement complex constructs in the classroom. This strategy further had to be amended due to social distancing health requirements as a result of COVID-19. Coach preparation for training remains critical. In addition, the coaches with an ability to facilitate the teacher training should lead this training, while those with less facilitation skills should support it. To answer this evaluation question, the evaluation team drew on findings presented in the previous questions of this Summative Evaluation. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 101 13.2 RSP COACHING Recruiting, selecting, training, and supervising coaches, scheduling and conducting coaching sessions To reach scale, there is a need to recruit sufficient coaches to reach schools and teachers while maintaining an appropriate ratio, which was a requirement reported by the province, district, CA/SAs and head coaches. In addition, there is a need to have sufficient head coaches in order to increase the frequency of visits to coaches at schools, as reported by a CA/SA and one of the head coaches. The coaches recruited must, according to almost all the CA/SAs and indicated by the District Officials, have a background in education, experience in Foundation Phase teaching and be trained as a coach. In addition coaches should have content knowledge about HL and EFAL, be familiar with CAPS and ideally be able to facilitate teacher training (not only learner-based classroom based training). Very few people are both EFAL and HL language literacy experts, coaches and trainers, and therefor may prioritize different aspects based on their skill set. Critical elements for effective coaching practice once recruited, and important for taking the RSP to scale, as reported by District Officials and the head coaches, include: • Every coach must understand the SOPs • Transport is essential for each coach to visit the schools and to be able to access it before school hours in order to arrive at the school on time for the first lesson, and for head coaches • Intensive coach training which is implemented over a number of days or virtual sessions to allow for greater explanations, deeper learning and exploring key concepts Head coach and a CA/SA raised the concern in 2019 which remains relevant in 2020 and for scale, that as numerous teachers have been involved in the EGRS and therefore know how to teach early grade reading, they should not be included in the RSP coaching – especially as other schools and teachers should be prioritized. Furthermore, the Advisor indicated that at some point it is important to let teachers get on with teaching without a coach. The training and mentoring of coaches is important and requires skills and experienced trainers and mentors. The current cohort of coaches have been trained and have experience, and some of them should be included in the planning for scale, and in implementing future coaching interventions. They could further be a resource for the District CA/SAs to increase capacity in the district to support teachers in need of mentoring or improvement, and to bring new teachers into the RSP strategies to improve early grade reading in the Foundation Phase. For scale, the selection and recruitment of a Coach with the required skills and experience is important. Once recruited, it is essential that they are briefed and the induction process sufficiently enables them to implement the program with fidelity. Training in the role of the coach and to build their competency in the coaching practice (such as being reflective, asking for feedback before giving advice) is critical. The coach training modules can be scaled to new coaching recruits. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 102 Ratio of coaches to teachers According to the Design Evaluation Report (page 73), the RSP ratio of coaches to teachers is 1:60 and the recommended performance standard is 1:30. The coach to teacher ratio continues to be variable, as reported by the coaches (Table 33). Eight of the fourteen coaches met the RSP design specification to support 60 teachers or less (highlighted in green in the table). Six of the fourteen coaches are expected to support more than 60 teachers. Coach 4 supports the least amount of teachers, supporting 45, while Coach 16 is expected to support 76 teachers - 16 more teachers than the design specification. The RSP coach to teacher ratio averaged 1:58 (minimum 1:45; maximum 1:76). Although nine met the RSP standard of 1:60, all coaches exceeded the recommended norm of 1:3056 , which reduced the amount of time that a coach could spend with each teacher. Only one coach had the same number of teachers from 2019 to 2020, while for six coaches the number of teachers they supported increased, and for four coaches the number of teachers they supported decreased. This is due to a number of factors; including the resignation of one coach, the promotion of one coach to head coach, and the addition of two new coaches, some coaches were also reassigned schools for easier logistical purposes. However, this switch of coaches and teachers does have implications for the ability to build relationships between coaches and teachers, and ideally a coach and teacher would be paired for the entire duration of the project. Table 33: Coach to school to teacher ratios from 2019 to 2020 Coach: School: Teacher ratio Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 2019 1:10:56 1:9:43 1:8:55 1:8:46 1:7:50 1:8:40 202057 = 1:8:56 1:8:58 1:8:66 1:10:45 1:6:46 NA Resigned Coach 7 Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 2019 1:10:52 1:12:61 1:14:57 1:10:55 1:7:57 1:11:56 2020 1:10:55 1:12:59 1:14:60 1:10:64 1:9:62 1:11:52 56 Hofmeyer, J. (2019) Literature review of Coaching in Education: From ‘Knowing to Doing’. Zenex Foundation Report. South Africa 57 Based off monitoring data from FPD Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 103 Coach 13 Coach 14 Coach 15 Coach 16 Coach 17 2019 1:8:57 Promoted from coach to Head coach 1:8:68 Head coach Not coaching in 2019 Not coaching in 2019 2020 1:8:54 Head coach Head coach 1:10:76 1:9:64 Data Note: Coach 15 was a Head coach and therefore this ratio is not applicable The RSP ratio exceeds the recommended norm, and this affected the amount of time that a coach can spend with each teacher. For delivery at scale, it is important to consider the coach ratio as this will affect the dosage and ultimately the quality of the RSP intervention. Given that coaches are inconsistent in reporting the frequency of teacher observation and support, the scaling of the RSP requires standardizing and capturing the visits by teacher (i.e., not only by school). 13.3 LTSM It is imperative that each teacher has their own copy of the LTSM and other materials. In 2020, there was an improvement in the dissemination of materials. What appears to work well is the distribution of the material directly to each teacher during the training. Teachers who are absent then receive the material directly from the coaches (which works effectively as the materials go into the correct teachers’ hands), or the school (often the principal) fetches the materials (but there is less fidelity in the materials actually reaching the teacher). 13.4 EMBEDDING TEACHER PRACTICE It is essential that for delivery at scale, that teachers understand that the RSP is an integral and systemic part of their teaching. There remains a perception by some teachers (as reported previously) that once the coaching support ends, that the intervention ends. However, the close-out phase of the RSP as designed by the FPD consortium was intended to inform school principals and SMT of both the achievements of the project, as well as indicate that the program needs to continue within the schools. The teacher training for Term 4 further highlighted to teachers that the materials and strategies applied in the classroom were, and should be implemented the following year. Finally, the District CA/SAs have a sense of ownership of the materials and potentially are able to support teaching practice in the future. It is essential that for delivery at scale, that teachers understand that the RSP is an integral and systemic part of their teaching. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 104 13.5 DISTRICT OWNERSHIP OF RSP Comparing the findings from 2019 to 2020 there has been an increase in the district and CA/SAs ownership and commitment to the RSP. Although there remains unclear role separation or ways of complimenting services between Advisors and the coaches. In addition, teachers are often unclear of the different roles, and how the coach and Advisors interact. For example, a few teachers reported that the coaches and Advisors give contradictory advice on classroom practices and this leaves them confused. As a result they do what they feel is right. The sense of ownership by District Officials is one of the most important and critical elements for the deepening of teacher practice, sustaining the implementation of the RSP, and for taking the RSP to scale. Without this buy-in, the RSP is unlikely to be sustained or be taken to scale. The support and facilitation of ownership by the national DBE was essential to the successful increase in district level ownership of the program. The CA/SAs involvement in the development of the quality materials was critical for the high level of ownership they articulated to the evaluator. For delivery at scale, it will be important to ensure that there is clarity of roles and boundaries, and that the value of coaching is understood and accepted by CA/SAs in addition to the Teachers, Principals, and SMT members. 13.6 COLLECTING QUALITY DATA FOR ON-TIME DECISION MAKING AND MANAGEMENT Scaling requires standardized indicators, monitoring and reporting systems which are established at the beginning of an intervention and which are used for decision making and tracking the activities, outputs and outcomes. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 105 14. WHAT TIME COMMITMENT IS REQUIRED TO IMPLEMENT AT SCALE? (QUESTION 12) Data sources: 1) FPD monitoring data 2) Cost data as per approved RSP budgets and actual expenditure figures as per management accounts on January 31, 2021 3) Key informant interviews 4) Cost benchmarks researched for the DBE by Venter and Sharif, 2020 5) Data Driven Districts (DDD) enrolment for Term 4, 2020 https://dbedashboard.co.za 6) RSP Close-out Report, March 31, 2021 The final RSP close-out audit had not yet been concluded by April 30, 2021. The RSP required level of effort from a consortium of four service providers that had the relevant subject expertise, as well as the relevant capacity to implement it in 252 schools. Key respondents from the DBE and the consortium indicated that some inefficiencies are created when many organizations are involved in the delivery of a program. Each organization, for example, had to allocate level of effort to project management, finance, human resources etc. Respondents recognized, however, that there are very few organizations in South Africa that have the human resource capacity and subject specialization to deliver an early grade reading intervention on the scale of the RSP. The RSP budget made provision for the involvement of persons in 75 staff positions and 12,438 person-days spread over 30 months. This excludes the team of freelancers that were involved in LTSM development tasks, and staff from the DBE and Provincial Education Department. The budget indicates that the RSP team involved fourteen coaches and two head coaches, six persons involved in the SMT training component and eight persons involved in the coach training. For each of the individuals involved in delivering the RSP intervention services, approximately two support personnel (including project managers) were involved. Support staff held positions in project management, logistics, administration, information technology, finance, human resources and monitoring and evaluation. Key respondents indicated that it is expected that a donor funded project would have onerous administrative and financial oversight requirements, which required additional staff. Beyond the involvement of the staff from the FPD consortium, some time was spent by four DBE staff members and other staff at the Provincial Education Department to implement the program. The RSP required approximately 13 days per annum from a DBE research director, 105 days split over two years from a lead researcher at deputy director level, a full￾time project assistant, and approximately five days over two years from champions at DBE senior management level. Estimates for staff from the PED was not collected, but the RSP, Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 106 as it was designed, required more than a full working months’ time (28.5 days per annum) commitment from a Curriculum Advisor responsible for supporting Setswana Home Language and required almost a full months’ (18.5 days per annum) time commitment for supporting English First Additional Language – this is time over and above their normal responsibilities. The RSP also had a time cost to participants. The RSP required a time commitment from teachers and SMT members beyond what is prescribed by SACE’s framework for continuous professional teacher development. Had the program been implemented as planned, each teacher would have spent about 48 hours per annum in training, and those involved in the coaching intervention, would have spent an additional 18 hours in coaching activities. While an analysis of the time and staff allocations of the RSP provide some insight into the requirements of delivering an Early Grade Reading project of this size and scope, there are some limitations to generalizing beyond the RSP. Some Key respondents indicated that the human resource allocation for some tasks, for example, monitoring and evaluation, may have been under-budgeted, while others indicate that better management of the human resources may have resulted in greater efficiencies. FPD personnel in management roles mentioned long working hours, especially when travelling to visits schools and attending provincial meetings. The time and cost impact of COVID-19 disruptions on the workload of managers and coaches were not unpacked in detail in this report. Reportedly, managers spent between 10 and 12 hours a day while working at home and would sometimes even work over weekends to plan and report. Key respondents indicate that the COVID-19 pandemic increased the workload of coaches and head coaches who had to adapt and develop new strategies, but given that they were unable to travel to schools it is possible that at least some of the time that were allocated to coach support may have been under-utilized in the period when schools were closed. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 107 15. IS THE RSP COST EFFICIENT? (QUESTION 9) An inspection of the cost data shows that budget management and cost controls were not optimal. An analysis of the budgeted time shows duplication in roles between different consortium partners, and a high ratio of support personnel to personnel responsible for direct service delivery. These factors, together with the poor fidelity of implementation, allows the evaluation team to conclude that the RSP was not cost-efficient. While the time and cost analysis attempted to understand the costs of this scaled-up initiative in two districts, the evaluation team advises against carrying out future cost￾effectiveness analysis using this cost data. The RSP financial costs analyzed in this report is not an indication of what this intervention should have costed. This is for two reasons: 1) Using the budgeted figures is inaccurate since the budget erroneously under-budgeted some of the significant cost items, 2) the actual expenditure figures are not a good measure of the implementation costs because far less travel and training took place due to COVID-19 restrictions. This implies that the costs could be either significantly underestimated or overestimated when using the RSP budget and actual expenditure. On the one hand, it may be that the costs are underestimated in this report when using the budget figures. There were inconsistencies in the budget that indicates some significant cost drivers were under-budgeted. The actual expenditure figures also underestimate the true costs of implementing the RSP design, because the treatment fidelity was low and the COVID-disruptions eliminated the need for significant budget line items such as venue, accommodation and travel costs. It is possible that for the level of coaching and training to have taken place as per design, the costs may have been significantly higher. On the other hand, it may be that some of the costs are overestimated in this report. There are examples of budget inefficiencies that could have been better managed and the consortium duplicated effort for project managers, finance, human resources; the number of support staff provided for in this budget is extremely high (about two support people for each person involved in service delivery). Interview data indicate that with less stringent requirements of USAID compliance and more efficient management of staff, significant savings may have been possible. To answer this evaluation question, the evaluation team drew on findings presented in the previous questions of this Summative Evaluation. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 108 SECTION 8: DISCUSSION The purpose of the Summative Evaluation was to provide an overall assessment of the RSP fidelity of the intended design to actual implementation and to determine whether the RSP can be effectively scaled. SCALING READING PROGRAMS The underlying evaluation purpose was to examine if RSP effectively scaled the EGRS I and what lessons it would offer to provinces interested in improving education quality through coaching. Importantly, the reason for scaling early grade programs, such as the RSP, is to scale quality innovations of the intervention that have a positive effect on teaching practice and, thus on, children learning to read. Research58 indicates that programs are more likely to scale if they: 1. Are built on established evidence/outcomes, or in other words, the pathways to change need to be built on evidence that the intervention elements each work (e.g. lesson plans are used by teachers AND learners can read better). 2. Include an evaluator/academic researcher in the design process to use feedback loops to help test, iterate, and improve the innovation over time. While both the DBE and the evaluators were involved, this point refers to an even deeper involvement, ensuring that M&E data is actually used to create mini-feedback loops and, as a result project leaders, implement data driven decisions and adaptive management. Embedding a Developmental Evaluator59 in the project would have been even better. Finally, there needs to be reading assessment data available to ensure that indicators are not just outputs, but outcomes. The learner assessment data should create active feedback loops should be at every level of the project: o With learners, so teachers can work with parents/caregivers to support reading acquisition and help with diagnosing learning difficulties from an early age o In a classroom, so teachers can easily identify learners that are either excelling or struggling and need extension or intervention o In a grade, so that HoDs can help teachers identify and resolve teaching issues related to reading, and stronger teachers can mentor those with less experience or expertise 58 https://www.cgdev.org/blog/case-evidence-based-innovation-and-implications-usaid-and￾beyond#.YFMG2to81XM.linkedin 59 “Developmental Evaluation (DE) was first proposed by Michael Quinn Patton with the support of colleagues who have wrestled with the problem of dealing with complexity in human systems and the need to provide structured, useful, actionable information to make decisions to support innovation.” https://www.cense.ca/developmental-evaluation-a-short-introduction/ Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 109 o In a school, so that SMT members can see how learners are performing and run PLCs with teachers to address common issues and take action if the school requires more resources or expertise o In a district, so that officials can target assistance and support to schools struggling to perform rather than rolling out blanket programs to all schools o In a province, do that programs can be rolled out to needy schools and policies developed to support reading acquisition, based on evidence o In the DBE, to analyze and compare reading results against policy and other be and to compare with international comparative studies such as PIRLS 60data and the Southern and Eastern Africa Consortium for Monitoring Educational Quality (SACMEQ) studies, etc. 3. Leverage established distribution networks (government or otherwise). By 2020, RSP was working with external outsourced service provider to deliver LTSM. Possibly LTSM could be distributed through PED standard channels (e.g., how the workbooks are distributed). 4. Collaborate with existing institutions to: o Build local capacity: Potential opportunities would include a strong link to SACE so that educators and SMT member earn CPD points and link to local universities for training. In particular, there is a need for pre-service foundation phase teachers to learn the techniques needed to teach reading effectively. Ensure all in-service training is actually assessed so that performance can be monitored. As data is zero rated for educational sites, explore more on-line training opportunities, with providers, such as COURSERA, who have assessment built into their system. o Bring evidence to bear on policy: One of the issues surfaced in this evaluation is the lack of performance benchmarks. This evaluation raised many relevant questions such as: How many sentences should learners write each day? The evidence should lead to revised CAPS policy requirements and on the design of future reading programs. 5. Have a low cost per person reached: as discussed in other sections of this Summative report, for the RSP and other reading programs to be scaled, it is critical to ensure that the reading program is affordable. 60 Progress in International Reading Literacy Study (PIRLS) is an international comparative assessment that measures student learning in reading. Since 2001, PIRLS has been administered every 5 years. PIRLS documents worldwide trends in the reading knowledge of 4th-graders as well as school and teacher practices related to instruction. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 110 SCALING RSP As stated in the Formative Implementation Report in 2019, delivering the RSP at scale is affected by a number of factors including the extent of teacher training required, the availability of good coaches, and the logistical arrangements around developing, printing, packaging, and distributing LTSM. Scaling the RSP to three Grades and two subjects (Setswana HL and EFAL) simultaneously affected the ability of FPD to deliver and monitor the RSP interventions. The challenges to uptake presented previously are not necessarily ‘unique’ challenges to take the RSP to scale, as such challenges are faced by many educational programs. The implementation evaluation findings indicate that the RSP in its current design cannot be effectively scaled to all primary schools in the North West. Typically, there are four types of scaling according to McLean and Gargani: 1. scaling out (replication), 2. scaling up (growth of the intervention), 3. depth (enhanced results and practice) and 4. sustainability (setting up for scale which is sustained)61 . The RSP attempted the first two simultaneously. While FPD presented a “Sustainability Strategy”62, it was actually a plan for hand-over of RSP resources to the DBE. Based on the evidence, a number of elements within the RSP can be scaled. There are a number of successes emerging from the RSP and the foundations are in place firstly, for HL and EFAL to be deepened in the districts, and secondly for key elements to be replicated and scaled to ultimately, improve early grade reading outcomes in learners. Scaling Out LTSM: Increasing reach of materials through dissemination Lesson Plans The teacher lessons plans are valued by teachers and district officials. These stakeholders consider the lesson plans to be a high quality product that enhances teacher practice. CA/SAs indicated that the eight-week structured lessons plans are more advantageous than the ten-week lessons plans (which is the length of time of a term). Teachers are seldom able to utilize all ten weeks because of administration, strikes, sport events and other such activities). CA/SAs feel ownership because they were involved in designing the lesson plans. Lessons plans could be disseminated to all teachers in the province. This should, however, be accompanied with teacher training. Altogether 78 percent of teachers indicated they use 61 Khulisa adapted the framework of McLean, R. and Gargani, J. 2019. Scaling Impact; Innovation for the public good. Routledge. 62 FPD presented this “strategy” at the August 27, 2020 PMT meeting. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 111 lesson plans daily. Where the lesson plans were not used, there seemed to be a link with not attending (or not being invited to) training. LTSM materials Teachers’ use of RSP LTSM in the classroom differs based on their confidence and understanding of how to use the materials. Once teachers see their learners improve their reading skills, they reported being more committed to using the materials continuously. To increase uptake in the classroom, some teachers need additional support and the intervention needs to target this cohort of teachers. An assessment feedback loop would reinforce use of LTSM. Scale LTSM dissemination to directly reach the intended teacher at the training Teachers preferred that the LTSM be delivered directly at the training session. This would allow monitoring of delivery, as teachers sign the register. The full package should be available at the start of the year rather than per term. For scaling, it is imperative that at the start of the project there is a detailed and agreed upon database of schools and teachers. Quarterly orientation sessions should be held as it should be expected that there will be new teachers continuously (from the RSP experience only 47% of teachers attended at least six of the seven training sessions). Scaling Up the Coaching Investment: Build on the investment in the RSP coaches The four excellent coaches demonstrated how coaching can provide valuable support to identified and/ targeted schools and teachers. These coaches are a resource in the district and province, and could be integrated into existing and new early grade reading programs. An alternative strategy to take the program to scale is to integrate the coaches into the district system thereby capacitating and strengthening the system. The coaches could work as curriculum advisors thereby increasing support to teachers. This approach has resource requirements and budgetary implications. The coaches’ job description should not be simply procedural but focused on reading outcomes. The RSP and UNICEF coaching training modules are available to capacitate new coaches. HODs, SMT members, principals and curriculum advisors could be trained on these modules to increase their skills, capacity and ability to support teachers in early grade reading practice. Scaling to Deepen Lasting Classroom Practice: Scale the SMT engagement through focusing on how SMT members support early grade reading teaching practice The SMT intervention was not integrated sufficiently into RSP, and operated fairly independently from the coaching, teacher training and LTSM interventions. Teacher respondents noted that SMTs are important to promote early grade reading practice. Therefore, the focus of the SMT support should be on “how to be a supportive SMT” and to enable teachers to implement the LTSM in their classrooms. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 112 Sustaining Scale: Scale and sustainability will not be achieved if District Officials do not contribute to the design and content of the materials to support curriculum delivery The ownership and engagement of CA/SAs is critical. Five of six district officials became champions of the LTSM and engaged with teachers and the coaches at the training. Integrating officials into the coach training and developing materials increased ownership. This took time, but their involvement from the start contributed to the LTSM and commitment to support teachers. Multi-stakeholder programs need strong management and monitoring structures which hold all stakeholders accountable No project with stakeholders can be managed without clear management processes and accountability levels, structures, monitoring and reporting processes. There needs to be clear consequences if implementation does not meet the fidelity and quality standards. Management message should be very clear, to avoid broken telephone syndrome seen in the RSP. The structure of the DBE national Steering Committee and PMT was deemed an important, effective and efficient mechanism for the oversight of the RSP. The PMT involved key stakeholders from various levels and departments within government, as well as implementers, evaluators and donors provided a management structure that allowed for: • Clear communication between all stakeholders • Feedback Loops, promoting adaptive management • A joint planning and management platform for decision making • Accountability and responsibility In addition, the complexity of the RSP implementation requires a strong project management, monitoring and reporting systems and individuals within an organization. The selection of an implementation partner is critical to implement a program with fidelity to achieve the desired outcomes. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 113 LESSONS FROM THE RSP FOR PED REPLICATION Key requirements for PED’s to replicate the RSP or early grade reading programs in provinces include: 1. Efficient project management and structures geared to adaptive management Early grade reading projects require effective and efficient management structures and strong project management and leadership which is solution orientated (they need to be able to adapt service delivery to meet outcomes and not only be compliance orientated). Project management should build relationships between partners in Consortium, with government officials, engage with school managers to facilitate collaboration and ownership of the project. Furthermore, the project manager should encourage and facilitate active participation, ensure efficient, timely communication, encourage feedback and allow, and encourage on the ground initiative and innovation to overcome barriers within project parameters. The implementing organization should, in the design of the project, establish clear roles and responsibilities, an efficient and timely M&E system, appropriate communication and feedback mechanisms, and detail the sustainability plan for embedding outcomes in the education system once the project is completed. Ideally, the project manager and or a strong implementation lead should be placed in the locality of the project, and have authority to make decisions and adapt management processes or project activities to respond to changing dynamics and to overcome barriers to implementation. 2. Coordinate roles and establish effective communication conduits Programs that require collaboration and coordination between multiple stakeholders and partners should establish clear roles and responsibilities aligned to early grade reading activities and results. For the effective implementation of such collaboration (between individuals, structures and organizations), communication conduits and protocols need to be established. For example, RSP implementation requires different levels and types of collaboration between project implements, national and provincial DBE, District officials including CA/SAs, SMT, coaches and teachers. 3. Strong M&E systems to drive data-driven performance management A well designed and efficient M&E system should be established, and agreed to by all stakeholders, at the start of the project. The system should: a. Provide timely data for project management decisions to monitor and improve performance. b. Ensure the system provides feedback loops and triangulates data in near to real time for use by decision makers to improve performance. c. Use data to move beyond compliance by including data on the quality of the activities and results at the outcome level, thereby enabling data driven decision making. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 114 d. M&E should be incorporated as an integral part of senior management and project management. Therefore it is necessary to have a senior evaluator who has the authority to direct and improve M&E capacity and systems, as well as inform and guide data driven decisions. Sufficient capacity is required with the M&E team to enable data driven performance (i.e., one person is unable to fulfil this function for a large and critical project of this nature). e. M&E should be integrated into all aspects of the project management cycle and practice: from design, budgeting, planning and implementation, reflection of practice, decisions and reporting. Attention to the details of the project, for example, confirming an agreed list of schools and teachers, using GIS maps to guide resource allocation to schools, use of secondary data sets and using baseline data to guide differentiated practice by coaches can contribute to strong implementation practice and cost efficiencies. The M&E Framework, ToC, indicators should relate to the data collected and requirements for data driven decisions. Effective and efficient data collection methods should be implements, and an integrated mobile system is recommended. f. Ensure that quality benchmarks are clearly demarcated, allowing outcome data to be measured (for example, rather than measuring how many teachers are using the lesson plans, measure how many learners are reading at benchmark level). 4. Focus capacity to build quality coaches who implement with an optimal teacher ratio a. The quality of coaches is driven by their selection, training and supportive management. Head coaches should not only focus on compliance, but on the ongoing professional development of coaches by providing feedback, assisting in overcoming barriers coaches face in schools, supporting coaches in providing services to teachers, and in building a community of practice between coaches. b. Project management should enable coaches to provide meaningful support to enhance teacher practice, rather than only on compliance. c. The international benchmark of coach to teacher ratio is 30:1. A higher ratio is only possible if differentiated coach services are provided to teachers in a school: for example, individual development plans identify teachers who need less support which allows the coach to focus more on teachers who require additional support. d. Communication and feedback by coaches to teachers should be included in the reporting system to allow for triangulated data and to monitor changing teacher practice over time. 5. Teacher use of learner assessments and benchmarks to measure changes and classroom uptake To determine teacher uptake and the effectiveness of RSP interventions, it is necessary to determine how well learners are doing. This requires firstly, providing baseline data to individual schools and teachers in order for teachers and managers to know how their school, grades and learners are doing. Facilitated workshops on what these results mean Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 115 and how schools and teachers can respond – and how the intervention provides solutions – is recommended. Secondly, assessments need to be used consistently throughout the intervention timeframe to monitor changes and allow for adaptations or differentiated services to meet needs. Assessment results should therefore be fed into the RSP data. Thirdly, benchmarks are required to confirm learner performance. However, benchmarks and clear targets are required to determine dosage of interventions, specifically for coaching interventions. The use of benchmarks and assessment or performance data should feed into both the early grade reading program M&E system, as well as the DDD Dashboards. This will contribute to managers and decision-makers making data-driven decisions rather than relying on observations and anecdotal evidence by teachers. 6. Clearly define all elements of the project and how all role players will work together to achieve the agreed goal. a. All aspects of a program intervention should be clearly articulated in the ToC, and all role players should agree to and support the implementation of activities to achieve the ToC. The ToC should include feedback loops for data, communication and adaptive management. For example, how data is triangulated and reported back to the teacher who is able to adapt strategies to address specific gaps in learner early grade reading performance. b. If structures, content or relationships are not defined the intervention is less likely to succeed. For example, it is necessary to clarify what is meant by SBW (for example, the frequency, content, structures, processes, etc.); and define requirements for WhatsApp coaching (frequency, type of messages, set up content messages etc.). Furthermore, ensure implementers know what the minimum requirements are, while encouraging implements to go beyond the minimum and interaction. c. Communication is critical to transfer not only administration and logistics activities to stakeholders and partners, but also how the desired outcomes and changes will occur, for example, by using platforms that engage implementers and beneficiaries in dialogues where changes in behavior is promoted and challenges and solutions can be shared. 7. Projects should be designed taking into account how change will occur, the length of time needed for the change to be brought about, and how change will be sustained a. A two-year project is too short a timeframe to bring about behavior change which teachers, SMTs and districts can sustain. The RSP, as a two-year project, was only able to ‘pilot’ or ‘test’ an approach and therefore the lessons and practice need to firstly guide the further development of teacher practice in the district, and secondly inform how to scale and sustain in that district. In new districts, replication of the project should allow for approximately two years to set up structures, relationships and activities before the depth and quality of delivery matures to enable a focus on outcomes and shift from a project focus to a sustained intervention. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 116 b. The design should include a strategy on sustainability and how the project will be closed out. This requires considering what materials, structures, relationships and practices need to be built and embedded in the education system for sustainability. For example, coaching practice should be integrated into district plans and operations through coordinated communication, reporting and official engagement between coaches, CA/SAs and SMTs. RSP reporting should be part of the existing report structures, for example, include a new field in the DDD so as to strengthen the existing system. Activities should be included in the district and provincial official calendar of events. c. The RSP program should be institutionalized into the school timetable (and therefore coaching sessions are not seen as an add-on but rather part of timetable and teaching practice). Such an approach requires school and district to support inclusion into timetables. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 117 RSP IMPROVEMENT SUGGESTIONS: TEACHERS When asked if they could change one thing about the RSP intervention, what it would be and why, 170 teacher respondents provided various suggestions, outlined below. They are summarized by topic of recommendation, not by order of importance. 1. Time: The teachers indicate that there is too little time allocated to some of the concepts and activities and the learners are unable to complete them. They also mention that at times too many topics are discussed per day and the learners’ attention spans are too short to retain all information. Evaluator comment: This speaks to the need to further test the materials and ensure sufficient time and to establish a benchmark. 2. Differentiated lesson plans: The teachers recognize that not all learners function at the same level and have mentioned that the RSP intervention does not cater for slower learners. This became particularly apparent during group guided reading where learners are dependent on each other to finish tasks. “I’m happy about RSP, but it does not cater for the slower learner.” (Grade 1 Teacher, 2020) “RSP must accommodate slower learners, because they take time to learn and understand, it only has room for the achievers” (Grade 2 Teacher, 2020) Evaluator comment: This speaks to the need to differentiate the training and support provided to teachers to enable them to cater to learners different needs. 3. Home Language: The teachers complimented the HL phonics lessons, however noted that there were many incorrectly spelled words and asked for other Setswana language experts to look at, and revise, the material provided.63 Evaluator comment: This speaks to the importance of the involvement of CA/SA in reviewing the materials to ensure standardization of Setswana use. 4. Handwriting component: While some teachers indicated that they need more time to spend with their learners on handwriting skills, others firmly state that writing cursive in Setswana is a waste of time and should not be focused on. Evaluator comment: This speaks to the importance of training teachers on the different components of language acquisition and how they all build on each other. 5. Lesson plan preparation: The teachers indicated frustration when it comes to rewriting or creating lesson plans,64 when they mentioned lesson plans are part of the 63 The evaluators note that there is an ongoing debate about Setswana (dialects, inconsistent spelling, etc.) 64 Its one of the compliance checks that seem to be done, has the teacher written a lesson plan. Those checking seem to require the teacher to handwrite the lesson plan, thus causing frustration. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 118 RSP project. They would like to be presented with weekly plans that can be adapted, but not to be completely re-written by the teachers. Evaluator comment: This speaks to the importance of integrating project materials with department requirements for teachers to write their own lesson plans. 6. Training time – Several teachers stated that it is not ideal for them to miss school time to attend RSP training sessions. They advocated for teacher workshops during holidays or after school so that it does not interfere with class time. Evaluator comment: This speaks to the importance of identifying the optimal time for training of teachers. While this was a recommendation provided by teachers, the RSP tried to train over the holidays during its first year of implementation, but teachers were unavailable and an agreement was reached to train over the first week of Term, where it would be easy for schools to accommodate looking after learners from other grades while their teachers were at training. 7. Expansion: Many teachers stated that they were so impressed with the RSP and that they would like a similar intervention for mathematics. “I would change nothing, but ask USAID to do the same with mathematics” (Grade 2 Teacher, 2020) Evaluator comment: This speaks to the importance of the providing support and mentorship to teachers across subjects to improve teaching and learning. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 119 RSP IMPROVEMENT SUGGESTIONS: SMT When asked if they could change one thing about the RSP intervention, what it would be and why, 43 SMT respondents provided various suggestions, outlined below. They are summarized by topic of recommendation, not by order of importance. 1. Time: SMT respondents felt that the RSP workload/content takes too much time and effort for the Foundation Phase teachers. Evaluator comment: This speaks to the importance of principals understanding time commitment requirements of teachers. 2. Training time: While the original RSP design called for residential training during school holidays, the revised implementation processes pulled teachers out of school. The SMT respondents requested that training should not interfere with teaching time. Evaluator comment: This speaks to the importance of identifying the optimal time for training of teachers. While this was a recommendation provided by SMT members, the RSP tried to train over the holidays during its first year of implementation, but teachers were unavailable and an agreement was reached to train over the first week of term, where it would be easy for schools to accommodate looking after learners from other grades while their teachers were at training. 3. Differentiated lesson plans: because learners are not at the same level in terms of learning and reading, SMT respondents suggested: a. Group Guided Reading should accommodate more learners per session (i.e., usually Group Guided Reading involves four to five learners, and therefore SMTs recommend more learners are involved) b. Reduce the number of sight words as this is too much information on a weekly basis for teaching and learning. The RSP materials did not state how many sight words should be taught per day or week. c. Teaching Phonics for two days is restrictive for learning. However, the RSP indicates Phonics should be taught for 15 minutes per week, usually five minutes per day over two or three days depending on the Grade. Evaluator comment: This speaks to the need to differentiate the training and support provided to teachers to enable them to cater to learners different needs. 4. Frequency of coaching visits: while in 2019 SMT respondents would like the frequency of visits to be reduced, in 2020 they felt that visits should increase. Evaluator comment: This speaks to the need for SMT members to be aware of support visits expected from the coach in their school. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 120 5. Training on the use of tablets: in the coaching process there should be a slot allocated to training on tablet use. Evaluator comment: This speaks to the need for SMT members to also receive on-site support on how to implement what they have learned during their PLCs. 6. Coach to work with Curriculum/Subject Advisor: recommendation that the school support should be done in conjunction with the department officials (CA/SAs), for example, RSP does not explicitly explain how school based assessments should be conducted, and in certain instances, the lesson and curriculum tracker for RSP contradicts with the school assessments planned by the schools and CA/SAs. Evaluator comment: This speaks to the need for interventions to closely integrate and align the work of coaches and CA/SA for sustainability. 7. SMT respondents requested feedback from coaches on how their teachers’ performance to be able to support them better. “Maybe increase the time that the coach visits the school so that teachers may spend more time learning from the coach. The SMT should also be involved in engaging with the coach so that they also can learn. The coach should be able to talk to me as a principal about what is expected and guide me where needed.” (SMT member, 2020) Evaluator comment: This speaks to the need for SMT members be more involved in the implementation process. This will allow for better support provided to teachers and more alignment between SMTs, coaches and teachers. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 121 SECTION 9: CONCLUSION There was insufficient implementation fidelity for the LTSM, coaching and SMT interventions. The differentiated practice between the interventions is negligible due to poor implementation resulting in insufficient dosage and uneven quality of coaching, and insufficient integration of the SMT support into the overall RSP implementation practice. Consequently, the coaching, LTSM and SMT intervention results are very similar. The evaluation found that four of the 14 RSP coaches could be judged as excellent, and who go beyond meeting administrative targets to provide quality coaching services, which strengthen teacher professional development and classroom practice. Six of the 14 RSP coaches were judged as average, and four of the 14 RSP coaches were judged as weak. Ratings were determined against RSP coach job descriptions, and emerging best practice for literacy coaches in South Africa. The SMT intervention was not implemented with the expected fidelity. Some SMT members and teachers perceived a value in the SMT intervention particularly as the training included modules on supporting early grade reading in the classroom and school. Overall there was an improvement in teacher uptake of LTSM and activities in classrooms since 2019. Across the interventions and strategies, approximately 75 percent of teachers are implementing the activities. However, there is room for greater improvement and better implementation of LTSM. The implementation of the RSP was not efficient. Although implementation improved in 2020, there remained many aspects where improvements could have enhanced dosage and quality of the interventions. However, there are aspects of the RSP that can be scaled. Accurately tracking the implementation fidelity of a large-scale reading intervention provides beneficial information for the identification of implementation challenges, and for adaptive management. To realize the potential of using monitoring data for adaptive management, an integrated monitoring system with real time, or near-real time, insights is required. Well-designed Information and Communications Technology (ICT) systems that are robust even in low-connectivity scenarios, could streamline the collection of data about LTSM delivery, training attendance and participation in coaching activities. Importantly, insights from such data could be made available to those with the role of monitoring and supporting implementation. The way in which the DDD Dashboard is integrated with existing EMIS data systems, and the way in which it provides role-specific insights, could be a model to investigate for reading initiatives that are rolled out by PEDs. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 122 SECTION 10: RECOMMENDATIONS DBE should: 1. Develop strategies and implement plans to address teaching and learning losses due to COVID-19 to mitigate the risk of long-term consequences on learners’ mastery of early grade reading. 2. Design and implement plans and programs to support SMT and teachers to promote their and learners wellbeing 3. Should schools face closure once again due to COVID-19, issue clear directives as to how schools and teachers should 1) communicate with parents; 2) ensure learners have clear tasks to complete and 3) measure this performance against a benchmark (such as completion of exercises, number of days) can continue to provide teaching and learning opportunities to learners to prevent further learning loss. 4. Ensure that programs – whether designed with differentiated approaches or not – are implemented to meet reach, dosage and quality standards. Without the required dosage and quality, programs are less likely to lead to the desired change. 5. During the design of programs confirm the roles and responsibilities of coaches and CA/SAs. There is a need to establish clear communication protocols to reduce conflicts in advice. A clear strategy is required as to how coaches and advisors complement each other. 6. Ensure that there is clarity about performance benchmarks. Teachers should: 1. Address gaps in learner literacy development lost due to school closures as a result of COVID-19, such as vocabulary development – guided by DBE, but teachers must not wait for DBE/SMT instruction; teachers should be equipped to take initiative to support learners in instances such as school closures 2. Be empowered to take ownership of program and methodologies in their classrooms and understand that the RSP is an integral and systemic part of their teaching 3. Attend and participate in the required dosage of training events, coaching sessions (virtual or physical) and SBWs 4. Implement the RSP and COVID-19 risk mitigation strategies on a daily basis in the classroom as designed 5. Actively pursue their engagement in reading support programs and ongoing professional development SMTs should: 1. Enable the same teachers to attend all training sessions 2. Support teacher classroom practice Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 123 3. Encourage a culture of reading in the school 4. Implement COVID-19 risk mitigation strategies to prevent long-term consequences on learner’s mastery of early grade reading 5. Attend and participate in SMT trainings to strengthen management and functionality of schools 6. Continue to engage and facilitate coaching practice within their schools 7. Work with coaches to enable uptake by teachers of coach support and advice PEDs and other Implementers should: 1. Base the program manager in charge of the Early Grade Reading program in the province (not in another province). 2. Provide administrative support for the Early Grade Reading program should also be in the province. 3. Include in program design a risk matrix with clear mitigation strategies, and which includes a basic disaster management approach. 4. Integrate the SMTs and principals’ strategies and approaches into the overall design and implementation of the program. 5. Gain SMT and principal commitment to ensure that the same teachers attend all the required training. 6. Set up SOPs to strengthen communication and working collaboratively between coaches and SMTs. This will ensure that SBWs are integrated into the school’s timetable and could help develop a school-based PLC 7. Ensure coaches provide developmental feedback to teachers and principals especially towards removing barriers to classroom practice, and supporting the teacher to implement the coaching advice. 8. Design good and clear management processes and accountability levels, structures, monitoring and reporting processes and consequences if implementation does not meet the fidelity and quality standards. 9. Specify roles, procedures and boundaries, and that the value of coaching is understood and accepted by Subject/Curriculum Advisors in addition to the Teachers, Principals, and SMT’s. 10. Ensure that there are strong induction processes for new coaches that enables them to implement the program with fidelity (recognizing that coach turnover and absenteeism is a serious issue). 11. Provide training in the role of the coach and to build their competency in the coaching practice (such as being reflective, asking for feedback before giving advice). This must be provided from the start of the intervention with continuous capacity building based on performance assessments and coaches’ individual developmental needs. 12. Deliver LTSM to each teacher, requiring sufficient numbers of printed materials as well as efficient distribution channels. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 124 13. Provide LTSM designed for Multi-grade schools and learners at different levels. 14. Consider PED messaging that explains that this is not “a project” but a systemic improvement as many teachers felt that the RSP was a “fad” and they could go back to previous teaching practices when the program is concluded. Early Grade Reading programs should: 1. Expend resources on selecting, training and supervising coaches 2. Ensure head coaches model good coaching practices with the coaches and provide developmental feedback and support to coaches 3. Provide differentiated support to teachers who are not implementing the early grade reading strategies to implement these strategies with fidelity and improve classroom practice. 4. Recruit literacy coaches that includes mapping skills against a comprehensive job description and competencies on interpersonal skills for quality practice. 5. Monitor LTSM use, as not all teachers actually unpacked and used the materials. 6. Disseminate LTSM materials with a training component. 7. Ensure the program is at least five years long so that behavior change is more likely to be integrated into teachers and other actors practice. 8. Put in place policies and guidelines that will ensure sustainability. 9. Integrate assessment (and feedback) into all levels of the program and ensure results are recorded in the monitoring system. For example, the following groups should be assessed both during and after their training sessions: a. Coaches b. Subject and Curriculum Advisors c. SMT members (the SMT portfolio is a step in this direction, but it was unclear if it was marked and feedback given, or if SACE provided credit for the course) d. Teachers e. District Officials f. Program managers/administrators g. Head coaches 10. Implement the rubric structure during the coach dry run during training for all dry runs. The rubrics will provide structured feedback to the coaches. MEL Recommendations for DBE, PED and implementers: Whether the MEL system relies on sophisticated technology options, or on available low￾tech solutions, a few key building blocks of data systems must be in place to run effective early grade reading programs. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 125 1. Develop an M&E framework using the intervention design as the basis for selecting indicators and setting targets. Include how the implementation, as well as the results of the intervention, will be monitored. 2. Programs must have an agreed to ToC before implementation or as part of the design. Indicators, targets and program planning should occur against this ToC. Programs should continue to test assumptions and amend the ToC as evidence is gathered to confirm/deny assumptions, change pathways and hypothesis. 3. Benchmark key indicators answering the question “what is adequate performance”. For example, how many pages of the workbook should be completed and by when? 4. Track indicators that can be used to track implementation progress and results from the beginning of the program. 5. Develop targets per indicator and review progress against targets. 6. Ensure the M&E system builds in testing learners as an integral part of the program and of the MEL system. These testing results should be provided to teachers, schools, coaches and districts. 7. Ensure there are output, outcome and impact indicators and targets for all the intervention components. Include indicators for both reach and dosage. Quickly adapt the indicators (and associated targets) if the intervention is changed due to disruptions. 8. Create and maintain master lists of all entities / groups and individuals that participate in the intervention. As a minimum requirement, a school master list, teacher master list and coach master list should be created and maintained for the duration of the program, ideally through a relational database. Start and end dates as well as the status of the participant (Active, inactive) is important to ensure new participants are included in all communication and program activities. Entities that leave the intervention should be flagged as inactive. 9. Develop a practical and efficient data management system – this includes both processes and systems. Document all data collection tools, processes, responsibilities and frequencies. 10. Decide on a system to use for data capture, storage and retrieval, and ensure that appropriate back-up and recovery plans are in place to reduce the risk of losing data and audit trails recording when data is changed. 11. Pay attention to data security issues and the protection of personal information. Data should be stored in a secure manner and should be easy to access and retrieve if required. A back-up and recovery plan will reduce the risk of losing data. 12. Require standardized indicators, a daily online reporting system and agreed unit of monitoring (under RSP the denominator for many of the indicators were inconsistent – teacher, school, classrooms, coaches). Examples of standardized indicators would include: a. Frequency of coach visits to each teacher by each school b. Complete files per teacher compiled by coach documenting the teacher’s developmental journey with the coaches help Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 126 c. Evidence of implementation fidelity by teacher and by language. 13. Assign data ownership and responsibilities for data collection, analysis, reporting and how data will be used at the start of the intervention. 14. Develop quality assurance processes and assign responsibilities to specific staff. 15. Draft job descriptions that include data ownership, data processing (data entry, data quality assurance, analysis reporting, and storage) responsibilities and criteria for data quality. Monitor these responsibilities through performance appraisals. 16. Ensure that training on the MEL system is provided to all those who will be responsible for capturing the information and using the insights from the data. 17. Log and track recommendations and their status in the data management plan. 18. Use baseline data collected for diagnostic purposes to guide differentiated support according to the reading level of learners in the school, and teacher competencies Costing Recommendations for DBE, PED and implementers: 1. Set up, and regularly, review early grade reading program budgets based on actual costs and benchmarks 2. Agree on budgets prior to or at start of intervention 3. Define budget line items clearly and include unit costs 4. Document clearly any changes to line items in an audit trail 5. Establish accurate financial management systems to monitor and track expenditure against actual activities Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 127 REFERENCES Buckley, P., Moore, B., Boardman, A. G., Arya, D. J., & Maul, A. 2017. Validating a Fidelity Scale to Understand Intervention Effects in Classroom-Based Studies. American Educational Research Journal, 54(6), 1378 -1413. https://doi.org/10.3102/0002831217726522 Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. 2007. A conceptual framework for implementation fidelity. Implementation science: IS, 2, 40. DOI: 10.1186/1748- 5908-2-40 Century, J., Rudnick, M., & Freeman, C. 2010. A Framework for Measuring Fidelity of Implementation: A Foundation for Shared Language and Accumulation of Knowledge. American Journal of Evaluation, 31(2), 199–218. https://doi.org/10.1177/1098214010366173 Department of Basic Education, National Assessment Circular 02 of 2020, 09/07/2020 Gustafson, M. 9 December 2020. The argument for returning to full daily attendance. Not published, received from the Department of Basic Education. Hoadley, U. November 2020. Schools in the Time of COVID-19: Impacts of the Pandemic on Curriculum. RESEP NON-ECONOMIC WORKING PAPER. Cape Town: South Africa. Hofmeyer, J. 2019 Literature review of Coaching in Education: From ‘Knowing to Doing’. Zenex Foundation Report. South Africa Khulisa Management Services, November 2019, Design Evaluation of the Reading Support Project commissioned by USAID https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf McLean, R. and Gargani, J. 2019. Scaling Impact; Innovation for the public good. Routledge. Mohohlwane, N., Taylor, S., and Shepherd, D. September 2020. COVID-19 and basic education: Evaluating the initial impact of the return to schools. National Income Dynamics Study (NIDS) – Coronavirus Rapid Mobile Survey (CRAM) Wave 2. Paper 11. Pretoria; South Africa National Evaluation Policy Framework (NEPF). 2011. Department of Basic Education, South Africa The National Income Dynamics Study – Coronavirus Rapid Mobile Survey (NIDS-CRAM) Wave 2; 2020 SA Education Stats, 2016. https://www.education.gov.za/Portals/0/Documents/Publications/Education%20Statistic%20 SA%202016.pdf?ver=2018-11-01-095102-947 Taylor, S., Cilliers, J., Prinsloo, C., Fleisch, B., & Reddy, V. 2017. The Early Grade Reading Study: Impact evaluation after two years of interventions. Retrieved online https://www.jet.org.za/clearinghouse/projects/primted/resources/language-and-literacy￾resources-repository/egrs-technical-report-13-oct-2017.pdf Van der Berg, S & Spaull, N. 2020. Counting the Cost: COVID-19 school closures in South Africa & its impacts on children. Research on Socioeconomic Policy (RESEP). Stellenbosch University. Stellenbosch. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: SUMMATIVE EVALUATION 128 Venter, F.H.J & Sharif, S. (2020) Analysis of the financial implication of scaling an early grade reading program in South Africa. DBE and UNICEF: South Africa ANNEX 1: SCOPE OF WORK AND AMENDMENTS 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 4 of 41 SECTION C – DESCRIPTION/ SPECIFICATIONS/ STATEMENT OF WORK C.1 INTRODUCTION Despite the government of South Africa’s (GoSA) large investment in basic education – ZAR 223.891 billion (approximately $15.99 billion) in 2017/18 – roughly 17% of the national budget – South Africa continues to face challenges providing a quality education in the majority of the country’s schools and its education indicators continue to lag behind that of its peers. In international comparative reading tests South Africa consistently performs at the bottom with nearly 80% of Grade 4 students unable to read with comprehension in the language of their choice (PIRLS 2016). The GoSA considers education to be one of its highest domestic priorities and one of the greatest long-term challenges facing the country, as is evident in the National Development Plan of which the number one outcome is improving the quality of basic education (DBE, 2013). To support the GoSA, USAID/SA is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve reading skills of primary grade learners which, at the time of publication, was in line with the continuation of Goal 1 of USAID’s Global Education Strategy, aiming for 100 million children worldwide with improved reading skills. The newly published US Government Basic Education Strategy (2019-2023) also prioritizes improved quality of instruction in basic education, and USAID’s new Education Policy (November 2018) continues to prioritize improved early grade reading outcomes. PERFORM will contribute to these shared goals through implementing education interventions in support of three objectives: ● Improved primary grade reading instruction; ● Increased quality of educational administration and support; and ● GoSA support for reading initiatives built at district, provincial and/or national level. PERFORM will use a demonstration effect and rigorous evaluation to support GoSA in bringing proven education solutions to scale, thus increasing the impact and value of national investments in the education sector. This will be done by piloting and testing local solutions which show promise to improve the reading skills of primary grade learners. Close collaboration with GoSA will aim to support buy-in for new reading interventions and to ensure GoSA has the data necessary to decide how best to take successful, cost-effective literacy improvement interventions to scale across the country. In order to deliver rigorous evaluations under PERFORM, the PERFORMANCE Indefinite Delivery Indefinite Quantity (IDIQ) single holder contract was awarded to Khulisa 1 UNICEF, “Education Budget South Africa 2017/2018” United Nations Children’s Fund, Brooklyn Pretoria, P. 2 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 5 of 41 Management Services to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID/SA in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from interventions. The contractor will also be responsible for relaying this information to GoSA, the education research community, and other education sector stakeholders. This dedicated education sector evaluation IDIQ contract will provide cohesion across all evaluations of PERFORM activities, as well as a meta-analysis of outcomes across the project. It will also allow for the flexibility to adapt to changes and challenges in PERFORM activities and to address additional reading related research and analysis requests from USAID that develop during the course of PERFORM’s implementation, both in response to the needs of DBE and otherwise. PERFORMANCE will help to fill a critical research gap by providing rigorous analysis in target areas related to improving the quality of language and literacy skills of primary grade learners in South Africa and the region. USAID/SA found there is little data available on the impact of teacher training programs on student learning outcomes, including literacy. There is also little rigorous analysis available on the effectiveness of school principals, district authorities, communities, and parental engagement to increase student learning. Furthermore, research studies on the cost-effectiveness of quality teacher training and support, school management and administration is even more limited, especially that which estimates the cost￾effectiveness of various types of programming in developing countries. In general, the quality and methodological rigor of the research that is available varies so significantly that it does not offer a sound empirical base for program design in South Africa. Rigorous research that does exist is so context-specific that it is not necessarily generalizable. PERFORMANCE aims to fill this crucial research gap with rigorous evaluations, studies and assessments. The IDIQ contract has three distinct objectives: ● Objective 1: Design and conduct evaluations, surveys, studies and sector assessments to inform and improve education sector activities. ● Objective 2: Make recommendations for viable future education sector activities. ● Objective 3: Strengthen the community of practice surrounding education research, monitoring, evaluation, and learning for languages and literacies. This task order under PERFORMANCE will require the contractor to conduct a design and implementation evaluation on a USAID funded activity titled the Reading Support Project (RSP) implemented by Foundation for Professional Development (FPD). C.2 BACKGROUND History of Early Grade Reading Study (EGRS) In order to address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province of South Africa. The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades. The three EGRS interventions include: ● A teacher training intervention: The first intervention provided teachers with lesson plans aligned to the National Curriculum Statement Grades R-12 (NCS) 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 6 of 41 including the Curriculum and Assessment Policy Statements (CAPS), as well as additional quality reading materials and training at centralized workshops twice a year. ● An on-site teacher training and coaching intervention: The second intervention (implemented in a different group of 50 schools) provided teachers with the same set of lesson plans and reading materials as the first intervention but additionally provided ongoing support to teachers through specialist on-site coaching and small cluster training sessions. ● A parental intervention: The third intervention (implemented in a different group of 50 schools) involved weekly meetings with parents to discuss the importance of learning to read in the early grades and to empower parents with the knowledge and tools to enable them to become more involved in their child’s literacy development. The core of the EGRS project is a comparison of the cost-effectiveness of three promising interventions to improve reading outcomes in learners’ home language (Setswana). Each intervention was implemented in a separate group of 50 schools with a further 80 control schools where ordinary schooling continued (230 quintile 1 – 3 schools). A formal impact evaluation methodology known as a Randomised Control Trial (RCT), complemented by a 60- classroom observation study and eight detailed case studies, enabled the researchers to estimate the impact of each intervention on measures of reading, and to understand where, how and why different elements of the intervention models worked or did not work. Findings from the EGRS revealed, “…small to moderate impacts of both the Training and Coaching interventions on Setswana reading outcomes at the end of Grade 1” after one year of intervention (Department of Basic Education, 2017). After two years of intervention, the findings showed a substantial positive impact of the coaching intervention while the training and parental involvement interventions showed small positive impacts (less than half the size of the coaching intervention). In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention which showed substantial positive impact. In response to this request, USAID and FPD modified the Reading Support Program (RSP) to include the selected EGRS components. This modification was finalized in October 24, 2018. Currently through the RSP, the successful on-site coaching program is being implemented in 165 of the original 230 schools and it is planned that all schools in the districts of Ngaka Modiri Molema and Dr Kenneth Kaunda (287 schools) will be provided with the basic learning program of lesson plans, additional reading materials, and teacher training. PERFORMANCE IDIQ On November 8, 2017, the PERFORMANCE single holder IDIQ contract was awarded for an ordering period of five years. Objective 1 in the IDIQ contract is for the contractor to research and conduct evaluations, surveys, studies and sector assessment to inform and improve education sector activities. In line with this first objective, USAID, along with its partner the South Africa Department of Basic Education, desires a design and implementation evaluation on the USAID funded activity titled the Reading Support Project (RSP). A design evaluation is important towards understanding both how an intervention works and whether it is likely to achieve its outcomes and eventual impact while an implementation evaluation, as defined by South African National Evaluation Policy Framework (NEPF, 2011), “aims to evaluate whether an intervention’s operational mechanisms support achievement or not and understand why. It looks at activities, outputs, and outcomes, use of resources and the causal links. It 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 7 of 41 builds on existing monitoring systems, and is applied during program operation to improve the efficiency and efficacy of operational processes. It also assesses the quality of the indicators.” Since early EGRS impact results showed substantial impacts on learner results through coaching and provision of learning materials, it is important to evaluate the design and implementation of RSP as it scales up this component of EGRS. Can this intervention maintain its positive impacts? Evaluation of RSP’s implementation is critical to understanding the impact data which will be collected at a later date. Also, by conducting a design and implementation evaluation, formative feedback to the program implementers could be provided to ensure that any deviations from the program are identified and addressed. Additionally, USAID could determine whether the RSP program is being delivered with fidelity, learn about the unique challenges that are being introduced through the scaling up of this project, gain information on an impact evaluation by triangulating data on the quality of services provided with findings on the impact of those services. C.3 PURPOSE AND OBJECTIVES Purpose USAID is seeking to explore whether the EGRS can be effectively scaled up. The modified RSP has been designed to test this hypothesis. However, the RSP is implemented by a different consortium of service providers than the original EGRS and it includes additional components so that RSP is not purely EGRS. Also, RSP is implementing the EGRS components in two languages, Setswana and English as a First Additional Language (EFAL). For this reason, it is necessary to clarify the design of the RSP before conducting an implementation evaluation. The purpose of the design evaluation is to: 1) Document the theory of change of the RSP. 2) Assess if the RSP design departs in material ways from the EGRS. 3) Assess if the adapted theory of change is likely to result in the anticipated outcomes.. Following the design evaluation, an implementation evaluation will be conducted to: 1) Provide formative feedback for USAID and the program implementers to ensure that any deviations from the program are identified and addressed. 2) Establish whether the RSP program is being delivered with fidelity. 3) Learn about the unique challenges that are being introduced through scaling up. 4) Inform an impact evaluation by providing the possibility for triangulating data on the quality of the interventions with findings on the impact of those interventions. 5) Assess whether the underlying assumptions used to develop the ToC are still valid Objectives The objectives of the Design and Impact Evaluation are as follows: 1) To identify any risks to the successful implementation of the RSP 2) To provide early findings to strengthen the implementation of the RSP 3) To ascertain whether the allocated time and resources were sufficient to successfully implement the RSP 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 8 of 41 4) To determine whether the FPD Consortium is implementing its activities as planned 5) To determine whether impact-level change can realistically be expected with RSP’s theory of change, its design and within the timeframe of the RSP implementation. 6) To explore whether and how the RSP theory of change could be strengthened to produce the intended outcomes 7) To validate the data collected through the RSP M&E system 8) To determine whether teacher training and coaching is leading to any early changes in teaching methods (e.g. group guided reading, phonics, use of chapters in lesson plans to document implementation) 9) To ascertain what dosage of support teachers are receiving from coaches (i.e. number of visits, time spent per visit) 10) To determine whether coaches are supporting teachers and reporting on their coaching visits 11) To determine the contribution of volunteering in generating change in school management 12) To determine whether the SMT support can be linked to changed support practices in school. C.4 EVALUATION QUESTIONS For the design and implementation evaluation, the following set of questions, with allocated percentage ranked on importance, are proposed to be answered in the formative and summative reports, as appropriate: Design (10%) 1) Is the RSP theory of change likely to lead to the anticipated results? a) What is the impact theory of change? b) What is the process theory of change? c) What are the critical assumptions in the theory of change? d) How does the design differ from the EGRS? e) Is the design uniform or customized for different schools (functionality, level of engagement)? f) Does the design make provision for exit and sustainability? Fidelity (50%) 2) To what extent did the FPD consortium implement the intended intervention design with fidelity? a) Did the Training / Coaching / SMT Training / District Training / Volunteer support happen as planned (i.e. the anticipated number of contact sessions, the specified duration, targeted number of participants, quality of input, within the planned time frame). 3) Is there evidence of RSP uptake by teachers in class? a) Is there evidence of RSP practices being implemented in class by teachers? b) Are the RSP LTSM and classroom libraries 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 9 of 41 i) Available for teachers and learners to use in the classrooms? ii) Is there evidence that the RSP LTSM and classroom libraries are being used as intended. iii) Is there a working system for students to check out books from the classroom libraries? iv) Is the quality of the materials durable enough for the classroom environment? 4) Is there evidence of changed practices based on the SMT support? 5) Were there any major divergence from the assumptions in the theory of change? 6) How could the management and implementation structures be improved? 7) How could the M&E systems be improved? Scalability (20%) 8) What are the unique challenges of delivery at scale with regards to a) RSP Teacher Training (material development, logistics, quality control) b) RSP Coaching (recruitment, selection, training, and supervision of coaches, scheduling and conducting coaching sessions) c) RSP Materials (development, printing, distribution, use) d) Involving the Provincial / District officials Costs (20%) 9) Is the RSP cost efficient? 10) What is the actual vs. anticipated unit cost? 11) How does the RSP unit costs compared to the EGRS unit costs and how are they different? 12) What time commitment is required from other stakeholders to implement at scale? 13) Which efficiencies could have been realized? 14) Which economies of scale could be realized in Year 2 and beyond? C.5 EVALUATION DESIGN AND METHODOLOGY The contractor is requested to propose a robust and detailed evaluation design and methodology. However, USAID does require the following components to be included in the proposed evaluation design. Should the contractor decide not to include one or more of the below components, the contractor must explain the omission. Also, the contractor has the ability to propose additional methods not included on the list below but should explain the benefits of the proposed method and how it compliments the mix of methods already requested by USAID towards achieving the stated objectives. Proposed evaluation design and methodology must include: 1) A full design review – consisting of a comprehensive desk review of all design documents as well as the involvement of a small panel of experts who will provide an assessment of the degree to which the design is likely to lead to the anticipated results. 2) An expert material review in year 1 and year 2, which includes an expert review of all materials developed, including lesson plans, graded readers, posters, big books and teacher/coach training materials, at the time of the evaluation, and a series of teacher focus groups on all of the materials that are in use at the time of the evaluation. 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 10 of 41 3) A full cost and time analysis towards the end of year 1 and towards the end of year 2 to provide formative input during the planning for year 2, and will check whether the costing and time management options were implemented. 4) Process mapping in year 1 to identify any sources of weakness in the implementation. 5) Review of FPD monitoring data and data quality assessment - A fidelity review through a full data quality audit. It is anticipated that the year 1 review can inform planning for year 2, and the year 2 review can provide an overall assessment of the program. 6) An interrogation of coaching practices, documenting the standard operating procedures of the coaching process, through interviews with coaches, teachers, and trainers including case studies in year 1. This will provide valuable feedback on the practicalities of coaching and the way in which coaches use their time. When it is triangulated with interviews with teachers, an adequate picture of coaching practices can be built. C.6 RESULTS AND DELIVERABLES Please refer to F.2 of this RFTOP for the required deliverables. C.7 CROSS CUTTING THEMES The following guidance is provided with respect to alignment with the US Government Basic Education Strategy 2019-2023, USAID’s Education Policy (November 2018) and Education Evaluation Policy, using local systems, sub-awards and incorporating gender considerations in evaluation activities. Where applicable and feasible for this task order the Contractor must consider all these themes in achieving contract goals and objectives and apply them to the overall performance of the Contract. The plan for doing this should be set out in the proposal and in the evaluation design report. C.7.1 Alignment with the USG Basic Education Strategy (2019-2023), the USAID Education Policy (2018) and USAID’s Education Evaluation Policy It is anticipated that the contract will be financed 100% from Basic Education (BE) funds. Funding for BE activities must meet all statutory requirements and align with the USG Basic Education Strategy available at https://www.usaid.gov/education/usg-strategy and USAID’s Education Policy (2018) and corresponding implementation guidance, including USAID’s Education Evaluation Policy. In particular, direct BE funding must address the six Key Principles of the USAID Education Policy as well as Priorities 1 and 2. Key Principles: ● Prioritize country-focus and ownership ● Focus and concentrate investments on sustainable results ● Strengthen systems and develop capacity ● Work in partnership and leverage resources ● Drive decision-making and investments using evidence and data ● Promote equity and inclusion 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 11 of 41 Relevant priorities: ● Children and youth, particularly the most marginalized and vulnerable, have increased access to quality education that is safe, relevant, and promotes social well-being. ● Children and youth gain literacy, numeracy, and social-emotional skills that are foundational to future learning and success. C.7.2 Using Local Systems The Contractor must engage local institutions in every step of implementation when feasible, building technical and management capacity within targeted institutions and communities where possible, working with or through host country systems when appropriate, and transferring managerial and activity implementation responsibility to local institutions where feasible. C.7.3 Sub-awards A means of establishing partnerships with local institutions is through the use of sub-awards. C.7.4 Incorporating Gender in Design Activities The contractor shall ensure that relevant gender issues are explored through the design process, as appropriate. The contractor shall further ensure that all evaluation topics or research questions that result from the design process are sensitive to gender. All proposed topics must require that the data be disaggregated by sex to enable analysis on relevant gender issues in language education and outcomes and to answer other relevant gender-based evaluation questions. C.8 GENERAL MANAGEMENT AND ADMINISTRATIVE APPROACH The contractor must provide all general management and administrative support necessary to perform the contract and achieve the above results. This includes, but is not limited to, the following: 1. The contractor will provide overall management and administration of the contract, including home office support and administrative services. The contractor will provide both the key personnel specified in the contract and additional personnel, long-term and short￾term, necessary to meet recurring general management and administrative support needs under the contract. 2. The contractor will procure or lease facilities, supplies and services as necessary to perform the contract. 3. The contractor will provide the planning necessary for performance of the contract. 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 12 of 41 4. The contractor will get the necessary ethical clearance and permission from authorities to conduct research in government sites, particularly where students will be involved. Requirements for ethical clearance may vary by province and target provinces are not yet determined. The contractor will be responsible for investigating and fulfilling the requirements for such clearance. 5. The contractor will provide oversight, quality control, and general technical support of all services and deliverables provided pursuant to the contract. This includes the provision of copywriters/editors that are familiar with US American English and US American report writing standards. 6. The contractor will provide and assure the proper, efficient, and uniform use of modern management and accounting practices, information technology (IT), communications, reporting, human resource management, property control, security, records, and other administrative processes and systems required under the contract. C.9 STANDARDS OF PERFORMANCE This section defines the performance requirements to which the contractor shall be held, establishes the performance levels or standards, and defines how these performance standards will be measured and verified. The contractor will be continually evaluated against these standards. Key Performance Indicators and Assessment Plan CONTRACTOR EXPECTATIONS KEY MEASURES OF PERFORMANCE PLAN FOR VERIFYING PERFORMANCE i.Technical Quality of Service The Contractor will deliver professional, high quality services that responds to the contract requirements. Consistency and accuracy of the Contractor’s work is demonstrated at all times throughout the period of performance of the contract, including quality of deliverables. Reports and presentations to USAID and other counterparts reflect professional quality standards in writing, data collection, and analysis. Official acceptance of submitted reports, and other deliverables by the COR via written correspondence, copies saved in COR file. ii. Schedule The Contractor’s Results and deliverables are performed and/or delivered to USAID according to the specified Deliverables and task order results are completed by dates identified in Section F. Date of receipt of submitted reports and presentations via physical delivery or electronic 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 13 of 41 timeline. The Contractor will provide timely answers and feedback on all requests, issues and or questions raised by the COR and/ or the CO. No more than fifteen calendar days transpires between submission of comments, questions, or issues to Contractor and a response received by USAID. submission, copies saved in COR file. iii. Cost Control The Contractor’s work plan (Inception report) and budget are adequate and result in the completion of all deliverables and tasks as outlined in Section C. The Contractor must provide the deliverables or outputs described in Section C and F and comply with all contract requirements, performing to the highest standards under the terms of the Contract. Completion of Contract without any modifications or amendments related to task order price due to contractor performance. iv. Business Relations Sound working relationship between the Contractor and USAID/South Africa Avenues of communication are clear; Effective communication practices and team management are evident and there are no documented instances of problems arising due to management of key personnel or the team; Logistical aspects pertaining to Section C tasks and deliverables are clearly designed, well￾thought out processes, are organized and implemented so that tasks and deliverables are efficiently completed. No documented problems or issues arise due to Key personnel management or communication issues; No documented problems or issues arise due to logistical issues (within the control of the Contractor) as documented by the COR. Performance Evaluation Criteria, Ratings and Standards 1. Exceptional Performance meets contractual requirements and exceeds many to USAID/South Africa’s benefit. The contractual performance of the required results was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective. 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 14 of 41 2. Very Good Performance meets contractual requirements and exceeds some to USAID/South Africa’s benefit. The contractual performance of the required results were accomplished with some minor problems for which corrective actions taken by the Contractor were effective. 3. Satisfactory Performance meets contractual requirements. The contractual performance of the required results contains some minor problems for which corrective actions taken by the Contractor appear or were satisfactory. 4. Marginal Performance does not meet some contractual requirements. The contractual performance of the required results reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor’s proposed actions appear only marginally effective or were not fully implemented. 5. Unsatisfactory Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the required results contains a serious problem(s) for which the Contractor’s corrective actions appear or were ineffective. C.10 PROVIDE DATA, TECHNICAL MATERIALS, AND OTHER INFORMATION The Contractor will provide USAID with data, technical materials, and other relevant materials produced in the execution of this contract in line with USAID’s Open Data Policy as outlined in ADS 579 http://www.usaid.gov/sites/default/files/documents/1868/579.pdf. This includes pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives, as well as data and information needed for reporting under the relevant foreign assistance objectives, areas and elements. Pedagogical Materials and Technical Inputs When applicable the Contractor must provide pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives. Examples of technical inputs to be provided to USAID include scripted lesson plans, supplementary readers, assessment instruments, observation tools, training guides, workshop reports, radio programs, assessment tools, sampling frames, photographs, videos, and other recordings. The Contractor must transmit technical materials to the relevant TOCOR and submit them to the USAID Development Experience Clearinghouse (https://dec.usaid.gov/). Data for Reporting Under Foreign Assistance Objectives When applicable the Contractor is required to provide datasets and codebooks that include data on student learning outcomes and information needed to estimate the number of unique pupils benefiting from program activities over the life of the program. The Contractor may be responsible, in collaboration with USAID, for obtaining country level memoranda of 72067419RFTOP00003 Design and Implementation Evaluation of the Reading Support Project PERFORMANCE IDIQ 72067418D00001 Page 15 of 41 understanding that allow for the sharing of the datasets and other data with USAID, as well as public access to the data through the partner organization, where possible. Within 90 days of the completion of data collection, the Contractor must transmit requested data to USAID. The transmittal shall be according to the following specifications: ● Datasets should be complete, clean, and final, and include any derived or secondary variables used to calculate indicator values provided in assessment reports. ● Datasets must be cleansed of Personally Identifiable Information (PII) prior to transmittal to USAID. PII includes any information that could be used to identify an individual student, teacher, or administrator for whom data have been collected. ● Datasets will include all variables included in the initial data collection, with the exception of any data that must be edited or cleaned to protect the privacy and anonymity of students, teachers, or administrators represented in the data. ● If variables are edited or removed in order to protect the privacy and anonymity of research subjects, steps should be taken to ensure that sufficient information is retained to allow analyses that require grouping students by school, or track schools/students across datasets if appropriate. ● Data must be transmitted along with relevant supporting materials and instruments. This includes questionnaires and other instruments, codebook, data dictionary, information on sample design, setup and weights, assessment reports, performance management plans or other materials that describe the structure of the assessment and/or program, and any other information a researcher may need when working with the data. ● Learning Assessment data can be transmitted in formats including Stata, SPSS, SAS, R, or an open and machine readable format. Supporting documents can be transmitted in MS Office or an open and machine readable format. ● The Contractor must provide information on the number of pupils benefiting from the program, disaggregated by sex and grade for each year that the program is active ● Datasets will be delivered through email, addressed to the relevant TOCOR. The Contractor may also be directed by USAID to submit data and related documents to a third party site (e.g. https://sartdatacollection.org) ● All prerequisites to providing the complete, cleaned datasets must be completed by the Contractor prior to the provision of the dataset to USAID, such as review and approval by Missions and host country governments, as appropriate. C.11 MATERIALS Materials developed under this contract are subject to FAR 52.227-14 (RIGHTS IN DATA – GENERAL), and the Government of South Africa and others will be granted a paid-up, nonexclusive, irrevocable, worldwide license (under the Creative Commons By “CC BY” http://creativecommons.org or otherwise as USAID may determine) to reproduce, prepare derivative works and distribute copies to the public. Any material not first produced in the performance of this contract is subject to clause FAR 52.227-14(c)(2). [END OF SECTION C] Modification No. 02, 72067419F1900015 MODIFICATION OF CONTRACT Page 1 of 2 1. MODIFICATION NUMBER 2 2. EFFECTIVE DATE OF See Block 15 3. CONTRACT NUMBER: 72067419F00015 4. EFFECTIVE DATE OF CONTRACT: 06/21/2019 5. GRANTEE: Khulisa Management Services Pty(Ltd) 26 7th Avenue Parktown North, 2193 Johannesburg, South Africa DUNS NO. : 636714107 TIN NO. : LOC NO. : 6. ADMINISTERED BY: Office of Acquisition and Assistance USAID/Southern Africa 100 Totius Street Groenkloof, 0181 Pretoria, South Africa 7. FISCAL DATA: Amount Obligated: $0.00 Budget Fiscal Year: Operating Unit: REED/EDU Strategic Objective: Team/Division: Benefiting Geo Area: Object Class: 8. TECHNICAL OFFICE: USAID/SA/REED 9. PAYMENT OFFICE: Office of Financial Management-Controller USAID/Southern Africa P.O. Box 43 Groenkloof 0027 Pretoria, South Africa 10. FUNDING SUMMARY: Obligated Amount Total Est. Amt. Amount Prior to this Modification: $849,790.00 $849,790.00 Change Made by this Modification: $0.00 $ 0.00 New/Current Total: $849,790.00 $849,790.00 11. DESCRIPTION OF MODIFICATION: The purpose of this modification is to: a) Amend objective 12 of the statement of work; b) Add an evaluation question; and c) Change the key personnel. As a result, the task order is modified as follow: (CONTINUATION OF MODIFICATION ON PAGE 2) 12. THIS MODIFICATION IS ENTERED INTO PURSUANT TO THE AUTHORITY OF FAA of 1961 AS AMENDED. EXCEPT AS SPECIFICALLY HEREIN AMENDED, ALL TERMS AND CONDITIONS OF THE GRANT REFERENCED IN BLOCK #3 ABOVE, AS IT MAY HAVE HERETOFORE BEEN AMENDED, REMAIN UNCHANGED AND IN FULL FORCE AND EFFECT. 13. CONTRACTOR: X IS EFFECTED HEREIN IS NOT REQUIRED TO SIGN THIS DOCUMENT TO RECONFIRM ITS AGREEMENT WITH THE CHANGES 15. THE UNITED STATES OF AMERICA U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 14. GRANTEE: (Name Typed or Printed) BY: TITLE: DATE: 12/8/2020 | 14:52 SAST Director Michael Ogawa BY: (Name Typed or Printed) TITLE: DATE: Caroline Hillas Regional Contracting Officer Modification No. 02, 72067419F1900015 Page 2 of 2 1) Section C3. OBJECTIVES AND RESULTS, objective 11 DELETE “To determine the contribution of volunteering in generating change in school management” and REPLACE with “To determine the contribution of the School Management Team intervention in generating change in school management”. 2) Section C4. EVALUATION QUESTIONS, Fidelity, ADD an additional question “To what extent did COVID-19 affect the implementation of the RSP and how did the program respond and adapt?”. 3) Section F.4 KEY PERSONNEL, Senior Project Manager, DELETE Ms. Katharine Tjasink and REPLACE with Ms. Margaret Roper. Except as herein modified, all other terms and conditions remain unchanged and in full force and effect. [END OF MODIFICATION No. 02] Modification No. 03, 72067419F00015 MODIFICATION OF CONTRACT Page 1 of 2 1. MODIFICATION NUMBER 3 2. EFFECTIVE DATE OF See Block 15 3. CONTRACT NUMBER: 72067419F00015 4. EFFECTIVE DATE OF CONTRACT: 06/21/2019 5. GRANTEE: Khulisa Management Services Pty(Ltd) 26 7th Avenue Parktown North, 2193 Johannesburg, South Africa DUNS NO. : 636714107 TIN NO. : LOC NO. : 6. ADMINISTERED BY: Office of Acquisition and Assistance USAID/Southern Africa 100 Totius Street Groenkloof, 0181 Pretoria, South Africa 7. FISCAL DATA: Amount Obligated: $0.00 Budget Fiscal Year: Operating Unit: REED/EDU Strategic Objective: Team/Division: Benefiting Geo Area: Object Class: 8. TECHNICAL OFFICE: USAID/SA/REED 9. PAYMENT OFFICE: Office of Financial Management-Controller USAID/Southern Africa P.O. Box 43 Groenkloof 0027 Pretoria, South Africa 10. FUNDING SUMMARY: Obligated Amount Total Est. Amt. Amount Prior to this Modification: $849,790.00 $849,790.00 Change Made by this Modification: $0.00 $ 0.00 New/Current Total: $849,790.00 $849,790.00 11. DESCRIPTION OF MODIFICATION: The purpose of this modification is to extend the period of performance by six weeks from March 31, 2021 to May 14, 2021. As a result, the task order is modified as follows: (CONTINUATION OF MODIFICATION ON PAGE 2) 12. THIS MODIFICATION IS ENTERED INTO PURSUANT TO THE AUTHORITY OF FAA of 1961 AS AMENDED. EXCEPT AS SPECIFICALLY HEREIN AMENDED, ALL TERMS AND CONDITIONS OF THE GRANT REFERENCED IN BLOCK #3 ABOVE, AS IT MAY HAVE HERETOFORE BEEN AMENDED, REMAIN UNCHANGED AND IN FULL FORCE AND EFFECT. 13. CONTRACTOR: X IS EFFECTED HEREIN IS NOT REQUIRED TO SIGN THIS DOCUMENT TO RECONFIRM ITS AGREEMENT WITH THE CHANGES 15. THE UNITED STATES OF AMERICA U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT 14. GRANTEE: (Name Typed or Printed) BY: TITLE: DATE: Michael Ogawa 3/30/2021 | 17:06 SAST Director BY: (Name Typed or Printed) TITLE: DATE: Caroline Hillas Regional Contracting Officer Page 2 of 2 1) Cover page, No.8 Estimated Completion Date, DELETE March 31, 2021 and REPLACE with May 14, 2021. 2) Section F2 Period of Performance DELETE March 31, 2021 and REPLACE with May 14, 2021. 3) Section H, ADD the following Requirement: PROCUREMENT OF “COVERED MATERIAL” (June 2020) 1. Except as provided in paragraph 2 below, and notwithstanding anything in this contract to the contrary, no funds under this contract] may be used for the procurement of “Covered Material” as listed below without the prior written consent of the Contracting Officer. For purposes of this special contract requirement, “Covered Material” shall consist of the following: ● N95 Filtering Facepiece Respirators, including devices that are disposable, half-face-piece, nonpowered, air-purifying particulate respirators intended for use to cover the nose and mouth of the wearer to help reduce the wearer’s exposure to pathogenic, biological, airborne particulates; ● Other Filtering Facepiece Respirators (e.g., those designated as N99, N100, R95, R99, R100, P95, P99, or P100), including single-use, disposable, half-mask respiratory protective devices that cover the user's airway (nose and mouth) and offer protection from particulate materials at an filtrationefficiency level equivalent to an N95 filtering facepiece respirator according to Section 84.181 of Title 42 of the Code of Federal Regulations (CFR); ● Elastomeric, air-purifying respirators and appropriate particulate filters/cartridges; ● PPE surgical masks, including masks that cover the user's nose and mouth and provide a physical barrier to fluids and particulate materials; ● PPE gloves or surgical gloves, including those defined at Sections 880.6250 (exam gloves) and 878.4460 (surgical gloves) of Title 21 of the CFR and such gloves intended for the same purposes; ● Ventilators; and ● COVID-19 test kits that are meant for the United States market. For clarity, non-medical grade masks, including cloth masks, are not included in the list of Covered Material above. Further, USAID may modify the list of Covered Material from time-to-time, in writing; any such changes to the list shall apply prospectively. 2. The restrictions set forth in paragraph 1 above shall not apply to the procurement of Covered Material: (a) for the protection of and use by the Contractor’s or subcontractor’s staff; or (b)for the safe and effective continuity of USAID-funded programs, including for the protection of beneficiaries, provided that such items are manufactured locally or in the same geographical region as the country in which USAID is providing assistance, as defined by the U.S. Department of State’s regional system (Africa, East Asia and the Pacific, Europe and Eurasia, Near East, South and Central Asia, and Western Hemisphere), and provided that such items are not, and could not reasonably be expected to be, meant for the United States market. The CO may change the exemptions set forth in this paragraph in writing; any such changes shall apply prospectively. Page 3 of 3 3. “Staff” for the purposes of the Exception in 2(a) is defined as any individuals receiving financial compensation from the Contractor or subcontractor. 4. For each purchase of Covered Material under Exception 2(b), the Contractor must provide the CO with contemporaneously dated documentation that the order of Covered Material is not meant for, and could not reasonably be meant for, the U.S. market. The CO must then upload the statement into ASIST. This documentation can take the form of a simple email verification from a vendor or a brief, contemporaneously dated, written statement or e-mail from the Contractor confirming its conversation with the vendor. (End of Special Contract Requirement) Except as herein modified, all other terms and conditions remain unchanged and in full force and effect. [END OF MODIFICATION No. 03] IV ANNEX 2: PROFILES OF EVALUATORS Jennifer Bisgard, Project Director Ms. Jennifer Bisgard co-founded Khulisa Management Services in 1993. An expert in M&E and organizational development, she leads evaluations and capacity building assignments in the education, and democracy and governance sectors. She has 20+ years of experience leading evaluations and research assignments for USAID, including impact evaluations, performance assessments, program/project design/management and data analysis, including leading evaluations, such as the USAID/SA Evaluation IDIQ and serving as Project Director on the previous PERFORMANCE Task Orders. Jennifer has co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned” published by Sage Publishers in August 2018. The book is edited by Kylie Hutchinson, with a forward by Michael Quinn Paton. The book features 22 case studies of evaluation failures, including ours which is based in South Africa. Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993. She has served on boards for the: African Evaluation Association (AfrEA), International Organization for Cooperation in Evaluation (IOCE) and South African Monitoring and Evaluation Association (SAMEA). She has a Master’s Degree in Social Change and Development from Johns Hopkins University. Margaret Roper, Senior Evaluator Ms. Margaret Roper fulfils the position of Deputy Director and senior MEL Specialist. She has extensive experience in program development, M&E and knowledge sharing in social policy and practice, social justice, social protection, child protection, school and community health and safety, educational development, and behavior change. Since 1993, Margaret has worked in the non-profit, government, donor and business sectors in Southern Africa. She has held positions in the Secretariat for Safety and Security, the South African Police Service, Health and Development Africa, Mott MacDonald, Inkanyezi Initiative and consulted for a range of organizations including Oxfam Australia, UNICEF and government departments such as Basic Education, Social Development, Correctional Services and Gauteng Department of Community Safety. Margaret belongs to SAMEA, AfrEA and to the International Society for the Prevention of Child Abuse and Neglect (ISPICAN). She has presented at ISPICAN conferences in Turkey (2012) and India (2011); and Pathways to Resilience IV, at an international conference held in Cape Town, 2017, on South Africa’s Response to Enabling Child Resilience Through Expanding Community-based Services She has a Masters in Social Research from the University of Sheffield (United Kingdom), and a Bachelors of Primary Education and Education (Honors) from the University of the Witwatersrand. Benita Williams, Senior Evaluation Specialist Benita Williams is a South African evaluator and current director of Benita Williams Evaluation. Over the past eighteen years, she has conducted various evaluations of education, youth development, income generation and health initiatives in Southern Africa for corporate donors, government departments and community-based NGOs. She has a keen interest in evaluation methods and evaluation design. She has experience in the sophisticated qualitative and statistical analysis of education data, voluntary counselling and testing/ health behavior survey data and skills￾audits. Recently her focus has been on the evaluation of education support initiatives related to early childhood education, whole school development in public schools, and the training of teachers in mathematics, physical science and English subject areas. She is team lead for an evaluation of a Zenex Foundation Secondary School development project across three provinces, and the evaluation of the Anglo American Education program that targets schools and ECD centers in eight areas across the country. She was part of a team that evaluated an early -childhood focused social￾franchise movement and also collaborated with RESEP at the University of Stellenbosch on a Public Expenditure Tracking Study in ECD. Ms. Williams frequently works as Developmental Evaluator and has done evaluations informed by Utilization Focused Evaluation, Outcome Mapping, Realistic Evaluation and Systems Evaluation theories. Ms. Williams has academic training in the field of Research Psychology at Master’s level (University of Pretoria – Dissertation not completed) and is currently pursuing a Master’s in Development Studies at the University of Johannesburg. In 2009-2010, Benita served as executive secretary for the African Evaluation Association, a network of evaluation associations and evaluators across Africa. Benita was the founding treasurer for the South African Monitoring and Evaluation Association (SAMEA) a voluntary association of evaluators and evaluation users in SA, and served until September 2009. Benita has published in peer-reviewed evaluation publications. Daleen Botha, Data Quality Analyst Daleen Botha is a director and evaluator at Benita Williams Evaluation. Her more than 15-year career includes more than ten years’ experience in M&E, mostly working in the education sector. Daleen has worked with government institutions in South Africa such as the South African Council of Educators and the Gauteng Department of Education. She has also worked with community￾based NGOs like the Sumbandila Trust and private foundations such as the Zenex Foundation. She is adept at qualitative and quantitative analyses, with extensive experience of developing M&E frameworks, Theories of Change, Logical Frameworks and Result chains, as well as data and project management. Daleen holds a BSc Honors degree in Computer Science as well as a post-graduate diploma in M&E from the University of Stellenbosch. She is a member of the South African Monitoring and Evaluation Association (SAMEA). Leticia Taimo, Evaluation Coordinator (Mid) Ms. Leticia Taimo joined Khulisa in 2015 as an intern, and is now a Senior Associate with 5+ years’ evaluation, project management and coordination experience. She has successfully participated and coordinated multiple education evaluation, research and assessment plus data collection projects in in South Africa for a variety of stakeholders (private sector, NGOs, government and international donors). In particular, she is developing a strong expertise in Early Childhood Development. She has presented on how to use technology to monitor ECD at the SAMEA Conference 2017, presenting evaluation findings at the “CSI that Works” 2017, facilitating an interactive session on M&E at the Play Conference 2017 and introducing M&E concepts at BRIDGE’s Early Childhood Development Community of Practice in 2018. Leticia has an MSc Development Studies from the School of Oriental and African Studies, University of London, and is also ISO 9001:2008 certified in Data Quality Management Systems Assessment. Leticia was awarded the Mandela Rhodes Scholarship in 2013 and the Commonwealth Scholarship in 2014 as recognition of her commitment to social change in the African continent. Heather Dixon Ms. Heather Dixon is an Associate at Khulisa Management Services. She has worked on evaluation projects in the fields of human trafficking, education, energy and electrification, funded by international donors, government, foundations and NGOs. She has successfully lead and managed remote studies in Kenya and Ethiopia. Ms. Dixon has experience in desktop review/research, tool design, data collection (including interviews and observations), qualitative and quantitative data analysis, report writing and project coordination. Prior to joining Khulisa, she worked on the implementation of establishing a reading coalition to address illiteracy in children in South Africa. This involved stakeholder negotiation, coordination, logistics and data collection and analysis. Heather is an Honors graduate in History and Political Studies (cum laude) from Rhodes University, and is also a Mandela Rhodes scholar. She can speak English and Afrikaans, and has an elementary level of proficiency in French. She resides in Johannesburg and is a South African citizen. David Ndou David Ndou serves as Emergent Evaluator and Associate at Khulisa Management Services’ Health division. He has gained over five years of experience in M&E, Evaluations, Data Quality Assessments (DQAs) and Data Quality Reviews (DQRs), project management and support on assignments for USAID, The Global Fund, and FHI 360. Recent assignments with relevant project assistance and support experience includes the USAID/Zambia Scaling Up Nutrition Learning and Evaluation (SUN LE) contract as well as the DQR for the Global Fund-funded HIV, TB and Malaria grant in Somalia. He also serves as fieldwork coordinator the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) IDIQ, USAID/Southern Africa’s Design and Implementation Evaluation of the Reading Support Project (RSP) task order, as well as the USAID Power Africa East Africa Regional Energy Program (EAEP) task order in Kenya and Ethiopia. David is an Honors graduate in Public Management and Administration from the University of South Africa He is fluent in English and Tshivenda, plus speaks isiZulu. He resides in Johannesburg, South Africa and is a South African national. Carole Metekoua, Statistician Carole Metekoua is a process-oriented data analyst and statistician with four years of experience specific to Monitoring and Evaluation (M&E). She has significantly contributed to learning and evaluation projects as well as research projects by managing, cleaning, transforming, analyzing and visualizing big data in Python, STATA, and R. Her experience working with Khulisa, includes serving as Data Analyst on the current long-term USAID Zambia Scaling Up Nutrition Learning and Evaluation (SUN LE) assignment, developing the data analysis plan, generating key indicators and conducting inferential analyses and interpretation of findings. She also served as Data Quality Reviewer with Khulisa during 2017-2018 to provide quality assurance for Global Fund data quality reviews in Tanzania, Mozambique, Sri Lanka, Indonesia and Burkina Faso. Previously she served as Researcher at Wits Health Consortium, focused on data management, analysis and presentation. She recently completed her Master of Science (MSc) degree in Epidemiology, specializing in Public Health Informatics. ANNEX 3: DISCLOSURE OF CONFLICT OF INTEREST READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Jennifer Bisgard Date October 11, 2019 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Margaret Roper Date October 11, 2019 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Benita Williams Date October 11, 2019 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Daleen Botha Date November 15, 2019 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Leticia Taimo Date October 11, 2019 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Heather Dixon Date March 01, 2021 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name David Ndou Date March 01, 2021 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Carole Metekoua Date November 15, 2019 ANNEX 4: EXECUTIVE SUMMARY OF FORMATIVE EVALUATION 2019 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |i EVALUATION REPORT – EXECUTIVE SUMMARY FORMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA February, 2020 Contract No: 72067418D00001, Order Number: 72067419F00015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd. Contract No: 72067418D00001, Order Number: 72067419F00015 ii READING SUPPORT PROJECT EVALUATION REPORT FORMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA February, 2020 Photo Credit: Brandon Barnard for USAID AUTHORS Jennifer Bisgard (Project Director) Katharine Tjasink (Senior Project Manager) Benita Williams (Senior Evaluator) Leticia Taimo (Evaluation Coordinator) Margaret Roper (Senior Evaluator) Jacqui Dornbrack (Reading Expert) Nombulelo Baba (Coaching Expert) CONTACT DETAILS Katharine Tjasink 26 7th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: ktjasink@khulisa.com Web Address: www.khulisa.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contract No: 72067418D00001, Order Number: 72067419F00015 iii ACKNOWLEDGEMENTS Khulisa Management Services (Pty) Ltd would like to thank the United States Agency for International Development (USAID) for supporting the development of this report through the funding mechanism which has enabled the evaluation to take place and through their ongoing engagement with the evaluation team. Khulisa specifically thanks USAID for their contribution and support in the design and delivery of the evaluation, and their input into the content and review of all reports. Khulisa would also like to thank the Department of Basic Education for their input into the evaluation design and the North West Provincial Education Department for their input and continuing endorsement. Khulisa also thanks the Foundation for Professional Development Consortium for their collaboration, support, and input into various evaluation processes. Finally, Khulisa appreciates the time and effort by all who participated in the evaluation, without which this evaluation would not be possible, including the coaches, teachers, and school management teams that we interviewed in the North West Province of South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 4 EXECUTIVE SUMMARY Background The Reading Support Project (RSP) is an educational program designed to be implemented in 263 schools2 in Dr. Kenneth Kaunda and Ngaka Modiri Molema districts in the North West Province of South Africa over the period January 1, 2019 – September 30, 2020. The Foundation for Professional Development (FPD) implements the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), and one international partner, Voluntary Services Overseas (VSO), referred to in this report as the FPD Consortium. The project is intended as a scale-up of the South African Department of Basic Education’s (DBE) Early Grade Reading Study (EGRS I3 ) which was an randomized control trial (RCT) with 230 Quintile 1 to 3 schools in the same two districts from 2015 - 2017. The RSP focuses on improving the pedagogical practices of Grade 1 to 3 teachers in how they teach reading in Setswana Home Language (HL) and English First Additional Language (EFAL) with the ultimate outcome to improve the reading skills of Foundation Phase learners. The FPD Consortium uses a cascade training approach to roll out the project. Once a term (four times a year), Consortium members train RSP reading coaches, who then train teachers in district-based geographic clusters. One-day dry runs, whereby coaches have an opportunity to practice their newly acquired knowledge and skills, are a feature of this training approach. The RSP provides teacher training, and Learning and Teaching Support Materials (LTSM), to all schools involved in the project. These materials include lesson plans, flash cards, EFAL and HL lesson plans, readers, posters, flash cards, and other supplementary material provided per term. A random selection of schools additionally receive reading coaches, and another set of randomly selected schools receive both reading coaches and support to the School Management Team (SMT). The underlying theory, in summary, is that through the provision of LTSM, training, coaching, and SMT support, teachers receiving the interventions will improve their practices in the classroom. Purpose For the period 2019 - 2020, USAID commissioned a Design and Implementation Evaluation of the RSP. The purpose of the Design Evaluation4 was to ensure that the design of the RSP, as it evolved, was well understood prior to conducting an Implementation Evaluation. The purpose of the Implementation Evaluation is to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, 2 The analysis presented in Annex 8 indicates that in October 2019 FPD was implementing the RSP in 256 schools. 3 The Early Grade Reading Study ran between 2015 and 2017 in North West Province. Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. 4 A separate Design Evaluation Report is available https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf. Contract No: 72067418D00001, Order Number: 72067419F00015 5 and whether there is fidelity to the intended design in the actual implementation of the program. Evaluation Questions The key Implementation Evaluation questions include: To what extent did the FPD Consortium implement the intended intervention design with fidelity? Is there evidence of RSP uptake by teachers in class? Is there evidence of changed practices based on the SMT support? Was there any major divergence from the assumptions in the Theory of Change? How could the management and implementation structures be improved? How could the M&E systems be improved? What are the unique challenges of delivery at scale? These seven questions are answered in this report, with the purpose of informing programming in 2020. An eighth question - Is the RSP cost efficient? - is addressed in the Summative Report5 . Methods The Implementation Evaluation comprises a mixed methods approach involving the collection, collation, triangulation, and valuing of both qualitative and quantitative primary evaluation data. Khulisa collected data from 60 randomly selected schools including 20 schools that are receiving LTSM and teacher training only (LTSM schools), 20 schools receiving LTSM, teacher training and coaching (Coaching schools), and 20 schools receiving LTSM, teacher training, coaching and SMT support (SMT schools). Primary data collection methods include group discussions with 65 teachers in 13 schools, interviews with 15 Provincial and District officials, 157 SMT members, and 58 Principals. All fourteen reading coaches were shadowed and interviewed for a week, and both head coaches were interviewed. Survey data was collected from 304 Foundation Phase teachers. Classroom observations were conducted in 242 Grade 1 and Grade 3 classrooms and the evaluation assessed 238 Grade 1 and Grade 3 learner workbooks (DBE workbooks and other learner books). A data quality assessment was conducted at 58 schools, classroom library data was collected from 123 classrooms, and classroom LTSM was assessed in 126 classrooms. These multiple sources of data were collated and compared. Khulisa used thematic analysis to analyze group discussion data and interview data, and summary statistics to analyze the quantitative data collected through the evaluation. The data was valued by the evaluation team in collaboration with experts (investigator triangulation). Data was collected using several different tools and instruments. 5 To be published on the DEC in May 2021. Contract No: 72067418D00001, Order Number: 72067419F00015 6 Key Findings Evaluation Question 1: To what extent did the FPD Consortium implement the intended intervention design with fidelity? According to the RSP monitoring data, the dosage and/or coverage of the intervention is sufficient in terms of number of teacher training sessions delivered, numbers of participating teachers, and number of SMT training workshops delivered. However, the RSP implementation was below the anticipated dose, per the RSP design specifications, in the following areas: full training dosage (only delivered to about half of the anticipated number of teachers); the targeted number of teachers reached with coaching; the number of individual coaching sessions held with teachers, the anticipated number of SBWs held after school, the number of SMT members trained, and the availability of LTSM. Evaluation Question 2: What evidence is there of RSP uptake by teachers in class? There is some evidence of RSP uptake by teachers in the classroom. However, there are some gaps. Less than 45 percent of all schools have reading mats (not a project requirement but indicative of good teaching practice). Most classrooms have HL and EFAL word walls while all schools have a low rate of displaying children’s work. The practices that slow teaching down were seen most often in the SMT and LTSM classrooms. In all three groups, there was still choral reading, but this was lowest in SMT classrooms. There was great variation across the schools with reference to teaching book knowledge and print concepts. In seven of nine criteria linked to teachers explicitly developing learners’ concepts about print and printed material, the SMT classrooms had the highest number of observations of preferred practices. There was little difference in the teaching of book knowledge and print concepts across Grade 1 and 3, although Grade 3 children do not need to be taught about book covers, spines, front and back and reading direction. There was more use of English in the SMT and Coaching classrooms as compared to the LTSM classrooms. In the SMT classrooms, teachers most often modelled the correct term, highlighted English terms, and encouraged the use of English compared to the other groups. The explicit teaching of vocabulary and spelling was also higher in the Coaching and SMT classrooms than in LTSM classrooms. Teaching of handwriting and copying from the board was low overall across the interventions. Learners writing more than five sentences was observed most often in LTSM classrooms. There was little difference in the observation of teachers reading aloud to children and Group Guided Reading was similarly observed in less than 50 percent of schools across all groups. The use of post-reading questions was highest in the SMT classrooms. Evaluation Question 3: Is there evidence of changed practices based on the SMT support? The classroom observation findings suggest that there is variable uptake of RSP practices, as evidenced in better practices in the classroom, with SMT classrooms providing more opportunities for language and literacy development than the other groups. Contract No: 72067418D00001, Order Number: 72067419F00015 7 Qualitatively, SMT members in almost half the schools (nine out of 20) reported that the SMT training assisted the Head of Departments (HODs) to improve their work in the classrooms and encourage reporting to SA-SAMS on a weekly basis. Evaluation Question 4: Were there any major divergences from the assumptions in the Theory of Change? The Theory of Change (ToC) assumptions surfaced in the Design Evaluation remain relevant and appropriate. In some areas, the assumptions are not being met. This includes in the areas of coach training (coaches were unable to execute dry runs during the last coach training), teacher training (fewer teachers attended than expected), LTSM delivery (not all teachers received LTSM and some teachers did not receive all LTSM), and dosage of coaching support. The coaching shadowing showed that coaches performing good coaching practices were adequately planning and preparing for visits to the schools, and engaged appropriately with the school principal, SMT, and the teachers. Weaker coaches were compliant with reporting but lacked skills and experience. Indications are that the coaches are monitoring and checking teacher implementation against the lesson plans and documenting their observations. Without the school-based workshops (SBWs), the coaching dosage is currently too low to support differential impact. The unmet dosage assumption remains the greatest threat to the RSP ToC. Evaluation Question 5: How could the management and implementation structures be improved? Management and implementation structures must be improved to (1) ensure the proper dosage and coverage is delivered, (2) track implementation and flag issues as they arise, (3) ensure quality of delivery and (4) ensure good reception of the RSP in schools. Evidence from the teacher group discussions, teacher surveys, and classroom observations all found that while each teacher is supposed to receive a full set of lesson plans including all LTSM, not all teachers received all materials. There are not adequate processes in place to check (i) if teachers have received materials in schools where coaching does not take place and (ii) for those teachers who miss training in schools where coaching does not take place, if those teachers have received their materials. The findings also raised a question around whether the LTSM packages contained ALL the RSP LTSM for all grades. The case studies illustrated that the Head Coaches are not providing adequate support to the coaches in the field. Coaches interviews confirmed that nine of the 14 Coaches felt that they did not receive adequate support from their Head Coach. The coach shadowing showed that all 14 coaches were welcome in the schools and were met with cooperation and support from the schools. It was clear that all coaches were familiar with the school environment and had been to the schools before. All 14 coaches acknowledged the positive support from the SMT. Evaluation Question 6: How could the M&E systems be improved? The RSP monitoring and evaluation (M&E) systems need to be improved from indicator design, to tool design, to data collection and management, and reporting. The Monitoring, Contract No: 72067418D00001, Order Number: 72067419F00015 8 Evaluation, and Learning (MEL) plan needs to be updated to reflect the final iteration of the RSP design. Numerous individual improvements are required within the M&E System to produce reliable data for tracking implementation in 2020. Evaluation Question 7: What are the unique challenges of delivery at scale? Delivering the RSP at scale is affected by a number of factors including the extent of teacher training required, the availability of good coaches, and the logistical arrangements around developing, printing, packaging, and distributing LTSM. Scaling the RSP to three grades and two subjects (Setswana HL and EFAL) simultaneously affected the ability to deliver and monitor RSP interventions. Involving provincial and district officials in the RSP is key to long term sustainability. However, this relationship takes time to build and nurture. Involving provincial and district officials in the process is worthwhile but time consuming both for the officials and for the RSP implementers. Recommendations The evaluation report provides specific recommendations for improvement of the RSP in 2020. These are summarized below: − The RSP M&E system needs to be improved, including improvement of data collection tools, data collection and data entry processes, data collation, and reporting. − RSP LTSM need to be quality reviewed and strategies need to be put in place to address the perceived fast pace of the lesson plans. Processes must be put in place to ensure teachers receive all LTSM. − The roles and responsibilities of head coaches should be better defined and there needs to be more strategic engagement between the head coach and coaches. − Coach skills should be improved through intensive training, support, and role clarification. Weaker coaches require targeted support. Monitoring the coaching process closely is key towards improving the dosage and coverage of coach support to teachers. − The RSP needs to deepen its engagement with district officials to ensure their buy in and cooperation, and should increase reporting to district and provincial officers. − To effectively scale the RSP, subject and curriculum advisors should be trained, roles and boundaries should be clarified, and teachers must understand that the RSP is an integral and systemic part of their teaching. There needs to be more focused engagement between the RSP and the DBE before the project is taken to scale. Contract No: 72067418D00001, Order Number: 72067419F00015 ANNEX 5: METHODOLOGY AND STUDY PROTOCOL METHODOLOGY PLAN AND STUDY PROTOCOL Design and Implementation Evaluation of the Reading Support Project 9 September 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd in collaboration with the South African Department of Basic Education Contract No: 72067418D00001, Order Number: 72067419F00015 III METHODOLOGY PLAN AND STUDY PROTOCOL DESIGN AND IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT 9 September 2019 Contract Number: 72067418D00001, Order Number: 72067419F00015 Cover Photo: School Books, Photo Credit: StockSnap, available from Pixabay DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contract No: 72067418D00001, Order Number: 72067419F00015 IV AUTHORS Jennifer Bisgard (Project Director) Katharine Tjasink (Senior Project Manager) Leticia Taimo (Evaluation Coordinator) CONTACT DETAILS Katharine Tjasink 26 7th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: ktjasink@khulisa.com Web Address: www.khulisa.com Contract No: 72067418D00001, Order Number: 72067419F00015 V TABLE OF CONTENTS Introduction and Background ......................................................................................................1 Introduction .............................................................................................................................1 Background .............................................................................................................................1 Evaluation Overview................................................................................................................1 Identifying Information .............................................................................................................3 Research Questions ...................................................................................................................4 Design ........................................................................................................................................4 Intervention Design..................................................................................................................4 Evaluation Design ...................................................................................................................6 Design evaluation (Inception phase)........................................................................................6 Implementation Evaluation ......................................................................................................7 Study Population.........................................................................................................................9 Sampling Plan.............................................................................................................................9 Instruments ...............................................................................................................................12 Phase 1: Coach Shadowing and School Piloting ...................................................................12 Phase 2: Fieldwork at Sample Schools..................................................................................13 Data Handling and Collection....................................................................................................16 Design Evaluation (Inception phase) .....................................................................................16 Implementation Evaluation ....................................................................................................16 Fieldworker Recruitment....................................................................................................16 Training and preparation for fieldwork ...............................................................................17 Fieldwork...........................................................................................................................18 Data preparation and reporting..........................................................................................20 Ethical Considerations ..............................................................................................................21 Contract No: 72067418D00001, Order Number: 72067419F00015 VI ACRONYMS AHL African Home Language CFR Code of Federal Regulations CSV Comma Separated Value DBE South African Department of Basic Education DQA Data Quality Assessment EFAL English First Additional Language EGRS Early Grade Reading Study GoSA Government of South Africa GPS Global Positioning System FPD Foundation for Professional Development HOD Head of Department ICT Information and Communications Technology IDIQ Indefinite Delivery/Indefinite Quantity ODK Open Data Kit LOE Level of Effort LTSM Learning and Teaching Support Materials PERFORM Practical Education Research for Optimal Reading and Management PERFORMAN CE Practical Education Research For Optimal Reading and Management: Analyze, Collaborate, Evaluate PIRLS Progress in Reading and Literacy Study QASP Quality Assurance Surveillance Plan RSP Reading Support Project SMT School Management Team SOP Standard Operating Procedures Contract No: 72067418D00001, Order Number: 72067419F00015 VII TO Task Order USA United States of America USAID United States Agency for International Development VSO Voluntary Service Overseas LIST OF TABLES Table 1 Potential Sampling Techniques and Data Sources.......................................................11 Table 2 Education researcher profiles.......................................................................................16 Table 3 Illustrative one-day plan for data collection...................................................................19 LIST OF FIGURES Figure 1 Proposed Coach Shadowing Process .........................................................................18 1 INTRODUCTION AND BACKGROUND INTRODUCTION Khulisa Management Services Pty Ltd. (Khulisa), is pleased to present this Methodology Plan and Study Protocol to the United States Agency for International Development (USAID) for a Design and Implementation Evaluation of the Reading Support Project. BACKGROUND To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching, and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified the existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned a design and implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). EVALUATION OVERVIEW The aim of the evaluation is to support USAID and the Government of South Africa (GoSA) in evaluating the extent to which the RSP is being implemented with fidelity, with a particular focus on whether it is being implemented in line with GoSA’s predecessor program, the Early Grade Reading Study (EGRS). The evaluation will: 1. Identify any risks to the successful implementation of the RSP 2. Provide early findings to strengthen the implementation of the RSP 3. Ascertain whether the allocated time and resources were sufficient to successfully implement the RSP 2 4. Determine whether the FPD Consortium is implementing its activities as planned 5. Determine whether impact-level change can realistically be expected with RSP’s theory of change, its design and within the timeframe of the RSP implementation 6. Explore whether and how the RSP theory of change could be strengthened to produce the intended outcomes 7. Validate the data collected through the RSP M&E system 8. Determine whether teacher training and coaching is leading to any early changes in teaching methods (e.g. group guided reading, phonics, use of chapters in lesson plans to document implementation) 9. Ascertain what dosage of support teachers are receiving from coaches (i.e. number of visits, time spent per visit) 10. Determine whether coaches are supporting teachers and reporting on their coaching visits 11. Determine the contribution of volunteering in generating change in school management 12. Determine whether the School Management Team (SMT) support can be linked to changed support practices in schools. Under this Task Order (TO), the evaluation team has a responsibility to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. This is important for a number of reasons. First and foremost, this information is critical towards accurately attributing any impact to the program. If successful, the program can be repeated in another context as the elements of implementation will be known. If unsuccessful, the evaluation will provide some insight into whether this was due to design failure (i.e. poor plausibility of the pathways to impact) or to implementation failure (i.e. the design was plausible but the implementation lacked the robustness required to deliver the intended impact). Second, this information will assist the implementers of the RSP - the Foundation for Professional Development (FPD) Consortium – in understanding any gaps in implementation or identifying pockets of success. The data will be used in a formative manner to give the program the best possible chance to lead to impact. The design evaluation will entail a mini literature review, a review of project documents, surfacing the RSP Theory of Change and comparing it to the EGRS Theory of Change; interviews with key stakeholders and a design workshop with an expert panel hosted at the DBE in Pretoria. Findings from this design phase will be included in a Design/Inception Report. The implementation evaluation will be conducted in the North-West province of South Africa. The settings within which the research will take place include Quintile 1 – 3 schools,6 which use Setswana as the language of learning and teaching in the Foundation Phase (Grade 1-3 classrooms). The population of interest includes: 6 In South Africa, public schools are grouped into quintiles, which describe the wealth of the school, and therefore how much government funding they are entitled to. Quintile 5 schools are the wealthiest schools in the country, while Quintile 1 schools are the poorest. Quintile 1 – 3 schools do not charge school fees, and a large proportion of these schools participate in the National School Nutrition program where learners receive a meal at the school. These schools are legally not allowed to charge school fees, and are usually the worst performing in the system. 3 • Teachers in Grades 1 to 3 in the participating schools in the North-West province; • Coaches providing support to participating schools; • SMT members in participating schools; and • The DBE and North West Provincial and District officials in Kenneth Kaunda and Ngaka Modiri Molema districts. The study findings will be directly applicable to the program funders, implementers, the North￾West province Department of Education, as well as the DBE. The information generated through this evaluation will add to the evidence base for policy decisions about the most effective way to support the teaching of Home Language and English as First Additional Language (EFAL) in the Foundation Phase, in South Africa. Research authorization is not required as the DBE has an agreement with USAID. The DBE in turn is mandated by laws to have oversight on program implementation in schools. IDENTIFYING INFORMATION Dr Stephen Taylor, Director of Research Coordination, Monitoring and Evaluation from the DBE is the Principal Investigator for the EGRS I, supported in this role by Dr Janeli Kotzé and Ms. Nompumelelo Mohohlwane. Khulisa is responsible for conducting the design and implementation evaluation. Ms. Jennifer Bisgard is the Project Director, supported in this role by Ms. Katharine Tjasink (Senior Project Manager) and Ms. Leticia Taimo (Evaluation Coordinator). The DBE is contactable at Tel: +2712 357 4156 | Call Centre: 0800 202 933 and Khulisa is contactable at Tel: +2711 447 6464. This activity is funded by USAID under Contract Number 72067418D00001 and Task Order number 72067418F00010 4 RESEARCH QUESTIONS Design Evaluation Implementation Evaluation 1. Is the RSP theory of change likely to lead to the anticipated results? 1. To what extent did the FPD consortium implement the intended intervention design with fidelity? 2. What evidence is there of RSP uptake by teachers in class? 3. Were there any major divergences from the assumptions in the theory of change? 4. How could the management and implementation structures be improved? 5. How could the M&E systems be improved? 6. What are the unique challenges of delivery at scale? 7. Is the RSP cost efficient? DESIGN INTERVENTION DESIGN7 The RSP’s strategy was revised taking into account EGRS evidence that the Coaching intervention showed a substantial positive impact after two years of intervention (end of Grade 2) and the DBE’s interest to expand the evidence based around in-school coaching model. In a written request to USAID, the DG specifically recommended that the RSP support the EGRS model in the Dr Kenneth Kaunda and Ngaka Modiri Molema districts that were earmarked for scaling after the RSP pilot. The DG further expressed the view that building on the EGRS model would ensure continuity in the schools that were involved in the EGRS in these two districts. The revised overarching goal of the RSP is to improve the reading skills of primary grade learners in African Home Languages (AHLs), as well as in English as a first additional language (EFAL). The project objective is to improve language and literacy content knowledge and pedagogy of primary grade teachers in AHLs as well as in EFAL. By end of the four-years, the project aims to achieve the following planned results: • Result 1: Increased capacity of curriculum advisors (CA) • Heads of department (HODs) to support primary grade teachers in language and literacy in African Home Languages as well as English First Additional Language. (Component 1) • Result 2: Increased capacity of principals or deputy principals to support primary grade teachers in language and literacy in African Home Languages as well as English First Additional Language. • Result 3: Improved quality and use of LTSM for African Home Languages as well as English First Additional Language in the primary grades. (Component 3) [with inclusion of EGRS structured lesson plans] 7 Information retrieved from Attachment 5 Revised Reading Support Activity 26 July 2019 5 • Result 4: Increased competency and improved practice of primary grade teachers to teach in AHL and EFAL. [new Result area; primary focus area for revised strategy] The revised strategic approach is designed to improve teacher effectiveness and quality by means of improving capacity of teachers through a structured learning program and specialist on-site coaching for better learner results in AHLs and EFAL. [Significant change with focus on the teacher and coaching.] The Project specifically aimed to improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of Learner Teacher Support Materials (LTSM); and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning. [unchanged] The [RSP] proposed strategic approach is designed to build teacher capacity to teach early grade reading in AHL and EFAL focussing on grades one to three [significant revision through direct engagement with teachers in school via coaching and pivot from focusing on capacitating Heads of Department (HoDs) to perform a coach-like role for teachers in their schools] and strengthen supervisory and management systems for AHL and EFAL literacy at the primary grade level. The revised RSP model is a direct capacity building model for teachers with supplemental systems strengthening activities to foster a more enabling environment for the coaching program and to promote sustainability of practices. Component 4 [new] is introduced and funded through: • Savings in limiting geographic expansion to two districts in North West; the project is no longer envisioned for Limpopo; • Savings in limiting the AHL material to Setswana (developed for pilot) by continued focus on North West; • Savings in offering a combined SMT (Principals and HODs) programme (Components 1 and 2 in the RSP’s original version) • Implementation of a fundraising strategy to supplement available USAID and DBE funds. • Savings from reduced volunteer engagement in the SMT programme • Savings resulting from the reallocation of vehicles from projects that closed down to RSP 6 EVALUATION DESIGN This study is divided into two parts: a design evaluation (July-August 2019) and an implementation evaluation (September 2019 – November 2020). DESIGN EVALUATION (INCEPTION PHASE) The assignment will begin with a mini literature review to explore extant reports, documents, performance indicators, and literature relevant to the subject area of the evaluation. The review will then examine the intervention itself and the data generated from the intervention thus far. This will help the evaluation team to understand the perspective of the implementers and the progress that the program has made to date. The review will also describe and assess the project’s intervention logic, including assumptions about how success will be achieved. It will help to inform tool development and will inform the evaluation team broadly about the intervention. Khulisa will conduct process mapping in 2019 to understand and outline the processes that are being followed to deliver the RSP. The process maps will unpack the process theory of change in more detail and indicate the activities and workflows relevant to key processes such as: • Recruitment, training, deployment, and supervision of trainers/coaches • Delivering training and coaching to teachers, SMTs and District Officials • Developing, printing, and delivering lesson plans and other Learning and Teaching Support Materials (LTSM) to schools and teachers. Khulisa will conduct a full design review in 2019 to document the RSP theory of change based on interviews, the process mapping, a design workshop, and the mini literature review. This will be compared to the EGRS theory of change. A panel of local and international literacy experts will review the RSP theory of change and assumptions and express an opinion on the feasibility of achieving the expected results. Approximately 30 stakeholders involved in the RSP will be invited to attend the workshop, which will be held at the DBE in Pretoria. The design review and process mapping will inform a fidelity review by comparing the RSP Standard Operating Procedures (SOPs) against actual implementation and will feed into the cost and time analysis, highlighting inefficiencies or redundancies that could be improved in 2020. A design evaluation report will be submitted in 2019 at the end of the inception phase of the assignment. The design review will document and contrast the EGRS and RSP impact theories of change, the process theories of change, and also draw comparisons based on a map of the systems in which the EGRS and RSP functions. This will help to identify if there is a material departure from the original outcome design, from the implementation model, or if the system within which the programs operate differ in complexity. The design review report will indicate how this information will be used in the implementation phase of the assignment. 7 IMPLEMENTATION EVALUATION As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling up the RSP. The purpose of the implementation evaluation is to determine the degree of fidelity of implementation. Khulisa will carry out a fidelity review drawing on secondary data provided through the FPD Consortium’s monitoring systems, supported by a data quality assessment (DQA) in 2019 and 2020 to determine the extent to which the data provided through the program monitoring systems is relevant, effective, timely, and efficient. The monitoring data (attendance data / visit logs / coach feedback / participant feedback forms, etc.) of the FPD consortium will be reviewed to determine if the teacher training / teacher coaching / SMT training / district training / volunteer support happened as planned – i.e. that the anticipated number of contact sessions were conducted for the specified duration (dosage), with the targeted number of participants (coverage), with quality of input, within the planned time frame. The DQA will be carried out on the monitoring data within each sampling arm (See Sampling Plan section). Based on the finalized RSP M&E Plan, and RSP M&E data review, key indicators will be developed and tools designed to ensure sufficient and appropriate evidence is collected by education researchers at sampled schools and with coaches to verify the M&E data. The DQA will document findings, draw conclusions, and make recommendations to improve data quality in 2020. In Khulisa’s prior DQA work, questions have scaled responses (i.e. “No-Not at all“, “Partly”, “Yes, completely”, “N/A”) with supporting narrative substantiating the response. Thus, the DQA tool will (i) streamline data synthesis across DQA tools, (ii) calculate performance scores at each level, (iii) produce graphs to describe strengths and weaknesses of the various indicators, and (iv) develop focused recommendations to improve program performance in specific domains. Case studies will be developed in 2019 and 2020 to document the typical week of a coach using monitoring data, coach observation, and interviews with the coach, teachers, supervisors, and district officials. The assessments will look for top performing coaches and their characteristics, middle and then potentially poor performing or low performing coaches and reasons why. The coaching case studies will comprise an assessment of all 14 coaches (case studies) to inform an early implementation report. More in-depth case studies will be developed and included as part of the formative implementation evaluation report. The evaluation team will survey teachers who previously had coaches under the EGRS as well as teachers who have never had a coach, and will compare and contrast the amount and quality of coaching received. These teachers will be drawn from the coaching intervention and coaching plus SMT samples (See Sampling Plan section). Assuming coaches also fulfil the role of trainers, interviews with all coaches/trainers will provide valuable feedback on their experience of training. In addition, two mid-level education researchers will conduct training observations (covering both districts) observing: 1) Subject Advisor Steering Committee meeting on 29-30 August, 2019 2) Coach training on 25-27 September where coaches are trained to train teachers for Term 4 and to support teachers during Term 4 3) Teacher training for Term 4 on 30 September to 2 October, conducted by the coaches trained in the previous week. 8 At least one training observation researcher will be fluent in Setswana and therefore able to conduct observations during the Setswana sessions. Observations, triangulated with teacher group interviews, will allow for a better understanding of the quality of the training. The evaluation team will carry out interviews with the SMT with all arms to compare the support experienced by teachers in the SMT treatment group to those outside of this group (Section 2.5, Sampling). The SMT will provide insight into coaching (where relevant) and the role of the Voluntary Services Organization (VSO) in providing mentors (where relevant), and will be asked to describe any resultant changes within the school – both in terms of teacher practice and school management. Using a structured interview guide, education researchers will gather information across a range of assessment domains. The education researchers will conduct classroom observations in all schools within each sampling arm (See Sampling Plan section) to understand how the different treatment options are leading to possible changes in the classroom. The classroom observations will be carried out in two Grades, in Home Language (HL) and English First Additional Language (EFAL). Information from the teacher survey and teacher group discussions will be triangulated with the findings. Khulisa will triangulate the information gleaned from coach shadowing and coach interviews, data from the SMT interviews, and classroom observations, with a small sample of teacher group discussions (See Sampling Plan section). Teachers will help evaluate the coaching and SMT support received and its effect on their teaching practices. To supplement the teacher group discussions, the evaluation team will administer a short survey to teachers to rate the RSP along dimensions such as quality and quantity of services received. The results will be compared from 2019 to 2020 to gage improvements. In the sample schools that have been allocated a classroom library, as part of the classroom observations specialist education researchers will conduct an assessment of classroom libraries. In addition, researchers will conduct a learner workbook assessment from a sample of 4 learner workbooks (i.e. DBE workbooks and learner exercise books) per sample school. This will comprise a comprehensive review of the DBE workbooks (from the start of the third term until the date of the assessment, where available) as well as a review of learner exercise books (from the start of the school year until the date of assessment, where available). The learner workbook assessment will take place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks will be drawn will be the same classes in which the Classroom Observations were conducted. The teachers of those classes will be asked to provide the workbook of the best learner in the class. Khulisa will carry out a full cost and time analysis toward the end of 2019 and again in 2020. For the cost analysis, we will analyze the budget and expenditure figures of the RSP (compiled by the FPD Consortium) and compare it to the EGRS budget (compiled by the DBE). Where possible, the audited records of the FPD Consortium could be used. For the time analysis, the process map will be used to identify the main activities in the delivery of the RSP. Key stakeholders from FPD and the DBE will be asked to provide estimates of the time that is required to carry out the RSP. This will be contrasted with the Level of Effort (LOE) estimates in the budget, and where possible, verified through key informant interviews (KIIs) with stakeholders in the field including the North West DBE. This will allow the evaluation to surface any misalignment between the planned time investment as per the budget and the actual time investment for project staff. The analysis will also reflect estimates of the time required of 9 other role players to implement the program. This information will provide formative input on LOE during the planning for 2020, will identify potential cost efficiencies that could be achieved, and in 2020 will check whether the costing and time management options were actually implemented. STUDY POPULATION The RSP is being implemented in the North-West province of South Africa, in the districts of Dr Kenneth Kaunda and Ngaka Modiri Molema. The implementation evaluation will be conducted in the two districts. The settings within which the research will take place include Quintile 1 – 3 schools,8 which use Setswana as the language of learning and teaching in the Foundation Phase (Grade 1-3 classrooms). Data will be collected across a sample of 60 treatment and control schools. The population of interest includes: • Teachers in Grades 1 to 3 in the participating schools in the North-West province; • Coaches providing support to participating schools; • SMT members in participating schools; and • North West Provincial officials and District officials in Kenneth Kaunda and Ngaka Modiri Molema districts. SAMPLING PLAN As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling up the RSP. The purpose of the implementation evaluation is to determine the degree of fidelity of implementation. Unlike the quantitative evaluation of the EGRS, the sample will not assess schools against extant characteristics such as whether they performed well or poorly or whether they had large learning gains in the EGRS. Since the RSP program is the larger scale up of the EGRS study, the sampling frame will also include schools that did not form part of the original sample of EGRS schools. 8 In South Africa, public schools are grouped into quintiles, which describe the wealth of the school, and therefore how much government funding they are entitled to. Quintile 5 schools are the wealthiest schools in the country, while Quintile 1 schools are the poorest. Quintile 1 – 3 schools do not charge school fees, and a large proportion of these schools participate in the National School Nutrition program where learners receive a meal at the school. These schools are legally not allowed to charge school fees, and are usually the worst performing in the system. 10 For school-level sampling, the actual sampling strategy was defined in consultation with the DBE and USAID, to draw a sample of 60 schools9 . In total, 14 schools were selected from Dr Kenneth Kaunda District and 44 from Ngaka Modiri Molema district to reflect the distribution of schools per district. The sample of 60 schools is comprised as follows: 1. 10 schools receiving coaching, who previously were control schools; 2. 10 schools receiving coaching & SMT training, who previously were control schools; 3. 10 schools receiving coaching, who previously were EGRS coaching schools; 4. 10 schools receiving coaching & SMT training, who previously were EGRS coaching schools; and 5. 20 schools currently receiving LTSM only, who were previously EGRS parental involvement schools)? It was also decided that these schools would be selected randomly from amongst all eligible schools matching the above criteria. The sample was drawn by Stephen Taylor (Director: Research Coordination, Monitoring and Evaluation, DBE) on 13 August 2019 using a statistical software called STATA. The list of 251 schools participating in RSP as at 13 August 2019 was used as a starting point. A number of different “strata” were classified in order to group all the schools into the 5 categories of school listed above, as well as to identify schools which did not fit into any of those categories and therefore would not be eligible for the sample. Schools not eligible for the sample included the 34 schools who had previously been part of the EGRS “Training” intervention, and the 85 RSP schools which were not previously part of EGRS and therefore have no relevant comparison group. Each school was then assigned a random number using STATA’s “runiform” command. Next, the list of schools was sorted in order first of strata and then in terms of the random number. Within each strata, therefore, the order of schools is strictly random. The first 10 schools within each of the above strata (except for the 5th strata where the first 20 schools were needed) were then selected for inclusion in the implementation evaluation sample. Finally, the last 4 schools within each of the five strata were also selected to serve as schools to be used by Khulisa for simulation and piloting of the survey instruments. These 20 schools were thus also randomly selected. It turned out that a fair number of schools who have received classroom libraries were included in the sample even though this was not a criterion upon which the sample was selected. The schools which have received classroom libraries were indicated in a list of sampled schools provided by the DBE to Khulisa. The sample is too small to be representative or to allow generalization, but should be big enough to allow for identification of consistent patterns within and between the groups. The evaluation team is adapting the tools and methods that the DBE used to gather data during the EGRS and, where relevant, will use the same or similar tools and approaches. 9 Together with the pilot schools, the number of schools will total 66 11 Within each school, a list of sampling techniques is described below. Table 1 Potential Sampling Techniques and Data Sources Data Source Sampling Method Coaching case studies Mid-level researchers will shadow all 14 coaches, which is the total population. The aim of these interviews is to gain qualitative insight into a “week in the life of a coach”. Coaching assessment and interviews Total population sampling of all 14 coaches Group discussion with teachers The evaluation team will identify Grade 1, 2 and 3 teachers to participate in a group discussion at 12 sample schools. The team will aim for maximum variation and consider variables such as gender, age, whether the respondent received previous EGRS interventions, subject (Home Language and EFAL) to ensure that the sample provides a range of views that are relevant to the evaluation questions. In schools with very few Grade 1, 2, and 3 teachers, all teachers will be included in the group discussions. KIIs with principals, SMT, district officials, and other identified stakeholders The evaluation team will use purposive sampling to engage key informants as the aim of these interviews is to gain qualitative insight into the implementing agency and other key informants’ perspective on the project and sampling for proportionality is not the main concern. The team will use a framework for selecting the final respondents that will include variables such as gender, role in the program, etc., to ensure that the sample can adequately address the evaluation questions. Where relevant, the evaluation team will use snowball sampling to identify other useful candidates for study. Classroom observations Teaching practices will be observed in one Grade 1 HL class and one Grade 1 EFAL class, as well as one Grade 3 HL class and 1 Grade 3 EFAL class per sample school (per the consultation with the DBE and USAID).Classes within those Grades/subjects will be randomly sampled. Workbook and Learner Exercise Book analysis The learner workbook assessment will take place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks will be drawn will be the same classes in which the Classroom Observations were conducted. This will make 4 learner workbook assessments per school and 240 in total across the 60 sample schools. Data quality assessment (DQA) School-level data collected through the FPD monitoring system will be cross-checked with the data available at the school. As part of the assessment, data about all coaches / trainers and the delivery of all materials will be assessed. Short survey with teachers All Grade 1 to 3 teachers at each of the subsample schools will be administered a short survey. Interviews with the principal and SMT All principals and available SMT members at each of the subsample schools will be interviewed. 12 INSTRUMENTS Data collection will occur in two phases: 1) coach shadowing and instrument piloting 2) fieldwork at sample schools. Training observation will be conducted between the two phases. The instruments required for each phase are described in the section below. The administration and completion of all the instruments administered throughout the evaluation will be exclusively in English. The bulk of these instruments will be administered using an electronic tablet and data capturing will therefore happen directly upon the administration of the questionnaires, or captured into electronic format on the same day as the school visit. PHASE 1: COACH SHADOWING AND SCHOOL PILOTING The tools to be used during the coach shadowing and piloting activities will be developed by Khulisa in collaboration with our Coaching Expert, and will be reviewed by the DBE before piloting and data collection. These include: 1. Coach consent form: This tool comprises an information sheet and consent form for coaches to participate in the coach shadowing data collection activity. 2. Coach administrative checklist: This tool comprises a data quality assessment of coaching administrative activities against the Standard Operating Procedures (SOPs). The tool will enable fieldworkers to capture what coaches do at the school, and whether this is in alignment with the SOPs. 3. Coaching observation tool: part A (school level data): This tool comprises a checklist for each school that the coaches visit over a four-day period. The tool observes coach arrival and exit times, whether they conduct an afternoon workshop with the teachers and other related activities, which will be compared against the SOPs. 4. Coaching observation tool: part B (lesson observation): This tool comprises a checklist for each lesson that the coaches observe. The checklist will check the coach’s routine in the lesson observation against the SOPs. 5. Coaching observation tool: part B (coach interview): This tool comprises a structured template for conducting and documenting an interview with each of the coaches. This template will capture the coaches’ experience of coaching, their experiences working with the schools, and the level of support that they receive from head coaches. 6. Teacher structured group discussion consent form: This tool comprises an information sheet and consent form for teachers to participate in a group discussion with the fieldworker. 7. Teacher structured group discussion guide: The teacher group discussion guide will be used to collect data from teachers to help evaluate the coaching and SMT support received and its effect on their teaching practices. Group discussions will be conducted with teachers from a sub-sample 12 of schools during the coach shadowing week. Items in the guide include: whether teachers are receiving adequate support from the coaches whether they're receiving afternoon, workshops, and what the issues are around receiving support in different grades, and in HL and EFAL. The results will be collated and reported in the aggregate, but qualitative evidence will be used to support findings from other data collection points. The tool will also collect information 13 on the level of support that the SMT provides in the school, as well as the quality of the RSP LTSM and training support. 8. Training observation tool: This tool will guide education researchers’ data collection at the coach training scheduled for 25 and 26 September 2019 followed by a dry-run with the coaches on 27 September 2019, the teacher training from 30 September – 2 October 2019 and the reference group scheduled for 29 and 30 August 2019. Data collected through this tool will be triangulated with teacher group discussions, allowing for better understanding the quality of the training. PHASE 2: FIELDWORK AT SAMPLE SCHOOLS Khulisa will work closely with the DBE to adapt existing EGRS tools that can be used for this evaluation. The following instruments will be administered across the sample of 60 schools: 1. SMT consent form; 2. Teacher consent form; 3. SMT interview guide; 4. Classroom observation guide; 5. Classroom walk-through checklist; 6. Learner workbook and exercise book assessment tool; 7. Survey with teachers; 8. Classroom library assessment; 9. School DQA tool. The tools are outlined as follows: 1. SMT consent form The SMT members (who are available for interview) will be given a consent form which will provide them with the full details of the study as well as capture consent. This form will allow the SMT to indicate whether they are willing to participate in a short 20-minute interview. The consent form provides information on the following before obtaining consent: a. Who we are b. What we are doing c. Your participation d. Confidentiality e. Risks/discomforts f. Benefits g. Who to contact if you have been harmed or have any concerns h. Consent 14 2. Teacher consent form Each teacher asked to complete the teacher survey will be given a teacher consent form, which will provide them with the full details of the study. This form will allow the teacher to indicate whether she/ he is willing to complete the survey. In addition, the teacher consent form will be administered to those whose lessons will be observed by the fieldworker. The teacher consent form provides information on the following before obtaining consent: a. Who we are b. What we are doing c. Your participation d. Confidentiality e. Risks/discomforts f. Benefits g. Who to contact if you have been harmed or have any concerns h. Consent 3. SMT interview guide The SMT interview aims to gather information from all sampled schools to compare the support experienced by teachers in the SMT treatment group to those outside of this group. The SMT will provide insight into coaching (where relevant) and the role of the Voluntary Services Organization (VSO) in providing mentors (where relevant), and will be asked to describe any resultant changes within the school – both in terms of teacher practice and school management. Using a structured interview guide, education researchers will gather information across a range of assessment domains. The SMT interview guide collects information on the following areas: a. SMT training b. Establishment of Professional Learning Communities c. RSP Coaching d. RSP LTSM 4. Classroom observation guide The Classroom observation guide will be administered in all schools within each sampling arm to understand how the different treatment options are leading to possible changes in the classroom. The classroom observations will be carried out in two Grades, in Home Language (HL) and English First Additional Language (EFAL). This guide will draw heavily from the EGRS classroom observation instruments. The evaluation team will review the EGRS instruments and note where the EGRS observation instrument picked up differences between the control and intervention groups. The questions related to these observation items will be included in the tool. 15 5. Classroom walk-through checklist Junior researchers will conduct a classroom walk-through in all sampled schools to observe classroom facilities and materials. This instrument will be used to guide the classroom walkthrough, and comprises a checklist of items and their presence/ absence and quality/ quantity. 6. Learner workbook assessment tool This tool will provide a guideline/checklist on reviewing learner workbooks to gain a sense of the complexity and completeness of learner work completed during class. 7. Teacher survey To supplement the teacher group discussions, teachers will be required to complete a short survey to rate the RSP along dimensions such as quality and quantity of services received. The results will be compared from 2019 to 2020 to gage improvements. The tool captures teachers’ experience with the RSP LTSM, training, coaching, SMT support and classroom libraries (the latter where relevant) 8. Classroom library assessment This tool will comprise a structured template, containing information on: • Whether there is evidence that the classroom libraries are being used as intended; • Whether there is a working system for students to check out books; and • Whether the quality of the materials are durable enough for the classroom environment. The tool will be used to rate the quality of classroom libraries, where they exist. 9. School, District/Province and FPD DQA tools Each of these tools will be tailored and used with respective groups (teachers, SMT, district, province, and FPD consortium) to collect data related to the RSP implementation. The monitoring data (attendance data / visit logs / coach feedback / participant feedback forms, etc.) of the FPD consortium will be reviewed to determine if the teacher training / teacher coaching / SMT training / district training / volunteer support happened as planned – i.e. that the anticipated number of contact sessions were conducted for the specified duration (dosage), with the targeted number of participants (coverage), with quality of input, within the planned time frame. Questions will include scaled responses (i.e. “No-Not at all”“, partly”“, Yes, completely”“, N/A”) with supporting narrative substantiating the response. 10. Cost & time analysis interview guide (structured) This tool with provide a structured guide with questions that gather information on estimates on the Level of Effort required to carry out RSP activities, time and processes taken for activities. The process maps developed during the design evaluation will serve as a foundation for developing key questions. The tool will be administered to the FPD consortium and stakeholders in the field including the North West DBE. 16 DATA HANDLING AND COLLECTION Data collection, analysis, and reporting will be carried out in two evaluation processes (design/inception and implementation evaluation): DESIGN EVALUATION (INCEPTION PHASE) In the design evaluation, evaluation team members will collect data from primary and secondary sources: • Primary data collection includes interviews with key stakeholders and a design workshop with FPD consortium members, provincial/district representatives, the evaluation team, the DBE and USAID. • Secondary data sources include project documents provided by the DBE and FPD, as well as literature explored in the mini-literature review. The data collected will be consolidated and submitted in a design report submitted to the client. IMPLEMENTATION EVALUATION Fieldworker Recruitment Khulisa will use a mix of Gauteng and North-West Setswana speaking fieldworkers or education researchers (herewith used interchangeably) with the experience and qualifications necessary to successfully carry out this assignment. Minimum standards are required for the recruitment of high-quality fieldworkers with the experience and expertise necessary to conduct this assignment. These differ between the mid-level education researchers and the junior education researchers, and were agreed upon with the DBE. Table 2 Education researcher profiles Level Profile/Qualification Requirement Rationale Mid Minimum 5 Years’ experience with bachelor degree OR 10 years’ experience without degree with experience in the education sector, previous experience with conducting observations, familiarity with electronic data collection. Ideally university post-graduate candidates or recent post-graduates such as PhD students (education, languages, etc.). Fluency in Setswana and English. • Ensures an in-depth understanding of Foundation Phase teaching and classroom practices. • Conducting classroom observations and coach case studies requires a specialized approach. Junior Minimum 3 Years’ experience with bachelor degree OR 5 years’ experience without degree with experience in the education sector, familiarity with electronic data collection. Ideally university candidates/recent graduates (education, languages, other). Fluency in Setswana and English. • Ensures the ability to collect the data on workbooks, classroom libraries, classroom walk-through and verify data as part of the DQA Fieldwork for 2019 will be conducted over two phases, as described in the Fieldwork section of this document. Mid-level education researchers will shadow all 14 coaches from the coaching intervention and coaching plus SMT intervention groups during phase 1, and phase 2 will entail school level data collection and contextual data collection (i.e. interviews district/provincial level and other key stakeholders). Schools will be mapped using Global Positioning System (GPS) coordinates provided by RSP and clustered according to their location. Education researchers will be sent out to conduct the fieldwork in and will be assigned to the clusters and reviewed based on ongoing fieldworker recruitment and assessment. Upon completion of fieldwork training, the fieldwork schedule will be finalized. The Project Manager will monitor adherence to the fieldwork schedule on a daily basis in order to be reactive to unexpected challenges in the field, adjusting the schedule accordingly. Education Researchers will report directly to the Fieldwork Manager and Fieldwork Coordinator who in turn will work in cooperation with the Evaluation Coordinator, and Project Manager. The Fieldwork Manager and the Fieldwork Coordinator will offer technical, logistical, and administrative support remotely to the Education Researchers. Khulisa has found that creating constant, open lines of communication between project staff and education researchers allows the team to quickly and effectively address challenges in the field. Khulisa will create a WhatsApp group for education researchers and project staff to discuss issues and solutions on a real-time basis. The Fieldwork Manager will monitor adherence to the fieldwork schedule on a daily basis in order to be reactive to unexpected challenges in the field, adjusting the schedule accordingly. Training and preparation for fieldwork Fieldwork for the implementation evaluation will take place in two phases, further described in the Fieldwork Section of this document. Below are the training and preparation requirements described per phase. Phase 1 training: Coach shadowing and school piloting Khulisa will compile a preliminary fieldworker list prior to fieldworker training. In planning for fieldworker training, Khulisa will compile a fieldworker training manual which will include information on the project background, the data collection methodology, all instruments and tracking protocols, and the use of electronic devices and software for data collection. Khulisa will also obtain and configure all handheld devices. Khulisa will hold a pre-training meeting to ensure the trainers are familiar with the tools, processes, and procedures for the training workshop. Khulisa will host the five day training workshop at the Khulisa offices in Johannesburg. A group of 10-12 mid-level education researchers will be trained, from which the final 8 researchers will be selected based on their performance and a post-training evaluation. Three days of the workshop will be dedicated to the instruments, while the fourth day will entail fieldwork simulation, and the fifth will focus on administration and logistical arrangements. The last day of training will look at the degree to which the researchers are able to accurately assess learner workbooks, conduct classroom and training observations, and assess classroom libraries. This will ensure high-quality data collection. Most instruments will be adapted from instruments used by the DBE in EGRS. However, the coach shadowing tools used during phase 1 require development. Phase 2 training: Fieldwork at sample schools For phase 2, Khulisa will follow pre-workshop preparations followed in phase 1, as well as an inter-rater reliability quality check. Khulisa will host a five-day fieldworker training workshop at a training venue in the North-West province. The 8 mid-level education researchers (used during phase 1) as well as 8 junior level education researchers will attend the training, from which the final 12 fieldworkers (6 mid and 6 junior) will be selected based on their performance and a post-training evaluation. The training structure remains similar to phase 1 training however, this will include simulation data collection in schools as part of the training. The tools to be used during this phase will have been piloted prior to data collection. The use of electronic devices and digital software for data collection will be built into both training workshops. Fieldwork Fieldwork for the Implementation Evaluation will take place in two phases, described below. Phase 1: Coach Shadowing and school piloting (2 September 2019 to 13 September 2019) This includes Mid-level education researchers shadowing all 14 coaches and piloting the tools. In week one of coach shadowing, seven (7) education researchers will shadow one coach each over a period of four days. In week two of coach shadowing, seven (7) education researchers will shadow one coach each over a period of four days. The timing of this exercise is demonstrated below. Figure 1 Proposed Coach Shadowing Process In addition to coach shadowing, seven mid-level education researchers will administer the evaluation tools and conduct all other school-level evaluation activities in one pilot school each during day 5 of week 1. The eighth fieldworker will conduct their pilot test during day 5 of week 2. This will allow for field-testing of the tools in 8 pilot schools in total. In week 2, six (6) researchers will conduct teacher group discussions during Day 5, i.e. one education researcher will go to two different schools on their Friday and conduct two different teacher group discussions. By the end of Week 2, the team would have collected data from 12 teacher group discussions. The 12 schools will be drawn from the coach shadowing schools. The sample for the pilot has been drawn by the DBE from schools which are outside of the main sample for fieldwork (i.e. pilot schools will be a subset of those schools that were not Week 1 Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5 Researcher 6 Researcher 7 Day 1 Day 2 Day 3 Day 4 Day 5 District 1, School 1 District 1, School 2 District 1, School 3 District 1, School 4 District 2, School 5 District 2, School 6 District 2, School 7 Week 2 Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5 Researcher 6 Researcher 8 Day 1 Day 2 Day 3 Day 4 Day 5 District 1, School 9 District 1, School 10 District 1, School 11 District 2, School 12 District 2, School 13 District 2, School 14 District 2, School 8 District 1, Coach 1 District 1, Coach 2 District 1, Coach 3 District 1, Coach 4 District 2, Coach 5 District 2, Coach 6 District 2, Coach 7 District 1, Coach 9 District 1, Coach 10 District 1, Coach 11 District 2, Coach 12 District 2, Coach 13 District 2, Coach 14 District 2, Coach 8 selected to be part of the main sample). At the same time, sampling will be purposive to include school location. This will ensure that education researchers are logistically able to get to the school on time to conduct the piloting and teacher group discussions. Phase 2: Fieldwork at sample schools (7 October – 18 October 2019) For the full fieldwork in the sample schools, 6 teams of education researchers (one mid-level paired with one junior-level researcher – i.e. 12 fieldworkers in total) will carry out the assignment. The research teams will be split into two districts. Khulisa understands that there may be more schools in Ngaka Modiri Molema district, so teams may not be distributed evenly across districts (e.g. currently allocated 3 teams per district). To increase efficiency, sampled schools will be mapped using Global Positioning System (GPS) coordinates clustered according to their location. Each team will be assigned to clusters of selected schools. All 12 researchers will conduct school visits over two weeks. In addition, one team (2 people) will spend one week observing teacher training during the school holidays. This will occur during the school holidays prior to fieldworker training. Another team (2 people) will carry out all provincial and district-level DQA activities as well as return to any schools that could not be accessed on an initial visit. This will occur the week after fieldwork ends. A final team (2 people) will spend one week interviewing head coaches and subject advisors. Below is an illustrative fieldwork plan for one-day data collection involving a junior and mid￾level education researcher at a school level. Table 3 Illustrative one-day plan for data collection Time Activity Day before school visit 1. Confirm the venue of the school as well as the estimated travelling time. 2. Call the school to find out exactly where the school is from a landmark. 3. Charge the tablets overnight 4. Arrange all the documents and other things you will need - Classroom observation guide (including use of LTSM) - Classroom walk-through checklist - Teacher FGD guide covering o use of LTSM - all treatment arms plus the control arm o teacher training – teacher training and coach treatment arms o coach support – coach treatment arm - School DQA tool - Classroom library assessment tool (administered to all schools in the sample that have a classroom library - SMT interview guide - Rapid coaching / RSP quality survey - Teacher and SMT consent forms - Stationery 5. Ask accommodation for a packed breakfast if you have to leave before breakfast is served TBC – based on distance from school 6. Leave accommodation to arrive at the school by 7h00 This gives you enough time to complete introductions and arrange the day’s activities 7:00 7. Introduce yourself and purpose of visit to principal, Foundation Phase HOD, and teachers. 8. Obtain consent from teachers and the SMT to participate in the study 9. Confirm venues for interviews and observations Time Activity 10. Ask the principal to show you where the venues are for the various assessments No later than 8:00 11. Begin school data collection Mid-level education researcher: Classroom Observation, HL Junior Researcher: DQA against SOPs and FPD monitoring data, administer teacher surveys – surveys include feedback on SMT and coaches (capture teacher surveys electronically) Around 10:00 Mid-level education researcher: Classroom Observation, EFAL By 12:00 Mid-level education researcher: SMT interview(s) Junior Researcher: conduct a classroom walk-through to determine visibility of LTSM, assess classroom libraries, assess 4 learner workbooks (capture classroom walk-through/library assessment & workbook analysis electronically) 13:00 11. Say goodbye to principal, teachers, HODs and sign log book 13:30 12. Travel to accommodation for the night 15:00 13. Capture notes electronically: teacher group discussions (mid), principal & SMT notes: (junior) 16:00 14. Update evaluation coordinator on the day’s events 16:15 15. Capture classroom observation electronically (mid), Capture DQA findings in the assessment template, including notes, electronically (junior) 17:00 16. Prepare for next day as per start of this schedule *Teacher Group Discussions will be conducted during Phase 1 of fieldwork, in Week 2 of the Coach Shadowing. The tools/instruments will be administered using electronic devices, where relevant. Fieldworkers will have one laminated copy of the SMT and teacher questionnaire, which will be handed to the teacher/ SMT member while the interview is conducted to allow the teacher/ SMT member to follow along. Completed instruments will be automatically and instantly uploaded to a Khulisa-based server, allowing the Project Manager to monitor data completeness and quality in real-time. The tablets will be distributed to each fieldworker. One tablet will remain at Khulisa with the Project Manager to spot-check any relevant queries. Data preparation and reporting Before data collection, the Project Manager and Evaluation Coordinator will develop and follow a Quality Assurance Surveillance Plan (QASP), which will be developed in collaboration with USAID and the DBE. During fieldwork, the Project Manager will check, screen and clean the delivered data on an ongoing basis to ensure quality, accuracy, and completeness. This approach serves to 1) decrease the amount of time required post-fieldwork to prepare the data for submission and analysis, and 2) allow for ongoing quality control and real-time course correction. The data will be extracted in CSV format on a daily basis and shared with the DBE and USAID, on request. Education researchers will communicate daily with the PM, Evaluation Coordinator and Fieldwork Manager on the days’ activities. During Phase 2, at the end of every day, education researchers will capture notes electronically before submitting to the Fieldwork Manager. Within a timeframe agreed between the DBE and USAID, Khulisa will transmit the relevant data and technical materials to USAID, as required. ETHICAL CONSIDERATIONS In line with the Common Federal Policy for Protection of Human Subjects in research (the “Common Rule”) – 22 CFR 225, Khulisa’s practice is to uphold high ethical standards that are aligned with the type of respondents involved in an evaluation. Adult Participation The objectives of the study will be explained to adult participants (e.g. principals, teachers) before they make an informed decision on whether or not to participate. All participants will be expected to indicate their informed consent before taking part in the study. Participants will also have the freedom to answer or not to answer any question. Participants will be assured of confidentiality of information since no data for an individual respondent will be published and questionnaires will only be handled by members of the DBE and Khulisa, including trained fieldworkers. While the names of participants will be captured, this information will only be used to link data with a particular participant and for the purposes of returning to the correct participant to conduct follow-up research. In data sets that are submitted to USAID, all identifying information will be removed. Child Participation Khulisa is sensitive to the fact that while no child will participate in this evaluation, the environments in which the evaluation will take place will have young children present. Photos taken will not focus on children’s faces, and Khulisa will ensure compliance with internationally recommended ethical standards on involvement of children in research. The ‘do no harm’ principles shall be applied and address the possible risks that children might encounter when they are participating in development projects. ANNEX 5.1: DATA SOURCES AND RESPONDENTS 1 DATA COLLECTION SOURCES Data Source Data Collection Number of Respondents 2019 2020 Project documentation Researchers reviewed project documents shared by FPD, including monitoring data, reports, Project Management Meeting (PMT) minutes, training materials, etc. All relevant program documentation was sent by FPD consortium partners to the evaluators All relevant program documentation was sent by FPD consortium partners to the evaluators Coaching observations, interviews and case studies Researchers shadowed the population of coaches in the RSP. The aim of these interviews was to gain qualitative insight into a “the life of a coach” and understand changes in implementation across the years 14 Coaches shadowed and interviewed; 2 Head Coaches interviewed 14 Coaches and 2 Head Coaches shadowed and interviewed Group discussion with teachers Mid-level researchers identified 65 Foundation Phase teachers to participate in a group discussion at 13 sample schools (range from two to nine teachers in an interview). In schools with very few Foundation Phase teachers, all teachers were included in the group discussions 65 Foundation Phase teachers participated in group discussion Khulisa adapted the data collection approach to minimize COVID-19 risk due to social distancing. This included removing the group discussions with teachers and including relevant questions as part of the teacher survey Surveys with teachers Mid-level researchers administered surveys to all Foundation Phase teachers in each sample school 304 teachers (of which 105 were Grade 1 teachers, 103 were Grade 2 teachers, and 96 were Grade 3 teachers) participated in Teacher Survey 311 teachers (of which 102 were Grade 1 teachers, 99 were Grade 2 teachers, and 110 were Grade 3 teachers) participated in the Teacher Survey 2 Data Source Data Collection Number of Respondents 2019 2020 Key Informant Interviews and survey with school principals and SMT members Mid-level researchers conducted interviews with all sample school principals and SMT members, where they were available 157 SMT members participated in Key Informant Interviews 58 Principals participated in school Data Quality Assessment interview 198 Principals and SMT members completed a Principal/SMT survey (149 from the sample schools and 49 from other RSP schools present at the 5th Professional Learning Community (PLC)) 57 Principals participated in school Data Quality Assessment interviews Key Informant Interviews with district officials, and other identified stakeholders The evaluation team used purposive sampling to engage key informants as the aim of these interviews was to gain qualitative insight into key informants’ perspective on the project and sampling for proportionality was not the main concern 2 DBE officials and 3 USAID staff participated in Key Informant Interviews (KII) 15 Provincial and District officials participated in KIIs 4 FPD staff members and 6 consortium members participated in KIIs 6 Provincial and District officials and 2 DBE officials participated in KIIs 4 FPD staff members and 5 consortium members participated in KIIs Classroom observations Teaching practices were observed in one Grade 1 HL class and one Grade 1 EFAL class, as well as one Grade 3 HL class and one Grade 3 EFAL class per sample school (per consultation with the DBE and USAID). Classes within those Grades/subjects were randomly sampled Classroom observations were conducted in 242 classrooms, of which 119 were Grade 1 classrooms and 120 were Grade 3 classrooms. Three classroom observations did not capture the grade correctly and were therefore excluded from all grade-disaggregated analyses Classroom observations were conducted in 214 classrooms, of which 104 were Grade 1 classrooms and 110 were Grade 3 classrooms 3 Data Source Data Collection Number of Respondents 2019 2020 Workbook and Learner Exercise Book analysis The learner workbook assessment took place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks were drawn were the same classes in which the Classroom Observations were conducted 238 learner workbooks (DBE workbooks and other learner books) were assessed, of which 118 are Grade 1 and 120 are Grade 3 learner workbooks 214 learner workbooks (DBE workbooks and other learner books) were assessed, of which 104 were Grade 1 classrooms and 110 were Grade 3 learner workbooks Data quality assessment (DQA) School-level data collected through the FPD monitoring system was cross-checked with the data available at the school DQA data collected from 5810 schools DQA data collected from 59 schools Classroom library assessment and classroom walkthrough assessments Classroom-level data on class libraries and LTSM was collected from the same classrooms that participated in the classroom observations Data from classroom libraries in 2020 was only collected in classroom library schools Classroom library data collected from 123 classrooms across 58 schools Classroom LTSM assessed in 126 classrooms (62 Grade 1 classrooms and 64 Grade 3 classrooms) across 57 schools Classroom library data collected from 82 classrooms across 30 classroom library schools Classroom walkthrough assessments were combined with classroom observations in 2020 and were completed in the 214 classrooms observed 10 In one school, no SMT members were present at the school (they had all gone to an appointment with the department of education) to collect the school DQA data and other data related to SMT. No information on why DQA data was not collected from the other school. 4 Data Source Data Collection Number of Respondents 2019 2020 School Functionality assessments School environment observation data was collected from sampled schools to understand preparedness, management and adherence to COVID-19 protocols and other key contextual indicators such as nutrition, health and hygiene and school management Not part of tools administered in 2019 School functionality tool administered in 59 schools Coach WhatsApp group chat analysis Asked Coaches to share the data from WhatsApp groups they had created to understand how coaches interacted with teachers during school closures in 2020 Not part of the formative evaluation methodology in 2019. Received data from 9 coaches and 1 Head Coach, and analyzed 14 Coach led-WhatsApp groups. Text files from groups were analyzed qualitatively (to understand when groups were started, what were the conversations themes, what kind of multi-media was shared, the role of the coach and how teachers responded to the group, the purpose of the group, how the group responded to COVID-19 challenges) and quantitatively (frequency of posts per individual in the group to identify dominant participants/influencers, amount of teachers in the group and frequency of coach interactions in the group) 5 Data Source Data Collection Number of Respondents 2019 2020 Observations from trainings, reference group meetings and other key events Observed key events/meetings that took place either in-person or virtually (online) to get a better understanding of project implementation and maximize communication and data collection for the evaluation One experienced Mid-level researcher attended one coach training (Term 4, 2019) and a selection of teacher trainings in term 4, 2019 The Khulisa team attended the following: 1 in-person coach training session, 2 RSP online coach training sessions, 3 online UNICEF coach training sessions, 2 online reference group sessions, 4 PMT meetings and 1 teacher training session for Term 4 in Mafikeng and Lichtenburg, and the Principals’ meeting A Khulisa team member attended the PLC meeting to present and administer the SMT surveys 6 RESPONDENTS FROM 2019 Interviews Respondents No. Type of stakeholder Male Female Key Informant Interviews USAID 3 Audience 1 2 DBE 2 Audience 0 2 FPD Consortium Interviews FPD respondents 4 Primary 2 2 Molteno respondents 2 Primary 0 2 OUP respondents 2 Primary 1 1 VSO respondents 2 Primary 1 1 North West Provincial Department of Education Interviews Curriculum Advisors 6 Audience 0 6 Subject Advisors 2 Audience 0 2 Director 1 Audience 0 1 Provincial coordinator for African languages 1 Audience 0 1 Chief Education Specialist 1 Audience 0 1 Project Coordinator 1 Audience 0 1 Director: Professional Educator Development Services 1 Audience 1 0 Deputy Chief Education Specialist (Project Coordinator): Dr. Kenneth Kaunda 1 Audience 0 1 Chief Education Specialist Professional Support: Dr. Kenneth Kaunda 1 Audience 0 1 Head coach interviews Head Coach: Dr. Kenneth Kaunda 1 Primary 0 1 Head Coach: Ngaka Modiri Molema 1 Primary 0 1 7 Coach interviews RSP Literacy Coaches 14 Primary 2 12 Interviews Respondents No. Type of stakeholder Male Female Teacher surveys Sample school Foundation Phase teachers 304 Primary Mixed, but predominantly female Teacher Group Interviews Pilot school Foundation Phase teachers 65 Primary Not captured School Principal interviews Sample school principals 58 Primary Mixed SMT member interviews Sample school SMT members 157 Primary Mixed, but predominantly female 8 RESPONDENTS FROM 2020 Interviews Respondents No. Type of stakeholder Male Female Key Informant Interviews DBE 2 Audience 1 1 FPD Consortium Interviews FPD respondents (individual) 4 Primary 2 2 Molteno respondents (individual) 1 Primary 0 1 OUP respondents (group) 3 Primary 1 2 SMT mentor (individual) 1 Primary 0 1 North West Provincial Department of Education Interviews Curriculum Advisors 5 Audience 0 5 Chief Education Specialist 1 Audience 0 1 Head coach interviews Head Coach 1 Primary 1 0 Head Coach 1 Primary 0 1 Coach interviews RSP Literacy Coaches 14 Primary 1 13 Teacher surveys Sample school Foundation Phase teachers 311 Primary 7 304 School Principal and SMT member interviews and surveys Sample school principals and SMT members 198 Primary 66 132 ANNEX 5.2: EVALUATION TOOLS 2020 ANNEX 5.2.1: COACH TOOLS Semi-Structured Coaching Observation Tool: Lesson Observation Page 1 of 5 Tool 1: Semi-Structured Coaching Observation Tool Purpose of coach shadowing: 1. Fidelity of implementation against program SOP and Job Description (criteria table in report) 2. Quality of coaching with teachers 3. Change of coaching in practice since 2019 observation 4. Achievement of outcomes (against ToC & against other models & HL/EFAL differences) 5. Lessons to strengthen practice, replication and scale RESEARCHER NAME: Prepopulate COACH NAME: Prepopulate Number of teachers coach supports Number of schools coach supports Lesson Observation (to be completed per lesson observed) Date of visit School Name: Prepopulate School EMIS Number: Prepopulate Teacher Name Prepopulate Teacher National Id/ Passport Number Teacher Gender Pre populate Lesson Observed (select one) □ Home Language □ EFAL Grade (select one) □ Grade 1 □ Grade 2 □ Grade 3 Which lesson week is being taught? Semi-Structured Coaching Observation Tool: Lesson Observation Page 2 of 5 Is the lesson week on track with the revised curriculum? Y/N PLANNING & PREPARATION 1. Did the coach circulate their schedule for the month of November 2020 to the teachers? a. Yes i. If yes, when? __________(insert date) b. No – they did not circulate their schedule to the teachers c. No – they did not develop a schedule for the month of November 2020 d. Any comments? 2. Does the teacher briefly show the coach which lesson is going to be presented from the lesson plan? a. Yes, ____________(insert length of time in minutes they engaged on this) b. No c. N/A – fieldworker did not observe d. Any comments? 3. Did the coach tell the teacher about planned visit? 4. Did the coach set up direct communication with the teacher (not through school)? 5. Did the coach reconfirm the visit before arriving? 6. Is the coach prepared for the visit with this teacher? 7. Has the coach reviewed notes, checked WhatsApp messages sent since last visit? 8. Does the coach have a development plan for this teacher? Evidence? LESSON OBSERVATION Lesson observation start time: While you observe the coach’s routine in the classroom, please make relevant notes on what the coach does. Lesson observation end time: SCHOOL BASED POST OBSERVATION DISCUSSION a. How long does the coach spend providing feedback to the teacher? b. Does the coach complete a lesson observation form? Y/N c. Does the Coach share the lesson observation form with the teacher after the discussion? Y/N Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Semi-Structured Coaching Observation Tool: Lesson Observation Page 3 of 5 Not observed Partially or unsuccessfully executed Completely and successfully executed Comments FEEDBACK PROCESS Before providing his/her own observations, the coach asks the teacher to identify the things they felt went well The coach offers his/her observations of things that went well The coach asks the teacher to identify things they would do differently in the future The coach asks questions to elicit clarification and prompt reflection regarding areas of improvement The teacher successfully identifies action steps The coach offers suggestions The coach guides the teacher to identify solutions for problem areas in the form of action steps The coach takes the teacher through the lesson observation document The coach covers the following topics under the lesson observation document: a) Classroom management b) Curriculum coverage c) The extent to which the teacher follows the weekly routine d) The extent to which the lesson complied with the expected methodology Semi-Structured Coaching Observation Tool: Lesson Observation Page 4 of 5 Not observed Partially or unsuccessfully executed Completely and successfully executed Comments CONTENT The coach helps teachers identify solutions to potential barriers to implementation The coach offers suggestions that are appropriate and reasonable in scope The coach provides examples of correct implementation with suggestions The coach provides a rationale for each suggestion The teacher successfully identifies action steps COMMUNICATION The coach offers suggestions for improvement in a constructive way The coach provides positive feedback The coach avoids judgement or bias when providing observations and suggestions The coach uses questions to prompt reflection rather than telling the teacher their problem areas The coach and the teacher set goals for the next visit For fieldworker: Please note the goals discussed in the comment box Semi-Structured Coaching Observation Tool: Lesson Observation Page 5 of 5 ADDITIONAL OBSERVATION AND EVIDENCE Semi-Structured Coaching Observation Tool: Professional Learning Community Page 1 of 3 Tool 2: Needs based workshop held in the afternoon Semi-Structured Coaching Observation Tool PROFESSIONAL LEARNING COMMUNITY [Please complete this tool with information collected at each school you visited with the coach] A. CONFIRMATION 1. Did the coach conduct a needs-based workshop in the afternoon for this school? □ Yes (then answer the following questions) □ No (skip to QA) □ Not observed (skip to QB) QA. IF NO: If there was no workshop in the afternoon, what were the reasons for not having it? (please provide brief description based on what the coach says) Were there any plans made for another workshop? If so, please state when it was scheduled for. QB. IF NOT OBSERVED, Why was the workshop not observed? B. Information Date Of Needs Based Workshop Observed Time Of Workshop: BEGIN TIME: END TIME: School Name: School EMIS Number: Number Of Teachers Who Should Be Attending The Workshop Number Of Teachers Participating Semi-Structured Coaching Observation Tool: Professional Learning Community Page 2 of 3 Focus Of RSP (Select One) □ Home Language □ EFAL Teachers From The Following Grade(s) (select all that apply) □ Grade 1 □ Grade 2 □ Grade 3 □ Phase Workshop structure □ Single School □ Cluster of Schools Any other notes? B. PLANNING AND PREPARATION FOR NEEDS BASED WORKSHOP 12.Did the coach circulate their schedule for the month of November 2020 to the school principal? a. Yes i. If yes, when?_______(insert date) b. No – they did not circulate their schedule to the principals c. No – they did not develop a schedule for the month of September 2019 Any comments? 13. The coach explains the purpose of the needs based workshop YES/NO 1. The coach has prepared a plan for the needs based workshop YES/NO C. NEEDS-BASED WORKSHOP 1. Please take a photo of the coach conducting the needs-based workshop with teachers 2. Please take photo of the attendance register 3. What was the content of the workshop? (please provide a brief description of what was covered) 4. How did the coach facilitate the needs-based workshop? (i.e. quality of facilitation) 5. How did the coach facilitate opportunities for teachers to reflect on their practice? 6. Did the coach complete the After School Workshop / Needs based workshop tool? Yes/No 6.1. If Yes, Please take a photo of the completed tool 6.2. If Yes, How detailed was the report? 7. Please describe the engagement of the teachers during the workshop. 8. What evidence was there that the workshop contributed to teachers’ development? D. PROFESSIONAL LEARNING COMMUNITY 1. Has the coach established a PLC at this school? Or is there one in the school? Why or why not? 2. How frequently does the PLC meet? 3. Does the coach facilitate the PLC? (i.e., Principal, HOD?) 4. To what extent do teachers participate in the PLC? Semi-Structured Coaching Observation Tool: Professional Learning Community Page 3 of 3 5. Do teachers reflect on their practice during the PLC? Explain how coach facilitated this, why or why not … 6. Is there evidence of PLC being held? (i.e., Photo of records, etc.). If yes, please take a photo Semi-Structured Coaching Observation Tool: Coach Interview Page 1 of 4 Tool 3: Semi-Structured Coaching Observation & Coach Interview Purpose of coach interview: 1. Determine changes from previous year and effect of training and support 2. Fidelity of implementation against program SOP and Job Description (criteria table in report) OBSERVATION DETAILS RESEARCHER: COACH NAME: DATE OF INTERVIEW SECTION A: COACH EXPERIENCE In this section, you will ask questions to the coach and write down the answers in this document, and then transfer this to the electronic tool. Before starting the interview, please turn on Otter on your tablet to record the discussion. Coaching Experience 1. Which of the following training sessions did you attend? [prepopulate] 2. Which of the following teacher training sessions did you participate in? 3. What was your role or engagement during these training sessions? 4. What effect did the COVID 19 lockdown and restrictions have on your role and function as a coach? (Prompt: not just on planning and implementation, but on content of RSP, role as a coach, teacher support during the lockdown etc.) 5. 5. How did you support teachers during school closures? 6. How many schools do you support? 7. How many teachers do you support? 8. How many of the teacher you support are male? And how many are female? 9. How do you identify and record the needs of the teachers? Has this changed from 2019? Why or why not? 10. Have you rated the schools and teachers for differentiated support? Has this changed from 2019? Why or why not? 11. If yes, what criteria do you use to rate the schools and teachers? 12. How many times do you support a teacher per term? Has this changed from 2019? Why or why not? 13. Do you visit some teachers more than others, and if so why? Has this changed from 2019? Why or why not? 14. Do you manage to visit all teachers you plan to visit per month? If no, explain what challenges you face. Has this changed from 2019? Why or why not? Semi-Structured Coaching Observation Tool: Coach Interview Page 2 of 4 15. Have you provided any catch up training to teachers who missed the training on LTSM? If yes, how long has this training been and how have you organised and delivered it? If no, why not? Has this changed from 2019? Why or why not? 16. How do you build a trust relationship with the teachers? 17. Do you share your classroom observations with the principal? Why or why not? If yes, how does that make the teachers feel? 18. Do you provide virtual/remote support for teachers? If yes, how? Training and Support to Coaches 1. What kind of support do you get from the Head Coach? What, if anything, has changed since last year? 2. Do you feel that you are adequately supported by the Head Coach? What, if anything, has changed since last year? 3. Do you send your coach monthly plan to the Head Coach? YES / NO [why not?] 4. Does the Head Coach approve the Monthly Plan you develop? YES / NO [why not?] 5. Does the Head Coach provide feedback to you on your monthly plan? YES / NO [why not?] 6. Do you feel that you are adequately supported by the School Management Team? What, if anything, has changed since last year? 7. Have you attended any developmental training? What, if anything, has changed since last year? If yes, what was the content? 8. Do you feel that your developmental needs are met? If no, why not? What, if anything, has changed since last year? 9. Did you receive training on the classroom libraries that have been distributed to some RSP schools? (Probe: If yes, was it useful? If no, why not?) What, if anything, has changed since last year? 10. What has been the most difficult/challenging aspect of your role as a coach? What, if anything, has changed since last year? 11. How do you deal with these difficulties/challenges? What, if anything, has changed since last year? 12. Is support readily available to deal with challenges that are beyond your power? What, if anything, has changed since last year? 13. Is there a system in place to ensure that coaches do get to the schools? If yes, how does it work? What, if anything, has changed since last year? Summative evaluation questions 14. If you could change one thing about the RSP coaching programme, what would that be and why? What, if anything, has changed since last year? 15. What is your understanding of the RSP needs based workshop? 16. What is your understanding of a PLC? 17. If you think of the teachers you coach, is there a difference in how male and female teachers respond to you as a coach? Why or why not? 18. Who in the schools in general actively leads the support of reading in the Foundation Phase? Please explain. (Prompt: power dynamics, positions of authority, champions etc.) Semi-Structured Coaching Observation Tool: Coach Interview Page 3 of 4 SECTION B: SUPPORT VISITS PLANNING AND PREPARATION The Standard Operating Procedures (SOPs) for coaching under the Reading Support Project (RSP) outline the procedures to be followed by coaches before, during, and after each classroom support visit. Please check with the coach the below: 1. What informs the coach’s monthly/weekly and daily plans? (i.e., reports, teacher profiles, feedback sessions, etc.) 2. How does the coach prepare for each visit? 3. Did the coach draw up a schedule for the month of November 2020? a. Yes. Started on ____________ (insert date) and submitted on _______________ (insert date) b. No, the coach did not draw up a schedule for November 2020 c. Any comments? 4. Did the coach send their schedule for the month of November 2020 to the head coach? d. Yes. Started on ____________ (insert date) and submitted on _______________ (insert date) e. No, the coach did not send their schedule for the month of November 2020 to the head coach f. NA – The coach did not draw up a schedule for the month of November 2020 g. NA – The coach does not recall h. Any comments? 5. Did the coach receive feedback on their schedule for the month of November 2020 from their head coach? a. Yes a. If yes,__________________ (insert date) b. No, the coach did not receive feedback from the head coach on their schedule for the month of September 2019 c. NA, the coach either did not develop a schedule or did not send their schedule for the month of November 2019 to the head coach d. Any comments? 6. Did the head coach sign off the coach’s schedule for the month of November 2020? a. Yes a. Please provide when received feedback____________ (insert date) b. No, the head coach did not sign off on the coach’s schedule for the month of November 2019 c. NA, the coach either did not develop a schedule or did not send their schedule for the month of November 2019 to the head coach for signature d. Any comments? 7. Did the coach circulate their schedule for the month of November 2020 to the Subject Advisors? a. Yes _______(insert date) and name of the subject advisor ________________ in __________________ district b. No – they did not circulate their schedule to the Subject Advisors Semi-Structured Coaching Observation Tool: Coach Interview Page 4 of 4 c. No – they did not develop a schedule for the month of November 2019 d. Any comments? 8. In the month of November, did the coach call each teacher to confirm each visit to each school? a. Yes, called all teachers b. Yes, but only some teachers c. No d. Any comments? 9. In the month of November, did the coach call each teacher to check curriculum coverage (progress to date)? a. Yes, all teachers b. Yes, but only some teachers c. No d. Any comments? 10. In the month of November, did the coach call each teacher to re-confirm his or her appointment? a. Yes, called all teachers b. Yes, but only some teachers c. No d. Any comments? 11. In the month of November, did the coach create/review the teacher profiles for each teacher before going to the school? a. Yes, and researcher was able to observe it b. Yes, but only some teachers c. No d. Any comments? SECTION C: EVIDENCE OF COACH OBSERVATION 1. Please take a photo of the completed Coach Lesson Observation Plan SECTION D: COMMENTS AND OBSERVATIONS RSP Head Coach Shadowing and Interview Tool Page 1 of 6 Tool 4: RSP HEAD COACH INTERVIEW QUESTIONNAIRE & SHADOWING 2020 Purpose of Head Coach Interview and Shadowing: 1. Fidelity of implementation against program SOP and Job Description 2. Quality of coaching with coaches 3. Achievement of outcomes (against ToC & against other models & HL/EFAL differences) 4. Lessons to strengthen practice, replication and scale RESEARCHER DETAILS RESEARCHER (prepopulated): HEAD COACH NAME (prepopulated): DATE: Head Coach Interview 1. Head Coach Support # NAME OF COACH (PREPOPULATED) NUMBER OF TIMES HEAD COACH VISITED COACH IN TERM 3 (SELF REPORTED BY HEAD COACH) NUMBER OF TIMES HEAD COACH VISITED COACH IN TERM 4 (SELF REPORTED BY HEAD COACH) TYPE OF SUPPORT HEAD COACH HAS PROVIDED FOR EACH COACH? OR - DEVELOPMENT PLAN IN PLACE FOR COACH? 1 2 3 4 5 6 7 8 9 10 2. Training fidelity and quality 2.1 Which of the following trainings did you attend (select all that apply): RSP Head Coach Shadowing and Interview Tool Page 2 of 6 2.1.1 RSP Coach Training 2.1.2 RSP Head Coach Training 2.1.3 Training from the RSP Monitoring and Evaluation advisor on the Standard Operating Procedures (SOPs) for recording, collation, reporting, storage, and management of the Reading Support Project data? 2.1.4 UNICEF Coach Training 2.1.5 Other 2.2 How, if at all, did these training events help you understand the objectives of the RSP? 2.3 How, if at all, did these training events help you to undertake your role as a Head Coach? 2.4 What training do you think a head coach needs to be able to better support coaches in schools? 3. Coaching Quality 1.1. What is your role as a head coach? 1.2. How has your role as the head coach changed due to the COVID-19 pandemic? 1.3. What is the quality of coaching of the coaches you supervise? 1.3.1.What needs to be improved? 1.3.2.What is your role in improving the quality of coaching practices? 1.4. Are the coaches adequately reporting against the criteria of the lesson plans etc.? How do you support them in reporting? 1.5. What makes a ‘good’ coach? If you look at the coach criteria sheet do these adequately cover the coach roles? What should be added or changed for the head coaches? 1.6. How would you respond in the following situations? 1.6.1.A coach faces a challenges in meeting with the teacher before the lesson to plan, what is your view on this and how can it be addressed? 1.6.2.A coach cannot hold the afternoon workshops – why is this? What can be done to address this? 1.7. Please describe the relationships between coach, head coach and the SMT? 1.8. What support (emotional, pedagogical etc.) can you provide to the coaches and how? 1.9. What do you find challenges in your role as Head Coach? 1.10. What do you think have been the successes of having Head Coaches in the RSP project? 1.11. Do you feel supported to fulfil your role as Head Coach? 1.12. Have there been any changes/improvement in implementation or management of the RSP since last year (2019)? (prompt: last year you and the other Head Coach mentioned that the two of you and the provincial manager shared a car, so transport was an issue, is this still the case?) 4. Coaching Fidelity & Adaptability (check against SOP/ToC) 4.1. How did you communicate with and support coaches during school closures? (i.e., WhatsApp groups?) 4.2. What were coaches required to submit to you during school closures? RSP Head Coach Shadowing and Interview Tool Page 3 of 6 4.3. Do you receive a monthly plan from each of the coaches? (confirm for each coach and validated with FPD data OR for the specific time period) # NAME OF COACH (PREPOPULATED) YES RECEIVE REGULARLY YES, BUT DON’T RECEIVE EVERY MONTH NO 1 2 3 4 5 6 7 8 9 10 4.4. What feedback or approval do you give them for each plan? How, when, etc. 4.5. Whose role is it to communicate the coach visits to schools? 4.6. On average, how many times per month do you visit each coach (Think of Term 1 and then Term 3 and 4)? 4.7. Do you differentiate your support based on the needs of the different coaches? 4.7.1. Do you collect lesson observation forms from your coaches? 4.7.2. If yes, how often do you collect them (per term)? 4.7.3. If yes, do you review the contents for quality assurance and come up with comments for the literacy coach, where applicable? 5. REPLICATION, SCALE & LESSONS 5.1. If the RSP went to all schools in the district and province, what are the critical elements that must be planned when it comes to the coaching element? (prompt: recruitment and selection of coaches, training, supervision, conducting coaching sessions, how many head coaches to coaches etc.) 5.2. Do you think that to scale-up RSP would be valuable? Why or Why not? Yes 1 No 0 Yes 1 No 2 RSP Head Coach Shadowing and Interview Tool Page 4 of 6 5.3. What factors constrain or enhance the scale up of the RSP training approach (e.g. language, number of people being trained, training environment, and other training practices) 5.4. What risks do you see in the scale up of the project if other provinces implemented the RSP? 5.5. Lessons for future or replication? 5.6. What are the lessons, emerging good practices for RSP and recommendations to inform scaling based on responses to COVID-19? RSP Head Coach Shadowing and Interview Tool Page 5 of 6 Head Coach Observation 6. HEAD COACH OBSERVATION (To be completed daily – complete one per day of shadowing) 6.1. Does the head coach complete a school visit log from? 6.2. Does the Head coach interact with the teachers/SMT? 6.3. The Head Coach was at the school the same amount of time that the Coach was at the school: YES / NO [skip pattern] 6.3.1. If Yes, then the Head Coach supported the Coach for the following: a) How Many EFAL lessons? (Insert numeric value) b) How many HL lessons? (Insert numeric value) c) Which Grades? Grade 1, 2 or 3 (select one) d) The after-school needs-based workshop? Yes 1 No 2 6.3.2. If No, then 6.4. Does the Head Coach (HC) provide feedback to the Coach? 6.5. Describe the manner and nature of the HC gives feedback to the coach? (constructive/instructive, is this during the lesson/after the lesson, is this in front of the teacher/privately, what is the content of the feedback [style of coaching/technical knowledge], etc). 6.6. Does the HC look at/take the lesson observation sheet to check after the Coach has completed it? 6.7. Does the HC make comments/provide feedback on the lesson observation sheet? Yes 1 No 2 Yes 1 No 2 The Head Coach Left before the Coach 1 The Head Coach was at the school longer than the coach to meet with teachers/SMT, etc 2 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Head Coach Shadowing and Interview Tool Page 6 of 6 6.8. Describe the nature between the coach and head coach (Prompt: There seems to be an open relationship/strained relationship, the feedback is well received/little feedback, trusting/mistrust, etc). 6.9. Does the head coach complete a report for every visit to a coach? 6.10. Has the head coach confirmed the establishment of a PLC in each school? 8. HEAD COACH OBSERVATION (Reflection on the two days of coach shadowing) 8.1 How many coaches per school did the head coach visit? 8.2 Describe how much the Head Coach spends most of their time during the shadowing days? (Admin vs. Meetings; Provincial representatives vs. travel vs. support to the coaches) 8.3 Reflect on the differences between the relationships between the HC and different coaches. (Prompt: Did the HC treat male/female coaches differently? Did the HC respect the coaches/did they seem respected by the coach?) 8.4 Did it seem that the HC differentiated their support to the coaches based on need? OTHER OBSERVATIONS: Yes 1 No 0 Yes 1 No 0 Coaches (insert numeric value) School (insert numeric value) All support to coaches seemed equal 1 “Weaker” coaches received less support 2 ““Weaker” coaches received more support 3 There were no “weaker” or “stronger ” coaches N/A ANNEX 5.2.2: SCHOOL TOOLS RSP Data Quality Assessment Survey Page 1 of 3 Tool 1: RSP SCHOOL DATA QUALITY ASSESSMENT (DQA) VERIFICATION LIST Identifying information: 1. District name: 2. School name: 3. School EMIS number: 4. Type of intervention: LTSM 1 Coaching 2 Coaching and SMT 3 4.1. If Coaching school: 4.1.1. Coach name: 4.1.2. Head coach name: NOTE: ASK THE SCHOOL PRINCIPAL TO VERIFY THE FOLLOWING INFORMATION. Please “tick” if the response is correct. If not, please enter the correct information. Question Response Teacher Training and LTSM 2 Number of Grade 1 teachers in the school: Number of Grade 2 teachers in the school: Number of Grade 3 teachers in the school: 2.1 Number of Grade 1 teachers: 2.2 Number of Grade 2 teachers: 2.3 Number of Grade 3 teachers: 3 Did the Provincial department communicate the training dates to your school via a circular? Yes 1 No 2 4 Were the RSP teacher training dates for the year shared with Principals (via SMS and/or email) and teachers during the first school term? Yes 1 No 2 5 Between Jan and Oct 2020, how many teacher training sessions were attended by schools’ Foundation Phase teachers: RSP Data Quality Assessment Survey Page 2 of 3 6 Where teachers did not attend RSP teacher training was any catch-up training and LTSM delivered to these teachers? Yes 1 No 2 7 For the fourth term, are there any Foundation Phase teachers at the school who did not get their RSP LTSM and Lesson plans? If yes, how many? Number Foundation Phase teachers without LTSM for Term 4: 8 Between Jan and Oct 2020, if LTSM was not delivered to the schools at any point, did the school arrange for LTSM to be picked up from the warehouse? Yes 1 No 2 Not Applicable 3 Coaching schools only 9 Between Jan 2020 and Oct 2020, how many times did the coach visit the school? (Note: The coach name will be provided to you. Please ask to see the school logbook. Between Jan and Oct 2020, how many RSP Literacy Coach visits are recorded?) 10 Between Jan 2020 and Oct 2020, how many times did the head coach visit the school? (Note: The head coach name will be provided to you. Please ask to see the school logbook. Between Jan and Oct 2020, how many RSP Literacy Coach visits are recorded?) 11 Between Jan 2020 and Oct 2020, were school-based afternoon workshops held by an RSP coach? (Note: Please ask to see the school logbook. Between Jan and Oct 2020, how many RSP Literacy Coach visits are recorded?) Yes 1 No 2 Classroom library project schools only 12 Did any teachers at the school receive a classroom library from RSP? If yes, how many? Number Grade 1 teachers: Number Grade 2 teachers: Number Grade 3 teachers: SMT schools only 13 Has your school received tablets as part of the training provided by RSP? (Note: Please ask to see the school asset register. How many RSP tablets are recorded (if the SMT received tablets, these should be recorded on the register)) Yes 1 No 2 RSP Data Quality Assessment Survey Page 3 of 3 14 Has your school received training sessions as part of the RSP? Yes 1 No 2 15 Any further comments? RSP Principal/SMT Survey Page 1 of 5 Tool 2: RSP PRINCIPAL / SCHOOL MANAGEMENT TEAM SURVEY Information Sheet and Consent Form Khulisa Management Services (Khulisa) is undertaking an evaluation funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on￾site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In addition, in 2020 we want to understand how the COVID-19 pandemic has affected the RSP implementation, and how the project has adapted its implementation. Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. YOUR PARTICIPATION We are asking you as a school principal/ SMT member to provide some background information that may underpin the teaching of reading in your school by participating in a group interview. The interview should not take longer than 20 minutes to complete. The questions are mainly about conditions and activities that may enhance or hinder the teaching of reading and language at the school and your experience participating in the RSP. We will also administer the intended evaluation instruments with some teachers from your school, including observing a HL and EFAL classroom in RSP Principal/SMT Survey Page 2 of 5 Grade 1 and in Grade 3. This year we also want to understand how the COVID-19 pandemic has affected your work and learn how you have adapted. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. CONFIDENTIALITY Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office, and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. The records from your participation that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. RISKS/DISCOMFORTS We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not to other Department officials at the circuit, district, provincial, or national offices. BENEFITS The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members, and teachers will never be attached to any of the information that you provide or the information released through reports or articles. WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Margaret Roper at Tel: 011 447 6464 ext. 3230, or e-mail at mroper@khulisa.com You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. 1. Identifying information RSP Principal/SMT Survey Page 3 of 5 1.1. School name: 1.2. School EMIS number: 1.3. District: 1.4. Principal/SMT member name: 1.5. Principal/SMT member surname: 1.6. Principal/SMT member date of birth: 1.7. 1.8. Principal/SMT member Sex: 1.9. What is your role in the school? 1.10. Are you also a teacher at the school? 1.11. If yes, what Grade do you teach Grade 1 1 Grade 2 2 Grade 3 3 Other 4 QUESTIONS SMT Training 1. Has your School Management Team received any in-service training on school leadership and / or school management in 2020? 1.1 If yes, who provided the training:___________ 2. Has your School Management Team received any in-service training on school supporting literacy development in 2020? 2.1 If yes, who provided the training:___________ 3. Has your School Management Team received Reading Support Project (RSP) in-service training in 2020? Yes 1 No 0 3.1 If yes, If yes, which RSP training workshops have you attended (Select all you attended)? Workshop 1 Workshop 2 Workshop 3 Workshop 4 1 2 3 4 Date DD/MM/YYYY Male 1 Female 2 Other 3 Principal 1 Deputy principal 2 Head of Department 3 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Principal/SMT Survey Page 4 of 5 3.2 From the workshops you attended, did you find any workshop more useful than others? Yes 1 No 0 3.2.1 If yes, which RSP training workshop did you find most useful (Select one)? Workshop 1 Workshop 2 Workshop 3 Workshop 4 1 2 3 4 3.3 What was the best part of participating in the training workshops for you? 3.4 Was the quantity of the training sufficient? Yes there were enough sessions provided and they covered the content adequately Yes, but the content felt rushed No, I would have liked to have more training sessions/workshops 1 2 3 4. Are you doing anything differently in your approach to teaching as a result of attending the RSP training? 5. If yes, please describe and provide an example of what you are doing differently Professional learning community 6. Have you established a Professional Learning Community at your school? (i.e. a community of practice; a meeting where you meet as a group to discuss topics or train teachers, etc.) 6.1 If yes, do you meet in person or is it virtual (e.g. WhatsApp)? 6.2 If yes, how often do you meet? 6.3 If yes, what topics do you cover? Yes 1 No 0 Yes 1 No 0 RSP Principal/SMT Survey Page 5 of 5 7. What were the challenges faced by the SMT during the COVID-19pandemic and schools closures? 8. Did the school implement any remote learning strategies during school closures? 8.1 If yes, what did the remote learning strategies entail? Please describe. 9. What strategies did you put in place at the school to support teachers with coping with the pandemic? (only applicable for principals) Coaching (only apply if Coaching school) 10. Do you feel Foundation Phase teaching practice has changed as a result of teachers’ interactions with their literacy coach? 10.1 If yes, how has it changed? 11. Would you recommend literacy coaches in schools? 12. Why / why not? (e.g. is the coach knowledgeable, reliable, good with the teachers, etc?) 13. How is your relationship with the coach different from that of the curriculum/subject advisor, if at all? 14. Are you aware of any schools based workshops the coach held with teachers in your school? Yes 1 No 0 15. If you could change one thing about the RSP literacy coaching, what would that be and why? Tablets (LTSM) (only for SMT schools) 16. Did you receive a tablet as part of the RSP SMT training? 16.1 If yes, what do you use the tablet you received for? 16.2 Have you experienced any challenges with the tablets? 16.2.1 If yes, what challenges did you experience? 16.3 What do you like the least about the tablet and materials loaded on it? 16.4 What do you like the most about the tablet and materials loaded on it? 16.5 Do you access and use the online / offline learning platform? Yes 1 No 0 17. If yes, how do you use it? Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Classroom Observation Tool Page 1 of 15 Tool 3: RSP CLASSROOM OBSERVATION TOOL Administer tool in: Demographics 1. Name of district 2. Name of school 3. School EMIS number 4. Name of fieldworker responsible for data collection 5. Date of visit 6. Day of the week 7. Name of teacher 8. Teacher’s Date of birth 9. Teacher Sex: Male 1 Female 2 Other 3 10. Teacher’s PERSAL number: __________ 11. Is the teacher observed the usual class teacher? Yes 1 No 0 12. Please select the Grade you are observing Grade 1 1 Grade 3 3 13. Subject observed Home Language 1 EFAL 2 14. If you experienced any problems gaining access to the classroom to observe or were unable to observe a FULL Setswana Home Language / EFAL lesson, note details below. 15. Did you experienced any challenges in gaining access to the classroom? 15.1. If yes, please note the details below. Yes 1 No 0 1 Grade 1 HL class 1 Grade 3 HL class 1 Grade 1 EFAL class 1 Grade 3 EFAL class RSP Classroom Observation Tool Page 2 of 15 16. Were you able to observe the FULL lesson? 16.1. If yes, note details below 17. Is this lesson taking place outdoors? Yes 1 No 0 Yes 1 No 0 18. Was there a delay in the lesson start? Yes 1 No 0 18.1. If yes, how many minutes was the lesson delayed by:_____ 19. Time lesson started:______ 20. Time lesson ended:______ Section A: Teaching and Learning Environment 21. How many learners are actually present in class? (start of the lesson) 21.1. How many boys are actually present in class? (start of the lesson):____ 21.2. How many girls are actually present in class? (start of the lesson):___ 22. Is the classroom clean? Yes 1 No 0 23. Please use your tablet to take a picture from one corner of the room, showing the layout of the room 24. Can the teacher move around the room comfortably and walk easily between all the desks/tables? Yes 1 No 0 24.1. If no to the question above, please explain RSP Classroom Observation Tool Page 3 of 15 25. Are relevant Covid-19 precautions taken in the classroom (e.g. distance between learners, teacher and learners wearing masks, etc.)? Yes 1 No 0 25.1. If yes, please select what you can observe (select all that apply): YES 1 NO 0 The teacher is wearing a mask over mouth and nose All learners are wearing masks over mouth and nose Most learners are wearing masks over mouth and nose Only some learners are wearing masks over mouth and nose The teacher is wearing a mask but DOES NOT cover mouth and nose All learners are wearing masks but DOES NOT cover mouth and nose Most learners are wearing masks but DOES NOT cover mouth and nose Only some learners are wearing masks but DOES NOT cover mouth and nose No learners are wearing masks The desks are spaced out (1.5 metres apart) Learners are NOT sharing desks Learners hand sanitize or wash hands before entering the classroom If required to work as a group, learners observe social distance (1.5. metres apart) When teacher interacts with the learners she/he observes social distance (1.5 metres apart) Learners are encouraged not to share materials (books, utensils, etc.). Other, specify 25.1.1. If Other, please provide detail of what else you can observe related to teacher and learner interaction in the classroom due to COVID-19: 25.1.2. What type of mask is the teacher wearing? (Skip pattern if select any of the teacher wearing mask questions above) YES 1 NO 0 Cloth mask Disposable mask RSP Classroom Observation Tool Page 4 of 15 Face shield Other (please specify) 25.1.3. What type of mask is the learners wearing? (Skip pattern if select any of the learners wearing mask questions above) YES 1 NO 0 Cloth mask Disposable mask Face shield Other (please specify) 26. Is there a reading carpet, mat (or similar ‘shared reading space’) evident in the classroom? (e.g. for story time) Note: This is not provided by the RSP Yes 1 No 0 26.1. If no to the question above, please explain 27. Is there a Home Language (HL) word wall in the classroom? Yes 1 No 0 27.1. If there is a HL word wall, in the classroom, are the words systematically organised (i.e. are words/flash cards organised and grouped together on the wall because of shared features e.g. phonetically) Yes 1 No 0 28. Is there an English First Additional Language (EFAL) word wall in the classroom? Yes 1 No 0 28.1. If there is an EFAL word wall, in the classroom, are the words systematically organised (i.e. are words/flash cards organised and grouped together on the wall because of shared features e.g. phonetically) Yes 1 No 0 29. How many different types of learner’s recent work or other recent learner-made objects are on display in the classroom? None 0 1-2 types 1 3 or more types 3 30. To what extent do the following activities slow down teaching and learning at the beginning or during the lesson: To a large extent To some extent Hardly at all Not at all 1 Handing out workbooks/readers/exercise books or other material/apparatus/equipment to learners? 3 2 1 0 RSP Classroom Observation Tool Page 5 of 15 2 Teacher consulting her notes or lesson plan? 3 2 1 0 3 Learners re-arranging the furniture, moving chairs/ desks, etc.? 3 2 1 0 4 Learners who do not have necessary pens, pencils, crayons, erasers, rulers and other necessary stationery moving around the class borrowing from classmates? 3 2 1 0 31. During the lesson, are there ‘outside’ interruptions to teaching in the classroom (e.g. intercom announcements, other teachers entering the room to talk to the teacher; learners bringing messages)? Yes 1 No 0 31.1. If yes, how many outside interruptions occur during the classroom observation? 1-2 1 3-5 2 More than 5 3 32. Is independent tasks/classwork given to the children? Yes 1 No 0 32.1. If yes, when the class is busy with independent written work or reading tasks (whether in groups or alone), does the teacher … Always Mostly Sometimes Never 1 Monitor all learners and check that they are doing what they are supposed to be doing? 3 2 1 0 2 Supervise quite closely less capable learners as they work? 3 2 1 0 3 Give additional tasks to learners who have completed their work/tasks and are coping well? (Note: Additional work can be given to the fast workers) 3 2 1 0 32.2. If yes, to what extent do learner appear to be engaged with independent written work? No learners 0 Some learners 1 Most learners 2 All learners 3 32.3. If yes, to what extent do learner appear to be engaged with reading tasks? No learners 0 Some learners 1 Most learners 2 All learners 3 33. Does any learner finish their work ahead of other children? Yes 1 No 0 33.1. If yes, what do learners who finish their work ahead of other children generally do? (select all that apply): YES 1 NO 0 They take a book from the book corner to read The teacher uses workbooks as a source or means for giving learners who finish classwork ahead of others opportunities to do more exercises They help other learners with their work They sit idle Other, specify RSP Classroom Observation Tool Page 6 of 15 34. Is there a difference in the way different learners engage with the teacher? (e.g. boys and girls, children with disabilities, etc.) 34.1. If yes, please explain what the difference in engagement is between the learners was like Yes 1 No 0 Section B: Discipline 35. In your observation, are there any learners’ not concentrating or paying attention? Yes 1 No 0 35.1. If yes, please select the number of boys:___ 35.2. If yes, please select the number of girls:___ 36. In your observation, are there any undisciplined or disruptive learners? Yes 1 No 0 36.1. If yes, please select the number of boys:___ 36.2. If yes, please select the number of girls:___ Section C: Listening and speaking 37. Does the teacher provide any front of class explanations, instruction or in class discussions? 37.1. When the teacher provides front of class explanations, instruction or in class discussions, to what extent do learners answer in unison or repeat responses after or with the teacher, or after or with other learners, rather than actually coming up with their own responses? Yes 1 No 0 Never 0 Sometim es 1 Mostl y 2 Alway s 3 37.2. If yes, for EFAL lessons, to what extent does the teacher model the correct terminology in English (even if explanations include code-switching)? [note skip pattern from Q13] Not at all 0 Hardly at all 1 To some extent 2 To a large extent 3 37.3. If yes, for EFAL lessons, does the teacher highlight or foreground the English terms for the concept (e.g. background, make meaning, illustrations)? [note skip pattern from Q13] Neve r 0 Sometim es 1 Mostl y 2 Alway s 3 RSP Classroom Observation Tool Page 7 of 15 37.4. If yes, for EFAL only, does the teacher encourage learners to use English terms when responding? [note skip pattern from Q13] Neve r 0 Sometim es 1 Mostl y 2 Alway s 3 38. Does the teacher explicitly develops learners’ concepts about print and printed material during the course of the lesson? Yes 1 No 0 38.1. If yes, does the teacher do any of the following (Select all that apply): YES 1 NO 0 1. Identify front/cover of a book 2. Identify the spine of a book 3. Identify the title / author / illustrator 4. Identify headings/sub-headings 5. Identify the beginning/end of sentences 6. Show that print moves from left to right 7. Identify full stops/other punctuation 8. Identify capitals 9. Identify bottom/top of page or picture 10. Identify page numbers 11. Show that pages turn from right to left Section D: Literacy and Language Development 39. Are there any extended texts (sentences or paragraphs including in textbook text or stories) are read or told? Yes 1 No 0 39.1. If yes, are learners encouraged and required to (Select all that apply): YES 1 NO 0 1. Retell, act out or summarise what they have read or what has been read to them (e.g. the story or plot)? For example, repeat a sequence of events in the story correctly. 2. Discuss or respond to pictures or illustrations? 3. Use pictures or illustrations as clues to aid understanding the text? 4. Identify and/or re-state the main theme/idea/principle? 5. Answer open-ended questions about what they have read or what has been read to them and give reasons for their answers (e.g. ‘Do you think she was right to…?’ and ‘Why do you think that?’) 6. Answer predictive/inferential questions (e.g., questions about what will happen next before the next part of the text is read; or ‘what would have happened if…? writing their own version of how they think a story will end, etc.)? 7. Answer literal, evaluative, appreciative, or critical questions? Section E: Vocabulary and spelling development RSP Classroom Observation Tool Page 8 of 15 40. Does any vocabulary development (learning new words and meanings, learning synonyms and antonyms) take place during the observation period? (e.g. learners making lists or personal ‘dictionaries’ or ‘word banks’ with new words) Yes 1 No 0 40.1. What is the evidence for your answer: 41. Is there any evidence of spelling development or testing of spelling during the lesson observation? Yes 1 No 0 41.1. What is the evidence for your answer: Section F: Opportunities to write 42. Do learners complete any writing/written tasks (not drawing) at all during the observation period? Yes 1 No 0 42.1. If yes, how many learners engage in writing/written tasks? Less than a quarter of the class 1 About a quarter of the class 2 About half of the class 3 Most/All learners 4 42.2. If yes, does the teacher check that learners are holding their pens/pencils and positioning their writing material? Yes 1 No 0 42.3. If yes, does the teacher check that learners are taking any corrective measures when learners are not doing this correctly? Yes 1 No 0 42.4. If yes, does the teacher check that learners are writing work in exercise books/jotters? Yes 1 No 0 42.5. If yes, does the teacher check that learners are writing work in writing work in pre￾printed workbooks? Yes 1 No 0 42.6. If yes, which of the following writing demands are made on all or most learners individually in the lesson? (Select all that apply) YES NO RSP Classroom Observation Tool Page 9 of 15 1 0 1. Colouring in 2. Drawing (own creation) 3. Copying a drawing/diagram (e.g. from the chalkboard, overhead, textbooks, etc.) 4. Drawing patterns in preparation for joined script or cursive writing 5. Copying vowels/letters/syllables 6. Writing vowels/letters/syllables (learners’ own productions without copying) 7. Copying paragraphs 8. Writing paragraphs dictated by the teacher 9. Writing their own paragraphs (each unique – i.e. creating their own text) 42.7. If yes, does the teacher ask that all or most of the learners copy words? Yes 1 No 0 42.7.1. If yes, how many words learners have to copy? 42.8. If yes, does the teacher ask that all or most learners write words? Yes 1 No 0 42.8.1. If yes, how many words learners have to write (select one only)? a) one or two individual words b) three to ten individual words c) more than ten individual words 42.9. If yes, does the teacher ask that all or most learners copy sentences? Yes 1 No 0 42.9.1. If yes, how many sentences learners have to copy (select one only)? a) one or two sentences b) three to five sentences c) more than five sentences (but not paragraphs) 42.10. If yes, does the teacher ask that all or most learners write sentences? Yes 1 No 0 42.10.1. If yes, how many sentences learners have to write (select one only)? a) one or two sentences b) three to five sentences c) more than five sentences (but not paragraphs) 42.11. How many different types of tasks/exercises involving writing text (can be letters, individual words, sentences, paragraphs but not drawing) does the teacher give learners to do during the lesson (including homework)? _____ 42.12. Are all learners involved in writing any cursive text during the lesson? Yes 1 No 0 42.12.1. If yes, please select which of the following cursive text writing demands learners have to do (select all that apply) YES 1 NO 0 RSP Classroom Observation Tool Page 10 of 15 1. Write letters 2. Write words 3. Write phrases (less than 6 words) 4. Write sentences (More than 5 words) 5. Write a paragraph Section G: Opportunities to read 43. Are learners given opportunities to read aloud? Yes 1 No 0 43.1. If yes, when learners are reading extended text aloud (sentences or paragraphs including in textbook text or stories), does the teacher provide learners with any of the following strategies for self-correcting (select all that apply): YES 1 No 0 1. Sound out a word 2. Guess a word 3. Use words similar to words that are already known, to work out what the word could be 4. Skip or read past difficult words and then go back and use the context/meaning in which a word is embedded to work out what the word could be 44. Are learners split into groups for reading? Yes 1 No 0 45. Does the teacher do shared reading with learners? Yes 1 No 0 45.1.1. If yes, per the lesson plan instructions, does the teacher ask the learners questions after reading (during shared reading)? Yes 1 No 0 Section H: Reading corner / Classroom Library 46. Is there a book collection/reading corner/box library or equivalent evident in the classroom (may be kept in the classroom cupboard/storage space)? [if no skip to question 46] Yes 1 No 0 46.1. If yes, please rate the existence, sufficiency, and quality of the books/materials in the reading corner: Small reading corner, not many books 1 Average reading corner, but quantity and quality of books could be improved 2 Classroom reading corner exists and is well stocked 3 46.2. Please take a photo of the reading corner 46.3. If yes, is there any evidence of the teacher referring to the book collection/reading corner/box library or equivalent? RSP Classroom Observation Tool Page 11 of 15 Please describe the evidence: Yes 1 No 0 46.4. If yes, is there any evidence of the teacher administering books to learners (i.e. checking out books)? Please describe the evidence: Yes 1 No 0 46.5. If yes, is there any evidence of the teacher retrieving books from learners (i.e. checking in books)? Please describe the evidence: Yes 1 No 0 46.6. [NOTE: if school is part of the classroom library project then ask this question] If yes, is there any evidence using the “five finger rule” to determine what level book the learner should be reading? Please describe the evidence: Yes 1 No 0 Section I: Classroom Facilities and Materials (Please rate the existence, sufficiency, and quality of the classroom infrastructure, facilities, and materials. You may need the teacher to show you the materials) 47. Is this the same classroom for both HL and EFAL for this grade? 48. Are there RSP materials (LTSM) available for the teacher to use in the classroom? Yes the materials are in the classroom and readily available for the teacher to use 3 Yes the materials are in the classroom, but kept in a cupboard 2 Some materials are there, as the teacher shares them with another teacher 1 The teacher did not receive the RSP materials 0 Yes 1 No 0 49. Are there any RSP Big Books EFAL*? (Please select one option) [ not required for Grade 3] Yes and in good quantity (20+) 3 Yes but could have more books (10-20) 2 Some books available, very poorly stocked (less than 10) 1 No books available 0 50. Are there any graded Readers EFAL (New Heights / AWEH! Readers) Yes and in good quantity (20+) 3 Yes but could have more books (10-20) 2 Some books available, very poorly stocked (less than 10) 1 No books available 0 51. Are there any Graded Readers HL (Vula Bula)? Yes and in good quantity (20+) 3 Yes but could have more books (10-20) 2 Some books available, very poorly stocked (less than 10) 1 No books available 0 RSP Classroom Observation Tool Page 12 of 15 52. Are there RSP posters (EFAL) available? No Yes, but poster(s) not on display Yes and on display 0 1 2 53. Are there RSP posters (HL) available? No Yes, but poster(s) not on display Yes and on display 0 1 2 54. Are the RSP posters visible to learners? Not applicable, posters not on display Few learners can see them well About half the learners can see them well All learners can see them well 1 2 3 4 55. Are RSP Theme Vocabulary Flashcards displayed (e.g. on classroom walls) Yes 1 No 0 56. Are RSP EFAL Sight Words Flashcards displayed (e.g. on classroom walls)? Yes 1 No 0 57. Are RSP HL Sight Words Flashcards displayed (e.g. on classroom walls)? Yes 1 No 0 58. Are RSP EFAL Consolidated Words Flashcards displayed (e.g. on classroom walls) EFAL? Yes 1 No 0 59. Are RSP HL Consolidated Words Flashcards displayed (e.g. on classroom walls) EFAL? Yes 1 No 0 60. Are the RSP flashcards visible to learners? Not applicable, flashcards not on display Few learners can see them well About half the learners can see them well All learners can see them well 1 2 3 4 61. Is there an Alphabet Frieze (EFAL) on display on the wall? Yes 1 No 2 62. Does the teacher have a Teacher Management and Storage File? No Files do not contain any resources Files contain few resources Files contain many resources 1 2 3 4 63. Does the teacher use a timetable incorporating weekly routine? Yes and it is the RSP timetable 2 Yes, but it is not RSP timetable 2 No 0 Section J: Lesson plans The section below requires that you check the Termly Lesson Plans RSP Classroom Observation Tool Page 13 of 15 64. Does the teacher have the RSP lesson plans for EFAL? Yes 2 No lesson plan available 0 No, but using another lesson plan 1 64.1. If yes, please take photo of the RSP EFAL lesson plan 65. Does the teacher have the RSP lesson plans for HL? Yes 2 No lesson plan available 0 No, but using another lesson plan 1 65.1. If yes, please take photo of the RSP HL lesson plan 66. Inside the lesson plans, check the assessment tasks with rubrics/ mark sheet available (Note: review the lesson plan from the week prior to this visit) No assessment sheet Not filled in Partially filled in with some ratings for learner tasks Fully filled in with all ratings for learner tasks 0 1 2 3 67. Inside the lesson plans, is there a Weekly reflection and weekly tracker? None Not filled in Partially filled in - teacher records some lessons completed / some notes on curriculum coverage / some catch up plans Fully filled in - teacher records all lessons completed, records curriculum coverage, and develops catch up plans 0 1 2 3 68. Inside the lesson plans, there is a curriculum tracker. Does the curriculum tracker in the EFAL lesson plan show signs of clearly being used? Yes 1 No 0 69. Inside the lesson plans, there is a curriculum tracker. Does the curriculum tracker in the HL lesson plan show signs of clearly being used? Yes 1 No 0 Compare the work actually covered in the lesson observed with the work outlined in the work plan for this time of the term/year 70. Was the content of the lesson/s observed in line with the work planned or indicated for the particular week in the term/year plan or series of lesson plans? Yes 2 No 0 Days not indicated on plan or plan too general 1 71. Was the content of the lesson/s observed in line with the work planned or indicated for the particular day in the term/year plan or series of lesson plans? Yes 2 No 0 Days not indicated on plan or plan too general 1 72. Did the teacher cover the entire lesson plan? Yes 2 No 0 No lesson plan made available 1 RSP Classroom Observation Tool Page 14 of 15 73. Did the teacher use the correct texts for the lesson? Yes 2 No 0 No lesson plan made available 1 Section K: Please take photos of the four walls 74. Photo of Wall 1 75. Photo of Wall 2 76. Photo of Wall 3 77. Photo of Wall 4 RSP Classroom Observation Tool Page 15 of 15 Section L: Comments Further comments If you have any additional comments you feel you should make about the lesson observed or about the observation schedule, write them below RSP School Functionality Tool Page 1 of 6 Tool 4: SCHOOL FUNCTIONALITY TOOL ***Take a photo of school sign 1. Section A: Food and Nutrition *** Please take a photo of the food preparation area before preparation 1.1. Do the children receive food at the school? Yes 1 No 2 1.2. Which grades receive food? R 1 2 3 4 1.3. Which meals are provided to the learners? Breakfast 1 Mid-morning Snack 2 Lunch 3 After-school snack 4 1.4. What time is the first meal provided? Enter time H H : M M 1.5. Which food groups are provided? Carbohydrates i.e. pap, samp, rice, potatoes 1 Fruits 2 Vegetables 3 Proteins i.e. beans, chicken, fish, meat 4 1.6. Is the food preparation area open and available for observation? Excellent – indoors, enough space for food handlers to work 1 Good - indoors, lack of space to work 2 Poor – roof only 3 Very poor – outside in the open 4 Can't rate due to being locked 99 1.7. Is the food preparation area clean? Very clean 1 Mostly clean 2 Mostly unclean 3 Very unclean 4 1.8. How is the food being cooked? Biogas/ electricity / gas 1 Open fire 2 Uncooked 3 Cooked off the school grounds 4 RSP School Functionality Tool Page 2 of 6 1.9. Is there a food garden at the school? Yes, the food garden is well established 1 Yes, but the garden was recently started 2 Yes, but the garden is not well maintained 3 No 4 2. Section B: Hygiene and Healthcare 2.1. Which of the following sanitation facilities does the centre/school have for learners? Select One Flush toilets 1 Waterless toilets e.g. VIP, urine diversion toilet etc. 2 Pit latrines, bucket systems, potties 3 None 4 2.2. How many toilets TOTAL are there for learners in Foundation Phase (Grades R to 3)? Enter number 000 2.3. How many UNUSABLE toilets are there for learners in Foundation Phase (Grades R to 3)? Enter number 000 2.4. Is toilet paper available for learners? Toilet paper available at the learners toilets 1 Learners must ask teacher for toilet paper 2 Learners bring toilet paper from home 3 Toilet paper is not available or use newspaper, paper etc. 4 *** Please take a photo of the school’s sanitation facilities (toilets, pit latrines, bucket systems, potties, etc.) MALE *** Please take a photo of the school’s sanitation facilities (toilets, pit latrines, bucket systems, potties, etc.) FEMALE Ensure you are accompanied by a school representative and that no learners are present 2.5. Are the learners' sanitation facilities clean? Yes 1 Moderately clean – can improve 2 No 4 2.6. Are these sanitation facilities safe? Yes (e.g. closed area, learners can't fall or hurt themselves, toilets monitored to prevent abuse etc.) 1 Moderately safe – can improve (e.g. broken windows, leaking pipes, access could be improved etc.) 2 No (e.g. learners at risk of falling into pits, broken toilets, learners can injure themselves, no access control, learners at risk of abuse) 4 2.7. Are there hand washing facilities? RSP School Functionality Tool Page 3 of 6 Yes, running water and soap / sanitizer available 1 Yes, running water but no soap available 2 Sanitizer, tippy tap, or other facility e.g. bucket 3 None available or natural source 4 2.7.1.If yes, are the hand washing facilities clean? Yes 1 Moderately clean – can improve 2 No 4 2.8. Are there additional handwashing facilities made available at the school in response to COVID-19?? 2.9. Is hand sanitizer available at the school? Yes 1 No 2 Yes 1 No 2 2.9.1.If yes, please select where you see hand sanitizer being used (select all that apply) 2.10. Is drinking water provided? Yes - water fountain or tap water available 1 Yes - water buckets and cups 2 Water available from river or other natural source 3 No drinking water available 4 Reception area 1 Principal’s office 2 Classrooms 3 Near lunch rooms 4 Near Toilets 5 Other (please specify) 6 3. Section C: School Environment 3.1. Is the school periphery secured? Periphery secure, security working well (e.g. gate access is controlled, security fence secure) 1 Reasonable but some systems breaking down (e.g. lack of access control, broken walls) 2 Clear efforts being made to secure learners and educators’ safety, but a lack of resources to do so properly (e.g. fence broken, no security access) 3 Security is a problem that constantly arises, evidence of breaching readily apparent 4 RSP School Functionality Tool Page 4 of 6 3.2. Rate the safety of the school area Learners and teachers are safe and secure (e.g. their health and wellbeing is cared for, they are not at risk of injury while at school, emergency procedures are clearly visible) 1 Moderately safe – can improve (e.g. glass on the ground, learners could hurt themselves on equipment, holes in the ground, signage lacking or in poor condition) 2 Learners and teachers are not safe and secure (they are at risk of injury, there is a lack of visible emergency procedures) 4 3.3. Is there a designated sport ground? Yes, and very well looked after - clearly used 1 Yes, poorly looked after but used 2 Yes, but in very poor condition 3 No 4 3.4. Did you observe any physical education class during school hours? Yes, observed numerous times 1 Yes, but only one session 2 No evidence of physical education 4 *** Please take a general photo of the school field 3.5. Is the school area clean? Yes 1 Moderately clean – can improve 2 No 4 3.6. Does the school have a library / multi-media centre? Yes 1 No 2 3.7. If yes, rate the condition of the library/ multi-media centre (resources) Yes, lots of books and clearly used 1 Yes, lots of books but not clearly used (e.g. covered in dust) 2 No, lack of books, few books, or dated books 3 No books at all 4 3.8. If yes, rate the condition of the library/multi-media centre (cleanliness) Very clean and tidy 1 Moderately clean – can improve 2 Not clean or tidy 4 *** Please take a photo of the library RSP School Functionality Tool Page 5 of 6 3.9. Are learners monitored on entry and exit for adherence to COVID-19 regulations (e.g. temperature screening, wearing of face masks, etc)? 3.10. Are teachers monitored on entry and exit for adherence to COVID-19 regulations? (e.g. temperature screening, wearing of face masks, etc) Yes 1 No 2 Yes 1 No 2 3.11. Is there anything else you would like to note regarding the school’s adherence to COVID-19 regulations? Please write it down here: 4. Section D: Teaching & Curriculum Delivery 4.1. Are teachers teaching and learners learning? All or most classes have teachers actively teaching and engaging learners; free play is supervised 1 Some classes have teachers actively teaching and engaging learners; free play not always supervised 2 Some classes have teachers, but they are not teaching and lack of discipline in class 3 No supervision, classrooms are disorganised, chaos 4 5. Section E: Learning and Teaching Materials 5.1. Are the teaching and learning materials in the storeroom/strong room accessible and being used? Materials are well organised and accessible 1 Materials are stored but not easily accessible 2 Materials are stocked but not used 3 No visible materials or resources 4 6. Section F: School Functionality & Management 6.1. Is the SGB functional? SGB meets regularly, has minutes, and makes meaningful decisions 1 SGB meets regularly and has minutes but no meaningful decisions made 2 SGB meets sporadically and doesn’t always have minutes 4 *** Please take a photo of the minutes 6.2. Add in the dates of the last two SGB meetings from the minutes Enter Date 1 D D M M Y Y Enter Date 2 D D M M Y Y RSP School Functionality Tool Page 6 of 6 6.3. Does the school have a referral network? Yes, display emergency numbers and other resources to refer learners to 1 Yes, display emergency numbers only 2 No, no list of networks on display 4 *** Please take a photo of the referral list 6.4. Is there at least ONE functioning computer for school administration connected to the internet? Yes, and connected to internet 1 Yes, but NOT connected to internet 2 Yes, but not working 3 No computer 4 7. Section G: School Rating of Functionality 7.1. How does this school compare to other schools in the North West? Above standard (atmosphere at school supports learning, culture of teaching, principal is proud of the school etc.) 1 Average (teaching and learning is happening, basic school management is in place) 2 Below standard (learning and teaching is not happening, school is not being managed) 4 7.2. Would I send my child to this school? I would happily send my own child to this school 1 It once was a school that I would have considered for my child but it is deteriorating 2 The school is working but I would not send my own child to this school 3 The department needs to take immediate action to fix this school 4 Please add anything you would like to note about this school *** Photo 1: Please take a photo of anything you find interesting *** Photo 2: Please take a photo of anything you find interesting RSP Learner Workbook and Exercise Book(s) Assessment Page 1 of 3 Tool 5: RSP LEARNER WORKBOOK AND EXERCISE BOOK(S) ASSESSMENT 1. Instructions and identifying information Instructions: − Fieldworkers must get the DBE workbook and the Learner Exercise Book/s: HL EFAL DBE Workbook Exercise Books DBE Workbooks Exercise books Grade 1 1 All (+/- 3) 1 All (+/- 3) Grade 3 1 All (+/- 3) 1 All (+/- 3) − For the DBE Workbook: the period under review is Beginning Term 4 (2 November 2020) up to 27 November 2020 − For the Learner Exercise books the period under review Beginning of the year January 2020 to 16 November 2020 (all exercise books needed except the test book) Ask the teacher to select one of the most proficient learners in her own class. Then request to see that learner’s DBE workbook/s, which will have all the work that the learner has done from the beginning of Term 4 (2 November 2020) up to 27 November 2020 in them, and all of the learner’s exercise books, which will have the work that the learner has done since the beginning of the year to 16 November 2020 (all exercise books needed except the test book). 1.1 District: 1.2 School Name: 1.3 School EMIS number: 1.4 Grade: 1.5. Teacher name and surname: 1.6. Teacher PERSAL number: Grade 1 1 Grade 3 2 RSP Learner Workbook and Exercise Book(s) Assessment Page 2 of 3 1.8 Subject 1.9 The name and surname of this learner is _______________________________________ 1.10 The gender of the learner is 2. DBE workbook assessment. Please count and record the following work that appears in this learner’s DBE workbook from the beginning of Term 4 (2 November 2020) up to 27 November 2020 Type of content counted (i) The overall number of pages on which the learner completed any exercises (ii) The overall number of pages students had marked their own exercises (in pencil) (iii) The overall number of pages the teacher had marked any exercises (usually in pen) (iv) The number of pages involving writing* exercises completed by the learner (v) The number of pages involving exercises comprising the writing of at least one full sentence (vi) The number of pages involving exercises comprising the writing of at least one paragraph * This is when the learner wrote at least one letter in an exercise. 3. Learner exercise book(s) assessment Please count and record the following work that appears in this learner’s exercise book/s from the beginning of the year until 16 November 2020. Please ask for all books in case the learner has started working in a new book sometime during the year. Use the dates provided in the books as a reference for the number of days. Type of content counted (i) The overall number of days on which the learner completed any exercises Home Language 1 EFAL 2 Male 1 Female 2 RSP Learner Workbook and Exercise Book(s) Assessment Page 3 of 3 (ii) The overall number of days students had marked their own exercises (in pencil) (iii) The overall number of days the teacher had marked any exercises (usually in pen) (iv) The number of days involving writing* exercises completed by the learner (v) The number of days involving exercises comprising the writing of at least one full sentence. (vi) The number of days involving exercises comprising the writing of at least one paragraph. * This is when the learner wrote at least one letter in an exercise RSP Classroom Library Assessment Page 1 of 3 Tool 6: RSP CLASSROOM LIBRARY ASSESSMENT Instructions Tool administered only in classroom library project schools Administer tool in same classrooms where classroom observation tool is administered Identifying information: 1. District name: 2. School Name: 3. Teacher’s name: 4. Teacher’s PERSAL number: 5. Grade in which classroom library assessment observed: Grade 1 □ Grade 3 □ Review of classroom book collection/reading corner/box library or equivalent. Administer this tool only in classroom library project schools (see your fieldworker schedule) 1. Is there a book collection/reading corner/box library or equivalent evident in the classroom (may be kept in the classroom cupboard/storage space)? Yes 1 No 2 2. How many picture books (including comics) are available? (Note: different titles i.e. not copies of the same book) 3. How many story books (narratives – mainly text) are available? (Note: different titles i.e. not copies of the same book) 4. Are the books in the library available in English? None (Poor) 0 Fewer than 5 1 5-10 2 11-20 3 More than 20 (Excellent) 4 None (Poor) 0 Fewer than 5 1 5-10 2 11-20 3 More than 20 (Excellent) 4 RSP Classroom Library Assessment Page 2 of 3 5. Are the books in the library available in Setswana? 6. Are the books in the library available in any other South African language? Yes 1 No 2 Don’t know/unable to determine language/s 3 6.1. If yes, please select what South African languages are available (Select all that apply): Afrikaans English isiNdebele isiXhosa isiZulu Northern Sotho Sesotho Setswana Siswati Tshivenḓa Xitsonga Other (please specify) 7. Do the books seem to be made available for regular use by learners in the class? Yes 1 No 2 8. Is there a reading poster on display where learners can see the covers of all the books in book collection/reading corner/box library or equivalent? 9. Do you see coloured stickers on the books? 9.1. If yes, How many books can you see with a green sticker (Note: Write the number of books you can see):__ 9.2. If yes, How many books can you see with a pink sticker (Note: Write the number of books you can see):___ Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 RSP Classroom Library Assessment Page 3 of 3 9.3. If yes, How many books can you see with a blue sticker (Note: Write the number of books you can see):__ 10. Is there a library/book progress or management chart on display? Yes 1 No 2 10.1. If yes, is the library/book progress or management chart filled out? Yes 1 No 2 10.1.1. How is the library/book progress or management chart filled out? With the learners’ names and a black, green, pink or blue sticker 1 With the learners’ names and a green, pink or blue sticker 2 Only with the learner’s names but indicates whether learners signed books out and in 3 Only with the learners’ names and no indication of signing books out or signing books in 4 11. Can you see and reading cards available for learners to take out books? Yes 1 No 2 12. Are the books well displayed and stored (i.e. to ensure they are durable for the classroom environment)? Well-preserved, well displayed/stored 1 Adequately displayed/stored; shows effort although could be improved 2 Tatty, poorly displayed/ stored 3 13. Write down any comments and observations: ANNEX 5.2.3: TEACHER TOOLS RSP Teacher Survey Page 1 of 22 Tool 1: RSP TEACHER SURVEY Information Sheet and Consent Form for Teachers Khulisa Management Services (Khulisa) is undertaking an evaluation funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). What we are doing To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In addition, in 2020 we want to understand how the COVID-19 pandemic has affected the RSP implementation, and how the project has adapted its implementation. Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. RSP Teacher Survey Page 2 of 22 Your participation We are asking you, as a language teacher, to complete a questionnaire where you will provide some information on how you teach reading in your class, as well as your feedback on your experience with the RSP by completing a questionnaire. This year we also want to understand how the COVID-19 pandemic has affected your work and learn how you have adapted. We may also require to observe your Home Language and English First Additional Language lessons, and conduct a classroom observation. This is similar to the coach shadowing process that you may have participated in October 2019.The questions mainly cover teaching practices and your experience with the intervention. Completion of the whole questionnaire should not take longer than 20 - 30 minutes. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. Confidentiality Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. Records that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. Risks/discomforts We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially and anonymously, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not other Department officials at the circuit, district, provincial, or national offices. Benefits The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members and teachers will never be attached to any of the information that you provide or the information released through reports or articles. RSP Teacher Survey Page 3 of 22 Who to contact if you have been harmed or have any concerns If you have concerns or questions about the evaluation, you may call the evaluation manager, Margaret Roper at Tel: 011 447 6464 ext. 3230, or e-mail at mroper@khulisa.com You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za CONSENT I _________________________________ hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I also understand that I can contact Khulisa should I not want to continue. I understand that my participation will remain confidential to anyone outside of the evaluation study and RSP programme. I understand that the information that I provide will be stored in accordance with the POPI Act and that evaluation findings will be communicated to senior DBE managers, the programme funders, and through articles in academic journals without making known my identity or that of the schools _______________________ Signature ________________ Date PLEASE START THE SURVEY ON THE NEXT PAGE RSP Teacher Survey Page 4 of 22 1. Identifying information 1.1. District name: 1.2. School name: 1.3. Teacher’s name: 1.4. Teacher’s surname: 1.5. Teacher PERSAL number: 1.6. Teacher date of birth: 1.7. Teacher Sex: 1.8. Are you the Foundation Phase Head of Department (HOD)? 1.9. What Grade do you teach? 1.10. Did you teach in this same school in 2019? 1.10.1. If no, were you part of an RSP school in 2019? 1.10.2. If no, in what month did you join the current school you are in? o January 2020 o February 2020 o March 2020 o April 2020 o May 2020 o June 2020 o July 2020 o August 2020 o September 2020 o October 2020 o November 2020 2. RSP Learning and Teaching Support Materials (LTSM) 2.1. RSP planned for each Foundation Phase teacher to get quarterly packs of Learning and Teaching Support Materials (LTSM) and Lesson Plans. 2.1.1. Did you receive these LTSM and Lesson Plans? 2.1.2. If No, please explain/specify why you did not receive the LTSM Date DD/MM/YYYY Male 1 Female 2 Other 3 Yes 1 No 0 Grade 1 1 Grade 2 2 Grade 3 3 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Teacher Survey Page 5 of 22 2.1.3. If yes, what terms did you receive LTSM for? Term 1 (January 2020) Term 2 and Term 3 (September 2020) Term 4 (November 2020) 1 2 3 2.1.4. If yes, which of the following LTSM did you receive between January and November 2020? (Please select ALL RECEIVED) with an “X”) Scripted lesson plans (Termly Lesson Plans) HL - plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Scripted lesson plans (Termly Lesson Plans) EFAL- plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (HL) Theme Vocabulary Flashcards (EFAL) Sight Words Flashcards (HL) 1 2 3 4 5 Sight Words Flashcards (EFAL) RSP Big Books (EFAL) RSP Graded Readers (EFAL) Handwriting poster Posters - Listening and Speaking 6 7 8 9 10 Assessment Record Book Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term Teacher Management and Storage files Classroom Library 11 12 13 14 2.1.5.How often do you use EACH of the LTSM outlined below? (Please mark one with an “X”) Not Applicabl e (I did not receive this LTSM) Neve r Approximatel y once a term Approximatel y once a month Approximatel y once a week Nearl y every day Every day RSP Teacher Survey Page 6 of 22 Scripted lesson plans (Termly Lesson Plans) HL 0 1 2 3 4 5 6 Scripted lesson plans (Termly Lesson Plans) EFAL 0 1 2 3 4 5 6 Theme Vocabulary Flashcards (HL) 0 1 2 3 4 5 6 Theme Vocabulary Flashcards (EFAL) 0 1 2 3 4 5 6 Sight Words Flashcards (HL) 0 1 2 3 4 5 6 Sight Words Flashcards (EFAL) 0 1 2 3 4 5 6 RSP Big Books (EFAL) 0 1 2 3 4 5 6 RSP Graded Readers (EFAL) 0 1 2 3 4 5 6 Handwritin g poster 0 1 2 3 4 5 6 Posters - Listening and Speaking 0 1 2 3 4 5 6 Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term 0 1 2 3 4 5 6 2.1.6.Did you find the LTSM useful? Yes Most of it 2 Yes Some of it 1 Not at all 0 2.1.7. If yes, which LTSM do you find the MOST USEFUL for Home Language (HL)? (Please select ONE with an “X”) RSP Teacher Survey Page 7 of 22 Scripted lesson plans (Termly Lesson Plans) HL - plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (HL) Sight Words Flashcards (HL) 1 2 3 2.1.8. Which LTSM do you find the MOST USEFUL for English First Additional Language (EFAL)? (Please select ONE with an “X”) Scripted lesson plans (Termly Lesson Plans) EFAL- plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (EFAL) Sight Words Flashcards (EFAL) RSP Big Books (EFAL) RSP Graded Readers (EFAL) 1 2 3 4 5 2.1.9.Overall, which LTSM do you find the MOST USEFUL? (Please select ONE with an “X”) Handwriting poster Posters - Listening and Speaking Assessment Record Book Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term Teacher Management and Storage files 1 2 3 4 5 Classroom Library I did not find any LTSM useful 6 7 3. Teaching and COVID-19 3.1. When did you return to school after schools reopened after COVID-19 closures? Before August 24th 2020 1 After August 24th 2020 0 3.2. When did you start teaching after schools reopened after COVID-19 closures? Date YYYY/MM/DD RSP Teacher Survey Page 8 of 22 3.3. During this year (2020), have you used any lesson plans or teaching support material in an electronic format, e.g. on a tablet or computer? 3.4. During school closures, did you send any materials home with your learners? 3.4.1. If yes, was this something that the school planned with all teachers or that you decided to do on your own only for your learners? 3.4.2. If yes, what kind of materials or resources did you make available to learners during COVID￾19 school closures? (Please mark all that apply with an “X”) DBE workbook s HL reading books EFAL reading books Worksheet s/ printed activities Links to online websites and resource s Other 3.5. If yes, how did you sent learners these materials? Whatsapp 1 Parent collected at the school 2 Sent home with learners before school closed 3 3.6. During school closures, did you communicate with your learners through their parents or caregivers? Yes 1 No 0 3.6.1. If yes, how did you communicate with them? WhatsApp 1 SMS 2 Telephone Calls 3 Email 4 Other 5 3.6.2. What did you communicate about (Select all that apply)? Sent them learning materials 1 Asked them about their wellbeing 2 Encouraged them to read with their children 3 3.7. Before the COVID-19 school closures, how much time did you spend teaching EFAL per week? (Please mark one with an “X”) 1 hour 1.5 hours 2 hours 2.5 hours 3 hours 3.5 hours 4 hours 4.5 hours 5 hours I don’t know 1 2 3 4 5 6 7 8 9 99 3.8. In the last month, how much time have you spent teaching EFAL per week? 1 hour 1.5 hours 2 hours 2.5 hours 3 hours 3.5 hours 4 hours 4.5 hours 5 hours I don’t know Yes 1 No 0 Yes 1 No 0 School planned all teachers sent materials home with learners 1 School planned but only some teachers sent materials home with learners 2 I decided to send the materials to my own learners 3 RSP Teacher Survey Page 9 of 22 (Please mark one with an “X”) 1 2 3 4 5 6 7 8 9 99 3.9. In the last month, how much time have you spent teaching HL per week? (Please mark one with an “X”) 6 hour 6.5 hours 7 hours 7.5 hours 8 hours 8.5 hours 9 hours 9.5 hours 10 hours I don’t know 1 2 3 4 5 6 7 8 9 99 3.10. In the last week, did you manage to teach all the learning areas to your class as planned? Yes 1 No 2 3.11. If yes, which learning area did you manage to mostly focus on? (Please mark one with an “X”) Home Language EFAL Life Skills Mathematics 3.12. If no. please share what your challenges are in teaching in all the learning areas to your class. 3.12.1. In your opinion, do you think the maximum number of students are attending your class/lessons under COVID-19 restrictions? Or could more learners attend? Yes the maximum number is attending my classes 1 No, more students can attend my classes 2 3.12.1.1. Why do you think this? 3.12.2. How many days were you unable to come to school in the last month on days you were scheduled to teach? RSP Teacher Survey Page 10 of 22 Number 3.12.2.1. Indicate the number of days for each of the reasons provided below. Illne ss Person al leave Teach er trainin g Official meetin gs Protes ts in area Union activiti es Communi ty memorial COVID-19 risk (comorbiditi es) COVI D-19 sick leave Isolatio n due to COVID￾19 exposu re Oth er 1 2 3 4 5 6 7 8 9 99 4. RSP Training 4.1. Between January and November 2020, did you attend training workshops on how to use the RSP lesson plans and other LTSM? Yes 1 No 0 4.1.1. If yes, which RSP training workshops have you attended (Select all you attended)? Term 1 (January 2020) Term 2 and Term 3 (September 2020) Term 4 (November 2020) 1 2 3 4.1.1.1. If selected Term 1, How valuable did you find the content of Term 1’s workshop? Extremely valuable Valuable Somewhat valuable Not at all valuable 4 3 2 1 4.1.1.2. If selected Term 2 and 3, How valuable did you find the content of this workshop? Extremely valuable Valuable Somewhat valuable Not at all valuable 4 3 2 1 4.1.1.3. wo If selected Term 4, Ho rkshop? w valuable did you find the content of Term 4’s Extremely valuable Valuable Somewhat valuable Not at all valuable 4 3 2 1 4.1.2. If you missed one or more of the training workshops, what were the reasons (select all that apply)? [note skip pattern to question 4.1.1] Reasons Tick all that apply I was sick and could not attend Personal reasons I did not receive communication on it I was at a different school before that was not part of the project RSP Teacher Survey Page 11 of 22 I was teaching a different grade My school did not receive the invitation There were no trainings due to COVID 19 lockdown and school closure Other, please specify If your reason is not on the options above, please specify why you missed one or more workshops: 4.1.3. From the workshops you attended, did you find any workshop more useful than others? 4.1.3.1. Is yes, which RSP training workshop did you find most useful (Select one)? Term 1 (January 2020) Term 2 and Term 3 (September 2020) Term 4 (November 2020) I found them all equally useful 1 2 3 4 4.1.3.1.1. Why did you prefer this workshop over the others you attended? Yes 1 No 0 RSP Teacher Survey Page 12 of 22 4.1.4. Please rate how useful the following training workshop content elements were for Home Language (Please mark one with an “X”): Very useful, I learned a lot Somewhat useful, learned some new things Not useful to me, but well delivered Not useful to me and needs improvement Weekly Routine 4 3 2 1 Phonics Lessons 4 3 2 1 Reading Lessons 4 3 2 1 Handwriting Lessons 4 3 2 1 Writing Lessons 4 3 2 1 Listening and Speaking Lessons 4 3 2 1 Assessment Tasks 4 3 2 1 Revised Annual Teaching Plans 4 3 2 1 4.1.5. Please rate how useful the following training workshop content elements were for EFAL: (Please mark one with an “X”) Very useful, I learned a lot Somewhat useful, learned some new things Not useful to me, but well delivered Not useful to me and needs improvement Weekly Routine 4 3 2 1 Phonics Lessons 4 3 2 1 Reading Lessons 4 3 2 1 Handwriting Lessons 4 3 2 1 Writing Lessons 4 3 2 1 Listening and Speaking Lessons 4 3 2 1 Assessment Tasks 4 3 2 1 Revised Annual Teaching Plans 4 3 2 1 4.1.6. As part of the training workshops, you were trained on the revised Annual Teaching Plans to help you implement the revised curriculum. Please describe how useful those were for you and how they helped you in the classroom: 4.1.6.1. If you could change one thing about the training on the revised Annual Teaching Plans what would it be and why: RSP Teacher Survey Page 13 of 22 4.1.7. Did you receive the RSP LTSM for Term 4 at the training in November 2020? 4.1.7.1. If no, how did you receive your LTSM for Term 4 in November 2020? I had the LTSM delivered to my school My coach brought the LTSM for me when s/he came for a school support visit I received the LTSM from other teachers in my school who attended the training I did not receive any LTSM 1 2 3 4 Other (please specify):____________________________________________________________________________ 4.1.7.2. If no, when did you receive the LTSM for Term 4 (November 2020)? A week after the training Two weeks after the training A month after the training Only at the next training session Never 1 2 3 4 5 Other (please specify):_____________________________________________________________________________ 4.1.8. Did you receive the LTSM in time for you to use in the classroom in Term 4 (November 2020)? 5. RSP Coaching 5.1. Do you receive visits from an RSP reading/ literacy coach? Yes 1 No 0 N/A (did not attend training) 2 Yes 1 No 0 Yes 1 No 0 RSP Teacher Survey Page 14 of 22 If yes, please respond to all questions on this next section. If no, skip to Section 6. RSP Classroom Library 5.1.1.If yes, please rate the statements below related to your experience with the coach (Select one option in each row): (Please mark one with an “X”) To a very large extent To a large extent Neutral/ Not Applicable To a limited extent Not at all My coach communicates the planned coaching visits with me 5 4 3 2 1 My coach communicates the planned coaching visits with me sufficiently in advance such that I am able to accommodate coaching visits in my schedule 5 4 3 2 1 My coach’s observations and feedback have helped me grow as a teacher 5 4 3 2 1 I have gained helpful resources from my coach 5 4 3 2 1 My coach communicates regularly with me 5 4 3 2 1 My coach maintains open, two-way communication with me 5 4 3 2 1 My coach helps me overcome barriers to teaching and learning 5 4 3 2 1 My coach holds reflective feedback sessions with me 5 4 3 2 1 My coach sets action plans after each feedback session 5 4 3 2 1 The coach helped me implement the revised Annual Teaching plans 5 4 3 2 1 5.1.2. If yes, please write down the number of visits/observations your coach conducted with you? In term 4 only How many visits have you received from your coach How many times did your coach observe your Home Language lessons How many times did your coach observe your English First Additional Language lessons How many times did your coach give you feedback on your lessons RSP Teacher Survey Page 15 of 22 5.1.3. Has your coach interacted with you using a WhatsApp group during the schools closure period? 5.1.3.1. If yes, did you find the coaches’ support through WhatsApp useful? 5.1.4. Have you received any written feedback forms from your coach? 5.1.5. How (if at all) has the coach supported you to implement the revised Annual Teaching plans? 5.1.6. Are you aware that your coach should be having school based workshops in the afternoons when he/she visits the school? 5.1.7. Have you participated in any school-based afternoon workshops organised by your coach? 5.1.7.1. If yes, how many have you attended: 5.1.7.1.1. In term 1 (January-March 2020):________ 5.1.7.1.2. In term 3 (August-October 2020) only:_________ 5.1.7.1.3. In term 4 (November-December 2020) only:_________ 5.1.7.2. If yes, do you feel you would benefit from more workshops like these? 5.1.8. If yes, what is the content of these workshops? Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Teacher Survey Page 16 of 22 5.1.9. Did you receive catch up training from your RSP coach on any RSP LTSM workshops you missed? Yes 1 No 0 I did not miss any RSP LTSM training workshops 2 5.1.9.1. If yes, please select for which training workshops you received RSP coach catch up training: Term 1 Term 2 Term 3 Term 4 1 2 3 4 5.1.9.2. If yes, what was the quality of this catch up training? 5.1.10. Do you interact with your coach for support outside of school visits? 5.1.10.1. If yes, what platform do you mainly use to collaborate? WhatsApp 1 Calls or text messages 2 Face to Face 3 Other 4 If Other, specify_______________________________________________________________ 5.1.11. Were you part of the Early Grade Reading Study (EGRS) previously? 5.1.11.1. If yes, did you also receive coaching support from coaches affiliated with the Early Grade Reading Study (2015 – 2017)? Yes 1 No 2 Good 3 Adequate 2 Poor 1 Yes 1 No 0 Yes 1 No 2 I don’t know 3 RSP Teacher Survey Page 17 of 22 5.1.11.1.1. If yes, how does your experience with the RSP reading coach compare with your experience with the EGRS coach? The RSP literacy coach support is: Better quality than support provided through the EGRS coach Same or similar quality to support provided through the EGRS coach Poorer quality than support provided through the EGRS coach 3 2 1 6. RSP Classroom Library 6.1. Did you receive an RSP Classroom Library kit/box? If yes, please complete the following questions on this section. If no, skip to Section 7. School Management Team 6.1.1. Were you trained on the management and use of the classroom library? 6.1.2. Were you orientated on the library starter-kit and shown examples of how to use each component? 6.1.3. Do you feel you received enough support to use the classroom library? 6.1.4. Do you use the classroom library kit/box? 6.1.4.1. If no, why not? 6.1.4.2. If yes, how often do you make use of the classroom library? (Please mark one with an “X”) Once a month Every 2 weeks Once a week 2-4 times a week Everyday Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 Yes 1 No 0 RSP Teacher Survey Page 18 of 22 1 2 3 4 5 6.1.5. How helpful do you find having a classroom library? (Please mark one with an “X”) Not at all helpful Somewhat helpful Extremely helpful 1 2 3 6.1.6.You received an A5 plastic sleeve as part of your classroom library kit/box. How useful do you find this A5 plastic sleeve? (Please mark one with an “X”) Not at all useful Somewhat useful Extremely useful 1 2 3 6.1.6.1. Why do you find the A5 plastic sleeve useful/ not useful? 6.1.7. Have you appointed selected learners as classroom library monitors? 6.1.7.1. If yes, what duties have they fulfilled? (Please select all that apply with an “X”) Manage the books Book box and chart management Other 1 2 3 If Other, please specify:________________________________________________________________ 6.1.8. Do learners take books home? 6.1.8.1. If no, why not? Yes 1 No 0 Yes 1 No 0 RSP Teacher Survey Page 19 of 22 6.1.9. Are learners interested in taking books home? (Please mark one with an “X”) None Few learners Some learners Most learners All learners 1 2 3 4 5 7. School Management Team 7.1. Do you feel supported by your School Management Team? 7.1.1.If yes, please rate the statements below related to your general experience with the SMT i.e. the Principal, Deputy Principal, Foundation Phase Head of Department or Other SMT member (Select one option in each row): Not at all To a limited extent Neutral/ Don’t know To a large extent To a very large extent My SMT encourages reporting on SA-SAMS in the school 1 2 3 4 5 My SMT identifies and responds to bottlenecks and other issues 1 2 3 4 5 My SMT supports book clubs at the school 1 2 3 4 5 My SMT supports a culture of reading in the school 1 2 3 4 5 I receive supervisory support from my SMT 1 2 3 4 5 My SMT holds regular meetings/updates in my school to increase accountability 1 2 3 4 5 Yes 1 No 0 RSP Teacher Survey Page 20 of 22 My SMT supports teachers in the coaching process and ensure they have the space and time available for observations, individual feedback and afternoon workshops 1 2 3 4 5 My SMT observes my lessons 1 2 3 4 5 My SMT models any lessons I struggle to teach 1 2 3 4 5 My SMT has encouraged or praised me for a lesson well taught 1 2 3 4 5 7.1.2 If no, please explain what support you need from school management: RSP Teacher Survey Page 21 of 22 8. RSP experience 8.1. Were you part of the RSP in 2019? 8.1.1. If yes, how has your experience with the RSP been in 2020, compared to 2019? 8.1.2. Please explain why your experience was better/the same or worse? 8.2. Was there a curriculum practice that was difficult to implement given COVID-19 protocols such as social distancing? For example: Group Guided Reading, paired reading, etc.. 8.2.1. If yes, please explain what difficulties you had with implementing the curriculum practice. 8.3. On a scale from 1-10 (1 being least likely and 10 being most likely) how likely are you to recommend the RSP to a friend, colleague or others? 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Yes 1 No 0 Better than 2020 3 About the same as 2020 2 Worse than 2020 1 Yes 1 No 0 RSP Teacher Survey Page 22 of 22 8.4. If you could change one thing about the RSP what would it be and why? Thank you for your time and effort! ANNEX 5.2.4: TRAINING OBSERVATION TOOLS RSP Training Observation: Coach Dry Run Page 1 of 7 Tool 1: RSP TRAINING OBSERVATION: COACH DRY RUN Coach Dry Runs: 23 October 2020 The following aspects must be looked for when observing the coach dry runs for the Reading Support Project (RSP): 1. How confident the coaches are in the dry run in terms of delivering this training (confidence in the materials and confidence in delivering training, as evidenced in the training style, ability to answer questions, pacing through the training curriculum, etc.) 2. The degree of fidelity to the content of the RSP training materials Please note that you must complete an information sheet and a rating sheet for each coach (14 coaches in total) during the dry run sessions. RSP Training Observation: Coach Dry Run Page 2 of 7 Coach [NUMBER] Information Name of the coach (Coach 1) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach [NUMBER] Dry run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. RSP Training Observation: Coach Dry Run Page 3 of 7 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. RSP Training Observation: Coach Dry Run Page 4 of 7 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes RSP Training Observation: Coach Dry Run Page 5 of 7 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous RSP Training Observation: Coach Dry Run Page 6 of 7 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes RSP Training Observation: Coach Dry Run Page 7 of 7 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes RSP Coach Training Observation Page 1 of 4 Tool 2: Coach training observation Name(s) of the master trainer(s) conducting the training Date of observation Session observed (EFAL or HL) Venue of observation (e.g. Zoom) Number of coaches/teachers attending training Name of researcher/evaluator Start time of observation End time of observation The following tool is to be used when observing coach or teacher training provided by the Reading Support Project (RSP). While observing any training, the researcher/evaluator should consider the following: 1. Whether the facilitators are training participants effectively on subject knowledge (EFAL and HL literacy for Grades 1, 2 and 3) and pedagogical knowledge (i.e. how to teach literacy in these languages) 2. How confident the facilitators are in delivering this training (confidence in the materials and confidence in delivering training, evidenced training style, ability to answer questions, pacing through the training curriculum, etc.) 3. The degree of fidelity to the content of the RSP training materials as the train-the-trainer model is rolled out (i.e. are coaches training teachers according to what was done in their own training?) 4. Whether participants demonstrate more knowledge over the course of the training (e.g. asking good questions, engaging with the content) 5. What factors constrain or enhance the scale up of the RSP training approach (e.g. language issues, numbers of people being trained, training environment, other training practices, etc.) The questions on this tool are divided into the following categories: 1. Facilitator(s) knowledge and practice 2. Content of the training 3. Participant engagement 4. Session format 1. Facilitator(s) knowledge and practice 1.1. Is the facilitator effective in delivering/facilitating the training sessions (s)? (i.e. clear on training objectives, emphasizes key points, answers questions, manages participant’s engagement, etc. o Yes o Somewhat o No RSP Coach Training Observation Page 2 of 4 1.1.1.Please expand on facilitator effectiveness (i.e. describe how he/she facilitates session) 1.2. Was the facilitator knowledgeable about the topic? (e.g. does he/she appear like they came prepared, answer questions clearly, etc.) o Yes o Somewhat o No 1.2.1.Please expand on facilitator knowledge (i.e. provide examples of showing content knowledge, etc.) 1.3. Did the facilitator build on previous modules/knowledge of participants (e.g. make reference to other topics taught and make links to what participants should already know?) 1.4. Does the facilitator ask questions to participants that allows for learning and reflection? o Yes o Somewhat o No 1.4.1.Please expand on facilitator use of reflective pedagogy 1.5. Does the facilitator model the appropriate behaviour expected from participants? o Yes o Somewhat o No 1.5.1.Please expand on facilitator use of reflective pedagogy 2. Content of the training 2.1. Are the training sessions building the necessary skills, according to the purpose and objectives of the training? o Yes o Somewhat o No 2.2. Was the planned content covered during the training (Note: Check detailed course outline or agenda provided to see if content above was delivered.) 2.3. How was the session facilitated? What type of activities were done in class? 2.4. Does training differentiate per Grade in content delivery? o Yes o No 2.5. What Grades were covered during the training (if presented differentiated training)? Select all that apply o Grade 1 o Grade 2 o Grade 3 RSP Coach Training Observation Page 3 of 4 2.6. What Subject(s) are covered during the training observation? Select all that apply o Home Language (HL) o English First Additional Language (EFAL) 2.7. If so, approximately how much time spent in each (EFAL vs HL)? 2.7.1.EFAL: o Very little (only a quarter of the training session) o About half of the training session o Most of the training session o All of the training session 2.7.2.HL: o Very little (only a quarter of the training session) o About half of the training session o Most of the training session o All of the training session 2.8. What language(s) are used by trainers during the observation? (e.g. Setswana and/or English 2.9. Are the facilitators building participants’ content knowledge (i.e. in HL and EFAL literacy? o Yes o Somewhat o No 1.2. Are the facilitators building participants’ pedagogy (i.e. the practice of teaching literacy and teaching methods)? o Yes o Somewhat o No 3. Participant engagement 3.1. Is the style of training participatory? o Yes o Somewhat o No 3.2. Were participants engaged in training sessions? o Yes o Somewhat o No 3.3. Is there a high level of interest and excitement by participants? o Yes o Somewhat o No o Can’t tell (e.g. if virtual session and can’t see faces of participants) 3.4. How is engagement managed by the facilitator? 3.5. Do all participants get a chance to engage in session? Do all participants get a chance to give their thoughts and opinions on matters? (i.e. is there a balance between input, discussion and response?) o Yes o Somewhat RSP Coach Training Observation Page 4 of 4 o No 3.6. Did participants do the required pre-readings? (please explain) 4. Training format 4.1. How smoothly did the training it go? (i.e. think of online VS face to face training, how maximise participation, what was used to engage participants, were people late/on time, etc.) 4.2. Was the pace of the session adequate for learning and sharing reflections insights? 4.3. How is the session delivered/structured? E.g. facilitator presents only and then asks questions? Shows videos? Shares screen? Have activities built into the session? 4.4. Is there variety in how presentations are made? (i.e. the delivery of training is not monotonous) o Yes o Somewhat o No 4.5. Are the training materials used (e.g. presentations, videos, handouts, workbooks, etc.) well received by participants and are the learning points clear? Please explain Other notes: ANNEX 5.2.5: INTERVIEW TOOLS RSP Curriculum/Subject Advisor Semi-Structured Interview Guide Page 1 of 4 Tool 1: RSP Curriculum/Subject Advisor Semi-Structured Interview Guide Date of interview: Name of interviewer: 1. Identifying information: Role of advisor 1.1. Name: 1.2. Sex/Gender: Male Female Other 1.3. Grade(s) covered by Curriculum/Subject Advisor (Select all that apply) Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Other If other: (specify)_____________________ 1.4. Languages (Select all that apply) Setswana EFAL 1.5. District(s) in which Curriculum/ Subject Advisor works Ngaka Modiri Molema Dr. Kenneth Kaunda 2. Familiarity with RSP: 2.1. How familiar are you with the RSP programme? (prompt: please explain what you know about it) 2.2. Did you receive a tablet loaded with RSP materials? (note: only applicable for Curriculum Advisors) 3. Attendance at RSP Trainings 3.1. RSP conducted a number of Reference Group meetings this year. 3.1.1. Did you attend the following meetings (please select the ones attended): Reference Group meeting Tick all that apply 26 June 2020 13-14 July 2020 1-2 October 2020 Other If other: (specify)_____________________ If attended any sessions ask questions below: 3.1.1.1. Why did you attend these Reference Group meetings? 3.1.1.2. What did you enjoy or not about being part of these Reference Group meetings? 3.1.1.3. This year due to COVID-19 the meetings mainly took place online. How did these compare to the in-person meetings you used to have? (Probe: what did you like about online VS in-person, which one do you prefer most and why?) 3.1.1.4. Do you feel you received adequate communication on the reference group meetings? (probe: did you know of the dates well in advance? Did you understand what was expected of you?) 3.1.1.5. Do you feel that the issues you raised (if any) in the reference group on previous training was heard and responded to? Why or why not? (Probe: how?) 3.1.1.6. How was the content of the reference group meetings? (probe: what did you like most about them? What would you change about the content?) RSP Curriculum/Subject Advisor Semi-Structured Interview Guide Page 2 of 4 3.1.1.7. Was there sufficient time allocated for the different activities required during the reference group meetings? Why or why not? During the reference group meetings, you meant to contribute to developing and refining RSP materials such as the lesson plans, baseline assessments, revised Annual Teaching Plans and the weekly planning tool. 3.1.1.8. What was your role in developing/refining the materials? (probe: please discuss and provide examples of each of the materials) 3.1.1.9. Do you feel that your contributions to these materials were incorporated in the final version? (probe: please discuss and provide examples of each of the materials) 3.1.1.10. What would you change about the reference group meetings, if anything? 3.1.2. If you didn’t attend any Reference Group meetings, why was that? 3.2. RSP conducted a number of training sessions with coaches. 3.2.1. Did you attend the following coach training sessions (please select the ones attended): Coach training Tick all that apply 13-16 July 2020 21-23 October 2020 Head coach training 28-30 August 2020 Other If other: (specify)_____________________ 3.2.1.1. If yes, what did you enjoy or not about being part of these coach training sessions? 3.2.1.2. If yes, did you assist in training any coaches (Probe: please elaborate how you assisted: For example how did you provide feedback during coach dry runs, how did you engage during delivering training content, how did you train the coaches on the school baseline assessments, etc.)? 3.2.1.3. How well did you think the trainers had prepared? 3.2.1.4. How well do you think the coaches were trained in (Probe: ask official to provide detail on each of the items below): • EFAL lesson plans, + other LTSMs • EFAL curriculum content and skills • Sestwana lesson plans, + other LTSMs • Sestwana curriculum content and skills • facilitation skills (to deliver the teacher training) 3.2.1.5. How did you find the coaches’ dry runs in preparation for the teacher training? (Probe: did you notice improvement in coaches over time? Were the coaches comfortable?) 3.2.2. If you didn’t attend any coach trainings, why was that? 3.3. RSP conducted a number of training sessions with teachers. 3.3.1. Did you attend the following training sessions (please select the ones attended): Teacher training Tick all that apply Term 1 (in person): 20-28 January 2020 Term 3: September 2020 Term 4 : 3-5 November 2020 Other If other: (specify)_____________________ 3.3.1.1. If yes, why did you attend? RSP Curriculum/Subject Advisor Semi-Structured Interview Guide Page 3 of 4 3.3.1.2. If yes, what did you enjoy or not about being part of these training sessions? 3.3.1.3. Over the past two years, do you feel the training delivery has improved? Why or why not? (Probe: please share examples) 3.3.1.4. If you noticed improvement, what training areas improved the most? 3.3.1.5. If you noticed improvement, what training areas improved the least or remain problematic? 3.3.1.6. Did you provide any feedback on the teacher training over the past two years? Yes No 3.3.1.6.1. If yes, what feedback did you provide? And to whom? 3.3.1.6.2. If yes, who responded to your feedback? How promptly were issues addressed? 3.3.1.6.3. If yes, do you feel your feedback was incorporated in future trainings? 3.3.1.7. If yes, did you assist in training any teachers (please elaborate how you assisted)? 3.3.2. If you didn’t attend any trainings, why was that? 3.4. Can you comment on how the RSP activities have contributed to your professional development? 4. Perception of the RSP 4.1. What is your role in the RSP? 4.1.1. What is your role in relation to Subject Advisor / Curriculum Advisor / Coach / Teacher and SMT? 4.1.2. How is your role different to that of the Head Coach? 4.1.3. Has this role changed since 2019? 4.1.4. Have you been able to fulfil this role? Why/Why not? 4.2. What is the role of the RSP coaches? Did their role change as a result of COVID-19 and school closures? 4.3. In your view, how effective are the coaches? Have you seen a change between last year and this year? Please explain your answer 4.4. What are the main challenges to improving early reading in the schools you support? 4.5. What is working well in improving early grade reading? Please explain your answer 4.6. Is there evidence of RSP practices being implemented in class by teachers? Please explain your answer (evidence for this statement?) 4.6.1. Can you compare the implementation of RSP in the class by teachers last year (2019) to now? Please explain what has changed or not. 4.7. Have RSP LTSM been provided to all the participating schools? 4.7.1. Please explain / expand (are teachers using it? Have they seen the classroom libraries? Is it quality material in their opinion) 4.8. How could the RSP be made more efficient? 4.9. Do you think the RSP is a cost efficient programme? 4.9.1. What, if anything, do you regard as good value for money in the RSP? (Prompt: which part of the RSP do you think has brought about the biggest changes, why? If you had to invest the budget in reading support, what would you spend your money on?) 4.9.2. Were the activities you were expected to support the RSP (e.g. attending reference group meetings in person, attending coach and teacher training, attend online trainings or meetings) were funded in the existing budgets within the district? RSP Curriculum/Subject Advisor Semi-Structured Interview Guide Page 4 of 4 4.10. How were the time demands of engagement with the RSP? (Probe: do you think it was adequate? Too much of your time? Too little can would have liked to be more involved?) 4.11. If necessary, how could management and implementation structures / processes be improved? 4.11.1. Have these structures and processes improved since last year (2019) when you were last interviewed? 4.12. Do you receive reports about the RSP? Yes No 4.12.1. If so from whom do you receive them? 4.12.2. Does the content of these meet your needs? Could the monitoring and reporting system be improved? 4.12.3. What do you do with the information in the reports? Please explain 4.13. If the RSP went to every Foundation Phase in the District / Province, what would be the critical elements that the program MUST take into consideration for this to be achieved? (Examples: how train teachers, how ensure quality, how will materials be distributed etc.) 4.13.1. How should district and provincial officials be involved if the project went to all schools (scale)? 4.13.2. What time commitment would they need to give to RSP? 4.14. What are the key considerations for taking the coaching element of the RSP to scale? 4.15. Do you think RSP should be scaled to all the schools in the district and province? 4.16. What has worked well in the past for taking education (teaching and learning) initiatives to all schools in the district and province? 4.17. Have you noticed any changes in RSP delivery since last year (2019)? 4.17.1. What has been the most significant of these changes? 4.18. Discussion and reflection on key comments official made in interviews last year 5. COVID 19 5.1. What effect did the COVID 19 lockdown have on schools, teaching and learners? 5.2. What plans and actions were or are being taken to keep HL and EFAL on track against the curriculum? (e.g., were you required to provide additional support to schools/teachers? 5.3. Did you communicate more via WhatsApp or other digital/virtual platforms?) 6. Any further comments? RSP Provincial & District Official Interview Questionnaire Page 1 of 2 Tool 2: PROVINCIAL & DISTRICT OFFICIAL INTERVIEW QUESTIONNAIRE PERSONAL DETAILS Interview date Day Month Year Name of Interviewer Name of Interviewee (s) Interviewee Organisation / Department Interviewee Title / Designation QUESTIONS 1. How have you been involved in the RSP (if at all)? What is your role? 2. Do you feel adequately supported in order to fill your role and to support Subject Advisors? 3. How do the roles of coaches, Subject Advisors and district officials work together? 4. Are you aware of the Early Grade Reading Study (EGRS) and did you have any experience with the EGRS study? 5. In your experience what are the key similarities between the EGRS and the RSP? 6. What effect did COVID-19 have on the RSP in the Province/District? 7. Has the RSP been integrated into the provincial structures and processes and, if so, how? 8. Is the RSP more integrated this year than last year (2019)? 9. Are you seeing any positive changes in the schools as a result of the RSP? If yes, what are these and why you think they occurred? If no, why not? 10. Would you say that schools that have a supportive SMT do things differently compared to others? (e.g. how does the SMT play a part in encouraging literacy development) 11. Do you feel that the RSP literacy coaches have a unique role in supporting teacher literacy practices? If so, how is this role different to the support provided by HODs, SA’s, and other district or provincial support structures?” 12. Do you think the support of the coaches to teachers might have helped teachers with teaching during COVID-19? (e.g. virtual teaching, implementing COVID-19 precautions in classrooms, etc.) 13. Could the management and implementation structures of the RSP be improved? If so how? If not, what is working well in how the program is being managed? 14. Have there been any changes/improvement in implementation or management of the RSP since last year (2019)? 15. Do you receive reports on the RSP? If so, what type of reports and when do you receive them? And from whom? 16. Could the monitoring and reporting system be improved? Please explain 17. How has the monitoring and reporting system changed since 2019? 18. Are you able to use the reports you receive to make decisions? Does it meet your needs? 19. If the RSP went to scale in the district or province, how could this be achieved in terms of: a. RSP teacher training RSP Provincial & District Official Interview Questionnaire Page 2 of 2 b. RSP coaching c. RSP materials d. How to involve the district and provincial officials e. Quality f. Languages 20. What has been successful in terms of implementing RSP in the districts? What should then be replicated for scale? 21. What are the major challenges you foresee in scaling RSP in the district or province? 22. Do you think that to scale-up RSP would be valuable? Why or why not? 23. Is the RSP a cost effective project? Why or why not? 24. What value (for money) does the RSP bring to early grade reading? 25. Discussion and reflection on key comments official had mentioned last year: 26. What are the lessons from implementing the RSP during COVID-19 pandemic? Are there any good practices we can replicate when taking a project like the RSP to scale? 27. Any other comments and observations? Thank you. RSP Office Staff Interview Questionnaire Page 1 of 3 Tool 3: RSP OFFICE STAFF INTERVIEW QUESTIONNAIRE PERSONAL DETAILS Interview date Day Month Year Name of Interviewer Location of interview/ Type of interview o Phone o Skype/Zoom o In person Number of participants o Individual o Group Name of Interviewee (s) Interviewee Title / Designation Office/Organisation o FPD RSP Head Office o FPD RSP Provincial Office INFORMED CONSENT • My name is ______________. I am working with Khulisa Management Services, to conduct an independent evaluation of the Reading Support Project (RSP). • I am here today to ask some questions about the RSP implementation, your role in it, and your insights into what has worked/has not worked. • We have [60 - 90 minutes] for our time together. Are you available to respond to some questions during this time? • (Consent) This interview is entirely voluntary and you may choose not to participate. If you agree to participate, you can choose to stop at any time or to skip any questions you do not want to answer. Your answers and your participation in this interview are completely confidential. We will not share any information that identifies you with anyone outside of the evaluation team. • Please feel free to stop this interview at any time to ask questions you may have about this consent or anything else. • Do I have your consent to proceed? o Yes o No If Yes, proceed with the interview. RSP Office Staff Interview Questionnaire Page 2 of 3 QUESTIONS RSP implementation and management 1. How long have you been involved in the RSP? 2. What is your role in the RSP? 3. What role has FPD played in the RSP? 4. Did you feel adequately supported in order to fulfil your role in the RSP? Please explain and suggest anything that could have been done differently to support you 5. Please explain your role in relation to other staff members of RSP. i.e. who do you report to? Who do you manage? Who do you communicate with? 6. In your opinion, what has been the biggest success in the RSP implementation? What worked well? 7. In your opinion, did the recommendations from the Formative Implementation Evaluation and your (FPD) Improvement Plan change the implementation in 2020 in anyway? Please explain your answer. 8. To what extent you think the project outcomes have been achieved? Give examples (prompt: in terms of teacher practice, LTSM, learner outcomes contribution the coaches have made, SMT) 9. Could the management and implementation structures of the RSP be improved? 9.1. If so how? 9.2. If not, what is working well in how the program is being managed? 9.3. What would the ideal consortium and organisational structure be for RSP in replicating the program? 10. How did you engage with the other Consortium partners? 11. Do you think the consortium had the right partners – why or why not? [if relevant, ask Who else needed to be part of the RSP?] 12. How do you interact with the RSP/FPD head office OR provincial office (Note to: ask depending on office and position)? 13. Have there been any changes/improvement in implementation or management of the RSP since last year (2019)? 14. How do you think the RSP could be made more efficient? 15. Have you faced any challenges in implementing the RSP? If so, what were these? 16. Do you think the RSP will be sustained in the schools? Why or why not? 17. What do you think needs to happen or be strengthened for sustaining the RSP? 18. What effect did COVID-19 have on the RSP implementation in 2020? Please expand on any changes in implementation: what worked, what didn’t work? 19. Please expand on how the SMT training implementation took place. Any difference in the planned implementation due to not using the volunteers (VSO) anymore? Working with the Department of Education 20. How do you (if at all) work with the Department of Education national, provincial and district offices? 21. What is working well in relation to how you work with the Department of Education? 22. What could be improved? M&E and Learning 23. How has the monitoring and reporting system changed since 2019, if at all? 24. Could the monitoring and reporting system be improved? Please explain 25. How are learnings shared within the RSP consortium to inform program adaptation and improvement? 26. What M&E data do you see as essential/required to implement a project such as the RSP? (i.e. what data do you absolutely need to have? And in what format?). e.g. do you need reach data? Change in knowledge data? Sustainability RSP Office Staff Interview Questionnaire Page 3 of 3 27. How do you think the RSP could be made more efficient? 28. Do you think the RSP will be sustained in the schools? Why or why not? 29. What do you think needs to happen or be strengthened for sustaining the RSP? Cost & Time 30. Have you been working on the RSP implementation full time or part time? 30.1. If part time, how much time did you spend on RSP related work per day? Per week? Per month? 31. Did your time spent on the RSP change in the following periods? Please expand on how much time you spent on each period per week/month: 31.1. Term 1 (January-March 2020) 31.2. School closure (March-August 2020) 31.3. Term 3 and 4 (August-December 2020) 32. Do you think the RSP is a cost effective project? Why or why not? 33. What do you regard as good value for money in the RSP? (e.g is it the coaching? Lesson plans? Training, etc.) 33.1. Why or Why not? Scaling Up 34. What suggestions do you have for the DBE to take this project to scale? (expand to more provinces) 35. What can be scaled from the RSP? (Prompt: a. RSP teacher training b. RSP coaching c. RSP materials d. How to involve the district and provincial officials e. Quality f. Languages) 36. What are the major challenges you foresee in scaling RSP (into more schools in the districts, more provinces, etc.)? 37. Do you think that to scale-up RSP would be valuable? Why or why not? Lessons 38. What is the one thing you would do like to see done differently in the RSP? 39. What advice would you give to an organisation that is about to start the RSP? 40. What advice would you give to DBE in provinces and districts if they wanted to start the RSP? Conclusion of Interview Thank you for your time. This concludes the interview. Your insights have been very valuable and we are going to use the information that you provided to us to inform the findings in the evaluation report. Before I go, 41. Do you have anything else you would like to add, or you think we should know before we leave? 42. Who else do you think I should talk to that can provide a different viewpoint? 43. Do you have any questions for me? RSP Consortium Members Interview Questionnaire Page 1 of 3 Tool 4: RSP CONSORTIUM MEMBERS INTERVIEW QUESTIONNAIRE PERSONAL DETAILS Interview date Day Month Year Name of Interviewer Location of interview/ Type of interview o Phone o Skype/Zoom o In person Number of participants o Individual o Group Name of Interviewee (s) Interviewee Titles / Designation Office/Organisation o Molteno o Oxford University Press INFORMED CONSENT • My name is ______________. I am working with Khulisa Management Services, to conduct an independent evaluation of the Reading Support Project (RSP). • I am here today to ask some questions about the RSP implementation, your role in it, and your insights into what has worked/has not worked. • We have [60 - 90 minutes] for our time together. Are you available to respond to some questions during this time? • (Consent) This interview is entirely voluntary and you may choose not to participate. If you agree to participate, you can choose to stop at any time or to skip any questions you do not want to answer. Your answers and your participation in this interview are completely confidential. We will not share any information that identifies you with anyone outside of the evaluation team. • Please feel free to stop this interview at any time to ask questions you may have about this consent or anything else. • Do I have your consent to proceed? o Yes o No If Yes, proceed with the interview. RSP Consortium Members Interview Questionnaire Page 2 of 3 QUESTIONS RSP implementation and management 44. How long have you been involved in the RSP? 45. What is your role in the RSP? 46. Did you feel adequately supported in order to fulfil your role in the RSP? Please explain 47. In your opinion, what has been the biggest success in the RSP implementation? What worked well? 48. To what extent you think the project outcomes have been achieved? Give examples (prompt: in terms of teacher practice, LTSM, learner outcomes contribution the coaches have made, SMT) 49. Have there been any changes/improvement in implementation of the RSP since last year (2019)? 50. Have there been any changes/improvement in management of the RSP since last year (2019)? 51. Could the management and implementation structures of the RSP be improved? 51.1. If so how? 51.2. If not, what is working well in how the program is being managed? 51.3. What would the ideal consortium and organisational structure be for RSP in replicating the program? 52. Have you faced any challenges in implementing the RSP? If so, what were these and how did you overcome them? 53. Did you produce / deliver anything less than what was contracted? If so, what was this? And why did it occur? 54. Did you produce / deliver anything beyond what was contracted? If so, what was this? And why did it occur? 55. How was your experience working as part of the FPD consortium (e.g. how was the interaction between consortium members? How did you manage conflict? 56. How do you interact with the RSP/FPD head office and provincial office? 57. How have you interacted with the department of education (district, provincial, national level)? 58. What worked well in your interactions with the DBE (district, provincial, national level)? 59. How could the partnership between FPD consortium and DBE (district, provincial, national level) be strengthened? 60. What effect did COVID-19 have on the RSP implementation in 2020? Please expand on any changes in implementation: what worked, what didn’t work? M&E and Learning 61. How has the monitoring and reporting system changed since 2019, if at all? 62. Could the monitoring and reporting system be improved? Please explain 63. How are learnings shared within the RSP consortium to inform program adaptation and improvement? 64. What M&E data do you see as essential/required to implement a project such as the RSP? (i.e. what data do you absolutely need to have? And in what format?). e.g. do you need reach data? Change in knowledge data? Sustainability 65. How do you think the RSP could be made more efficient? 66. Do you think the RSP will be sustained in the schools? Why or why not? 67. What do you think needs to happen or be strengthened for sustaining the RSP? Cost & Time 68. Have you been working on the RSP implementation full time or part time? 68.1. If part time, how much time did you spend on RSP related work per day? Per week? Per month? 69. Did your time spent on the RSP change in the following periods? Please expand on how much time you spent on each period per week/month: 69.1. Term 1 (January-March 2020) RSP Consortium Members Interview Questionnaire Page 3 of 3 69.2. School closure (March-August 2020) 69.3. Term 3 and 4 (August-December 2020) 70. Do you think the RSP is a cost effective project? Why or why not? 71. What do you regard as good value for money in the RSP? (e.g is it the coaching? Lesson plans? Training, etc.) 71.1. Why or Why not? Scaling Up 72. What suggestions do you have for the DBE to take this project to scale? (expand to more provinces) 73. What can be scaled from the RSP? (Prompt: g. RSP teacher training h. RSP coaching i. RSP materials j. How to involve the district and provincial officials k. Quality l. Languages) 74. What are the major challenges you foresee in scaling RSP (into more schools in the districts, more provinces, etc.)? 75. Do you think that to scale-up RSP would be valuable? Why or why not? Lessons 76. If you could change one thing about the RSP, what would it be and why? 77. If you had to start this project again, what advice would you give to an organisation that is about to start the RSP? 78. If you had to start this project again, what advice would you give to DBE if they wanted to start the RSP? Conclusion of Interview Thank you for your time. This concludes the interview. Your insights have been very valuable and we are going to use the information that you provided to us to inform the findings in the evaluation report. Before I go, 79. Do you have anything else you would like to add, or you think we should know before we leave? 80. Who else do you think I should talk to in your organisation that can provide a different viewpoint or deeper insight? 81. Do you have any questions for me? ANNEX 5.3: COACHING CRITERIA Coaches were observed during lessons and for SBWs (if they occurred) in 2019 and 2020. The following aspects of lessons were observed. - Planning and Preparation; this relates to how prepared for the lesson observation the coach is (do they compare their notes on the teacher to previous observations, etc.) and whether the coach communicates their visits in advance to the teachers. - Pre-observation discussion; coaches were expected to engage with the teacher before the lesson began about the teacher’s own lesson preparation. - Classroom observation; this related to what a coach did during a lesson. Coaches might have checked learners’ books to see progress and teacher marking, coaches might have been active and followed along with group guided reading (if it occurred), and coaches should have been making detailed notes using the lesson observation tool (and in some cases a notebook as well) to be able to provide relevant feedback. - Post-observation discussion/Feedback: This is the most important stage of the coaching practice (and elaborated on is the case studies under coaching practice). This relates to how feedback is given to teachers after their lesson, and what feedback is given. Coaches were expected to encourage teachers to reflect on their own practice and identify action steps, with guidance from the coach. Critical feedback should be balanced with positive reinforcement and observation of progress, or elements conducted successfully in the lesson. Feedback should also cover; curriculum coverage, classroom management, and RSP methodology where applicable. The most variation is seen between coaches’ performance in this stage, and in how feedback is delivered. - School-Based Workshops: Coaches were expected to identify needs of teachers and conduct SBWs with all teachers in the Foundation Phase based on their observations. This was used as the main criteria for the rating the quality of the coaches by the FPD Consortium. However, SBWs varied in practice. If the coach conducted the SBW they were considered to have met the criteria. The matrix of skills and competencies for coaches used in the coach rubric and case studies was as follows: Skills and Competencies Excellent Average to good to Weak coaches to Meets all duties required of them Meets all criteria and in some cases, exceeds criteria Meets criteria to varying degrees. Might be stronger or weaker in different areas. Might have missed some steps, but outperformed in other areas Coach tended to do the bare minimum, or not meet criteria at all Maximizes time at school Identifies needs of teachers and holds School Based workshops Provided virtual support via WhatsApp Encourages teacher reflection Facilitates growth of teacher Quality of relationship with teachers/SMT Evidence of content knowledge/ RSP methodologies/ technical ability ANNEX 6: FIDELITY TABLES 1 6.1 Number of Schools Reached Number of schools reached with assistance Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of public and private schools receiving USG assistance. Baseline data 2019: 298 Annual Cumulative Planned target: 255 251 RSP Custom Indicator – None 2 Number of participating schools (2019 / 2020) Design specification Initial target was 298 schools revised to 263 from 298 in 201911, and then revised to 255 in 202012 . 2019 2020 Monitoring Data 256 schools participated 42 fewer than original specification 7 fewer than revised specification of 2019 Project schools. Actual Target Coaching 68 75 SMT 67 65 LTSM only 121 123 Grand Total 256 263 251 Schools participated 47 fewer than original specification 4 fewer than revised specification of 2020 Project schools. Planned Actual Target Coaching 65 67 69 SMT 68 65 66 LTSM only 121 119 120 Grand Total 254 251 255 Data note 2019: School master list as on 28 Oct 2019. 11 Final Evaluation Report: Design Evaluation of the Reading Support Project November 2019 12 FPD. MEL Plan May 2020. 3 2020: School master list provided on 29 Jan 2021. The 2020 Target specified in the MEL plan was of 255 (Target), but the RSP list of schools indicated that 254 schools were part of the plan (Planned), and the monitoring data showed that ultimately 251 schools participated (Actual). Note: One school (coded school 1) was retained on the school list but did not participate at all in 2020. (Email correspondence, 8 Feb 2021) 4 6.2 Number of teacher training sessions (2019 and 2020) Indicator -- Number of teacher training sessions held Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicator -- Number of teacher training sessions held. Annual Cumulative Planned target: A total of 4 JIT13 teacher training sessions (2*2 day and 1*1 day training sessions) scheduled annually. Duration = 6 days per year. 2019: 4 sessions with a duration of 6 days 2020: 3 sessions of shorter duration 13 Referred to in the RSP design as Just in Time (JIT) training 5 Design specification • 4 sessions per year • 6 days per year Number of teacher training sessions Term 1 Term 2 Term 3 Term 4 2019 2020 2019 2020 2019 2020 2019 2020 Monitoring Data (Actual data reported by FPD for period) Feb – One day Jan – No data* Apr – Two days No training – school closures Jun – Two days Sep – cluster training Sep – One day Total: Six days in 2019 Nov – cluster training Khulisa observation / Project documentation N/A N/A N/A N/A N/A N/A N/A One day Data note 1. Actual duration (hours) of session not reported. 2. Since training in term 3 of 4 2020 was conducted in clusters (due to COVID-19 restrictions) it is not possible to report on the length of the training session. Targets were not amended due to COVID-19. However, a training circular for Term 4 and Khulisa observations confirm that the training for Term 4 took place over one day. 3. *Data was not provided 6 6.3 Number of teachers participating in training sessions by Grade, School and Term Indicator -- Number of educators completed professional development activities Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of educators who complete professional development activities with USG assistance. Annual Cumulative Planned target: 1,578 2019: 1,882 Adjusted target was 155 schools with 6 teachers per school = 1,530 2020: 1,580 RSP Custom Indicator -- (note: attending at least one session) Output Indicator -- Number of participating teachers in training sessions Target: All educators from participating schools are expected to attend (note, the data is unclear if all educators from participating schools attended) 2019: 1,882 2020: 1,580 RSP Custom Indicator -- Outcome Indicator -- (#) Percentage of teachers who attended the full training program. Target: 1578 in 2019, but 1,530 adjusted target. Teachers from all qualifying schools are expected to attend training sessions to complete the CAPS curriculum. 28% (which is 547 of 1,925) teachers attended all seven sessions 7 Design Specification Original Design Specification: 263 schools with six teachers each = 1,578 Adjusted 255 schools with six teachers each = 1530 Number of teachers and schools attending training by Grade Grade 1 Grade 2 Grade 3 Total Target / Expected Number 2019 2020 2019 2020 2019 2020 2019 2020 Number of FPD linked Schools 278 251 270 244 283 248 25214 255 Number of teachers who attended training 661 537 586 517 624 525 1,882 1,580 1,530 Average number of teachers per school 2.38 2.14 2.21 2.12 2.20 2.12 6.79 6.27 6 Number of teachers who attended: All sessions (4 in 2019 and 3 in 2020) 283 357 264 345 279 357 826 1,059 1,530 Three sessions in 2019 204 195 200 599 8 Design Specification Original Design Specification: 263 schools with six teachers each = 1,578 Adjusted 255 schools with six teachers each = 1530 Number of teachers and schools attending training by Grade Grade 1 Grade 2 Grade 3 Total Target / Expected Number 2019 2020 2019 2020 2019 2020 2019 2020 Two sessions 115 127 102 120 108 121 325 369 One session 59 53 36 52 37 47 132 152 Data note 1. A complete list of teachers for all enrolled schools, which indicates the status of each teacher was not maintained during the program 2. Only teachers who participate at least once in the program are included in this analysis. 3. 2019: Teachers from non-project schools are included, in 2020 teachers from non-project schools are not included. 4. The teacher training data set include 6 teachers of one Primary school (coded school 4), a school that was removed from the project - training sessions for these teachers are excluded. 5. 2020: The data set used for this analysis was not extracted from the RedCap system. The data in the Excel data set is based upon source documents and thus more complete than the data in the RedCap system. (Email correspondence, Feb 8, 2021) 9 6.4 Number of teachers who attended training by term Design Specification 2019: 263 schools with six teachers each = 1,578 2020: Adjusted 255 schools with six teachers each = 1530 Number of teachers who attended training by term TERM 1 TERM 2 TERM 3 TERM 4 TOTAL 2019 2020 2019 2020 2019 2020 2019 2020 2019 2020 Monitoring Data 1,545 1,387 1,711 No training 1,377 1,288 1,368 1,392 1,882 1,580 Teacher survey respondents (self￾reported) 83.6% 89.6% 90.5% 80.1% 91.4% Term 2 and 3 combined 89.5% 91.2% 304 307 Data note 1. The design specification assumes that six teachers per school will be trained. However, the Term 2 data indicate that more than six teachers per school were trained. 2. The number of teachers trained (as per FPD monitoring data) includes participants from non-project schools, as well as individual teachers that attended more than one grade’s training. A significant variance in teacher attendance between training events is likely to have a significant resource implication. Such a variance requires further investigation. 3. 2019: Reported data: There is a discrepancy between the number of teachers trained per the monitoring data and the reported figures in the FPD indicator table, which is appended to FPD’s quarterly reports to USAID. Monitoring data is updated with late submissions even after reporting deadlines and additional data quality control is conducted after the reporting deadline. 4. 2019 and 2020: Since a teacher master list was not available only teachers who participate at least once in 2019 or 2020 are included in are this analysis. 5. 2019: Teachers from non-project schools are included. 6. 2020: The numbers represent only teachers from project schools. The teacher training data set include 6 teachers of one Primary school (coded school 4), a school that was removed from the project - training sessions for these teachers are excluded. 7. The data set used for this analysis was not extracted from the RedCap system. The data in the Excel data set is based upon source documents and thus more complete than the data in the RedCap system. (Email correspondence, Feb 8, 2021) 8. 2020: Small inconsistencies in the data were corrected where possible e.g. incorrect EMIS numbers for schools. 10 6.5 Number of teachers who participated per grade per year in coaching activity Design Specification Six Foundation phase teachers per school x 135 Coaching schools15 = 810 teachers Grade 1 Grade 2 Grade 3 Total Number of teachers who participated 2019 2020 2019 2020 2019 2020 2019 2020 Monitoring Data16 235 (87%) 279 (103%) 226 (84%) 247 (91%) 227 (85%) 230 (85%) 668 (82%) 756 (93%) Number and percentage of teachers who received17: Five and more individual coaching sessions 2 (1%) 10 (3%) 1 (0.5%) 8 (2%) Two to four individual coaching sessions 109 (29%) 189 (50%) 204 (57%) 171 (50%) One individual coaching session 109 (29%) 80 (21%) 92 (26%) 68 (19%) No individual coaching sessions 159 (42%) 100 (26%) 150 (42%) 108 (30%) Total number of teachers who participated in at least one coaching activity in 2019 or 2020 379 379 355 355 Data notes 1. 2019: The data on the number of supported teachers is entered from monitoring tools submitted by coaches and head coaches. It is likely that the figures presented here is an underestimate of the actual support visits, since the FPD M&E office is unable to enforce the submission of this data. 2. The total number of teachers who received coaching in 2019 might differ slightly from the previous table since some teacher records did not include a unique identifier and thus could not be merge across the 2019 and 2020 data sets. The 2019 grade 3 teacher data set did not include any unique identifiers and therefore could not be merged and reported in this format. 3. 2020: The data was extracted from the RedCap system, a total number of sessions per teacher for 2020 was provided. 15 Figures as per FPD database, November 2019. 16 This analysis made use of separate data sets for 2019 and 2020 participation- only teachers who participated in the specific year is counted. 17 The analysis used a merged dataset thus the total number of participants includes teachers who participated in 2019 only, in 2020 only or in both years. The grade 3 datasets for 2019 and 2020 could not be merged. 11 6.6 Number of individual coaching sessions per year Indicator -- Number / Percentage teachers receiving coaching sessions Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicator Target: 810 Output Indicator -- Number of teachers receiving at least one individual coaching session per teacher (teachers in the coaching intervention) 2019: 668 (82%) 2020: 759 (93%) RSP Custom Indicator -- Outcome Indicator -- % of teachers receiving individual coaching support (lesson observations for both languages and/or attendance to SBWs) over the year. Target: Educators are expected to receive at least 5 individual coaching sessions annually. 2019: 1% (3 received 5 sessions) 2020: 2% (18 teachers received 5 sessions) 12 6.7 Number of lesson observations conducted by coaches Indicator -- Number of lesson observations conducted by coaches Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicator Output Indicator -- Number of lesson observations conducted annually per teacher. Target: Each teacher is supposed to receive a minimum of 5 individual sessions over a year 810 teachers x 5 sessions = 4,050 individual coaching sessions per year 2019: 1,127 of 4,050 (28%) 2020: 1,778 of 4,050 (44%) Adapted target for this report Due to COVID-19 and data issues the calculation was amended based on actual data available and implementation of coaching over the 2 terms when coaches could provide coaching to teachers: Calculated for 3 terms in 2019 (target = 3,038) and 2 terms in 2020 (target = 2,025) Adapted Target 2019: 1,124 of 3,038 (37%) 2020: 1,782 of 2,025 (88%) Data note: According to this indicator teachers were expected to receive at least 10 individual coaching sessions over a period of 8 terms. However due to COVID-19 and data issues, data for 5 terms could be analyzed. Three terms in 2019 and two in 2020. The adapted target per teachers for 2019 is thus 3.75 and 2.5 for 2020 totaling 6.25 coaching sessions per teacher. For this analysis the target was rounded to 6. 13 Design Specification Four to six coaching sessions per teacher in 2019 and in 2020 Grade 1 Grade 2 Grade 3 Total Average number of individual coaching sessions (Lesson observations) per teacher per year 2019 2020 2019 2020 2019 2020 2019 2020 Monitoring Data 1.0 1.96 1.1 1.91 1.0 1.79 1.0 1.89 Number of individual coaching sessions Expected number of individual coaching sessions in 2019 and 2020 for 4 terms = 4,050. Adapted: 2019 for 3 terms = 3038 Adapted: 2020 for 2 terms = 2025 Monitoring Data 388 654 375 595 364 529 1,127 1,778 Teacher Survey Data 70 69 67 69 62 76 200 214** Data notes 1. *Data was provided based on Terms only and not by grade (Term 1 = 116; Term 2 = 636; Term 3 = 1213) 2. 2019 Data for Terms 1 to 3 only 3. Data notes: 2019 - Actual number of individual coaching session is likely an underestimate. The data is captured from monitoring tools submitted by coaches and head coaches, and it is reported that the monitoring tools are sometimes received late by the FPD M&E team. 4. Reported data: There is a discrepancy between number of sessions as per the monitoring data (1,127) and the reported figures in FPD indicator reporting table (1,965) which is appended to FPD’s quarterly reports to USAID. An interview with FPD staff indicates that they use a combination of coaches’ reports and captured lesson observation forms to report. 5. **Teacher survey data: 5 teachers from LSTM only schools reported receiving visit from coach in 2020, affecting final number reported. 14 6.8 Number of coaches who received professional development Indicator -- Number of coaches who received professional development Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of coaches (including head coaches) who receive professional development with USAID funding Baseline: 16 Annual Cumulative Target: 16 16 RSP Custom Indicator (Output) Number of coaches who complete training Target: 16 The coach training is defined as 3 days of training held quarterly and includes coach dry runs per term 16 Design specification 14 literacy and 2 head coaches receive training prior to the start of each term (3 days) Total of 16 coaches and 8 sessions Actual number of Coaches participated in coach training Term 1 Term 2 Term 3 Term 4 2019 2020 2019 2020 2019 2020 2019 2020 Monitoring Data 16 16 16 No monitoring data for this virtual training 16 16 16 16 15 6.9 Number of School Based Workshops (SBW) Indicator -- Number of School Based Workshops (SBW) Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicator Output Indicator -- Average number of SBW per school Target: Each school should have 15 SBWs per year 2020: 1.4 SBW per school (10%) RSP Custom Indicator -- Output Indicator -- Number of individual teachers attend at least one SBW per quarter. Target: 810 teachers Data not reported per quarter. RSP Custom Indicator - Outcome Indicator -- % of teachers attending School-Based Workshops (SBWs). Target: A target of SBWs conducted per school was set at 15 annually, teachers should attend all SBWs at the school. Adapted target for 2020: 7.5 2020: 600 of 796 (75%) attended at least one of the SBWs. 2020: No teachers attended 6 or more SBWs Data note: RSP monitoring data was not reported by quarter. The RSP Formative Report noted that the 2019 data was underreported. 16 2019 2020 Design Specification 8-12 workshops per school in 2019. By September, an average of 6.4 to 9.6 workshops per school would have been expected (terms 3 in 2019) Total number of school-based workshops in 2019: 1680 By September 2019, 1,344 of these workshops would have been expected. 15 SBW in 2020 per coaching school SBW were only conducted in two terms of 2020. Total number of SBW for a full year: 135*15 = 2,025 Total number of SBW for 2 terms: 2,025/2= 1,012 Total number of teachers participation sessions for full year = 810*15 = 12,150 Total number of teachers participation sessions for full year = 810*7.5 = 6,075 Data source: Monitoring data 2019 2020 Total number of SBW 194 (10% of target) Average number of SBW per school 1.4 (target = 15) 194 held in 135 schools Number of teacher participation sessions 359 949 (8% of target and 16% of adapted target) Number of teachers participating 107 (57% of all teachers) 111 (54% of all teachers) Average Number of teachers participating in SBWs per school 1.8 1.8 Data notes 1. 2019: The number of school-based workshops reported in the reported data includes other school visits as well e.g., delivery of material. 2. Data reported for period Feb to Aug 2019 3. Total number of school support visits provided by term (Term 1= 250; Term 2 = 406; Term 3 = 545) 4. 2020: Two data sets were received. The data in the Excel sheet was more comprehensive and complete, and was therefore used in this analysis. Number of school based workshops conducted per school as reported by principal in school survey 2019 2020 Number of schools reporting at least one SBW conducted 21 20 Number of SBWs conducted per school # of workshops : # schools 1: 9 2: 5 3: 3 4: 1 5: 1 6: 1 7: 1 8: 1 N/A 17 6.10 Number of SMT PLC sessions (2019-2020) Indicator -- Number of PLC sessions conducted Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicators - Number of PLC sessions conducted. Target: 5 PLCs including a wrap-up session planned for this intervention. Target: SMT from all participating schools expected to attend. 5 63 of 65 expected school sent SMT members to training 18 6.11 Number of tablets distributed to SMTs and number of school visits Indicator -- Number of tablets distributed Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of tablets distributed. Baseline: 129 Annual cumulative target: 334 111 (81%) RSP Custom Indicator – None18 N/A Number of tablets distributed Number of school visits Design Specifications 139 SMT members from 65 schools receive a tablet computer 65 schools receive support visits from a VSO volunteer (2019) Monitoring data 111 (81%) 2019: 18 schools (28%) 27 school visits as some schools received more than one visit. 2020: No school visits Data notes: Some tablets were listed as withdrawn and these were excluded from the final number. Evaluator comment: The evaluators were unable to confirm that the tablets were actually used for the project 18 The tablets pre-loaded with relevant content as well as the school support visit are include in the Theory of Change narrative and in the Design Specification. 19 6.12 Number of schools with participating SMT members Indicator -- Number of schools with participating SMT members Number of schools Design specification Revised to 65 in 2019 by end of project Monitoring Data (Actual data reported by FPD for period) 66 schools reached in total but 2 were coaching only schools19 and not supposed to attend the training 1 fewer than the revised specification One school’s SMT members did not attend any of the 5 PLC sessions. Reported Data Not reported as a separate indicator Data note: District officials and Facilitators were removed from the attendance list. Number of SMT members trained 2019 2020 Total Monitoring Data (Actual data reported by FPD for period) Session 1 = 110 / 103 (76%) Session 2 = 115 / 106 (79%) Session 3 = 85 / 83 (61%) Session 4 (Feb 2020) = 84 (62%) Session 5 (Nov 2020) = 63 (47%) 137 SMT members participated (101%) 3 schools - one representative 19 Two schools (schools code 2 and 3) are coaching only schools - SMT members from these two schools each attended one PLC. 20 Design specification 130 SMT members from 65 schools participate in training Reported (Reported number as per FPD Indicator reporting table to USAID) Not reported as a separate indicator Number of SMTs who attended: All sessions (3 in 2019 and 2 in 2020) 34 (25%) Four sessions 35 (26%) Number of individual SMTs members who submitted POE: All four 42 (31%) At least one 89 (65%) None 48 (35%) Data note Cleaned data set to removed District Officials and other program staff for 2019 data. Did not exclude the attendance of SMT members from coaching only schools. 21 6.13 Number of SMT completing professional development activities Indicator -- Number of SMT members complete professional development activity Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of education administrators and officials who complete professional development activities with USG Assistance (Principals/Deputy Principals, DHs) Baseline: 103* Annual Cumulative Planned target: 134 137 RSP Custom Indicator Output Indicator -- Number of SMT members completing at least one PoE Target: SMT members need to submit 4 PoE in order to successfully complete the course and earn the full amount of SACE points. 89 or 65% completed one PoE (see below for those reaching all four) Outcome Indicator -- - # (%) SMT completing professional development activities (including submission of PoE) Target: All participants to attend and submit PoEs and earn South African Council of Educator (SACE) continuing professional development points. Note from FPD MEL Plan: Each of the 4 SMT program modules are linked to a PoE assignment which must be completed and submitted for assessment. SMT are supposed to submit all 4 to earn the maximum amount of SACE points. The PoE will form part of the package to be submitted to SACE for CPTD points. Data note: The evaluators were unable to confirm if SACE points were actually awarded. 31% (41 SMT members) completed all four PoE 22 Outcome Indicator -- - # (%) SMT completing professional development activities (including submission of PoE) Target: All participants to attend and submit PoEs and earn South African Council of Educator (SACE) continuing professional development points. Note from FPD MEL Plan: Each of the 4 SMT program modules are linked to a PoE assignment which must be completed and submitted for assessment. SMT are supposed to submit all 4 to earn the maximum amount of SACE points. The PoE will form part of the package to be submitted to SACE for CPTD points. Data note: The evaluators were unable to confirm if SACE points were actually awarded. 31% (41 SMT members) completed all four PoE 23 6.14 LTSM Training and Distribution Indicator -- Number of primary school classrooms that receive LTSM Indicators (MEL plan of May 2020) Implementation USAID indicator: Number of primary school classrooms that receive a complete set of essential reading instructional materials with USG assistance Baseline data 2019: 1,501 classrooms Annual Cumulative Planned target: 1,530 2019: 1,071 (70%) 2020: Terms 1 and 4 = 1,117 (73%) and Terms 2 and 3 = 1,481 (97%) RSP Custom Indicator - Number of educators receiving lesson plans and complete LTSM package (by language, Grade, school) Target: 1,578 - LTSM packages for both EFAL and HL are distributed during training and all teachers are expected to receive the relevant packages. This indicator will also capture LTSM collected for educators who did not attend training, as well as LTSM delivered to LTSM only schools post-training. Records of LTSM received by teachers kept per school in 2019 as well as Terms 2 and 3 of 2020. Data Note: Data not provided by classroom 24 RSP Intervention Target number of schools Number of schools - No data available Number of schools - All LTSM received Number of schools with Shortage of LTSM 2019 2020 2019 2020 2019 2020 2019 2020 LTSM and Training Schools 123 120 11 (8%) T1, T4: 1 (1%) T2, T3: 0 (0%) 99 (74%) T1, T4: 84 (69%) T2, T3: 116 (96%) 23 (17%) T1, T4: 37 (31%) T2, T3: 5 (4%) Coaching Schools 140 135 9 (7%) T1, T4: 0 (0%) T2, T3: 0 (0%) 84 (65%) T1, T4: 101 (75%) T2, T3: 132 (97%) 37 (28%) T1, T4: 34 (25%) T2, T3: 3 (3%) Total 263 255 20 (8%) 183 (70%) 60 (23%) Data notes: The data file for Term 2 and 3 lists the LTSM collected per school, the data file for Term 1 and 4 lists the LTSM collected per teacher, which is summarized per school for the purposes of this analysis. The data for one Primary School was excluded for the purpose of this analysis. Inconsistencies between EMIS numbers and school names were fixed. There remained uneven reporting by teacher of receipt of the materials, as illustrated in the table below. Percentage of teachers reported receiving the materials 2020 GRADE INTERVENTION GROUP Grade 1 Grade 2 Grade 3 Coaching Coaching & SMT LTSM only Scripted Lesson Plans (Termly Lesson Plans) HL 96% 93% 95% 96% 95% 93% Scripted Lesson Plans (Termly Lesson Plans) EFAL 96% 93% 95% 96% 94% 93% Theme Vocabulary Flashcards (HL) 89% 83% 82% 83% 84% 87% Theme Vocabulary Flashcards (EFAL) 97% 98% 96% 98% 97% 96% Sight Words Flashcards (HL) 82% 79% 80% 78% 79% 84% 25 Percentage of teachers reported receiving the materials 2020 GRADE INTERVENTION GROUP Grade 1 Grade 2 Grade 3 Coaching Coaching & SMT LTSM only Sight Words Flashcards (EFAL) 94% 95% N/A 95% 92% 94% RSP Big Books (EFAL) 82% 81% N/A 59% 63% 61% RSP Graded Readers (EFAL) N/A 78% 62% 62% 64% 62% Handwriting Poster 57% 64% 61% 61% 63% 57% Posters - Listening and Speaking 73% 77% 67% 76% 75% 65% Assessment Record Book 48% 51% 38% 47% 48% 41% Curriculum Tracker/ Teacher Monitoring and Support Tool 68% 71% 66% 69% 73% 61% Teacher Management and Storage Files 22% 25% 16% 23% 21% 18% RSP Big Books (HL) (Term 4 2020) 79% 77% N/A 59% 64% 53% Booklets (Teacher-Guides) HL Power Point Presentation (Term 4 2020) 46% 48% 36% 47% 42% 39% 26 Percentage of teachers reported receiving the materials 2020 GRADE INTERVENTION GROUP Grade 1 Grade 2 Grade 3 Coaching Coaching & SMT LTSM only Classroom Library** 43% 42% 44% 41% 48% 40% **Libraries were distributed to 30 of the 59 sampled schools. 27 6.15 Curriculum Advisor participation in project activities Indicator -- Curriculum / Subject Advisors attend Reference Group Meeting Indicators (MEL plan of May 2020) Implementation USAID indicator: None RSP Custom Indicator Output Indicator -- Number of Curriculum / Subject Advisors attending Reference Group meetings Original Target: 16 Revised Target: 12 Reference group meetings are convened quarterly to quality assure EFAL and HL lesson plans and support material. Parallel sessions are held for each language, normally over two days for HL and one day for EFAL. Sessions are also attended by DBE for support supervision and technical advice. 8 to 10 RSP Custom Indicator Number of Curriculum / Subject Advisors attending Just in Time (JIT) teacher training sessions. Revised Target: 12 Curriculum / Subject advisors attend teacher training to observe and provide technical support to coaches, and to conduct sessions on School Based Assessment (SBAs). 8 or 9 28 RSP Custom Indicator -- Number of Curriculum Advisors receiving Tablets Revised Target: 12 RSP MEL Plan says: Curriculum Advisors will be given Tablets loaded with RSP LTSM for 2019 and 2020 13 provided to CAs and to the District Education officials. 29 Design specification 16 Curriculum Advisors • Participate in Reference Group Meetings • Attend teacher training sessions • Attend Coach training sessions • Receive tablet computers Actual number of District Officials (subject advisors) participated in Reference Group meetings Term 1 Term 2 Term 3 Term 4 2019 2020 2019 2020 2019 2020 2019 2020 Reference group meeting dates NA 9 Dec 2019 14-15 Mar 2019 26 Jun 2020 11 Jun 2019 1-2 Oct 2020 29/30 Aug 2019 20 Nov 2020 Monitoring Data No data 10 (63%) 10* (63%) 9 (56%) 10* (63%) 9 (56%) 8* (50%) 8 (50%) Reporting Data NA NA 10 NA 11 NA 8 NA Actual number of District Officials (subject advisors) participated in Teacher training sessions NA 9 (56%) NA No Training NA 9 (56%) NA 8 (50%) Actual number of District Officials (subject advisors) who attend coach training sessions 2 9 10 9 10 8 10 8 Actual number of District Officials (subject advisors) who received tablets 13 received in Aug of 2020 Data note * Data available for Meeting number 2, 3 and 4 in 2019 Data not available in 2019 for attendance at coach training sessions. Involvement in PLC was planned for 2020. Tablets were planned for distribution during Curricular Advisors training in 2020 2019 data: Small discrepancies between the monitoring data and the FPD indicator reporting table were found. This is the result of continued data cleaning that took place after the report deadline. 2020 data: No data for attendance of coach training sessions. 30 6.16 USAID Indicator Summary Table PROGRAM GOAL: To improve language and literacy content; knowledge and pedagogy of primary grade learners in AHL and EFAL I n d i c a t o r Data Source Baseline data FY 2020 QuarterlyStatus–FY2020 Annual Performance Achieved to Date (in %) Target Justification for Fiscal Year of this Annual Report Year Value Annual Cumulative Planned Target Annual Cumulative Actual Q1 (Oct￾Dec 2019) Q2 (Jan￾Mar 2020) Q3 (Apr – Jun 2020) Q4 (Jul-Sept 2020) Intermediate Result (IR): (Long term) Sub-IR: Standard Indicators Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance DDD 2019 58,253 56,576 N/A 58,253 56,576 TBD 56,576 Number of coaches (including head coaches) who receive professional development with USAID funding Project Records 2019 16 16 N/A 16 16 16 16 Number of educators who complete professional development activities with USG assistance Teacher training attendance registers 2019 1501* 1,530 N/A 1578 1530 N/A 1530 31 Number of education administrators and officials who complete professional development activities with USG Assistance (Principals/Deputy Principals, DHs) SMT training attendance registers 2019 103* 134 N/A N/A 134 N/A N/A Number of primary school classrooms that receive a complete set of essential reading instructional materials with USG assistance Teacher LTSM registers 2019 1501* 1,530 N/A 1,578 1,530 TBD 1,530 Number of primary or secondary textbooks and other teaching and learning materials (TLM) provided with USG assistance (LP=Lesson Plans) LTSM Distribution Forms 2019 105,000* 272,184 N/A 69,43 2 67,584 67,584 67,584 Number of public and private schools receiving USG assistance DBE Records 2019 298 255 N/A 263 255 253 253 Number of tablets distributed Asset Loan Forms 2019 129 334 N/A N/A 134 200 N/A 32 6.17 RSP Custom Indicators RSP CUSTOM INDICATORS OUTPUT INDICATORS DEFINITION Metric tools: Baseline Q1 Q2 Q3 Q4 TEACHER TRAINING 2019 (Oct￾Dec 2019) (Jan￾Mar 2020) (Apr – Jun 2020) (Jul-Sept 2020) Number of teacher training sessions held A total of 4 JIT teacher training sessions (2*2 day and 1*1 day training sessions) are scheduled annually to enable educators to cover the CAPS curriculum presented in the lesson plans. This is a cumulative indicator and deviations call for alternative measures to be implemented to ensure educators receive catch-up training and the relevant material JIT Training report/project reports 4 1 1 N/A 1 Number of participating teachers in training sessions JIT teacher training sessions will be held once per quarter and all educators from participating schools are expected to attend (number of expected teachers against actual attendance to be recorded) and catch-up training must be conducted for those who do not attend (number of teachers who receive catch up training to be recorded) JIT Teacher training attendance registers 1501* (avg) 1578 1530 N/A 1530 The number of schools where at least one teacher attended the grade 1 training AND at least 1 teacher attended the grade 2 training AND at least one teacher attended the grade 3 training The intervention targets 255 schools in 2020 and approximately 6 teachers per school, all of whom are expected to mandate teachers to attend training. This indicator will capture information on schools with at least 1 Grade-specific teacher attending all three training sessions, excluding FP DHs and Principals/Deputies who are encouraged to attend all training sessions. Only those who teach specific Grades will be counted on the relevant days JIT Teacher training attendance registers N/A 263 255 N/A 255 Number of educators receiving lesson plans and complete LTSM package (by language, Grade, school) LTSM packages for both EFAL and AHL are distributed during training and all teachers are expected to receive the relevant packages. This indicator will also capture LTSM collected for educators who did not attend training, as well as LTSM delivered to LTSM only schools post-training LTSM register of receipt of materials by each teacher 1501* 1578 1530 TBD 1530 COACHING INTERVENTION Number of coaches who complete training The coach training is defined as 3 days of training held quarterly and includes coach dry runs per term Coach training attendance registers 16 16 16 16 16 Number (%) of in-school support supervision visits provided by head coaches to coaches Head Coach observes each coach at least once a month, completes the Supervision Tool and develops an ISP for each coach. Head coaches should identify coach strengths and Head coach support supervision form N/A 14 14 N/A 14 33 RSP CUSTOM INDICATORS OUTPUT INDICATORS DEFINITION Metric tools: Baseline Q1 Q2 Q3 Q4 weaknesses and develop plans for further developmental support Number of teachers receiving at least one individual coaching session per teacher (teachers in the coaching intervention) A coaching session is defined as in-class lesson observation by coaches, including pre and post observation feedback using the standardized RSP tool. Observations will be aggregated per language, Grade Lesson Observation Form 668 810 810 TBD 810 Number of lesson observations conducted annually per teacher Each teacher is supposed to receive a minimum of 5 individual sessions over a year Coach support template TBD N/R N/R TBD TBD Average number of SBW per school? SBW is typically an afternoon session involving all FP educators, and informed by needs identified during individual coaching sessions. This platform is aimed at fostering a peer learning environment and for educators to share best practices Each school should have 15 SBWs per year SBW attendance registers TBD N/R N/R N/R TBD Number of individual teachers attend at least one SBW per quarter SBW is typically an afternoon session involving all FP educators, and informed by needs identified during individual coaching sessions. This platform is aimed at fostering a peer learning environment and for educators to share best practices SBW attendance registers TBD TBD TBD N/A TBD # (%) teachers meeting minimum support requirement as per RSP observation tool Assessment criteria to define competency levels of educators to structure support. Coaches to provide support based on need and this informs a differentiated support for educators. Minimum support requirements will be informed through identified capacity gaps in educators Numerator: number of educators who meet minimum support requirements Denominator: total number of educators in coaching schools Assessment criteria N/A N/A TBD TBD TBD # (%) teachers on track against curriculum tracking tool (sample) The lesson plan is used in line with a curriculum tracker to ensure teaching and learning occur at an acceptable pace. Catch-up plans must be in place where gaps have been identified. Coaches will collect curriculum coverage data in a sample of coaching schools. Numerator: Number of teachers (per Grade) on track using curriculum trackers Denominator: Total number of teachers sampled Curriculum trackers N/A N/A TBD TBD TBD SMT INTERVENTION 34 RSP CUSTOM INDICATORS OUTPUT INDICATORS DEFINITION Metric tools: Baseline Q1 Q2 Q3 Q4 Number of SMT completing at least one PoE SMT need to submit 4 PoE in order to successfully complete the course and earn the full amount of SACE points. Project Records N/A 40% 60% 80% N/A Number of PLC sessions conducted 5 PLCs including a wrap-up session are planned for this intervention. SMT from all participating schools are expected to attend. PLC attendance registers 3 1 1 N/A N/A SUBJECT ADVISORS Number of Subject Advisors attending Reference Group meetings Reference group meetings are convened quarterly to quality assure EFAL and AHL lesson plans and support material. Parallel sessions are held for each language, normally over two days for AHL and one day for EFAL. Sessions are also attended by DBE for support supervision and technical advice. Reference Group meeting attendance registers N/A 12 12 12 N/A Number of Subject Advisors attending JIT teacher training sessions Subject advisors attend teacher training to observe and provide technical support to coaches, and to conduct sessions on School Based Assessment (SBAs). Attendance registers N/A 12 12 12 N/A Number of Subject Advisors receiving Tablets SAs will be given Tablets loaded with RSP LTSM for 2019 and 2020 Asset Loan Forms N/A N/A 9 N/A N/A OUTCOME INDICATORS TEACHER TRAINING Indicator Definition (#)Percentage of teachers who attended the full training program Sessions are held each quarter, with Term 1 and Term 3 training over 2 days; and Term T2 and Term 4 over 1-day sessions. Teachers from all qualifying schools are expected to attend training sessions to complete the CAPS curriculum Teacher training templates (RedCap) N/A N/A N/A N/A TBD COACHING % of teachers receiving individual coaching support (lesson observations for both languages and/or attendance to SBWs) over the year Educators are expected to receive at least 5 individual coaching sessions annually. The differentiation of support will lead to the revision of these targets. They will be updated when the teacher profiling process has been concluded. Coaching support is both individual lesson observations and attendance to SBWs Coaching support template (RedCap) N/A N/A N/A N/A TBD % of teachers attending SBWs A target of SBWs conducted per school was set at 15 annually, however this figure still needs to be revised down based on the realistic number that can be achieved in light of the remaining days for coaching support. All FP teachers are expected to attend SBWs held in their schools. Coaching support template (RedCap) N/A N/A TBD TBD TBD 35 RSP CUSTOM INDICATORS OUTPUT INDICATORS DEFINITION Metric tools: Baseline Q1 Q2 Q3 Q4 % teacher improvement noted in intervention schools This indicator will focus on the teachers assessed to inform the differentiated support to be received by educators. Support plans will be developed and progression monitored over time. The criteria for support will be determined and linked to expected level of competency, and support plans will be developed per teacher and monitored Survey N/A TBD TBD TBD TBD Proportion of teachers reporting improvement in job satisfaction Job satisfaction in this case will be linked with the motivation to teach based on the support provided through the training to increase knowledge in content and methodologies. Factors influencing motivation such as school attendance, time on task will be measured using self-reports Teacher Survey N/A N/A N/A N/A TBD SMT INTERVENTION Proportion of SMT program participants reporting improved confidence, capacity and job satisfaction SMT will be given self-administered questionnaires to rate themselves on a Likert scale against the set criteria SMT Survey N/A N/A N/A TBD N/A # (%) SMT completing professional development activities (including submission of PoEs) Each of the 4 SMT program modules are linked to a PoE which must be completed and submitted for assessment. SMT are supposed to submit all 4 to earn the maximum amount of SACE points. These PoEs will form part of the package to be submitted to SACE for CPTD points. Project record of SMT completion of activities N/A N/A N/A >80% N/A ANNEX 7: EVIDENCE OF TEACHER UPTAKE AND LTSM IN CLASSROOMS 1 Table of Contents Table 1: Strategies utilized by teachers to assist learners to decode words ....................................2 Table 2: Strategies to promote comprehension of text for learners ...................................................3 Table 3: Opportunities to read by intervention group ...........................................................................4 Table 4: Percentage of where the correction of handwriting implements was observed................4 Table 5: Construction of patterns or letters by learners, across all intervention groups .................5 Table 6: Construction of own words by intervention type ....................................................................6 Table 7: Construction of sentences by intervention type .....................................................................7 Table 8: Comparison between Grade 1 EFAL and HL words written ................................................8 Table 9: Comparison between Grade 3 EFAL and HL sentence writing...........................................9 Table 10: Cursive writing of Grade 3 learners between intervention groups..................................10 Table 11: Vocabulary and spelling development of Grade 1 and Grade 3 classrooms combined10 Table 12: Tasks given to learners and learner management across intervention groupss ..........11 Figure 1 Availability of Teacher Management and Storage File.................................................12 Figure 2 Availability of Theme Vocabulary Flashcards ..............................................................12 Figure 3 Availability of Posters ..................................................................................................12 Figure 4 Visibility of posters ......................................................................................................12 Figure 5 Visibility of EFAL flashcards ........................................................................................12 Figure 6 Display of EFAL Sight Word Flashcards......................................................................12 Figure 7 Visibility of HL flashcards ............................................................................................12 Figure 8 Availability of HL Graded Readers ..............................................................................12 Figure 9 Display of Alphabet Frieze ..........................................................................................12 Figure 10 Availability of HL Big Books.......................................................................................12 Figure 11 Availability of EFAL Big Books ..................................................................................12 Figure 12 Availability of EFAL graded readers ..........................................................................12 Figure 13 Daily use of LTSM reported by teachers ...................................................................12 Figure 14 Use of EFAL Lesson Plans .......................................................................................12 Figure 15 Use of HL Lesson Plans............................................................................................12 Figure 16 Use of EFAL graded readers.....................................................................................12 Figure 17 Use of handwriting poster..........................................................................................12 Figure 18 Use of EFAL Big Books.............................................................................................12 Figure 19 Use of HL Big Books .................................................................................................12 Figure 20 Use of EFAL theme Vocabulary Flashcards ..............................................................12 Figure 21 Use of HL Theme Vocabulary Flashcards .................................................................12 Figure 22 Use of HL Sight Words Flashcards ...........................................................................12 Figure 23 Use of EFAL Sight Words Flashcards .......................................................................12 Figure 25 Use of Posters for Listening and Speaking................................................................12 Figure 24 Use of Curriculum Tracker/Teacher Monitoring and Support Tool .............................12 2 1. Classroom Practice Figures in green represent a positive change (either increase or decrease, depending on the item assessed represented by the arrow) from 2019 to 2020, while figures in red represent a negative change (either an increase or decrease represented by an arrow) from 2019 to 2020. Table 4: Strategies utilized by teachers to assist learners to decode words20 Strategies used by teachers: Coaching SMT LTSM only 2019 2020 2019 2020 2019 2020 Sounding out a word 74% (63 of 85)  85% (35 of 41) 88% (68 of 77) 88% (53 of 60) 80% (64 of 80)  86% (42 of 49) Guessing a word 21% (18 of 85)  24% (10 of 41) 49% (38 of 77)  33% (20 of 60) 30% (24 of 80)  14% (7 of 49) Using similar words that are already known by learners to work out the word 40% (34 of 85)  44% (18 of 41) 70% (54 of 77)  35% (21 of 60) 48% (38 of 80)  57% (28 of 49) Skipping a word and returning to it and using in context 12% (10 of 85)  7% (3 of 41) 23% (18 of 77)  13% (8 of 60) 14% (11 of 80)  10% (5 of 49) 20 2019 data was reanalyzed for consistency with 2020 analysis, and therefore some figures differ to what was presented in the Formative Report of 2019. The tools were revised and re-piloted as part of the revision process from the formative to the summative evaluation. The 2019 data was reanalyzed. ‘Learners are given an opportunity to read aloud’; was based on the 2019 items of ‘Whole class reads aloud together without teacher’, ‘Learners read aloud together in groups or pairs’ and ‘Learners read individually aloud to the class’. ‘Teacher does shared reading with the learner’ was based on the combined 2019 data of “Learners read aloud together in groups or pairs’ and ‘Learners read individually aloud to the class’. 3 Table 5: Strategies to promote comprehension of text for learners Item observed Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 The teacher asks the learners questions after reading 64% (54 of 85)  89% (25 of 28) 73% (56 of 77)  86% (30 of 35) 69% (55 of 80)  83% (24 of 29) Learners Encouraged to Retell, Act, Summarize Story 35% (30 of 85)  50% (20 of 40) 57% (44 of 77)  36% (16 of 44) 44% (35 of 80)  69% (36 of 52) Learners Encouraged to Discuss Illustrations 78% (66 of 85)  93% (37 of 40) 79% (61 of 77)  82% (36 of 44) 61% (49 of 80)  90% (36 of 52) Teachers Use Illustrations to Aid Understanding of the Text 69% (59 of 85)  80% (32 of 40) 81% (62 of 77)  77% (34 of 44) 71% (57 of 80)  77% (40 of 52) Learners Identify Main Theme of the Text 52% (44 of 85)  65% (26 of 40) 70% (54 of 77)  61% (27 of 44) 40% (32 of 80)  67% (35 of 52) Learners Answer Open Ended Questions 51% (43 of 85)  85% (34 of 40) 73% (56 of 77)  55% (24 of 44) 65% (52 of 80)  64% (33 of 52) Learners Answer Predictive/Inferential Questions 34% (29 of 85)  53% (21 of 40) 62% (48 of 77)  16% (7 of 44) 44% (35 of 80)  46% (24 of 52) Learners Answer Literal, Evaluative, Appreciative and Critical Questions 42% (36 of 85)  45% (18 of 30) 69% (53 of 77)  30% (13 of 44) 48% (38 of 80)  56% (29 of 52) 4 Table 6: Opportunities to read by intervention group2122 Item observed Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 Learners are given an opportunity to read aloud 60% (155 of 255) 63% (41 of 65) 65% (149 of 231) 77% (60 of 78) 58% (140 of 240)  70% (49 of 71) Learners are split into groups for reading 66% (56 of 85) 35% (23 of 65) 74% (57 of 77)  40% (31 of 78) 79% (63 of 80)  24% (17 of 79) The teacher does shared reading with the learners2324 68% (116 of 170) 48% (31 of 65) 74% (114 of 154)  49% (38 of 78) 63% (100 of 160)  47% (33 of 71) Table 7: Percentage of where the correction of handwriting implements was observed Item observed Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 Teacher checks that learners are holding pens and positioning writing material 24% (21 of 85) 25  26% (13 of 50) 16% (12 of 77)  25% (14 of 57) 17% (13 of 80)  46% (26 of 57) Teacher corrects learners when pens/writing materials are not held/positioned correctly 46% (39 of 85)  60% (30 of 50) 25% (19 of 77)  42% (24 of 57) 16% (13 of 80)  70% (40 of 57) 21 2019 data reanalysed for consistency, therefore some figures changed. In some cases 2019 data represents multiple combined items, as the observation tool had changed 22 The analysis in 2019 combined different items observed to provide this percentage, therefore although the percentages are an average, however this was still based on 242 total classrooms observed (85 coaching, 77 SMT, and 80 LTSM). 23 Combined 2019 data from “Whole class reading aloud with teacher”, and “individual guided reading practice.” 24 All 2019 values for this items rated “Sometimes”, “mostly”, and “always” were combined. 25 Data for 2019 was reanalysed for comparison to 2020 data analysis 5 Table 8: Construction of patterns or letters by learners, across all intervention groups Writing item observed Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 Most/all learners drawing patterns in preparation for cursive writing 24% (14 of 58)  8% (4 of 50) 7% (4 of 53)  7% (4 of 57) 14% (6 of 42)  5% (3 of 57) Own drawing 26% (15 of 58)  18% (9 of 50) 9% (5 of 53)  11% (6 of 57) 29% (12 of 42)  4% (2 of 57) Writing own vowels/ letters/syllables 21% (12 of 58)  30% (15 of 50) 38% (20 of 53)  33% (19 of 57) 29% (12 of 42)  46% (26 of 57) 6 Table 9: Construction of own words by intervention type Construction of own words by learners Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 Teacher expects most/all learners to write words 60% (51 of 85) 34% (17 of 50) 36% (28 of 77) 46% (26 of 57) 44% (35 of 80)  61% (35 of 57) 1-2 words observed by fieldworker during writing activity 43% (21 of 49)  24% (4 of 17) 32% (12 of 37) 7% (2 of 26) 29% (11 of 38)  37% (13 of 35) 3-10 words observed by fieldworker during writing activity 43% (21 of 49)  76% (13 of 17) 49% (18 of 37)  58% (15 of 26) 55% (21 of 38)  57% (20 of 35) 10+ words observed by fieldworker during writing activity 14% (7 of 49)  0% 19% (7 of 37)  35% (9 of 26) 16% (6 of 38)  6% (2 of 35) Total 100% N=49 100% N=17 100% N=37 100% N=26 100% N=38 100% N=35 Data note: The question in the tool was amended from 2019 to 2020, and the data was reanalysed for closer comparison. 7 Table 10: Construction of sentences by intervention type Construction of sentences by learner Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 Teachers expect most/all learners to write sentences 44% (21 of 48) 54% (27 of 50) 48% (19 of 40) 47% (26 of 57) 54% (20 of 37) 61% (35 of 57) 1-2 sentences observed by fieldworker during writing activity 36% (17 of 48)  33% (9 of 27) 38% (15 of 40)  35% (9 of 26) 30% (11 of 37)  46% (16 of 35) 3 – 5 sentences observed by fieldworker during writing activity 56% (27 of 48)  52% (14 of 27) 38% (15 of 40)  54% (14 of 26) 40% (15 of 37)  43% (15 of 35) More than 5 sentences observed by fieldworker during writing activity 8% (4 of 48)  15% (4 of 27) 25% (10 of 40)  11% (3 of 26) 30% (11 of 37)  11% (4 of 35) Total 100% N=48 100% N=27 100%26 N=40 100% N=26 100% N=26 100% N=35 Data note: Source of data is from classroom observations. The sample size (n) varies depending on what lesson and task was actually being observed. The data above the thick black line refers to teacher expectation, whereas data below the black line refers to fieldworker ratings of quantity. 26 Note: rounding of percentages of 37.5% (15 out of 40) to 38% increases total percentage to 101% 8 Table 11: Comparison between Grade 1 EFAL and HL words written Grade 1 comparison between EFAL and HL Coaching SMT LTSM EFAL HL EFAL HL EFAL HL Teacher expects all/most learners to write words 29% (2 of 7) 15% (2 of 13) 11% (1 of 9) 42% (5 of 12) 54% (7 of 13) 77% (10 of 13) Teacher expects learners to write 1-2 words 50% (1 of 2) 100% (2 of 2) 0% 0% 86% (6 of 7) 50% (5 of 10) Teacher expects learners to write 3 – 10 words 50% (1 of 2) 0% 100% (1) 80% (4 of 5) 14% (1 of 7) 50% (5 of 10) Teacher expects learners to write more than 10 words 0% 0% 0% 20% (1 of 5) 0% 0% Total 100% N=2 100% N=2 100% N=1 100% N=5 100% N=7 100% N=10 Data note: N = number of classroom observations per intervention and language (EFAL/HL) 9 Table 12: Comparison between Grade 3 EFAL and HL sentence writing Grade 3 comparison Coaching SMT LTSM EFAL HL EFAL HL EFAL HL Teacher expects all/most learners to write words 60% (9 of 15) 87% (13 of 15) 59% (10 of 17) 63% (12 of 19) 73% (11 of 15) 63% (10 of 16) 1-2 sentences 0% 31% (4 of 13) 20% (2 of 10) 33% (4 of 12) 18% (2 of 11) 0% 3 – 5 sentences 78% (7 of 9) 54% (7 of 13) 60% (6 of 10) 59% (7 of 12) 55% (6 of 11) 90% (9 of 10) More than 5 sentences 22% (2 of 9) 15% (2 of 13) 20% (2 of 10) 8% (1 of 12) 27% (3 of 11) 10% (1 of 10) Total 100% N=9 100% N=13 100% N=10 100% N=12 100% N=11 100% N=10 Data note: N = number of classroom observations per intervention and language (EFAL/HL) EFAL N= 47 HL N= 50 10 Table 13: Cursive writing of Grade 3 learners between intervention groups Coaching HL SMT LTSM EFAL HL EFAL HL EFAL HL All learners involved in writing any cursive text during the lesson 2019 50% (10 of 20) 33% (7 of 21) 70% (14 of 20) 37% (7 out of 19) 55% (11 of 20) 33% (7 out of 21) 41% (17 of 41) 56% (21 of 39) 44% (18 of 41) All learners involved in writing any cursive text during the lesson 2020 27% (4 of 15) 20% (3 of 15) 32% (6 of 19) 0% (0 of 17) 38% (6 of 16) 40% (6 out of 15) 23% (7 of 30)  16% (6 of 36) 39% (12 of 31) Data note: n= number of observations by intervention and HL/EFAL for Grade 3 learners only The 2019 was analyzed differently across items. Table 14: Vocabulary and spelling development of Grade 1 and Grade 3 classrooms combined Item observed Coaching SMT LTSM 2019 2020 2019 2020 2019 2020 There is evidence of vocabulary development during the observation period 80% (68 of 85)  71% (46 of 65) 82% (63 of 77)  78% (61 of 78) 75% (60 of 80)  72% (51 of 71) There is evidence of development/testing of spelling during the lesson observation 80% (68 of 85)  59% (38 of 65) 82% (63 of 77)  60% (47 of 78) 58% (46 of 80)  65% (46 of 71) The difference between the Grade 1 and Grade 3 learner development was minimal. 11 Table 15: Tasks given to learners and learner management across intervention groups Item observed Coaching SMT LTSM 2020 2020 2020 Independent tasks given to learner N = 214 65% (42 of 65) 63% (49 of 78) 63% (45 of 71) Teacher monitors all learners to check that they are doing what they are meant to be N = 135 Categories were combined for “mostly” and “always” 67% (28 of 42) 53% (26 of 49) 75% (33 of 44) Teacher supervises less capable learners N = 135 Categories were combined for “mostly” and “always” 55% (23 of 42) 33% (16 of 49) 66% (29 of 44) Learners are given additional tasks once they have completed their work N = 134 21% (9 of 42) 6% (3 of 48) 55% (24 of 44) Extent to which learners are engaged with their independent written work N = 136 Categories were combined for “most learners” and “all learners” 93% (39 of 42) 90% (44 of 49) 98% (44 of 45) Extent to which learners are engaged with independent reading work N = 134 Categories were combined for “most learners” and “all learners” 52% (22 of 42) 54% (26 of 48) 73% (32 of 44) Data notes: n= number of classrooms where observations occurred. Items were observed multiple times by fieldworkers during the actual lesson (i.e., the teacher may have used these strategies more than once during the lesson) therefore the percentages are reported for each time the strategy was used per intervention group. 12 2. Classroom Management 3. Availability of LTSM Figure 3 Availability of Theme Vocabulary Flashcards Figure 4 Availability of Posters Figure 2 Availability of Teacher Management and Storage File 13 Figure 5 Visibility of posters Figure 6 Visibility of EFAL flashcards 14 Figure 8 Visibility of HL flashcards Figure 7 Display of EFAL Sight Word Flashcards 15 Figure 10 Display of Alphabet Frieze Figure 9 Availability of HL Graded Readers 16 Figure 11 Availability of HL Big Books Figure 12 Availability of EFAL Big Books 17 Figure 13 Availability of EFAL graded readers 18 4. Usefulness of LTSM (as self-reported by teachers) Figure 14 Daily use of LTSM reported by teachers 19 Figure 16 Use of HL Lesson Plans Figure 15 Use of EFAL Lesson Plans 20 Figure 18 Use of handwriting poster Figure 17 Use of EFAL graded readers 21 Figure 20 Use of HL Big Books Figure 19 Use of EFAL Big Books 22 Figure 22 Use of HL Theme Vocabulary Flashcards Figure 21 Use of EFAL theme Vocabulary Flashcards 23 Figure 24 Use of EFAL Sight Words Flashcards Figure 23 Use of HL Sight Words Flashcards 24 Figure 26 Use of Curriculum Tracker/Teacher Monitoring and Support Tool Figure 25 Use of Posters for Listening and Speaking ANNEX 8: COACHING CASE STUDIES READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |1 EXCELLENT COACH CASE STUDY (COACH 1) 1. PROFILE Overall rating: Planning and Preparation: Pre-observation discussion: Classroom observation: Feedback/Coaching practice: School-Based workshops: Length of time in RSP as coach: 23 months by 2020 Coach training: All RSP Training “Building trust relationships involves being approachable. You need to be that person that someone can walk up to and talk to without thinking twice… I find that if you are empathetic and don’t give that judging attitude, then it’s easier to build that trust.” – Coach 1 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |2 2. COACH SHADOWING SUMMARY The coach was shadowed from October 14 to 15, 2020. Four classes were observed over the two days, at two different schools. On day one, three teachers from Grades 1-3 were observed. On day two the coach observed a Grade 1 HL teacher because the rest of the teachers were busy with term assessments, and were unavailable. The Foundation Phase educators all were available for the School Based Workshop (SBW) which occurred after school on the second day. According to the evaluator, the coach has shown tremendous growth over the last two years. They have become competent in RSP methodologies, and the coach has built the most active WhatsApp, virtual Professional Learning Community (PLC) with their teachers. The teachers were observed as being respectful and excited to work with the coach and therefore RSP. As a result of the coach’s passion and skills, the working relationship between the coach and teacher was strong enough to sustain the teachers during level five COVID-19 lockdown, which allowed the coach to provide psychosocial support. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |3 3. COACHING STANDARDS FIDELITY TO COACHING AND RSP PROGRAM The evaluator’s assessment is that the coach is diligent, professional and their overall awareness of the RSP methodologies is evident in their communication style. Accordingly, the coach was open to working with teachers on finding solutions that can influence a more structured and better way of working within the school environment. For example, the coach suggested the SMT consider changing the rotation system for the learners at one school. The coach was reflecting teachers’ concern that the learners were attending only one day a week, and thus, not retaining what they had learnt in the previous week. The coach is enthusiastic about their role, this is demonstrated by their approach to teacher’s visits and their feedback processes. The coach not only gives verbal communication, but also demonstrates key methodologies in their feedback session. This approach allowed the teachers to ask more questions based on that particular demonstration. QUALITY OF COACHING Planning preparation in 2020: The coach prepares in the same way that they did in 2019. The coach uses the classroom observation tool to prepare, they communicate with the school leadership and the teachers to confirm or track progress since the last visit. The coach submitted their October plan to the head coach, but did not receive feedback. The coach assumes that the plan was accepted since there were no objections from the head coach. The coach confirmed and re-confirmed their visit before they went to the schools. Pre-observation discussion in 2020: In all four observations, the coach communicates with the teachers in the same way that they did in 2019. The coach discusses the observation, confirms which lesson the teacher will teaching and if both learners and teachers are comfortable and ready. In two of the observations, they shared the lesson observation tool with the teachers, while the discussions were more informal for the rest of the “The coach was excellent, they were responsive, gave one-on-one feedback and demonstrates examples of good practice. The coach was detail-orientated and they explored how COVID-19 was affecting the teachers. They were good at communicating and relationship building. The teachers opened up to discuss their challenges with the coach, and the coach made sure to hear the teachers’ input before providing feedback.” – Associate Evaluator shadowing Coach 1 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |4 observations. Although they were informal, the teachers and the coach seemed to know what was about to happen, and the teachers were ready. Classroom Observation in 2020 On day one, Grade 1 EFAL was observed and the lesson was on week two. The theme of the week was “friends”. The teacher asked the question of the day, and used a puppet to illustrate that action. After the lesson, the coach gave advice on techniques for introducing the sounding of phonics. The coach suggest that for sight words, the teacher should use code switching to enable the further understanding of words. For Grade 2 EFAL, the observation was a week 5 lesson. The teacher began her lesson with the revision of phonics. After the lesson, the coach provided verbal examples and conducted role playing examples for teachers to model with learners. In the Grade 3 EFAL observation, a week six lesson was observed. The lesson was an introduction to phonics and the building of consonants. The coach illustrated how the reading exercise can be improved. After reading, the teacher should ask questions about the actual story that they have read. On day two, Grade 1 HL was a lesson from week two. The coach checked the learner exercise books against the ATPs, and to confirm the current week. The coach emphasized the need for the teacher to show learners how to write on the chalk board. The coach identified action steps to solutions and shared them with the teachers. In 2020, the coach is still a thorough observer and pays attention to detail. The coach is much more comfortable with RSP methodologies and also responsive to the new COVID-19 environment by constantly reminding teachers and learners of safety protocols. Overall, the grades 2 and 3 teachers were adhering to the lesson plan pacing per the revised curriculum, and the grade 1 teachers were lagging. School Based Workshops (2020) A SBW was conducted on the second day of the observation at the second school visited. Five Grades 1 to 3 teachers were present. The coach covered the following areas: • The school rotation system and its effect on both the learners and educators; • Key components when conducting the listening and speaking part of the lesson; • Grade 1 teachers should use lines on the chalk board when demonstrating writing. The coach demonstrated each point by role playing the scenario being discussed. The session was opened for questions and feedback from the teachers. The teachers requested the school principal and HOD to look closely at other possible options to the rotation system currently being implemented by the school. The coach felt that the current scheduling was detrimental to the progress of the learners, especially the foundation phase which is very important in setting the right start for the child. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |5 Modelling behavior and actions in the classroom situation were also discussed with the teachers. The coach emphasized that learners, especially in Grade 1, learn by copying. Therefore, it is important to alert learners to what must be written and where. The evaluator did not observe a SBW tool being completed. The coach’s approach was not only to explain their comments but to demonstrate. They emphasized the listening and speaking component of the lesson. The engagement was good. Teachers spoke about the challenges they faced on rotational attendance, and the time it takes to cover a specific theme. They raised the concern that learners forget what they learned in the previous week. 4. COACHING PRACTICE According to the evaluator, in 2020, the coach remains a respected mentor with strong interpersonal skills, both in person and online. The coach was observed to be adaptive, and was able to foster a learning community amongst the teachers. The schools welcomed the coach warmly. The coach was responsive and maintained the good working relationship they have built with the school leadership over the two years. The coach continues to believe in the importance of communication because it helped them build trust with the teachers over the years, especially during the COVID-19 lockdown. The coach ensured that their schools had received all relevant RSP documents and material at the beginning of the term. They acknowledged the challenges associated with the rotation system and the associated teacher anxieties about curriculum coverage. “The SBW is the after schools session. I identify needs based on the interaction with the teachers in some cases based on the questions and request from the teachers...” – Coach 1 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |6 The coach was observed to have become a confidant, and much more than just a coach, to the teachers. Finally, the coach pays attention to detail and cares about the wellbeing of the entire school community from the learners to their leaders e.g. principal, school management, etc. 5. TEACHER PERCEPTION OF COACH 1: Teachers were surveyed on their experience of and perception of their coach, this data is based on the responses from teachers who were at schools assigned to Coach 1. Eighteen teachers surveyed were from SMT and Coaching schools supported by this coach. All 18 teachers indicated that they received supporting visits from this coach. “Let me once remind you on the following • Your daily preparations • Time table to be CAPS compliant • Reading corners to be visible • Carpet for teaching and reading • Group leaders to give learners who are not reading occupational tasks • Discipline • Group guided reading core methodologies-pre reading-during reading - Post reading • Don't forget flashcards of the new words to be taught before reading commence • Do what u supposed to do according to the lesson plans. Formal assessment to be done as supposed to be done…Shared reading please do big book for Setswana” – Coach 1 via WhatsApp to teachers 2019/09/02 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |7 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |8 6. PROGRAM CONTEXT: COACH RESPONSE TO COVID-19 CONTEXT The COVID-19 pandemic and associated school closures meant that normal practice of supporting the schools could not be pursued. However, the coach indicated that remote support processes were implemented. This included using the WhatsApp groups with the teachers to communicate relevant information. The coach shared videos and documents. The coach also indicated that this platform was extended to psychosocial support, to find out if the teachers were well and coping with the current situation. The coach created the most active WhatsApp group and PLC of all the coaches over the two years. In 2020 alone, the group posted over 2000 times and the coach posted over 500 times of the total annual amount. 27 WHATSAPP ANALYSIS The WhatsApp group was created on August 26 2019 and closed on December 9 2020. The coach had started an earlier WhatsApp group in 2019, but changed to this group in August 2019. During the August 2019 the coach continued to add more teachers, sharing information on workshops and LTSM material distribution. The coach consistently followed up on how teachers were using the LTSM. In addition to the 51 teachers being coached, Coach 1 included their LTSM teachers in their WhatsApp group. This meant that these teachers were also being supported, but this meant fidelity to the TOC was compromised. During evaluation fieldwork, a teacher survey was conducted in 59 sampled schools. Of this, 17 teachers reported that they were supported by Coach 1 and 16 found the WhatsApp group useful. 27 Data was obtained from Coach WhatsApp groups. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |9 Typically the coach shared information about trainings, venues, visiting schedules. • The coach created a responsive and positive group, consistently reaffirming the teachers’ abilities and reassuring them that the coach is available to them for questions and support. • In September 2019, the group receives news about the untimely passing of one of the foundation phase teachers involved in the RSP. The group is saddened by the news and the coach suggested they attend the memorial as a unit, wearing similar colors. The coach mobilizes the group and asked for a financial contribution. As a group they decide to sing hymns at the memorial, arranging printing of lyrics for all involved that were not familiar with the particular hymn. After the memorial, the teachers thanked each other for the support provided and declared their unity as more of a family, rather than just a work-group. This demonstrates the safe space created by the coach. • To further solidify a family-type relationship between the group, the coach continued to share information on their upcoming visits, workshops and LTSM materials, reminding the teachers to rest during the festive season and to gear up for a productive 2020. “Colleagues Gudnite. I know that this coming week you busy with Assessment, Recording and Reporting I won't disturb you too much. Try to cover up…if I happen to come to your school I will only check the informal activities and curriculum coverage. IF I HAPPEN TO COME!!! Nanite. – Coach 1 via WhatsApp to teachers 2019/09/02 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |10 • The coach answered most questions the teachers have on the platform timeously and when not possible to do so, the coach explained why. This approach seemed to be appreciated, as the group had a high response rate. • At the beginning of the COVID-19 lockdown period there was very little RSP talk on the group. This period marked a more lighthearted time on the group, where bible verses, prayers, COVID-19 information and falsehoods regarding the pandemic were shared. The coach then started suggesting an exercise on lesson plans, to retain such information, teachers shared their lockdown experiences and discussed the impact of lockdown on teaching and learning. The coach continued to check in regularly and initiated weekly discussion topics on creating positive learning environments. • Between June and July 2020, the coach started sharing information on the revised ATPs, following up on LTSM material and preparing for baseline assessments. The continued follow up discussion allowed for refresher discussions between the teachers. Even when few teachers responded, the coach was able to use their responses and questions as a discussion point for the rest of the group, effectively involving all. • Besides RSP discussions, social interactions were also encouraged by the coach and many jokes and words of encouragement were shared. The evaluator noted that the group shows high levels of psycho-social support to each other, allowing everyone to freely express themselves and share experiences outside of RSP. The group became less active when an M&E staff member joined. Throughout the chats, the coach acted as an informant but not the dominant person in the group. Responsive teachers took the lead on many occasions. FACTORS THAT AFFECT COACHING PRACTICES The lockdown restrictions were a challenge to the coach, they were not able to visit the teachers as they would have liked. Migrating to virtual support was also a challenge to the teachers because they “Morning Colleagues...I have an idea of organizing reading competitions, and also English Drama competitions. Just a thought. You choose any book and dramatize it and or choose any book and learners read and do some recounting. Dreaming alone…let's think about it...we can do it within the school...invite parents and have an open day session of reading both Setswana and English.. 🙏🙏🙏🤦♂💃💃Enjoy your holidays ditsala...Rest lo tle lo lapologetswe. – Coach 1 via WhatsApp to teachers 2019/09/20 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |11 could not download some the information the coach shared in the group. Teachers had to change their classroom layout e.g. removing the reading carpets, posters on the wall was also a challenge. In comparison to 2019, the coach did not raise challenges related to distributing material and school visits. Although not a problem, the coach suggests that the role of the Head Coach can go beyond administration. The coach would suggest more feedback and reflection sessions with the Head Coach on the coach’s performance. HEAD COACH AND OTHER SUPPORT (2020) The coach expressed their thoughts about the changes regarding visits from the Head Coach. They confirmed that the visits are not communicated. Mostly, the coach will see the Head Coach at the school that they are visiting for the day without prior warning or communication. The coach stated that there is no formal process around the visit. The Head Coach observes and leaves the school immediately after that process, without providing feedback. The coach feels adequately supported by school SMT members and, at times, encourages teachers to speak to their Heads of Department (HODs) about resources and fund raising. Based on their experience with the allocated schools, the coach believes that school leadership wants RSP to succeed. “The Head Coach offers advice and support with any necessary information regarding the scheduled school visits, [as well as] administrative support. The Head Coach also visits my schools, this has changed from last year. ” - Coach 1 “The SMT in all my schools are very supportive and want the project to succeed so that our learners can have better options in life by improving the standard of education.” - Coach 1 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |12 7. COACH 1 RUBRIC: 28 2019 column – observations conduced in Term 1 in January 2019, 2020 column observations conduced in Term 3 in October 2020 29 The coach reported that before October 2020, they had not received a visit from the head coach. The FPD monitoring data says that the coach was visited on 6 November 2020. Coach 1 Criteria from Coaches’ Job Descriptions 2019 2020 Criteria from Coaches’ Job Descriptions 2019 2020 Qualified as teacher Yes Yes Monitors progress against lesson plans for the whole lesson Inadequate Fully Teaching experience Extensive Extensive Looks at learners’ books Inadequate Partial Foundation Phase experience: 21 years 22 years Monitors teachers’ assessment processes Partial Partial Ratio coach to schools 1: 7 8 Provides feedback to teachers Fully Fully Ratio coach to teachers 1: 50 51 Teachers reflect on their practice Partial Fully Setswana language proficiency Fluent Fluent Conducts needs-based workshops in the afternoons 0 of 4 1 of 2 Develop monthly visit schedules Inadequate Fully Sets up PLC Not Reviewed Fully Monthly Plan submitted to Head Coach Inadequate Fully Facilitates PLC Not Reviewed Fully Head Coach Approval Inadequate Fully Completes in-class coaching tools Partial Fully Coach tells teacher about planned visit Fully Fully Completes SBW / PLC tools Not Reviewed Inadequate Coach sets up direct communication channel with teacher (not through school) Fully Fully Completes training tools Not observed Not observed Coach re-confirms visit Fully Fully Participates in training sessions 16 days Not observed, monitoring data indicates participation in all sessions Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Partial Fully Head coach visits the coach at school28 None Once29 Follows steps of Class Observation Partial Fully Head coach provides support to coach Inadequate Inadequate READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |13 AVERAGE COACH CASE STUDY (COACH 5) 1. PROFILE Overall rating: Planning and Preparation: Pre-observation discussion: Classroom observation: Feedback/Coaching practice: School-Based workshops: Length of time in RSP as coach: 23 months by 2020 Coach training: All RSP Training 2. OBSERVATION SUMMARY Coach 5 was shadowed over two days from October, 19 – 20, 2020. A full observation was conducted on day one for Grade 1 HL and Grade 3 EFAL. On day two, a full observation was not conducted due to assessments taking place in the classroom. The coach observed the assessment session, rather than a teaching session. In 2020, like 2019, the coach continued to state that their role was to empower teachers, develop, support and assist teachers to be conversant in RSP across all schools. The coach “My role was to empower teachers and develop them so that they know what they must do, make them conversant with RSP, and know what to do when they are at their respective schools...” – Coach 5 “The coach was excellent, they provided good detail in forms while observing the class. The coach advocates for inclusive education, and wants to influence the process and include learners who are struggling. The coach provided good feedback.” – Evaluator shadowing Coach 5 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |14 further understood their role as needing to adapt and support teachers in a COVID-19 school environment. They included aspects of psychosocial support such as; to check in on the teachers. According to the evaluator, the coach integrated their experience in special needs and remedial education to assist teachers and learners. A comparison of the 2019 and 2020 observations illustrate a shift from an administrative style, to responsive coaching in order to meet the individual needs of teachers. 3. COACHING STANDARDS FIDELITY TO COACHING AND RSP PROGRAM The coach was allocated 6 schools, and 46 teachers in 2020 according to the monitoring data. This decreased from 7 schools, and 50 teachers in 2019. Over the two years, the coach’s administrative skills have somewhat improved, although there are still areas for improvement. The coach prefers an informal method of communication and support (for example, using WhatsApp). The coach was flexible in their approach to conducting classroom observations, they integrated teaching practices, and other intervention strategies from their qualifications and experience. Although this flexible and informal approach was well-received and contributed to positive relationship building with teachers, this could compromise fidelity of implementation of the program. In comparison to 2019, the coach monitors teacher progress and fidelity to the lesson plans in 2020, they have improved in their assessment of learner’s books and fully complete in￾class coaching tools. The coach was not observed providing teacher training. QUALITY OF COACHING Planning preparation in 2020: Coach 5 has continued in their planning and preparation duties, and providing support, however this moved to online platforms. Due to COVID-19, this had to move to a virtual platform – WhatsApp. The evaluator observed that the coach was more confident in preparing for teacher visits, and continued to share their schedule with the Head Coach. For the preparation stage, the evaluator observed that communication between coach and head coach has improved since 2019. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |15 Pre-observation discussion in 2020: For the observations on day one (two Grade 1 HL teachers and one Grade 3 EFAL teacher), the coach had a two minute discussion with each teacher before the lesson began. The coach checked in on teacher wellbeing, reflected back to previous discussion, and checked on teacher preparation. Similar to 2019, the coach does not go through the lesson plan, or have a formal discussion, but the evaluator observed that the teachers and the coach have built a comfortable working relationship, where the engagement is sufficient. For HL Grade 1 observation on day two, the coach observed the assessment of learners. Classroom Observation in 2020 On day 1 the coach observed three teachers. The evaluator observed the coach reviewing learner books in one of the four classes observed. After all three observations, the coach provided detailed group feedback. The coach assisted the teachers on how to work with sight words, adjusting the size of flash cards and assisting with shared reading. The coach interacted with the teachers by asking them questions and demonstrating best practice. The coach was reportedly a proponent of inclusive education by emphasizing repetition of activities, for example; encouraging the teacher to facilitate learners speaking and reading out loud. The focus of the observation on day two was on assessment. The coach asked the teachers if they understood how to run the assessment task and if they had any question. The assessment task was to break up the key components of words. The coach observed one teacher, a Grade 1 HL teacher administering the assessment. The teacher described the learners as the “slow” group. In 2020, the coach’s style was more interactive and engaging when compared to 2019. The teachers raised learning challenges, and the coach was able to draw on previous experience with special needs education to advice teachers suitably. In 2020, it was observed that the coach was more comfortable with the RSP methodologies, materials and outcomes. The coach was able to respond and identify best recommendations to teachers immediately when needed, in situ. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |16 Overall, while the coach did not give individual feedback (as they had done in 2019), they continued to give feedback in a positive manner. They are still very detail-oriented in group feedback sessions. School Based Workshops (2020) The coach did not hold SBWs at either school, stating that schools were busy with assessments. In 2019, the coach tried to organize an LTSM catch-up training for teachers who missed previous training sessions but this did happen due to an alleged teacher strike. The coach reported that they took the initiative to engage with a new teacher who had fallen behind in implementation and needed additional support. The coach reported that they utilized a needs-based approach to SBWs. 4. COACHING PRACTICE In 2020, the coach appeared to have a good working relationship with the teachers and the coach was aware of COVID-19 challenges. The coach was aware of curriculum coverage challenges, baseline assessments and knowledge gaps. The coach was proactive in addressing challenges and seemed to communicate well. The coach did not provide the evaluators with their WhatsApp chat so their communication with the teachers was not confirmed. The coach appeared well versed in the LTSM, passionate about assisting the teachers to contribute to learners overall development. Teachers appeared to be receptive to feedback and advice. The coach explicitly recognized the strain on the teachers due to COVID-19. The coach was encouraging, empathetic and helpful to teachers. However, the coach did not ask reflective questions. The coach was more confident in their strategies, attentive to the needs of the teachers, and was able to adapt to a COVID-19 environment. The coach remains a mentor to the teacher once they are in the classroom. The coach reported that they are passionate about education, which helped them to connect with the teachers and suggest coping strategies. The coach adhered partially to the RSP coaching observation tool. “It [The SBW] must informed by the needs of the teacher. If I have visited twelve teachers over three times, I can only do an SBW if I have identified it as a need for the teachers and thereafter do it once for all of them, not do it every visit. Other immediate needs will be addressed during the feedback session.” - Coach 5 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |17 5. TEACHER PERCEPTION OF COACH: Teachers were surveyed on their experience of and perception of their coach, this data is based on the responses from teachers who were at schools assigned to Coach 5. Twenty￾three teachers surveyed were from SMT and Coaching schools that were supported by this coach. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |18 6. PROGRAM CONTEXT: COACH RESPONSE TO COVID-19 CONTEXT In 2020, the coach reported that they were able to adapt to the COVID-19 environment and provide support to the teachers in terms of their well-being. Physical interaction was limited this year, but the coach reportedly relied on WhatsApp to communicate with the teachers and share relevant material, although this could not be confirmed by the evaluators. WHATSAPP ANALYSIS The coach indicated that they had a WhatsApp group with their teachers. The evaluation team were unable to access the content, as the coach indicated that they had deleted the WhatsApp chat history30 . During evaluation fieldwork, a teacher survey was conducted in 59 sampled schools. Of this, 11 teachers reported that they were supported by Coach 5 and all 11 found the WhatsApp group useful. FACTORS THAT AFFECT COACHING PRACTICES Most of the challenges documented in 2019 remain. The shortage of teachers continued and school infrastructural problems were still prevalent. New teachers, hired in 2020 had not been trained on RSP and were reportedly prioritized by the coach. COVID-19 meant that the coach was not be able to visit teachers as planned, and therefore reported providing virtual support via WhatsApp although this could not be confirmed by the evaluators. Overall, the coach has been able to maintain their successes from the previous year. Although the coach emphasized inclusive education, teachers were not always as resourceful or experienced to assist these learners as needed. COACHING SUPPORT (2020) This coach does not feel adequately supported by their Head Coach. While administrative support was provided and the head coach sometimes approved their plans, the coach reported feeling unsupported regarding curriculum implementation. The coach reported that 30 The evaluators requested the WhatsApp group download starting in October 2020 and repeated the request several times, including through a call to the coach. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |19 the only change in support from 2019 was that Head Coaches visited them at schools, however this coach reported that the Head Coach had not visited them in 2020. The coach reported that the only time curriculum support was provided from Head Coaches was when one coach had requested the Head Coaches to do a rerun in a workshop.31 The coach felt that SMT support had improved in 2020, comparted to 2019, but was still insufficient. The coach believes SMT now understand the role and value of the coaches in their school. 31 This information was reported by the coach in the interview, but it could not be verified from the monitoring data, or other data sources. “The way I understand our main role is curriculum implementation… There isn’t a day when they [the Head Coach] will make an offer to assist us with for instance, how to present writing in Grade 3. Curriculum based support of that nature isn’t available.” - Coach 5 “When we started it was all unclear but now that they see improvements and they hear from the teachers that the RSP program is really helping, the support [from SMT] has improved.” - Coach 5 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |20 32 2019 column – observations conducted in Term 1 in January 2019, 2020 column observations conduced in Term 3 in October 2020 33 The coach reported that before October 2020, they had not received a visit from the head coach. The FPD monitoring data says that the coach was visited on 6 November 2020. Coach 5 Criteria from Coaches’ Job Descriptions 2019 2020 Criteria from Coaches’ Job Descriptions 2019 2020 Qualified as teacher Yes Yes Monitors progress against lesson plans for the whole lesson Inadequate Inadequate Teaching experience Extensive Extensive Looks at learners’ books Inadequate Partial Foundation Phase experience 12 years 12 years Monitors teachers’ assessment processes Partial Inadequate Ratio coach to schools 1: 7 6 Provides feedback to teachers Fully Fully Ratio coach to teachers 1: 50 46 Teachers reflect on their practice Partial Partial Setswana language proficiency Fluent Fluent Conducts needs-based workshops in the afternoons 0 of 4 0 of 2 Develop monthly visit schedules Inadequate Fully Sets up PLC Not Reviewed Not reviewed Monthly Plan submitted to Head Coach Inadequate Fully Facilitates PLC Not Reviewed Not reviewed Head Coach Approval Inadequate Fully Completes in-class coaching tools Partial Partial Coach tells teacher about planned visit Fully Fully Completes SBW / PLC tools Not Reviewed Inadequate Coach sets up direct communication channel with teacher (not through school) Fully Fully Completes training tools Not observed Not observed Coach re-confirms visit Fully Fully Participates in training sessions 16 days Not observed, monitoring data indicates participation in all sessions Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Partial Fully Head coach visits the coach at school32 None Once33 Follows steps of Class Observation Partial Partial Head coach provides support to coach Inadequate Inadequate READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |21 COACH CASE OF POOR COACHING PRACTICE (COACH 7) 1. PROFILE Overall rating: Planning and Preparation: Pre-observation discussion: Classroom observation: Feedback/Coaching practice: School-Based workshops: Length of time in RSP as coach: 23 months by 2020 Coach training: All RSP Training 2. OBSERVATION SUMMARY In 2020, the coach conduct their observations differently due to COVID-19. Changes in the annual plan meant teachers improvised on teaching and assessments dates. The “One must be professional and patient enough, especially in the big schools where there are many classes and challenges. One needs to assure the teachers that I am not there to find fault, but I am actually there to support them…” – Coach 7 “The coach was very average. Their approach was procedural and they wanted to get things done and out of the way. They went ahead with lessons regardless of the assessment process and timing. They did seem to have a good relationship with the teachers, and their needs based workshop was good.” – Associate evaluator shadowing Coach 7 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |22 observation took place over two days at one school during October 12-13, 2020. Six classes were observed on the first day and a SBW was conducted on the second day. The visit was arranged by the coach and confirmed with the teachers, however, assessments still went ahead and the coach continued with the observation program as planned. As a result, instead of a lesson, the coach observed the assessments instead. According to the evaluator, the planning, pre-discussions and classroom observation elements were therefore compromised. Where full observations took place, the coach was less rigorous about checking the teacher’s every task and responsibility in comparison to 2019 where the coach was more thorough. According to the evaluator, the coach has shown growth in their ease with the program, they are able to improvise when necessary without unsettling the teacher and the observation- be it assessment or lesson. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |23 3. COACHING STANDARDS FIDELITY TO COACHING AND RSP PROGRAM In comparison to 2019, the coach’s administration skills seems to have deteriorated e.g. not using tools during observation, minimal communication trail between the coach and teachers over the two years. This, the evaluator believes, could partly be due to the assessments and subsequent compromised observations, as well as the program’s break during COVID-19 lockdown. The coach, however, still needs improvement on their post-observation discussions. After observing a teacher, the coach would give general feedback, and saved the rest of the feedback for the discussion on day two. Although the discussions were based on what was observed the previous day, it was not structured, and there was very little reflection on the teacher’s individual observations themselves. Despite these challenges, the coach was more detailed in their group feedback and showed their strength as a coach. They addressed challenges with reading, different ways of using the books and flash cards. The evaluator’s assessment of the coach is that they are overall a good communicator and knowledge sharer. The coach showed eagerness to work with the teachers, although they could not due to assessments. According to the evaluator, the coach’s willingness, almost insisting to still meet with the teachers after school on the second day, was commendable. Although the session was not structured and detailed, it allowed teachers to share some challenges and open up a platform to discuss those challenges. QUALITY OF COACHING Planning preparation in 2020: For each visit, the coach still prepares using a lesson plan like they did in 2019. According to the coach, they plan ahead so that if there is a teacher who is already in week five, they are able to prepare and lead the teacher into the next week. For the 2020 observations, the coach indicated that they had sent their schedule for October to the Head Coach, and received feedback on the October 6. The coach still communicates with the HOD, confirms the lesson plans, and availability of the teachers prior to the visit just like they did 2019. A missing element noted from 2019 was that the coach still hasn’t created teacher profiles in 2020. According to the coach, their intention was to see whether the program still continues in November, then they will identify schools and explain to the Head Coach that those schools need intervention. Pre-observation discussion in 2020: Apart from greeting the teachers, explaining the situation and getting consent from the teachers, pre observation discussions were not similar or as thorough as those observed in 2019. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |24 Classroom Observation in 2020 On day one, no formal/scheduled teaching took place because teachers were busy with assessments this year. The coach saw 6 teachers, each observation took 30-35 minutes. Where classes were busy with assessments, the coach did not do much except check the DBE books, learner workbooks, signed certain pages, and did not really engage with the teachers. The coach seemed concerned with how the teachers were coping with COVID-19, how many learners were attending since the rotational attendance, etc. Two Grade 3 lessons were observed. The Grade 3 EFAL lesson was on listening and reading. When there was a proper lesson in session, coach did not have one on one feedback session with any of the teachers, but rather saved feedback for the needs based workshop the following day. Made feedback on the day general; feedback – not specific. There was a time when the coach stepped into role play what the teacher had to do with the learners. The coach wasn’t correcting anything, they were was just participating. According to the evaluator, it would appear that the coach has gotten comfortable with the RSP material and guiding the teachers through it. This was why the coach was able to go through the workbooks and checklist with ease. Although it was not the usual observation in some classes, the coach knew how to work around it, put the teachers at ease, and still manage to somewhat accommodate the observation objectives. The coach still maintains a good relationship with the teachers, and even with the assessments, they still paid attention to some details and were always ready to step in to assist the teachers. School Based Workshops (2020) A SBW was held on day two, October 13, 2020. Due to COVID-19, the coach stated they were unable to organize more workshops this year. The SBW was based on the observations made on day one. The coach was at the same school over the two days of coach shadowing. The workshop was not focused on any particular subject but covered problematic aspects of both HL and EFAL Grade 1-3 observed the day before. The coach acknowledged the challenges of working in a post COVID-19 environment. They urged teachers to be aware of the need to continue with Learner workbook assessments and to remain positive. They furthermore covered challenges on curriculum coverage, baseline assessments, group guided reading and how to use homework to maintain engagement with learners. The coach emphasized coping mechanisms with COVID-19, encouraging the teachers to work together. They encouraged teachers to take advantage of the situation with the smaller classes, they kept referring to retaining learners (keeping them back a year), and so that teachers could maximize curriculum coverage. The coach is noted as making less comparison statements than 2019 for instance comparing teacher performances Only time the evaluator really observed the coach’s skills, was when the coach gave feedback in the needs-based workshop. From what was observed, all nine teachers he was coaching were there, although not all of them were observed during the two days. The session was interactive, and evaluator saw the coach’s expertise. Prior to that, he would just check books and say one or two things. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |25 4. COACHING PRACTICE According to the evaluator, the coach is recognized as a leader. This can be attributed to the coach’s good interpersonal skills with the teachers, and ability to maintain working relations with the schools. The coach admitted that the WhatsApp platform was a bit restrictive, but they still tried to reach out to the teachers when circumstances allowed, this was either through phone calls or school visits when the coach could. The evaluators note that the coach is well versed in the program, comfortable mentoring teachers and sharing knowledge. Most of the coach’s observations could not be conducted as planned, but the coach kept a positive attitude, and genuinely wanted to check in with the teachers post level five and four lockdown. The coach was prepared to deal with questions on curriculum coverage, baseline assessments and how to make the most out of the materials. They took the initiative to organize an interactive feedback session with the teachers to accommodate both the observation, and the wellbeing of the teachers on day two. The coach also provided additional support and motivation to the teachers but encouraged them to rely and support each other in difficult times. A notable area of development for the coach is improving their administrative adherence because the coach does not work with the observation tools as required. Although the coach has good interpersonal skills, the evaluator observed little communication between the coach and teachers outside the class visits. 5. TEACHER PERCEPTION OF COACH 7: Teachers were surveyed on their experience of and perception of their coach, this data is based on the responses from teachers who were at schools assigned to Coach 7. 12 teachers surveyed were from SMT and Coaching schools that were supported by this coach. All 12 of these teachers indicated that they received supporting visits from this coach. “My understanding of the SBW is that we have to support teachers where we have noticed weaknesses...” - Coach 7 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |26 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |27 6. PROGRAM CONTEXT: COACH RESPONSE TO COVID-19 CONTEXT Prior to COVID-19 lockdown, the coach met with teachers. After lockdown, the coach reported that they switched platforms, and reported that they relied more on WhatsApp. The coach said they emphasized to the teachers the importance of conducting baseline assessments of the learners when the schools reopened in August 2020. According to the coach, some teachers would call, however, providing help from those platforms sounds like it was a bit restricting for the coach, but they made it work. The main challenge with virtual support was that although the coaches had access to data and technology, the teachers were neither subsidized nor prepared. WHATSAPP ANALYSIS The coach seemed to have more than one WhatsApp group. They provided the downloaded data of one group. This group was initiated by the Coach in May 2020 and closed it on the November 12, 2020. It was one of the least active and responsive groups, with only 59 messages of which 41 were from the coach. On the group, the coach shared their visiting schedule, initiated motivational discussions with the teachers and tried to encourage interaction. The group was not interactive, only one teacher responded, which means the coach did most of the talking and information sharing. The coach acknowledged that data is a challenge for the participating teachers and encouraged them to join the discussion. The coach did not seem demotivated by the lack of responses from the teachers and continued to share information. The coach shared information on ATPs, whether or not teachers received material, tips for teachers and motivational quotes and talks, suggestions on using the LTSM material, and baseline assessments. During evaluation fieldwork, a teacher survey was conducted in 59 sampled schools. Of this, 10 teachers reported that they were supported by Coach 7 and all six found the WhatsApp group useful. READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |28 FACTORS THAT AFFECT COACHING PRACTICES According to the coach, COVID-19 was an obstacle. The coach reported they could not meet with the teachers who were unable to interpret the contents of the lesson plan without their assistance, especially in terms of assessment. The coach said some teachers would call and say that they are unable to use the plans, then the coach would say that it is difficult to explain telephonically. The coach would then suggest that the Subject Advisor might know more. The coach said meeting with the Subject Advisor was also a challenge because teachers given the lockdown restrictions. If they met once with the Subject Advisor, they know that they are applying everything they did in the training where the subject advisors are invited. This way, the coach cannot give teachers different stories when they meet them. According to the coach, some teachers were ‘lazy’. HEAD COACH AND OTHER SUPPORT (2020) The coach believes that support is readily available, especially in 2020. In February 2020, the coaches were told to report any problems immediately rather than wait for the monthly report, so that it could be attended to immediately. The coach felt the Head Coach supports them adequately. This is excluding infrastructure/technological challenges like data or other issues beyond the Head Coach’s control. The coach attended the UNICEF training. This covered almost everything including time and classroom management, which the coach found are the main obstacles in curriculum coverage. The coach therefore believes that their developmental needs are met. Some of the things the coach finds in a lesson plan are given in a brief way, but if they consult the developmental training material, it’s given in full. Sometimes the coach wishes they could share it. “The RSP lesson plans, however, do not allow for any laziness because every week, teachers are expected to do something. If you miss two days you can’t go forward, you must go back. Most teachers fall behind because of sick leave or meetings and lesson plans don’t allow for shortcuts. When you go through the plans at the training or during consultations, the teachers find it very difficult, they don’t like it that way because they have to go back to the missed lessons.” - Coach 7 “Yes, as far as I am concerned, my Head Coach supports me adequately. Apart from technology challenges like data or issues that are beyond them” – Coach 7 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |29 According to the coach, there is a system in place for ensuring that coaches get to schools, and assistance in terms of drawing a monthly plan. The plan tells you where you are supposed to be. This makes it easier for schools/teachers to find you when they have challenges. At the school environment, the coach depends entirely on the HOD of that particular phase. Even if the coach sends a reminder to the school, they make work through the HOD, thereafter they report that they send a WhatsApp to all of the teachers; but the coach’s first point of call is the HOD’s, especially If they have to consult privately on a teacher’s performance. “I would suggest that let the SMT, Principal , where there is Deputy Principal, let them attend the training with the teachers and the coaches so they are also informed on the information what is happening…” – Coach 7 READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |30 COACH 7 RUBRIC: 34 2019 column – observations conduced in Term 1 in January 2019, 2020 column observations conduced in Term 3 in October 2020 Coach 7 Criteria from Coaches’ Job Descriptions 2019 2020 Criteria from Coaches’ Job Descriptions 2019 2020 Qualified as teacher Yes Yes Monitors progress against lesson plans for the whole lesson Fully Fully Teaching experience Extensive Extensive Looks at learners’ books Fully Fully Foundation phase experience 15 years 15 years Monitors teachers’ assessment processes Fully Fully Ratio coach to schools 1: 10 10 Provides feedback to teachers Partial (but not appropriate ) Partial (but not appropriate) Ratio coach to teachers 1: 52 52 Teachers reflect on their practice Partial Partial Setswana language proficiency Fluent Fluent Conducts needs-based workshops in the afternoons 3 of 4 Not observed Develop monthly visit schedules Fully Fully Sets up PLC Not Reviewed Not reviewed Monthly Plan submitted to Head Coach Fully Fully Facilitates PLC Not Reviewed Not reviewed Head Coach Approval Inadequate Inadequate Completes in-class coaching tools Fully Not observed Coach tells teacher about planned visit Fully Fully Completes SBW / PLC tools Not Reviewed Not observed Coach sets up direct communication channel with teacher (not through school) Fully Fully Completes training tools Not observed Not observed Coach re-confirms visit Fully Fully Participates in training sessions 16 days Not observed, monitoring data indicates attendance all sessions Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Fully Fully Head coach visits the coach at school34 Not reported Once Follows steps of Class Observation Fully Fully Head coach provides support to coach Adequate Adequate READING SUPPORT PROJECT: SUMMATIVE EVALUATION COACH CASE STUDIES |31 DATA SOURCES FOR COACH CASE STUDIES Data Sources Coaching Observation Tool Part C, Coaching Observation Tool Part B, 2019 Evaluator debriefing interview September 2019 Coaching Observation Tool 2020; Coach interview 2020 Teacher Survey, 2020 Monitoring data, 2019 and 2020 Valuing Criteria based on RSP Standard Operating Procedures And RSP Coach Job Description Design Report (August 2019) Table 7: Assessing the Coaching Design Comparison of 2019 and 2020 data Variables of teacher￾centric coaching practices as defined in Zenex Literature Review on Coaching in Education (2019) ANNEX 9: RSP THEORY OF CHANGE ASSUMPTION RATINGS The RSP assumptions documented in the Design Evaluation are as follows, with assessment for 2020 on if the condition holds true for the desired change to occur: 1. Assumption Holds - Plausible and proven 2. Assumption Holds - Plausible and unproven 3. Assumption does not hold – not plausible 4. Assumption Adapted due to COVID-19 design change 5. Major divergence from the assumption based on implementation evidence (not related to COVID-19 effects) In addition, the assessment rates if the RSP implementation was sufficient, partial (some aspects sufficient but not consistent), insufficient (desired dosage not achieved) or unknown. The focus of this assessment is to measure the likelihood of the contribution to the desired change as stated in the SOPs and design evaluation: if the evidence overall indicates above 80% achievement of implementation it is deemed sufficient, below 50% insufficient, and 51-79% partial or insufficient evidence. Note: CA refers to Curriculum and Subject Advisors in the tables below. Icon Description Yes No Partial Divergence from assumption ? Unknown Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Materials Activities: All teachers receive necessary LTSM prior to the start of school term 70% of teachers received LTSM in 2019 and Term 1 and 2 of 2020 96% collected in 2020 Term 3 and 4 97% for coaching schools Term 2 and 4, 2020 (FPD Monitoring data) 99% of teachers reported receiving LTSM and Lesson plans at teacher training. (Teacher survey) Outputs: Teachers have the LTSM in the classrooms Classroom observation and coach shadowing confirmed teachers use of LTSM in classroom Lesson plans checked and verified by Curriculum and Subject Advisors (CAs) CA/SA participated in Reference Group meetings and provided input which was included in the lesson plans (KII) The EFAL and HL content is adequately paced, aligned with the curriculum, and integrates the LTSM District and CA were involved in the material development and refinement to ensure alignment and integration. No material review was undertaken. Outcomes It is possible to implement the strategies in the lesson plans with large classes35 Diversion from the Assumption – strategies differ and implementation dependent on group or class size 35 According to the OECD, smaller size classes are often seen as beneficial because they allow teachers to focus more on the needs of individual students, and reduce the amount of class time needed to deal with disruptions. According to the DBE Norms and Standards: a primary school classroom should not have more than 35 learners per educator. The Annual Performance Plan (2018) for the North West Education Department, however, indicates that 45 percent of learners are in classes with more than 45 learners. Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Classroom libraries Activities All teachers receive classroom libraries in time Classroom library intervention implemented in 2019. Formative report states materials had been received and being used, although not all schools had received all materials. All teachers receive training on how to select titles for learners, and how to manage the classroom libraries 93% of all teachers (who received a classroom library) indicated that they had been trained on them Outputs Classroom libraries contain a range of appropriate titles applicable to different reading levels A range of materials available. 40% of classrooms had more than 20 picture books available (across all grades). 70% had books in English, and 91% had books in Setswana. Classroom libraries are accessible and well-managed (using the book management system) to make sure all learners benefit 50% of classroom observations indicated that the management chart was available. 61% of classroom observations indicated that the management chart was not filled in. Outcomes The library management system works, and learners are able to take books home and return them in time 78% of teachers stated learners are taking books home Teachers are able to incorporate the classroom libraries without explicit mention in the lesson plans 83% of teachers said they receive sufficient support, however coaches drive classroom practice 40% of teachers found library somewhat helpful. Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) and without compromising the use of other LTSM Learners have the ability to read on their own No evidence (impact evaluation required) Teacher Training Activities Competent coaches deliver training with fidelity Quality of coaches training varies as observed during teacher training, however all teachers followed the agenda and slide presentation Training venues are adequate Observed at teacher training. Venues were changed during implementation if they were inadequate Teachers attend all four training sessions 44% of teachers attended all four training sessions in 2019 67% in teachers attended all 3 training sessions in 2020 (due to COVID-19 one was not held). 47% of teachers attended 6 out of 7 training sessions. Relevant teachers attend specific trainings (by Grade) Improvement in 2020 and therefore teachers attended the specific training by Grade (monitoring data) Adequate catch up sessions are implemented for those who missed training Divergence from assumption – catch up not clearly defined in SOP and therefore sharing and group feedback was reported as catch up sessions Time allocated for training is enough to cover content Insufficient time for training to cover full content as noted in training observations Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Sequencing HL and EFAL training is optimal, and complements teachers’ learning Assumption does not hold - HL and EFAL were alternated as first and second session during training and did not affect teacher participation. Evidence of teacher learning required. The sequencing of longer and shorter training sessions is optimal Differentiated training agenda observed No evidence to confirm optimal and no monitoring data on actual duration of sessions (hours) ? Outputs Teachers are empowered and motivated to deliver learning via scripted lesson plans 80.5% of teachers found the scripted lesson plans the most useful Home Language resource 79.9% of teachers indicated they used HL lesson plans every day, 81.9% indicated they used EFAL lesson plans every day No data on level of empowerment or motivation. Teachers refer to LTSM and noted exercises at specified periods per lesson plans Majority of teachers use 1 or more of the LTSM items during lessons as observed Teachers understand how to implement lesson plans and reference LTSM Teachers indicate frequent use (every day/nearly every day) of the materials as observed Teachers spend the allocated amounts of time in teaching EFAL and HL Adapted ATP and lessons times due to COVID-19 affected implementation of design standard. Limited observation conducted ? Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Teachers understand the technical aspects of teaching No evidence Teachers’ SACE continuous professional training and development (CPTD) points are allocated No evidence Outcomes Teachers implement as trained Insufficient evidence – self-reported data from teacher survey and limited classroom observations Inclusive education, pacing, curriculum coverage, and all related aspects being implemented fully One lesson was observed and not full delivery of package observed to provide sufficient evidence Elements targeted at SMT members (Foundation Phase HOD and Principal or Deputy) Activities Competent facilitators conduct training SMT training was not observed. Evidence based on SMT mentor interview and experience Support from District to encourage participation No monitoring data provided. Foundation Phase HOD and Principal or Deputy attend all training(s) [Minimized clashes with District activities] Only 5 schools attended all and 21 SMTs received full package (monitoring data) Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Clustering exercise adequately done No evidence Course responds to identified needs of SMT and incorporates practical strategies for them to fulfil their supervisory, administrative, literacy teaching in schools No data on needs of SMT provided Participants apply themselves to practical exercises No feedback on Portfolio of Evidence marks Outputs SMT are motivated and apply their knowledge in their respective schools Across the results SMT interventions are doing worse, or equal to the coaching and LTSM interventions. However, teachers in the SMT schools reported high levels of support. SMT have the necessary resources to carry out their tasks No evidence SMT receive the necessary support from international volunteers Divergence from assumption - model changed when VSO withdraw from program, and COVID-19 prevented SMT mentor to conduct visits to SMTs SMT encourage reporting on SA-SAMS in their schools 85% of all teachers reported that their SMTs encourage them to report on SA-SAMS. (Large extent and to a very large extent) with Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) the highest % of support through the Coaching and SMT intervention (both 49%). SMT receive supervisory visits from volunteers and maintain a positive working relationship Divergence from assumption - model changed when VSO withdraw from program, and COVID-19 prevented SMT mentor to conduct visits to SMTs Outcomes Noted reading culture in schools 91% of teachers in the SMT intervention reported that the SMT supports a culture of reading in schools (large extent and to a very large extent), compared to an 85% average across interventions Teachers receive supervisory support from SMT 85% of teachers in the SMT intervention reported receiving supervisory support, compared to an 82% average across interventions Regular meetings/updates in school to increase accountability (Reporting on SA-SAMS – evidence of using data for decision making) 88% of teachers in the SMT intervention reported this, compared to 84% average across interventions Schools sustain gains No evidence – requires long-term evaluation Elements targeted at Curriculum Activities Attending teacher training and coach training aligns with CAs skills development needs No data on CA development needs Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Advisors (CAs) CAs pay attention to training Coaches were scored during training on role play and given feedback by CAs. Clashes with District activities minimized to allow for maximum attendance The final SMT PLC clashed with District activity and therefore limited attendance by CAs and SMTs. No further data available over the duration of the program CAs receive tablets loaded with relevant and useful content All CAs reported tablets were received with all the HL and EFAL LTSM and material CAs participate in a Community of Practice (COP) in Year 2 No evidence and not included in the SOP – possible divergence of model Outputs CAs use the tablets and content for supporting reading in schools No evidence of use CAs understand the teacher training course and its practicality CAs attended teacher and coach training No evidence of understanding CAs perceive the course as critical Supportive overall of course, but no evidence defined as critical Trained CAs are empowered to apply their knowledge in literacy/pedagogy No evidence Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) CAs gain the expertise to enhance instructional leadership in schools No evidence CAs have the necessary resources to implement their roles and responsibilities No evidence Outcomes CAs apply their new knowledge No evidence Teachers receive capacity building on curriculum, and support from CAs The focus was on capacity building of teachers and coaches, no evidence of specific focus on capacity building of CAs to provide support to teachers. No evidence of CA support to teachers beyond interviews with CAs. Overarching There is sufficient integration across components and actors involved in different components to work together towards improving literacy outcomes Insufficient integration as excluded partners from key management activities (as reported by partners), SMT not sufficiently integrated into teaching and classroom practice; CA and coach relationship, roles and responsibilities not defined or integrated towards a collaborative approach to supporting teachers. The combination of components delivered to each school are sufficient to lead to improved learner outcomes in all classes Unproven – impact evaluation required ? Component Assumptions Assessment Evidence 2020 Plausible Proven implementation practice Sufficient implementation (likely to lead to change) Participants are willing to participate in RSP activities, and see the need for RSP Individual teacher and SMT member attendance does not meet design specification The table below reviews the assumptions for the coaching intervention to draw comparison to the findings of the Formative Evaluation (2019) Coaching assumption Findings 2019 Findings 2020 Plausible Proven Sufficient implementation Coaches have the necessary competency and skills to coach (has appropriate teaching, reading and coaching experience, familiar with the lesson plans and LTSM, able to establish good rapport) Partially. Six of the coaches have degrees (either in education or Setswana). Seven of the coaches have diplomas in education. One coach has a post-matric certificates in education and five modules of a degree program. Coaching observation data suggests that coaches are familiar with the lesson plans and LTSM, and many are able to establish good rapport with teachers. While the assumption holds, Coaches are not homogenous in terms of background and experience and may provide variable support to teachers. Assumption Holds - Plausible and proven Insufficient: COVID-19 affected the implementation and resulted in a divergence for Term 2 and 3 from the Theory of Change RSP improvement plan for 2020 included removing the two poorest coaches. Promoted one to head coach to replace poor performance head coach from 2019. Limited training was provided to new coaches, but insufficient orientation. Coach and head coach was provided virtually during 2020. Due to COVID-19 school closures new coaches could not build a strong build relationship with teachers as could only see them twice before lockdown. Head coach support to coaches viewed by coaches as insufficient: no guidance on monthly plans, no ongoing or developmental feedback on what doing right or how to improve, they provided administrative support but on the quality of coaching Differentiated levels of coaching practice remain. Coach to school and coach to teacher ratio is manageable No, higher coach to school and teacher ratio than recommended standard Assumption Holds - Plausible and proven Insufficient: Six coaches ratio’s to number of teachers increased, eight coaches met the design specification The design specification assumes that each school has approximately six Foundation Phase teachers to train. This assumption does not seem to hold. Using the full database of teachers who attended at least one FPD training session, the average number of teachers per school who attended the training, is closer to seven per school. Assumption Holds - Plausible and proven Insufficient: Average of 6.27 for 2020 teachers per school attended Teachers can schedule in coaches, and lesson observation takes place (minimum disruptions occur) Yes, although coaches need to communicate better with the SMT to arrange that lesson scheduling allows for sequential lesson observation. Assumption Holds - Plausible and proven Sufficient: In 2019, Term 1 and 4 in 2020, the coach and teacher communicate on scheduling and improvement noted from 2019. However, the school time tabling can affect teachers’ ability to schedule time for coaches, for example when assessment weeks are planned, and EFAL lessons across grades may be planned at the same time. Scheduling of coaching sessions was a concern to Unions as it was viewed to take time away from mathematics (reported by one teacher). Recommendation: Involve unions in providing recommendation on how the RSP can be replicate in schools Teachers present positive/learning attitudes and understand how the coach’s role is different from a curriculum advisor Partially. Teachers generally present positive attitudes to coaches but there is a lack of clarity on roles. Assumption Holds - Plausible and proven Partial: Teachers generally present positive attitude to coaches but the confusion on roles between coach and CA remains The coach conducts needs based school workshops every time he / she visits a school No, the FPD monitoring data on school￾based workshops is incomplete and it is likely that the number of sessions that took place is significantly underreported in the monitoring data. An analysis of the coaching observation data reveals that coaches conducted less than 50% of their planned visits during the observation week. Assumption Holds - Plausible and proven Partial: Inconsistent practice by coaches as seen in the monitoring data for Term 1 and Term 4; and supported by coach shadowing data. The data indicates a confusion by coaches of the understanding of the needs based workshop and feedback session, and group versus individual feedback. Recommendation: Clarity on differences between school-based workshops and feedback session required through defined indicators and reporting requirements. The coach is able to have individual coaching sessions with each teacher at least five times per year No. According to the FPD monitoring data, teachers received on average only one individual coaching visit in 2019, and not even 20% of the planned individual coaching sessions were delivered. Assumption Holds - Plausible and proven Insufficient: The monitoring data indicates on average coaches had one individual coaching session during 2020 (excluding the COVID-19 closure period). ANNEX 10: ANALYSIS OF LEARNER WORKBOOKS (2020) ANNEX 11: LIST OF RSP MATERIA ENGLISH FIRST ADDITIONAL LANGUAGE LTSM- RSP Grade 1 Grade 2 Grade 3 1 Lesson plans: The day-to-day lessons that the teachers need to teach throughout the term. Included in the lesson plans are: core methodologies; assessment rubrics; weekly routines (each term) Lesson plans: The day-to-day lessons that the teachers need to teach throughout the term. Included in the lesson plans are: core methodologies; assessment rubrics; weekly routines (each term) Lesson plans: The day-to-day lessons that the teachers need to teach throughout the term. Included in the lesson plans are: core methodologies; assessment rubrics; weekly routines (each term) Termly 3 Big Book stories (2 per term) Big Book stories (2 per term) 4 Flashcards: Printed sets of the words teachers need to teach theme vocabulary and phonics. Flashcards: Printed sets of the words teachers need to teach theme vocabulary and phonics. RSP provides theme vocabulary words (based on the themes in the DBE Rainbow Workbooks) as well as flashcards for the vocabulary in the Big Books and Readers. 5 Theme vocabulary: Provided to teachers in the form of flashcards with training on how to use the cards for revision games and extension activities. Theme vocabulary: Provided to teachers in the form of flashcards with training on how to use the cards for revision games and extension activities. 6 Not applicable Graded Readers: AWEH! Graded Readers: AWEH! Once off 7 Examples of learners’ work: Exemplars are provided to demonstrate how teachers can check for learners’ understanding of shared reading tasks, and to demonstrate the required standard of written work Examples of learners’ work: Exemplars are provided to demonstrate how teachers can check for learners’ understanding of shared reading tasks, and to demonstrate the required standard of written work Examples of learners’ work: Exemplars are provided to demonstrate how teachers can check for learners’ understanding of shared reading tasks, and to demonstrate the required standard of written work Built into lesson plans 8 Curriculum Tracker/ Teacher monitoring and support tool: A selfmonitoring instrument for teachers. This support tool summarizes the core activities in the lesson plans and helps teachers track and record the efficacy of their delivery of the EFAL curriculum. Curriculum Tracker/ Teacher monitoring and support tool: A self-monitoring instrument for teachers. This support tool summarizes the core activities in the lesson plans and helps teachers track and record the efficacy of their delivery of the EFAL curriculum. Curriculum Tracker/ Teacher monitoring and support tool: A self-monitoring instrument for teachers. This support tool summarizes the core activities in the lesson plans and helps teachers track and record the efficacy of their delivery of the EFAL curriculum. Termly Tracker – includes forms for the tracking of Group Guided reading SETSWANA HOME LANGUAGE LTSM- RSP Grade 1 Grade 2 Grade 3 Notes 1 Scripted lesson plans for each day with core methodologies sections (each term) Scripted lesson plans for each day with core methodologies sections (each term) Scripted lesson plans for each day with core methodologies sections (each term) Termly 2 Graded readers (Term 1 –Vula Bula 16 Titles; 20 copies each title ) Graded readers (Term 1 –Vula Bula 16 Titles; 20 copies each title ) Graded readers (Term 1 – Vula Bula 8 Titles; 20 copies Once off ** Special noting for future work (beyond RSP 2019-2020 scope) A Vula Bula anthology was developed for Sesotho, isiXhosa and is currently being finalized for Setswana. This will be used in the future 3 Book Register Exercise books set up as accession registers for the Vula Bula reading books. (Term 1) Book Register Exercise books set up as accession registers for the Vula Bula reading books. (Term 1) Book Register Exercise books set up as accession registers for the Vula Bula reading books. (Term 1) May use school’s accession system, or create sheets but keep record 4 Handwriting poster – demonstrate the form and directionality of lower and upper case letters (Term 1) Handwriting poster – demonstrate the form and directionality of lower and upper case letters (Term 1) Handwriting poster – demonstrate the form and directionality of lower and upper case letters (Term 1) Once off Gr 1 – print Gr 2 – print and cursive Gr 3 – print and cursive 5 Posters (2 per term) Posters (2 per term) Posters (2 per term) Once off Previously used posters, request use of Big Books moving forward Book Books: Quotation by Molteno Grade 1 - 3 books (8 titles available on Molteno website) Grade 2 - 5 books (8 titles available on Molteno website) Grade 3 - 4 Books (specify no. of titles) 6 Flash card words – sight words (each term) Flash card words – sight words (each term) Flash card words – sight words (each term) Termly Sight words, theme title, phonic sound and words 7 Reading words – linked to the Vula Bula Graded Readers (each term) Reading words (each term) Reading words (each term) Termly Theme words, Group Guided reading words 8 Assessment tasks with rubrics/ mark sheet –rubrics provided criteria for teachers to use to award objectives assessment rating for learner tasks (each term) Assessment tasks with rubrics/ mark sheet –rubrics provided criteria for teachers to use to award objectives assessment rating for learner tasks (each term) Assessment tasks with rubrics/mark sheet –rubrics provided criteria for teachers to use to award objectives assessment rating for learner tasks (each term) Built into lesson plans 9 Assessment record book- CAPS & SA-SAMS complaint assessment record tables. Teachers used these to record formal assessments of learners (each term) Assessment record book- CAPS & SA-SAMS complaint assessment record tables. Teachers used these to record formal assessments of learners (each term) Assessment record book- CAPS & SA-SAMS complaint assessment record tables. Teachers used these to record formal assessments of learners (each term) Termly 10 Curriculum tracker (each term) Curriculum tracker (each term) Curriculum tracker (each term) Termly To be displayed in the classroom on the wall 11 Weekly routine (each term) Weekly routine (each term) Weekly routine (each term) Termly Built into lesson plans – remains constant throughout the year. 12 Teacher Files (Management and storage files) Teacher Files (Management and storage files) Teacher Files (Management and storage files) Once off or termly– depending how lesson plans are bound. 13 Facilitators’ Guides -Detailed handbook for trainers to use when training teachers Facilitators’ Guides -Detailed handbook for trainers to use when training teachers Facilitators’ Guides -Detailed handbook for trainers to use when training teachers Termly 14 SMT Monitoring and Support Tool SMT Monitoring and Support Tool SMT Monitoring and Support Tool Termly 15 Display Boards – 4 different colored A2 display boards (laminated) to frame the work for the theme. One board for each of the following: - Phonics - Sight words - Theme words - Writing frame Display Boards – 4 different colored A2 display boards (laminated) to frame the work for the theme. One board for each of the following: Display Boards – 4 different colored A2 display boards (laminated) to frame the work for the theme. One board for each of the following: Once off ANNEX 12: STATEMENT OF DIFFERENCES There were no statements of differences at the time of writing this Summative Implementation Evaluation report.