READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |i EVALUATION REPORT FORMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA February, 2020 Contract No: 72067418D00001, Order Number: 72067419F00015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd. Contract No: 72067418D00001, Order Number: 72067419F00015 ii READING SUPPORT PROJECT EVALUATION REPORT FORMATIVE IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT IN SOUTH AFRICA February, 2020 Photo Credit: Brandon Barnard for USAID AUTHORS Jennifer Bisgard (Project Director) Katharine Tjasink (Senior Project Manager) Benita Williams (Senior Evaluator) Leticia Taimo (Evaluation Coordinator) Margaret Roper (Senior Evaluator) Jacqui Dornbrack (Reading Expert) Nombulelo Baba (Coaching Expert) CONTACT DETAILS Katharine Tjasink 26 7th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: ktjasink@khulisa.com Web Address: www.khulisa.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United Contract No: 72067418D00001, Order Number: 72067419F00015 iii ACKNOWLEDGEMENTS Khulisa Management Services (Pty) Ltd would like to thank the United States Agency for International Development (USAID) for supporting the development of this report through the funding mechanism which has enabled the evaluation to take place and through their ongoing engagement with the evaluation team. Khulisa specifically thanks USAID for their contribution and support in the design and delivery of the evaluation, and their input into the content and review of all reports. Khulisa would also like to thank the Department of Basic Education for their input into the evaluation design and the North West Provincial Education Department for their input and continuing endorsement. Khulisa also thanks the Foundation for Professional Development Consortium for their collaboration, support, and input into various evaluation processes. Finally, Khulisa appreciates the time and effort by all who participated in the evaluation, without which this evaluation would not be possible, including the coaches, teachers, and school management teams that we interviewed in the North West Province of South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 iv ABSTRACT From September 2019 to November 2019, Khulisa Management Services (Pty) Ltd conducted an Implementation Evaluation of the USAID-funded Reading Support Project (RSP). The primary aim of the Implementation Evaluation is to account for the adaption of the RSP and inclusion of EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in actual implementation. The key evaluation questions answered in this report include: 1. To what extent did the FPD Consortium implement the intended intervention design with fidelity? 2. Is there evidence of RSP uptake by teachers in class? 3. Is there evidence of changed practices based on the SMT support? 4. Was there any major divergence from the assumptions in the Theory of Change? 5. How could the management and implementation structures be improved? 6. How could the M&E systems be improved? 7. What are the unique challenges of delivery at scale? Khulisa employed a mixed methods design. Findings include: − The dosage, adherence, quality, participant responsiveness, and program differentiation (i.e., program fidelity) varies from being on target to under￾performance. − There is no major divergence from the Theory of Change assumptions. However, some assumptions are not being met due to issues with implementation. For example, fewer teachers attended training than expected, some teachers are not receiving all LTSM, and the dosage of coaching support is not sufficiently high to support behavior change. − Teachers in general are receptive and responsive to the materials. However, they face challenges in implementation. Effective coaches are contributing to improved delivery. Key words: South Africa; Education; Early Grade Reading; Implementation Evaluation; Teacher Training; Coaching Contract No: 72067418D00001, Order Number: 72067419F00015 v CONTENTS EXECUTIVE SUMMARY XIV 1. INTRODUCTION 1 1.1 SYNOPSIS OF TASK, PURPOSE, AND AUDIENCE 1 2. BACKGROUND 3 2.1 THE INTERVENTION 3 2.2 THE EVALUATION 4 2.2.1 DESIGN EVALUATION 5 2.2.2 IMPLEMENTATION EVALUATION 6 3. METHODOLOGY 7 3.1 INTRODUCTION 7 3.2 RESPONDENTS 7 3.3 SAMPLING 7 3.4 METHODOLOGICAL LIMITATIONS 11 3.5 BIAS AND CONFLICT OF INTEREST 12 4. FINDINGS 13 4.1 TO WHAT EXTENT DID THE FPD CONSORTIUM IMPLEMENT THE INTENDED INTERVENTION DESIGN WITH FIDELITY? 13 4.1.1 DOSAGE AND COVERAGE 14 4.1.2 ADHERENCE AND QUALITY 34 4.1.3 PARTICIPANT RESPONSIVENESS 52 4.1.4 PROGRAM DIFFERENTIATION 54 4.2 WHAT EVIDENCE IS THERE OF RSP UPTAKE BY TEACHERS IN CLASS? 57 4.2.1 COMPARATIVE FINDINGS 59 4.2.2 SUMMARY FINDINGS 82 4.3 IS THERE EVIDENCE OF CHANGED PRACTICES BASED ON THE SMT SUPPORT? 84 4.4 WERE THERE ANY MAJOR DIVERGENCES FROM THE ASSUMPTIONS IN THE THEORY OF CHANGE? 89 Contract No: 72067418D00001, Order Number: 72067419F00015 vi 4.5 HOW COULD THE MANAGEMENT AND IMPLEMENTATION STRUCTURES BE IMPROVED? 92 4.5.1 LTSM DELIVERY AND QUALITY 92 4.5.2 ROLES AND RESPONSIBILITIES OF HEAD COACHES AND COACHES 93 4.5.3 RECEPTION OF RSP IN SCHOOLS 96 4.5.4 RECEPTION OF RSP AT DISTRICT AND PROVINCE LEVEL 97 4.5.5 SCHOOL-BASED WORKSHOPS 99 4.6 HOW COULD THE M&E SYSTEMS BE IMPROVED? 101 4.6.1 RSP DATA COLLECTION 101 4.6.2 THE FPD DATA MANAGEMENT SYSTEM AND PROCESSES 107 4.6.3 RSP MEL PLAN 110 4.6.4 FEEDBACK FROM DISTRICT AND PROVINCIAL STAKEHOLDERS 117 4.6.5 FPD REPORTING 117 4.6.6 SUGGESTED IMPROVEMENTS 121 4.7 WHAT ARE THE UNIQUE CHALLENGES OF DELIVERY AT SCALE? 122 4.7.1 RSP TEACHER TRAINING 122 4.7.2 RSP COACHING 123 4.7.3 RSP MATERIALS (DEVELOPMENT, PRINTING, DISTRIBUTION, USE) 125 4.7.4 TEACHERS SEE RSP AS AN “ADDITIONAL REQUIREMENT” NOT AS A DBE REQUIREMENT 126 4.7.5 INVOLVING PROVINCIAL / DISTRICT OFFICIALS 126 5. SUMMARY RECOMMENDATIONS 127 5.1 RECOMMENDATIONS FOR MONITORING AND EVALUATION 127 5.2 RECOMMENDATIONS FOR LTSM 127 5.3 RECOMMENDATIONS FOR HEAD COACHES 128 5.4 RECOMMENDATIONS FOR COACHING 128 5.5 RECOMMENDATIONS FOR DISTRICT AND PROVINCIAL INTEGRATION 130 5.6 RECOMMENDATIONS FOR SCALE UP 130 Contract No: 72067418D00001, Order Number: 72067419F00015 vii 6. BIBLIOGRAPHY 131 7. ANNEXES 138 ANNEX 1: SCOPE OF WORK 139 ANNEX 2: EVALUATION TEAM PROFILES 149 ANNEX 3: CONFLICT OF INTEREST 154 ANNEX 4: EVALUATION RESPONDENTS 165 ANNEX 5: COACH CASE STUDIES AND RUBRIC 168 ANNEX 6: EVALUATION TOOLS 264 ANNEX 7: METHODOLOGY PLAN & STUDY PROTOCOL 515 ANNEX 8: CALCULATION OF PROJECT SCHOOLS 544 ANNEX 9: CALCULATION OF COACHING DOSAGE 547 ANNEX 10: REPORTED RSP PARTICIPATION BY CURRICULUM / SUBJECT ADVISORS 550 ANNEX 11: STATEMENT OF DIFFERENCES 554 ANNEX 12: TEACHER TRAINING MATRIX 555 Contract No: 72067418D00001, Order Number: 72067419F00015 viii TABLES Table 1: School Sampling Methods and Data Sources 9 Table 2: Number of teacher training sessions in 2019 15 Table 3: FPD monitoring data on Average number of teachers per school 15 Table 4: Number of participating teachers in training sessions in 2019 16 Table 5: Grade 1-3 teachers that attended 1, 2, 3, and all training sessions in 2019 18 Table 6: Teachers that received individual coaching in 2019 21 Table 7: Individual coaching sessions delivered in 2019 22 Table 8: Number of teachers involved in SBWs in 2019 24 Table 9: SBWs conducted in 2019 24 Table 10: SMT members involved in the RSP in 2019 27 Table 11: SMT engagements conducted in 2019 28 Table 12: District officials involved in RSP engagements in 2019 28 Table 13: Schools with shortages of LTSM. 30 Table 14: LTSM components as reported by teachers by grade 32 Table 15: LTSM components received as reported by teachers 33 Table 16: Teacher perceptions of coaching elements 37 Table 17: Teacher perceptions of coaching elements by grade 38 Table 18: Coaching case study - summary rubrics (coaches 1 to 7) 40 Table 19: Coaching case study - summary rubrics (coaches 8 to 14) 42 Table 20: Frequency of LTSM Use 48 Table 21: Teacher comments on LTSM usefulness and quality 49 Table 22: Teacher perceptions of usefulness of HL training components 51 Table 23: Teacher perceptions of usefulness of EFAL training components 52 Table 24: Lessons Observed by District, Grade, and Language 58 Table 25: Overall usage of English in EFAL classes 63 Contract No: 72067418D00001, Order Number: 72067419F00015 ix Table 26: Observations where strategies to read words aloud were observed 65 Table 27: Observations where strategies to read words aloud were observed by grade 66 Table 28: Observations where strategies for developing an understanding of text 67 Table 29: Observations where strategies for understanding HL and EFAL text were observed 68 Table 30: Observations where the teaching of vocabulary and spelling were observed 69 Table 31: Observations where the teaching vocabulary and spelling were observed by language 70 Table 32: Observations where the teaching of handwriting was observed 71 Table 33: Observations where the teaching of handwriting by language was observed 71 Table 34: Activities where copying from the board by learners is observed 72 Table 35: Construction of own words and texts by learners 73 Table 36: Comparing sentence writing in Grade 1 and Grade 3 HL 74 Table 37: Comparing sentence writing in Grade 1 EFAL and Grade 3 EFAL 74 Table 38: Teacher reading aloud (by intervention) 77 Table 39: Teacher reading aloud (by grade) 78 Table 40: Children in groups reading aloud (by intervention) 79 Table 41: Individual silent reading (by intervention) 81 Table 42: Self-reported changes in SMT practice due to RSP training 84 Table 43: SMT recommendations for RSP going forward 85 Table 44: SMT report of changes as a result of teacher-coach interactions 86 Table 45: Theory of Change Assumptions against Findings 90 Table 46 SBWs planned versus actual during coach shadowing 99 Table 47: RSP Data Sources 102 Table 48: Frequency versus type of RSP data 107 Table 49: RSP Indicator specification example - Coaching 111 Table 50: List of SMT related indicators in the RSP MEL Plan 111 Table 51: Comparing indicators in two different RSP MEL plan drafts 112 Contract No: 72067418D00001, Order Number: 72067419F00015 x Table 52: RSP output indicators specified in two RSP MEL plan 115 Table 53: RSP impact / outcome indicators in two different RSP MEL plans 115 Table 54: RSP Indicator specification example - SMTs 116 Table 55: RSP Indicator specification example – Lesson observations 117 Table 56: School Support Visits - Extract from RSP Quarter Two indicator reporting table118 Table 57: Meeting of Targets extract from RSP Quarterly indicator reporting table 118 Table 58: School Support Visits extract from RSP Quarter 4 indicator reporting table 119 Table 59: Reported figures for the indicator: Number of School Support Visits 120 Table 60: Coach to Teacher Ratio (Self-Reported) 124 Table 61 Potential Sampling Techniques and Data Sources 532 Table 62 Education researcher profiles 538 Table 63 Illustrative one-day plan for data collection 541 Table 64: Number of RSP schools as per DBE source 545 Table 65: Number of active RSP schools in a merged list of FPD and DBE schools 545 Table 66: Calculation of RSP coaching dosage 549 Table 67: Number of Reference Group meetings attended by interviewed Curriculum Advisors 551 Table 68: Number of Coach Training sessions attended by interviewed Curriculum Advisors 551 Table 69: Reported attendance of Curriculum Advisors at coach training sessions 552 Table 70: Number of teacher training sessions attended by interviewed Curriculum Advisors 552 Table 71: Attendance at teacher training sessions by interviewed Curriculum Advisors 552 Contract No: 72067418D00001, Order Number: 72067419F00015 xi FIGURES Figure 1: Teacher attendance at termly training sessions by grade 18 Figure 2: Comparing FPD reported teacher attendance with FPD monitoring data calculations 18 Figure 3: RSP training workshop attendance per teacher survey results 20 Figure 4: Number of coach visits per school, as reported by school principal 23 Figure 5: SBWs conducted per school as reported by principal 26 Figure 6: LTSM components received, reported by teachers (Teacher Survey) 31 Figure 7: Teacher experience of RSP literacy coach compared to EGRS I coach 36 Figure 8: Teachers who reported they received RSP LTSM at training 46 Figure 9: Teacher perspectives on the training content value (by grade) 50 Figure 10: Teachers who indicated that they received EGRS I coaching support 55 Figure 11: Choral responses by grade 62 Figure 12: Comparing Learner workbooks and amount of written material 75 Figure 13: Comparing Setswana HL and EFAL written work in DBE workbooks 75 Figure 14: Comparing time spent on Setswana and EFAL written work in learner exercise book 76 Figure 15: Groups reading aloud (by grade) 80 Figure 16: Groups reading aloud (by language) 80 Figure 17: Teacher perceptions of SMT support 88 Figure 18: FPD data sources reviewed 109 Figure 19: FPD Presentation on January to September 2019 Implementation Progress 110 Figure 20 Proposed Coach Shadowing Process 540 Contract No: 72067418D00001, Order Number: 72067419F00015 xii ACRONYMS AND ABBREVIATIONS BWE Bernita Williams Evaluation CAPS Curriculum and Assessment Policy Statement CA/SA Curriculum Advisor/Subject Advisor DBE Department of Basic Education Dr Doctor EFAL English First Additional Language EGRS Early Grade Reading Study FPD Foundation for Professional Development GoSA Government of South Africa GR Grade HL Home Language HOD Head of Department ICT Information and Communications Technology ID Identity IDIQ Indefinite Delivery Indefinite Quantity JIT Just in Time Khulisa Khulisa Management Services (Pty) Ltd KII Key Informant Interview LTSM Learning and Teaching Support Material M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning Molteno Molteno Language Institute NECT National Education Collaboration Trust Contract No: 72067418D00001, Order Number: 72067419F00015 xiii OUP Oxford University Press of South Africa PERFORM Practical Education Research For Optimal Reading and Management PERFORMANCE Practical Education Research For Optimal Reading and Management: Analyze, Collaborate, Evaluate PLC Professional Learning Community PSRIP Primary School Reading Improvement Programme RCT Randomized Control Trial RSP Reading Support Project SA-SAMS South Africa-School Administration and Management System SBWs School-based Workshops SLP Structured Learning Program SMT School Management Team SOP Standard Operating Procedure SoW Scope of Work ToC Theory of Change USAID United States Agency for International Development USG United States Government VSO Voluntary Services Overseas WSoE Witwatersrand School of Education WITS University of the Witwatersrand xiv EXECUTIVE SUMMARY Background The Reading Support Project (RSP) is an educational program designed to be implemented in 263 schools1 in Dr. Kenneth Kaunda and Ngaka Modiri Molema districts in the North West Province of South Africa over the period January 1, 2019 – September 30, 2020. The Foundation for Professional Development (FPD) implements the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), and one international partner, Voluntary Services Overseas (VSO), referred to in this report as the FPD Consortium. The project is intended as a scale-up of the South African Department of Basic Education’s (DBE) Early Grade Reading Study (EGRS I2 ) which was an randomized control trial (RCT) with 230 Quintile 1 to 3 schools in the same two districts from 2015 - 2017. The RSP focuses on improving the pedagogical practices of Grade 1 to 3 teachers in how they teach reading in Setswana Home Language (HL) and English First Additional Language (EFAL) with the ultimate outcome to improve the reading skills of Foundation Phase learners. The FPD Consortium uses a cascade training approach to roll out the project. Once a term (four times a year), Consortium members train RSP reading coaches, who then train teachers in district-based geographic clusters. One-day dry runs, whereby coaches have an opportunity to practice their newly acquired knowledge and skills, are a feature of this training approach. The RSP provides teacher training, and Learning and Teaching Support Materials (LTSM), to all schools involved in the project. These materials include lesson plans, flash cards, EFAL and HL lesson plans, readers, posters, flash cards, and other supplementary material provided per term. A random selection of schools additionally receive reading coaches, and another set of randomly selected schools receive both reading coaches and support to the School Management Team (SMT). The underlying theory, in summary, is that through the provision of LTSM, training, coaching, and SMT support, teachers receiving the interventions will improve their practices in the classroom. Purpose For the period 2019 - 2020, USAID commissioned a Design and Implementation Evaluation of the RSP. The purpose of the Design Evaluation3 was to ensure that the design of the RSP, as it evolved, was well understood prior to conducting an Implementation Evaluation. The purpose of the Implementation Evaluation is to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, 1 The analysis presented in Annex 8 indicates that in October 2019 FPD was implementing the RSP in 256 schools. 2 The Early Grade Reading Study ran between 2015 and 2017 in North West Province. Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. 3 A separate Design Evaluation Report is available https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf. Contract No: 72067418D00001, Order Number: 72067419F00015 xv and whether there is fidelity to the intended design in the actual implementation of the program. Evaluation Questions The key Implementation Evaluation questions include: 1. To what extent did the FPD Consortium implement the intended intervention design with fidelity? 2. Is there evidence of RSP uptake by teachers in class? 3. Is there evidence of changed practices based on the SMT support? 4. Was there any major divergence from the assumptions in the Theory of Change? 5. How could the management and implementation structures be improved? 6. How could the M&E systems be improved? 7. What are the unique challenges of delivery at scale? These seven questions are answered in this report, with the purpose of informing programming in 2020. An eighth question - Is the RSP cost efficient? - is addressed in the Summative Report4 . Methods The Implementation Evaluation comprises a mixed methods approach involving the collection, collation, triangulation, and valuing of both qualitative and quantitative primary evaluation data. Khulisa collected data from 60 randomly selected schools including 20 schools that are receiving LTSM and teacher training only (LTSM schools), 20 schools receiving LTSM, teacher training and coaching (Coaching schools), and 20 schools receiving LTSM, teacher training, coaching and SMT support (SMT schools). Primary data collection methods include group discussions with 65 teachers in 13 schools, interviews with 15 Provincial and District officials, 157 SMT members, and 58 Principals. All fourteen reading coaches were shadowed and interviewed for a week, and both head coaches were interviewed. Survey data was collected from 304 Foundation Phase teachers. Classroom observations were conducted in 242 Grade 1 and Grade 3 classrooms and the evaluation assessed 238 Grade 1 and Grade 3 learner workbooks (DBE workbooks and other learner books). A data quality assessment was conducted at 58 schools, classroom library data was collected from 123 classrooms, and classroom LTSM was assessed in 126 classrooms. The evaluation team also interviewed the DBE, FPD consortium members, and other key stakeholders. These multiple sources of data were collated and compared. Khulisa used thematic analysis to analyze group discussion data and interview data, and summary statistics to analyze the quantitative data collected through the evaluation. The data was valued by the evaluation 4 To be published on the DEC in May 2021. Contract No: 72067418D00001, Order Number: 72067419F00015 xvi team in collaboration with experts (investigator triangulation). Data was collected using several different tools and instruments. Key Findings Evaluation Question 1: To what extent did the FPD Consortium implement the intended intervention design with fidelity? According to the RSP monitoring data, the dosage and/or coverage of the intervention is sufficient in terms of number of teacher training sessions delivered, numbers of participating teachers, and number of SMT training workshops delivered. However, the RSP implementation was below the anticipated dose, per the RSP design specifications, in the following areas: full training dosage (only delivered to about half of the anticipated number of teachers); the targeted number of teachers reached with coaching; the number of individual coaching sessions held with teachers, the anticipated number of SBWs held after school, the number of SMT members trained, and the availability of LTSM. Evaluation Question 2: What evidence is there of RSP uptake by teachers in class? There is some evidence of RSP uptake by teachers in the classroom. However, there are some gaps. Less than 45 percent of all schools have reading mats (not a project requirement but indicative of good teaching practice). Most classrooms have HL and EFAL word walls while all schools have a low rate of displaying children’s work. The practices that slow teaching down were seen most often in the SMT and LTSM classrooms. In all three groups, there was still choral reading, but this was lowest in SMT classrooms. There was great variation across the schools with reference to teaching book knowledge and print concepts. In seven of nine criteria linked to teachers explicitly developing learners’ concepts about print and printed material, the SMT classrooms had the highest number of observations of preferred practices. There was little difference in the teaching of book knowledge and print concepts across Grade 1 and 3, although Grade 3 children do not need to be taught about book covers, spines, front and back and reading direction. There was more use of English in the SMT and Coaching classrooms as compared to the LTSM classrooms. In the SMT classrooms, teachers most often modelled the correct term, highlighted English terms, and encouraged the use of English compared to the other groups. The explicit teaching of vocabulary and spelling was also higher in the Coaching and SMT classrooms than in LTSM classrooms. Teaching of handwriting and copying from the board was low overall across the interventions. Learners writing more than five sentences was observed most often in LTSM classrooms. There was little difference in the observation of teachers reading aloud to children and Group Guided Reading was similarly observed in less than 50 percent of schools across all groups. The use of post-reading questions was highest in the SMT classrooms. Contract No: 72067418D00001, Order Number: 72067419F00015 xvii Evaluation Question 3: Is there evidence of changed practices based on the SMT support? The classroom observation findings suggest that there is variable uptake of RSP practices, as evidenced in better practices in the classroom, with SMT classrooms providing more opportunities for language and literacy development than the other groups. Qualitatively, SMT members in almost half the schools (nine out of 20) reported that the SMT training assisted the Head of Departments (HODs) to improve their work in the classrooms and encourage reporting to SA-SAMS on a weekly basis. Evaluation Question 4: Were there any major divergences from the assumptions in the Theory of Change? The Theory of Change (ToC) assumptions surfaced in the Design Evaluation remain relevant and appropriate. In some areas, the assumptions are not being met. This includes in the areas of coach training (coaches were unable to execute dry runs during the last coach training), teacher training (fewer teachers attended than expected), LTSM delivery (not all teachers received LTSM and some teachers did not receive all LTSM), and dosage of coaching support. The coaching shadowing showed that coaches performing good coaching practices were adequately planning and preparing for visits to the schools, and engaged appropriately with the school principal, SMT, and the teachers. Weaker coaches were compliant with reporting but lacked skills and experience. Indications are that the coaches are monitoring and checking teacher implementation against the lesson plans and documenting their observations. Without the school-based workshops (SBWs), the coaching dosage is currently too low to support differential impact. The unmet dosage assumption remains the greatest threat to the RSP ToC. Evaluation Question 5: How could the management and implementation structures be improved? Management and implementation structures must be improved to (1) ensure the proper dosage and coverage is delivered, (2) track implementation and flag issues as they arise, (3) ensure quality of delivery and (4) ensure good reception of the RSP in schools. Evidence from the teacher group discussions, teacher surveys, and classroom observations all found that while each teacher is supposed to receive a full set of lesson plans including all LTSM, not all teachers received all materials. There are not adequate processes in place to check (i) if teachers have received materials in schools where coaching does not take place and (ii) for those teachers who miss training in schools where coaching does not take place, if those teachers have received their materials. The findings also raised a question around whether the LTSM packages contained ALL the RSP LTSM for all grades. The case studies illustrated that the Head Coaches are not providing adequate support to the coaches in the field. Coaches interviews confirmed that nine of the 14 Coaches felt that they did not receive adequate support from their Head Coach. The coach shadowing showed that all 14 coaches were welcome in the schools and were met with cooperation and support from the schools. It was clear that all coaches were Contract No: 72067418D00001, Order Number: 72067419F00015 xviii familiar with the school environment and had been to the schools before. All 14 coaches acknowledged the positive support from the SMT. Evaluation Question 6: How could the M&E systems be improved? The RSP monitoring and evaluation (M&E) systems need to be improved from indicator design, to tool design, to data collection and management, and reporting. The Monitoring, Evaluation, and Learning (MEL) plan needs to be updated to reflect the final iteration of the RSP design. Numerous individual improvements are required within the M&E System to produce reliable data for tracking implementation in 2020. Evaluation Question 7: What are the unique challenges of delivery at scale? Delivering the RSP at scale is affected by a number of factors including the extent of teacher training required, the availability of good coaches, and the logistical arrangements around developing, printing, packaging, and distributing LTSM. Scaling the RSP to three grades and two subjects (Setswana HL and EFAL) simultaneously affected the ability to deliver and monitor RSP interventions. Involving provincial and district officials in the RSP is key to long term sustainability. However, this relationship takes time to build and nurture. Involving provincial and district officials in the process is worthwhile but time consuming both for the officials and for the RSP implementers. Recommendations The evaluation report provides specific recommendations for improvement of the RSP in 2020. These are summarized below: − The RSP M&E system needs to be improved, including improvement of data collection tools, data collection and data entry processes, data collation, and reporting. − RSP LTSM need to be quality reviewed and strategies need to be put in place to address the perceived fast pace of the lesson plans. Processes must be put in place to ensure teachers receive all LTSM. − The roles and responsibilities of head coaches should be better defined and there needs to be more strategic engagement between the head coach and coaches. − Coach skills should be improved through intensive training, support, and role clarification. Weaker coaches require targeted support. Monitoring the coaching process closely is key towards improving the dosage and coverage of coach support to teachers. − The RSP needs to deepen its engagement with district officials to ensure their buy in and cooperation, and should increase reporting to district and provincial officers. − To effectively scale the RSP, subject and curriculum advisors should be trained, roles and boundaries should be clarified, and teachers must understand that the RSP is an integral and systemic part of their teaching. There needs to be more focused engagement between the RSP and the DBE before the project is taken to scale. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |1 1. INTRODUCTION 1.1 SYNOPSIS OF TASK, PURPOSE, AND AUDIENCE This report, produced by Khulisa Management Services (Pty) Ltd. (Khulisa), is submitted under the Design and Implementation Evaluation of the Reading Support Project (Award Number: AID-674-A-16-00007) to the United States Agency for International Development (USAID) under Indefinite Delivery Indefinite Quantity (IDIQ) Contract Number: 72067418D00001, Order Number: 72067419F00015 (Scope of Work in Annex 1). The Reading Support Project (RSP) is an educational program designed to be implemented in 263 schools5 in two educational districts in the North West Province of South Africa (i.e., Dr Kenneth Kaunda and Ngaka Modiri Molema) over the period January 1, 2019 – September 30, 2020. It focuses on improving the pedagogical practices of Grade 1 to 3 teachers in how they teach reading in Setswana Home Language (HL) and English First Additional Language (EFAL) with the ultimate outcome to improve the reading skills of Foundation Phase learners. The Foundation for Professional Development (FPD) implements the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), and one international partner, Voluntary Services Overseas (VSO). The project is intended as a scale￾up of the Department of Basic Education’s Early Grade Reading Study (EGRS I 6 ) which was an RCT with 230 Quintile 1 to 3 schools in the same two districts from 2015 - 2017. For the period 2019 - 2020, USAID commissioned Khulisa to conduct a Design and Implementation Evaluation of the RSP. The purpose of the Design Evaluation7 was to ensure that the design of the RSP, as it evolved, was well understood prior to conducting an Implementation Evaluation. The purpose of the Implementation Evaluation is to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. This is important for a number of reasons: − It is critical towards accurately attributing any impact to the program. If successful, the program can be repeated in another context as the elements of implementation will be known. If unsuccessful, the evaluation will provide some insight into whether this was due to design failure or to implementation failure. 5 The analysis presented in Annex 8 indicates that in October 2019 FPD was implementing the RSP in 256 schools. 6 The Early Grade Reading Study ran between 2015 and 2017 in North West Province. Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. 7 A separate Design Evaluation Report is available https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |2 − Second, this information will assist the RSP implementers to understand any gaps in implementation or identify pockets of success. The data will be used in a formative manner to give the program the best possible chance to lead to impact. The Scope of Work (SoW) for this evaluation, in Annex 1, highlights the importance of the evaluation for learning and accountability, with USAID, the South African Department of Basic Education (DBE), and the RSP Implementers (i.e., The FPD Consortium) being the main target audiences. The North West Provincial Department of Basic Education is another key audience. It is expected that the FPD Consortium will use the 2019 evaluation results to formatively inform programming in 2020, that the DBE will use the results and lessons learned from the evaluation to inform their approach to improving the quality of reading, and that USAID and other donors will use the evaluation results to guide future investments in early grade literacy interventions and evaluations in South Africa. The Implementation Evaluation was carried out by an independent evaluation team (profiles of the team are included in Annex 2 and statements of conflict of interest are included in Annex 3). The team consists of evaluation specialists from Khulisa Management Services and Benita Williams Evaluation (BWE), as well as subject matter experts including two South African consultants with extensive experience working with early grade reading interventions in South Africa. Photo: Khulisa, RSP Classroom Wall Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |3 2. BACKGROUND 2.1 THE INTERVENTION8 To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study in two districts in the North West province of South Africa (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS I evaluated three Setswana HL interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The EGRS I targeted one Grade per year (Grade 1 in 2015, Grade 2 in 2016 and Grade 3 in 2017). The interventions ended in 2017 and EGRS impact results (Taylor, Cilliers, Prinsloo, Fleisch, & Reddy, 2017) showed substantial impacts on learner results through training, coaching, and provision of Learning and Teaching Support Materials (LTSM). In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS I specifically to scale up the coaching intervention, which showed the most impact. In response to this appeal, USAID requested that the FPD Consortium modify their existing cooperative agreement to include the selected EGRS I components. The revised RSP comprises groups of schools receiving interventions to improve the quality of teaching and learning in the classroom and, in turn, improve early grade reading. However, the RSP scales the EGRS I in terms of the number of schools involved, the number of Grades (Grades 1 to 3 simultaneously) and the number of subjects (Setswana Home Language/HL and English First Additional Language/EFAL). The schools were divided into three groups, each receiving a different combination of interventions, which include9 : Group 1: schools receiving LTSM and Teacher Training (referred to as the LTSM Intervention) Group 2: schools receiving Coaching, LTSM and Teacher Training (referred to as the Coaching Intervention) Group 3 (a subset of Group 2): schools receiving Coaching, LTSM, Teacher Training and a School Management Team training program referred to as the SMT Intervention RSP specifically aims to improve subject matter knowledge, promote more effective pedagogic practices, improve in-class time management, increase effective use of LTSM, and foster a school environment that supports teachers’ ability to implement the full curriculum and facilitate successful teaching and learning. In addition, a number of RSP classrooms are receiving 8 The Design Evaluation provides more information on the EGRS I and RSP interventions. See https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf 9 As per final RSP list of project schools provided to Khulisa and last updated in October 2019. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |4 classroom libraries through a joint program of the North West Department of Education and the University of Witwatersrand, School of Education (WSoE), delivered via the University of the Witwatersrand (WITS). 2.2 THE EVALUATION To support the Government of South Africa (GoSA), USAID is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve the reading skills of primary grade learners. Khulisa was awarded the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) IDIQ contract. The contract aims to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from the PERFORM interventions. Under the PERFORMANCE contract, USAID, along with its partner, the DBE, commissioned a Design and Implementation Evaluation of the RSP to evaluate the implementation of RSP as it scales up components of the EGRS I. The evaluation will provide a careful account of how the RSP was adapted to include EGRS I, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e., used to improve the program in 2020) and, in 2020, the results will be summative (i.e., compared to the 2019 results). The evaluation will: 1. Identify any risks to the successful implementation of the RSP. 2. Provide early findings to strengthen the implementation of the RSP. 3. Ascertain whether the allocated time and resources were sufficient to successfully implement the RSP. Design Evaluation Question: Is the RSP theory of change likely to lead to the anticipated results? Implementation Evaluation Questions: 1. To what extent did the FPD Consortium implement the intended intervention design with fidelity? 2. Is there evidence of RSP uptake by teachers in class? 3. Is there evidence of changed practices based on the SMT support? 4. Was there any major divergence from the assumptions in the Theory of Change? 5. How could the management and implementation structures be improved? 6. How could the M&E systems be improved? 7. What are the unique challenges of delivery at scale? 8. Is the RSP cost efficient? EVALUATION QUESTIONS Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |5 4. Determine whether the FPD Consortium is implementing its activities as planned. 5. Determine whether impact-level change can realistically be expected with the RSP theory of change, its design and within the timeframe of the RSP implementation. 6. Explore whether and how the RSP theory of change could be strengthened to produce the intended outcomes. 7. Validate the data collected through the RSP Monitoring and Evaluation (M&E) system. 8. Determine whether teacher training and coaching is leading to any early changes in teaching methods (e.g., Group Guided Reading, phonics, use of chapters in lesson plans to document implementation). 9. Ascertain what dosage of support teachers are receiving from coaches (i.e., Number of visits, time spent per visit). 10. Determine whether coaches are supporting teachers and reporting on their coaching visits. 11. Determine the contribution of volunteering in generating change in school management. 12. Determine whether the School Management Team (SMT) support can be linked to changed support practices in schools. 2.2.1 DESIGN EVALUATION According to the South African National Evaluation Policy Framework (NEPF, 2011), the purpose of a Design Evaluation, during implementation, is to “…assess the quality and implementability of the intervention design in practice.” A Design Evaluation is important towards understanding both how an intervention works and whether it is likely to achieve its outcomes and eventual impact. In the context of the RSP, the purpose of the Design Evaluation was to determine the differences between the design of the RSP and the EGRS, and whether the RSP design could support the impact goals of the EGRS. The final Design Evaluation report was submitted on November 22, 2019. It reviewed relevant project documents and literature (termed a mini literature review), summarized findings from interviews with key stakeholders and from a design workshop led by an expert panel and hosted at the DBE, surfaced the RSP ToC and compared it to the EGRS I ToC. The Design Evaluation found that the RSP ToC is based on evidence about reading acquisition and effective teacher development strategies, and clearly identifies the skills, knowledge, behaviors, and relationships that the RSP would need to lead to impact. However, the quantum of desired change is unspecified. Ultimately, the success of the RSP ToC depends on the uptake of RSP lesson plans, LTSM, and classroom libraries in the classroom. The program assumes that good quality of training and lesson materials are delivered10, and that teachers will have an incentive to try these out in class. For the schools receiving the 10 A review of LTSM was not commissioned as part of the Design Evaluation. Therefore, although a pedagogy expert conducted a cursory review of the materials, the evaluation team cannot comment on whether this assumption has been met. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |6 coaching intervention, the quality of the coaches, the nature and dosage of coaching may influence the degree to which it is able to replicate or exceed the success of the EGRS I. 2.2.2 IMPLEMENTATION EVALUATION An Implementation Evaluation, sometimes called a “process” or “performance” evaluation, is defined in the South African National Evaluation Policy Framework (NEPF, 2011) as follows: “[An Implementation Evaluation] aims to evaluate whether an intervention’s operational mechanisms support achievement or not and understand why. Looks at activities, outputs, and outcomes, use of resources and the causal links. It builds on existing monitoring systems, and is applied during program operation to improve the efficiency and efficacy of operational processes. It also assesses the quality of the indicators.” During implementation, this type of evaluation is often used “…to see whether a program or policy’s operational mechanisms support achievement or not, and understand why” (NEPF, 2011). It can be used to complement an impact evaluation by determining “…what quality and quantity of services the program provides so that this information can be integrated with findings on what impact those services have” (Rossi et al, 2004, p.177). USAID’s Evaluation Policy (USAID, 2012) calls an Implementation Evaluation a “Performance Evaluation”, and describes it as a type of evaluation that, “…focus[es] on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual.” USAID’s requirements for a Performance Evaluation are that it could be timed in such a way that the results are made available prior to key decision-making points, such as developing new strategies, designing new projects, and making decisions about new procurements. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |7 3. METHODOLOGY 3.1 INTRODUCTION The approved methodology comprises a mixed methods approach involving the collection, collation, triangulation, and valuing of both qualitative and quantitative primary evaluation data. See Annex 7 for the USAID approved Methodology and Study Plan. Fourteen coach case studies (Annex 5) provide a deeper dive into the practices, profiles, and performance of all RSP coaches. Other primary data collection methods used include group discussions with teachers in 13 schools, interviews with district officials, interviews with head coaches, interviews with the DBE, and interviews with FPD consortium members and other key stakeholders conducted during the Design Evaluation phase as well as the Implementation Evaluation. Multiple sources of data were collated and compared. Khulisa used thematic analysis to analyze group discussion data and interview data, and summary statistics to analyze the quantitative data collected through the evaluation. The data was valued by the evaluation team in collaboration with experts11 (investigator triangulation). 3.2 RESPONDENTS The stakeholders who participated in the evaluation (from the Design Report to the Formative Evaluation Report) are summarized in Annex 4. Overall, there were 630 respondents representing the DBE, USAID, the North West Provincial Department of Education (including district officials, provincial officials, and Curriculum Advisor/Subject Advisors – CA/SAs), the FPD Consortium, the coaches, head coaches, teachers, Principals, and SMT members. 3.3 SAMPLING As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling up the RSP. The purpose of the implementation evaluation is to determine the fidelity of implementation. Unlike the quantitative evaluation of the EGRS I I, the sample did not assess schools against extant characteristics such as whether they performed well or poorly or whether they had large learning gains in the EGRS I. As the RSP program is the larger scale up of the EGRS I, the sampling frame also included schools that did not form part of the original sample of EGRS I schools. 11 A pedagogy expert and a coaching expert. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |8 For school-level sampling, the actual sampling strategy was defined in consultation with the DBE and USAID, to draw a sample of 60 schools12 . In total, 16 schools were selected from Dr Kenneth Kaunda District and 44 from Ngaka Modiri Molema district to reflect the distribution of schools per district. The sample of 60 schools is comprised as follows: • Ten schools receiving coaching, who previously were EGRS I control schools. • Ten schools receiving coaching, who previously were EGRS I coaching schools. • Ten schools receiving coaching & SMT training, who previously were EGRS I control schools. • Ten schools receiving coaching & SMT training, who previously were EGRS I coaching schools. • Twenty schools currently receiving LTSM and teacher training only, who were previously EGRS I parental involvement schools). It was also decided that these schools would be selected randomly from amongst all eligible schools matching the above criteria. The sample was drawn by the Director for Research Coordination, Monitoring, and Evaluation at the DBE on August 13, 2019 using a statistical software called STATA. The list of 251 schools participating in RSP as of August 13, 2019 was used as a starting point. A number of different “strata” were classified in order to group all the schools into the five categories of school listed above, as well as to identify schools which did not fit into any of those categories and therefore would be ineligible for the sample. Schools ineligible for the sample included the 34 schools who had previously been part of the EGRS I teacher training intervention, which was not scaled up in RSP, and the 85 RSP schools which were not previously part of EGRS I and therefore have no relevant comparison group. Each school was then assigned a random number using STATA’s “runiform” command. Next, the list of schools was sorted in order first of strata and then in terms of the random number. Within each strata, therefore, the order of schools is strictly random. The first ten schools within each of the above strata (except for the 5th strata where the first 20 schools were needed) were then selected for inclusion in the implementation evaluation sample. Finally, the last four schools within each of the five strata were also selected to serve as schools to be used by Khulisa for simulation and piloting of the survey instruments. These 20 schools were thus also randomly selected. 12 In addition to the 60 sample schools, the DBE drew a sample of 20 schools within which to conduct pilot testing of tools and training simulations. From this sample of 20 schools, six schools were selected for pilot testing of tools and two for training simulations. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |9 A number of schools who received classroom libraries were included in the sample, even though this was not a criterion upon which the sample was selected. The sample is too small to be representative or to allow generalization, but should be big enough to allow for identification of consistent patterns within and between the groups. Within each school, a list of sampling techniques is described below. Table 1: School Sampling Methods and Data Sources Data Source Sampling Method No. Respondents Coaching case studies, assessment and interviews Mid-level researchers shadowed the population of coaches in the RSP. The aim of these interviews was to gain qualitative insight into a “Week in the life of a coach.” 14 Coaches shadowed and interviewed Group discussion with teachers Mid-level researchers identified 65 Foundation Phase teachers to participate in a group discussion at 13 sample schools (range from two to nine teachers in an interview). In schools with very few Foundation Phase teachers, all teachers were included in the group discussions. 65 Foundation Phase teachers participated in group discussion Surveys with teachers Mid-level researchers administered surveys to all Foundation Phase teachers in each sample school. 304 teachers (of which 105 were Grade 1 teachers, 103 were Grade 2 teachers, and 96 were Grade 3 teachers) participated in Teacher Survey. Key Informant Interviews with school principals and SMT members Mid-level researchers conducted interviews with all sample school principals and SMT members, where they were available. 157 SMT members participated in Key Informant Interviews. 58 Principals participated in school Data Quality Assessment interview. Key Informant Interviews with district officials, and other identified stakeholders The evaluation team used purposive sampling to engage key informants as the aim of these interviews was to gain qualitative insight into key informants’ perspective on the project and sampling for proportionality was not the main concern. 15 Provincial and District officials participated in Key Informant Interviews. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |10 Data Source Sampling Method No. Respondents Classroom observations Teaching practices were observed in one Grade 1 HL class and one Grade 1 EFAL class, as well as one Grade 3 HL class and one Grade 3 EFAL class per sample school (per consultation with the DBE and USAID). Classes within those Grades/subjects were randomly sampled. Classroom observations were conducted in 242 classrooms, of which 119 were Grade 1 classrooms and 120 were Grade 3 classrooms. Three classroom observations did not capture the grade correctly and were therefore excluded from all grade￾disaggregated analyses. Workbook and Learner Exercise Book analysis The learner workbook assessment took place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks were drawn were the same classes in which the Classroom Observations were conducted. 238 learner workbooks (DBE workbooks and other learner books) were assessed, of which 118 are Grade 1 and 120 are Grade 3 learner workbooks. Data quality assessment (DQA) School-level data collected through the FPD monitoring system was cross￾checked with the data available at the school. DQA data collected from 58 schools Classroom library assessment and classroom walk-through assessments Classroom-level data on class libraries and LTSM was collected from the same classrooms that participated in the classroom observations. Classroom library data collected from 123 classrooms across 58 schools Classroom LTSM assessed in 126 classrooms (62 Grade 1 classrooms and 64 Grade 3 classrooms) across 57 schools Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |11 3.4 METHODOLOGICAL LIMITATIONS There are a few limitations in the methodology, which stem from: Khulisa proposed including a material review in the evaluation but this was removed from the evaluation scope of work as the DBE and USAID felt it would be addressed as part of an ongoing larger literacy materials review outside of this evaluation. Unfortunately, this material review by another entity has not yet been completed, as was expected, and therefore leaves a gap in the evaluation. The Implementation Evaluation was based on a randomly selected sample of 60 schools. Only the FPD Consortium had any data at the population level (i.e., monitoring data). At the time of writing the report, the final list of schools receiving interventions was not well defined. The DBE and FPD Consortium presented the evaluation team with slightly different lists of schools. This is explored in Annex 8: Calculation of Project Schools. The evaluators used 263 as the base number of schools in all of the calculations presented in this report as this is what the May 2019 design specification detailed. FPD data as at October 2019 indicated that the project was active in 256 schools). The evaluation reviewed two different draft versions of the MEL plan and indicators. A final version of the MEL plan was not yet developed pending the finalization of contracting between USAID and FPD. These limitations have affected the evaluation in the following ways: − A basic underlying assumption in the RSP Theory of Change that the LTSM are of good quality and are sufficiently well organized for teachers to implement. The evaluation team is unable to test this assumption given the removal of the expert materials review from the scope of the evaluation. So that the team was able to understand the content of the materials, the evaluation team asked the pedagogy expert to conduct a cursory review of the EFAL materials and this expert, in collaboration with a Setswana junior researcher from BWEC, conducted a basic review of the Setswana materials. − Schools were selected for the Implementation Evaluation using random selection protocols. This enhances the generalizability of the findings. However, given that the Implementation Evaluation results are sample results, not population results, the evaluation team mainly had to rely on the secondary data collected through FPD’s monitoring systems to conduct calculations of dosage and coverage. − The RSP schools changed over time (i.e., attrition of schools / addition of new schools over the course of the year) and are not consistently identified between the FPD Consortium and DBE. All calculations with schools as the denominator therefore were based on assumptions of the numbers of schools receiving RSP interventions. − Working with different versions of the RSP MEL plan meant that the evaluation team had to make many assumptions. These findings are thus indicative rather than comprehensive. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |12 3.5 BIAS AND CONFLICT OF INTEREST The evaluation team was able to work freely and without interference and there were no known or unresolvable differences of opinion or conflicts of interest either within the evaluation team or between the evaluation team, the DBE and USAID. For the qualitative data collected through the evaluation, the evaluation team attempted to ensure that a range of stakeholders with different perspectives were interviewed, that the team provided a space for alternate views, and that the data was triangulated in an open and transparent manner (both data triangulation and investigator triangulation). Photo: Khulisa, Classroom Library Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |13 4. FINDINGS The following section discusses the Formative Evaluation findings with relevant evidence for each of the evaluation questions. Evidence is drawn from multiple sources (see Methodology Plan and Study Protocol in Annex 7). 4.1 TO WHAT EXTENT DID THE FPD CONSORTIUM IMPLEMENT THE INTENDED INTERVENTION DESIGN WITH FIDELITY? According to the RSP monitoring data, the dosage and/or coverage of the intervention is sufficient in terms of number of teacher training sessions delivered, numbers of participating teachers, and number of SMT training workshops delivered. However, the RSP implementation was below the anticipated dose, per the RSP design specifications, in the following areas: full training dosage (only delivered to about half of the anticipated number of teachers); the targeted number of teachers reached with coaching; the number of individual coaching sessions held with teachers, the anticipated number of SBWs held after school, the number of SMT members trained, and the availability of LTSM.. The adherence of the RSP to its intended design was variable. Dosage aside, quality of implementation was generally fair to good (across data sources) in certain areas such as teacher training and coaching support. District and provincial officials, teachers, and SMT members were responsive to RSP interventions. However, in a few cases the program was not well received (e.g., in particular districts or among particular teachers). The RSP was not differentiated in design, nor tailored to the baseline of reading performance in each school. However, coaching case studies revealed that at least two coaches differentiate support to schools and teachers of their own accord. Overall, the fidelity of implementation requires improvement (quantity and quality) in order to deliver the expected results per the Theory of Change. Also, many issues with data quality transpired (such as teachers counted as being trained that are not part of the intervention or comparison schools). There is also a deeper question of whether the impact theory is achievable particularly when it comes to the amount of intervention that can possibly be delivered within the limits of the time and resources available to the implementers. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: − RSP monitoring data. − RSP Standard Operating Procedures. − RSP Design Document. − Design Evaluation (reflecting against ToC). − Key Informant Interviews (KII) (District and Provincial staff, FPD Consortium, teachers, coaches, and head coaches). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |14 − Classroom observations in 119 Grade 1 and 120 Grade 3 classrooms (Grade not clear in three observations). − Teacher training observation. − Coach training observation. − Coaching case studies with 14 coaches. − Monthly plans for the month of coach observations submitted by each coach. − School Data Quality Assessment in 58 schools. − Classroom library assessments in 58 schools. − Classroom materials assessment in 57 schools − Learner workbook assessments with 118 Grade 1 and 120 Grade 3 learner workbooks. − Teacher surveys with 105 Grade 1, 103 Grade 2, and 96 Grade 3 teachers. Implementation fidelity is usually assessed along a number of criteria (Buckley et al., 2017) including: 1. Dosage (the frequency of program delivery) and coverage (number of target participants reached). 2. Adherence (whether program components were delivered as prescribed). 3. Quality of delivery (how well the program material was implemented). 4. Participant responsiveness (how well the instruction was received or perceived). 5. Program differentiation (the degree to which the program was differentiated according to type of intervention actually delivered, status of reading in the school, rural versus urban schools, etc.) Dane and Schneider (1998, cited in Century et al., 2010) recommend that all of these criteria be assessed to understand the integrity of a program. This report explores these criteria using the RSP monitoring findings as a base, triangulates these data with other sources of information, and comments on the quality of the FPD monitoring data as part of a general assessment of data quality. 4.1.1 DOSAGE AND COVERAGE Dosage is a measure of the amount of time spent on program components (e.g., on teacher training, coaching, etc.), or the amount of intervention received by program participants, while coverage is a measure of whether all the targeted participants are reached with an intervention (Carroll et al., 2007; Century et al., 2010). This section disaggregates dosage and coverage by component of the RSP including teacher training, individual teacher coaching, SBWs, SMT training and support, district engagements, and LTSM delivery. TEACHER TRAINING Number of teacher training sessions According to FPD monitoring data, the FPD consortium delivered the anticipated number of training sessions. The planned number of teacher training sessions took place in 2019, but Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |15 the two-day training sessions were held in Terms 2 and 3, instead of in Terms 1 and 3. No data on the actual duration of the training sessions (i.e., number of hours) were available. Table 2: Number of teacher training sessions in 2019 Design Specification Planned teacher training sessions Monitoring Data Actual teacher training sessions reported in FPD monitoring data for period Feb to Sept 2019 Reported Data Reported number of training sessions as per FPD Indicator reporting table to USAID Term 1 - Two days Term 2 - One day Term 3 - Two days Term 4 - One day = Six days in 2019 Feb – One day Apr – Two days Jun – Two days Sep – One day = Six days in 2019 This is not reported on as a separate indicator Data notes: Actual duration of sessions is not reported. Design specification versus FPD monitoring data versus FPD reported data. Number of participating teachers in training sessions According to FPD monitoring data, there is more than an average of six Foundation Phase teachers per school. The design specification assumes that each school has approximately six Foundation Phase teachers to train. This assumption does not seem to hold. Using the full database of teachers who attended at least one FPD training session, the average number of teachers per school who attended the training is closer to seven per school. Table 3: FPD monitoring data on Average number of teachers per school Grade 1 Grade 2 Grade 3 Total Grade 1 to 3 Total number of schools linked to teachers attending the training sessions 278 270 283 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |16 Grade 1 Grade 2 Grade 3 Total Grade 1 to 3 Total number of teachers who attended training 661 597 624 1882 Average number of participating teachers per school 2.38 2.21 2.20 6.79 According to FPD monitoring data, the number of trained teachers is roughly on target if an assumption is made that each school had six Foundation Phase teachers. As above, although the sample schools show a lower number, the actual number of teachers per school may be closer to seven in the population of schools. FPD trained between 1,369 teachers and 1,711 teachers in each training event. The number of teachers trained seems to be roughly on target. Small discrepancies between the monitoring data and the reported data were found – this can be attributed to data quality control which continues after quarterly reporting deadlines to USAID13 . Table 4: Number of participating teachers in training sessions in 2019 Design Specification Targeted number of teachers Monitoring data Actual number of teachers that attended training (Feb – Sep 2019) Reported Data Reported number of teachers trained on Scripted lesson pans and LTSM (Feb – Sep 2019) as per FPD Indicator reporting table to USAID 263 schools with six teachers each = 1,578 Term 1 = 1,545 Term 2 = 1,711 Term 3 = 1,377 Term 4 = 1,369 Term 1 = 1,442 Term 2 = 1,747 Term 3 = 1,369 13 This figure is reported in the indicator table. The notes that accompany the table explain that the 1,442 figure excludes teachers from other grades and non-project schools. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |17 Design Specification Targeted number of teachers Monitoring data Actual number of teachers that attended training (Feb – Sep 2019) Reported Data Reported number of teachers trained on Scripted lesson pans and LTSM (Feb – Sep 2019) as per FPD Indicator reporting table to USAID Individual teachers that attended all four training sessions = 826 Data notes: Targeted number of teachers: The design specification assumes that six teachers per school will be trained. However, the Term 2 data indicate that more than six teachers per school were trained. The number of teachers trained (as per FPD monitoring data) includes participants from non-project schools, as well as individual teachers that attended more than one grade’s training, An accurate calculation of the number of project teachers trained was not possible with the monitoring data available at the time of writing this report. A significant variance in teacher attendance between training events is likely to have a significant resource implication. Such a variance requires further investigation. Reported data: There is a discrepancy between the number of teachers trained per the monitoring data and the reported figures in the FPD indicator table, which is appended to FPD’s quarterly reports to USAID. Monitoring data is updated with late submissions even after reporting deadlines and additional data quality control is conducted after the reporting deadline. Teacher attendance at training in Terms 3 and 4 was lower than in Terms 1 and 2. No systematic differences were found in the attendance of teachers from different grades. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |18 Figure 1: Teacher attendance at termly training sessions by grade Figure 2: Comparing FPD reported teacher attendance with FPD monitoring data calculations According to the FPD monitoring data, the full training dosage was only delivered to about half of the teachers. Only 826 out of the total number of teachers that attended any training session (1,882), attended all four training sessions, meaning that the number of teachers that received the full training program is far lower than planned. Approximately 1,425 of teachers attended at least three training sessions. Table 5: Grade 1-3 teachers that attended 1, 2, 3, and all training sessions in 2019 Number of training sessions attended: Grade 1 Grade 2 Grade 3 Total Grade 1 to 3 Cumulative Total Number of teachers that attended all training sessions 283 264 279 826 826 521 599 463 450 283 466 530 454 447 264 558 582 460 472 279 0 200 400 600 800 Term 1 Term 2 Term 3 Term 4 All sessions Attendance of Teacher Training Sessions by Grade Grade 3 Grade 2 Grade 1 1545 1711 1377 1369 826 1442 1747 1369 0 500 1000 1500 2000 Term 1 Term 2 Term 3 Term 4 All sessions Attendance of Teacher Training Sessions (Total Grade 1 - 3 teachers) FPD Reporting Total Calculated Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |19 Number of training sessions attended: Grade 1 Grade 2 Grade 3 Total Grade 1 to 3 Cumulative Total Number of teachers that attended three training sessions 204 195 200 599 1425 Number of teachers that attended two training sessions 115 102 108 325 1750 Number of teachers that attended only one training session 59 36 37 132 1882 Total number of teachers training in the teacher monitoring data set 661 597 624 1882 Data note: There is not a complete teacher list for all enrolled schools, which indicates the status of each teacher. However, the FPD monitoring results do not reflect the findings of the Teacher Survey, in which more than 80 percent of teachers reported attending each of the termly training sessions (see Figure 3 below). This may be a factor of the composition of the sample rather than an indication of a difference between the actual number of SBWs attended by teachers and the monitoring reported data. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |20 Figure 3: RSP training workshop attendance per teacher survey results The teacher survey also asked “If you missed one or more of the training workshops, what was the reason?” Of the 66 teachers who responded to this question, 30 (45.4%) noted that they did not attend due to being sick or for personal reasons, 13 noted that they had moved to a new grade or a new school (19.6%), 18 (27.2%) noted that they did not attend due to miscommunication about the training, and 5 (7.5%) cited other reasons. TEACHER INDIVIDUAL COACHING Number of participating teachers in individual coaching / lesson observation sessions According to FPD monitoring data, far fewer than the targeted number of teachers are reached with coaching. The number of teachers that were individually supported at least once in 2019, are 668. This is lower than the target of 780 teachers. Given that there are likely more than an average of six Foundation Phase teachers per school, the coverage of the coaching support is inadequate. 83.60% 90.50% 91.40% 89.50% Term 1 Term 2 Term 3 Term 4 % of respondents RSP Training Workshops Attended Teacher Survey Respondents Yes No NA N=304 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |21 Table 6: Teachers that received individual coaching in 2019 Design Specification Targeted number of teachers Monitoring Data Actual number of teachers supported at least once (with Individual coaching or School Based Visit) in period Feb to Aug 2019 Reported Data Number of teachers supported as per FPD Indicator reporting table to USAID Six Foundation phase teachers per school x 135 Coaching schools14 = 810 teachers GR 1 = 235 GR 2 = 226 GR 3 = 227 Total = 668 Not reported Data Notes: The data on the number of supported teachers is entered from monitoring tools submitted by coaches and head coaches. It is likely that the figures presented here is a gross underestimate of the actual support visits, since the FPD M&E office is unable to enforce the submission of this data. Number of individual coaching sessions Various data sources show that far fewer individual coaching sessions took place than what was planned. The FPD monitoring data on individual coaching sessions (also referred to as lesson observations) is incomplete and it is likely that the number of sessions that took place is significantly underreported. According to the FPD monitoring data, teachers received on average only one individual coaching visit in 2019, and not even 20 percent of the expected individual coaching sessions were delivered. The figures reported by FPD in the indicator reporting table that is submitted to USAID on a quarterly basis, also indicates a very low number of individual coaching sessions (referred to as lesson observations). Table 7 below summarizes this data. 14 Figures as per FPD database, November 2019. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |22 Table 7: Individual coaching sessions delivered in 2019 Design Specification Planned number of individual coaching sessions Monitoring Data Number of individual coaching15 sessions (Feb to Aug 2019) Reported Data Number of individual coaching sessions as per FPD Indicator reporting table to USAID Four to six sessions per teacher in 2019 By September 2019 an average of 3.2 to 4.8 individual coaching sessions per teacher would be expected (Refer to Annex 9 for calculations) Average number of individual coaching sessions per teacher GR 1 = 1.0 GR 2 = 1.1 GR 3 = 1.0 Not Reported Expected number of individual coaching sessions in 2019- 4,032 (Refer to Annex 9 for calculations) Number of individual coaching sessions conducted: GR 1 = 388 GR 2 = 375 GR 3 =364 Total (Term 1 to Term 3) = 1,127 Number of individual coaching sessions conducted (Indicator is: Number of lesson observations conducted, including teacher feedback) Term 1 = 116 Term 2 = 636 Term 3 = 1,213 Total (Term 1 to Term 3) = 1,965 Data notes: Actual number of individual coaching session is likely an underestimate. The data is captured from monitoring tools submitted by coaches and head coaches, and it is reported that the monitoring tools are sometimes received late by the FPD M&E team. Reported data: There is a discrepancy between number of sessions as per the monitoring data (1,127) and the reported figures in FPD indicator reporting table (1,965) which is appended to FPD’s quarterly reports to USAID. An interview with FPD staff indicates that they use a combination of coaches’ reports and captured lesson observation forms to report. 15 Also referred to as lesson observations Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |23 In our evaluation sample, the number of reported visits per school varied significantly, but only one fifth received at least one visit per month from a coach. Survey data from school principals on the number of visits received from a coach between February and October 2019 (this could include visits for drop off of materials, individual coaching sessions, or for SBWs) in a sample of 38 Coaching schools, indicates that only eight out of 38 schools (21%) reported receiving at minimum one coaching visit per month i.e., at least nine visits in the period February to October 2019. About 20 of the 38 (52%) schools reported receiving five to eight visits and ten of the 38 schools (26%) reported receiving fewer than five visits for the period. Figure 4: Number of coach visits per school, as reported by school principal SCHOOL-BASED WORKSHOPS Number of participating teachers in SBWs FPD Monitoring data about the number of participants in SBWs show far fewer teachers were reached in SBWs than anticipated. 0 0 2 3 5 5 4 7 4 3 1 2 2 0 1 2 3 4 5 6 7 8 0 Visits 1 Visit 2 Visits 3 Visits 4 Visits 5 Visits 6 Visits 7 Visits 8 Visits 9 Visits 11 Visits12 Visits20 Visits Number of Schools Number of coach visits reported by school principal Reported number of coach visits conducted per school - school survey data (n=38) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |24 Table 8: Number of teachers involved in SBWs in 2019 Design Specification Targeted number of teachers supported in SBWs Monitoring Data Actual number of teachers supported at least once in period Feb to Aug 2019 Reported Data Number of participating teachers in SBWs as per FPD Indicator reporting table to USAID Six Foundation phase teachers per school x 135 Coaching schools = 810 teachers GR 1 = 119 GR 2 = 98 GR 3 = 94 Total = 311 Not Reported Data notes: Actual number of teachers that participate in SBWs is based on attendance registers submitted by coaches. Number of school-based workshops FPD Monitoring data on the number of SBWs is unreliable. The FPD monitoring data on SBWs is incomplete and it is likely that the number of sessions that took place is significantly underreported in the monitoring data. The data indicate that an average of 0.3 to 0.4 SBWs were conducted in the period Feb to August 2019. To meet the target set in the RSP design specifications, a minimum of seven visits per school would have been expected. Table 9: SBWs conducted in 2019 Design Specification Planned number of SBWs as per design specification Monitoring Data Number of SBWs (Feb to Aug 2019) Reported Data Number of SBWs as per FPD Indicator reporting table to USAID 8-12 workshops per school in 2019. By September, an average of 6.4 to 9.6 workshops per school would have been expected. (Refer to Annex 9 for calculations) Average number of SBWs per teacher GR 1 = 0.4 GR 2 = 0.3 GR 3 = 0.3 Not Reported Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |25 Design Specification Planned number of SBWs as per design specification Monitoring Data Number of SBWs (Feb to Aug 2019) Reported Data Number of SBWs as per FPD Indicator reporting table to USAID Total number of SBWs in 2019: 1, 680 By September 2019, 1, 344 of these workshops would have been expected. (Refer to Annex 9 for calculations) Number of SBWs GR 1 = 139 GR 2 = 114 GR 3 = 106 Total (Term 1-Term 3) = 359 Number of school support visits (possibly includes other aspects of support not only SBW’s ) Term 1 = 250 Term 2 = 406 Term 3 = 545 Total (Term 1 to Term 3)= 1,201 % of planned versus actual support visits Term 1 = 60% Term 2 = 87% Term 3 = 93% Data notes: The number of SBWs reported in the reported data includes other school visits as well e.g., delivery of material. Despite gaps in the FPD monitoring data, it is likely that far fewer SBWs took place than planned. Data reported in FPD indicator table indicates that 1,201 school based support visits (which could also include other tasks like material delivery) took place in Terms 1 to 3 of 2019 – This is less than what the design specifies. Data collected at school confirm that SBWs are an exception rather than a rule. In our evaluation sample, 17 out of the 38 schools (44%) reported that no SBWs took place in the period February to October 2019. Frequent SBWs seem to be only occurring in about five of the 38 schools for which this data was collected. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |26 Data collected during coach shadowing observations also confirms that SBWs are not the norm. Of the 43 SBWs planned for the four days of coach observation, only 19 took place. Figures reported by FPD in quarterly reports to USAID are open to ambiguous interpretation. FPD reported that 60 percent of the planned school support visits took place in Term 1, 87 percent in Term 2 and 93 percent in Term 3. The FPD indicator does not report on SBWs in Coaching schools only. More discussion on the limitations of the FPD reported data is contained in section 4.6.3 MEL plan and 4.6.4 FPD reporting. SMT TRAINING AND SUPPORT Number of participating SMT members FPD monitoring data indicate that fewer SMT members are trained than planned – This is because schools do not always send a principal and an HOD to workshops. Only 66 people (out of a targeted 130) (50.77%) attended all three training sessions. FPD monitoring data indicates that the targeted number of tablets were distributed. In the SMT interviews, 18 of the 20 SMTs indicated they had received the tablets. Although not a project requirement, it is notable that in six schools these tablets were not registered on the school asset register. FPD monitoring data indicate that only a third of the schools have so far received an SMT support visit from a VSO volunteer. 17 9 4 3 1 1 1 1 1 0 2 4 6 8 10 12 14 16 18 0 Workshops 1 Workshops 2 Workshops 3 Workshops 4 Workshops 5 Workshops 6 Workshops 7 Workshops 8 Workshops Number of Schools Number of coach visits reported by principal Reported number of school-based workshops conducted per school - school survey data (n=38) Figure 5: SBWs conducted per school as reported by principal Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |27 Table 10: SMT members involved in the RSP in 2019 Design Specification Targeted number of SMT members Monitoring Data Actual Number of SMT members trained / supported for period Feb to Sept 2019 Reported Data Number of SMT members reached as per FPD Indicator reporting table to USAID 130 SMT members from 65 schools participate in training SMT members trained Launch = 45 (34.6%) Workshop 1 = 110 (84.6%) Workshop 2 = 115 (88.5%) Workshop 3 = 85 (65%) Persons who attended all three workshops = 66 (50.8%) Number of SMT’s attending PLCs Term 1 =0 Term 2 = 111 Term 3 = • Session 1 - 112 • Session 2 - 8916 130 SMT members from 65 schools each receive a tablet computer Tablets distributed = 129 Not Reported 65 schools receive support visit from a VSO volunteer 18 schools visited (27 school visits in total, some schools received more than one visit) Not Reported Small discrepancies between the monitoring data and the FPD indicator reporting table were found. The reported data includes other officials that attended training, while the indicator is intended to only report on SMT members that attended. Number of SMT training and support sessions FPD monitoring data indicates that SMT training seems to be on track, but support visits are lagging. Three of the planned SMT training sessions took place in 2019, so the training is 16 The comments section of the indicator reporting table lists 85 participants. FPD explained that the figure of 89 also included district officials that attended the training. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |28 on track to being completed as planned. Only a third of the schools received a SMT support visit from a VSO volunteer. Table 11: SMT engagements conducted in 2019 Design Specification Planned number of SMT training / support sessions Monitoring Data Number of SMT training/ support sessions (Feb to Sep 2019) Reported Data Number of SMT training / support sessions as per FPD Indicator reporting table to USAID Five to eight SMT training sessions in two years Three workshops (May, Jul and Sep) Three PLCs One visit per school = 65 visits in total 18 visits Not Reported DISTRICT ENGAGEMENTS Number of participating District Officials FPD Monitoring Data about the Curriculum advisors’ involvement in the program was incomplete. Fewer district officials are participating in the RSP engagements than planned. Table 12: District officials involved in RSP engagements in 2019 Design Specification Targeted number of District Officials Monitoring Data Actual number of District Officials involved in 2019 Reported Data Number of District Officials involved in 2019 as per FPD Indicator reporting table to USAID 16 Curriculum Advisors attend Coach training sessions Data not available Not Reported 16 Curriculum Advisors participate in Reference group meetings Meeting 1 = no data Meeting 2 = 10 Number (and %) of Curriculum Advisors who completed professional development activities Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |29 Design Specification Targeted number of District Officials Monitoring Data Actual number of District Officials involved in 2019 Reported Data Number of District Officials involved in 2019 as per FPD Indicator reporting table to USAID Meeting 4 = 10 Meeting 4 = 8 Term 1 = 10 Term 2 = 11 Term 3 = 8 16 Curriculum Advisors Trained / involved in PLC N/A – Planned for 2020 only N/A 16 Curriculum Advisors receive tablet computers No Data Not Reported. The DBE and FPD note that tablet provisioning to Curriculum Advisors did not take place in 2019. These tablets were planned for distribution only during the Curriculum Advisor training in 2020. Data notes: Small discrepancies between the monitoring data and the FPD indicator reporting table were found. This is the result of continued data cleaning that took place after the report deadline. LTSM DELIVERY According to FPD monitoring data, lesson plan, and LTSM availability in RSP schools is not on target. Only 70 percent of the schools reported having received all lesson plans and all LTSM. This is based on a reconciliation conducted by coaches in October of 2019 and was done at the level of school, rather than at the level of teacher. The reconciliation was necessary because the data collected during the delivery of materials were inadequate to track material coverage. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |30 Table 13: Schools with shortages of LTSM. RSP Intervention Target number of schools Number of schools - No data available Number of schools - All LTSM received Number of schools with Shortage of LTSM LTSM and Training Schools 133 11 (8%) 99 (74%) 23 (17%) Coaching Schools 130 9 (7%) 84 (65%) 37 (28%) Total 263 20 (8%) 183 (70%) 60 (23%) According to survey data collected by the evaluation team from 302 teachers, more than 90 percent of teachers reported having received lesson plans and flashcards, but only about 62 or 63 percent received the Big Books and Graded Readers. There is a possibility that some of this data is not entirely reliable. For example, if the teachers received scripted lesson plans (which 93 percent of them indicated that they had), then they also would have received the curriculum tracker/teacher monitoring and support tool as it is in the same book (although only 75 percent of teachers indicated they had this), so 18 percent of teachers made a reporting error here. The number of LTSM received, as reported by teachers, between January and October 2019 is listed in the Figures and Tables below. Figure 6 disaggregates this data by district, Table 14 by Grade, and Table 15 by intervention group. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |31 Figure 6: LTSM components received, reported by teachers (Teacher Survey) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |32 Table 14: LTSM components as reported by teachers by grade LTSM reported as received by teachers in the teacher survey (by grade) Grade 1 Grade 2 Grade 3 Scripted Lesson Plans (Termly Lesson Plans) HL 94.2% 95.1% 91.7% Scripted Lesson Plans (Termly Lesson Plans) EFAL 98.1% 98.1% 89.6% Theme Vocabulary Flashcards (HL) 94.2% 93.1% 89.6% Theme Vocabulary Flashcards (EFAL) 99.0% 94.1% 93.8% Sight Words Flashcards (HL) 88.5% 89.2% 88.5% Sight Words Flashcards (EFAL) 89.4% 79.4% 90.6% RSP Big Books (EFAL) 83.7% 75.5% 24.0% RSP Graded Readers (EFAL) 32.7% 79.4% 78.1% Handwriting Poster 63.5% 58.8% 68.8% Posters - Listening and Speaking 69.2% 68.6% 79.2% Assessment Record Book 48.1% 50.0% 54.2% Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term 76.9% 76.5% 71.9% Teacher Management and Storage Files 10.6% 16.7% 11.5% USB Stick with Songs Used in Lesson Plans 1.9% 3.9% 1.0% Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |33 Table 15: LTSM components received as reported by teachers LTSM components reported as received by teachers in the Teacher Survey (by intervention group) Coaching Coaching & SMT LTSM only Total Scripted Lesson Plans (Termly Lesson Plans) HL 94.9% 90.6% 95.9% 93.7% Scripted Lesson Plans (Termly Lesson Plans) EFAL 94.9% 91.5% 93.9% 93.4% Theme Vocabulary Flashcards (HL) 98% 85.8% 93.9% 92.4% Theme Vocabulary Flashcards (EFAL) 99% 95.3% 92.9% 95.7% Sight Words Flashcards (HL) 93.9% 83% 89.8% 88.7% Sight Words Flashcards (EFAL) 86.7% 85.8% 86.7% 86.4% RSP Big Books (EFAL) 58.2% 54.7% 73.5% 61.9% RSP Graded Readers (EFAL) 56.1% 67% 65.3% 62.9% Handwriting Poster 63.3% 63.2% 64.3% 63.6% Posters - Listening and Speaking 70.4% 71.7% 74.5% 72.2% Assessment Record Book 54.1% 40.6% 58.2% 50.7% Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term 78.6% 67.9% 79.6% 75.2% Teacher Management and Storage Files 13.3% 17.9% 7.1% 12.9% USB Stick with Songs Used in Lesson Plans 2% 3.8% 1% 2.3% Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |34 Subject/Curriculum Advisors also reported differences in the extent to which all schools and teachers had received the materials. In one sub-district, all schools were reported to have received the materials. In schools that had received classroom libraries, they were reported as being used. However, a number of schools reported that they had not received all the materials. 4.1.2 ADHERENCE AND QUALITY Adherence refers to whether "…a program service or intervention is being delivered as it was designed or written” and quality is defined as “…the manner in which a teacher, volunteer, or staff member delivers a program” (Mihalic, 2004, cited in Carroll et al., 2007, p.2). Adherence, together with quality, also refers to how well the program material was implemented (Buckley et al, 2017). Carroll et al., (2007, p.6) note that, “If the content of an intervention is delivered badly, then this may affect the degree to which full implementation is realized. In studies evaluating fidelity the provision of extensive training, materials, and support to those delivering an intervention is an implicit acknowledgement that effort is required to optimize the quality of the delivery of the intervention being evaluated” Importantly, adherence to program design is substantially influenced by contextual factors such as: − Program complexity (which becomes particularly important when looking at scaling interventions), − Strategies for facilitating proper implementation (guides, manuals, training, monitoring, feedback, etc.), and − Participant responsiveness (how well the program is accepted and taken up by participants). Dosage and coverage are two aspects to adherence, covered in the section above. This section looks specifically at the content of the interventions and the associated quality of delivery, with a focus on training, coaching, delivery of LTSM, and delivery of SMT support. COACH TRAINING OBSERVATION Key findings from the coach training observation include: 1. All the coaches received the necessary materials. 2. All the coaches received a training presentation in a digiband. 3. HL material needed revision, as there are spelling errors. 4. Some pages in the DBE and Lesson Plans do not correlate. 5. HL Lesson Plan pages are not intact and need to be re-glued. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |35 6. Coaches were concerned about a dry run on the last day as they felt they were not provided enough time to prepare. 7. Coaches reported that they needed time to travel and to prepare for the actual teacher training. 8. A compromise was reached to have the dry run for only half a day, and with demonstrations from three facilitators rather than actual coach dry runs. 9. Not all coaches attended the dry runs. Despite the fact that the master trainers were well organized and presented well on the materials, the coach training observation revealed that coaches were not ready to provide training to teachers. In contrast to the findings of the Design Evaluation, the preparation of coach trainers for training was inadequate, as the dry runs were not carried out and coaches did not receive feedback on their plans for coaching teachers the following week. In addition, the content of the training did not cover coaching methods and approaches. It dealt mainly with content. This finding has implications for the quality of teacher training for Term 4. TEACHER TRAINING OBSERVATION The teacher training observation findings are summarized as follows: 1. In all teacher training sessions, the coach had to facilitate individually – there was no co￾facilitation as agreed upon at the coach training. 2. In most cases, fewer teachers attended than were on the register for training. Of 340 expected teachers, the observer noted that only 254 attended. In one training venue, seven teachers attended that were not on the attendance list. 3. Teachers received materials but there were some shortages in some cases. In seven of eight training venues, all teachers received their LTSM. In one training venue, seven teachers did not get EFAL lesson plans. Those who were not on the list did not get LTSM. 4. In all eight training venues, EFAL materials were packaged in a wrapper. With the HL materials, only flashcards were wrapped but the lesson plans and other material were not. Charts and posters were packaged in rubber bands. 5. The facilitator’s manual was not used - the coaches indicated that they only have an EFAL manual and not a HL manual. 6. The training style was generally participatory, with a few exceptions. 7. Some coaches went through lesson preparation with the participants taking into cognizance the lesson plans provided, therefore, all the aspects of the lesson plan including the use of LTSM and assessments were taken into account. Others simply read through the slides. 8. Some coaches did an in-depth presentation and discussion on paired and independent reading while others read through the slides. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |36 9. Most coaches looked at an aspect and showed how to do it in HL and after that in EFAL. When dealing with a subject the coach switched to the relevant language, which forced the participants to use the language. 10. In most sessions, the pace of training was right for learning. COACHING Teacher Surveys Surveys with 102 teachers that received coaching interventions, and had previously received coaching support from EGRS I coaches, indicated that these teachers felt the RSP coaches were as good as, or of better quality, than EGRS I coaches (92.1% of the sample – see Figure 7 below). Figure 7: Teacher experience of RSP literacy coach compared to EGRS I coach Tables 16 and 17 below demonstrate teachers’ perceived experience of various coaching elements, across the three grades and by intervention. When looking at the results by intervention group (Table 16), there is a clear difference between teachers in the Coaching schools and teachers in the SMT schools with the latter more frequently reporting positive experiences with their coach “to a very large extent.” In the SMT group, the three practices with the highest number of teachers rating “to a very large extent” include (i) coach observations and feedback helping them grow as a teacher (65.4% of respondents), (ii), coaches maintaining open and two-way conversation (65.4% of respondents), and (iii) coaches communicating the planned coaching visits sufficiently in advance that teachers are able to accommodate coaching visits (62.6% of respondents). 7.8% 57.8% 34.3% RSP literacy coach support is of a poorer quality than support provided through the EGRS coach RSP literacy coach support is of the same or similar quality to support provided through the EGRS coach RSP literacy coach support is of a better quality than support provided through the EGRS coach Teacher perceptions of how the RSP reading coach compares with the EGRS I coach (Respondents who received an EGRS and a RSP coach) N=102 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |37 By comparison, in the Coaching group, the three practices with the highest number of teachers rating “to a very large extent” include (i) teachers having gained helpful resources from their coaches (48.9% of respondents) (ii), coaches communicating the planned coaching visits sufficiently in advance that teachers are able to accommodate coaching visits (48.9% of respondents), and (iii) coaches regularly communicating with teachers (46.7%). Table 16: Teacher perceptions of coaching elements By Intervention Group Coaching Coaching & SMT Not at all / to a limited extent / neutral To a large extent To a very large extent Not at all / to a limited extent / neutral To a large extent To a very large extent My coach communicates the planned coaching visits with me sufficiently in advance such that I am able to accommodate coaching visits in my schedule 10.9% 40.2% 48.9% 0.9% 36.4% 62.6% My coach’s observations and feedback have helped me grow as a teacher 8.7% 45.7% 45.7% 3.7% 30.8% 65.4% I have gained helpful resources from my coach 6.5% 44.6% 48.9% 6.6% 31.8% 61.7% My coach communicates regularly with me 15.2% 38.0% 46.7% 11.2% 33.6% 55.1% My coach maintains open, two-way communication with me 17.4% 38.0% 44.6% 9.3% 25.2% 65.4% My coach helps me overcome barriers to teaching and learning 18.5% 41.3% 40.2% 6.5% 40.2% 53.3% My coach holds reflective feedback sessions with me 13.0% 44.6% 42.4% 7.5% 33.6% 58.9% My coach sets action plans after each feedback session 19.6% 42.4% 38.0% 10.2% 35.5% 54.2% *Note: One teacher in an LTSM reported receiving an RSP coach. This teacher may have moved from a coaching school into an LTSM school. When looking at the results by grade group (Table 17), there are no clear substantial differences between the grades. To some extent, more teachers in the Grade 2 group reported positive experiences with their coach “to a very large extent” more often than the other grades (above 50% of Grade 2 respondents). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |38 Table 17: Teacher perceptions of coaching elements by grade By Grade Grade 1 Grade 2 Grade 3 Not at all / to a limited extent / neutral To a large extent To a very large extent Not at all / to a limited extent / neutral To a large extent To a very large extent Not at all / to a limited extent / neutral To a large extent To a very large extent My coach communicates the planned coaching visits with me sufficiently in advance such that I am able to accommodate coaching visits in my schedule 8.6% 38.6% 52.9% 0.0% 32.8% 67.2% 7.9% 44.4% 47.6% My coach’s observations and feedback have helped me grow as a teacher 7.1% 40.0% 52.9% 3.0% 32.8% 64.2% 7.9% 41.3% 50.8% I have gained helpful resources from my coach 10.0% 38.6% 51.4% 3.0% 40.3% 56.7% 6.4% 34.9% 58.7% My coach communicates regularly with me 20.1% 32.9% 47.1% 12.0% 31.3% 56.7% 6.4% 44.4% 49.2% My coach maintains open, two-way communication with me 12.9% 35.7% 51.4% 10.5% 31.3% 58.2% 15.9% 27.0% 57.1% My coach helps me overcome barriers to teaching and learning 11.4% 54.3% 34.3% 13.4% 32.8% 53.7% 11.1% 34.9% 54.0% My coach holds reflective feedback sessions with me 8.6% 47.1% 44.3% 9.0% 35.8% 55.2% 12.7% 33.3% 54.0% My coach sets action plans after each feedback session 17.1% 41.4% 41.4% 11.9% 37.3% 50.7% 14.3% 38.1% 47.6% Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |39 Coaching case studies The 14 coaching case studies are provided in Annex 5 and the findings summarized in Table 18 and Table 19 (key presented below). Key Word Description Extensive Met the criteria / all elements AND the researcher noted additional activities, for example tasks, level of engagement with teacher or school Fully Consistently met the criteria / all elements of the coach observation tool were noted by the researcher to have been completed by the coach Partial The criteria / element / activity was partially completed (met some of the criteria, or was not implemented to the desired standard, or implementation was inconsistent) Inadequate Did not meet the criteria/element or activity, for example did not do the activity or did not do it to the desired standard Not reviewed This element was not included in the field work observation or interview tool Not observed The researcher noted that this element was not observed during the coach observations visits Not reported This element was not documented by the researcher Unclear It was unclear despite the researcher trying to gather evidence (for example, it was unclear if the head coach did provide feedback to the coach on the schedule or not) () Reason for rating explained or further explanation to inform evaluation or RSP Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |40 Table 18: Coaching case study - summary rubrics (coaches 1 to 7) Criteria from Coaches’ Job Descriptions Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 Coach 7 Qualified as teacher Yes Yes Yes Yes Yes Yes Yes Experience in teaching Extensive Extensive Extensive Extensive Extensive Extensive Extensive Ratio coach to schools 1: 10 9 8 8 7 8 10 Ratio coach to teachers 1: 56 43 55 46 50 40 52 Setswana language proficiency Fluent Fluent Fluent Fluent Fluent Fluent Fluent Develop monthly visit schedules Fully Fully Fully Fully Inadequate Fully Fully Monthly Plan submitted to Head Coach Fully Fully Fully Fully Inadequate Fully Fully Head Coach Approval Partial Partial Partial Fully Inadequate Fully Inadequate Coach tells teacher about planned visit Fully Fully Fully Fully Fully Fully Fully Coach sets up direct communication channel with teacher (not through school) Fully Fully Fully Fully Fully Fully Fully Coach re-confirms visit Fully Fully Fully Fully Fully Partial Fully Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Partial Partial Fully Fully Partial Fully Fully Follows steps of Class Observation Fully Fully Fully Fully Partial Partial Fully Monitors progress against lesson plans for the whole lesson Fully Fully Fully Fully Inadequate Partial Fully Looks at learners’ books Fully Fully Fully Fully Inadequate Partial Fully Monitors teachers’ assessment processes Fully Fully Fully Fully Partial Partial Fully Provides feedback to teachers Fully Fully Fully Fully Fully Fully Partial (but not appropriate) Teachers reflect on their practice Partial Inadequate Partial Fully Partial Fully Partial Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |41 Criteria from Coaches’ Job Descriptions Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 Coach 7 Conducts SBWs after school 2 of 4 0 of 0 1 of 4 2 of 4 0 of 4 0 of 4 3 of 4 Sets up PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Facilitates PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Teachers reflect on their practice Not observed Not observed Not observed Fully Not observed Not observed Inadequate Completes in-class coaching tools Fully Partial Fully Fully Partial Fully Fully Completes After School Workshop / PLC tools Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Completes training tools Not observed Not observed Not observed Not observed Not observed Not observed Not observed Participates in training sessions 16 days 16 days 16 days 16 days 16 days 16 days 16 days Head coach visits the coach at school (since January 2019) Once Once Three times None None Yes Not reported Head coach provides support to coach Inadequate Inadequate Inadequate (only admin support) Inadequate Inadequate Adequate Adequate Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |42 Table 19: Coaching case study - summary rubrics (coaches 8 to 14) Criteria from Coaches’ Job Descriptions Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 Coach 13 Coach 14 Qualified as teacher Yes Yes Yes Yes Yes Yes Yes Experience in teaching Extensive Extensive Extensive Extensive Extensive Extensive Extensive Ratio coach to schools 1: 12 14 10 7 11 8 8 Ratio coach to teachers 1: 61 57 55 57 56 57 68 Setswana language proficiency Fluent Fluent Fluent Fluent Fluent Fluent Fluent Develop monthly visit schedules Partial Fully Fully Fully Fully Fully Fully Monthly Plan submitted to Head Coach Fully Fully Fully Fully Unclear Fully Fully Head Coach Approval Inadequate Inadequate Inadequate Inadequate Inadequate Fully Fully Coach tells teacher about planned visit Fully Fully Partial Partial (tells SMT) Unavailable Fully Fully Coach sets up direct communication channel with teacher (not through school) Fully Fully Partial (no airtime) Inadequate Inadequate Fully Fully Coach re-confirms visit Fully Fully Partial Inadequate Inadequate Fully (no airtime) Inadequate Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Fully Inadequate Fully Inadequate Inadequate Fully Inadequate Follows steps of Class Observation Fully Fully Fully Partial Partial Partial Partial Monitors progress against lesson plans for the whole lesson Fully Fully Fully Fully Partial Fully Partial Looks at learners’ books Fully Fully Partial Fully Partial Inadequate Inadequate Monitors teachers’ assessment processes Fully Fully Fully Partial Inadequate Inadequate Partial Provides feedback to teachers Fully Fully Fully Partial Inadequate Fully Fully Teachers reflect on their practice Fully Fully Partial Partial Inadequate Partial Partial Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |43 Criteria from Coaches’ Job Descriptions Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 Coach 13 Coach 14 Conducts SBWs after school 2 of 3 4 of 4 2 of 4 1 of 0 (2 schools attended) 0 of 4 1 of 0 1 of 4 Sets up PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Facilitates PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Teachers reflect on their practice Fully Fully Partial Inadequate Inadequate Inadequate Inadequate Completes in-class coaching tools Fully Fully Fully Fully Fully Fully Fully Completes After School Workshop / PLC tools Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Completes training tools Not observed Not observed Not observed Not observed Not observed Not observed Not observed Participates in training sessions 16 days 11 days 6 days 16 days 16 days 11 days 11 days Head coach visits the coach at school (since January 2019) Once Occasional None Once None Yes Three times Head coach provides support to coach Inadequate Partial Inadequate Inadequate Inadequate Inadequate Partial Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |44 The coaching case studies revealed that coaching skills and practice vary. Some coaches are strong administratively and/or very thorough but directive in their feedback, providing their own suggestions rather than facilitating a guided discussion. Other coaches are stronger in their interpersonal skills and develop nurturing relationships with teachers but tend to be less consistent across classrooms or less strong administratively. In general, the coaches are adequately planning and preparing for the visit to the schools, and engaging appropriately with the school principal, SMT, and the teachers. Classroom observations are occurring, where the coach has an opportunity to monitor the teacher’s ability to implement the lesson plans. Coach 12 had the weakest coaching competencies. This coach supports 11 schools and 56 teachers. The coach was observed for a period of four days, at three schools, and interacting with nine individual teachers. As per the coach case studies and rubric (Annex 5), all scores except for two were either partial, inadequate or overlooked. The only two times the coach was rated fully compliant was for submitting the monthly schedule and for completing the coaching observation tools. During the week of observations, the coach had planned to conduct four SBWs, but did not complete any. When asked to describe her role as a coach she noted that she was, “Coaching only to support the RSP project.” The researcher further indicated that she was resistant to the coaching observations and was not forthcoming on sharing her experience as a coach. In five classes, the coach did not check whether the teacher acknowledged learners experience and prior knowledge, but she did check this in three classes. On a positive note, the coach paid particular attention to how learners with barriers are catered for by the teachers and in one instance encouraged a teacher to note individual learners to provide with a remedial lesson. However, teacher feedback was weak. In the feedback sessions with teachers, the coach did not ask the teacher to identify things they felt went well, nor did she ask the teacher what they would have done differently, nor did she explicitly ask questions that would encourage the teacher to reflect in all observed instances. Furthermore, in two instances the researcher notes that the coach provided negative feedback, and that in three instances providing of constructive feedback was only “Partially executed.” It was noted that her interpersonal skills and her administrative skills were areas for growth. Coach 4 demonstrated the highest or best competencies. This coach supports eight schools and 46 teachers. She was observed for a period of four days working with four schools and 11 teachers. As per the coach case studies rubric, she was rated “Fully” for all for all her coaching duties except for the tasks relating to Professional Learning Communities (PLCs). Of three planned after school workshops, she held two. The researcher described her as “Friendly, welcoming, respectful, humble, and very eager to help… Has a good relationship with the teacher… She goes the extra mile.” The coach was consistent across classrooms. For example, she checked whether LTSM was available, relevant, and well organized in all 11 observations. In all 11 observations, she checked that relevant reading methodologies were applied. In only three classes, the coach was not observed checking whether learners with barriers were provided for or not. In all observations, the coach made notes of whether the lessons were clear, and in all observations, the coach checked whether lesson introductions Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |45 were creative and whether they acknowledged learners’ experience and prior knowledge. In all observations, she gave her comments on what had gone well and where the teachers could improve. In eight observations, she asks questions to the teachers that prompt reflection, and assists the teachers in coming up with actions steps they can take. In all sessions the coach and teacher worked together to set goals for the next visit. As one of the reasons for her success, the coach noted that success factors included, “Building a trusting relationship with the teacher and providing practical modelling of how to implement lessons when they are having challenges. Not being judgmental but helpful.” Successes and challenges The availability of vehicles for use by the coaches to get to the school is a success. Seven coaches specifically cited that because they are provided with transport and fuel, this is a positive, enabling factor for them to be able to do their job. While this is an important factor to coaches, one coach mentioned he did not always have access to the vehicle, and that there was a lack of support from the asset manager which impeded him from being proactive in offering solutions to teachers. Another coach also noted her challenge with the project car, she indicated that she is only able to fetch the vehicle from the parking lot at 7 a.m., which then means that he can only arrive at schools at 8 a.m., due to how far away the schools are. Many coaches indicated good and timely communication, both of their monthly plans and log sheets, with the principals and HODs as a factor that contributed to successful coaching practice. All but one of the coaches, indicated the importance of building trusting relationships with teachers and the SMTs, and many gave instances of how having a good relationship with those stakeholders helped them to coach better. Six coaches noted the helpfulness of a WhatsApp Group with their teachers, which enabled more efficient communication, but also meant that they could provide remote assistance. However, two coaches raised a challenge that they were not provided with airtime to phone teachers, and that not all teachers are on WhatsApp. Other factors identified by coaches17 as moderators of success include: − Conflicting events happening on the same day at a school. For example, district meetings, school trips, and sports/entrepreneurship days when learners are not in class. − Missing LTSM at some schools, or no libraries, which makes it difficult for the coach to monitor how the teacher uses these resources. − Teachers not being able to remain after school to attend workshops as they will then miss their transport home. − Teacher shortages or the school has appointed un/under qualified teachers, which makes it more difficult to coach teachers. 17 Not all coaches cited these factors consistently. This is a summary of all factors identified by coaches. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |46 − Overcrowding in classrooms, making it difficult for coaches to observe teachers and for teachers to teach effectively. − The elements that Coaches believe can contribute to the success of coaching include: o Arriving early to the training of the teachers to discuss the lesson with the teacher and ensure all is prepared. o Having transport which makes travelling to schools easy. This helps to arrive at the school on time. o Being welcomed by the teachers who are open to observation. AVAILABILITY AND USE OF LTSM According to FPD monitoring data, lesson plan, and LTSM availability in RSP schools is not on target. As noted earlier in the report, survey data collected by the evaluation team from 302 Grade 1 to 3 teachers, shows that more than 90 percent of teachers reported receiving lesson plans and flashcards between January and October 2019. Comparatively only 62 to 63 percent reported having received the EFAL Big Books and Graded Readers. Of the 304 Grade 1 to 3 teachers in the sample who reported receiving LTSM, 97 percent reported that they received their LTSM at the teacher training. Figure 8: Teachers who reported they received RSP LTSM at training Yes 97% No 1% NA 2% Percentage of teachers who reported receiving RSP LTSM at the training Teacher Survey Respondents Yes No N=304 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |47 There were only ten respondents who indicated that they did not receive their LTSM at the training. Of these, two did not receive any LTSM, six received their materials from other teachers in the school that attended training, and two indicated that they had their LTSM delivered to their school. In the district and provincial interviews, three of the CA/SAs commented on the high quality of the LTSM materials and the importance of equipping teachers with resources they can use. One CA/SA indicated that it is necessary to induct new Heads of Department (HODs) in the use of the materials. The coaching case studies revealed some issues with the LTSM. In one observation, the researcher noted that one teacher did not have a ‘Lesson Presentation’. The teacher indicated that she did not know how to compile one since she had not attended any RSP training or workshops. The researcher observing coach 12 noted that “The teachers she coaches generally lacked lesson plans” and two teachers made comments about the LTSM charts falling off the walls. The researchers noted that there were not enough books for all the children in some classrooms. In terms of the quality and usefulness of the lesson plans and LTSM, group discussions in 13 schools consistently revealed that teaching aids were useful to teachers. However, they noted that the pace of the lesson plans was too fast and some teachers had problems with the time allocated to writing and/or phonics. Teachers who responded to the Teacher Survey noted that they used the Scripted Lesson Plans most frequently (81.8% of respondents reported using the HL lesson plans daily and 80.5% of respondents reported using the EFAL lesson plans daily), followed by the Theme Vocabulary Flashcards (62.6% of respondents reported using the HL flashcards daily and 61.3% of respondents reported using the EFAL flashcards daily), and Sight Word Flashcards (62.3% of respondents reported using the HL flashcards daily and 59.6% of respondents reported using the EFAL flashcards daily). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |48 Table 20: Frequency of LTSM Use Frequency of LTSM Use NA Never Once a term Once a month Once a week Nearly every day Every day Scripted Lesson Plans (Termly Lesson Plans) HL (N=302) 2% 2.3% 1.7% 0% 1% 11.3% 81.8% Scripted Lesson Plans (Termly Lesson Plans) EFAL (N=302) 3% 1.3% 2% 0% 1.7% 11.6% 80.5% Theme Vocabulary Flashcards (HL) (N=302) 5.3% 2% 1.7% 0% 6.6% 21.9% 62.6% Theme Vocabulary Flashcards (EFAL) (N=302) 4% 1.3% 2% 0.7% 5.6% 25.2% 61.3% Sight Words Flashcards (HL) (N=302) 7% 2.3% 1.7% 0% 4.3% 22.5% 62.3% Sight Words Flashcards (EFAL) (N=302) 6% 2% 2% 0.3% 6% 24.2% 59.6% RSP Big Books (EFAL) (N=302) 29.5% 3% 2.6% 0% 11.6% 23.8% 29.5% RSP Graded Readers (EFAL) (N=302) 22.8% 3% 1% 0.7% 6.3% 23.8% 42.4% Handwriting Poster (N=302) 22.2% 2.6% 2.3% 1% 9.3% 24.8% 37.7% Posters - Listening and Speaking (N=302) 11.9% 1.3% 2% 1% 5.3% 31.1% 47.4% Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term (N=302) 12.3% 4.6% 3.% 3% 21.9% 14.9% 40.4% When asked “If you could change one thing about the RSP what would it be and why?” teachers who filled in the teacher survey consistently mentioned timing of lesson plans as a specific issue needing attention. These findings are supported by interviews with SMT members, who noted concerns with the allocation of time to implement all learning activities. Table 21 below documents teacher comments on LTSM per the group discussions and Teacher Survey. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |49 Table 21: Teacher comments on LTSM usefulness and quality Teacher Comment on RSP LTSM Teacher group discussions (13 schools) Teacher surveys (n = 304) Lesson plans are too fast, there are too many elements, or the lesson plans are too time consuming 12 out of 13 schools 43% of teacher respondents (130 of 304) Slower learners are unable to keep up with the pace of lesson plans 4 out of 13 schools 8% of teacher respondents (25 of 304) Teaching aids (phonics posters, word cards, etc.), big books, and/or graded readers were of good quality and useful 12 out of 13 schools NA Too little time specifically allocated to writing and/or phonics 3 out of 13 schools 13% of teacher respondents (39 of 304) Writing tasks too difficult, particularly for Grade 3 EFAL NA 7% of teacher respondents (21 of 304) QUALITY AND USEFULNESS OF TEACHER TRAINING In the Teacher Survey, teachers were asked to rate the value of the content of the RSP training. Per Figure 9 below, most teachers across the grades found the training content to be “valuable” (58.7% of respondents in Grade 1, 59.0% in Grade 2 and 66.3% in Grade 3). Approximately a third of the teachers found the training content to be “extremely valuable” (32.7% of respondents in Grade 1, 38.0% in Grade 2 and 30.4% in Grade 3). Fewer than 9 percent of teachers rated the training content “somewhat valuable” and no teachers rated the training content “not at all valuable.” Generally, most respondents appeared satisfied with the training. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: FORMATIVE EVALUATION REPORT |50 Figure 9: Teacher perspectives on the training content value (by grade) Tables 22 and 23 below break this down into the usefulness of specific components of the training, across the three grades and by subject (HL and EFAL). Altogether, more than 80% of teachers across grades and subjects rated the training components either “Somewhat useful, I learned some new things” or “Very useful, I learned a lot.” Most teachers rated the training components as “very useful” across all three grades and both languages, with Grades 2 and 3 HL and EFAL having the highest numbers of teachers rating the training components “Very useful.” Comparatively, there were fewer Grade 1 teachers, across HL and EFAL, who rated the components “Very useful,” although these numbers were still high. Few teachers rated any of the components as “Not useful to me, I already knew the content but well delivered” (less than 11% of respondents) and even fewer (less than 9% of respondents) rated the training components, “Not useful to me and needs improvement.” Interestingly, handwriting had the highest numbers of respondents indicating that the component was not useful, both for HL and EFAL. 3.3% 66.3% 30.4% 3.0% 59.0% 38.0% 8.7% 58.7% 32.7% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Somewhat valuable Valuable Extremely valuable Teacher perspectives on value of the content of training workshops Grade 1 Grade 2 Grade 3 n = 296 51 Table 22: Teacher perceptions of usefulness of HL training components Home Language Grade 1 Grade 2 Grade 3 Somewhat Useful Very Useful Somewhat Useful Very Useful Somewhat Useful Very Useful Weekly Routine 37.5% 56.7% 34.7% 62.4% 26.1% 69.6% Phonics Lessons 33.7% 62.5% 19.8% 76.2% 20.7% 75.0% Reading Lessons 48.1% 46.2% 25.7% 66.3% 31.5% 62.0% Handwriting Lessons 37.5% 55.8% 28.7% 59.4% 38.0% 53.3% Writing Lessons 44.2% 49.0% 33.7% 62.4% 32.6% 59.8% Listening and Speaking Lessons 30.8% 66.3% 21.8% 74.3% 23.9% 72.8% Assessment Tasks 38.5% 59.6% 30.7% 63.4% 32.6% 60.9% 52 Table 23: Teacher perceptions of usefulness of EFAL training components English First Additional Language Grade 1 Grade 2 Grade 3 Somewhat Useful Very Useful Somewhat Useful Very Useful Somewhat Useful Very Useful Weekly Routine 40.4% 53.8% 35.6% 59.4% 26.1% 66.3% Phonics Lessons 29.8% 64.4% 30.7% 66.3% 20.7% 71.7% Reading Lessons 50.0% 44.2% 30.7% 61.4% 25.0% 65.2% Handwriting Lessons 40.4% 51.9% 32.7% 47.5% 42.4% 40.2% Writing Lessons 44.2% 50.0% 38.6% 56.4% 34.8% 53.3% Listening and Speaking Lessons 35.6% 60.6% 33.7% 61.4% 26.1% 68.5% Assessment Tasks 42.3% 54.8% 34.7% 61.4% 30.4% 59.8% 4.1.3 PARTICIPANT RESPONSIVENESS Participant responsiveness is defined as “How far participants respond to, or are engaged by, an intervention” (Carroll et al., 2007, p. 3), or how well the instruction was received or perceived by participants (Buckley et al., 2017). INTERVIEWS WITH DISTRICT AND PROVINCIAL STAKEHOLDERS Per the sampling methodology in Section 3.5, 15 provincial and district officials participated in KIIs. The provincial and district officials indicated that the RSP has not been implemented for sufficient time to be able to see or note uptake by teachers or changes in learners, nor have officials had an opportunity to visit the schools (five respondents). One official had visited an RSP school and noted that the teacher was following all the lesson steps and instructions. Some officials noted that when reviewing class performance from the start of the year, there was an improvement in learners’ marks. One CA/SA noted that at the end of the year they would undertake an analysis of results and compare this to previous years’ learner results. Their assumption is that there will be an improvement given the EGRS I results and the fact that the two projects are similar. 53 All six CA/SAs indicated that teachers are using the RSP materials. They noted changes in the classroom layout, materials on display such as reading charts, increased planning of lessons by the teachers, teachers following the routines, increased confidence of teachers teaching reading, and better management of classroom libraries. One of the challenges noted by a CA/SA was the management of teachers to keep to the recommended time allocation for activities during a lesson, and another noted the confusion among teachers regarding the use of RSP and Primary School Reading Improvement Programme (PSRIP) materials. Another CA/SA indicated that the attitude of teachers would determine the success of them implementing the materials: some teachers have a negative attitude and therefore despite excellent materials, they will not use them. However, she noted that with the support of the coaches and continuous engagement, this negative attitude could change. One district official indicated that they have seen a change in the teachers from the first training at the beginning of the year to now, as they can see that teachers are applying the training and using materials in the classroom. Further, the official indicated that the RSP has contributed to reducing teachers’ work load as they do not need to spend as much time on lesson planning as lesson plans are provided with the materials, and that teachers can monitor their own progress thereby giving more time to focus attention on the learners. Three CA/SAs noted that the step-by-step instructions are contributing to teacher practice and ability to use the RSP materials. One CA/SA indicated that Group Guided Reading remains a challenge for teachers to implement, and that further support, and clarity, is needed on the methodology, processes and materials. According to CA/SAs, the biggest moderating factors hindering uptake include large classroom sizes, which is challenging for teachers to manage; teachers not receiving all the materials; principals moving teachers across grades every year; time management; inexperienced or under qualified teachers, or no pool of qualified teachers who are teaching in the Foundation Phase with the basic skills needed to teach reading. TEACHER RESPONSES The findings from the teacher group discussions corroborate the findings noted above: − In eight (8 of 13) schools, teachers noted that the training was appropriate and relevant. − Teachers in eight schools noted that the training time was too short for learning, and this led to confusion in some cases. − Teachers in two (2 of 13) schools noted that demonstrations were inadequate and that a simulation in schools would be more conducive to learning. Overall, teachers felt that training improved from term to term, with Term 3 being much better than the previous terms. In the Teacher Survey, 92 percent of teachers (n = 304) rated the teacher training workshops as “Valuable” (59.9%) or “Extremely Valuable” (32.6%). When asked to rate which terms’ training 54 was the most useful, Term 3 was cited by 44 percent (44.1%) of teachers and Term 4 by 33 percent (31.3%) of teachers. Term 1 was the least useful training, according to teachers. Most teachers in the group discussions reported that they had received adequate support from their coach. The biggest complaint was the little time dedicated to training on complex constructs. On the Teacher Survey, 94 percent of teachers who reported having received RSP literacy coaches (n = 199), noted that their coaches’ feedback helped them grow as a teacher to a large or very large extent. Eighty Seven percent (87%) of teachers also reported that their coach helped them to overcome barriers to teaching and learning to a large or very large extent. On the Teacher Survey, respondents that reported having previously had an EGRS I coach (n = 102), 92.2 percent noted that their RSP coach was of the same (57.8%) or better (34.3%) quality. Eighty Five percent (85%) of teachers who filled in the Teacher Survey (n = 304) reported that they would likely recommend RSP to a friend, colleague, or others (rating of six out of ten or above). 4.1.4 PROGRAM DIFFERENTIATION Program differentiation is defined as “The degree of contrast between treatment and control strategies and/or activities [emphasis added]” (Buckley et al., 2017, p. 5). The Summative Evaluation, together with the Impact Evaluation results for the RSP should be able to say something about which components are the most essential for impact. This Formative Evaluation looked at program differentiation specifically in terms of the degree to which the program was differentiated within each of the treatment and comparison conditions. Ideally, instead of a “one size fits all” approach, the program would be implemented differently (e.g., light touch versus more intensive support) according to (i) status of reading in the school, (ii) rural versus urban schools, (iii) previous EGRS I schools versus new schools, etc. Of the 102 surveyed teachers who reported receiving RSP coaching support, 89 percent reported that they also received support from coaches affiliated with the EGRS I (see Figure 10 below). Given the large numbers of teachers who had previously interacted with EGRS I coaches, RSP interventions should have been differentiated according to prior exposure (e.g., previous EGRS I schools receive a lighter touch intervention than those who were previously in a control condition in the EGRS). 55 Figure 10: Teachers who indicated that they received EGRS I coaching support Learner reading performance baseline data was collected as part of the sustainability evaluation of the EGRS. Although this baseline included RSP schools, there was no use of baseline results to allow FPD and the Coaches to differentiate support according to the reading level of learners in the school. The coaching observation found that only two of the fourteen coaches differentiated their support to teachers using a color-coded system. Coach 10 stated that she is not always sure how to differentiate her support as she said, “With RSP, I am not sure how the schools and teachers get their ratings. I am not sure if it is the Head Coach who rates the schools and teachers.” Three important notes about differentiation in the context of the delivery of the RSP include: − Without high levels of SMT support, the two coaching conditions are very similar. − Without proper dosage and quality of coaching, it is important to explore the extent to which these interventions are differentiated in practice. Yes 89% No 11% Received coaching support from coaches affiliated with the Early Grade Reading Study (2015-2017) Teacher Survey Respondents Yes No N=102 56 Photo: Khulisa, LTSM on a desk in RSP classroom 57 4.2 WHAT EVIDENCE IS THERE OF RSP UPTAKE BY TEACHERS IN CLASS? There is some evidence of RSP uptake by teachers in the classroom. However, there are some gaps. Less than 45 percent of all schools have reading mats (not a project requirement but indicative of good teaching practice). Most classrooms have HL and EFAL word walls while all schools have a low rate of displaying children’s work. The practices that slow teaching down were seen most often in the SMT and LTSM classrooms. In all three groups, there was still choral reading, but this was lowest in SMT classrooms. There was great variation across the schools with reference to teaching book knowledge and print concepts. In seven of nine criteria linked to teachers explicitly developing learners’ concepts about print and printed material, the SMT classrooms had the highest number of observations of preferred practices. There was little difference in the teaching of book knowledge and print concepts across Grade 1 and 3, although Grade 3 children do not still need to be taught about book covers, spines, front and back and reading direction. There was more use of English in the SMT and Coaching classrooms as compared to the LTSM classrooms. In the SMT classrooms, teachers most often modelled the correct term, highlighted English terms, and encouraged the use of English compared to the other groups. The explicit teaching of vocabulary and spelling was also higher in the Coaching and SMT classrooms than in LTSM classrooms. Teaching of handwriting and copying from the board was low overall across the interventions. Learners writing more than five sentences was observed most often in LTSM classrooms. There was little difference in the observation of teachers reading aloud to children and Group Guided Reading was similarly observed in less than 50 percent of schools across all groups. The use of post-reading questions was highest in the SMT classrooms. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: − Classroom observations in 119 Grade 1 and 120 Grade 3 classrooms (Grade unclear in three observations). − Classroom library assessments in 58 schools and classroom materials assessments in 57 schools. − Assessments of 118 Grade 1 and 120 Grade 3 learner workbooks. The evaluation team conducted 242 classroom observations in 60 sample schools in the districts of Dr Kenneth Kaunda and Ngaka Modiri Molema in the North West Province. Per Table 24, and in keeping with the sampling methodology, more observations occurred in the Ngaka Modiri Molema district. There was a relevantly equal division of Grade 1 and Grade 3 classes observed and, within these, similar numbers of HL (Setswana) and EFAL classes observed. With the exception of five lessons, the usual class teacher was observed. 58 Fieldworkers had good access to the classroom and only in four cases were there any issues with access. Table 24: Lessons Observed by District, Grade, and Language District Number of lessons observed Grade One* Grade Three* Language HL EFAL HL EFAL Dr Kenneth Kaunda 58 15 15 14 15 Ngaka Modiri Molema 184 44 44 46 46 Total 242 59 59 60 61 *When broken down by Grade, the number of classrooms observed tallies 239. This is due to the fact that in three classroom observations, grade could not be distinguished. Of the 60 sample schools in which lessons were observed, the sample was split into three groups for the purpose of analysis: − Group 1: Schools where teachers are receiving coaching (referred to as Coaching schools) – a mix of ten previous EGRS I control schools and ten previous EGRS I coaching schools. − Group 2: Schools where teachers are receiving coaching and the SMT is receiving support (referred to as SMT schools) – a mix of ten previous EGRS I control schools and ten previous EGRS I coaching schools. − Group 3: Twenty schools that have only received teacher training and LTSM (referred to as LTSM schools) - who were previously EGRS I parental involvement schools. The primary purpose of the classroom observations was to identify differences in instructional practices among the three different groups (Coaching schools, SMT schools, and LTSM schools), to identify which practices appeared more successful and to identify issues that need to be addressed in 2020. For all tools with a scale element (e.g., where fieldworkers rated from “never seen” to “always seen”), the evaluation team created a composite of all the positive ratings (i.e., added together all responses indicating “sometimes,” “mostly,” or “always”) when indicating where a practice was “observed” and “not observed.” Only counting “always” as the equivalent to “observed” skews the results quite substantially. 59 Additional comparisons were made on selected aspects of reading and writing instruction across Grade 1 and Grade 3 classes as well as across HL and EFAL lessons. 4.2.1 COMPARATIVE FINDINGS INTRODUCTION The RSP lesson plans were designed with particular activities taking place on each day of a given week. Teachers in all three groups (Coaching schools, SMT schools, and LTSM schools), were trained on how to implement the RSP lesson plans. Therefore, on particular days particular activities would not be observed – not because teachers were not doing what they were meant to but because the activity was not part of the lesson plan for the day. The fact that this is consistent across the intervention groups means that the findings can still be compared in relation to one another. However, the findings are not comprehensive in terms of whether teachers generally engage with specific activities, or not, in the classroom. Where possible, classroom observation results from the RSP have been compared to the EGRS I I classroom observation results. While the same tools were used for both analyses18, the data from the EGRS I classroom observation study is not always reported in a comparable manner to the RSP classroom observation data. In addition, the methodology differs quite substantially between the two projects. The RSP evaluation analyzed two Grade 1 and two Grade 3 classrooms per school in 20 schools from each of the Coaching, SMT and LTSM school groups (i.e., four lessons per school), split evenly across two language lessons – HL and EFAL. Comparatively, in the EGRS I evaluation, one lesson was observed per school, in one grade (Grade 2) and one language (HL). In total, the RSP evaluation analyzed observations from 239 classrooms, (242 lesson observations were conducted, but only 239 classrooms were captured by grade) compared to 60 lesson observations in the EGRS I. Additionally, the RSP and EGRS I researchers were not the same between the two observations. Thus, differences in scores may be at least partly due to fieldworker effects. It must therefore be cautioned that the results are not directly comparable, but a comparison does provide a point of reference for the RSP results. The findings are interpreted in this context. TEACHING AND LEARNING ENVIRONMENT Cleanliness of classrooms and movement ability In the RSP classes, most classrooms were clean across all schools, with clean classrooms found in the SMT schools (98.7%), Coaching schools (85.9%), and LTSM schools (84.8 %). With a few exceptions, teachers were able to move easily between the desks. 18 The RSP classroom observation comprised a subset of the items from the EGRS I classroom observation tool 60 Reading carpet/mat The presence of a reading mat is important in the Foundation Phase classroom as it provides a more informal and intimate space for the teacher to do shared reading with Big Books as well as Read Alouds. Reading mats were not provided as part of the RSP LTSM, but teachers are encouraged to source and use these. In the RSP classrooms, reading mats were found in fewer than half of all schools – in Coaching schools, only 44.7 percent of the classrooms had mats, compared to 35.9 percent of the SMT classrooms and 41.8 percent of the LTSM classrooms. The presence of a reading mat does not indicate whether the mat is being used, but these figures indicate that at least half of all classes do not provide a dedicated space for the teacher to do Shared Reading and Read Alouds. By comparison, in the EGRS I classrooms reading mats or reading carpets were observed in 75 percent of Coaching schools, 65 percent of Training schools, and 10 percent of Control schools (DBE, 2017, p.39). The numbers of classrooms with reading mats were low but fairly consistent across the RSP groups whereas the EGRS I evaluation saw large differences based on treatment status. Print rich classes Children need to be able to see new words repeatedly in order to commit them to memory. The words should also be arranged logically (either thematically, structurally or phonically). In the RSP observations, there was a slightly higher prevalence of HL and EFAL word lists up in the Coaching and SMT classrooms. Regarding the systematic arrangement of the word lists, this was found more often in the Coaching classrooms for HL, as well as for EFAL. However, none of these differences was significant when comparing schools by group and by grade. It is important for the motivation and confidence of children that they see their work displayed, as well as for their understanding that writing is purposeful and has an audience other than the teacher. Overall, around half of the RSP classrooms were observed displaying children’s work, with the LTSM classrooms displaying learners’ work most often (53.8%). This difference was significant in Grade 3 classrooms (Fisher's Exact = 0,026). By comparison, very few classrooms observed in the EGRS I displayed learners’ work on the walls (DBE, 2017, p.39). Teaching slowed down Various studies in SA have shown (Hoadley, 2012; Taylor, 2012) that time is often wasted on unnecessary activities during lessons. In this observation, five practices that slow down teaching were documented: 1. Handing out books: Seen to some/a large extent mostly in LTSM classrooms (30.1%), and SMT classrooms (29.9%) and to a lesser degree (21.2%) in Coaching classrooms. 61 2. Teacher consulting notes: Seen to some/a large extent mostly in SMT classrooms (28.6%) and in LTSM classrooms (23.8%) and to a lesser degree (14.1%) in Coaching classrooms. 3. Rearrangement of desks and chairs: Seen to some/a large extent mostly in Coaching classrooms (36.5%) and lower in other classrooms (SMT, 28.6%, and LTSM 26.3%). 4. Children not having the necessary stationery: Seen to some/a large extent mostly in LTSM classrooms (26.3%) and SMT classrooms (26.0%) and least in Coaching classrooms (18.9%). 5. Outside interruptions: Most (82.4%) Coaching classrooms had no outside interruptions to lessons during the observation session (compared to 77.9% of SMT classrooms and 67.5% of LTSM classrooms). In the EGRS I, handing out books slowed down teaching to “some/a large extent” in 50 percent of Control classrooms, 60 percent of Training classrooms, and 55 percent of Coaching classrooms - almost double the amount this was seen in RSP classrooms. Teachers consulting their notes slowed down teaching to some or a large extent in ten percent of Control and Training classrooms, and 35 percent of Coaching classrooms. Learners re-arranging the furniture wasted time in 35 percent of Control classrooms, five percent of Training classrooms, and ten percent of Coaching – comparatively this was seen more often in the RSP classrooms. Learners not having the necessary stationery slowed down teaching to some/large extent in 45 percent of Control classrooms, in 25 percent of Training classrooms and 20 percent of Coaching classrooms (DBE, 2017, p.40). Overall, there was no clear pattern of time wastage practices between the groups in either the RSP or the EGRS I. When some learners are finished with their work. Learners complete tasks at different rates and it is important that there are set activities that children know to do when they have completed tasks so as not to sit around idly waiting for their peers to finish. The RSP classroom observation data shows that, − In approximately two thirds of all observations, children in SMT classrooms went to the book corner to choose a book (64.9%, comparable to the 64.8% in Coaching classrooms) while in 56.3 percent of LTSM classrooms this practice was observed. − In approximately half (54.1%) of Coaching classrooms, learners who had finished before others worked in workbooks (versus 46.8% in SMT and 45% in LTSM). − Peer assistance (learners helping other learners once they had finished) occurred more often in Coaching classrooms (78.9%) than in the other two intervention groups (70.1% in SMT schools and 62.5% in LTSM). The numbers of children who sat idly after finishing work were similar across all groups (51.8% in Coaching classrooms, 55.6% in SMT classrooms, and 51.3% in LTSM classrooms). 62 LISTENING AND SPEAKING Choral responses Choral responses refer to the entire class responding in unison, a practice that is particularly prevalent in South African classes (i.e., choral reading instructional practice as opposed to systematic reading teaching practice). In the South African context, this practice can be pedagogically unsound as it does not facilitate learners’ ability to decode texts and is often misinterpreted by the teacher who assumes they can read when they have simply learnt to say the words by heart. The teacher cannot know which children understand and which do not (Fleisch, 2016). In the RSP classroom observations, learners were observed repeating the text off by heart or simply repeating what was read by the teacher or by other learners who could read (without actually engaging in reading) in 73.1 percent of LTSM classrooms, 69.4 percent of Coaching classrooms, and 58.4 percent of SMT classrooms. However, this was seen to varying degrees – from less than a quarter of the class (in approximately a third of classrooms across all three groups) to all of the learners (less than 8% of classrooms). There was no difference by language (HL or EFAL) but there was a substantial difference between the grades, with choral responses seen more often in Grade 1 than Grade 3 classrooms. Figure 11: Choral responses by grade It was mostly in the Coaching classrooms that the teacher tried to make sure that a wide variety of learners had a chance to answer questions by actively trying to involve those learners who did not voluntarily participate (80.0%). This was lower in the LTSM classrooms (55%) and SMT classrooms (57.9%). 67.8% 49.6% 75.4% 54.5% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0% Grade 1 Grade 3 Choral responses By grade Learners reading aloud repeat the text off by heart (not engaging with reading) Learners repeat what is read by teacher/other learners (not engaging with reading) 63 Explicitly teaching book knowledge and print concepts Children need to be taught the various parts of a book, the print conventions of reading from left to right and top to bottom and the role of illustrations in meaning making. This does not need to be taught during every single lesson and once it has been taught (usually in Grade R and 1), it does not need to be repeated every time a book is read. There was no definite pattern in the data of any group (Coaching, SMT or LTSM) explicitly teaching all print concepts. It is to be expected that some print concepts are taught more regularly and frequently in Grade 1. These include teaching various parts of the book and page direction. Yet the data shows that the teaching of print concepts is generally being taught as frequently in Grade 3 as in Grade 1. There is a slightly different pattern in EFAL classrooms where teaching of print concepts is generally being taught more often in Grade 3 classrooms compared to Grade 1 classrooms. In the EGRS I, there was also no obvious pattern of one group teaching print concepts consistently more often than the others (EGRS, 2017, p.42). Use of English in the Classroom Three items in the classroom observation tool refer to the use of English in the classroom. Looking at the use of English in EFAL RSP classrooms, generally, the Coaching classrooms and the SMT classrooms appeared to use English “to a large extent/mostly/always” more often in the classroom than the LTSM schools. A closer examination of the data generated for the disaggregated Grade 3 EFAL class is telling, as it is necessary that teachers use English sufficiently and regularly to expose Grade 3s to the language and its academic terms and concepts. Table 25 shows that in the SMT classrooms, teachers most often modelled the correct term, highlighted English terms, and encouraged the use of English compared to the other groups. Table 25: Overall usage of English in EFAL classes Item % seen in EFAL Coaching SMT LTSM Grade 1 Grade 3 Grade 1 Grade 3 Grade 1 Grade 3 Teacher models correct English term 70.0% 52.4% 78.9% 84.2% 55.0% 57.1% English term highlighted 50.0% 52.4% 89.4% 84.2% 55.0% 47.6% 64 Item % seen in EFAL Coaching SMT LTSM Grade 1 Grade 3 Grade 1 Grade 3 Grade 1 Grade 3 Use of English encouraged 55.0% 80.9% 84.2% 94.8% 75.0% 76.1% Photo: Khulisa, Foundation Phase Classroom 65 LANGUAGE AND LITERACY DEVELOPMENT This section covers two important aspects of language and literacy development: how often the teacher models strategies on how to read words when reading to the children (usually during Shared Reading) and how often the teacher uses strategies on how to develop understanding during reading. Teacher Models Strategies on How to Read Words Table 26 below shows that the teachers in the SMT schools were observed modelling all four reading aloud strategies more often than in the other schools. Teachers in the Coaching schools were observed modelling reading aloud strategies the least. It is important to note that different texts require different strategies so it would not be relevant to use every strategy for every text. Table 26: Observations where strategies to read words aloud were observed Strategies modelled by Teacher Coaching SMT LTSM Sounding out a word 74.1% 88.3% 80% Guessing a word 21.2% 49.4% 30% Use similar words that are already known to work out word 40.0% 70.1% 47.5% Skip a word and return after using context 11.8% 23.4% 13.8% There was virtually no difference between the strategies teachers used in HL and EFAL classrooms (i.e., there was a less than 2% difference in observations for each strategy listed in Table 26), with the exception of sounding out a word, which was observed in 8.2 percent more HL classrooms than EFAL classrooms. When disaggregating by grade (Table 27), there is a clear distinction between the grades with strategies to read aloud words being observed more often in Grade 3 than Grade 1 classrooms. 66 Table 27: Observations where strategies to read words aloud were observed by grade Strategies modelled by Teacher Grade 1 Grade 3 Sound out a word aloud 78.0% 83.5% Guess a word 14.6% 40.5% Use similar words to work out a word 47.5% 56.2% Skip & contextualize the word 11.0% 21.5% In the EGRS I classroom observation study, as in RSP, sounding out a word was used more often than the other strategies. This was observed in 50 percent of the Control schools, 60 percent of Training schools, and 70 percent of Coaching schools. Guessing a word was used in only 10 percent of Control and Coaching schools, and in 15 percent of Training schools (EGRS, 2017, p.42), which is approximately half the number of times this was observed in RSP classrooms. The strategy of using similar words that are already known to work out a word was observed in 35 percent of EGRS I Control schools, 15 percent of Training schools, and 40 percent of Coaching schools. The strategy for skipping a word and then coming back to it to figure it out in context was observed in only 5 percent of Control schools, and 10 percent of schools in each intervention group (EGRS, 2017, p.42). Although not directly comparable, as a point of reference it is interesting that all four strategies were observed more often across the RSP groups than the three EGRS I groups. Teacher Uses Strategies to Develop Understanding during Reading Table 28 illustrates that teachers in RSP SMT classrooms consistently outperformed teachers in the other two groups in terms of supporting learners to develop meaning from a text. Asking children various types of questions and getting them to retell what has been read is what develops understanding (which is the key purpose of reading). 67 Table 28: Observations where strategies for developing an understanding of text Item observed % Coaching SMT LTSM Learners Encouraged to Retell, Act, Summarize Story 35.3% 57.1% 43.8% Learners Encouraged to Discuss Illustrations 77.6% 79.2% 61.3% Teachers Use Illustrations to Aid Understanding of the Text 69.4% 80.5% 70.9% Learners Identify Main Theme of the Text 51.8% 70.1% 40.0% Learners Answer Open Ended Questions 50.6% 72.7% 65.0% Learners Answer Predictive/Inferential Questions 34.1% 62.3% 43.8% Learners Answer Literal, Evaluative, Appreciative & Critical Questions 42.4% 68.8% 47.5% When disaggregating by language (Table 29), these practices were observed to the same degree or slightly more often in HL classrooms than EFAL classrooms, with the exception of learners answering predictive and inferential questions, which was observed in 18.7 percent more EFAL classrooms than HL classrooms. 68 Table 29: Observations where strategies for understanding HL and EFAL text were observed Item observed % HL EFAL Learners Encouraged to Retell, Act, Summarize Story 46.2% 43.3% Learners Encouraged to Discuss Illustrations 74.8% 70.0% Teachers Use Illustrations to Aid Understanding of the Text 78.2% 68.3% Learners Identify Main Theme of the Text 57.1% 49.2% Learners Answer Open Ended Questions 63.9% 60.8% Learners Answer Predictive/Inferential Questions 50.4% 69.2% Learners Answer Literal, Evaluative, Appreciative & Critical Questions 54.6% 49.2% When disaggregating by grade, these practices were observed consistently more often in Grade 3 than Grade 1 classrooms. 69 The differences between the EGRS I groups were less distinct in comparison to the RSP, where there was more variation. As in the RSP, teachers were observed supporting learners to develop meaning from a text more often in the most intensive intervention (in this case, Coaching). Although not directly comparable, percentage-wise these strategies were observed more often across the RSP groups than the EGRS I groups. The most noticeable difference was in learners answering predictive/inferential questions, whereby no classrooms in the EGRS I Control group were observed conducting this practice compared to approximately a third (34.1%) of classrooms in the RSP LTSM group. Developing vocabulary and spelling The explicit development of vocabulary is important as the more words children recognize and know the meaning and structure of, the better able they are to understand what they have read. In the RSP classroom observation, teachers from Coaching and SMT classrooms were observed developing vocabulary and spelling more than 80 percent of the time. This was comparatively not seen as often in the LTSM schools. Table 30: Observations where the teaching of vocabulary and spelling were observed Item observed Coaching SMT LTSM There is evidence of vocabulary development during the observation period 80.0% 81.8% 75% There is evidence of development/testing of spelling during the lesson observation 80.0% 81.8% 57.5% Breaking this down by language and grade, there were slightly more observations of vocabulary development in Grade 3 classrooms versus Grade 1 classrooms whereas there was no obvious pattern in spelling development by grade. Overall, there were a similar number of observations of vocabulary development in HL classrooms (79.8%) and EFAL classrooms (77.5%), whereas there was slightly more evidence of spelling development in HL classrooms (76.5%) than EFAL classrooms (69.2%). 70 Table 31: Observations where the teaching vocabulary and spelling were observed by language Language HL EFAL Grade 1 Grade 3 Grade 1 Grade 3 There is evidence of vocabulary development during the observation period 78.0% 81.7% 71.2% 83.6% There is evidence of development/testing of spelling during the lesson observation 78.0% 75.0% 64.4% 73.8% In comparison, in the EGRS I, vocabulary development was observed in 20 percent of Control classrooms, 40 percent of Training classrooms, and 60 percent of Coaching classrooms (EGRS, 2017, p.42). Spelling development was observed in 35 percent of Control and Coaching classrooms, and 60 percent of Training classrooms. Opportunities to write In the Foundation Phase classes, children need to be taught how to hold a pencil or pen and how to form the letters correctly and efficiently with correct direction and spacing. This is referred to as teaching handwriting. In Grade 1, children learn how to write print and by the end of Grade 3, they should have been taught how to write in cursive script. In addition to the mechanics of writing, it is also expected that children are able to formulate their own ideas in writing. This type of writing is about the content and the organization of this content, and not about the technical aspects of forming letters. Per Table 32, when looking across grades and intervention groups, teachers checking whether learners are holding their pens and positioning their writing material correctly was observed more often in Grade 1 across intervention groups. The pattern was the same with teachers correcting learners’ writing position, with the exception of Coaching classrooms where there was a marginal difference between the grades. No Grade 1 learners in SMT classrooms were observed drawing patterns. This practice was observed more often in Coaching classrooms. As expected, the pattern changes with cursive writing, where Grade 3 learners were much more often observed using cursive writing than Grade 1 learners. 71 Table 32: Observations where the teaching of handwriting was observed Item % observed Coaching SMT LTSM GR 1 GR 3 GR 1 GR 3 GR 1 GR 3 Teacher checks that learners are holding pens & positioning writing material 28.6% 19.5% 18.9% 12.8% 28.2% 4.9% Teacher corrects learners when pens/writing materials are not held/positioned correctly 42.9% 46.3% 29.7% 20.5% 23.1% 9.8% All learners involved in writing any cursive text during the lesson 2.4% 41.5% 2.7% 53.8% 2.6% 41.5% Most/all learners drawing patterns 20.0% 25.9% 0.0% 13.3% 15.0% 13.6% Interestingly, when disaggregating by language, teaching of handwriting was observed more often in HL than EFAL classrooms and, with some activities, substantially so (Table 33). Table 33: Observations where the teaching of handwriting by language was observed Item % observed HL EFAL Teacher checks that learners are holding pens & positioning writing material 29.4% 8.3% Teacher corrects learners when pens/writing materials are not held/positioned correctly 37.0% 20.8% All learners involved in writing any cursive text during the lesson 17.1% 13.0% Most/all learners drawing patterns 31.9% 20.0% 72 Copying from the board While it is important that children learn to copy text from the board, it is often hugely time consuming and not of major cognitive benefit, especially in Grade 3. In the classroom observations, between 36.8 and 53.3 percent of classrooms overall had no evidence on writing demands made on all or most learners individually. Of the remainder of the classrooms in which these practices were observed, the prevalence of copying from the board was more or less similar across groups. However, it was higher for copying letters and syllables, plus copying more than ten words, in SMT schools (Table 34). Table 34: Activities where copying from the board by learners is observed Item observed % Coaching SMT LTSM Coloring in 19.0% 7.5% 28.6% Copying a drawing 22.4% 22.6% 26.2% Copying vowels, letters & syllables 36.2% 52.8% 38.1% Copying 1-2 words 3-10 words 10+ words 40.9% 50% 9.1% 32.5% 45% 22.5% 41.2% 50% 8.8% Copying sentences 1-2 3-5 +5 44.4% 38.9% 16.7% 43.6% 38.5% 17.9% 39.5% 44.7% 15.8% In the EGRS I classroom observation (EGRS, 2017, p.44), these practices were observed in 20 percent of schools, or less, with learners copying vowels/letters/syllables, words, and sentences seen much more often in Training and Coaching classrooms than Control classrooms. 73 Writing own words and texts Writing sentences is expected from Grade 1 and far more so in Grade 3. The construction of three to five sentences was seen in 56.3 percent of Coaching classrooms as opposed to 40.5 percent in LTSM classrooms and even lower in SMT classrooms, at 37.5 percent. However writing more than five sentences was seen the most in LTSM classrooms (29.7%), followed by 25 percent of the time in SMT classrooms and only 8.3 percent in Coaching classrooms. The lower rate of observation of teachers requiring children to write five or more of their own sentences, particularly in the Coaching classrooms (only 8.3%), is notable (Table 35). Table 35: Construction of own words and texts by learners Writing item observed Coaching SMT LTSM Own drawing 25.9% 9.4% 28.6% Writing own vowels/letters/syllables 20.7% 37.7% 28.6% Writing words 1-2 3-10 10+ 42.9% 42.9% 14.3% 32.4% 48.6% 18.9% 28.9% 55.3% 15.8% Writing sentences 1-2 3-5 5+ 35.4% 56.3% 8.3% 37.5% 37.5% 25% 29.7% 40.5% 29.7% However, since the writing demands of Grade 3 are higher than those of Grade 1, a comparison between the sentence writing across the grades may be more informative than breaking this down by intervention group alone. Table 36 compares sentence writing in HL, between Grade 1 and Grade 3 across intervention groups. The table illustrates that in classes in Coaching schools, both Grade 1 and Grade 3 classrooms were mostly observed writing three to five sentences, and in SMT Schools Grade 1 classrooms were mostly observed writing one to two sentences and Grade 3 classrooms were mostly observed writing three to five sentences. 74 In LTSM schools, Grade 1 classes were equally split between writing one to two, and three to five sentences, and Grade 3 classes were mostly observed writing three to five sentences. In Coaching and SMT classrooms, no Grade 1’s were observed writing more than five sentences, but 14.3 percent of Grade 1 classes in LTSM schools were observed writing more than five sentences. Table 36: Comparing sentence writing in Grade 1 and Grade 3 HL Item Home Language Coaching GR 1 GR 3 SMT GR 1 GR 3 LTSM GR 1 GR 3 1-2 sentences 44.4% 13.3% 75.0% 6.7% 42.9% 9.1% 3-5 sentences 55.6% 73.3% 25.0% 66.7% 42.9% 54.5% +5 sentences 0.0% 13.3% 0.0% 26.7% 14.3% 36.4% Table 37 compares sentence writing in EFAL, between Grade 1 and Grade 3 across intervention groups. The table illustrates that across intervention groups, the majority of Grade 1 classes were observed writing one to two sentences. In the Coaching intervention, more than half of the Grade 3 classrooms were observed writing three to five sentences (57.1%), and less than half of the Grade 3’s in LTSM classrooms were writing three to five sentences (44.1%). In SMT classrooms, just under a third (30%) of Grade 3 classes were writing three to five sentences, but half (50%) of classes were observed writing more than five sentences. Table 37: Comparing sentence writing in Grade 1 EFAL and Grade 3 EFAL Item EFAL Coaching GR 1 GR 3 SMT GR 1 GR 3 LTSM GR 1 GR 3 1-2 sentences 87.5% 28.6% 83.3% 20% 77.8% 17.6% 3-5 sentences 12.5% 57.1% 0% 30% 22.2% 44.1% +5 sentences 0.0% 15.4% 16.7% 50% 0% 38.2% 75 The learner workbook assessments found a significant difference between the intervention groups in terms of the amount of written material observed in learner workbooks, per Figure 12 below: Figure 12: Comparing Learner workbooks and amount of written material Per Figures 13 and 14, in both the DBE workbooks and learners’ own work, there was significantly more writing in Setswana HL than EFAL. Figure 13: Comparing Setswana HL and EFAL written work in DBE workbooks 8.22 10.06 9.67 Coaching (N = 80) Coaching & SMT (N = 77) LTSM only (N = 81) Average number of pages with work Number of pages in the DBE workbooks involving writing of at least one full sentence p-value: 0.028 10.38 8.21 Home Language (N = 121) EFAL (N = 117) Average number of pages with work DBE workbook assessment - average number of pages with written work Home Language (N = 121) EFAL (N = 117) p-value: 0.016 76 Figure 14: Comparing time spent on Setswana and EFAL written work in learner exercise book Since writing is essential and supports the acquisition of reading, it is critical that children write every day. Interestingly, across all intervention groups and grades (except Grade 1’s in the LTSM intervention), there were more observations of children writing five or more sentences in EFAL versus HL. The greatest difference was observed in the SMT intervention. In SMT classrooms there were no observations of learners in Grade 1 HL classrooms writing five or more sentences, compared to 16.7 percent of Grade 1 EFAL classrooms where learners wrote five or more sentences. This pattern was also evident in Grade 3 classrooms in SMT schools between EFAL and HL whereby in 26.7 percent of Grade 3 HL classrooms, learners were observed writing five or more sentences, compared to Grade 3 EFAL, where in 50 percent of the classrooms, learners wrote five or more sentences. In comparison, the EGRS I found that in Control classrooms, only eight teachers (40%) expected learners to engage in two or more written tasks during the observed lessons, compared to 11 teachers in Training schools (55%) and 17 teachers in Coaching schools (85%), (EGRS, 2017, p.25). The tasks teachers gave learners most often involved writing sentences of three words or less, and copying sentences. The exercises per writing activity in the EGRS I report are reported in the categories of “short sentences,” “long sentences”, “whole paragraphs,” and “longer extended texts,” so are not comparable with the RSP evaluation results (EGRS, 2017, p.27). 53.01 41.01 22.33 36.74 31.77 25.99 14.36 The overall number of days on which the learner completed any exercises The overall number of days the teacher had marked any exercises (usually in pen) The number of days involving writing* exercises completed by the learner The number of days involving exercises comprising the writing of at least one full sentence. Learner exercise book(s) assessment - time spent on written work Home Language (N = 121) EFAL (N = 117) p-value <0.001 77 The learner exercise books were also assessed and reported on differently in the EGRS I classroom observation study vs. the RSP (EGRS, 2017, p.27). The study reported the average number of pages of written work in DBE workbooks in total, while the RSP evaluation reported on the number of pages in DBE workbooks that contained writing of at least one full sentence. These results are therefore not comparable. Opportunities to read Children need multiple opportunities to read every day and they need to have access to various types of reading. This section explores the reading opportunities that researchers observed across the three types of schools. Teacher reading aloud Reading aloud to children is an important part of language, vocabulary, and reading enjoyment. It is also an essential aspect of developing listening skills. The classroom observations noted when teachers would read aloud to the class, but when they did not have the text (so could not follow along), and this was observed less than 50 percent of the time across any of the intervention groups (Table 38). In the Coaching group, the teacher was observed reading aloud (sometimes, mostly, or always) 30.6 percent of the time, 31.2 percent of the time in the SMT group, and 33.8 percent of the time in the LTSM group. Classroom observations also noted when teachers read aloud to the class, but with the class following along silently in their own texts. This is important for children to hear the correct pronunciation of words, to have a sense of how fluent readers pace themselves, and to hear words that they may not have seen before. This kind of reading aloud by the teacher was only observed 32.9 percent of the time in the Coaching group and 37.7 percent of the time in SMT group, but more often (50.1% of the time) in the LTSM group. Table 38: Teacher reading aloud (by intervention) Item Coaching SMT LTSM Teacher reading aloud to the class 30.6% 31.2% 33.8% Teacher lead reading with class following silently 32.9% 37.7% 50.1% When breaking this down by grade (Table 39), there was a negligible difference between Grade 1 and Grade 3 in teachers reading aloud to the class. However, teachers leading reading with the class following silently was observed substantially more often in Grade 3 than Grade 1 classrooms. 78 Table 39: Teacher reading aloud (by grade) Item Grade 1 Grade 3 Teacher reading aloud to the class 33.1% 30.6% Teacher lead reading with class following silently 32.2% 47.9% When breaking this down by language, teachers reading aloud to the class was seen more often in EFAL classrooms (36.7%) than HL (26.9%), while teachers leading reading with the class following silently was observed approximately the same amount of times in both languages (40.8% in EFAL and 39.5% in HL). The EGRS I classroom observation study (EGRS, 2017, p.29) reported that the teacher mostly or always led reading seven out of 20 times in Control classrooms (35%), four out of 20 times in Training classrooms (20%), and seven out of 20 times in Coaching classrooms (35%). In the EGRS I, the teacher mostly or always read aloud to learners without the learners following silently in only three out of 20 observations in the Control classrooms (15%), two out of 20 observations in Training classrooms (10%), and five out of 20 observations in Coaching classrooms (25%). These items appear to have been observed to the same or a lesser degree in the EGRS I classrooms compared to the RSP. However, the EGRS I scores combine the “mostly” and “always” categories whereas the RSP includes the “sometimes” category in the calculation of whether the item was observed or not observed. Thus, these scores are not directly comparable. Children in groups reading aloud Having children read in small groups or pairs is the most effective and efficient way of the teacher assessing who can and cannot read and of providing individual input for reading improvement. Children reading in groups or pairs reading aloud was observed (sometimes, mostly, or always) in 65.9 percent of Coaching classrooms, 63.7 percent of SMT classrooms and 70 percent of LTSM classrooms (Table 40). It is interesting that Group or Paired reading was observed mostly in the LTSM schools and encouraging that SMT and Coaching classrooms were observed applying Group Guided and Paired methodology in more than 60 percent of observed classrooms. It is hoped that all three types of schools will increase their application of this critically important type of reading. A primary objective of Group Guided Reading is to provide children with a reading text at their level to improve fluency and confidence. Therefore, it is expected that different groups have different levelled books. Classroom observations observed when learners were split into groups, and then whether different groups read graded readers of different ability levels. Within 79 the groups or pairs of learners reading aloud, learners were reading graded readers of different ability levels in 40 percent of the Coaching classrooms, 46.8 percent of the SMT classrooms, and 37.5 percent of the LTSM classrooms. In only approximately half of these groups, there were roughly the same number of learners in each group. One would have hoped to see a higher number of differentiated readers in the schools where Coaching is happening but there is not a major difference across the three intervention types. Table 40: Children in groups reading aloud (by intervention) Item Coaching SMT LTSM Learners read aloud together in groups or pairs 65.9% 63.7% 70.0% Different groups read graded readers of different ability levels 40.0% 46.8% 37.5% Grouped learners reading: roughly same number of learners in each group 41.2% 51.9% 50.0% When breaking this down by grade and language, these items were observed consistently more often in Grade 3 than Grade 1 classrooms, and much more often in HL than in EFAL classrooms (Figures 15 and 16). 80 Figure 15: Groups reading aloud (by grade) Figure 16: Groups reading aloud (by language) Comparatively, in eight EGRS I Coaching schools, children were observed “mostly/always” engaging in Group Guided Reading (40%), this was observed in six schools in Training schools 65.3% 33.9% 36.4% 67.8% 47.9% 58.7% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Learners read aloud together in groups or pairs Different groups read graded readers of different ability levels Grouped learners reading: roughly same number of learners in each group Groups reading aloud (by grade) Grade 3 Grade 1 73.1% 48.7% 52.1% 60.0% 33.3% 43.3% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Learners read aloud together in groups or pairs Different groups read graded readers of different ability levels Grouped learners reading: roughly same number of learners in each group Groups reading aloud (by language) EFAL HL 81 (30%), and in one school in the Control schools (5%), (EGRS, 2017, p.29). Again, these results are not directly comparable to those of the RSP - the classroom observation tool also had a “sometimes” observed category, as well as a “never” observed category, and the EGRS I study did not report on these two categories. Individual reading aloud Giving children an opportunity to read aloud in class, while not the most efficient way of hearing children read, is still beneficial. Children reading individually aloud to the class was observed (sometimes, mostly, or always), in 60 percent of observations in Coaching schools, 66.3 percent in SMT schools and 50.1 percent in LTSM schools. When breaking this down by grade and language, this item was, again, observed consistently more often in Grade 3 (63.6%) than Grade 1 (53.4%) classrooms, and much more often in HL (73.1%) than in EFAL (49.2%) classrooms. In the EGRS I results, it is reported that children reading aloud and individually (mostly/always), was observed equally in Control and Training schools, with two schools being observed in each (10% of observations), and in was observed (mostly/always) in four schools in the Coaching intervention (20%) (EGRS, 2017, p.29).19 Individual silent reading (independent reading) Independent silent reading is the ultimate goal of reading and yet it was seldom seen in the classes. It is noted, however, that it is unlikely that a teacher will choose this type of reading when being observed. Table 41: Individual silent reading (by intervention) Item Coaching SMT LTSM Learners read individually silently independently 24.8% 33.8% 35.1% For the Coaching classrooms, it was seen 24.8 percent of the time, 33.8 percent of the time in the SMT classrooms, and 35.1 percent of the time in the LTSM classrooms (sometimes, mostly, or always). This was seen much more often in Grade 3 (43.8%) than Grade 1 (18.6%) classrooms and much more often in HL (54.6%) than EFAL (32.5%) classrooms. The EGRS I classroom observation study only reported mostly/always scores. In the EGRS I, learners reading silently and individually (mostly and always), was observed in two schools in 19 These results are not directly comparable to those of the RSP, because the Classroom Observation tool also had a “Sometimes” observed category, as well as a “Never” observed category, and the classroom observation study on EGRS did not report on these two categories. 82 the Control schools (10%), but was not observed20 in either intervention group (EGRS, 2017, p.29). Individual Guided Reading Individual guided reading practice, where a learner reads aloud so that the teacher can monitor their reading, give comments, make corrections, and ask questions, was also observed. This practice is important and best done during Group Guided Reading where the teacher should listen to each child read aloud for a few minutes. Individual guided reading was observed (sometimes, mostly, or always) in 61.2 percent of Coaching schools, 75.4 percent in SMT schools, and 57.6 percent of the time in LTSM schools. It was seen much more often in Grade 3 (70.2%) than Grade 1 (58.5%) classrooms. However, it was seen more often in EFAL (57.5) than HL (45.4%) classrooms, which is inverse to the pattern in other items within the “opportunities to read” category. Post reading questions The Curriculum and Assessment Policy Statements (CAPS) and good reading practice suggest that teachers conduct pre-reading, during reading and post-reading questions about the text. This three-phase questioning develops understanding of texts. This observation recorded teachers’ use of post-reading questions. Generally, most teachers are using post-reading questions with the majority being in the SMT schools (72.7%), followed by 68.8 percent in LTSM schools and 63.5 percent in Coaching schools. This was seen more often in Grade 3 (72.7%) than Grade 1 (63.6%) classrooms and more often in HL (72.3%) than EFAL (64.2%) classrooms. 4.2.2 SUMMARY FINDINGS 6. Reading mats: Less than 45 percent of all schools had reading mats and there was little difference between the various types of interventions. 7. Word walls: Most classrooms had HL and EFAL word lists while the prevalence thereof and ordering of words into some kind of system was highest in the Coaching schools. 8. Displaying of children’s work: All groups had a low rate of displaying children’s work yet the LTSM classrooms had the highest of the three. 9. Practices that slow down teaching: With the exception of one practice (i.e., rearrangement of desks and chairs), the practices that slow teaching down were seen most often in SMT and LTSM classrooms. 10. Choral reading: In all three groups, there was still choral reading but it is lowest in the SMT classrooms. 20 It might have been observed “sometimes” but the EGRS I classroom observation study does not report on the “sometimes” category in this instance. 83 11. Book knowledge & print concepts: There was great variation across the classrooms with reference to teaching book knowledge and print concepts with no obvious pattern being shown by any one type. Concerning is that there was little difference in the teaching of book knowledge and print concepts across Grade 1 and 3. It is expected that Grade 3 children do not still need to be taught about book covers, spines, front and back and reading direction. 12. Use of English: There was a higher use of English in the SMT classrooms as compared to the Coaching and LTSM classrooms. 13. Teaching reading for understanding: A positive finding of the observations is that the focus of teaching reading with understanding was much higher in the SMT classrooms. However, it is noted that the way the items have been grouped in the instrument are problematic. 14. Vocabulary and Spelling: The explicit teaching of vocabulary and spelling was higher in the Coaching and SMT classrooms than in LTSM classrooms. 15. Writing: Teaching of handwriting and copying from the board was low overall across the interventions. However, teaching of handwriting was consistently observed more often in HL than EFAL classrooms. Learners writing more than five sentences was observed most often in LTSM classrooms. 16. Reading Aloud: There was little difference in the observation of teachers reading aloud to children across all three intervention types. 17. Group Guided Reading: Group or Paired reading was observed mostly in the LTSM schools and SMT and Coaching classrooms were observed applying Group Guided and Paired methodology in more than 60 percent of observed classrooms. Group guided reading practices were observed consistently more often in Grade 3 than Grade 1 classrooms, and much more often in HL than in EFAL classrooms. 18. Post reading questions: The use of post reading questions was highest in the SMT classrooms, and was seen more often in Grade 3 than Grade 1 classrooms, and more often in HL than EFAL classrooms. 84 4.3 IS THERE EVIDENCE OF CHANGED PRACTICES BASED ON THE SMT SUPPORT? The classroom observation findings suggest that there is variable uptake of RSP practices, as evidenced in better practices in the classroom, with SMT classrooms providing more opportunities for language and literacy development than the other groups. Qualitatively, SMT members in almost half the schools (nine out of 20) reported that the SMT training assisted the HODs to improve their work in the classrooms and encourage reporting to SA-SAMS on a weekly basis. Altogether, 64 SMT members took part in SMT group interviews in the 20 sample schools receiving Coaching and SMT interventions. Of the 20 schools, two schools stated that they had not received any form of SMT training. Eighteen of the school SMTs reported that they had received tablets although, as noted in this report, a review of the school asset registers did not corroborate this finding. In response to whether the SMT is doing anything differently as a result of attending the training, the following comments were captured from the SMT members interviewed in this evaluation: Table 42: Self-reported changes in SMT practice due to RSP training Self-reported change in practice Number of schools Improved teaching and learning: training assisted the HODs to improve their work in the classrooms. Training encouraged the use of reporting to SA:SAMS on a weekly basis. RSP meetings are being held in the schools and language policies being streamlined accordingly. Learners taking library books home to read with parents. Educator are encouraged to use LTSM 9 of 20 schools Enhanced support structures: SMTs conduct monitoring visits to supervise and support educators. Educators receive feedback on anything related to assessments, through the moderation of their work, feedback, and suggestions for improvement. 4 of 20 schools Management and Leadership Skills: improved planning and monitoring of school finances, accountability has increased (sessions are held with teachers). Positive change among educators when they experience positive leadership. SMT started developing policies due to the training presentations. 5 of 20 schools *Two of the SMT schools reported that they did not receive SMT training When asked if they could change one thing about the RSP Coaching, what it would be and why, the SMT members provided various recommendations, outlined in Table 43, below: 85 Table 43: SMT recommendations for RSP going forward Recommendations Number of schools Time: the allocation of time to implement all learning activities is limited. The RSP work load/content is too much for the Foundation Phase teachers. “We need more visits from the coach because we are not yet conversant with the RSP.” 8 of 20 school respondents Reduce coaching visits to one per quarter per teacher 1 of 20 school respondent Differentiated lesson plans: because learners are not at the same level in terms of learning and reading, Group Guided Reading should accommodate more learners. In addition, there is need to reduce sight words as it is too much information on a weekly basis. Teaching Phonics for two days is restrictive for learning. 5 of 20 school respondents Training on the use of tablets: in the coaching process there should be a slot allocated to training on tablet use 1 of 20 school respondent Literacy Coach to work with CA/SA: recommendation that the visits to the school should be done in conjunction with the department officials (CA/SA), for example RSP does not explain how assessments should be conducted in certain instances the tracker contradicts with the school assessments 1 of 20 school respondent No need for change: identified the program as functioning positively and influencing both teachers and learners in a positive manner. 2 of 20 school respondents Non-responses to this question 2 of 20 school respondents SMT members were asked if they felt that Foundation Phase teaching had changed as a result of the teacher’s interactions with their literacy coach (Table 44). 86 Table 44: SMT report of changes as a result of teacher-coach interactions Reported change in the school as a result of Teacher-Coach interactions Number of schools Implementation of learning strategies to enhance learning- • Coaches have helped encourage teacher confidence. • Educators have been empowered to teach reading and other components better. • Learners’ reading skills have improved, hence a marked improvement in language teaching in the Foundation phase. • All components of language are covered during lesson presentations. • Learners are able to construct good sentences when given a word and can eloquently read. • Improved time management, example applied in shared reading and Group Guided Reading. 14 of 20 school respondents Negative changes: • There is a focus on the intelligent learners and the slower ones are left behind, teacher always behind with their work schedules and are not able to adhere to routines as suggested in the lesson plans. • Some teachers are of the view that the coach visits should be reduced. • RSP involves a lot of administration, and it negatively affects learners on subjects that are not in RSP. 3 of 20 school respondents Comparison to EGRS I: Although teachers were doing their work better, SMT felt that changes have not occurred, as the program is similar to EGRS. However, there is a positive sentiment to shared reading and Group Guided Reading 1 of 20 school respondent The following are examples of positive changes experienced by the SMT: “We implement all the learning strategies to enhance learning.” SMT member “We have realized a change. Our teachers are a bit confident. We are impressed about the Foundation Phase learners’ performance. That is one of the highlights, even the vocabulary has improved.” SMT member “Educators are now empowered to teach reading and other components better.” SMT member “Accordingly, SMT agreed that coach was knowledgeable and introduced a marked improvement within language teaching in the Foundation Phase. As a result of RSP, LTSM was being properly utilized. The SMT wholeheartedly recommends coaches to 87 other schools based on their lived experiences with a coach in their school. The Principal concluded that they were all happy with coach in their school because coach was reliable, trustworthy, friendly, and easy to work with.” SMT member “Yes, it has changed. We used to help the learners to read but without following any structure. We now follow a proper structure, such as time management, especially when learners are reading books. For example, we apply time management when learners are doing group guided and shared reading.” SMT member There were also negative changes according to SMT members: “RSP is demanding such that we ignore other subjects. Mathematics and Life Skills are suffering. After school we need to stay behind to prepare for the next day and forget other subjects.” SMT member “Although RSP involves a lot of administration and it negatively affects the performance of learners on subjects that are not in RSP. The time allocations per lesson are unreasonable, limiting, and restrictive. When I attended training, the coaches said the lesson plan starts from the day the school opened. The tracker contradicts this. The teachers find it difficult to follow the lesson plans. The program is good but has its disadvantages.” SMT member 88 From the perspectives of 278 teachers in the sample (Figure 17), teachers reported that support from their SMT was generally good across all intervention conditions. Figure 17: Teacher perceptions of SMT support 2.5%8.6%12.60% 1.10% 1.80% 10.8% 11.5% 6.10% 4.30% 5.40% 8.3% 18.0% 20.90% 7.20% 9.40% 47.8% 43.2% 36.70% 38.10% 42.40% 30.6% 18.7% 23.70% 49.30% 41.00% My S MT o b s e r v e s m y l e s s o n s My S MT m o d e l s a n y l e s s o n s I s t r u g g l e t o t e a c h My S MT s u p p o r t s b o o k c l u b s a t t h e s c h o o l My S MT s u p p o r t s a c u l t u r e o f r e a d i n g i n t h e s c h o o l I r e c e i v e s u p e r v i s o r y s u p p o r t f r o m m y S MT Experience with SMT Teacher Survey Respondents Not at all to a limited extent Neutral/Don't know To a large extent To a very large extent N=278 89 4.4 WERE THERE ANY MAJOR DIVERGENCES FROM THE ASSUMPTIONS IN THE THEORY OF CHANGE? The Theory of Change assumptions surfaced in the Design Evaluation remain relevant and appropriate. In some areas, the assumptions are not being met. This includes in the areas of coach training (coaches were not able to execute dry runs during the last coach training), teacher training (fewer teachers attended than expected), LTSM delivery (not all teachers are receiving LTSM and some teachers are not receiving all LTSM), and dosage of Coaching support (see Section 4.1.1). Given the centrality of the Coaching intervention to the RSP, this section reviews the assumptions around coaching outlined in the Design Report against the evidence from the Implementation Evaluation. Coaches performing good coaching practices are adequately planning and preparing for visits to the schools, and engaging appropriately with the school principal, SMT, and the teachers. Weaker coaches are compliant only with reporting but lack skills and experience. Indications are that the coaches are monitoring and checking teacher implementation against the lesson plans and documenting their observations. Without the SBWs, the coaching dosage is currently too low to support differential impact. The unmet dosage assumption remains the greatest threat to the RSP Theory of Change. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: − Coach shadowing observations. − Coach interviews. − Review of coach qualifications. − Coach training observations. − FPD Monitoring Data. − Interviews with district and provincial officials. − Interviews with Head Coaches. − Teacher Surveys. The findings for coaching practice are rated against the ToC activity assumptions, the following is noted (Table 45): 90 Table 45: Theory of Change Assumptions against Findings Assumption Findings Data Source Coaches have the necessary competency and skills to coach (has appropriate teaching, reading and coaching experience, familiar with the lesson plans and LTSM, able to establish good rapport) Partially. Six of the coaches have degrees (either in education or Setswana). Seven of the coaches have diplomas in education. One coach has a post-matric certificates in education and five modules of a degree program. Coaching observation data suggests that coaches are familiar with the lesson plans and LTSM, and many are able to establish good rapport with teachers. While the assumption holds, Coaches are not homogenous in terms of background and experience and may provide variable support to teachers. Coach shadowing observation and interview with coach, review of coach qualifications Coach to school and coach to teacher ratio is manageable No, higher coach to school and teacher ratio than recommended standard Coach shadowing observation and interview with coach, FPD monitoring data The design specification assumes that each school has approximately six Foundation Phase teachers to train. This assumption does not seem to hold. Using the full database of teachers who attended at least one FPD training session, the average number of teachers per school who attended the training, is closer to seven per school. FPD monitoring data Teachers can schedule in coaches, and lesson observation takes place (minimum disruptions occur) Yes, although coaches need to communicate better with the SMT to arrange that lesson scheduling allows for sequential lesson observation. Coach shadowing observation, interview with coach, teacher surveys Teachers present positive/learning attitudes and understand how the coach’s role is different from a curriculum advisor Partially. Teachers generally present positive attitudes to coaches but there is a lack of clarity on roles. Coach shadowing observation and interview with coach, teacher surveys, district and provincial interviews 91 Assumption Findings Data Source The coach conducts needs / school based school workshops every time he / she visits a school No, the FPD monitoring data on SBWs is incomplete and it is likely that the number of sessions that took place is significantly underreported in the monitoring data. An analysis of the coaching observation data reveals that coaches conducted less than 50% of their planned visits during the observation week. Coach shadowing observation, FPD monitoring data The coach is able to have individual coaching sessions with each teacher at least five times per year No. According to the FPD monitoring data, teachers received on average only one individual coaching visit in 2019, and not even 20% of the planned individual coaching sessions were delivered. FPD monitoring data 92 4.5 HOW COULD THE MANAGEMENT AND IMPLEMENTATION STRUCTURES BE IMPROVED? Management and implementation structures should be improved to (1) ensure the proper dosage and coverage is delivered, (2) track implementation and flag issues as they arise, (3) ensure quality of delivery and (4) ensure good reception of the RSP in schools. This section discusses how management and implementation structures can be improved and is broken into the following subsections: LTSM delivery and quality, roles and responsibilities of different stakeholders, reception of RSP in schools (and with the districts and province), and SBWs. To answer this evaluation question, the evaluation team triangulated information from a variety of sources including: − FPD Monitoring Data. − Teacher Group Discussions. − Teacher Survey. − Interviews with Coaches. − Interviews with Head Coaches. − Interviews with district and provincial officials. 4.5.1 LTSM DELIVERY AND QUALITY The evaluation team found evidence of issues with the delivery of the LTSM. Evidence from the teacher group discussions carried out the week of 9 to 13 October, teacher surveys, and classroom observations all show that while each teacher is supposed to receive a full set of lesson plans including all LTSM, not all teachers received all materials. There are not adequate processes in place to check (i) if teachers have received materials in schools where coaching does not take place and (ii) for those teachers who miss training in schools where coaching does not take place, if those teachers have received their materials. The findings also raise a question around whether the LTSM packages contain ALL the RSP LTSM for all grades. The underlying assumption that all teachers receive CAPS-aligned lesson plans and a complete package of quality LTSM for both EFAL and Setswana HL may not be fully met. RECOMMENDATION FOR MANAGEMENT/IMPLEMENTATION: It is advised that teachers check their LTSM packages when they receive them, and that they sign for each LTSM type separately to ensure they receive the full package. The evaluation team recommends that the RSP M&E framework capture whether teachers have received ALL LTSM, note where there are issues, and provide feedback to the LTSM developers. Teachers were generally very happy with the LTSM received, but there could be some improvements made, which include: − Providing audio/CD for songs they have to use in class, rather than USB stick. − Ensuring that lesson plans are more durable, as currently pages tear easily. 93 − Ensuring that lesson plans are checked against DBE workbooks, as teachers have noticed misalignment. The Implementation Evaluation findings support the findings of the Design Evaluation report, which suggest that the lesson plans are demanding in terms of what it requires teachers to cover in each lesson. At least two schools noted that the curriculum itself is challenging for teachers to implement, and that this is reflected in the compacted RSP lesson plans. In the Teacher Survey, 43 percent of teacher respondents (130 of 304) noted that the pace of the lessons was too fast for learners. The implication for the RSP theory of change is that even if the LTSM are delivered, teachers trained, and lesson plans implemented, there may not be sufficient time for learners to absorb the information and learn. This may particularly affect slower learners. RECOMMENDATION: While the pace of the lesson plans is determined by the curriculum, RSP should consider strategies and tactics for the coaches and teachers to address the pace issue. 4.5.2 ROLES AND RESPONSIBILITIES OF HEAD COACHES AND COACHES HEAD COACH ROLES The Coach and the Head Coach relationships are reportedly mostly negative or neutral. The case studies illustrate that the Head Coaches are not providing adequate support to the coaches in the field. Interviews with the coaches confirmed that nine of the 14 Coaches felt that they did not receive adequate support from their Head Coach. The Head Coach role seems to be compliance based, primarily tracking documentation and time management. There appears to be confusion as to what the role of the Head Coaches is. But overall, coaches would appreciate having more contact with the Head Coaches than what they currently receive. “I wish to know their [Head Coach] role or what they should be supporting us on.” Coach 6 “I received feedback as the Head Coach asked why I didn’t have the school-based workshop on my plan. I did not have the workshop mainly this month is Assessment period. The workshops are planned per teacher’s needs.” Coach 6 “…the challenge I see is that our Head Coaches don't have transport to come out to schools. For my Head Coach to come through to me, [they] needs to get a lift from one of the coaches so that he or she can bring her to where I am. And then if anything, I would have to take [them] back to the provincial office.” Coach 8 The findings in the case studies suggest that the Head Coaches have only visited the coaches once since they were trained, and in one case not at all. In addition, the interaction with the Head Coach primarily takes place during provincial or project meetings, and this is when 94 feedback is received and further mentoring or training occurs. The Coaches report that the Head Coaches are not submitting the timetable of visits to the schools, and in some cases, coaches are having to do this directly. There appears to be some confusion as to how this is supposed to occur: in the coach job descriptions, this is their role, however in the Standard Operating Procedures (SOPs) it is the role of the Head Coach. Interviews with Head Coaches show a different picture. Head Coaches indicated21 that their role is to monitor, guide, support and provide feedback to the seven coaches they each supervise. They review the attendance registers, plans, and observation reports submitted by the coaches. In addition, their role is to liaise with the district and provincial DBE officials on the needs of the project and teachers. If the Head Coaches are unable to provide support or address an issue with the coaches, then FPD is involved. One Head Coach indicated that there is no specific training for Head Coaches, they participated in the coach training where they were trained on the tools. They are involved in the ongoing training of the coaches. The Head Coach indicated that they visit the coaches once a month, sometimes twice a month, and they provide different support to the coaches depending on what they need. A Head Coach indicated that the coach reporting tool needed to be improved. One Head Coach indicated that the coaches have improved over the year, and although there are still some areas that coaches need to develop, “They are getting there.” Areas identified that coaches need to develop include facilitation skills to manage their time, materials and to multi-task. The Head Coach further acknowledges that coaches face challenges especially as schools and teachers have competing priorities. They further indicated that the quality of coaching varies across the coaches. The Head Coaches indicated that they receive the monthly plans from the coaches based on the template they provide. The Head Coach reviews and if necessary, they will arrange a meeting with the coach to address the gap such as in lesson observations or reporting. The Head Coach communicates school visits with the district via email and a composite report is submitted to province. Monthly meetings are held with the coaches. As noted earlier in this report, a challenge which affects the level of support the Head Coaches can provide to the coaches is the lack of dedicated transport for each of the Head Coaches. They share one vehicle with the RSP project staff and therefore are unable to visit the coaches at the schools as frequently as necessary. To overcome this, they arrange to go with the coaches but many of the coaches and schools are far away from their office in Lichtenburg. They therefore rely on coaches sending messages via WhatsApp to them and they respond to these to address challenges, questions, or planning issues. The SBWs are also viewed by the Head Coaches as only necessary when there is an identified school need. They indicated that cluster-based school workshops are difficult to hold due to schools being far apart. 21 Evidence in this section is provided from a key informant interview held with each of the two Head Coaches 95 The Head Coaches indicated they work closely with the district CA/SAs: “We work closely with CA/SAs. Even at training as CA/SAs, they [are] always there. We also slot in CA/SAs to present certain topics as they are experts. And before printing they review materials.” Head Coach B RECOMMENDATION: RSP to: − Strengthen the engagement between the Head Coach and Coaches, including confirming roles and engagement with the school for planning purposes. − Clarify Head Coaches’ transportation issue so they can monitor and support Coaches appropriately. − Provide training for Head Coaches. COACH ROLES, RESPONSIBILITIES AND TIMES The Coach shadowing process noted that not all Coaches observe the whole lesson for example: − Some spent limited time on lesson observation (five to ten minutes). − Some spent a little more time during observation (up to 15 minutes). − Some spent the entire lesson with the teacher. Researchers also noted that the coaches do not record their notes on the RSP form as it is too short and restrictive, leaving no space for comments. Instead, coaches write their notes in their notebook and use those notes to give the teachers feedback. All 14 coaches provide feedback to teachers. But the quality of the feedback varies by coach. For example: − One coach only provided feedback session to all the teachers in a group after school. − One coach provided both individual feedback in class and held a school-based workshop after school and showed a video. − One coach interrupted the teacher giving the lesson to demonstrate how it should be done, while the other coaches would wait until the end of the session for this. Nonetheless, in all group discussions, teachers referred to the coaches as very helpful, as the quotes below illustrate. “During the feedback session, the coach is very friendly and shows us where we made mistakes and how to correct them without judgment.” School 2 “It is helpful because as you are talking, the coach shows you your mistakes, the coach shows you how to correct them. And maybe somewhere does some practicals for you on how to do something.” School 6 “…it helps you to rectify what you were not doing right, like my one-on-one session the coach guided me on Group Guided Reading, and I started seeing the light.” School 7 96 “When we blunder, our coach demonstrates for us on the spot so that we can understand what we are doing wrong. Like when I was doing shared reading, the coach took over from me and taught the learners to show me how it should be done.” School 7 “For me I was embarrassed because anyway I was not doing what the coach taught me but I felt it was good for the coach to demonstrate what the coach wanted me to do.” School 7 “… is like a mother to us and learners.” School 4 “… is humble she is approachable she is helpful.” School 4 “… is hands on and does presentation when you do not understand.” School 4 “You are free when [they] is present.” School 4 “With coach interaction is good as compared to the CA/SAs because the coach develops you.” School 6 RECOMMENDATIONS: Further capacity building is required of coaches in the following areas (to be built over the next six months) to ensure teachers get optimum support. These recommendations for RSP include: − Ensure consistency of observations and documentation in expanded coaching tools. − Record deeper insight and detail of observations aligned to teaching and learning methodology. − Improve reflective practice for teachers to reflect on own practice and find solutions for themselves to develop their own action plans. − Promote reading beyond teacher and classroom. − Build skills to assess teacher needs and establish SBWs. − Engage and plan for Professional Learning Communities (PLCs) (perhaps through the SBWs). In addition, it is necessary to address the hours the coaches work to align more closely with the school hours to maximize efficiency of time spent at school and engagement with teachers. 4.5.3 RECEPTION OF RSP IN SCHOOLS The coach shadowing showed that all 14 coaches were welcome in the schools, they are met with cooperation and support from the schools. It was clear that all coaches were familiar with the school environment and had been to the schools before. All 14 coaches acknowledged the positive support from the SMT in the schools in which they work. One of the coaches indicated he was able to communicate freely with the principal and HODs, and that he was allowed to make photocopies and use other resources at the school. 97 In one school, the evaluation team (researcher, coaching expert and quality assurance supervisor) observed that one of the teachers at School 13 did not seem comfortable with the coach, apparently meeting her for the first time. While researchers noted the positive relationships, two of the eight Khulisa researchers stated that “Although the coaches were professional, teachers still saw them as CA/SAs.” 4.5.4 RECEPTION OF RSP AT DISTRICT AND PROVINCE LEVEL Provincial and district officials provided their take on the current management and implementation practices of the RSP (a full list of respondents is available in Annex 4). Three respondents felt the RSP was partially integrated with district and provincial activities because how HL and EFAL are taught has been integrated into the DBE and CA/SAs are involved in the program while another twelve respondents felt that the RSP was fully integrated because provincial officials receive reports on RSP, and officials visit schools that are part of the project. One respondent felt that the RSP provides external support for the CA/SAs and has clear outcomes, regular meetings, and reporting structures. The project report is part of the National Strategy for Learner Attainment report from the province; schedules are received for training and school visits, and the RSP forms part of the Curriculum Forum. In addition, training is reviewed for teacher development. Five respondents indicated that CA/SAs were allocated to the schools for them to support the RSP, which suggests that the RSP may not be as fully part of the provincial structures and processes as described in the interviews. However, the FPD monitoring data indicates that not everyone is fully integrated as per the design specification. For example, only particular provincial and district officials are involved. One of the district officials indicated that the RSP management and implementation structures were working. She attributed this to the RSP appointing staff from within the DBE district system, and therefore they had existing relationships, contacts and understood the operations of education in the district. Consequently, the staff were able to engage officials and implement the project. The communication and existing reporting structure and detail was viewed as appropriate, and she acknowledged that the staff add value to improving reading in the early grades without the District having to support them. This is in contrast to the feedback from CA/SAs who indicated there is a need to improve communication between themselves, district, coaches, and the RSP project management. One CA/SA provided the example of one school having been selected and attended the training, only to be withdrawn from the RSP with inadequate communication and consultation. The involvement of the SMT was noted by another CA/SA as being critical to the ongoing support, ownership, and management of the RSP. On the other hand, another senior district official indicated that given the role and mandate of CA/SAs and Specialists in monitoring the professional development of teachers, it is necessary that there is agreement of agreed standards against which teachers are monitored. She further 98 suggested that the CA/SAs should receive tablets with these agreed standards to ease monitoring and reporting. A further recommendation was for joint training of coaches and CA/SAs as this would strengthen the relationships, build collaboration and to address the concern that “Coaches seem to be working in isolation.” A Head Coach corroborated the need for and to strengthen coaches and CA/SAs working together, and to address the complaints from each group. The District further highlighted that they do call for meetings on the RSP to receive progress and details, and have mandated one official to be the official representative on the project. One of the strengths of the project according to one official is the joint planning sessions and the steering committee meetings where the RSP is discussed in detail. However, another official indicated that reports are received by the Province and District from the CA/SAs on a monthly basis, but these are not official RSP reports – which it appears he would prefer official RSP reports. A second provincial official indicated that the CA/SA reports need improvement and could include input from the coaches. A provincial official indicated that quarterly reports are received from the RSP; however, more frequent reporting is preferable. One official indicated that all schools need to have a coach, and he perceived this to be a gap in the implementation of the project. A further suggestion from another Provincial official was to bring the SMT more on board, as well as bringing other programs together, such as PSRIP and National Education Collaboration Trust (NECT), because the different programs are confusing and “Causing problems.” One of the CA/SAs reported that the RSP project manager in the district did not have the skills to build personal relationships and listen or engage appropriately with the CA/SAs. The CA/SA noted, “He lacks the thing of managing properly. We get brushed off and he shifts us around. We are treated like machines.” As a result, the CA/SAs in this particular district have stepped back from the RSP. RECOMMENDATIONS: RSP needs to: 1. Increase collaboration between coaches and district officials through for example, joint training, and reporting. 2. Increase frequency of official RSP reports to district and provincial officers. 3. Consider how to develop a greater integrated response between programs in the district. 99 4.5.5 SCHOOL-BASED WORKSHOPS The data shows that workshops were held during the coach shadowing weeks, but in an inconsistent way (not in all schools visited) and did not accurately reflect what was on the coaches’ monthly plan. Four of 14 coaches did not hold any SBWs. Table 46 illustrates the number of workshops held by coaches during coach shadowing: Table 46 SBWs planned versus actual during coach shadowing Coach Workshops planned for the days observed (as per coach monthly plan) Workshops held during days of Coach Shadowing Difference Coach 1 4 2 -2 Coach 2 0 0 0 Coach 3 4 1 -3 Coach 4 4 2 (one combined workshop between two schools) -2 Coach 5 4 0 -4 Coach 6 4 0 -4 Coach 7 4 3 -1 Coach 8 3* 2 -1 Coach 9 4 4 0 Coach 10 4 2 -2 Coach 11 0* 1 +1 Coach 12 4 0 -4 Coach 13 0 1 +1 Coach 14 4 1 -3 Total 43 19 -24 * These coaches were only observed for three days. Denominator adjusted accordingly 100 Coaches report that they only conduct workshops when they have seen a common problem, based on the coaches’ discretion, and not fixed to a schedule. However, coaches’ monthly schedules had daily SBWs, as reflected in the table above, and the schedule was not adhered to in ten of the 14 coaches observed. In addition, where workshops are held, they are often used as a joint-feedback session, replacing the one-on-one feedback provided after the lesson observations. But, even when scheduling workshops, coaches face a number of challenges in delivering workshops to teachers such as: teachers being absent or busy with other work commitments, and in certain areas there are limitations regarding public transport, thus teachers use common transport to get home and have to leave school at a certain time. RECOMMENDATION: RSP to put systems in place to ensure that the needs are being correctly diagnosed and that the consequent workshops are held. It is further recommended that the coaches document the training subject. Finally, that Head Coaches monitor these SBWs. Photo: Khulisa, RSP Classroom Library Wall Display 101 4.6 HOW COULD THE M&E SYSTEMS BE IMPROVED? The RSP M&E systems need to be improved from indicator design, to tool design, to data collection and management, and reporting. The Monitoring, Evaluation, and Learning (MEL) plan needs to be updated to reflect the final iteration of the RSP design. Numerous individual improvements are required within the M&E System to produce reliable data for tracking implementation in 2020. 4.6.1 RSP DATA COLLECTION A range of data sources are outlined in the MEL plan, and the forms and data collection instruments for these were verified by the evaluation team. Table 47 indicates the means of verification or data sources that FPD collects. These are documented in the RSP MEL Plan. The evaluation team’s observations on these means of verification are presented in the right hand column. 102 Table 47: RSP Data Sources Data source description as per RSP MEL plan Evaluator Observations 1. Attendance registers for coach, teacher, and SMT training: Each cluster training venue has its own attendance register which the participants allocated to that venue should complete. Ensure that all participants enter all the required fields completely (name, surname, gender, Identity number, designation, grade taught, contact details, signature) against the name of the school. Handwritten entries (particularly names of schools) are strongly discouraged. Blank attendance register forms were used for Terms 1, 2 and 3 training sessions. In Term 4, the teacher training forms were pre￾populated. Teacher training registers do make provisions for completing a designation and grade taught. Registers do not provide for multi-day events. The new pre-populated registers do not capture additional teaches who attend the training. 2. Attendance report form: an excel spreadsheet has been developed for electronic data entry and storage of attendance data. This excel has similar fields to the hard copy attendance register, however it also has fields for coaches/data entry personnel to mark attendance per term. This will make it easy for coaches to have quick access to information on individual teachers in their cluster, as well as track attendance over time. This template will be updated on a quarterly basis after each training has been completed. Data is captured in Excel worksheets named: • Teacher Training Grade 1 • Teacher Training Grade 2 • Teacher Training Grade 3 Only attendance is captured, not attendance per day. Identity (ID) number is used as unique identifier but not always completed. The spreadsheet did not link to a master list of teachers. It does not have a field to indicate if a teacher started late / dropped out of the training program. 103 Data source description as per RSP MEL plan Evaluator Observations 3. LTSM material register: learning and teaching support material will be packaged and distributed per teacher by Grade at the training venues. These include EFAL and HL lesson plans, readers, posters, flash cards, and other supplementary material provided per term. Teachers will sign a register confirming receipt of the LTSM, and this will be used as evidence of distribution. For those LTSM distributed at the cluster training venues where the contact person is not the designated coach; the contact person responsible for receiving material will sign a receipt form, and this will be checked and kept by the coach responsible for the training cluster venue. These will be kept by the coach and used to ensure that all schools and teachers in all grades have received the relevant packages. Shortages will be recorded where applicable, and efforts made to ensure that these are met within two weeks of end of training. LTSM dispatched from the project warehouse will be recorded in a logbook and records filed at the Provincial office. In Terms 1 and 2, a blank form was used for teachers to acknowledge and sign receipt of the FAL and HL LTSM materials. The form was not completed consistently and could not be used to determine LTSM distribution to individual teachers. In Terms 3 and 4, a revised form was used with columns for School name, Teacher name, Lesson Plans HL, lesson plans EFAL and a signature to acknowledge receipt of lesson plans. Coaches were tasked with data collection. However, this was not used to determine LTSM distribution. In October 2019, an LTSM audit was completed by the coaches, in both LTSM and coaching schools to determine which schools received incomplete materials, and which schools did not receive material at all. Data are captured per school in Excel worksheets: • LTSM distributed • Schools with shortages • Schools with no shortages Remedial actions and dates are captured for schools with shortages. 4. Tablet distribution: an asset loan form will be signed by Principals/Deputy Principals, Heads of Department, and District officials receiving tablets. The tablet will be assigned to the school and not an individual and taken care of by the Principals/DH receiving it. Should the Principal/HOD resign or take up a new post elsewhere, it is expected that the tablet will be handed over the new SMT member who will then carry on with project activities. The responsibility for safekeeping of and maintenance of the tablets will remain with the school and designated personnel throughout the life of the project. Form not observed The evaluation team checked if the tablets were documented on the school asset registers, and this was apparent in only a few cases. 104 Data source description as per RSP MEL plan Evaluator Observations 5. Lesson observation forms: will be completed by the literacy before, during, and after conducting a lesson observation. Attached to it, is a teacher feedback form, which will be completed after the observation to structure feedback provision. The teacher can keep a copy where feasible. Head coaches are to review the contents of the lesson observation form for quality assurance and come up with comments for the literacy coach where applicable. Completed forms observed The forms were not always completed in full and the Head Coach did not always complete the form in full. Form does not ask the coach to check that LTSM is available and in use. Data are captured in Excel worksheets named: • Teacher Support Grade 1 • Teacher Support Grade 2 • Teacher Support Grade 3 However, the data is incomplete and not used for formal reporting. Instead, the coach reports are used for reporting, but these are not disaggregated by teacher. 6. In- School support forms: are to be filled in for every individual school visit conducted, on-site at the schools. These should be signed by the requisite personnel in the spaces provided and stamped. Form not observed 7. School support visit log form: A visit log form has been developed and will be signed by the teacher, Principal and coach or international volunteer and will have information on the teacher/SMT supported, time in and out, as well as signatures, and stamped. Form not observed 105 Data source description as per RSP MEL plan Evaluator Observations 8. School Needs-Based Workshop forms: a School Based Workshop (SBW) attendance register will be completed for all SBWs conducted. This will show the date, names of teachers attending the training, school names as well as the contents of the SBW. Completed forms observed. The form includes fields for coaches to complete the grade and language pertaining to the SBW but this is not always completed in full. Data are captured in Excel worksheets named: • Teacher Support Grade 1 • Teacher Support Grade 2 • Teacher Support Grade 3 9. Workshop/Training Evaluation forms: will be handed out, and completed after each teacher training and SMT PLC/workshop and collected by the responsible facilitator. These forms focus on evaluating the quality of training including quality of the facilitation, training materials (presentations etc.) as well as the overall training venue/facility. It is the facilitator’s responsibility to ensure that these forms are completed, and data entered onto a reporting Excel sheet and shared with the M&E advisor after each training. Forms not observed In Terms 1 and 2, coaches were tasked with distributing feedback forms to a sample of 10% teacher who attended the training. In Term 3, focus group sessions were held. Coaches were again asked to nominate participants. Captured data were not observed. 10. Focus Group Discussions (FGD) at teacher training: will be conducted periodically, on a 6-months basis. The FGD will be conducted by trained (part-time) data collectors at selected venues. This information will provide qualitative in-depth feedback on the training as well as additional support where applicable. No Focus Group Discussion Guide was observed In Term 3, focus group sessions were held. Coaches were again asked to nominate participants. Captured data were not observed. 106 Data source description as per RSP MEL plan Evaluator Observations 11. Pre and Post-tests: will be administered to all teachers during training. These will assess basic comprehension of the expected implementation and content of the training. This would include knowing the routine, different components of reading and methodologies. The tests would be administered at the start of training and toward the end of training. The tests will be marked by the partner responsible and results shared with the PMT. The results of these tests will inform implementation. A baseline self-assessment for teachers was not observed. Captured data were not observed. 12. Monthly planning forms: are to be filled in by literacy coaches, international volunteers, and Head Coaches. Literacy coaches are to submit their monthly plans to Head Coaches, and Head Coaches to the Provincial Manager, and ultimately, the Project Manager. The planning forms are a schedule of site visits as they detail the date, activity to be conducted and other information necessary for scheduling purposes. The plans can and should be reviewed on a weekly basis by the team to ensure actual progress is in line with planned, and to revise the plan where necessary. A monthly plan for school visits was observed but not always adhered to. 13. Monthly report forms: are to be completed once monthly, providing an aggregation of all activities conducted over the past month, as well as plans for the next month. These are to allow for timely response to implementation issues on the ground. A completed monthly Head Coach report was observed. Quantitative data per school or per teacher is not recorded. Data are recorded per coach. Narrative data is provided on the successes / issues / challenges at schools. 14. Monthly progress update report: is to be filled in by consortium partners, detailing activities undertaken in the previous month and plans for the next month by the 5th of the following month; using the agreed upon and circulated template. This template is in line with the USAID quarterly report template. This was not observed 107 The table delineating the frequency of data collection does not distinguish between primary data, and reports based on this data. The table indicates primary data sources that are collected on a daily basis, and then indicates how these results are reported on a weekly, monthly, quarterly, and annual basis. Many of the primary data sources listed above are not represented in this list. Table 48: Frequency versus type of RSP data Frequency Type of Information / Data Daily Lesson observation School support visit log data School – based Workshop Head Coach Supervision data Weekly Summary of school visits Feedback from Monthly Aggregation of visits conducted, lessons learned Consolidated project implementation updates from all consortium partners plus report on planned activities Consolidated Provincial progress update plus planned activities for the next month Quarterly USAID Report Annually USAID Annual Report Data flow is inadequately articulated, and challenges in the data flow process results in incomplete data, and inaccurate reporting. The RSP MEL Plan does not articulate where original data collection forms are kept, which information is captured into which database, and the timelines for these are not included in the M&E plan. Interviews with FPD staff indicate that data and reports from the field are not submitted timeously by the coaches, Head Coaches, and provincial coordinators. This leads to incomplete reporting. The evaluation team learnt that some of the data entry and quality control is conducted at the head-office, and that some of the records are kept at provincial level. 4.6.2 THE FPD DATA MANAGEMENT SYSTEM AND PROCESSES As of October 2019, FPD used Microsoft Excel Spreadsheets to manage the entry and storage of its data. A more sophisticated database is necessary to adequately manage relational datasets, and to maintain an audit trail when records are changed. The data extracted for the analysis in this report contained duplicate data, and up-to 108 date master lists of the participating schools and teachers – with an indication when a school or teacher started on the project – is not kept. Currently, the Standard Operating Procedures are not adequate to ensure that data capturing and management is done well. There is no documented procedure for data quality control and this is conducted informally. FPD has commissioned the development of a relational database and capturing system to consolidate data and address the current shortcomings of the data system. User manuals and Standard Operating Procedures will be developed for the new system. The initial system training for coaches will be conducted in December 2019 and a test version of the system will be made available for coaches and capturers to get a feel for the system and provide input towards enhancements. Data from the current data sets will be used to prepopulate the system, thus data will not be recaptured. The set of FPD spreadsheets made available to the evaluation team are represented below – Each Spreadsheet name is listed, together with an indication of the number of records in the list. Training data originate from the attendance registers completed at each training event. Teacher support visit data originate from the reports submitted by coaches. Data about LTSM originate from a material audit coaches completed in October of 2019. The list of data reviewed is graphically portrayed in Figure 18 below. 109 Figure 18: FPD data sources reviewed 110 Listed versus expected units do not match. In some instances, there are discrepancies between the number of units for which data is available in each of the spreadsheets, and the number of units that are expected – For example – 256 schools are listed, but according to specification, 263 should be listed, and according to the DBE, 251 should be listed. See Annex 8 for an investigation of school numbers. There is no process for reconciling discrepancies between the number of expected units and listed units. 4.6.3 RSP MEL PLAN The March 2019 RSP Monitoring and Evaluation plan does not align with the May 2019 RSP design specification. A planned update will be completed by January 2020. The March 2019 Monitoring and Evaluation plan has not been updated to reflect the agreements as per the May 2019 design specification for the RSP. Particularly the targets need to be updated, and in some instances, the indicators too. Important program components not reported in Quarterly indicator reporting table: For example – there is no indicator in the Quarterly indicator reporting table that tracks the delivery of SBWs – yet this data is collected and reported on elsewhere: Figure 19: FPD Presentation on January to September 2019 Implementation Progress 111 Source: FPD Presentation to the EGRS I Improvement Plan steering committee, 2019 11 21 Indicators that do not match the design specification targets: The indicator below, which is not reported in the quarterly indicator reporting table but is included in the indicator specification, provides an example of a mismatch between the target as per the design specification and the M&E plan. The design specification indicates that coaches are expected to visit each school on a monthly basis. The specification assumes that coaches need to visit only once per term. Table 49: RSP Indicator specification example - Coaching Indicator % schools, teachers, language where coach is providing min. 1 supervision visit in past Q [50%] Definition Schools in the coaching intervention, who receive at least one lesson observation visit/support in the past quarter Purpose To determine coverage Numerator Number of schools that received at least one visit in the past quarter Denominator Total number of schools assigned to coaching intervention Disaggregate by Schools, (classrooms, language), grade taught, gender Data Sources Lesson observation forms Responsible FPD, Coaches Consideration Only schools assigned to coaching intervention Source: RSP MEL Plan 2019 (March 2019 version) Incorrect focus of indicators: In the case of the SMT component in the RSP, the indicators are disproportionately focused on ICT capacity development, while this is only small part of the focus of the SMT program. Table 50 below lists the SMT related indicators in the RSP MEL Plan 2019. Table 50: List of SMT related indicators in the RSP MEL Plan # SMT enrolled in SMT course: Number of DHs and Principals/Deputy principals registered for training, and given tablets Number of SMT attending ICT course: Number of DHs and Principals/Deputy principals attending training on ICT Number (and %) of SMT showing improvements in ICT skills/competency: Pre and post training assessments to measure changes in knowledge level % SMT receiving school support visits: Principals and DHs who received a support visit from international volunteers over the past quarter (include planned and unsuccessful visits) Number of SMT attending ICT course: Number of DHs and Principals/Deputy principals attending training on ICT Number (and %) of SMT showing improvements in ICT skills/competency: Pre and post training assessments to measure changes in knowledge level Source: RSP MEL Plan 2019 (March 2019 version) There are no indicators that track how many of the participants received the full dosage of RSP support. 112 The number of teachers trained in Term 1, 2, and 3 is reported, but an indicator that checks how many of the participants attended all the training sessions is necessary to track implementation fidelity. Missing Indicator Specifications: There is not a complete set of indicator specifications for the indicators that are reported on a quarterly basis to USAID. The Table 51 indicates the text of the 21 indicators listed in the Indicator Reporting Table which is submitted on a quarterly basis to USAID. Two versions of the RSP M&E plan were reviewed; a version provided in Feb 2019 and a version provided in July 2019 (Dated March 2019). Table 51 also indicates where no specifications were available. Indicator Inconsistencies: In some instances, there were inconsistencies between the indicators as per the Indicator Reporting Table, and the Indicator specifications in the M&E plans. If indicator specifications were available, the title of the indicator is provided in the table. Table 51: Comparing indicators in two different RSP MEL plan drafts Indicator Number22 Indicator text in Quarterly Indicator Reporting Table Indicator Specification Draft RSP MEL Plan Feb 2019 Indicator Specification Draft RSP MEL Plan March 2019 1 Number of Curriculum Advisors enrolled in literacy course No Specification *Mismatched* J % of Curriculum Advisors completing professional development activities 2 Number of primary school educators enrolled in project (DHS) g Number of teachers enrolled in project, by Grade *Mismatched* D ES.1-6 23 Number of primary or secondary school educators who complete professional development activities with USG assistance 3 Number of education administrators and officials enrolled in professional development activities (principals/deputy principals) j # SMT enrolled in SMT course O # SMT enrolled in SMT course *Mismatched* G ES.1-12 Number of education administrators and officials who complete professional development activities with USG assistance 22 Since the indicators and indicator specifications are not numbered in the Indicator Reporting table or the M&E plans, and the order in which the indicators appear is not the same order in which the indicators specifications are presented, the evaluation team allocated indicator and indicator specification numbers and references. 23 Indicators with a number such as ES. 1-1 Are standard indicators prescribed by USAID. These indicators have more complete indicator reference sheets available. 113 Indicator Number22 Indicator text in Quarterly Indicator Reporting Table Indicator Specification Draft RSP MEL Plan Feb 2019 Indicator Specification Draft RSP MEL Plan March 2019 4 Number of SMT (HODs and principals) attending PLCs No Specification No Specification 5 Number of SMT enrolled in ICT training *Mismatch* m Number of SMT attending ICT course *Mismatch* Q Number of SMT attending ICT course 6 Number (%) of SMT showing improvement in ICT skills/competency n Number (and %) of SMT showing improvements in ICT skills/competency R Number (and %) of SMT showing improvements in ICT skills/competency 7 Number of school support visits No Specification for coaches’ support visits to schools *Mismatch* l % SMT receiving school support visits No Specification for coaches’ support visits to schools *Mismatch* P % SMT receiving school support visits 8 % of planned versus actual school support visits *Mismatch* q % schools, teachers, language where coach is providing min. 1 supervision visit/teacher in past Q [50%] *Mismatch* U % schools, teachers, language where coach is providing min. 1 supervision visit in past Q [50%] 9 Number of schools enrolled in the project, by intervention No Specification No Specification 10 % of learners achieving 50% and more (by language) b Percentage of learners achieving 50% and more (by AHL, EFAL) [>75%] *Mismatch* A ES.1-1 Percent of learners who demonstrate reading fluency and comprehension of grade level text at the end of Grade 2 with USG assistance 11 Number of teachers enrolled in project, by grade and gender g Number of teachers enrolled in project, by Grade L Number of teachers enrolled in project, by Grade 12 Number of teachers trained on scripted lesson plans and LTSM (by grade, term, and gender) h # of teachers trained on scripted lesson plans and LTSM M # teachers trained on scripted lesson plans and LTSM (by grade, term, and gender) 114 Indicator Number22 Indicator text in Quarterly Indicator Reporting Table Indicator Specification Draft RSP MEL Plan Feb 2019 Indicator Specification Draft RSP MEL Plan March 2019 13 Number of teachers receiving LTSM *Mismatch* o # primary textbooks and other teaching and learning material provided with USG assistance *Mismatch* F ES.1-11 Number of primary school classrooms that receive a complete set of essential reading instructional materials with USG assistance 14 Number of coaches who receive initial training No Specification No Specification 15 Number of coaches receiving on-going professional development p Number of coaches who receive on-going professional development S Number of coaches who receive on-going professional development 16 Number of lesson observations conducted, by grade and language (EFAL, AHL), including teacher feedback *Mismatch* r Number of lesson observations conducted by language, grade T # of lesson observations conducted, by Grade and language (EFAL, AHL), including teacher feedback 17 Number (%) planned versus actual lesson observation by school, grade, language No Specification No Specification 18 Number (%) of teachers on track against curriculum tracking tool No Specification No Specification 19 % teachers reporting improved confidence, job satisfaction (by intervention) i % teachers reporting improved knowledge, confidence and job satisfaction N % teachers reporting improved knowledge, confidence and job satisfaction 20 Number (%) of CAs who complete professional development activities e % of CAs completing professional development activities No Specification 21 Average number of learners per teacher, by grade No Specification No Specification Unreported indicators. There are output indicators in the RSP M&E plans for which specifications were developed, that are not reported quarterly to USAID. Some of these indicators are USAID standard indicators which intends to report the cumulative reach at the 115 end of the RSP project, but it is worthwhile to keep track on the progress throughout the program. Table 52: RSP output indicators specified in two RSP MEL plan Type of indicator Indicator Specification Draft M&E Plan Feb 2019 Indicator Specification Draft M&E Plan March 2019 Output – USAID Standard indicator B ES.1-3 Number of learners in primary schools or equivalent non-school based settings reached with USG education assistance Output – USAID Standard indicator c Number of learners reached in reading programs at the primary level with USG assistance C ES.1-5 Number of learners reached in reading programs at the primary level with USG assistance k % SMT completing the project developmental activities Output – USAID Standard indicator o # primary textbooks and other teaching and learning material provided with USG assistance E ES.1-10 Number of primary or secondary textbooks and other learning and teaching materials (LTLM) provided with USG assistance s Number of schools using Information and Communication Technology (ICTs) due to USG support Impact or Outcome indicators: There are a few Impact / outcome indicators in the RSP M&E plans for which specifications were developed, that are not included on the Indicator reporting table to USAID. These should inform the impact evaluation of the RSP and data sources and calculations can be added to the Indicator specifications. Table 53: RSP impact / outcome indicators in two different RSP MEL plans Type of indicator Indicator Specification Draft M&E Plan Feb 2019 Indicator Specification Draft M&E Plan March 2019 Impact / Outcome a % learners who demonstrate reading fluency and comprehension of grade level text at the end of Grade 2 with USG assistance NA Impact / Outcome d % schools that showed improvement with evidence (coaching/correct use of LTSM/capacitated Principal/Deputy Principal & HODs) NA Impact / Outcome f % Curriculum Advisors as showing improvement in Roles and Responsibilities K % Curriculum Advisors showing improvement in Roles and Responsibilities 116 Type of indicator Indicator Specification Draft M&E Plan Feb 2019 Indicator Specification Draft M&E Plan March 2019 Impact / Outcome NA H Difference in learner performance (between baseline and end-line results) Impact / Outcome NA I % schools that showed improvement with evidence (coaching/correct use of LTSM/capacitated Principal/Deputy Principal & HODs) Disagreement in the unit of analysis in indicator and data source: In some instances, the indicator specifications do not clearly outline the data source, and the details of the calculations. The indicator specification below, for example, indicates that the denominator is the total number of SMT enrolled i.e., the individual is the unit of analysis. The data source is a school support visit form, which indicates how many school visits are done. If either the principal or the HOD are visited during a support visit, one support visit will be counted. Table 54: RSP Indicator specification example - SMTs Indicator % SMT receiving school support visits Definition Principals and DHs who received a support visit from international volunteers over the past quarter (include planned and unsuccessful visits) Purpose To determine coverage Numerator Number of SMT who received a school visit Denominator Total number of SMT enrolled in course Disaggregate by Cadre, gender, school Data Sources School support visit forms Responsible FPD, Volunteers Consideration Only SMT enrolled in course, in participating schools. Calculate planned versus actual visits conducted from this data Source: RSP MEL Plan 2019 (March 2019 version) Data sources not updated: The indicator below, for example, indicates that a lesson observation form will be used as the data source. However, an interview with FPD staff indicate that coaches’ reports are used as the data source. 117 Table 55: RSP Indicator specification example – Lesson observations Indicator # of lesson observations conducted, by Grade and language (EFAL, AHL), including teacher feedback Definition A lesson observation when a coach is witnessing a teacher when communicating the learning objectives to learners, both verbally and in writing and concluded with teacher feedback Purpose To ensure that structured learning is occurring in the classrooms as per training received Numerator Number of observations conducted, by language and grade Denominator Disaggregated by Language grade Data Source Lesson observation forms Responsible Coaches, head coaches, FPD Consideration Only schools involved in the coaching intervention will be reported on Source: RSP MEL Plan 2019 (March 2019 version) 4.6.4 FEEDBACK FROM DISTRICT AND PROVINCIAL STAKEHOLDERS One official indicated that there is sufficient monitoring of the teachers by the coaches and CA/SAs. Two CA/SAs indicated that they would like to know in detail what the coaches are doing; not just the planned visits, but what is happening in the schools. One CA/SA recommended a monitoring tool for them so that it is explicit what they need to monitor as part of the DBE. Another CA/SA indicated that the Foundation Phase HODs need a standardized tool to monitor implementation and teacher development. 4.6.5 FPD REPORTING FPD reports their progress on a quarterly basis, using a Quarterly Indicator Reporting Table that accompanies their narrative reports. An investigation into two of the reported indicators, show some areas for further follow-up. Assessment of targets met: Tables 56, 57 and 58 below indicate what is reported for the indicator “Number of school support visits” and “Percentage of planned versus actual school support visits.” These indicators essentially express the same information in different ways. However, the way in which the calculations are done, and targets are interpreted, leads to different assessments – For the indicator: “Number of school support visits” the target was not met in quarter two. But for the indicator “Percentage of planned versus actual school support visits” the target was met and exceeded. No indicator specifications are available for these indicators, to interrogate the calculations. There are indicator specifications for SMT school support visits, but these do not seem to apply since the reported figures indicate how much coach support of some kind had been provided to schools. 118 Table 56: School Support Visits - Extract from RSP Quarter Two indicator reporting table Source: RSP Quarterly Indicator Reporting table September 2019 The comments about meeting targets are presented in more detail below: following: Table 57: Meeting of Targets extract from RSP Quarterly indicator reporting table Source: RSP Quarterly Indicator Reporting table September 2019 A similar disjuncture is found for the fourth quarter reporting 119 Table 58: School Support Visits extract from RSP Quarter 4 indicator reporting table Relationship between indicators are not always clear. Without indicator specifications, it is not clear how the “number of schools support visits” differ from “lesson observations (with feedback)” and “SBWs” A note in the table indicates, however, that even school visits to deliver LTSM (to Coaching schools, or other schools) are included in this reported figure. The figures reported in the Quarterly indicator reporting table do not always tally. Table 59 indicates that there are discrepancies in the way in which the annual cumulative planned value, differs from the sum of the quarterly values. This might be because of missed targets in the second quarter, or because the targets were revised in Quarter Three. It is not made clear in the report. 120 Table 59: Reported figures for the indicator: Number of School Support Visits FY 2019 QUARTERLY STATUS – FY 2019 Annual Cumulative Planned Annual Cumulative Actual QUARTER TWO (January to March) QUARTER THREE (April - June) QUARTER FOUR (July - September) Target Actual % Achieved Note Target Actual % Achieved Note Target Actual % Achieved Note FPD Report W24:1,120 X:N/A A: 420 B: 250 C:60% 25 - Not Met D: 468 E: 406 F: 87% 26Revised Target G: 590 H: 545 I: 93% 27.No assessment Khulisa Comment Should be 1,478 (A+D+G)28 Should be: 1,201 (B+E+H) Calculated as B/A Confirmed Concur Calculated as E/D Confirmed Should read: Target not met Calculated as H/G Confirmed Should read: Target not met Source: RSP Quarterly Indicator Reporting table September 2019 The item W Annual Cumulative planned total appears to be miscalculated. It should actually be 1,478 if the quarterly targets (A+D+G) is added together. It is possible that the inconsistency is related to the note about the revised target in Quarter Three. It is not clear if the annual cumulative target is on par with the expectations in the RSP Design Specification. The item W Annual Cumulative planned total is reported to be 1,120. This amounts to 75 visits per coach (15 coaches) for the nine-month period - or roughly ten visits per month – excluding January and July. There are, however, 22 working days per month. Expressed differently, this comes to roughly eight visits to each of the 130-140 coaching schools over the period January to September 2019. However, it is possible that visits to drop off or audit LTSM at non-coaching schools would also be included in this figure. 24 Letters preceding the reported values were added by the evaluation team, to facilitate ease of reference. 25 Text in the Indicator Reporting table: Target Not Met. School visits in the first quarter were primarily to deliver LTSM to schools, for those that had not received it at the training. Visits were also to conduct SBWs and for introductory meetings between coaches, SMT and teachers. 26 Text in the Indicator Reporting table: Target has been revised to align with the minimum standard as proposed by DBE and adopted by the project. 27 Text in the Indicator Reporting table: The main reason for coaches not meeting their targets is based on targeting; where targets set do not take into account the actual number of days available for coaching (considering own training and preparations and conduct of teacher training), and other disruptions which cannot be anticipated either from the school or the Provincial office 28 Target as revised in Quarter Three not reflected 121 4.6.6 SUGGESTED IMPROVEMENTS Data Management System and processes The evaluation team recommends that FPD continue to pursue the development of a relational database and data entry system. Data should be maintained with an adequate audit trail. This will help to reconcile discrepancies between expected number of units, and the listed number of units in each data source. In terms of Data Management, the implementers need to specify and document the data flow, data quality control processes, version control processes, access control processes, relational data processes, the process of archiving hard copy data, and improve on reporting. The RSP SOPs should be amended to improve the quality of data collected, quality control of collected data, correct capturing of data, quality control on the data capturing process, and better reporting. The SOPs should also specify roles and responsibilities for the M&E system to ensure accountability. RSP MEL plan The RSP MEL plan needs to be revised to align with the May 2019 RSP design specification. The evaluation team suggests that the update ensure that important program components (e.g., SBWs) are included in the Quarterly indicator reporting table. Indicators and targets should be checked to ensure that they align with the RSP design specification. Some of the indicators are incorrectly focused – for example, the SMT indicators seem to be overly focused on ICT skills. The Indicator specifications need to be reviewed – there are missing indicator specifications, indicator inconsistencies, and unreported indicators. Impact and outcome indicators can be specified in more detail since the impact evaluation component has already collected baseline data. The RSP indicators should be reviewed to align the unit of analysis in the data, in the specifications, and the reports. The evaluation team strongly recommends that additional indicators be developed that track how many of the participants received the full dosage of RSP support. FPD Reporting USAID and FPD should identify and correct contradictory reporting about meeting targets, and inconsistencies in the reported figures. When the results are presented, it would help to articulate the relationship between different indicators (such as school support visits, lesson observations, and SBWs). 122 4.7 WHAT ARE THE UNIQUE CHALLENGES OF DELIVERY AT SCALE? Delivering the RSP at scale is affected by a number of factors including the extent of teacher training required, the availability of good coaches, and the logistical arrangements around developing, printing, packaging, and distributing LTSM. Scaling the RSP to three grades and two subjects (Setswana HL and EFAL) simultaneously affects the ability to deliver and monitor RSP interventions. Involving provincial and district officials in the RSP is key to long term sustainability. However, this relationship takes time to build and nurture. Involving provincial and district officials in the process is worthwhile but time consuming both for the officials and for the RSP implementers. 4.7.1 RSP TEACHER TRAINING JUST IN TIME TRAINING One of the most critical, and difficult, aspects of delivering training to teachers at scale is that there is often insufficient time and/or budget to provide direct training from master trainers to teachers. The Design Evaluation found that, as a result, many early grade reading programs turn to a cascade model of training. The RSP has adopted this cascade training approach whereby master trainers’ cascade training to coaches, who then train teachers - called Just in Time (JIT) training. The literature indicates that while cascade training is likely to be more cost-efficient, it is likely that efficiency gains may be offset against losses in terms of effectiveness. This is a necessary strategy when implementing at a larger scale, but it may result in variable quality of training. One of the risks identified in other JIT projects is that JIT training programs tend to focus mainly on content, while pedagogy and coherence between reading methods is typically not adequately covered given the time constraints. The results from the coach and teacher training observations, as well as the classroom observation data, support this concern. Important concepts such as Group Guided Reading were covered in the training presentation to teachers, but there was insufficient time to allow for modelling, practice, and information sharing between teachers. Coaches themselves did not receive adequate instruction on the pedagogical aspects of delivering the lesson plans. Classroom observations found that teachers struggled to implement complex constructs in the classroom. 123 COACH PREPARATION FOR JIT TRAINING The Formative Evaluation findings generally support the findings of the Design Evaluation in the area of coaches’ preparation to deliver JIT training to teachers. However, there were some major divergences, described in more detail in Section 4.1.2 in this report. INVOLVEMENT OF CURRICULUM ADVISOR/SUBJECT ADVISORS S IN JIT TRAINING The CA/SAs interviewed in this evaluation felt they needed to be more involved in the training of the teachers as they are experts in teacher professional development, and are therefore able to pick up on key elements and give appropriate advice. Furthermore, one CA/SA indicated that there should be more joint visits with the coach after the training to build a better relationship and collaborate more with individual teacher development. Two CA/SAs indicated that the district and provincial officials should also be invited to the trainings and coordination meetings. One CA/SA indicated that the teacher training groups needed to be smaller to allow for deeper involvement. CA/SAs in general felt that the coaches were not sufficiently equipped or experienced to conduct training: one indicated that a coach read through the material and did not facilitate it which was unacceptable when training adults and qualified teachers. A few indicated that previous training worked well when they conducted it and having the same group of trainers moving between the sub-districts. A district official indicated that proper planning and consultation for the training of teachers and better coordination is required for implementation. Both CA/SAs and District officials indicated that budgets should cover accommodation and transport of teachers and officials to participate in the training sessions, as many travel long distances to attend. OTHER CONSIDERATIONS Some teachers are not Setswana speaking and to teach HL they must be. The provincial department is currently profiling Foundation Phase teachers to develop Setswana HL teachers as specialists. It is critical to nurture Setswana as a subject, and this requires Setswana specialist teachers. 4.7.2 RSP COACHING RECRUITMENT, SELECTION, TRAINING, AND SUPERVISION OF COACHES, SCHEDULING AND CONDUCTING COACHING SESSIONS To reach scale, there is a need to recruit sufficient coaches to reach schools and teachers while maintaining an appropriate ratio, which was a requirement reported by the province, district, CA/SAs and head coaches. In addition, there is a need to have sufficient Head Coaches in order to increase the frequency of visits to coaches at schools, as reported by a CA/SA and one of the Head Coaches. The coaches recruited should, according to almost all the CA/SAs and indicated by the district officials, have a background in education, experience in Foundation Phase teaching and be trained as a coach. 124 Critical elements for effective coaching practice once recruited, and important for taking the RSP to scale, as reported by district officials and the Head Coaches, include: − Every coach understands the Standard Operating Procedures − Transport is essential for each coach to visit the schools and to be able to access it before school hours in order to get to the school on time for the first lesson, and for Head Coaches − Intensive coach training that all should go through which is done over a number of days (as currently it is too short) to allow for greater explanations, deeper learning and exploring key concepts Head Coach and a CA/SA raised the concern that as numerous teachers have been involved in the EGRS I and therefore know how to teach early grade reading, they should not be included in the RSP coaching – especially as other schools and teachers should be prioritized. Furthermore, one respondent indicated that at some point it is important to let teachers get on with teaching without a coach. RATIO OF COACHES TO TEACHERS According to the Design Evaluation Report (page 73), the RSP ratio of coaches to teachers is 1:60 and the recommended performance standard is 1:30. The coach to teacher ratio is variable, as reported by the coaches. This is between 43 and 68 teachers per coach. The maximum number of schools supported is 14 and the minimum is seven per the Table 60. Furthermore, the total number of schools that the coaches have reported that they support totals 130, this is six less schools than the DBE’s reported number of coaching schools (136). The evaluation team then compared these self-reported numbers with the numbers of schools and number of teachers that the coaches report that they support at the end of each of their monthly plans (listed as a total, not as how many teachers were supported that month). Some coaching plans indicated that coaches supported more or fewer schools than they reported. However, the total between the coaches was 131, still five less than the number of coaching schools reported by the DBE. The total number of teachers that all the coaches reported they support is 753. However, according to their monthly plans the total number of teachers they support is 716. Only four coaches reported the same number of schools and teachers as was in their monthly plans. The RSP ratio exceeds the recommended norm, and this will affect the amount of time that a coach can spend with each teacher. For delivery at scale, it is important to consider the coach ratio as this will affect the dosage and ultimately the quality of the RSP intervention. Table 60: Coach to Teacher Ratio (Self-Reported) Coach Coach self-reported number of Schools that receive coaching Coach self￾reported Number of Teachers coached Coach self-reported frequency of visit per teacher ** process map = 10-15 per year / at least five teachers per day Coach 1 10 56 Three times per term Coach 2 9 43 Three teachers per day Coach 3 8 55 Three times per term 125 Coach Coach self-reported number of Schools that receive coaching Coach self￾reported Number of Teachers coached Coach self-reported frequency of visit per teacher ** process map = 10-15 per year / at least five teachers per day Coach 4 8 46 Two times per month Coach 5 7 50 Two or three times per term Coach 6 8 40 Two or three times per term Coach 7 10 52 Four times per term Coach 8 12 61 Two times per term Coach 9 14 57 One or two times per month Coach 10 10 55 Three times a term Coach 11 7 57 Two times per term Coach 12 11 56 Three times a term Coach 13 8 57 More than four times per term Coach 14 8 68 Once per term TOTAL 130 753 RECOMMENDATION: Given that coaches are inconsistent in reporting the frequency of teacher observation and support, RSP should consider standardizing and capturing the visits by teacher (i.e., not only by school). 4.7.3 RSP MATERIALS (DEVELOPMENT, PRINTING, DISTRIBUTION, USE) There were different views between the DBE officials on how the LTSM can be taken to scale. According to three officials, nothing needs to change. However, an increase in the print run is required as there are not enough materials. Without their own copy, teachers are not able to teach reading effectively. The distribution of LTSM should be efficient and the monitoring of distribution is critical. There is theft of materials at schools and therefore, annually, there is a need to ensure sufficient resources are at each school. It was further noted that it is important that CA/SAs continue to be involved in the development of materials before printing and distribution so they can check the quality, and there is a need for new Big Books. RECOMMENDATION: To take to scale means materials should ultimately be provided for Multi-grade schools, which are not part of the RSP. This will require lesson plans to help the school and teachers improve reading. 126 4.7.4 TEACHERS SEE RSP AS AN “ADDITIONAL REQUIREMENT” NOT AS A DBE REQUIREMENT There was one incident reported of resistance by teachers towards the RSP: one researcher reported that teachers see the RSP as not part of the DBE and thus an add-on to their work. RECOMMENDATION: It is essential that for delivery at scale, that teachers understand that the RSP is an integral and systemic part of their teaching. 4.7.5 INVOLVING PROVINCIAL / DISTRICT OFFICIALS CURRICULUM ADVISOR/SUBJECT ADVISORS BUY-IN AND ROLE CONFUSION The CA/SAs are not clear on the role and boundaries of the coaches, they say they are “Tired of these people [coaches] giving advice to teachers” (quote from one coach during an interview). Similarly, preliminary findings show that teachers are also unclear to the difference between the Coach and CA/SA. “Support provided by the coach is what is liked the most and the thing liked least is the role ambiguity challenges such as not knowing who to listen to between the coach and the subject advisor.” School 3 The CA/SAs indicated that for the RSP to expand and remain sustainable, it is necessary to train the CA/SAs as this will increase support and ownership for the RSP, and there after the CA/SAs train the teachers with the support of the coaches. Two provincial officials highlighted that this would contribute to the sustainability of reading support in the district, as officials will be placed in the driving seat of improving reading support and break the silo approach between the coaches and CA/SAs. It is important to clarify the roles of CA/SAs, coaches and Head Coaches before taking the RSP to scale. District and CA/SAs noted that sharing of information to enhance learning is important. One CA/SA indicated that there is “Nothing in it [the RSP] for me,” and that this needs to be addressed with more focused engagement between the RSP and DBE before the project is taken to scale. It was suggested by one DBE official that there be greater involvement of provincial and district officials in the RSP and that this could be achieved through the Provincial Curriculum Forum. This Forum was deemed a necessary mechanism for scale up. The official further indicated that there is a need to create a Steering Committee at district level for the involvement of district and provincial officials. In a number of interviews with the provincial, district, and CA/SAs, respondents indicated the necessity of integrating reading support projects that are running in the province through forming a consortium which will reflect the Minister’s approach to the NECT, and that an integrated approach will increase support enormously and reduce role confusion. One provincial official noted that the barriers to effective teaching should also be addressed, such as overcrowding classrooms, for the RSP to be more efficient. The size of classes should be taken into account for scaling the project, as this will affect coach to teacher ratios, distribution of materials, teacher training and sub￾district engagement. RECOMMENDATION: For delivery at scale, it will be important to ensure that there is clarity of roles and boundaries, and that the value of coaching is understood and accepted by CA/SAs in addition to the Teachers, Principals, and SMT’s. 127 5. SUMMARY RECOMMENDATIONS The following are the recommendations that are integrated in the report. They are summarized by topic of recommendation, not in the order of importance. 5.1 RECOMMENDATIONS FOR MONITORING AND EVALUATION This evaluation identifies a number of recommendations for improving the M&E for the RSP 1. The evaluation team recommends that FPD continue to pursue the development of a relational database and data entry system. Data should be maintained with an adequate audit trail. This will help to reconcile discrepancies between expected number of units, and the listed number of units in each data source. 2. In terms of Data Management, the implementers need to specify and document the data flow, data quality control processes, version control processes, access control processes, relational data processes, the process of archiving hard copy data, and improve on reporting. 3. The RSP Standard Operating Procedures should be amended to improve the quality of data collected, quality control of collected data, correct capturing of data, quality control on the data capturing process, and better reporting. The SOPs should also specify roles and responsibilities for the M&E system to ensure accountability. 4. The RSP MEL plan needs to be revised to align with the May 2019 RSP design specification. The evaluation team suggests that the update ensure that important program components (e.g., SBWs) are included in the Quarterly Indicator Reporting Table. Indicators and targets should be checked to ensure that they align with the RSP design specification. Some of the indicators are incorrectly focused – for example, the SMT indicators seem to be overly focused on ICT skills. 5. The Indicator specifications need to be reviewed – there are missing indicator specifications, indicator inconsistencies, and unreported indicators. Impact and outcome indicators can be specified in more detail since the impact evaluation component has already collected baseline data. The RSP indicators should be reviewed to align the unit of analysis in the data, in the specifications, and the reports. The evaluation team strongly recommends that additional indicators be developed that track how many of the participants received the full dosage of RSP support. 6. USAID and FPD should identify and correct contradictory reporting about meeting targets, and inconsistencies in the reported figures. When the results are presented, it would help to articulate the relationship between different indicators (such as school support visits, lesson observations, and SBWs). 5.2 RECOMMENDATIONS FOR LTSM The RSP LTSM are valued by teachers and many are used in RSP classrooms. There are a number of recommendations for the development and delivery of RSP LTSM as follows: 128 1. It is advised that teachers check their LTSM packages when they receive them, and that they sign for each LTSM separately to ensure they receive the full package. 2. The evaluation team recommends that the RSP MEL framework capture whether teachers have received ALL LTSM, note where there are issues, and provide feedback to the LTSM developers. 3. While the pace of the lesson plans is determined by the curriculum, RSP should consider strategies and tactics for the coaches and teachers to address the pace issue. 4. The LTSM need to be quality reviewed – both in terms of durability of materials and content. a. Some teachers reported that the lamination was too hard to allow teachers to put RSP LTSM up as the weight of the lamination pulled the materials off the classroom walls. b. The use of Sesotho words in the lesson plans was raised as an issue for some teachers, as was the use of incorrect Setswana words in the materials. 5.3 RECOMMENDATIONS FOR HEAD COACHES The role of the Head Coach is critical for providing timely, adequate support to Coaches. The Formative Evaluation Findings suggest that the following recommendations need to be implemented: 1. Strengthen the engagement between the Head Coach and coaches, including confirming roles and engagement with the school for planning purposes. 2. Clarify Head Coaches’ transportation issue so they can monitor and support Coaches appropriately. 3. Provide training for Head Coaches. 5.4 RECOMMENDATIONS FOR COACHING The findings suggest that RSP should: 1. Revise Coach Job descriptions/contracts to state that they should be at school by 7:30 a.m. (rather than start work at 8:30 a.m.), and accordingly ensure that their vehicles can be collected from the parking lots in time. 2. Ensure consistency of observations and documentation in expanded coaching tools. 3. Record deeper insight and detail of observations aligned to teaching and learning methodology. 4. Improve reflective practice for teachers to reflect on own practice and find solutions for themselves to develop their own action plans. 5. Promote reading beyond teacher and classroom. 6. Build skills to assess teacher needs and establish SBWs. 7. Engage and plan for PLCs (perhaps through the SBWs). 8. Address the hours the coaches work to align more closely with the school hours to maximize efficiency of time spent at school and engagement with teachers. 129 9. Put systems in place to ensure that the needs are being correctly diagnosed and that the consequent workshops are held. It is further recommended that the coaches document the training subject. Finally, that Head Coaches monitor these SBWs. 10. Given that coaches are inconsistent in reporting the frequency of teacher observation and support, RSP should consider standardizing and capturing the visits by teacher (i.e., not only by school). 11. The evaluation team recommends that, for 2020, FPD, and the DBE review the time set aside for coach training and consider how much time is required for adequate training. This should be separated into time required for teaching coaches how to coach (this may be covered by the upcoming coach training course) and to the time required for coaches to familiarize themselves with the LTSM and what is required of them to train teachers in the use of that LTSM. Additionally, coaches need to be advised that they will need to complete a dry run at training, and should receive a copy of the program as well as the training slides and materials in advance of the training. 12. The evaluation team recommends that, for 2020, sufficient time be dedicated to cover both EFAL and HL, that concepts such as shared reading and Group Guided Reading should be covered in separate sessions with links created between the sessions. 13. Schools need to be taught about the importance of having and using a reading mat in the Foundation Phase. 14. Teachers need to be advised about the importance of displaying children’s work. 15. A closer observation needs to be conducted as to why the SMT schools have a higher incidence of practices that slow down teaching. 16. Teachers need to be coached about the importance of hearing children read individually or in small groups. 17. Far more coaching needs to be done on assisting teachers with Group Guided Reading. 18. It should be noted that the teaching of basic book knowledge is only required in Grade R and 1 and does not need to be repeated every day. More time needs to be spent on reading for understanding, rather than a focus on technical aspects of print, especially in Grade 3. 19. Teaching reading for understanding is essential but more clarity is needed on the types of questions teachers ask and the ways they facilitate interaction with the text. This needs to be reflected in the observation tool. 20. Teachers need to be encouraged to teach vocabulary explicitly and consistently. 21. More focus needs to be placed on coaching teachers to help children write their own texts. This should be happening in Grade 1 already. 22. Teachers should be coached on the importance of reading aloud to children (not as in Shared reading but as in a Read Aloud). 23. Teachers need more guidance to choosing graded readers, allocating learners to various ability groups, and facilitating reading within groups. 130 24. Teachers need to ask pre-during and post questions and the observation instrument should reflect these three aspects of questioning. 5.5 RECOMMENDATIONS FOR DISTRICT AND PROVINCIAL INTEGRATION Specific recommendations emanating from interviews with district and provincial respondents suggest that the RSP needs to: 1. Increase collaboration between coaches and district officials through for example, joint training, and reporting. 2. Increase frequency of official RSP reports to district and provincial officers. 3. Consider how to develop a greater integrated response between programs in the district. 5.6 RECOMMENDATIONS FOR SCALE UP Interviews with district, provincial, and CA/SAs indicated a number of specific recommendations for scaling up the RSP. In summary, the RSP should: 1. Train the CA/SAs as this will increase support and ownership for the RSP, and break the silo approach between the coaches and CA/SAs. 2. Ensure that there is clarity of roles and boundaries, and that the value of coaching is understood and accepted by CA/SAs in addition to the Teachers, Principals, and SMTs. 3. Provide materials to multi-grade schools, which are not part of the RSP. This will require lesson plans to help the school and teachers improve reading. 4. Ensure that teachers understand that the RSP is an integral and systemic part of their teaching. 5. Share information to enhance learning. 6. Have more focused engagement between the RSP and DBE before the project is taken to scale. 7. Engage the Provincial Curriculum Forum as a necessary mechanism for scale up. 8. Create a Steering Committee at district level for the involvement of district and provincial officials. 131 6. BIBLIOGRAPHY Baker, L. Breher, MJ. & Guthrie, JT. (Eds) (2000) Engaging Young Readers: promoting achievement and Motivation. Guilford Press. Bean, R. M., Draper, J., Hall, V., Vandermolen, J., & Zigmond, N. (2010) Coaches and coaching in Reading First Schools: A reality check. The Elementary School Journal, 111(1), 87-114. Bold, T., Kymani, M., Mwiru, G., Ng’ang’a, A., & Sandefur, J. (2013) Scaling up what works: Experimental evidence on external validity in Kenyan education. Working paper no. 321, Center for Global 318 Journal of Educational Change (2018) 19:293–3211 3. Development, Washington, DC. Retrieved June 18, 2018, https://www.cgdev.org/publication Buckley, P., Moore, B., Boardman, A. G., Arya, D. J., & Maul, A. (2017). Validating a Fidelity Scale to Understand Intervention Effects in Classroom-Based Studies. American Educational Research Journal, 54(6), 1378 -1413. https://doi.org/10.3102/0002831217726522 Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation science: IS, 2, 40. DOI: 10.1186/1748-5908-2-40 Century, J., Rudnick, M., & Freeman, C. (2010). A Framework for Measuring Fidelity of Implementation: A Foundation for Shared Language and Accumulation of Knowledge. American Journal of Evaluation, 31(2), 199–218. https://doi.org/10.1177/1098214010366173 Churr, C. (2015) Realization of a child's right to a basic education in the South African school system: Some lessons from Germany, PER Volume 18 number 7 http://dx.doi.org/10.4314/pelj.v18i7.01 Available from: http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S1727-37812015000700002 Clarke, J., & Dede, C. (2009) Design for scalability: A case study of the River City curriculum. Journal of Science Education and Technology 18(4), 353-365. Coburn C. E. (2003) Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher 32(6), 3-12. Coenders, F., & Verhoef, N. (2019) Lesson Study: professional development (PD) for beginning and experienced teachers, Professional Development in Education, 45:2, 217- 230, DOI: 10.1080/19415257.2018.1430050 Davies, R. (2013) Planning evaluability assessments: A synthesis of the literature with recommendations. Working Paper 40. London: UK Department for International Development. Deacon, R. (2016) The initial teacher education research project: Final report Johannesburg: JET Education Services: www.jet.org.za 132 Department of Basic Education (2017). Technical Report: The Early Grade Reading Study: Classroom observation study. Pretoria, South Africa. Department of Basic Education (12 June 2018) Technical Report: The Early Grade Reading Study: Impact evaluation after two years of interventions. Pretoria, South Africa. Department of Basic Education (August 2017) Summary Report Results of Year 2 Impact Evaluation: The Early Grade Reading Study (EGRS). South Africa. Department of Basic Education, Annual Report 2017/2018. Available from: https://www.gov.za/sites/default/files/gcis_document/201810/dbe-201718-annual-report.pdf Department of Basic Education. (December 2017) Response to release of Progress in International Reading Study 2016 results. South Africa. Available from: https://www.education.gov.za/Newsroom/MediaReleases/tabid/347/ctl/Details/mid/5986/ItemID/5522 /Default.aspx Department of Higher Education (5 December 2018) 2018 Global Competitiveness Fact Sheet. South Africa. DevTech Systems, Inc. (2017) Northern Education Initiative Plus Mid-term Evaluation. https://pdf.usaid.gov/pdf_docs/PA00N894.pdf Dubeck. M.M. & Gove. A. (2015) The Early Grade Reading Assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development 40 p.315–322. Elish-Piper, L., & L'Allier, S. K. (2011) Examining the relationship between literacy coaching and student reading gains in Grades K to 3. The Elementary School Journal, 112(1), 83e106. http://dx.doi.org/10.1086/660685. Elliott, T.A. (2016) An overview of education and drama in South Africa, in Research in Drama Education: The Journal of Applied Theatre and Performance, 21:1. Pp.57- 59, DOI: 10.1080/13569783.2015.1127150 Fleisch, B. (2016). System-wide improvement at the instructional core: Changing reading teaching in South Africa. J Educ Change 17, 437–451. https://doi.org/10.1007/s10833-016- 9282-8. https://resep.sun.ac.za/wp-content/uploads/2017/10/ReSEP-WP-05-2016- Hoadley.pdf Fleisch, B. & Schöer, V. (2014) Large-scale instructional reform in the Global South: insights from the mid-point evaluation of the Gauteng Primary Language and Mathematics Strategy. SA Journal of Education, Vol 34, No 3. Fleisch, B., Schöer, V., Roberts, G., & Thornton, A. (2016) System-wide improvement of early-grade mathematics: New evidence from the Gauteng primary language and mathematics strategy. International of Educational Development, 49, 157–174. https://doi.org/10.1016/j.ijedudev.2016.02.006 Fleisch, B. & Dixon, K. (Forthcoming) Identifying mechanisms of change in the early Grade Reading Study in South Africa in South African Journal of Education. 133 Funnell, S.C. & Rogers P.J. (2011) Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass. Green, Parker, Deacon and Hall. (2011) South African Journal of Childhood Education 2011 1(1): 109-122. Guskey, T. R. (2002). Professional development and teacher change. Teachers and teaching, 8(3), 381-391. Guthrie, J. T., Wigfield, A., Barbosa, P., Perencevich, K. C., Taboada, A., Davis, M. H., & Tonks, S. (2004. Increasing reading comprehension and engagement through concept￾oriented reading instruction. Journal of educational psychology, 96(3), 403. Hanushek, E.A. & Wößmann, L. 2007. “The Role of Education Quality in Economic Growth.” Policy Research Working Paper 4122, World Bank, Washington, D.C. http://www￾wds.worldbank.org/servlet/ WDSContentServer/WDSP/IB/2007/01/29/000016406_20070129113447/Rendered/PDF/wp s4122.pdf. Hartell, C. G., Steyn, M. G., & Chetty, M. (2015) Towards Equality and Equity in Education: Assessing an Initiative to Strengthen Teacher Professional Development in South Africa (SPECIAL ISSUE: Equality and Equity in Education). 17(2), 73-93. Hayes, D. (2000) Cascade training and teachers' professional development. ELT Journal Volume 54/2: Oxford University Press. Hivos TOC Guidelines (n.d.). Theory of Change Thinking in Practice https://knowledge.hivos.org/theory-change-guidelines. Hoadley, U. (2012) What do we know about teaching and learning in South African primary schools? Education as Change, Taylor & Francis. Howie, S.J., Combrinck, C., Roux, K., Tshele, M., Mokoena, G.M., and McLeod Palane, N. (2017) PIRLS Literacy 2016: South African Highlights Report. The Centre for Evaluation and Assessment, Faculty of Education, University of Pretoria http://www.saep.org/wp￾content/uploads/2017/12/pirls-literacy-2016-hl-report-3.zp136320.pdf ILO / Ballester, T. & Esheikhi, A. 2019. ILO Working Paper Number 29: The Future of Work: A Literature Review. Joyce, B. & Showers, B. (1982) The coaching of teaching. Educational Leadership, 40(1):4-8. Kasch J., van Rosmalen P., Kalz M (2009). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science (23)9, 845-867. Khulisa Management Services (Pty) Ltd. (2015) Performance Evaluation of USAID/Malawi Early Grade Reading Activity. https://pdf.usaid.gov/pdf_docs/PA00KBNS.pdf Khulisa Management Services, November 2019, Design Evaluation of the Reading Support Project commissioned by USAID https://pdf.usaid.gov/pdf_docs/PA00WJHJ.pdf 134 Kim, Y.-S. G., Boyle, H. N., Zuilkowski, S. S., & Nakamura, P. (2016). Landscape Report on Early Grade Literacy. Washington, D.C.: USAID. Lynch School of Education International Study Center. (2017) What Makes A Good Reader: PIRLS 2016 International Results in Reading. Downloaded from https://nicspaull.files.wordpress.com/2017/12/p16-pirls-international-results-in-reading.pdf McIlwraith, H. (ed.) (2013) The Cape Town Language and Development Conference: Looking beyond 2015; Language Rich Africa: Policy dialogue https://www.britishcouncil.org.za/sites/default/files/language_rich_africa_policy_dialogue_brit ish_council.pdf McLean, R & Gargani, J. (2019) Scaling Impact: Innovation for the public good. Routledge & International Development Research Centre. Mohangi, K., Krog, S., Stephens, O. & Nel, N. (2016). Contextual challenges in Early Literacy Teaching and Learning in Grade R Rural Schools in South Africa, in Per Linguam: A Journal for Language Learning, 32(1); 71-87. University of South Africa. Murray, S. (2002) Language issues in South African education: An overview. In R. Mesthrie (Ed.), Language in South Africa, Cambridge University Press: Cambridge. National Education and Development Unit. (February 2018) NEEDU Policy Brief: Habits of Highly Effective Instructional and Facilitative Leaders: Leadership Practices That Support Teaching and Learning. Available from: https://www.education.gov.za/Portals/0/Documents/Publications/NEEDU%20POLICY%20B RIEF%20SERIES/Policy%20Brief%208-Instructional%20leadership_.pdf?ver=2018-07-02- 103215-710 National Institute of Child Health and Human Development. (2000) National Reading Panel Report: Developing early literacy: report of the national early literacy panel, Lonigan & Shanahan. Oliveira, C., Lopes, J. & Spear-Swerling, L. (2019) Teachers’ academic training for literacy instruction. European Journal of Teacher Education, Volume 42, 3. Organization for Economic Cooperation and Development (OECD) (2016) PISA Results. Organization for Economic Cooperation and Development (OECD) (July 2015) OECD Economic Surveys: South Africa Overview. Patton, M.Q. (2008) Utilization-focused evaluation. Thousand Oaks, CA: Sage publications. Pflepsen, A. (2018) Coaching in Early Grade Reading Programs: Evidence, experiences and recommendations. A Global Reading Network Resource. Prepared by University Research Co., LLC. (URC) under the Reading within REACH initiative for USAID’s Building Evidence and Supporting Innovation to Improve Primary Grade Assistance for the Office of Education (E3/ED). Available at www.globalreadingnetwork.net Piper, B. & Zuikowski, S.S (2015) Teacher Coaching in Kenya: Examining instructional support in public and non-formal schools. 135 Piper, B., Destefano, J., Kinyanjul, E.M & Ong’ele, S. (2018) Scaling up successfully: Lessons from Kenya’s Tusome National Literacy program. Journal of Educational Change 19, 293-321. Pouezevara, S. R. (Ed.) (2018) Cultivating dynamic educators: Case studies in teacher behavior change in Africa and Asia. (RTI Press Publication No. BK-0022-1809). Research Triangle Park, NC: RTI Press. https://doi.org/10.3768/rtipress.2018.bk.0022.1809. Primary School Reading Improvement Programme (PSRIP). http://nect.org.za/publications/case-studies/psrip_booklet.pdf/view. Progress in International Reading and Literacy Study (PIRLS) (2016) http://pirls2016.org/pirls/summary/ Reeves, C., Heugh, K., Prinsloo, C. H., Macdonald, C., Netshitangani, T., Alidou, H., & Herbst, D. (2008) Evaluation of literacy teaching in the primary schools of the Limpopo Province. Pretoria, South Africa: Human Sciences Research Council in Association with the Department of Language Education at the University of Limpopo. Rutgers L. (2012) Coaching Foundation Phase Literacy Teachers as leaders in a school in the Western Cape Province: A Professional Development Strategy http://scholar.sun.ac.za Sailors, M., Hoffman, J. V., Matthee, B., Pearson, P. D., & Beretvas, N. S. (2007) Evaluating the impact of the “Learning for Living” project on the development of literacy skills of learners in low-income communities in South Africa. The enterprise of education, 121- 143. Shin, J.K. & Crandall, J. (2014). Teaching Young Learners English: from Theory to Practice. Boston, MA: National Geographic Learning & Heinle Cengage Learning. Snow, C.E.; Burns, M.S. & Griffin, P. (1998). Preventing Reading Difficulties in Young Children. Washington D.C.: National Research Council. Social Impact, Inc. (2013) Midterm performance evaluation of the USAID/Jamaica basic education project: in support of the Jamaica Education Transformation Project (ETP). https://pdf.usaid.gov/pdf_docs/PDACX310.pdf Sonmark, K., Revai, N., Gottshalk, F., Deligiannidi, K. & Burns, T. (2017) Understanding teachers’ pedagogical knowledge: report on an international pilot study. OECD Education Working Papers, No. 159, OECD. Paris. South African Book Development Council (SABDC) & the DG Murray Trust. (2016) National survey into the reading and book reading behavior of adult South Africans. TNS, South Africa. Available from: http://sabookcouncil.co.za/wp-content/uploads/Final-Report-NRS￾2016.pdf Spaull, N. (2017) The unfolding reading crisis: The new PIRLS 2016 results. Available from: https://nicspaull.com/2017/12/05/the-unfolding-reading-crisis-the-new-pirls-2016-results/ Posted on December 5, 2017 136 Stanovich, Keith E. (1986) Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 22, 360-407. Stoop, Chrizell. (2017). Children's rights to mother-tongue education in a multilingual world: a comparative analysis between South Africa and Germany. PER: Potchefstroomse Elektroniese Regsblad, 20(1), 1-35. https://dx.doi.org/10.17159/1727-3781/2017/v20n0a820 Synthesis of Findings and Lessons Learned From USAID-Funded Evaluations (March 2, 2018) Education Sector, 2013-2016. https://pdf.usaid.gov/pdf_docs/PA00T1HD.pdf Taylor, S., Cilliers, J., Prinsloo, C., Fleisch, B., & Reddy, V. (2017). The Early Grade Reading Study: Impact evaluation after two years of interventions. Retrieved online https://www.jet.org.za/clearinghouse/projects/primted/resources/language-and-literacy￾resources-repository/egrs-technical-report-13-oct-2017.pdf The University of Florida Lastinger Centre for Learning, Learning Forward & Public Impact. Coaching for Impact: Six Pillars to Create Coaching Roles That achieve their potential to improve Teaching and Learning. The William & Flora Hewlett Foundation. (2014). Learning to improve Learning: Lessons from Early Primary Interventions and Evaluations in India and Sub-Saharan Africa. https://www.hewlett.org/wp-content/uploads/2016/08/2014-02- 14_Learning%20to%20Improve%20Learning%20Synthesis% 20for%20Publishing_Edited_0.pdf Theory of Change Academy (n.d.) https://changeroo.com/TOC-academy/posts/expert-TOC￾quality-audit-academy. Trialogue South Africa. https://www.trialogueknowledgehub.co.za/index.php/education￾overview United Nations Children Fund (UNICEF) South Africa. Education Budget South Africa 2017/2018. Available from: www.unicef.org/southafrica United States (2019). US Government Education Policy. United States. (2018). Draft USAID Education Policy. USAID (2018) Report: Coaching in Early Grade Reading Programs: Evidence, Experiences and Recommendations. USAID (2019) Early Grade Reading Barometer. Available from: https://earlygradereadingbarometer.org/ USAID (March 2, 2018) Synthesis of Findings and Lessons Learned from USAID-Funded Evaluations: Education Sector, 2013-2016... https://pdf.usaid.gov/pdf_docs/PA00T1HD.pdf USAID. 2013. Midterm performance evaluation of the USAID/Jamaica basic education project: in support of the Jamaica Education Transformation Project (ETP). Social Impact, Inc. https://pdf.usaid.gov/pdf_docs/PDACX310.pdf 137 Van der Berg, S., Burger, C., Burger, R., de Vos, M., du Rand, G., Gustafsson, M., Moses, E., Shepherd, D., Spaull, N., Taylor, S., van Broekhuizen, H., & von Fintel, D. (2011) Low Quality Education as a Poverty Trap. Stellenbosch: University of Stellenbosch. Verhoeven, L. & Snow, C. (2001) Literacy and Motivation. Lawrence Erlbaum Association Publishers, London. Weaver, L. & Cousins, J.B. (2004) Unpacking the participatory process. Journal of Multidisciplinary evaluation, 1(1), pp.19-40. World Bank. (2007) Refer to Hanushek & Wößmann. World Bank. (2018) World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. doi:10.1596/978-1-4648-1096-1. License: Creative Commons Attribution CC BY 3.0 IGO 138 7. ANNEXES Annex 1: Scope of Work Annex 2: Evaluation Team Profiles Annex 3: Statements of Conflict of Interest Annex 4: Evaluation Respondents Annex 5: Coach Case Studies Annex 6: Evaluation Tools Annex 7: Methodology Plan and Study Protocol Annex 8: Calculation of Project Schools Annex 9: Calculation of Coaching Dosage Annex 10: Reported RSP Participation by Curriculum Advisors/Subject Advisors Annex 11: Statement of Differences Annex 12: Teacher Training Matrix Contract No: 72067418D00001, Order Number: 72067419F00015 139 ANNEX 1: SCOPE OF WORK Contract No: 72067418D00001, Order Number: 72067419F00015 140 SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK C.1 INTRODUCTION Despite the government of South Africa’s (GoSA) large investment in basic education – ZAR 228.8 billion (approximately $15.25 billion) in 2016/17 – roughly 16% of the national budget – South Africa continues to face challenges providing a quality education in the majority of the country’s schools and its education indicators continue to lag behind that of its peers. In international comparative reading tests South Africa consistently performs at the bottom with nearly 80% of Grade 4 students unable to read with comprehension in the language of their choice (PIRLS 2016). The GoSA considers education to be one of its highest domestic priorities and one of the greatest long-term challenges facing the country, as is evident in the National Development Plan of which the number one outcome is improving the quality of basic education (DBE, 2013). To support the GoSA, USAID/SA is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve reading skills of primary grade learners which, at the time of publication was in line with the continuation of Goal 1 of USAID’s Global Education Strategy, aiming for 100 million children worldwide with improved reading skills. The newly published US Government Basic Education Strategy (2019-2023) also prioritizes improved quality of instruction in basic education, and USAID’s new Education Policy (November 2018) continues to prioritize improved early grade reading outcomes. PERFORM will contribute to these shared goals through implementing education interventions in support of three objectives: ● Improved primary grade reading instruction; ● Increased quality of educational administration and support; and ● GoSA support for reading initiatives built at district, provincial and/or national level. Interventions will be designed and implemented in partnership with GoSA to focus on building teacher knowledge and skills that can improve literacy pedagogy, strengthen and improve education management for better reading outcomes, and involve community members in supporting reading outcomes. PERFORM will use a demonstration effect and rigorous evaluation to support GoSA in bringing proven education solutions to scale, thus increasing the impact and value of national investments in the education sector. This will be done by piloting and testing local solutions which show promise to improve the reading skills of primary grade learners. Close collaboration with GoSA will aim to support buy-in for new reading interventions and to ensure GoSA has the data necessary to decide how best to take successful, cost-effective literacy improvement interventions to scale across the country. In order to deliver rigorous evaluations under PERFORM, the PERFORMANCE Indefinite Delivery Indefinite Quantity (IDIQ) single holder contract was awarded to Khulisa Management Services to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID/SA in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from interventions. The contractor will also be responsible for relaying this information to GoSA, the education research community, and other education sector Contract No: 72067418D00001, Order Number: 72067419F00015 141 stakeholders. This dedicated education sector evaluation IDIQ contract will provide cohesion across all evaluations of PERFORM activities, as well as a meta-analysis of outcomes across the project. It will also allow for the flexibility to adapt to changes in PERFORM activities and to address additional reading related research and analysis requests from USAID that develop during the course of PERFORM’s implementation, both in response to the needs of DBE and otherwise. PERFORMANCE will help to fill a critical research gap by providing rigorous analysis in target areas related to improving the quality of language and literacy skills of primary grade learners in South Africa and the region. USAID/SA found there is little data available on the impact of teacher training programs on student learning outcomes, including literacy. There is also little rigorous analysis available on the effectiveness of school principals, district authorities, communities, and parental engagement to increase student learning. Furthermore, research studies on the costeffectiveness of quality teacher training and support, school management and administration is even more limited, especially that which estimates the cost-effectiveness of various types of programming in developing countries. In general, the quality and methodological rigor of the research that is available varies so significantly that it does not offer a sound empirical base for program design in South Africa. Rigorous research that does exist is so context-specific that it is not necessarily generalizable. PERFORMANCE aims to fill this crucial research gap with rigorous evaluations, studies and assessments. The IDIQ contract has three distinct objectives: ● Objective 1: Design and conduct evaluations, surveys, studies and sector assessments to inform and improve education sector activities. ● Objective 2: Make recommendations for viable future education sector activities. ● Objective 3: Strengthen the community of practice surrounding education research, monitoring, evaluation, and learning for languages and literacies. This task order under PERFORMANCE will require the contractor to propose design concepts for pertinent evaluation and assessment activities and to deliver such design reports to USAID. This Task Order will also require the management and reporting deliverables as discussed in the IDIQ Contract and laid out below in Section C.3. C.2 BACKGROUND On November 8, 2017, the PERFORMANCE single holder IDIQ contract was awarded for a performance period of five years. To accomplish the objectives of the PERFORMANCE IDIQ as listed above in section C.1,, the contractor will provide design and delivery services to USAID through conducting desk reviews, stakeholder dialogues and other scoping and research activities on specific topics upon written request by the TOCOR. The findings, conclusions and recommendations from these reviews and engagements will be presented to USAID as design reports. Based on the design reports, new task orders may or may not be solicited by USAID under the IDIQ contract. Furthermore, the IDIQ contract has core support administrative and reporting requirements that will be met through this task order. Contract No: 72067418D00001, Order Number: 72067419F00015 142 C.3 OBJECTIVES AND RESULTS The objective of this task order contract is twofold: (1) to administer core support services for the PERFORMANCE IDIQ as laid out below; and (2) to produce, upon request from USAID, design reports to be used for the development of future task orders for research and/ or community strengthening activities through a process of desk reviews, stakeholder dialogues and other scoping and research activities. To achieve these objectives, the contractor must produce the following intended results: 1. Five design reports presenting the findings, conclusions and recommendations based on desk reviews, stakeholder dialogues and other scoping and research activities on specific topics, upon written request by the TOCOR. 2. Reports and deliverables or outputs as specified by the IDIQ Contract in Section F.5, and incorporated here: REPORTS AND DELIVERABLES OR OUTPUTS In addition to the requirements set forth for submission of reports in Section I and in AIDAR clause 752.242- 70, “Periodic Progress Reports,” under the first task order issued to this IDIQ, the Contractor shall submit the following deliverables or outputs to the COR specified in accordance with Section I: (a) Contents Of Periodic Progress Reports (1) Performance Monitoring Reports (PMRs). The Contractor shall submit performance reports at intervals agreed upon with the COR (Assume the COR requests annual reports), summarizing progress of the major activities in process during the period of implementation of the contract, indicating any problems encountered, and proposing remedial actions as appropriate. In addition, the Contractor shall include a section which discusses any salient programmatic trends that can be distilled from major activities that are in-process or recently completed; and highlight unresolved or ongoing administrative/ bureaucratic constraints to the Contractor's optimal performance; an update of which will be used as part of future award evaluations. (2) Contract Financial Report. The contractor shall submit a contract financial report including the following: (i) Notification of New Task Order Report Content: (1) Task Order number, (2) Mission/Bureau contracting the task order, (3) Period of Performance, (4) CO, (5) COR, (6) Ceiling Price, and (7) Initial Obligated Amount. (ii) Modifications to Existing Task Orders Report Content, to include a short description of any of the following changes to existing task orders: Incremental funding, (1) Time extensions, (2) Change of CO and/or COR, and (4) Completion of work. Contract No: 72067418D00001, Order Number: 72067419F00015 143 The above reports shall be submitted within 30 days of the three-month periods (calendar quarters) ending on March 31, June 30, September 30, and December 31. The contractor shall submit the reports to the COR identified in Section G of the Schedule, who will forward the reports to the responsible Contracting Officer. The contractor shall promptly notify the Contracting Officer and COR of any problems, delays, or adverse conditions which materially impair the contractor's ability to meet the requirements of the contract. (b) Accrual Reports The contractor shall submit an estimated accrual report to the cognizant technical office for the task order: (1) Total amount obligated, (2) Total amount invoiced for, (3) Total amount expended but not yet invoiced for, (4) Remaining unexpended funds. The contractor shall submit these reports to the COR identified in Section G of the Contract on March 10, June 10, September 10, and December 10. (c) Language of Reports and Other Deliverables All reports and other deliverables shall be in American English, unless otherwise specified by the task order. (d) Meetings The contractor shall schedule and attend semi-annual meetings with the COR. (e) Reports (1) The cover page of all deliverables required hereunder shall include the USAID Identity (or the name of the Agency written out) prominently displayed, the contract number (see the cover page of this contract), Contractor name, name of the USAID project office (USAID/Southern Africa/Regional Environment, Education and Democracy Office), the publication or issuance date of the document, document title, author name(s), and activity name (PERFORMANCE). Descriptive information is required whether Contractor furnished products are submitted in paper or electronic form. All materials shall include the name, organization, address, telephone number and email address of the person submitting the materials. (2) Hard copy reports shall be prepared on non-glossy paper (preferably recycled and white or off white) using black print. Elaborate art work, multi-color printing, and expensive bindings are not to be used. Whenever possible, pages shall be printed on both sides (see also Section E of this contract). (3) Electronic formats shall be submitted with the following descriptive information: (i) Operating system format, e.g., Windows or Macintosh compatible; (ii) Name of application software used to create the files, e.g., Microsoft Word 2010; (iii) The format for any graphic and/or image files included, e.g., TIFF compatible; and (iv) Any other necessary information, e.g., special backup or data compression routines/software used for storing/retrieving submitted data. (f) Distribution to COR and Briefing Contract No: 72067418D00001, Order Number: 72067419F00015 144 Distribution of Task Orders to Basic Contract COR: One copy of each new task order issued under this IDIQ contract must be forwarded to the IDIQ COR within one week of receipt by the Contractor. C.4 CROSS CUTTING THEMES The following guidance is provided with respect to alignment with the US Government Basic Education Strategy 2019-2023, USAID’s Education Policy (November 2018) and Education Evaluation Policy, using local systems, sub-awards and incorporating gender considerations in evaluation activities. Where applicable and feasible for this task order the Contractor must consider all these themes in achieving contract goals and objectives and apply them to the overall performance of the Contract. C.4.1 Alignment with the USG Basic Education Strategy (2019-2023), the USAID Education Policy (2018) and USAID’s Education Evaluation Policy It is anticipated that the contract will be financed 100% from Basic Education (BE) funds. Funding for BE activities must meet all statutory requirements and align with the USG Basic Education Strategy available at https://www.usaid.gov/education/usg-strategy and USAID’s Education Policy (2018) and corresponding implementation guidance, including USAID’s Education Evaluation Policy. In particular, direct BE funding must address the six Key Principles of the USAID Education Policy as well as Priorities 1 and 2. Key Principles: ● Prioritize country-focus and ownership ● Focus and concentrate investments on sustainable results ● Strengthen systems and develop capacity ● Work in partnership and leverage resources ● Drive decision-making and investments using evidence and data ● Promote equity and inclusion Relevant priorities: ● Children and youth, particularly the most marginalized and vulnerable, have increased access to quality education that is safe, relevant, and promotes social well-being. ● Children and youth gain literacy, numeracy, and social-emotional skills that are foundational to future learning and success. C.4.2 Using Local Systems Contract No: 72067418D00001, Order Number: 72067419F00015 145 The Contractor must engage local institutions in every step of implementation when feasible, building technical and management capacity within targeted institutions and communities where possible, working with or through host country systems when appropriate, and transferring managerial and activity implementation responsibility to local institutions where feasible. C.4.3 Sub-awards A means of establishing partnerships with local institutions is through the use of sub￾awards. The Contractor may subcontract as outlined and referenced in the PERFORMANCE IDIQ base contract. Any other subcontracts must be pre-approved by the CO. C.4.4 Incorporating Gender in Design Activities The contractor shall ensure that relevant gender issues are explored through the design process, as appropriate. The contractor shall further ensure that all evaluation topics or research questions that result from the design process are sensitive to gender. All proposed topics must require that the data be disaggregated by sex to enable analysis on relevant gender issues in language education and outcomes and to answer other relevant gender￾based evaluation questions. C.5 GENERAL MANAGEMENT AND ADMINISTRATIVE APPROACH The contractor must provide all general management and administrative support necessary to perform the contract and achieve the above results. This includes, but is not limited to, the following: 1. The contractor will provide overall management and administration of the contract, including home office support and administrative services. The contractor will provide both the key personnel specified in the contract and additional personnel, long￾term and short-term, necessary to meet recurring general management and administrative support needs under the contract. 2. The contractor will procure or lease facilities, supplies and services as necessary to perform the contract. 3. The contractor will provide the planning necessary for performance of the contract. 4. The contractor will get the necessary ethical clearance and permission from authorities to conduct research in government sites, particularly where students will be involved. Requirements for ethical clearance may vary by province and target provinces are not yet determined. The contractor will be responsible for investigating and fulfilling the requirements for such clearance. 5. The contractor will provide oversight, quality control, and general technical support of all services and deliverables provided pursuant to the contract. This includes the provision of copywriters/editors that are familiar with US American English and US American report writing standards. Contract No: 72067418D00001, Order Number: 72067419F00015 146 6. The contractor will provide and assure the proper, efficient, and uniform use of modern management and accounting practices, information technology (IT), communications, reporting, human resource management, property control, security, records, and other administrative processes and systems required under the contract. C.6 STANDARDS OF PERFORMANCE This section defines the performance requirements to which the contractor shall be held, establishes the performance levels or standards, and defines how these performance standards will be measured and verified. The contractor will be continually evaluated against these standards. Key Performance Indicators and Assessment Plan CONTRACTOR EXPECTATIONS KEY MEASURES OF PERFORMANCE PLAN FOR VERIFYING PERFORMANCE i. Technical Quality of Service The Contractor will deliver professional, high quality services that responds to the contract requirements. Consistency and accuracy of the Contractor’s work is demonstrated at all times throughout the period of performance of the contract, including quality of deliverables. Reports and presentations to USAID and other counterparts reflect professional quality standards in writing, data collection, and analysis. Official acceptance of submitted reports, and other deliverables by the COR via written correspondence, copies saved in COR file. ii. Schedule The Contractor’s Results and deliverables are performed and/or delivered to USAID according to the specified timeline. The Contractor will provide timely answers and feedback on all requests, issues and or questions raised by the COR and/ or the CO. Deliverables and task order results are completed by dates identified in Section F. No more than fifteen calendar days transpires between submission of comments, questions, or issues to Contractor and a response received by USAID. Date of receipt of submitted reports and presentations via physical delivery or electronic submission, copies saved in COR file. iii. Cost Control The Contractor’s work plans and budget are adequate and result in the completion of all deliverables and tasks as outlined in Section C. The Contractor must provide the deliverables or outputs described in Section C and F and comply with all contract requirements, performing to the highest standards under the terms of the Contract. Completion of Contract without any modifications or amendments related to task order price due to contractor performance. Contract No: 72067418D00001, Order Number: 72067419F00015 147 iv. Business Relations Sound working relationship between the Contractor and USAID/South Africa Avenues of communication are clear; Effective communication practices and team management are evident and there are no documented instances of problems arising due to management of key personnel or the team; Logistical aspects pertaining to Section C tasks and deliverables are clearly designed, well thought out processes, are organized and implemented so that tasks and deliverables are efficiently completed. No documented problems or issues arise due to Key personnel management or communication issues; No documented problems or issues arise due to logistical issues (within the control of the Contractor) as documented by the COR. Performance Evaluation Criteria, Ratings and Standards 1. Exceptional Performance meets contractual requirements and exceeds many to USAID/South Africa’s benefit. The contractual performance of the required results was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective. 2. Very Good Performance meets contractual requirements and exceeds some to USAID/South Africa’s benefit. The contractual performance of the required results were accomplished with some minor problems for which corrective actions taken by the Contractor were effective. 3. Satisfactory Performance meets contractual requirements. The contractual performance of the required results contains some minor problems for which corrective actions taken by the Contractor appear or were satisfactory. 4. Marginal Performance does not meet some contractual requirements. The contractual performance of the required results reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor’s proposed actions appear only marginally effective or were not fully implemented. 5. Unsatisfactory Contract No: 72067418D00001, Order Number: 72067419F00015 148 Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the required results contains a serious problem(s) for which the Contractor’s corrective actions appear or were ineffective. C.7 PROVIDE DATA, TECHNICAL MATERIALS, AND OTHER INFORMATION The Contractor will provide USAID with data, technical materials, and other relevant materials produced in the execution of this contract in line with USAID’s Open Data Policy as outlined in ADS 579 http://www.usaid.gov/sites/default/files/documents/1868/579.pdf. This includes pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives, as well as data and information needed for reporting under the relevant foreign assistance objectives, areas and elements. Pedagogical Materials and Technical Inputs When applicable the Contractor must provide pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives. Examples of technical inputs to be provided to USAID include scripted lesson plans, supplementary readers, assessment instruments, observation tools, training guides, workshop reports, radio programs, assessment tools, sampling frames, photographs, videos, and other recordings. The Contractor must transmit technical materials to the relevant TOCOR and submit them to the USAID Development Experience Clearinghouse (https://dec.usaid.gov/). C.8 MATERIALS Materials developed under this contract are subject to FAR 52.227-14 (RIGHTS IN DATA – GENERAL), and the Government of South Africa and others will be granted a paid-up, nonexclusive, irrevocable, worldwide license (under the Creative Commons By “CC BY” http://creativecommons.org or otherwise as USAID may determine) to reproduce, prepare derivative works and distribute copies to the public. Any material not first produced in the performance of this contract is subject to clause FAR 52.227-14(c)(2). [END OF SECTION C] Contract No: 72067418D00001, Order Number: 72067419F00015 149 ANNEX 2: EVALUATION TEAM PROFILES Contract No: 72067418D00001, Order Number: 72067419F00015 150 Jennifer Bisgard, Project Director Ms. Jennifer Bisgard co-founded Khulisa Management Services in 1993. An expert in M&E and organizational development, she leads evaluations and capacity building assignments in the education, and democracy and governance sectors. She has 20+ years of experience leading evaluations and research assignments for USAID, including impact evaluations, performance assessments, program/project design/management and data analysis, including leading evaluations, such as the USAID/SA Evaluation IDIQ and serving as Project Director on the previous PERFORMANCE Task Orders. Jennifer has co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned” published by Sage Publishers in August 2018. The book is edited by Kylie Hutchinson, with a forward by Michael Quinn Paton. The book features 22 case studies of evaluation failures, including ours which is based in South Africa. Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993. She has served on boards for the: African Evaluation Association (AfrEA), International Organization for Cooperation in Evaluation (IOCE) and South African Monitoring and Evaluation Association (SAMEA). She has a Master’s Degree in Social Change and Development from Johns Hopkins University. Katharine Tjasink, Senior Project Manager Ms. Katharine Tjasink joined Khulisa in 2012, as Regional Technical Coordinator on the Farmer Voice Radio project. In April 2014, she took up the position of Senior Associate in the Education and Social Development Division, and currently serves as Associate Director. She is an experienced M&E and research professional with 10+ years’ experience conducting and managing M&E and research assignments for a number of clients, including USAID. Katharine has successfully led multiple evaluations and data collection projects, including evaluations of early grade language and literacy projects in South Africa. She has in-depth knowledge of USAID’s activity reporting, financial and project management requirements and evaluation policy. Prior to joining Khulisa, Ms. Tjasink worked with a radio and Information and Communication Technology (ICT)-based agricultural program funded by the Bill and Melinda Gates Foundation. In 2013, she was selected as one of four award recipients, from a pool of 60 applicants from 25 developing countries, to attend and present her unique “meta-story” evaluation methodology at the 27th annual American Evaluation Association (AEA) conference in Washington D.C. She holds a Master’s Degree in Research Psychology from the University of the Witwatersrand. She is also ISO 9001:2008 certified in Data Quality Management Systems Assessment. Leticia Taimo, Evaluation Coordinator (Mid) Ms. Leticia Taimo joined Khulisa in 2015 as an intern, and is now a Senior Associate with 5+ years’ evaluation, project management, and coordination experience. She has successfully participated and coordinated multiple education evaluation, research, and assessment plus data collection projects in in South Africa for a variety of stakeholders (private sector, NGOs, government, and international donors). In particular, she is developing a strong expertise in Early Childhood Development. She has presented on how to use technology to monitor ECD at the SAMEA Conference 2017, presenting evaluation Contract No: 72067418D00001, Order Number: 72067419F00015 151 findings at the “CSI that Works” 2017, facilitating an interactive session on M&E at the Play Conference 2017 and introducing M&E concepts at BRIDGE’s Early Childhood Development Community of Practice in 2018. Leticia has an MSc Development Studies from the School of Oriental and African Studies, University of London, and is ISO 9001:2008 certified in Data Quality Management Systems Assessment. Leticia was awarded the Mandela Rhodes Scholarship in 2013 and the Commonwealth Scholarship in 2014 as recognition of her commitment to social change in the African continent. Margaret Roper, Senior Evaluator Ms. Margaret Roper fulfils the position of Deputy Director and senior MEL Specialist. She has extensive experience in program development, M&E, and knowledge sharing in social policy and practice, social justice, social protection, child protection, school and community health and safety, educational development, and behavior change. Since 1993, Margaret has worked in the non-profit, government, donor, and business sectors in Southern Africa. She has held positions in the Secretariat for Safety and Security, the South African Police Service, Health and Development Africa, Mott MacDonald, Inkanyezi Initiative and consulted for a range of organizations including Oxfam Australia, UNICEF and government departments such as Basic Education, Social Development, Correctional Services, and Gauteng Department of Community Safety. Margaret belongs to SAMEA, AfrEA and to the International Society for the Prevention of Child Abuse and Neglect (ISPICAN). She has presented at ISPICAN conferences in Turkey (2012) and India (2011); and Pathways to Resilience IV, at an international conference held in Cape Town, 2017, on South Africa’s Response to Enabling Child Resilience Through Expanding Community-based Services She has a Masters in Social Research from the University of Sheffield (United Kingdom), and a Bachelors of Primary Education and Education (Honors) from the University of the Witwatersrand. Benita Williams, Senior Evaluation Specialist Benita Williams is a South African evaluator and current director of Benita Williams Evaluation. Over the past eighteen years, she has conducted various evaluations of education, youth development, income generation, and health initiatives in Southern Africa for corporate donors, government departments, and community-based NGOs. She has a keen interest in evaluation methods and evaluation design. She has experience in the sophisticated qualitative and statistical analysis of education data, voluntary counselling and testing/ health behavior survey data and skills-audits. Recently her focus has been on the evaluation of education support initiatives related to early childhood education, whole school development in public schools, and the training of teachers in maths, physical science, and English subject areas. She is team lead for an evaluation of a Zenex Foundation Secondary School development project across three provinces, and the evaluation of the Anglo American Education program that targets schools and ECD centers in eight areas across the country. She was part of a team that evaluated an early -childhood focused social-franchise movement and also collaborated with RESEP at the University of Stellenbosch on a Public Expenditure Tracking Study in ECD. Ms. Williams frequently works as Developmental Evaluator and has done evaluations informed by Utilization Focused Evaluation, Outcome Mapping, Realistic Evaluation, and Contract No: 72067418D00001, Order Number: 72067419F00015 152 Systems Evaluation theories. Ms. Williams has academic training in the field of Research Psychology at Master’s level (University of Pretoria – Dissertation not completed) and is currently pursuing a Master’s in Development Studies at the University of Johannesburg. In 2009-2010, Benita served as executive secretary for the African Evaluation Association, a network of evaluation associations and evaluators across Africa. Benita was the founding treasurer for the South African Monitoring and Evaluation Association (SAMEA) a voluntary association of evaluators and evaluation users in SA, and served until September 2009. Benita has published in peer-reviewed evaluation publications. Jacqui Dornbrack, Panel Expert Jacqui Dornbrack holds a PhD (Applied Linguistics) and a Master’s in Education (Language and Literacy). She is an Independent Literacy Consultant and has consulted for both Zenex and Funda Wande this year. In 2018, she was the Content Director for the Funda Wande, responsible for the development, writing, and design of course material for teachers as well as video scripts to capture good classroom practice. The Funda Wande Course entails 13 modules on teaching reading and writing in the Foundation Phase in both HL and EFAL. Before she joined Funda Wande, Jacqui worked as a Foundation Phase literacy coach for Pearson (Western Cape Department Of Education [WCED] 100 Schools Improvement project) in five schools in Delft and Kuilsriver and as the Programme Advisor for Shine Literacy (an Non-Profit Organization (NPO) who works in schools to support reading and writing in Grades 2 and 3). Before this, she worked in academia as a teacher educator for University of Cape Town (UCT) (Postgraduate Certificate in Education [PGCE] Intermediate Phase) and University of the Western Cape (UWC) (Intermediate Phase) and has supervised 12 post graduate students in the field of language education. Jacqui has also been a school teacher and has presented at numerous local and international conferences and has eight peer reviewed articles and is on the review board of two education journals: Reading and Writing (South African) and English Practice and Critique (New Zealand). Janet Orr, Panel Expert Janet Orr is a highly experienced educator providing quality basic education program services for youth in both the United States and developing countries in Africa, the Middle East, and Asia. She served as the English as a Second Language lecturer when Ohio University/USAID began to establish the Department of Primary Education at the University of Botswana. Janet’s M.Ed. specialization in teaching primary school reading and English language learning was earned at the University of Illinois. She expanded her knowledge into program services and evaluation as the Associate Director of the Center for Equity and Excellence in Education at the George Washington University. She applied those skills in an overseas context when she served as USAID’s Education Specialist in Tanzania and Sri Lanka. Most recently, she is focused on consulting services to design effective instructional programs using research and data gained through program evaluation. Janet served as an Education/Reading Specialist in three Early Grade Reading Mid-Term Evaluations: Jamaica’s Education Transformation Project (ETP), Malawi EGRA, and Nigeria’s Northern Education Initiative Plus. Contract No: 72067418D00001, Order Number: 72067419F00015 153 Nombulelo Baba, Panel Expert Nombulelo Baba holds a BA (Hons) degree in Development Studies from the University of the Western Cape. She is currently affiliated with the National Collaboration Education Trust (NECT), working as NRC Coordinator. Nombulelo Baba has worked as a teacher educator at READ Educational Trust and Matthew Goniwe School of Leadership and Governance. She has also served as Subject Advisors Coach at the NECT during which time she was instrumental in reviewing and improving the design of the coaching models. She has reviewed coaching tools to ensure effectiveness, evaluated coaching models and TOC, plus supervised EFAL coaches practicing in schools. As part of developing coaches and Subject Advisors, she was involved in developing and training coaches on case studies that are relevant to their context. Daleen Botha, Data Quality Analyst Daleen Botha is a director and evaluator at Benita Williams Evaluation. Her more than 15- year career includes more than ten years’ experience in M&E, mostly working in the education sector. Daleen has worked with government institutions in South Africa such as the South African Council of Educators and the Gauteng Department of Education. She has also worked with community-based NGOs like the Sumbandila Trust and private foundations such as the Zenex Foundation. She is adept at qualitative and quantitative analyses, with extensive experience of developing M&E frameworks, Theories of Change, Logical Frameworks and Result chains, as well as data and project management. Daleen holds a BSc Honors degree in Computer Science as well as a post-graduate diploma in M&E from the University of Stellenbosch. She is a member of the South African Monitoring and Evaluation Association (SAMEA). Carole Metekoua, Statistician Carole Metekoua is a process-oriented data analyst and statistician with four years of experience specific to Monitoring and Evaluation (M&E). She has significantly contributed to learning and evaluation projects as well as research projects by managing, cleaning, transforming, analyzing and visualizing big data in Python, STATA, and R. Her experience working with Khulisa, includes serving as Data Analyst on the current long-term USAID Zambia Scaling Up Nutrition Learning and Evaluation (SUN LE) assignment, developing the data analysis plan, generating key indicators and conducting inferential analyses and interpretation of findings. She also served as Data Quality Reviewer with Khulisa during 2017-2018 to provide quality assurance for Global Fund data quality reviews in Tanzania, Mozambique, Sri Lanka, Indonesia and Burkina Faso. Previously she served as Researcher at Wits Health Consortium, focused on data management, analysis and presentation. She recently completed her Master of Science (MSc) degree in Epidemiology, specializing in Public Health Informatics. Contract No: 72067418D00001, Order Number: 72067419F00015 154 ANNEX 3: CONFLICT OF INTEREST 155 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Jennifer Bisgard Date October 11, 2019 156 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Katharine Tjasink Date October 11, 2019 157 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Leticia Taimo Date October 11, 2019 158 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Margaret Roper Date October 11, 2019 159 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Benita Williams Date October 11, 2019 160 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Jacqui Dornbrack Date October 11, 2019 161 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Janet Orr Date October 11, 2019 162 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Nombulelo Baba Date October 11, 2019 163 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Daleen Botha Date November 15, 2019 164 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Carole Metekoua Date November 15, 2019 165 ANNEX 4: EVALUATION RESPONDENTS 166 Interviews Respondents No. Type of stakeholder Male Female Key Informant Interviews USAID 3 Audience 1 2 DBE 2 Audience 0 2 FPD Consortium Interviews FPD respondents 4 Primary 2 2 Molteno respondents 2 Primary 0 2 OUP respondents 2 Primary 1 1 VSO respondents 2 Primary 1 1 North West Provincial Department of Education Interviews Curriculum Advisors 6 Audience 0 6 Subject Advisors 2 Audience 0 2 Director 1 Audience 0 1 Provincial coordinator for African languages 1 Audience 0 1 Chief Education Specialist 1 Audience 0 1 Project Coordinator 1 Audience 0 1 Director: Professional Educator Development Services 1 Audience 1 0 Deputy Chief Education Specialist (Project Coordinator): Dr Kenneth Kaunda 1 Audience 0 1 Chief Education Specialist Professional Support: Dr Kenneth Kaunda 1 Audience 0 1 167 Interviews Respondents No. Type of stakeholder Male Female Head coach interviews Head Coach: Dr Kenneth Kaunda 1 Primary 0 1 Head Coach: Ngaka Modiri Molema 1 Primary 0 1 Coach interviews RSP Literacy Coaches 14 Primary 2 12 Teacher surveys Sample school Foundation Phase teachers 304 Primary Mixed, but predominantly female Teacher Group Interviews Pilot school Foundation Phase teachers 65 Primary Not captured School Principal interviews Sample school principals 58 Primary Mixed SMT member interviews Sample school SMT members 157 Primary Mixed, but predominantly female 168 ANNEX 5: COACH CASE STUDIES AND RUBRIC 169 COACH CASE STUDY 1 PROFILE29 Qualifications Master of Business Administration (2012) Bachelor of Education (Honors) (2006) Further Diploma in Education (2002) Diploma in Education Primary (1995) Experience of teaching Chief Examiner, Department of Basic Education (2012- 2014) Senior Education Specialist Foundation Phase Literacy, Department of Education (2008-2015) Head of Department, Primary (2000-2008) Teacher, Primary (1996-1999) Experience and training in coaching Experience gained as a Subject Advisor. RSP training provided by OUP and Molteno on the lessons plans and components of language. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:10 Ratio of coach to RSP teachers 1:56 Self-reported frequency of RSP coaching a teacher “I visit each school at least twice a month, however if more than 9 teachers are at a school then visits are three times a month.” On average a teacher is seen three times a term. 29 Qualifications and experience of coaches have been reduced to try and anonymise the profiles 170 Data Source She was shadowed in the week of September 2, 2019, during which she was interviewed and observed at two schools working with 12 individual teachers. The Researcher was joined by the Khulisa Associate on September, 4 2019 at for quality assurance purposes. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “As a coach I am conducting classroom visits, observing lessons, I provide in class and in school coaching. I facilitate PLCs for teachers to support them to reflect on the opportunities for improvement. I also conduct just in time training in lesson plans to prepare teachers for the subsequent term. At the beginning of every term we do training, so that the teachers know what they are supposed to do inside the classrooms. We also participate in train the trainer session whereby they equip us with the relevant support on what to do with the teachers when we go to schools and during our training. And we also do some reporting whereby we report every month on what transpired during our visit. After training we also do some reports on how the training was, telling them about challenges, where to improve and coming up with recommendations. We report to our Head Coaches, we just send emails to them on [a] monthly [basis] and after the trainings.” Coaching practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The researcher reported that all the teachers who she coached seemed comfortable with her and had clearly met her before. According to the researcher the teachers appeared to both trust and listen to her. This coach is warmly welcomed by the school and feels part of the staff, and is not treated as an outsider. She feels supported by the School Management Team and often goes straight to the class rather than having to report to a SMT member. She attributes this to the importance of communication. In her words “If you communicate well, they will give the respect.” Being honest with the teachers is necessary, for example: “If you were supposed to come and then you … attend a meeting somewhere, communicate with the teacher.” Other elements of building a trusting relationship include respecting the teachers, being approachable, being humble and not coming in as “This monster and when you come it’s like you are a police: teachers shivering when you come, we need to avoid these things.” Responsibility is also important, being accountable, and confidentiality is critical. She indicated that as a coach, one needs to keep some things to oneself. 171 There is a trusting relationship between the coach and the teachers. She indicated that teachers share their problems with her, and when she asks why they do not tell the principal, they say she is a mediator, they trust her. She sits with them and tries to address the problem, but if it is beyond her control, then she informs the principal. The teachers accept this: “They feel at home” in this situation. The coach reports issues that need attention to the principal, particularly those that affect the ability of the teacher to execute their duties. Examples include when classes are overcrowded, the lack of a carpet in the classrooms in the Foundation Phase, a lack of LTSM on the walls or that learners do not have the books or charts. COACHING STANDARDS THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • She shares the monthly schedule with the teachers, even though this is not part of her role as it should be done by the Head Coach (but reportedly this is not happening). This is usually done through WhatsApp directly with the teachers. 2. Pre-observation discussion • In six observations the teachers briefly show the coach the lesson to be presented. The coach also informs the teacher of what she will be doing during that lesson. However, in six instances the teachers did not share the lesson plan with her, as sometimes the lesson began immediately on her arrival. • She checks that the teacher understands the content of the lesson in all 12 observations and makes notes in her coaching form. • She walks around the classroom and selects learner books. She goes through them to understand the learner’s progress in the lesson plans in all 12 observations. She also checks that flashcards and reading books are available. 3. Classroom Observation • In all 12 observations, the coach observes if LTSM is available, relevant and well organized, and monitors if it is being used properly. She notes this in her coaching tool. • The coach checks whether Drop All and Read (DAR) / Drop Everything and Read (DEAR) is included in the timetable in all 12 observations. In three observations, DAR/DEAR was not in the school timetable. In one school, the teacher was using a 2018 timetable that did not have DAR on it, and the coach noted this for follow up after the lesson. • According to the researcher: She observes, in all 12 observations, if learners were working in well organized groups during group guided reading. She was able to guide the teacher on how to manage group guided reading when she noticed one teacher was not following the scripted lesson plan methodology. Her intervention helped to make the new content clearer for the learners. Similarly, in another classroom, she noted that the teacher read the story rather than letting the learners read it and correcting when necessary: feedback was provided to the teacher at the end of the lesson rather than interrupting the lesson. 172 • The coach documented in her notes when one teacher did not acknowledge learners’ experiences or prior knowledge at the start of the lesson, as she checks whether teachers do this in all 12 observations. This teacher, however, was observed to begin the lesson with a creative and relevant song. In another situation, she noted how the teacher gathered prior knowledge about the circus using questions to the learners. • She successfully checked (in 11 out of 12 observations) and noted when teachers were using reading strategies and processes during lessons; if focused activities are integrated within components of language; if opportunities were created for the application of new knowledge for learners; and if and how learners with learning barriers were differentiated, for example if the teacher used flashcards. She continuously observed the teacher and learners and wrote in her notes. In the only instance where she did not check whether reading strategies were utilized, the researcher notes that “This might be due lack of enough space and the coach used her notebook (she has a big A4 notebook where she writes notes during observation). And her having to take over teaching might have distracted her not to write as she was busy with the leaners.” • The coach furthermore verified in 11 observations whether the teacher was continuously assessing and giving developmental feedback to the learners and if the planned assessment activities assessed learner development and skills. In nine observations the coach also checks whether learners were assisted to reflect on their new knowledge, skills, attitudes and values. She observed the teacher’s skills and recorded the teacher’s ability in her coaching documents, however, she noted that sometime there is not sufficient space to record her observations. In a few instances (three) some items were left blank on the coach tool, or no justification was provided (however this was the exception rather than the norm). • The coach noted that the outcomes of the lesson were not made clear by the teachers in six lessons. • She used the lesson plan to follow the activities of the teachers. 4. School based post-observation discussion • In all 12 observations, the coach asks the teacher to identify things they felt went well in the lesson. However, in three observations, the researcher notes that this did not come before the coach gave her own feedback, but rather during the feedback session. In all 12 instances, the coach shares her observations of things that went well, for example that the teacher’s assessments were developmental, and she takes the teacher through the lesson observation document during the feedback. The focus of the feedback is primarily on the extent to which the lesson complied with the core methodology. She offers appropriate and reasonable suggestions and explains the rationale providing good examples to guide the future implementation. • In all 12 observations, the researcher did not observe the coach asking the teacher for her insights into what she would do differently in the lesson. Nor did she guide the teachers to identify solutions for themselves through questions to elicit clarification. The researcher observed that it is primarily the coach who talks and the teacher who listens. The coach did not always (only in six of 12 observations) provide feedback on classroom management or curriculum coverage to the teacher. • She seems to ensure she offers suggestions for improvements in a constructive way – she is professional in her feedback and in a way that aims to build the teacher’s competency. She is neutral when offering feedback to the teachers. She gives feedback in a calm voice and with a smile. 173 • The coach and the teacher together set goals for the next month in four feedback sessions. This was as a result of a challenge or problem the teacher faced which the coach identified. For example, when time management was a problem or group reading practices were not followed according to the lesson plan. In one coaching session the teacher indicated she will catch up on the reading activities she had not done with the learners, and in two other coaching sessions teachers agreed that on the next coaching visit the teacher will do the shared and group guided reading, because she now knows how they are supposed to be done. 5. After School Workshops Four needs-based after school workshops were planned during the week of the coach shadowing, however only two were facilitated. The coach indicates that she does hold catch-up training sessions for teachers who missed the training on LTSM. “I just get those schools that did not attend the training and we meet at a central place and then I train them the same way that I train during my main training. But it will be just for a day per group.” CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach noted the following challenges in providing coaching services to teachers in the schools: • The most difficult part of being a coach, according to the coach, was providing the training to the teachers the first time as there were “So many teachers” and she was alone providing the training and distributing the books at the same time. This was particularly challenging as some teachers take more books for themselves. However, this has changed, and some teachers now help the coach. • Events beyond the coach’s control influence if she is able to support to all the teachers during a month. For example, school trips, meetings or other events such as entrepreneurship days when learners are not in class. This coach noted the following successes in providing coaching services to teachers in the schools: • Arriving early to the training venue and setting everything up so that all goes according to plan, and preparing in advance. • The use of WhatsApp groups and messaging for teachers, and to send audio messages. • She indicated she did provide virtual training, however further information is required. COACHING SUPPORT The coach receives support from the RSP project manager, Head Coach, colleagues and the office. She feels that sufficient resources are provided to enable her to travel, communicate and record coaching activities using the tablet and forms. 174 The Head Coach has only visited her once at a school, and during this visit the head coach provided guidance. They meet during the provincial meetings where coaches can share their challenges, talk about their frustrations, and discuss what they do in class. This is when the head coach coaches them and tells the coaches how to do things. She views this support as not being adequate. EVALUATION OBSERVATIONS This coach appears to be well-respected by teachers and other school officials. Her advice and suggestions are well received by teachers. She could strengthen her practice by using questions to prompt and elicit teacher reflection, encourage teachers to identify actions themselves that could improve their practice. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 175 COACH CASE STUDY 2 PROFILE Qualifications Advanced Certificate in Education (2009) Further Diploma in Education (2006) Diploma in Education (1993) Experience of teaching Teacher, Primary School (1994–2000) Head of Department, Primary School (2000–2018) Experience and training in coaching RSP training provided by OUP and Molteno on the lessons plans and components of language. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:9 Ratio of coach to RSP teachers 1:43 Self-reported frequency of RSP coaching teachers Three teachers per day / ten teachers over a two-day period Data Source The Coach was shadowed in the week of September 2, 2019, during which she was interviewed and observed at four schools working with nine individual teachers (however one teacher fell ill during the lesson and the observation was terminated). The Researcher was joined by the Khulisa Coaching Expert and Khulisa Associate on September 2, 2019 for quality assurance purposes Duration of practicing as an RSP coach 9 months 176 Coach’s understanding of their role as a coach: “My role is to assist and support teachers and HOD’s for the own development.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach assists and develops the teachers and thereby gains their trust. It is important that confidentiality is maintained. She talks to the teachers to overcome any challenges she, and those that the schools, face. She is seen to be outgoing and relates well to people. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach plans in advance of going to the school, and if for any reason she cannot attend she will make a catch-up plan to cover the work. • She submits her plans to the Head Coach and receipt is acknowledged, but the Head Coach does not sign off or give feedback. She does not circulate to the Subject Advisors, as this is the role of the head coach. • She calls all the schools to inform them about her visits, and she sends a reminder of the dates before visiting the school using WhatsApp. 2. Pre-observation discussion • The coach looks at the learner books to gain an understanding of the progress of the lesson plans (8). • She holds a discussion with the teacher when the teacher shares the lesson plan, which is usually before the lesson. In one observation (out of 8) the teacher did not share the lesson plan and it was not discussed. 3. Classroom Observation • The coach observes if the LTSM is available, relevant, well organized and is being implemented by the teachers. However, it is difficult for the researcher to determine if the coach assessed if the LTSM is relevant or well organized, or being implemented – for example in 8 of the classes she observes it is available but did not document in her coaching tool notes if it is relevant to the lesson being taught. • She checks that the teacher understands the content of the lesson, and this was noted in the coach’s documents (8). With one teacher, she provided suggestions for improvement. • She did not check if the DAR/DEAR was in the timetable and if it was being followed (8). • She observes the teacher to check if learners’ experience and prior knowledge is acknowledged; the introduction is creative and within the context of the planned learning content, and if the outcomes of the lesson are clear. She notes her observations and suggestions for improvement in the coach documents (8). 177 • While the lesson is being implemented, she follows the lesson plan (except in one instance when the plan was not provided to her). During the lesson, she checks and notes down that the routines are being adhered to; the core methodologies for reading focus are used, reading strategies and processes are used during the lesson (except in one lesson it was not clear if she observed this from her notes); if reading skills are taught through focused activities, and if focused activities are integrated within components of language. For example, the coach noted when the teacher read difficult words with the learners before reading the story. She also checked if opportunities for application of new knowledge were created for learners by the teacher. • In four observations she checked that learners with barriers were being catered for, and provided suggestions. For example, she noted how these learners were being supported during one observation, and in another class how the teacher involved them in the lesson activities. She did not check or note down if the learners with learning barriers were catered for in two instances. • The coach checked if the learners were assessed and given feedback, if the planned assessment activities assessed learner development and skills, and if the learners were assisted to reflect on new knowledge, skills, attitudes and values. For example, she noted that learners were assessed in one classroom, but the teacher did not provide feedback to them. • In one classroom, the teacher fell ill after about 10 minutes into the lesson, and the coach then took over the lesson to allow the teacher to leave and report to the Principal and SMT.30 4. School based post-observation discussion • She did not ask any of the teachers if they could identify things that went well after the lesson (8), rather she shared her observation of things that went well during the lesson. • She asks the teacher to identify things they thought they could do differently (8), but she did not ask questions to elicit clarification, prompt reflection by the teacher or to encourage them to identify problem areas (8). • She offers her suggestions which are then discussed with the teacher, and she provides the rationale and explanation as to why she is making the suggestion. She discussed actual and potential barriers for implementation with the teacher, and offered solutions with explanations. • The teacher and the coach worked in partnership to identify solutions, and agreed an action plan (8). • The teacher was taken through the lesson observation document and this was discussed after the lesson (8). • The coach noted and provided feedback on classroom management (6) and curriculum coverage (7) to the teachers, and in all instances, she checked that the weekly routine was being followed. • She provides constructive feedback based on the teacher and learner activities during the lesson. 30 The coach and researcher did not record observations for the 10 minutes of the lesson. The principal confirmed that this teacher was undergoing medical treatment 178 • No goals were set for the next visit, only the date was scheduled (8). 5. After School workshops • The coach did not facilitate any needs-based workshops during the coach shadowing period, and she had not planned any either. • The coach indicates that she utilizes the needs-based after school workshops to train teachers who missed the original LTSM training. “For example, Klerksdorp schools recently had strikes, I went back to train the educators.” CONTEXT FACTORS THAT AFFECT COACHING PRACTICES This coach noted the following challenges in providing coaching services to teachers in the schools: • The condition of the classrooms is not conducive for learners, as the floors are in bad condition and overcrowded – in one school there are 60 learners in the Grade 1 class. Teachers were reportedly asked to contribute money from their own pockets to fix the classroom floors. • If teachers are not present at the school, it is necessary to discuss with the HOD or principal to re-schedule the next visit so that all the teachers can be seen during the month. • There is limited time to do all the coach administrative duties. • The Subject Advisors are not clear of the boundaries or their roles, and say they are “Tired of these people” giving advice to the teachers. This coach noted the following successes in providing coaching services to teachers in the schools: • The coaches are provided with transport to make their work easy. It is best to go to the schools early in the morning before the lessons start. • Sharing the monthly plans with the Head Coach, regular communication with the Head Coach using WhatsApp, and submitting the log sheet to the Head Coach and Principal. • Providing schools and teacher support using WhatsApp. COACHING SUPPORT The coach received a support visit from the Head Coach once, in August 2019. She feels that she does not get sufficient support from the Head Coach. She indicated she receives support from the SMT, and they often make sure that the suggestions she makes to improve teaching and learning are implemented by the teachers. She has not attended any developmental training but did receive training on classroom libraries. However, she was advised not to dwell too much on the library content. 179 EVALUATION OBSERVATIONS The coach is viewed as being professional and her work ethic is positive, and she is able to give extensive feedback. Her skill to encourage teachers to reflect back during feedback could be strengthened, as well as to ensure that she covers all the items on her coaching tool. ________________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 180 COACH CASE STUDY 3 PROFILE Qualifications Advanced Certificate in Education, School Leadership (2012) Advanced Certificate in Education, Curriculum and Professional Development (2009) Further Diploma, Management (2000) Diploma, Primary Education (1993) Experience of teaching Deputy Principal Foundation Phase, Primary School (2010-2018) Acting Principal, Primary School (2010-2011) Head of Department, Primary School (2000-2009) Foundation Phase Teacher, Primary School (1999-2000) Foundation Phase teacher, Primary School (1994-1999) Experience and training in coaching Certificate, Mentorship (2015) RSP training provided by OUP and Molteno on the lessons plans and components of language. Received training in the use of classroom libraries Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:8 (was 1:10) Ratio of coach to RSP teachers 1:55 Self-reported frequency of RSP coaching a teacher Three times per term “I visit some teachers more, for those that are weak.” 181 Data Source She was shadowed in the week of September 9, 2019, during which she was interviewed and observed at three schools working with 12 individual teachers. The researcher was joined by the Khulisa Associate on September, 10 2019 for quality assurance purposes. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “Assist teachers in the classroom, assist the teachers where they need skills, with assessments and hold training sessions.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach has good relationships with the teachers, and the teachers are able to express themselves and engage with the coach. She demonstrates good coaching skills and provides practical examples and is ready to share her experience and suggestions. She has an excellent relationship with the school, where the principal and HOD support her engagement in the school. One teacher asked her to come to her class. She has good communication skills and is professional in her engagement with the school management and teachers. She builds a trusting relationship with the teachers through allowing teachers to voice out issues pertaining to the lesson plans, allowing teachers to raise problems they face in classroom management, and communicating with the teachers. If a teacher phones her, she will assist the teacher and give them help telephonically. The coach felt that she was adequately supported by the HOD and Deputy Principals’ at the schools. She indicated that they accommodate her visits even though they may have planned an activity with the teachers. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • She uses the previous observation reports to strategies on how to develop the teacher as she is able to identify areas teachers need support in. This informs the next month’s plan. • The plan is sent to the Head Coach and input is supposed to be received from the Head Coach, but she said this does not happen, so she does her own planning. She received feedback previously from the Head Coach to indicate that she should 182 communicate directly with the school about the dates, and that she could continue with her plan. • One teacher was not expecting the coach to visit and was in the middle of another lesson, but she quickly changed over to the Setswana lesson. 2. Pre-observation discussion • In all ten observations no discussion was held before the lesson, rather the teacher indicated where in the lesson plan they were and shared the lesson plan. In one school, the coach did not have the lesson plan. • The coach did not check if the teachers understood the content of the lesson in all ten observations, rather she observed the lesson and noted the teachers understanding and communication of the content with the learners. 3. Classroom Observation • In all ten observations, the coach looks at the learner books to understand progress in the overall lesson plan, and to check if learners are being assessed. In one classroom she also looked at the DBE workbooks. • In all of the classes, she follows the teachers’ lesson plan while the teacher implemented the lesson. Seven of the teachers shared the lesson plan but there was no discussion on the lesson, and in three classes the teacher did not share the lesson plan. • She observes if the LTSM is available, relevant to the lesson, if it is well organized and if it being used properly in six classes, but in three all these criteria were not observed. In one class the researcher noted that the coach did not check if the LTSM was being used properly, but completed the other tasks. One teacher indicated to the coach that there is a shortage of readers in the class. • She also checked if the Drop all and read (DAR) / Drop everything and read (DEAR) is on the timetable and being followed. She followed the reading against her reader during the class. At one school all the teachers said that reading was done every day immediately after the morning assembly. • In one school she complimented the teacher on the “Beauty of the library” and took photos of the space. • She observed the teachers’ implementing the lesson plan, and checked if they acknowledged learner’s prior knowledge and experience, if the introduction to the lesson was creative and within the context of the planned learning content, and adherence to the routines in all ten classes. The coach takes notes while observing. • The coach checked that the teacher understands the content of the lesson which she noted in her observation tool. • She observed and was able to report if the outcomes of the lesson were clear in all ten observations. However, in the lesson when the teacher was not expecting the coach, the coach did not have the lesson plan to follow, and the full lesson could not be given by the teacher. Therefore, she was not able to complete her coaching observation report as some items were not observed. • The coach observes if the core methodologies are being implemented, that the new content is clear to learners, that the reading strategies and processes are used during the lesson, and that reading skills are taught through the planned focused activities. She did this in all but three situations as noted by the researcher. She requested one teacher to do a reading activity with the learners as a focused 183 activity, and another to do a reading lesson next time she visited. In four classes she noted that the teachers were adhering to the lesson plans. • She observed the teacher and how the learners were engaged in the practical activities, as well as looking at the learner books to confirm that focused activities are integrated within components of language, as well as that the purpose of the lesson is being achieved. • The coach checks that the learners are working in well organized groups. She walks around the classroom during the lessons and asked the teachers about the learners who looked lost, or provided guidance to the teacher when children were sitting in rows rather than in groups for reading. • In one lesson, she was aware of a group of learners who were inactive, and she spoke to the teacher during the lesson about them. The teacher focusses on the slower learners by calling them by name. She further observed the presentation of the lesson in relation to how learners were interacting and what remedial action the teachers were implementing to address learning challenges. • The coach monitored in all cases that the teachers continuously assess and give developmental feedback to learners, and if the planned assessment activities assess learner development and skills. She also identified if learners were assisted to reflect on new knowledge, skills, attitudes and values. The coach asked one teacher to do the assessment activity, whereupon the teacher took the checklist and assessed learners. 4. School based post-observation discussion • Before giving her observations, she asks the teacher to identify what went well (in seven observations), and in the other three she began by telling the teacher what she thought went well in the lesson. In nine observations she did not ask the teachers to identify processes that could be done differently in the future. With one teacher, she asked her to identify what she would do differently in future. • Although she asked all the teachers for feedback, she did not prompt for teacher reflection. The coach offered her own observations before the teachers had an opportunity to respond. She did not guide the teacher to identify solutions for themselves, rather the teachers agreed with her suggestions. There were two teachers who reflected on her practice and gave her opinion of what worked well during the lesson. • The coach offers her observations of things that went well, and congratulated the teachers on their teaching. She provides practical examples of what the teachers can do, for example on how to give attention to slow learners. She takes each teacher through the lesson observation document and they discuss it. Her suggestions are appropriate and reasonable, and she gives a rationale for each suggestion (except in one situation when the teacher was rushing and did not make the time to discuss). According to the researcher, two teachers were noted to thank her for her suggestions. • Feedback on classroom management and curriculum coverage was provided to all ten teachers, and she gave guidance on how to overcome barriers to implementation. For example, she discussed with a teacher how the teacher could go about undertaking revision with the learners, and in another case discussed how the teacher could assist weaker learners. In one class, the coach discussed how to the teacher could help learners read, as the coach identified a few learners who could not yet read. With another teacher, she indicated that guided reading was done well. 184 • She regularly congratulates and praises teacher for what they are doing well, and was viewed as being diplomatic, giving negative feedback constructively, and she gave factual feedback, which as a result did not make the teachers feel offended. Consequently, the teachers accepted and supported her suggestions. • She covers topics of classroom management, curriculum coverage and the extent to which the teacher is following the weekly routine in her feedback to the teachers. • The coach and the teacher identified action steps and goals for the next visit. But, according to the researcher, she is quick to give suggestions rather than prompting the teacher to reflect and decide for themselves. • She gives suggestions for improvement in a constructive way, and according to the researcher, tries to avoid judgments and bias by giving factual and concrete suggestions. • The coach and teacher spoke about the next visit, and goals and activities were set. The agreed goals included the teacher working more with weak learners, doing revision of the previous two weeks lessons, attendance at a workshop, agreeing the group reading activity to be observed during the next coaching visit, using the tracking took and giving more attention to reading fluency. 5. After school workshops • She had indicated on her monthly plan that she would facilitate five afternoon workshops during the coach shadowing period, one of which was held. • The coach indicates that she held a one-on-one catch-up training with a teacher who missed the LTSM training, however no planning has been done for the other teachers who missed the LTSM training. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES This coach noted the following challenges in providing coaching services to teachers in the schools: • Some schools lack LTSM and library resources, and31 some schools are unable to develop a reading corner in the classroom because the classes are overcrowded. • The length of time for teaching the lesson plans is too short, and the coaches cannot then see how the teacher implements the complete lesson. • Teachers are not always available due to union meetings, sick leave, department workshops or school activities, for example the celebration of Women’s Day. • Making an appointment with individual teachers is problematic, the appointment and arrangements of dates must be made with the school so that all the relevant teachers are available. This coach noted the following successes in providing coaching services to teachers in the schools: • Using WhatsApp to communicate with teachers. 31 Further detail on the LTSM material is provided in the preliminary findings report 185 • The RSP car is available for use to schools that are far away and if they are close, she uses her own vehicle. • She shared that it is sometimes32 necessary to support the teacher by setting up the classroom library. COACHING SUPPORT The Head Coach provided administrative support during the LTSM training, and when she visits the school she will talk to the teachers. The Head Coach has visited her three time this year, once she said she would come but did not arrive. Usually if she is coming, the Head Coach will phone her in the morning and let her know. When school-based issues are reported, for example the overcrowding of the classroom, she reported it to the Principal, and wrote it on her report to the Head Coach who indicated this was the Department of Education’s issue. She indicated that this was beyond her control. EVALUATION OBSERVATIONS The coach indicated that the teachers are good at what they do, and in most cases the learners are able to read and are articulate, so she indicated that there has not been a need for the needs-based workshops. This coach has good relationships with teachers, is experienced and understands the content and process of teaching language and literacy. She is beginning to engage with the teachers to prompt them to reflect on their practice. However, she could encourage and prompt teachers to come up with solutions themselves, rather than providing suggestions immediately to which the teacher agrees on. She is good at supporting teachers with suggestions; however, she does not demonstrate what to do in practice or give sufficient guidance on how suggestions can be implemented in practice. ______________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 32 No names or details of actual schools or classrooms were provided 186 COACH CASE STUDY 4 PROFILE Qualifications Master of Technology: Educational Management (2002) Bachelor of Technology: Education Management (1999) Higher Education Diploma (1998) Secondary Education Diploma (1994) Experience of teaching Circuit Manager (2016-2019) School Principal, Primary School (2000-2006) Teacher (1996-2006) Experience and training in coaching RSP training provided by OUP and Molteno on the lessons plans and components of language. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:8 Ratio of coach to RSP teachers 1:46 Self-reported frequency of RSP coaching a teacher Each teacher is visited twice a month although some teachers are not available due to workshops, meetings, trainings or SADTU meetings. New teachers or teachers who have not attended the training are visited more frequently. Data Source Coach 4 was shadowed in the week of September 2, 2019 during which she was interviewed and observed at four schools working with eleven individual teachers. The Researcher was joined by the Khulisa Coaching Expert on September 3, 2019 for quality assurance purposes 187 Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “Guiding and supporting educators on the effective implementation of strategies and processes. The coach measures the performance of the teachers in line with the lesson plan prescription.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials She was described by the researcher as being “Friendly, welcoming, respectful, humble and very eager to help. She is not judgmental. She does not impose on the teacher, and has a good relationship with the teacher. She has an open door policy. Good personality, and also established a good rapport with the researcher. She goes the extra mile.” She does not share her classroom observations with the Principal as they are confidential. The teachers appear to enjoy working with her, as she is passionate and positive about being a coach. Coaching Standards THE FIVE STAGES OF DEVELOPMENT OF THE COACH This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach submits their monthly plan to the Head Coach. For this month, the Head Coach provided feedback, and the coach had to adjust her plans to accommodate provincial and Head Office activities. Once revised, she submitted to the Head Coach, but had not yet received feedback. • The coach sends the schedule of visits for the month to the School Principal, Department Heads and Foundation Phase teachers and in all 11 observations, the visits are confirmed using WhatsApp groups. The coach then calls each teacher to re-confirm his or her appointment. 2. Pre-observation discussion • She trains the teacher on the content of the scripted lesson plan, and sets goals and targets with the teachers, which are then monitored over the month. • In all 11 observations, the coach and the teacher briefly talked about the lesson plan and compared it to the weekly routine before the lesson, and checked if the teacher understood the content of the lesson in, however in one situation this did not occur. • She checks the learner books in eight observations, and in three situations she checked a sample of the books. 188 3. Classroom Observation • The coach checks the lessons against the RSP tool to determine the curriculum coverage during the visit to the school. She checks that the teacher is on track, and if not, what the challenges are. They discuss how these can be addressed, and how the teacher can catch up lessons. • The coach, when in the classroom, supports the teacher and is fully engaged in the lesson. The coach is able to demonstrate the lesson to assist the teacher to improve their teaching. • In all observations, the coach checks that the LTSM is available, that it is relevant to the lesson, and if the teacher understands the content of the lesson. The coach observed that the LTSM was well organised for the planned components and were being properly used for the lesson plans (11 observations). • The routines for Drop All and Read (DAR) and Drop Everything and Read (DEAR) were noted to be well established in all but one class where this was not observed. She also checks that learners were well organised in groups. In only one class the coach noted that the leaners were not well organised in groups as some learners were reading in the front of the class while others were idling with nothing to do. • The coach always checks if the teacher’s introduction acknowledged learners’ experience and prior knowledge, this was noted for example to be through the use of dialogue (1), questions (2) and showing a birthday card and discussing days of the week (1). In two classes, the coach noted that a vibrant song was used to introduce the lesson, but the song was not relevant to the content of the lesson. • The introduction was noted to be creative and vibrant in class, and the researcher noted the use of songs by four teachers. However, in one situation the coach did not note this in her tools, as observed by the researcher. • She observes and makes notes on whether the outcomes of the lesson were clear in all situations. • The coach always notes that the relevant core methodologies were being used to help learners understand the content based on the lesson plan being taught. • The coach noted that the teachers were using the reading strategies and processes continuously during the lesson, that reading skills were being taught through focused activities which were integrating components of the language. In one class, the coach did not observe or record these according to the researcher. • She noted that learners with learning barriers were provided with differentiated activities in eight observations, such as learners in seven classes being given more time and opportunities to read. In three classes the coach was not observed checking or noting this, and in another the coach noted this, but gave no developmental feedback to the teacher. • She further documented for all teachers if the teacher created opportunities for learners to apply new knowledge, for example, one teacher gave learners an opportunity to play the characters in a dialogue to apply the new knowledge. • The teachers were also observed in nine classes to determine if they were giving continuous assessment to the learners (in five classes, the coach observed that the teacher needs to give more developmental feedback). The researcher did not observe the coach observing the teacher doing the continuous assessment, but noted the assessment activities that were planned by ten of the teachers, would 189 assess the learner development and skill. In one class planned assessment activities were not observed. • She further checks if learners are assisted to reflect on new knowledge, skills, attitudes and values – she noted that in one class the teacher used a written activity, and in another class the teacher asked questions. In one class the coach did not check this element. 4. School based post-observation discussion • Before providing her own observations, she asked the teachers to identify things they felt went well and what they would do differently in future (10). However, for one teacher this did not happen, as the coach only gave her own feedback. • The coach offers her observations of things that went well (11) and offers her suggestions on what to do differently or areas for improvement. For example she recommended to one teacher, that they give more development feedback to learners when they answer correctly to encourage others. For another teacher, she gave guidance on how to control groups that are not in the front with the teacher. • She is able to ask questions to elicit clarification and prompt reflection regarding areas of improvement and for the teacher to identify solutions for themselves, as demonstrated in eight observations. The teacher is given an opportunity to identify action steps they can take (8). She is using questions to prompt reflection rather than telling the teacher their problem areas, but this was not observed in four classrooms. With one teacher, the researcher noted that the coach demonstrated the correct implementation of the lesson plan but did not engage with the teacher on the teacher identifying what to change, nor provide a rationale or help the teacher to successfully identify action steps. However, it was noted that she provided the demonstration in a constructive way. • She goes through the lesson observation document with all the teachers explaining her observations and insights after the lesson. The coach goes through areas of the teacher’s classroom management, curriculum coverage and the extent to which the teacher follows the weekly routine, and notes if they are on track and adhering as prescribed. The extent to which the lesson complied with the core methodologies are also noted (10). However, this was not observed in one class. • She demonstrates lessons for teachers during the feedback session, and shares ideas and suggestions for improvement with the teacher when necessary. Her suggestions were viewed as being appropriate and reasonable in scope by the researcher. She shares her rationale for her observations, and she provides relevant examples. She was observed avoiding judgement and bias when providing her observations and suggestions. • The next visit was scheduled, and goals were set for the teacher to work on in all instances. 5. After school workshops • Training on LTSM is provided for those that never attended training by the coach. The training comprised of a 30 minute session to demonstrate to the teachers, followed by sitting with the teacher discussing lesson plans and demonstrating what they need to do. • She planned three workshops for the week of coach shadowing, and two were held, which included one combined workshop between four schools. 190 CONTEXT FACTORS THAT AFFECT COACHING PRACTICES This coach noted the following challenges in providing coaching services to teachers in the schools: • Resistance by some of the teachers to the RSP. • Difficulty in scheduling coaching due competing priorities, for example department meetings, or memorial services teachers attend. Coaches cannot dictate time or priorities to the schools. • Non-alignment of times when the coach must attend school between 8:00 a.m. and 4:00 p.m. however the school begins at 7:30 a.m. and ends at 2:00 p.m. • Teachers are not able to remain after school due to transport arrangements. • Lack of support to coaches to address these challenges. This coach noted the following successes in providing coaching services to teachers in the schools: • The use of the company fleet (vehicles) and card for fuel and services. • Schools stamp the log sheet to confirm the coach attended the school. • The use of WhatsApp groups including video clips to teachers. • Building a trusting relationship with the teacher and providing practical modelling of how to implement lessons when they are having challenges. • Not being judgemental, but helpful. COACHING SUPPORT The coach does not feel supported by the Head Coach but feels adequately supported by the School Management Teams. The Head Coach provides resources and cascades information from the national and provincial offices. EVALUATION OBSERVATIONS This coach consistently checks and guides the teachers in their implementation of the lesson plans, and provides constructive and relevant feedback to coach and improve the teacher’s delivery of the lessons. There are only a few instances when this is not done. She is using reflective methods and questions to encourage the teachers to reflect on their practice and to come up with solutions for themselves – this was only observed not to occur in one instance. __________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; researcher debriefing interview September 2019 191 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 192 COACH CASE STUDY 5 PROFILE Qualifications Bachelor of Education (Honors) in Learner Support (Current) Advanced Certificate in Education, Human Relations and Social Development (2009) Further Diploma in Education Management (2006) Junior Primary Teacher’s Diploma (1997) Experience of teaching Inclusive Education Advisor, Education Department (2017–2018) ECD Specialist, Education Department (December 2013) Teacher, Primary School (2012 – 2013) Teacher, Primary School (1997 – 1998), (2000 – 2010) Principal, Primary School (1999) Experience and training in coaching RSP training provided by OUP and Molteno on the lessons plans and components of. This coach indicated that “coincidental or unplanned” training on classroom libraries took place. This coach also has experience as a Soul Buddyz Facilitator. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:7 Ratio of coach to RSP teachers 1: 50 Self-reported frequency of RSP coaching a teacher Two or three times per term 193 Some teachers are prioritized if necessary, for example teachers who have been absent for some time due to problems such as illness. Data Source The coach was shadowed in the week of September 2, 2019 during which she was interviewed and observed at three schools working with seven individual teachers (one teacher was observed for both EFAL and HL). The researcher was joined by the Khulisa Associate on September 5, 2019 at Doornbuilt for quality assurance purposes. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “To support teachers in the teaching of language, support the teaching of learners with barriers, intervention where teachers need support with curriculum and demonstrate lessons for teachers.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach values the relationships she has with her teachers, and works hard to earn their trust by keeping her promises to them and by being honest. She only reports positive matters to their principals so that their confidence in her, and the confidentiality between them, is built up and maintained. When she meets with her teachers she is received with warmth and ‘’Works as a partner and colleague.’’ Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach says that she plans her activities for the month, and diarizes these. She says that she usually shares her monthly activities with the Head Coach and teachers, however, the schedule for the month of the research visit was not available. The researcher is uncertain about how often monthly planning takes place, and indicates that at times the coach offers ambiguous responses about it, but that the coach also blames the Head Coach for this issue. • The coach calls all teachers prior to the session to confirm that it will take place, and teachers are contacted to check their curriculum coverage. • Teacher profiles for some teachers were created/viewed before the visit. 2. Pre-observation discussion 194 • The coach looked at learners’ books to gain an understanding of overall progression for only one teacher out of the seven, for their Home Language and EFAL observation. • The coach does not usually have a formal conversation with the teachers about where they are in their lesson plans, nor does she look at their lesson plans, (only recorded twice out of eight observations), but in four observations the researcher noted that despite this, there was a common understanding between the coach and the teacher on the lesson for the day. • When the coach does explicitly discuss the lesson plan (three out of the eight lessons), it is discussed more on a thematic level rather than specific details of the lesson. • For three out of the eight observations, it was noted that the lesson commenced without much communication between the coach and the teacher. 3. Classroom Observation • In three instances, the researcher noted that the coach briefly utilizes the monitoring tool to make observations about the lesson. In two instances the researcher noted that no formal notes were made by the coach. The coach self-reported that she uses a notebook to highlight areas where intervention is needed. • In six instances, it was not evident to the researcher that the coach checks to see whether LTSM is available, relevant or well organized. Her approach seems sporadic, as some of her classroom monitoring tools indicate that she did check LTSM (such as one dated August 21). In one instance, the coach checks this thoroughly with the teacher and they discuss which posters and flash cards will be used for the lesson. • In one instance, she did not check whether there was DAR/DEAR scheduled in the timetable. DAR/DEAR was not recorded for four observations – but instead ‘’READING FOR FUN’’ was listed. In three instances, timetables were not available for the classes, either because the school did not have resources, or because the timetable had to be redesigned. • This coach has completed courses on ‘Learners with special needs’, which she utilizes frequently to support her teachers. In two separate instances she advised a teacher on how to support a learner with a speech defect, in another instance she observed which learners had barriers to learning and in another observation the coach noted a girl with a sight problem who was asked to read from the board – and then indicated that a follow-up would need to be done. • In all but one classroom observation, the coach checked to see if reading strategies and core reading methodologies of Shared Reading, Group Guided Reading, Paired/Independent Reading and Phonics were utilized. In two instances the coach intervened to guide the teacher; once during a phonics lesson and several times during a group guided reading session. In one observation, the coach emphasized the importance of these strategies, but they were not utilized in class. In one instance, the teacher code-switched as a way to teach reading, and the coach verified this as an appropriate strategy. • In all eight observations the coach checked – either formally or orally – whether learners are continuously assessed, given developmental feedback and whether the planned assessment activities assess learner development and skills. In four instances the coach emphasized formative, informal assessment. In another instance the coach highlighted that teachers are more inclined to summative assessment, and 195 in another observation the coach stated that teachers need more training to fully consider formative assessment. 4. School based post-observation discussion • In one instance feedback and post-observation discussion did not happen at all as there was a training workshop at another school. • In all observations (seven out of the original eight) the coach made comments to the teacher on how to improve their teaching based on what she had observed. Comments made were usually around classroom management, but in some cases on methodologies utilized or curriculum coverage as well. Comments ranged from re￾emphasizing particular elements of the RSP, to providing specific strategies such as dividing learners into different groups, the suggestion of a reading mat for the classroom or maintaining eye contact with the learners. • The coach provides feedback in a positive way, as twice she complimented the teachers before she identified the area for growth. One teacher was told that they “Followed Phonics Steps very well” and that “Letter formation gaps were identified.” Another teacher was praised for an orderly classroom before the coach said that the teacher should pay more individual attention to students. • In two observations it was noted that “Teachers felt free to discuss with the coach,” in one instance the comment is that the “Teacher did not talk much during the feedback session.” • In four instances the researcher comments that the teachers were required to sign the Lesson Observation Document, but that the teacher had used a separate document to write down the coaching points. In one observation the researcher noted that the Lesson Observation Document that was used was an old one that was dated 08–21-2019. • No goals were set in any of the seven observations for the next visit. In one observation it was noted that goal setting was “Not planned for” in that session. 5. After School Workshops • The coach indicated that she was planning to do a workshop the week after the researcher was observing her (week ending September 13, 2019), but that she was not sure how often nor how long these workshops should be. • The coach reported that she had previously provided catch-up training to teachers who missed the LTSM training. She reported that after approval, and invitation to affected teachers via the Head Coach, the training was cascaded. The researcher did not observe this training. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The researcher and the coach noted the following challenges in providing coaching services to teachers in the schools: 196 • Shortage of teachers and the appointment of unqualified teachers. • In four observations (two of these were about the same teacher) it was noted that the classes did not have a teacher for Term 1 or Term 2 (implying formal education only began in the third term). It was then noted that these three teachers were only qualified to teach Grade – R (teaching two Grade 3 classes, and two Grade 1 classes). • One school was noted to have a poor environment which the coach and researcher felt would have a negative impact on teaching and learning (dirty classrooms, broken windows, missing doors). • Vandalism and community members stealing or damaging LTSM. • The researcher noted that the schools are not in close proximity, and that the coach might get tired driving such long distances and that it was not feasible to arrive at the school as the school day was starting. The researcher also felt that the number of schools and teachers that the coach was supporting was too high. The researcher and the coach noted the following successes in providing coaching services to teachers in the schools: • Team teaching or demonstrations by the coach to overcome some challenges • The use of WhatsApp to provide remote support • Identification of learners with barriers and assisting with remedial action • The researcher noted that teachers said that they feel their children are more advanced compared to those who have not been part of the project. • The researcher noted that the coach was working to change both the physical aspects of the classroom (including getting boxes for LTSM), but also changing the teachers teaching by building their confidence COACHING SUPPORT This coach does not feel adequately supported by their Head Coach. She indicated that she receives about “Five to ten percent support on curriculum” from the Head Coach, and “A bit more’’ support for administration. She does feel supported by the School Management Teams, but notes that due to limited financial constraints that support is strained. She has not received any training since the initial RSP training. She notes that her training on classroom libraries was “Unplanned or coincidental”. EVALUATION OBSERVATIONS The researcher describes this coach as passionate about her job and coaching, and that she takes her work seriously. This coach seems to use her discretion with each teacher as she varies how she approaches the five stages of coaching practice. She works hard to establish and maintain good relations with them, so that they trust her and would listen to her advice. Her approach is not very structured or rigorous in terms of following/checking lesson plans, or schedules, and the researcher did not think she started on time. An area for development for this coach would be on the administrative and organizational side of coaching. 197 However, of this coach, the researcher said “Once in class she does her duties remarkably well. She understands issues of curriculum, classroom management, learner management as well as LTSM utilization.” _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 198 COACH CASE STUDY 6 PROFILE Qualifications Adult Basic Education and Training Certificate (1999) Secondary Teachers’ Certificate (1990) Primary Teachers’ Certificate (1969) Experience of teaching Teacher, Subject Head (English), High School (1983–1997) Teacher, Primary School (1972–1982) Experience and training in coaching Reading Coach, EGRS North West: Setswana (2017) Coach: Foundation Phase teachers in HL, EFAL and Mathematics, Class Act Educational Services (2011–2013) Trainer: Teachers–reading and writing, 3 Read Educational Trust (2001–2004) Facilitator, Open Learning Group (1998–2001) RSP training provided by OUP and Molteno on the lessons plans and components of language. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:8 Ratio of coach to RSP teachers 1:40 Self-reported frequency of RSP coaching a teacher Two or three times per term, depending on the number of teachers per schools. “Sometimes in big schools I see a teacher once … I see the weaker ones more.” She acknowledges that she is supposed to see each teacher once a month. 199 Data Source Coach 6 was shadowed in the week of from October 21, 2019. She was interviewed and observed at two schools working with four individual teachers during six lessons. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “A critical friend. A mentor, I empower teachers.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach feels that she builds trust by being very open with teachers, and by making sure they understand what is expected of them. She says she shows empathy and sensitivity to individual issues, so that teachers can confide in her. The researcher describes her interaction with teachers positively: that she works well with them and that a good working relationship exists between them. She “Explains with respect” and often congratulates and thanks teachers at the end of the lesson. Once, when teacher saw the coach, “She was so happy and [she] even hugged her.” The researcher further describes the coach as being objective, good at providing facts and giving examples of what should be changed and “Her feedback is constructive.” He elaborates that she is very vocal and can come across as strong when providing feedback, but explains her reasoning well. For example, when she corrects teachers, “She has a way of telling the teachers their wrongs [so] that they will agree and admit that they are wrong although negative at times”. It seems she attempts to handle providing critical feedback diplomatically, as she “Will try to make it not look bad by congratulating teacher after another attempt” when correcting teachers in front of learners, to ensure that learners are taught correctly. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach prepares for school visits by drawing up a schedule and sending it to the Head Coach. Once approved, she distributes the schedule to teachers/schools. For September, it was circulated on August 29, 2019. • She sends WhatsApp messages to all teachers about the scheduled visits, but does not call–she explained that she only calls when there is a diversion. 200 • For the month of September, the coach did not call each teacher to check curriculum coverage, nor reviewed teacher profiles, as she was ill (absent for two weeks). 2. Pre-observation discussion • Most of the time, the teacher briefly shows the coach which lesson is going to be presented from the lesson plan (four times). Once, the coach had to request the plan first. During the other two lessons, no discussion around the lesson plan took place before the start of the lesson. • The teachers usually do not introduce the coach to the class before starting with the lesson (not observed six times). The researcher noticed that the teacher greeted the coach twice, and during the other four lessons, started or continued with the lesson. Once, the teacher did not seem comfortable with the amount of visitors in the classroom. • The coach manages to discuss the lesson plan with the teacher properly most of the time (observed during four of the six lessons). However, she fails to allow the teacher to properly share the lesson plan for the day; during three lessons, this was not observed, and during the other three lessons the teacher handed the lesson plan over, but did not share it properly. • She can improve with regards to checking learner books to understand progress in the overall lesson plan, as this step was not observed four times. The researcher notes that during three of these lessons, only reading/oral activities took place, and during the two other lessons, the coach walked over to tables to review learner books. 3. Classroom Observation • Regarding lesson preparation, the coach does well to observe if LTSM is available and relevant, plus if the teacher understands the content of the lesson (all observed six times). For example, she mentioned the faintness of the pictures and suggested more legible copies. She also corrected the teacher when an incorrect answer was given and explained the correct answer. However, although she did check the LTSM on the wall/checked flash cards, and followed the lesson plan throughout, it was not clear whether she properly observed whether LTSM is well organized (not observed four times). • Although the coach always seems to checks if the routine for DAR/DEAR is well￾established (noticed six times), there was no timetable and the coach indicated that DAR/DEAR is not practiced at the school (during all six lessons). • The coach almost always checks if learners are working in well-organized groups (fully observed five times). The researcher notes that the coach moves around the arranged groups of learners, observing them, their work or the flash cards. During the sixth lesson, it was noted that learners were seated in rows, labelled with animal names–the coach indicated that she prefers this grouping to when tables are placed together. • During lesson implementation, the coach always observes if LTSM is properly used, and makes notes on whether the outcomes of the lesson are clear (both steps noticed six times)–she follows the lesson plan, checks the LTSM against the lesson, observes intently, constantly makes notes and objected once when LTSM was used incorrectly by the teacher. However, most of the time it was not clear whether she specifically checked whether the introduction acknowledges learners’ experiences and knowledge, whether it is creative or within the context of the planned learning content (each step was completely observed once during the six lessons). During 201 one of those lessons, she did intervene when the teacher was incorrect on a specific point. According to the researcher, she handled the correction appropriately and congratulated the teacher after the lesson was re-taught as per the coach’s request. • She performs well with regards to checking if routines are adhered to (by checking the lesson plan throughout the lesson and making notes–observed six times) and if focused activities are integrated within components of language (observed six times). For example, she discussed fluency with the teacher twice, corrected the teacher when an error occurred relating to riddles, and assisted learners with pronunciation when reading. • The coach also manages to check consistently if opportunities for application of new knowledge are created for learners (observed six times). The researcher notes that she encourages teachers to let learners use their own words and sentences (three times) and repeat riddles for learning. Most of the time, the coach checks if reading strategies and processes are used during lesson (observed four times). For example, she was actively observing the teacher while reading and suggested that the teacher should include learners and answer questions. She also highlighted that sight words should be read first, and observed learners doing shared reading. • Areas of improvement for this coach, with regards to teaching and learning, include checking if reading skills are taught through focused activities–the researcher did not fully observed this during four of the six lessons. However, once, she reminded the teacher what emergent reading skills are. Secondly, she rarely checks if learners with learning barriers are catered for via differentiated activities (fully observed twice). During those two lessons, she walked between learners and spoke to the teacher about learners who were placed far from others. • She is reasonably consistent in checking if the core methodologies for reading focus are used to make new content clear to learners (observed five times). For example, she spoke to the teacher about methodologies and made sure that the teacher follows proper methodologies twice, sat next to a group doing Group Guided Reading on the carpet, and discussed incorrectly used methods during the feedback session. • With regards to assessment, the coach consistently checks if learners are assisted to reflect on new knowledge, skills, attitudes and values (observed six times). For example, she advised the teacher to allow the learners to come up with own words and sentences (four times), observed learners reading and once asked the teacher to repeat an activity and let learners answer questions. • She can improve on checking if learners are continuously assessed and given developmental feedback (not observed during three lessons), and whether planned assessment activities assess learner development and skills (not observed during four lessons). However, she did occasionally perform these checks well, for example by checking learners’ books while they were writing (three times), and discussing the checklist with the teacher when Shared Reading was performed. 4. School based post-observation discussion • The coach seems to handle the feedback process well. Most of the time, she completed all the required steps (observed five times during six lessons). Interestingly, during one lesson, none of the feedback process steps were recorded as observed, although it was well-covered in the other classes76 . She offers her 202 observations of things that went well (five times), for example by congratulating the teacher (three times), and explaining what she liked about the lesson. She asks the teacher to identify things they would do differently in the future (especially relating to reading), offers suggestions, and asks questions to elicit clarification and reflection on areas of improvement, almost all of the time (all observed five times). For example, she sometimes asks teachers to reflect on a specific activity on “How to do things better”. • Most of the time, she asks the teacher to identify the things they felt went well, first (observed four times), takes the teacher through the lesson observation document, (observed five times), and guides the teacher to successfully identify action steps (observed five times). Once, she asked the teacher to rate her own lesson and explain her rating. However, it is not clear whether she guides the teacher to identify solutions for problem areas through action steps properly–as this was fully observed during half of the lessons (three times). • The coach covers the required topics under the lesson observation document well, as this step was observed successfully during five of the six classes. On classroom management (covered five times), she emphasizes aspects such as having a print rich environment, black board space, and learner discipline. For curriculum coverage (five times), she confirms which week is being covered. On the extent to which the lesson complies with the core methodology, the coach gave constructive feedback, explain topics such as Group Guided Reading, Shared Reading, teaching sight words before reading, and using methodologies in full. • The coach performs well relating to content, as she almost always covers all steps successfully (all observed five out of six times). She helps teachers identify solutions to potential barriers to implementation (five times), and offers suggestions that are appropriate and reasonable in scope (five times), for example using posters and correcting the arrangement of books in the box, or explaining why sight-words should be done before reading. • She almost always provide examples of correct implementation with suggestions (five times). For example, she covers Group Guided Reading, how to keep other learners busy, Shared Reading and the Point-Ask method. The coach usually guides the teacher to successfully identify action steps (five times), and provides a rationale for each suggestion offered (five times), such as explaining why vocabulary words should be taught first, or why she would rate the teacher a certain way. • Her communication skills are strong. She offers suggestions for improvement in a constructive way (observed five times), providing positive comments first, followed by “Negative comments”, but in a polite and objective manner. She avoids judgement or bias when providing observations and suggestions, focusing on specific points or going through the lesson observation tool, speaking for example about phonemic awareness and sight words. • She also uses questions to prompt reflection rather than telling the teacher their problem areas (observed five times). The researcher explains that she prompts teachers a lot, or asks teachers to rate themselves first. Then she explains why what rating she would give the teacher, and why. The coach usually sets goals with the teacher for the next visit (observed five times). For example, they agree that the teacher will start the lesson with an introduction song, teach words before reading, use the steps in the lesson plan to teach reading, or teach vocabulary of words before letting learners read, and so forth. • The researcher further elaborates generally around the coach’s feedback process. He notes that when the coach provides feedback, she will, for example, ask the 203 teacher to explain how a certain aspect should be taught, and then ask him/her to describe how they (the teacher) performed, sometimes guiding the teacher to realize where they went wrong. The coach also asks teachers to rate themselves, and then provides her own rating with rationale of the specific score. The researcher notes that sometime teachers then lower their own rating, because they realized that they were not performing the task correctly. 5. After school workshops • The coach did not facilitate any needs-based cluster training during the coach shadowing period, although four sessions were planned. • The coach indicated that no teachers missed the LTSM training but that two newly appointed teachers were “Inducted into RSP” in September and October. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach noted the following challenges in providing coaching services to teachers in the schools: • She does not always manage to visit the teachers she has planned for the month due to: school activities, teacher absenteeism and personal commitments. • The coach feels that not all her developmental needs are met, for example she lacks IT skills (such as excel), for which training has not been provided. • Another difficult aspect of the role she mentioned, is working with teachers who have negative attitudes and who do not implement tasks as required. She does address this problem with principals directly. • The coach also mentions break-ins at schools as challenges. • She feels that creating print-rich environments on the walls, with what they have, is also a challenge. The researcher seconds this point, by stating that most of the classes are not very print-rich. He elaborates that some classrooms have some words on their walls, sometimes the wall or paint does not allow pictures or posters to stick. The researcher noted the following challenges: • Although it seems that the WhatsApp group platform improves the distribution of schedules and communication between the coach and teachers, it was noted that the coach circulated the approved school visit schedule for the month of September to teachers/schools only two working days before the beginning of the month (August 29, 2019)–which is very short notice. • During all six lessons, the researcher observed twice that the teacher is not successfully following the RSP routine. Once, the researcher highlighted that the teacher does not seem to be following the RSP routine properly, and in another lesson, the teacher was “Not motivated at all to teach and learners seem not to be used to [the] RSP routine.” • Some concerns around infrastructure are noted. For example, one class does not have a carpet, and learners pull/drag chairs to go to reading area, while another classroom does not have enough books for everyone to read simultaneously. 204 • Although the researcher provides positive feedback around most teachers’ performance, once, serious concerns were raised. He states that the teacher “Is not doing her job at all and learners are lost and cannot read well.” Apparently the coach indicated that this teacher is sickly and absent from school most of the time. He elaborates that there seems to be a mutual understanding between the coach and the teacher, but that “Work is not done and there is a need for proper monitoring and support.” This coach noted the following successes in providing coaching services to teachers in the schools: • The coach did not explicitly note any successes. • The researcher emphasizes that the coach is skilled, knowledgeable and have a good relationship with teachers. Providing detailed feedback to teachers in an effective, constructive manner, is one of her strengths. He feels that “With enough SMT monitoring and support, the school can be on the right track.” • During the six lessons, the researcher notes four times that teachers seem used to following the RSP routine–that the teacher was making sure that the learners were actively participating in the activities, and once started the class with riddles. “Learners enjoyed themselves.” COACHING SUPPORT The coach feels adequately supported by the Head Coach, who approves the school visit schedule, accompanies her on school visits, provides advice via WhatsApp and helps when problems outside the coach’s control, occurs. She also feels adequately supported by the SMTs. However, generally, she does not feel support is readily available to deal with challenges that are beyond her power. EVALUATION OBSERVATIONS She is an experienced coach and teacher, having been coaching since 2011, having 25 years of teaching experience (including 15 years as Foundation Phase teacher). The researcher describes the coach as skilled and knowledgeable, and that she knows her curriculum and advises teachers well. As explained in the sections above, she does provide constructive criticism to teachers when providing feedback, but according to the researcher she is “Very experienced in coaching and with methodologies as she explains it so well when giving feedback.” Based on the input provided, she seems to be passionate about improving learning in these schools. The researcher also notes that she is a strong-minded individual, and that she did come across as negative when she initially called to set up the shadowing appointment. However, when they met, she was very different, and she “Seems to be true to her job.” When their interview started, she “Was not open, but as [the] interview progressed, she opened up.” She refused to discuss certain topics, such as district officials and what she would change about the RSP program. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations 205 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 206 COACH CASE STUDY 7 PROFILE Qualifications Bachelor of Education (Honours) (2013) Advanced Certificate in Education (2009) Certificate in Education Management Development (2003) Certificate in Managing Performance and Discipline at Schools (2003) Higher Education Diploma (1999) Secondary Education Diploma (1996) Secondary Education Certificate (1992) Primary Teachers Course (Inservice) (1986) Experience of teaching Principal, Primary School (2005-2017) Principal, Primary School (1999-2004) Teacher, Middle School (1989-1999) Teacher, Secondary (1981-1998) Experience and training in coaching RSP training provided by OUP and Molteno on the lesson plans and components of language. The coach indicated that he received “Training offered by ladies from Wits” in reference to classroom libraries, and that “They promised to come and offer more training and that has not transpired.” Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:10 Ratio of coach to RSP teachers 1:52 207 Self-reported frequency of RSP coaching a teacher Four times per term for each teacher. If the coach deems the teacher to be progressing slowly, he might visit them more. Data Source Coach 7 was shadowed in the week of September 2, during which he was interviewed and observed at three schools working with 11 individual teachers (12 observations, one teacher was observed for EFAL and HL). The researcher was joined by the Khulisa Coaching Expert on September, 5 2019 at Doornlaagte for quality assurance purposes Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: The researcher notes that the coach sees his role as a coach to assist and support the Foundation Phase teachers and find solutions for the challenges they face. Furthermore, he assists them with the interpretation of the lesson plans, observes the teachers and advises them on how to implement new strategies so as to improve reading in schools. Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach is described as a leader and that this skill helps him to coordinate the teachers. The researcher wrote of the coach that “His patience helped in building relationships, building trust, he seems like somebody who understand(s) their needs.” He navigates the teaching and learning environments well. He is described as empathetic, patient and having high emotional intelligence. The researcher wrote in one observation that even the learners were happy to see the coach. He emphasises the importance of keeping promises and timeous communication in building relationships with his teachers. He does not report on classroom activities with the principal, but says he is supported by the SMT because they “Allow’’ him to work with their teachers. There is some role clarification needed here, as the coach is not clear that he should be reporting to the principals as well. The coach also notes that there is “No proper working relationship’’ with the Subject Advisors. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation 208 • The coach utilizes the monthly plan, and sends these to the Head Coach, however at the time of interview had yet to receive feedback on the monthly plan. The coach also circulated the monthly plan to all the teachers before the 1st of the relevant month. • The coach indicates that he calls the teachers on a weekly basis to confirm visits, but that he prefers to discuss curriculum coverage face-to-face. • The coach has never created or reviewed teacher profiles. • In planning for the session the coach, communicates with the Foundation Phase HoD, checks the lesson plan, schedules his sessions and confirms the availability of his teachers prior. 2. Pre-observation discussion • In all 12 observations it was noted that the coach checked learners’ books and noted their progress as either sufficient or reasonable. • The coach was rigorous in the checking of learners’ books, discussing the lesson plan and ensuring the teacher identified the activity to be covered for the period. • While checking the progress of the learners, he was sure to ask for clarity from the teachers where he did not understand, and also compared the learners’ progress to the lesson plans and where the teacher indicated the class was to ensure alignment. 3. Classroom Observation • The researcher observed the coach over two Grade 1 EFAL lessons, four Grade 1 HL lessons, three Grade 3 EFAL lessons and three Grade 3 HL lessons. • Two teachers (one Grade 3 and one Grade 1) were observed for both their EFAL and HL lessons. • In all 10 classrooms, the coach checked to determine whether LTSM was relevant, available and well-organized. The coach also noted areas for additional support or the conditions of the LTSM. In one class the coach noted that LTSM was well organized, but that the space for storage was not sufficient. In two instances, the teachers told the coach that they were still running short of books. The LTSM was well-organized except for in two instances, one where the EFAL books were “Just mixed with the HL books,’’ and another where it was noted as being poorly packed due to no shelves and stored in a dirty and dusty classroom. • In all 12 observations the coach ensured that the teachers understood the contents of the lesson, and in three observations asked further questions to verify this. • In all 12 observations, the coach checked the routine to determine whether DAR/DEAR is adhered to and being followed. In seven observations, DAR/DEAR is well adhered to and implemented fully. • In two observations, DAR/ DEAR is being implemented with challenges, but with one teacher struggling to adhere to it all the time (with no specific reason given), and in another instance DAR/DEAR is adhered to, but not fully “Because the learners are using common transport from this farm school.” This is a significant issue as in another school, the coach noted a reason for the success of DAR/DEAR as “It was not a challenge for learners because they do not have to rush for common transport and have time to read instead.” • In three instances DAR/DEAR is established and complied with however resistance is noted. In one instance, “The teacher mentioned that the subject advisors are adamant that they need to use the timetable according to what and how they 209 suggest and not according to what the RSP suggest(s).” In the two other instances similar challenges are observed; “The subject advisors and life skills facilitators are giving the teacher a hard time regarding the use the timetable” and “The timetable should be done by the SGB, so that arguments and confusions are avoided.” • In all 12 observations the coach checked whether LTSM is being used properly (and in only one instance was a teacher confused about its usage), checked that the introduction to the lesson was creative and acknowledged learners’ prior knowledge, and the coach noted that the lesson outcomes were clear. • In all 12 observations the coach checked to see whether routines are adhered to, and in all cases they were. One teacher indicated that she did sometimes struggle with the routines, but that she tries her best, and another teacher maintained she had to balance the quality of teaching and not rush as some learners were slower than others. • In all 12 observations the coach checked whether core methodologies, reading strategies and reading skills were taught and utilized. • In three instances the researcher did not observe the coach checking whether focused activities are integrated within components of language (vocabulary, word recognition, fluency etc.). In five instances the researcher noted that the coach did check this and that “The teacher made sure that the components of the language were used. For instance, idioms were used as part of vocabulary and word recognition.” • In two instances the researcher did not observe whether the coach checked to see whether learners with barriers are catered for or not. In the other observations it was noted that the coach did check this, and that learners were accommodated for but it was not illustrated how. In one instance it was noted that the learners with barriers are not catered for, and the coach indicated that he would give it more attention. • In all 12 observations the coach checked whether learners are continually assessed and given developmental feedback, whether the planned assessments assess learner development and skills and whether or not learners are assisted to reflect on their learnings. 4. School based post-observation discussion • In three observations, the researcher did not observe the coach asking the teacher to share what had gone well or how they felt the lesson had gone, in four observations the researcher noted that “The coach proceeded to offer his observation without affording the teacher any opportunity to identify what went well.” In one observation the researcher noted that the coach encouraged the teacher to share and in another three observations the teacher “Indicated that according to her everything was going on track.” • The coach did not always offer suggestions, or encourage discussion and reflection on the part of the teacher regarding improvements. His approach leans a lot more towards instructing the teachers where they should improve, if at all – in only six out of the 12 observations did the coach provide practical suggestions. One of these suggestions was that the teacher should “Attend the training whenever scheduled and that she should also seek assistance from him or other teachers who seem to be hands on with RSP.” In all instances he did not guide the teacher to come to solutions in their areas for growth. • In only two instances did the coach prompt the teacher to identify solutions for the problems; however the researcher noted that this was difficult for them. In one 210 instance the researcher wrote that “The coach did not guide the teacher, instead he suggested that she must learn from a grade two teacher who is executing her duties excellently.” The coach seems to provide suggestions by comparing teachers to other schools or teachers regularly, as twice his suggestions made reference to how other schools were implementing the programme. • The researcher wrote that “The coach was not in any way biased,” for three of the observations, and that “The coach was not judgmental at all” for the nine remaining observations. • In all 12 observations no goals were set for the next session, but dates were agreed upon for the following visit. 5. After school workshops • The coach planned four after school needs-based workshops, but conducted three. • The needs identified for the first workshop were reflection on activities in the lesson plans, and challenges around DAR/DEAR implementation. The needs identified for the second workshop were also around reflection and group gradual reading, at this workshop – “There was a concern from the teachers that there is a need for the coaches and specialist to have a meeting and discuss the issues that affect the implementation of RSP and other subjects.” • On the last workshop it was noted that; “The workshop went well as compared with ones he held with other schools. The content of the workshop included the mistakes in the content of the lesson plan, e.g. He noted some error in certain activities where South Sotho words were used and urged the teachers to observe and work around that, instead of repeating the same mistakes and feeding the learners with wrong information. It was a phase intervention training.” • The coach indicates that he has never offered catch up training for teachers who might have missed LTSM training, he indicates that his schedule is “Tight.” CONTEXT FACTORS THAT AFFECT COACHING PRACTICES This coach and the researcher noted the following challenges in providing coaching services to teachers in the schools: • Use of vehicles and a lack of support from the asset manager. This impedes the coach from being proactive and offer solutions to colleagues and teachers, i.e. material distribution. • Teacher absenteeism, union meetings and school events interfering with monthly plans at the schools. • Specific challenges in one school, which were - lack of electricity - lack of water “The two mentioned challenges, which were observed, are so deep that learners are not attending classes as they are supposed to be. For instance, lack of water, affects sanitation, the toilets are blocked and there was no drinking water. Although the HoD is busy with the relevant departments to sort out these issues (challenges) its long term effects will have a negative impact on the learners.” 211 This coach noted the following successes in providing coaching services to teachers in the schools: • The researcher noted that the coach created scenarios whereby the learners were provided an opportunity to read in the class, and the coach would observe their reading ability. The coach then provided coaching to the teacher as to what elements should be addressed i.e. pronunciation, handling the book. • The researcher noted that the coach utilized “Skinner reinforcement methodology – the coach uses incentives (sweets) for learners to entice them to read.” COACHING SUPPORT This coach feels adequately supported by the Head Coach, he says the support he gets from the head coach is developmental and he “Learns the ropes of how to navigate through.” He believes the SMTs are supportive as they allow him to work with the teachers in their schools. This coach attended both the SOP training and the RSP Term 4 Training. While he feels that his developmental needs are met, he still needs to be trained on how to use the tablet and does not believe that the lesson plan is user friendly. EVALUATION OBSERVATIONS This coach is incredibly meticulous and thorough in his approach to coaching. He was strong in administration, organized and well planned during the planning and preparation stage, and checked nearly every task and responsibility of the teacher during pre-observation discussion and classroom observation. An area for development for this coach is in the post-observation discussion, it was here that the coach began to be less thorough in the steps and indicators. The discussions were much less reflection or discussion based, and seemed to be more about him leading the conversation. By his own reporting, the teachers were incredibly compliant and on track in their teaching, both in terms of methodology and curriculum coverage; however the only positive comments that the researcher noted that he made were about the classroom walls being clean, with the LTSM well organized. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Field worker debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 212 COACH CASE STUDY 8 PROFILE Qualifications Bachelor of Arts Education (1994) Experience of teaching Setswana Teacher, (2018-2018) Setswana Teacher High School (2016-2018) Teacher, High School (1996-1999) Teacher, High School (1994-1996) Experience and training in coaching RSP training provided by OUP and Molteno on the lesson plans and components of language. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:12 Ratio of coach to RSP teachers 1:61 Self-reported frequency of RSP coaching a teacher Each teacher is visited two or three times per term Data Source Coach 8 was shadowed in the week of September 9, 2019, during which she was interviewed and observed at three schools working with nine individual teachers. Duration of practicing as an RSP coach 9 months 213 Coach’s understanding of their role as a coach: “Guiding and supporting educators to implement strategies and processes of RSP. The coach measures the performance of the teachers with the way they do their work in accordance with the lesson plan.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials She indicated that she builds a trusting relationship with the teachers through continuous communication, by not being judgmental when teachers make mistakes, and by demonstrating activities when necessary. She indicated that she does not share observations of teachers practice with the school principal or the HOD as this is confidential information. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • She has rated the schools and teachers for differentiated support as their needs vary, for example some teachers require the lessons to be demonstrated to them, while other require needs-based workshops. The criteria to identify the different support is based on the teacher’s ability to follow the lesson plan. New teachers to the school who have not attended the teacher training require additional support. Catch up training is provided if a teacher did not attend the teacher training. • She develops a monthly plan for coaching activities and submits the schedule by email to the head coach, however no feedback is received. She informs the school, Heads of Department, and teachers through WhatsApp about visits (9). Before the visit, she telephones the teachers to confirm the visits or any changes. 2. Pre-observation discussion • The teacher and coach discussed the lesson plan prior to the start of the lesson, and compared the lesson to the weekly routine (9). The teacher shared the lesson plan with the coach (9), discussed it (9) and when necessary a catch-up plan was discussed if the lesson plans did not match the required weekly plan (3). Teachers are able to explain what they will be doing and where they are in the weekly routine. • She did not check the learner books in three observations, but did in the other six observations. • The coach did observe if the LTSM is available, relevant, and well organised, which in (8) observation it met these standards, and in one observation it was noted that the LTSM was well organised according the various components of the lesson. • She further checked that the teacher understood the content of the lesson, which the teacher did in nine instances. • The coach checked if the Drop all and Read (DAR) and Drop Everything and Read (DEAR) was on the timetable and well established (9). It was noted in one 214 observation that the teacher and learners started reaching without delay, and in another classroom the reading activity was ready on the mat to proceed. • The coach noted that the learners were working well in organised groups in nine observations. An example was when the coach noted that the group that was not reading was busy with a written assessment and quiet. 3. Classroom Observation • The coach checks the curriculum coverage during her visits to the school (9) and where necessary indicated if there was a need for the teacher to develop a catch up plan (3) and in one school the teacher was also doing catch-up from the previous week. • She checks that the LTSM is properly used and records this in her coach notes. (9) • In nine observations the coach checked that the teacher’s introduction to the lesson acknowledged the learner’s experiences and prior knowledge, for example through giving the meanings of words (2), however in two instances the learners’ knowledge was not acknowledged. • The coach noted that in nine classrooms the teacher gave a creative and relevant introduction, for example using (5) songs (but the song was not relevant in one situation) or clapping hands to match the number of syllables for each word (1). • She observed and made notes on whether the outcomes of the lesson are clear, which she indicated was the case (9). • The coach looked at the lesson plan during the lesson to follow the routines and pacing, which she indicated were well adhered to in nine situations. • The methodologies for reading are checked to make sure the new content is clear to learners (9), however the coach noted that one teacher mixed group guided with shared reading. • She did not check that reading strategies and processes were used during the lesson in two lessons, however this was then completed in seven observations. The coach noted for one teacher the need to complete the steps properly against the time. • She checked that reading skills were taught through focused activities (6) and that they are integrated within components of language. In two instances it was noted that learners were able to correct themselves when reading. In three instances the coach did not check this, as the learners were doing written assessments during her visit. • She further noted if the teacher differentiated for learners with learning barriers (6) and indicated that the teacher must help learners to build confidence (1). In one class the teacher encouraged the quieter learners to participate, and in the same class a learner with sight challenges was placed in front of the teacher to help him to see and participate. Another teacher was noted to be doing her best to help learners with barriers by involving them and assisting them. In three classes the coach was not observed doing this. • Opportunities for application of new knowledge were created for learners (9), and examples were noted, for example learners were told to tell what each word meant (2) or to form sentences using the new words (3). • Learners were assisted to reflect on new knowledge, skills, attitudes and values through the use of questions (7), for example to speak openly about what they knew on the lesson focus. However, in two instances, this was not checked. In one 215 observation, the coach checked that learners were able to arrange pictures according to logical sequence of events. • Learners were noted to be assessed continuously by the teachers throughout the lesson (9), however developmental feedback was not always provided by the teacher and the coach noted this twice, and in another instance the coach noted the teacher was assessing the learners but not praising them when they answered correctly (1). 4. School based post-observation discussion • Before giving her own observations, the researcher indicated that she asks the teacher to identify things that they felt went well during the lesson (9) and then asks the teacher to identify things they would do differently in the future (9). The researcher noted that the coach does not dictate to the teachers, but instead asks the teacher to commit to what they will do differently (9). The coach recorded when the teacher was happy with the way learners answered questions and understood the sounds (2), and in one class the teacher noted that learners understood the beginning, middle and end sounds. • The coach offers her observations of things that went well (9) and commented on the relevancy of the song as an introduction (2), or the use of the tracker during reading (1), or acknowledging what the teacher did well, for example focusing on the question for the day. • She asks questions to elicit clarification and prompt reflection regarding areas of improvement to the teacher (8), and she provides feedback and offers suggestions (9), for example the teacher must cut sight words out and display them individually in the classroom (2), or that the teacher must read the word three times and let learners repeat it after her (1). In one instance the coach asked the teacher about the mix up of group guided reading and shared reading. In one situation the coach did not provide a rationale for her suggestions according to the researcher. • The teacher is guided by the coach to identify solutions for problems areas in the form of action steps (9), which were successfully identified (9). • The coach takes the teacher through the lesson observation document (9). Classroom management (9), curriculum coverage (9) and the extent to which the teacher follows the weekly routine (9) were discussed. In one observation, the coach indicated to the teacher that lessons must not be skipped, and the routine must be followed, and in another the teacher was complimented on her ability to work with one group while other groups were busy with other activities (1). A suggestion was made by the coach to one teacher to choose a leader for each group to assist in classroom management. • The coach noted the extent to which the lesson complied with the core methodology (9) to make the lesson clear to the learner. • The researcher observed the coach helping teachers to identify solutions to potential barriers to implementation, and offered her suggestions that were appropriate and reasonable (9). For example, she suggested to one teacher to start with sight words for learners and to read in a low voice so that other learners cannot hear them, and with another teacher to motivate learners, focus on the question of the day, and integrate various components into the lesson. One teacher was told not to code-switch but to use “Proper Setswana” – which the researcher reported was given in a constructive way. An area for further development for the teacher is to strengthen developmental assessments. 216 • She is able to give examples and demonstrations (9), for example she demonstrated developmental feedback (1). The researcher noted that she provides feedback using a friendly and positive approach, avoiding judgement or bias. The coach does use questions to prompt reflection (9). • The coach and teacher set goals for the next visit (9), for example for a teacher to emphasize developmental feedback, or to appoint a group leader. 5. After school workshops • A needs-based workshop was facilitated in two of the three schools observed, and the coach had planned five for the week. The coach used the workshop as an opportunity to provide feedback to the teachers in the schools. The workshops were held according to grades. • Workshop content ranged from group guided reading, shared reading, phonics, how to provide developmental feedback to learners, giving learners a chance to reflect on what they have learnt, and highlighted areas identified by the coach during lesson observations. • The coach notes that she spends “Two days in a school – one for HL and one for EFAL” offering catch-up training for teachers who missed the initial LTSM training, CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The following challenges in providing coaching services to teachers in the schools were noted: • Teachers use local transport to travel and are unable to remain at school in the afternoons for the needs based workshops. She indicated that this is beyond the control of the coaches. • Coaching visits are cancelled due to union events, memorial services, strikes and department meetings, therefore coaching support is not provided. • When teachers leave the school, it means that the coach must “Start again” and train the new teachers (she reported that two teachers had left and she had to re￾train their replacements). • Some teachers show resistance to the RSP, thereby making it difficult for coaches to support them. This coach noted the following successes in providing coaching services to teachers in the schools: • She uses videos when visiting teachers for discussion after the lesson to guide and improve their practice. • The project provides a vehicle and fuel. • The coaches must sign log sheets which the school stamps to confirm the coach has attended the school. 217 COACHING SUPPORT The Head Coach has only visited her once for an observation, however she reported that no support was provided. The Head Coach has only provided resources. She does not feel that the Head Coach provides adequate support, however the School Management Team are reported to be very supportive and eager to help her. EVALUATION OBSERVATIONS She was described by the researcher as “Friendly, welcoming, respectful, humble and very eager to help. She is not judgmental. She does not impose to the teacher [and] has a good relationship with the teacher”. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 218 COACH CASE STUDY 9 PROFILE Qualifications Advanced Diploma in Education Management (2009) B Tech Education Management (2003) Diploma in Education (1993) Experience of teaching Head of Department, Primary School (2018–present) Teacher and Acting Head of Department, Primary School (2015–2017) Teacher, Primary School (2013–2015) Teacher, Primary School (1993–2012) Experience and training in coaching Training: Outcomes based assessment RSP training provided by OUP and Molteno on the lessons plans and components of language. Coach has not been trained in the use of classroom libraries, “I haven’t heard of such trainings.” Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:14 schools Ratio of coach to RSP teachers 1:57 teachers Self-reported frequency of RSP coaching a teacher “Once or twice a month.” She does visit some teachers more often than others, if a need exists. Data Source The coach was shadowed in the week of September 2, 2019 during which she was interviewed and observed at three schools working with 219 six individual teachers. The researcher was joined by the Khulisa Coaching Expert on September 4, 2019 for quality assurance purposes. Duration of practicing as an RSP coach 4 months (New coach) Coach’s understanding of their role as a coach: “It's all about giving teacher and learner support: Spearheading literacy interventions. In my case, the RSP program. Co-teaching and working with individual teachers and phases.” The coach further explains that she conducts classroom support visits and school based workshops at the end of each school day, draws monthly and termly schedules and facilitates teacher orientation on RSP projects for Foundation Phase teachers. She also completes four tools, visit logs, registers and the lesson development document. She checks “The quality of learning by giving feedback to every educator always.” She checks coverage of work. Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach seems to have a good relationship with the schools. She notes that she aims to build a trusting relationship with teachers by adhering to her visit schedule and co-teaching with teachers, while listening to their challenges and attending to their needs. The researcher adds that she has an established professional relationship which makes her interact amicably with especially the teachers. He notes that she works collaboratively, observes and listens, takes teacher suggestions and is approachable. Her openness helps with creating very good relationships with teachers and SMTs, and that they respect, trust and listen to her. However, she feels that creating a credible relationship with the concerned schools and teachers in particular is the most challenging aspect of her role as coach. The coach mentioned that her Head Coach sometimes mediates between her and the SMTs, which could indicate some concerns. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation 220 • The coach created and uses the group WhatsApp to disseminate the monthly schedule for planned visits to teachers and other relevant information to schools (6). She tries to build a trusting relationship with teachers by planning ahead. • She sends out the schedule of planned visits, and reconfirms planned visits beforehand (through WhatsApp, and by calling some via phone). However, for September visits, this was done only two days before the beginning of the planned visits (6). • The schedule is sent to teachers, school administration and the Head Coach during the week before the visits are scheduled to start. She does not seem clear on whether the Head Coach should be signing off on the schedule beforehand, and she does not feel it is part of her role to circulate the schedule to the Subject Advisors. • When asked whether she creates or reviews teacher profiles before going to each school, she responded that she does not, as it is not part of her role. 2. Pre-observation discussion • She does not regularly review learner books as part of the pre-observation, as this was observed twice out of six lessons. However, the researcher noted that she looked at and understood the overall lesson plan, and evidence of her and the teacher’s preparedness to RSP implementation was checked. • The teacher and coach successfully discuss the planned lesson, their own preparedness to RSP resources and/or learner readiness in four instances (out of six). Significant planning was observed by the researcher. • Usually, the teacher briefly shows the coach which lesson is going to be presented from the lesson plan before the lesson starts. This step is not always completed properly, before the start of the lesson (twice out of six lessons). In one of these instances, the coach was given the lesson plan about ten minutes after the start of the lesson. 3. Classroom Observation • At the start of the observation, she consistently observes if LTSM is available, relevant and well-organized (6), for example, sorted according to term and/or in packs on a table. • She always checks if the teacher understands the content of the lesson–in all six observations, she notes that the lesson is well-presented to learners and/or content clearly understood by the teacher. It was noted that content was planned at learner’s level (once). • She also checks if the routine for DAR/DEAR is well established (all six times), and visible on the timetable (three times) and/or the teacher’s model book (once). It is observed that DAR/DEAR is scheduled either daily or three times a week, but at one observation, DAR/DAR is not well established. • She checks that LTSM is properly used/fully utilized (observed six times) and indicates that she has conducted catch up training to teachers on LTSM, school by school. • During all observed lessons, she checks if learners are working well in organized groups (usually of five to six learners per group), that the introduction is creative to motivate learners, and within the context of the planned learning content. The latter is observed through a song, poem or roleplay (five times). It was noted that once, learners were divided according to ability, which can cause labelling. 221 • The coach does not always successfully/fully check that the introduction acknowledges learners’ experiences and prior knowledge, as this was observed twice during the six lessons, at “Moderate achievement.” But she observes and makes notes on whether the outcomes of the lesson are clear, usually based on learners’ responses (6), such as learners being able to make predictions, etc. • With regards to teaching and learning, she always checks if the routines are adhered to by following the lesson plan, if core methodologies for reading focus are used to make new content clear to the learners plus if reading strategies and processes are used during lessons (6). The researcher noted that core methodologies were used in implementing RSP resources twice, while well￾presented guided reading, pair reading and/or peer learning were observed, and noted a “Substantial achievement” twice. The researcher notices that she performed fantastically when joining during group guided reading. • With regards to checking if reading skills are taught through focused activities, this was observed twice out of six observed lessons, and viewed as a developmental area. Once, this was not observed at all, and during three lessons it was partially observed or unsuccessful. The researcher noted that sight word reading was limited, fluency reading was not modelled by the teacher and once, and lesson plan steps were not followed. • Checking if focused activities are integrated within components of language is performed–but once, during the six lessons observed, only partially.The researcher notes that word games/puzzles were used effectively, including for the building of words, and that the language component received maximum attention. • The coach’s checks on whether learners learning with barriers are catered for via differentiated activities was successfully completed half of the time (i.e., three times, via extra reading class). The researcher felt that barriers being addressed can be better specified better (three times), and that the coach could improvise word building games in alignment with RSP materials. • She does check if opportunities for application of new knowledge are created for learners, as this was observed five out of six times. When observed, role playing, and modelling were mentioned and it was noted that she executed this step outstandingly. • Assessment checks were observed during six observed lessons, where she observed that learners are continuously assessed and given developmental feedback: this occurred three times but sometimes only glanced at. With checking if learners are assisted to reflect on new knowledge, skills, attitudes and values (observed three out of six times, mostly via roleplay), she showed skill at assisting learners to give a recount about their own experiences. • She does check whether the planned assessment activities assess learner development and skills, as it was observed during all six lessons (including assessing spelling and writing and giving feedback via corrections). 4. School based post-observation discussion • Although she provides immediate feedback, the coach sometimes hesitates to ask the teacher to identify the things they felt went well, before providing her own observations. This step was observed as being completely executed only twice (during six lessons), although when she does ask, she does performs it excellently. Examples highlighted includes having discussions to address a teacher’s concerns or reading a recounting story. 222 • She shares observations of things that went well (observed during all six lessons), sometimes using Bloom’s taxonomy questioning. For example, she indicated that classroom management went well or that learners were successfully involved in a debate. • The coach is prone to not asking the teacher to identify things they would do differently in the future, in this case, during two of the six lessons. Twice, she did so partially‒for example, when pacing routines according to time, was highlighted. • Most of the time, she asks questions to elicit clarification and prompt reflection regarding areas of improvement – this was observed five out of six times. For example, she prompted reflections through recount reading and reflection and to specify reading strategies. She also offers suggestions, such as teaching from the known to the unknown, and working according to the stipulated time. • She guides the teacher to identify solutions for problem areas in the form of action steps, resulting in the teacher successfully identifying action steps, but only half the time (observed three times). For example, used excellent experiences to explain vocabulary of the day and/or contextual discussions aimed at improvement occurred twice. • Although she takes the teacher through the lesson observation document (occurring five times), the researcher observed once that she could have given teacher chance to read lesson observation notes. • The coach covers the following topics under the lesson observation document, well: classroom management (6), curriculum coverage (5), the extent to which the teacher follows the weekly routine (6), as well as the extent to which the lesson complied with the core methodology (5). In fact, with regards to the last topic, substantial achievement was noted, including that comments were provided against each core method, while a recount text was read, and literal question used to active learner cognition – although once it was not observed that she covered the last topic. Generally, she seems to adhere well to the lesson observation document requirements. • With regards to content, the coach helps recipients identify solutions to potential barriers to implementation, almost all of the time (observed five times), such as lesson preparations, reading sheets and enrichment activities. Although, the researcher noted that the teacher should specify which intervention are being addressed, and that during one lesson, it was not observed. • She is strong in offering suggestions that are appropriate and reasonable in scope, for example using different forms of genre in teaching comprehensive strategies (e.g., a recount text, an instruction text, a riddle text, etc.), as it was observed six times. • She also provides examples of correct implementation with suggestions, using excellent illustrations such as Mind-Maps or using body movement to explain a point, almost all of the time (observed five times). • On providing a rationale for each suggestion, plus successfully identifying action steps, her performance seems mixed. During the six lessons, she partially and completely executed both of these steps three times each. It was noted that she once performed providing rationale for each suggestion, outstandingly, and sometimes taking suggestions and mapping the way forward. • The coach has fair communication skills, with some developmental areas. She does offer suggestions for improvement in a constructive way, being noticed most of the time (5), although she can do this more with more specifics/detail. 223 • She always avoids judgement or bias when providing observations and suggestions (6), being fact-based and focused on requirements of the lesson observation tool, although her feedback can sometimes lean towards being too general. • Most of the time, she uses questions to prompt reflection rather than telling the teacher their problem areas, as this was observed during five lessons. It was noted that she successfully executed a rational two-way discussion with questions and consensus more than once, but that she does not always have it written down in her notes. • She almost always sets goals for the next visit with the teacher (five), for example that vocabulary work should be taught from context with concrete examples, reading strategies, etc. However, once it was noted by the researcher that this was not observed at all, nor written down in the observation tool. 5. After school workshops • The coach has facilitated four needs-based workshops (cluster training), of five planned, during the coach shadowing period. The researcher noted that the content of the workshop was aligning the use of LTSM to the lesson plan, adhering to certain routines such as group guided reading and Drop All and Read. • The coach also notes that she provided catch-up LTSM training for teachers who might have missed the initial training. She indicated these trainings took an hour and a half, and that she went school by school. • She identifies teachers’ needs by observing lessons and checking what teaching and learning resources or materials are being used in the class. She also mentioned that she uses school needs-based workshops as one of her key strategies to deal with challenges in her role as coach. Finally, she suggested that lesson demonstrations during needs-based workshops should be increased. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach and the researcher noted the following challenges in providing coaching services to teachers in the schools: • She feels that creating a credible relationship with the concerned schools and teachers in particular is the most challenging aspect of her role as coach – but she attempts to use negotiation or offering school needs-based workshops as strategies to deal with these type of challenges. • The coach noted that she is sometimes not able to visit all the teachers she plans to visit per month, due to teachers not being available (attending meetings, workshops, and so forth). This affects her visiting schedule. She mentions that some teachers need to be always reminded. Also, she sometimes fails to visit schools due to circumstances beyond her planning (no explanation provided). The researcher notes that she was quick to adapt and change schools when one school wasn’t available or busy. • She suggests that the number of coaches should be increased so that there is sufficient time for visits to teachers in need. • She mentions that she feels that her developmental needs are met only to a limited extend (no explanation provided), and that she needs more training in the field of 224 coaching. The researcher substantiated this, by noting that she needs more orientation, workshops on methodologies of teaching and reading in the classroom and more assistance in applying RSP resources and methodologies. • She uses the car provision for school visits, which can affects her ability to get to schools if the schools are far. The researcher noted that she is only able to fetch the vehicle from the parking lot at 7 a.m., which means she can only arrive at a school around 8 a.m. due to distances. The researcher noted the following challenges: • The researcher notes that there was one teacher that had problems controlling her class, for whom the coach should have provided more motivation. • A concern around classroom corners and RSP library resources was noted by the researcher. • Although it seems that the WhatsApp group platform improves the distribution of schedules and communication between the coach and teachers, it was noted that the coach circulated the approved school visit schedule for the month of September to teachers/schools only two working days before the beginning of the month–which is very short notice and could cause problems. This coach noted the following successes in providing coaching services to teachers in the schools: • The coach pays attention to the adequate use of LTSM and RSP resources – in fact, she motivates teachers and learners to fully utilize the available RSP resources by aligning them to their daily lesson plans. • She facilitated all the planned needs-based workshops during the coach shadowing period (four). She is one of two coaches who were able to execute all the planned needs-based workshops during that period. It seems that she is effectively using these workshops as tool to address her relationship with teachers and overcome challenges in the role. • Based on the researcher’s observations, she provides feedback and performs good discussions with teachers after lessons, which are collaboratively in nature, and invites teacher’s opinions and suggestions. She also listens well, has good facilitation skills and encourages open dialogue during workshops. COACHING SUPPORT The coach explains that she does receive some support from the Head Coach. The Head Coach sometimes accompanies her to schools, chairs meetings and mediates between her and the SMTs. However, she only feels adequately supported to a limited extent, because the Head Coach is often not available, and often only through phone calls and WhatsApp. She usually does not receive feedback or approval from the Head Coach on the school visit schedules. However, she does feel that there is support readily available to deal with challenges beyond her power. EVALUATION OBSERVATIONS Although this coach has been in the RSP coaching role for only four months, she has a lot of classroom experience and has demonstrated a variety of skills and strengths during the 225 observation period – she works collaboratively, is observant, and listens well. She is also thorough, planning in detail and following the lesson plan “Step by step”, based on the researcher’s observations, she provides feedback and performs good discussions with teachers after lessons, which are collaboratively in nature, and invites teacher’s opinions and suggestions. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 226 COACH CASE STUDY 10 PROFILE Qualifications ACE Setswana teaching foundation phase (2013) ACE in Curriculum and Professional (2009) Teachers Higher Diploma (2001) Teachers Diploma (1989-1991) Experience of teaching Grade 3 Teacher and Foundation Phase HOD, Primary (2004-2015) Foundation Phase Teacher, Primary (1999–2004) Intermediate Phase Teacher, Primary (1992-1999) Experience and training in coaching EFAL and HL Language Coach, Foundation Phase, NECT North West Province (2015-2018) RSP training provided by OUP and Molteno on the lessons plans and components of language. Coach has not been trained in the use of classroom libraries. Setswana language proficiency Intermediate in reading, writing and speaking Ratio of coach to RSP schools 1:10 Ratio of coach to RSP teachers 1:55 Self-reported frequency of RSP coaching a teacher Three times per term “Sometimes it depends on the number of teacher in a particular school.” Data Source Coach 10 was shadowed in the week of September 9, 2019, during which she was interviewed and observed at three schools working with 11 individual teachers. 227 Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “My role as a coach is to assist and support teachers…I often demonstrate, present and show the teachers videos on how to use the components, so that they can improve on their teaching skills.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach is reserved and “Not a talker”. The researcher said that she and the coach were able to get along and build good rapport despite the coach not coming across as “Friendly”. She is also described as passionate about education and teaching. She said that she works to build positive relationships with the teachers by keeping their information as confidential as far as possible, and avoids criticizing or judging them. She does not share her observations with the principal. She is described as “Hands-on”. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the Researcher’s observations. 1. Planning and preparation • The coach completed her monthly schedule and submitted it timeously to the Head Coach. She did not receive feedback, nor signoff on the schedule from the Head Coach. She also indicated that the Head Coach does not sign off on her Lesson Observation Tool. • There is a need for role clarification, the coach is under the impression that the Head Coach should be the one to circulate the monthly plan to Subject Advisors (not done). She is also not sure how schools or teachers get rated for additional support, and is not sure whether the Head Coach should be doing this or not, or what the system to rate the schools is. • The coach said “I visit the teachers equally. Emphasis is put on teachers that needs more assistance and support.” • The coach does not call teachers in advance because they do not get airtime to make these calls, they only receive data to utilize WhatsApp. She struggles with this as some teachers are not on WhatsApp – but she did communicate with all of them that are on WhatsApp. • The researcher observed the coach utilizing/creating teacher profiles. 228 2. Pre-observation discussion • In all 11 observations the coach looked at learner books to understand the progress in the lesson plan, she discussed the plan with the teachers and the teachers shared their lesson plan for the day. • For three observations the teacher was in Week 7, in three observations the teacher was in Week 8, one teacher was in Week 9, two teachers were in Week 10, and two were not stated • The researcher noted that one teacher does not have a ‘Lesson Presentation’. The teacher indicated that she did not know how to compile one since she had not attended any RSP training or workshops. 3. Classroom Observation • In seven observations the coach checked whether LTSM was available, relevant and well-organized, in one observation the coach did not check LTSM at all. In two observations the coach checked whether LTSM was available and relevant, but did not check for its organization, and in one observation LTSM was checked to be available, but its organization and relevance were not checked. In three of these observations it was the noted that LTSM was not well organized or that the display needed work. • In six observations it was clear that the coach checked to make sure that the teacher understood the content of the work, and then in three observations she provided suggestions; “She advised the teacher to use Vula Bula readers for shared reading and enlarge the text so that the learners can see and read after the teacher”. In four observations it was not observed that the coach checked to ensure that the teacher understood the lesson content. • In one observation the coach did not check whether DAR/DEAR was part of the school timetable, but she did check in the ten other observations. Of these, DAR/DEAR was only in the timetable in one instance, where it was not – the coach encouraged the teacher to find a place for DAR/DEAR activities in the timetable. • The researcher notes that in five observations the coach checks whether the introduction of the lesson acknowledges learners’ prior knowledge and experience, and in six observations the coach does not check this. However, in nine of the observations the comment is that the teacher starts the lesson with a song, and only twice is the comment one that implies prior knowledge is acknowledged. • In all the observations the coach checked to determine whether routines were properly followed or not. In seven observations the routines were not properly followed, three of these were due to poor time management. Four teachers adhered to the routines. • In all 11 observations the coach did check whether or not core methodologies, and reading strategies were employed. However, the use of these techniques is not always adequate, there are seven comments about the quality of the teaching in regard to these techniques – ranging from the techniques not being employed at all, not adequately used or issues with how they are used such as; “Learners were not asked questions during the reading.”, “The picture walk was not done with the learners before reading. They only discussed the title page at the end of the story reading” and “The learners were not sitting down when reading. Learners must be sat (sp.) [seated] when reading so that they can be relaxed.” • The coach only noted twice whether reading skills were taught or not, but did not check this nine times. The integration of focused activities and language 229 components was checked by the coach nine times, however the only component ever mentioned was that of ‘sight words’, and whether those were utilized, or explained or not. • In three observations the coach did not check whether or not learners with barriers are supported or not, in eight observations the coach did check. In only one observation were these learners catered for – but only by means of guidance by the teacher. In the other seven observations these learners were not catered for. In one observation the teacher then gave a suggestion for improvement. The researcher noted of another observation that “The coach checked and indicated that the learners with learning barriers were not guided, assisted and catered for. Most of them were playing and not paying attention.” • In all 11 observations the coach checked whether learners were continuously assessed and given feedback or not, in five of these instances the coach observed that there were some missing exercises and activities in learners’ books. In one she indicated that “Most of the activities were done, marked and corrections made.” • In one observation the coach did not check the planned assessment activities at all, and in 10 the coach did not check whether the assessment activities assessed learner development and skills, but checked whether they were in line with the Lesson Plan. For two of these, the coach notes that some activities were missing. • The coach did not check in any of the 11 observations whether or not learners are assisted to reflect on new knowledge, skills, attitudes and values. • The Coach indicated that she does not complete a Lesson Observation Plan and that “She only has observation sheet she completes per schools”. 4. School based post-observation discussion • In all 11 observations the coach asked the teacher to identify things they went well. In all 11 observations the coach also makes an observation about something she liked or thought went well. Although, these observations had less to do with the teachers’ teaching. One positive comment was praising the teacher for their use of flashcards, one was a comment on the learners being well behaved, one comment was that the classroom was “Foundation Phase friendly”, and the other eight comments were about the posters displayed on the walls. • The researcher noted that the teacher never asked questions to elicit clarification and prompt reflection regarding areas of improvement by the teacher. Three times the researcher noted that this did not happen “Because she adopted a directive model of coaching.” In five instances the teachers were asked what they thought they could have done differently, but in six instances they were not asked this. • In all 11 observations the coach offered multiple suggestions to the teacher that were practical and useful ranging from suggestions on classroom management to methodology. Each teacher was given a list of suggestions based on the observation. • In the post-observation discussion, the coach seemed to take more of an instructive role. The comments from all 11 observations imply little reflection such as, “She [the coach] talked while the teacher was listening and not given a change to offer her own suggestions” (made four times), “The coach did not ask questions. She talked while the teacher listened and not given a chance to offer suggestion” (noted three times) or “The coach did not use question to prompt reflection. She talked while the teacher was listening and not given a chance to offer her own suggestion.” • The researcher noted that in 11 observations feedback was not bias or judgmental and was based on the learner’s and teacher’s activities. 230 • In none of the 11 observations were goals set between the teacher and the coach, however for all of them they agreed they would next meet in October following the school holiday. 5. After school Workshops • The coach conducted two needs-based workshops out of the four she had planned on her monthly plan. The needs-based workshop was not conducted at the third school because the Grade 1 and Grade 2 teachers were not available (Grade 1 had interviews and Grade 2 had emergency at home). • The coach also reported that she has conducted catch-up training for those teachers who had missed the training on LTSM, and that this training normally takes two to three hours. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach and the researcher noted the following challenges in providing coaching services to teachers in the schools: • The coach noted that she struggled to deliver the LTSM and carry a lot of boxes, “All these is not part of my duties.” • The coach is not given airtime to call teachers, only data for WhatsApp. • The researcher noted that the schools are in very remote areas, at least two and a half hours away. • The researcher noted that the coach does not have time for admin, and that coaches would like a day to do this admin. • The researcher noted that the coaches are not monitored or supported by the Head Coach, and that feedback on the performance of the coach should be given. This coach noted the following successes in providing coaching services to teachers in the schools: • The coaches are provided with cars which makes getting to the schools easier. COACHING SUPPORT The coach indicates that she does not get any support from the Head Coach. She believes Head Coaches should avail themselves to the coaches four times a month. She says the School Management Team supports her as they allow her to go and support the teachers. She says the only time she received support is when they are in the office. She indicated that she had not attended developmental training, “Only a curriculum training and workshop”. Regarding the curriculum she feels her developmental needs have been met, although not in the areas of report writing and planning. She has not received any training on classroom libraries. EVALUATION OBSERVATIONS 231 This coach is a well-balanced and well-rounded coach, regarding most areas of coaching. She is experienced at coaching, and this comes through with the detailed suggestions she gives to teachers. She is described as passionate about education and teaching. An area for growth could be in the way in which she gives feedback, while she is sure to give positive comments as well as suggestions, she is quite instructive and the teachers might benefit from more of a discussion or being asked more questions which lead them to reflect on ways forward. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 232 COACH CASE STUDY 11 PROFILE Qualifications Advanced Certificate in Education (2011) Higher Diploma in Adult Basic Education (ABET) (2010) Primary Teachers Certificate (1973) Experience of teaching Acting HoD, Primary School (2015-2016) Teacher, Grade 1, Combined School (2014) Teacher, Grade 5 and Grade 7 English, Primary School (2014) Teacher, Grade 1 to Grade 6, Primary School (1977-1998) Experience and training in coaching Provide training to supervisors, VHS Production (2017–2018) Coached and mentored Foundation Phase Teachers, Gauteng Primary Literacy Math’s Strategy (GPLMS) (2011-2014) Setswana, English FAL and Mathematics Coach, Class Act Educational Services (2010-2013) Mentor School Managers and Manage Mentoring Programs in Schools, ETDP-SETA (2010) Supervisor, Khara Gude (Adult Upliftment Education Program) (2008- 2010) Educator Empowerment Trainer, Heineman Publishers (2007–2008) RSP training provided by OUP and Molteno on the lessons plans and components of language. Trained in the use of classroom libraries, “The library trainer also does rounds in schools just like the coaches. She has the same material that we supplied to the schools. She is responsible for the library and we are not.” Setswana language proficiency Fluent in reading, writing and speaking 233 Ratio of coach to RSP schools 1:7 Ratio of coach to RSP teachers 1:57 Self-reported frequency of RSP coaching a teacher The coach indicates that this depends on the time frame. “I see a teacher in two month[s]. In a term I see one teacher twice.” Data Source Coach was shadowed in the week of 9 September, 2019 during which she was interviewed and observed at two schools working with seven individual teachers. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “My role as a coach is more of supporting a teacher, I am not directly involved with the learners. Mine is to support the teacher - I support her with whatever I trained her for. I hold one on one sessions with the teacher explaining to her probably where I observed she lacks. Remember she also has experience, she has been trained, because I am a coach I trained as a coach...I must see that the teacher does what is in the lesson plan and where she doesn’t understand I remain with her in the afternoon. Before I leave the school… I call a meeting and we discuss and support each other because even these teachers, they have different strengths.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach indicates that she does not “Deal” with subject advisors, and communicates with HODs instead of teachers leading up to her coaching sessions. The coach has a good relationship with their Head Coach, who she indicates “Communicates” and that if she has a problem she is “Free to call her.” She values confidentiality between her and the teachers, she emphasizes to them that what they discuss is only between them and that she does not even tell the SMT. She also tells the SMT that her work with the teachers is confidential because her work is personal and “Developmental.” The researcher notes that it “Appears teachers appreciate the relationship” that the coach has with them. 234 Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach does develop a monthly plan, which she submits to the Head Coach and to the HODs. She indicates that it is the responsibility of the HODs to then circulate this plan to the teachers. The coach does not get feedback from the Head Coach on this plan, however she does not expect feedback either – “To say what? Because I tell her what am going to do and am not expecting her to reply.” However, before she does the plan the Head Coach communicates with the coaches – “She sends a template with days that are taken (training, meetings) and I plan for my visits based on the available days.” • The coach does not call teachers in advance, she indicates that she provides her feedback to them one-on-one. • The coach does not create or review profiles for the teachers. 2. Pre-observation discussion • In all seven observations the coach checked the DBE workbooks of the learners for their progress, sometimes signing them as well. • In three observations the coach discusses the lesson plan with the teacher and allows the teacher to share their plan for the day • In one observation the researcher notes that the discussion and sharing was only partially done “Because the teacher brought it to the coach and opened where she was and started the lesson without any discussion.” • In four observations the discussion of the lesson plan, and sharing by the teacher did not happen. In three of these instances the researcher thinks this was because of the time that they arrived at the classroom - “Because the lesson started as soon as we entered the classroom” and “Might be due to us arriving while another lesson was on.” 3. Classroom Observation • In all seven observations the coach checked whether LTSM was available, relevant and well-organized. In all seven classrooms the comments by the researcher indicates that the LTSM was relevant and displayed in the classroom. In only one instance were some charts (but other posters were still hung up) not on display, when the coach asked about these, the teacher indicated that “They fall.” • In all seven observations the coach checks to make sure the teacher understands the lesson. • In all seven observations the coach checked that DAR/DEAR was incorporated, and in all seven classrooms DAR was in the timetable which was pasted on the classroom wall. • In all seven observations the coach checks that the introduction of the lesson acknowledges learner’s experiences and prior knowledge, that the introduction is creative and within the context of the plan and the coach checks whether the outcomes of the lesson are clear. The coach’s notes then confirm that in all seven classrooms, all teachers met these criteria. 235 • In all seven observations the coach follows the lesson plan and checks to see if routines are adhered to. In all seven observations, teachers were adhering to routines, however, in one classroom “Pacing” was noted as a challenge. • In all seven observations the coach checks if the core methodologies for reading, reading strategies and reading skills are utilized and taught, and whether activities are integrated within the components of language. In only one instance it was noted that core methodologies for group guided and shared reading are not used, otherwise these methodologies and strategies were utilized. In one observation it was noted that phonics were taught during the group guided reading, and in another observation learners were retelling stories from their Vula Bula books; “The teacher made the learners to retell stories read without looking at the books and asked them questions based on what they said and for her to get more information.” • In two observations the coach did not check whether learners with barriers are catered for or not. In five observations the coach did check whether they are catered for. In one of these, learners are not catered for and the teacher “Mentioned that she has 3 leaners that are being referred to an institution for special needs. They cannot focus and disrupts the class.” In the other four instances, it was indicated that learners with barriers are catered for, but in three of these it was not indicated how they are catered for. In the other, the teacher indicates that she “Attends [to] the learners with barriers daily from 1:10 p.m. till 1:30 p.m.” • In four observations the coach checks if learners are continually assessed and given developmental feedback, whether the planned assessment activities assess learner development and skills and whether or not learners are assisted to reflect on their learning. In these four observations it was also noted that these observations were all positive. • In two observations the coach did not check if learners are continually assessed and given developmental feedback, nor whether the planned assessment activities assess learner development and skills, and also did not check whether or not learners are assisted to reflect on their learning. • In one observation the coach checked whether learners are continually assessed and given feedback – in this case the learners were “Assessed through exercises.” But in this observation the coach did not check whether the assessment activities actually assess learner development and skills, nor whether learners are assisted to reflect on their learning. 4. School based post-observation discussion • In six observations the coach asks the teacher to identify the things they felt went well, before providing her own observations. These ranged from “Using the poster, to the assessment of the learners, to Group Guided Reading (two teachers), and the phonics (three teachers). • In all seven observations the coach then identified her observations of things that went well in the lesson. • In five out of seven observations the coach then asked the teacher to identify what they might do differently in the future. • In one observation the teacher did not ask the teacher on what they thought went well, or to identify what they would do differently in the future. • In all seven observations the coach offers suggestions for where the teacher could improve. However, in all seven observations the coach did not ask questions of the teacher to prompt reflection, and did not guide the teacher to identify solutions for problem areas (so the teacher then could not successfully identify action steps 236 either). In one instance the researcher notes that the “Directive model of coaching [was] used,” and in another that the “Coach [was] doing the talking.” • All suggestions given were noted to be appropriate and reasonable in scope, and a rationale was always provided to the teacher. • In all the observations the researcher notes that the coach takes the teacher through the lesson observation document to give feedback. • Reflecting on her own feedback process the coach states, “Because I am in the classroom, I am actually looking and experiencing what the teacher is doing and the learners too, how they react to the teacher. That is why when I give feedback, I discuss it with the teacher and because I have seen if she could have one it probably the learners would have understood. But if the teacher is too fast, I will show her that if you had gone a little bit slower because somewhere, I don’t think the learners understood the instructions the teacher is trying to give to them.” • In six out of seven observations the researcher comments that the feedback the coach gives is “Very brief.” The researcher then comments that; “It’s hard to grasp much from it,” she “Left some questions unanswered,” the coach “Is very brief when writing her observation notes as well as when she gives feedback and does not address all the questions making it hard to observe her,” and “As usual the coach was very brief on her feedback and did not cover all the questions on her observation document.” Despite this comment however, the lowest the researcher scored the coach was an eight out of ten (twice out of seven observations). The coach also scored nine three times and ten twice. • In all seven observations the teacher and the coach set goals for the next visit. These range from being practical and specific, such as “She [the teacher] will manage her class and make sure that those learners who are not involved in group guided reading are fully engaged in another activity” to broader goals that are perhaps less achievable in a month, such as, “During the next visit learners will be able to read with understanding,” “Learners will be perfect when it comes to sounding words, and coming up with new words without difficulties,” or “Learners speaking and reading fluently with understanding.” 5. After school workshops • The coach planned one after school needs-based workshop, and then conducted two - combining two schools, as the coach observed similar challenges across the teachers in the schools regarding guided and shared reading. • However, on catch-up training for teachers who might have missed the LTSM training, the coach said; “We give the teachers the catch-up plan, we don’t have to conduct the catch up plan….We explain the catch up plan to those who missed training during my school visits.” CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The researcher and the coach noted the following challenges in providing coaching services to teachers in the schools: • The schools are far apart, affecting the ability of the coach to provide support. The researcher notes that the coach is “Not provided with accommodation to stay there. Spends weekend in Gauteng and during week in North West.” 237 • The coach does not have enough time for admin, and is only allocated one hour per day to complete admin. • There are sometimes “Riots” or violence in the communities where the coach is meant to go, meaning she can no longer get to the school. This coach noted the following successes in providing coaching services to teachers in the schools: • Availability of transport has helped the coach in supporting the school, “That’s the main thing, we have transport all the time.” • When there is violence in the community, the coach is notified and can then visit another school – “Fortunately because of my good relationship with the teachers, the SMT would know that they are expecting me, they would phone and that is when I can make a deviation and go to another school.” • The coach is free to call her Head Coach if she has a problem, who she says communicates with her. COACHING SUPPORT The coach “Cannot complain” with the level of support she receives from the Head Coach. “My head coach supports me by coming to my schools once or never, depending on her schedule…When we are in shortage of material, she is able to deliver the material to where I am, while I am training teachers. She also attends the training…Once per month we also meet at the office to discuss whatever we want to discuss and share with her, together with other coaches.” The coach also feels supported by the SMT of all her schools. “Each time when I send my plan, they respond to me, they have open doors to me, I have never heard them complaining. They are very much supportive in anything.” The coach has not attended any additional training other than RSP training. EVALUATION OBSERVATIONS The coach is described by the researcher as “A great coach in terms of her personality, very friendly and humble.” The coach is very consistent in the Pre-observation discussion, and the Lesson Observations, checking consistently across schools and sections of the observations. An area for growth could be on creating and reviewing teacher profiles to plan her support. During the coaching sessions, an area for growth could be the attention paid to learners with barriers, as this was not always observed, and perhaps the coach could provide teachers with more ideas on how to support these learners. The coach has developed good relationships with all those she works directly with; the teachers, Head Coach and SMT. During the post-observation discussion, the coach then becomes less consistent as she does not always speak to all areas, and is described as “Brief.” An area for growth here would be on the reflective part of the feedback. While the coach gives suggestions and compliments the teachers on areas where they did well, the coach does not ask the teacher many questions to encourage their reflection and does not guide them either. _______________________________________________________________________ 238 Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 239 COACH CASE STUDY 12 PROFILE Qualifications Bachelor of Education (Honors), Education Training and Development (2003) Bachelor of Technology in Educational Management (1998) National Higher Diploma in Educational Management (1997) University Diploma in Education (Primary) (1994) Experience of teaching Foundation Phase Teacher, Primary School (1995-2009) Experience and training in coaching Class Act Educational Services, Coach / Mentor (2017) Matthew Goniwe School of Leadership and Governance, Inclusion Coach and Facilitator (2015) Class Act Educational Services, Facilitator (2010-2013) Class Act Education Services, Material Developer (Part-time: 2012- 2013) RSP training provided by OUP and Molteno on the lessons plans and components of language. Has not received training in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:11 Ratio of coach to RSP teachers 1: 56 Self-reported frequency of RSP coaching a teacher Twice per term. Some teachers are visited more frequently depending on the need and required intervention. 240 Data Source The coach was shadowed in the week of September 2, 2019, during which she was interviewed and observed at three schools working with nine individual teachers. However, one teacher was not available as she had to attend an SGB meeting during the planned observation. In addition, in all three schools this was a planned assessment week and therefore six teachers were not planning for a lesson, however at least four of them implemented a demonstration lesson. In four instances the researcher was not able to observe a feedback session by the coach to the teachers. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: The coach described her role as a coach to be “Coaching only to support RSP project.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials This coach does not share her observations with the school principal, and although she indicated that the SMT is supportive, no further insights were provided during her interview with the researcher. The researcher indicated that she was resistant to the coaching observations and therefore not forthcoming on sharing her experience. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The monthly work plan sets out the activities for the month, however during the researcher’s interview with the coach she was not able to produce the plan. This was later received by the evaluators. It is unclear if the plan was sent to the Head Coach or Subject Advisors and if any feedback was received. • The coach indicated that the monthly plan supports her in preparing for the visits to the schools. 2. Pre-observation discussion • The coach reviewed the learners’ books seven times, but did not undertake this in two observations. She did review the assessment sheets as the school during two of the assessment classes across two of the schools. 241 • The coach discusses the lessons plan with the six of the teachers and in one situation she explained to the teacher that teachers are required to write a lesson preparation document and keep it in a file. In another instance the teacher had to get back to an SGB meeting and therefore the observation was suspended. There was no discussion in two observations. The coach asked for the lesson plan in one instance, and this was not explicitly done in another observation. • Although in six class it was an assessment week, all of the teachers were asked to demonstrate a lesson which was then discussed (however, the researcher and coach did not provide explicit notes on these lessons during the coach shadowing observations). • The coach observed if the LTSM is available in four classes and the relevancy to the lesson in three situations. This was not always observed or relevant during three of the assessment classes, nor was it undertaken during two of the lesson observations. • The researcher notes that the coach did not explicitly observe if the LTSM was well organized during seven observations, however it was done in two class although the coach did not then observe if it was well organized in one instance. • She checks the teacher understands the content of the lesson in seven observations, and in one situation they discussed the assessment test sheet. In one observation this was discussed during the planning discussion prior to the lesson. • In eight classes in two schools, the Drop all and Read (DAR) and Drop Everything and Read (DEAR) were not clearly indicated on the time-table, instead the time￾table indicated “READING FOR FUN.” • The coach checks if learners are working well in organized groups which occurred in two observations, although the researcher rated this as being partially done in five instances. But this was not relevant during two of the assessment classes as leaners were being individually assessed, and it was not clearly recorded during one observation. • During lesson implementation, the coach observes and notes in five classes if LTSM is being properly used, once again this was not relevant for the two assessment classes. Although the coach noted the availability of readers in five of the classes where the teacher used this for a comprehension exercise (3). 3. Classroom Observation • The introduction acknowledges learners’ experiences and prior knowledge was checked by the coach five lessons, and once by reviewing the lesson preparation document of the teacher. However, in three classes this was not observed. • The coach checks the introduction is creative and relevant to the context of the planned learning content which was noted in two lessons, however, in two lessons this was not explicitly done by the coach. In one lesson she reviewed the lesson preparation document of the teacher, and in another three classes this was not demonstrated due to assessments. • The coach observes in five classes and makes notes on whether the outcomes of the lesson are clear, and in once instance by reviewing the lesson preparation document of the teacher, and another by observing adherence to the time-table. She makes notes on her observation tool. • She checks the routines are adhered to during the lesson against the lesson plan in six instances, and once by reviewing the lesson preparation document provided by the teacher. 242 • The coach checks if the core methodologies for reading are used to make the content clear to learners which was observed in five classes, and in two of the lessons the teacher demonstrated during the assessment week. • She further checks if reading strategies and processes are used during the lesson, which occurred seven times out of the nine classes (it was done in two of the assessment classes). • The coach also checked twice if reading skills are taught through focused activities, including during two of the demonstration lessons during the assessment class. This was not observed during five of the classes. She also checked seven times if focused activities are integrated within the components of language, which were noted three times to be adopted from the prescribed lesson plan and used for the lesson preparation. During the two full lesson observations the coach noted if focused activities are integrated within components of language. • The coach showed a particular interest on how learners with barriers are catered for by the teachers, and observed this in seven classes. In one instance the learners were requested to read individually by the teacher, and another teacher was encouraged to note individual learners and provide a remedial lesson. • She checked in seven classes if opportunities for application of new knowledge are created for learners, and this was not noted in one class. • The coach did observe and highlight the assessment types (formal, summative, informal and continuous), and noted that both summative and formative assessments are catered in five classes. • Learners were continuously assessed in all eight classrooms. • The coach checks if the planned assessment activities assess learner development and skills in all classes. With one teacher she stressed formative and informal assessments, in another she shared the assessment sheet with the researcher, and in a further observation she noted that the teacher asked the learners to read in pairs as the assessment activity. • She further checked twice if learners are assisted to reflect on new knowledge, skills, attitudes and values, but this was not explicitly catered for in six classes. 4. School based post-observation discussion • The researcher was not able to observe the coach in post-observation discussion in three instances, and the coach did not consent to a photograph of her providing feedback to the teacher. • The coach did not ask the teacher to identify the things they felt went well before providing her own observation in all five observations. She immediately offered her observations of things that went well. • The coach did not ask the teacher to identify things they would do differently in the future in all five instances, but the researcher noted in two instance this was undertaken as a dialogue. • The researcher noted that it was not evident in the interaction between the coach and the teacher in all five situations, that the coach asked questions to elicit information and prompt reflection regarding areas of improvement. • The coach offers her suggestions and referred to past interventions she had had with all five teachers. 243 • She does not guide any of the five teachers to identify solutions for problem areas in the form of action steps, nor is an opportunity created for the teacher to identify action steps for themselves. • The coach does take all the teachers through the lesson observation document, and the teacher had to sign the document to acknowledge this process. • She did not cover classroom management in three of the feedback sessions observed, but in two instance the coach did focus on feedback to the teacher- both of which pertained to the discipline of learners. • Curriculum coverage was not stressed or addressed by the coach in any of the five feedback sessions. • The extent to which the teacher follows the weekly routine was checked by the coach in four instances, but was not explicitly discussed. This was not observed by the research in one instance. • She did not check in two of the lessons the extent to which the lesson complied with the core methodology, which this was not relevant for three of the assessment classes observed. • The coach twice helped the teachers to identify solutions to potential barriers to implementation, and a discussion was noted with one teacher on how to address readers who are slow in reading acquisition. Her suggestions were noted by the researcher to be appropriate and reasonable in scope in three instances. • She did not provide examples of correct implementation with suggestions in four of the five feedback sessions observed. When she did provide a suggestions, this was provided with a rationale for each suggestion. • However, the researcher noted two situations when the coach provided negative feedback. In all instances the researcher noted she avoids judgement and bias. • Her suggestions for improvement were twice provided in a constructive way, however the researcher noted was partially executed in three of the instances. • In all five observations the teacher was not given an opportunity to identify action steps. The coach does not use questions to prompt reflection, and no goals were set between the teacher and the coach in any of the observations. 5. After school workshops • School-based workshops are provided for two hours after school to provide catch￾up LTSM training. • Needs-based after school workshops were included in her monthly plan. Four needs-based trainings were planned during the week of coach shadowing, none of which were held. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES She noted the following challenges in providing coaching services to teachers in the schools: • Unforeseen circumstances at the school prevent her from visiting the school and teachers as per the monthly plan. 244 • The appointment of unqualified teachers and the shortage of teachers at a school is the most challenging aspects of her role as a coach. To overcome this challenge, she conducts demonstration teaching sessions. She indicated that further support was not readily available to deal with challenges that were beyond her power. • The long distances to get to a school makes it difficult to get to the school for the start of the day. • The large number of schools and teachers make it difficult to visit and coach to meet the required operational procedures. • Providing feedback to the Principals is difficult as must uphold confidentiality. • It is necessary to train SMT’s on the RSP so that they understand the program and are then able to provide necessary support. This coach noted the following successes in providing coaching services to teachers in the schools: • Coaches must be honest and keep their promises to teachers as this contributes to building a trusting relationship. • Coaches must provide support and retain confidentiality on professional and personal issues of the teachers. • The daily logbook must be signed by the SMT, and coaches sign in at the school, which means that coaches do get to the schools. COACHING SUPPORT The researcher indicated that the coach received “Situational” support from the Head Coach but no developmental support, nor has she attended any developmental training. She does not feel adequately supported by the coach, but indicated she does receive support from the SMTs. However, she provided no further insights to the researcher. EVALUATION OBSERVATIONS The researcher indicated that the coach has relevant experience and qualifications, however she could strengthen her interpersonal skills, and the LTSM in the classrooms she observes could be displayed and organized more. The teachers she coaches generally lacked lesson plans, and if they were available, she would go through them with the teacher. Her administration could be strengthened given that the monthly plan was not available. She is authoritative and provides feedback directly without engaging the teachers to reflect on their own lesson and classroom practice. She understands the curriculum and lesson plan and is able to outline the steps and demonstrate delivery. She was noted by the researcher to be able to effectively identify learners with barriers and provide remedial actions. _______________________________________________________________________ Data Sources: Coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the Coaching Design; RSP Coach Job Description 245 COACH CASE STUDY 13 PROFILE Qualifications Monitoring, Evaluation and Research Skills Certificate (2016) Bachelor of Arts (BA) Honors degree: Management and Development (2012) Bachelor of Education Honors: Education Psychology (2001) Experience of teaching Deputy Chief Education Specialist: PL5, (2011–2016) Senior Education Specialist: PL3, (2006-2011) Head of Department: PL2, Primary (2000–2006) Teacher, PL1, Primary (1996‒1999) Teacher, PL1, Primary (1993‒1995) Experience and training in coaching Experience as Facilitator (ACE, OFTD; ECD NQF Level 4, SANTS College and NQF Level 4 and 5, Orbit College). RSP training provided by OUP and Molteno on the lessons plans and components of language for the training. Trained in the use of classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:8 Ratio of coach to RSP teachers 1:57 Self-reported frequency of RSP coaching a teacher More than four times per term. She visits some teachers more often than others based on their need. 246 Data Source She was shadowed in the week of September 2, 2019 during which she was interviewed and observed at two schools working with nine individual teachers (four of which were observed twice). The researcher was joined by the Khulisa Associate on September 3, 2019 for quality assurance purposes. Duration of practicing as an RSP coach 9 months Coach’s understanding of their role as a coach: “To train teachers in the content, train teacher on the use of LTSM, observe lessons, assist with challenges, close gaps and conduct school needs-based workshops.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The researcher notes that the coach does not present herself as an authority, but rather as a co-worker and a colleague to the teachers–that she has a feminine or motherly touch: “The teachers said she's like, a sister, a mother, a friend, a peer” and that they can rely on her “More than their own principal.” The coach views herself as someone who relates with people, “Sympathize, empathize and respect people. I do not make them feel I am superior.” In general, the teacher and the school welcomed the visit, and the coaching process is appreciated and valued by the coach and the teacher. Twice, it was noted that schools were flexible regarding the timetable–periods were adjusted twice to accommodate and enable her to observe teaching, even though it was an assessment week (twice). Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach circulated the September 2019 visit plan to the teachers, Heads of Department, Principals and Head Coach. • She reported during the interview that she uses color coding to help profile and prioritize her visits, and she checks the lesson document observations on the SOPs document to prepare for each visit. For example, for the month of September, she spent time on teacher profiles, but concentrated more on the red-coded teachers, as they need the most attention. 2. Pre-observation discussion • At the start of most of the lessons, the relevant files and documentation, including the lesson plan and preparations, the DBE book, etc., was already on the table 247 ready for the coach’s perusal, or given to the coach upon arrival (ten out of 13 observations). The researcher notes a specific time that the teacher shows the coach the lesson from the lesson plan, twice. Once, the teacher did not show or provide the coach with the lesson plan. • The researcher observes that the teacher does not introduce the coach to the class, most of the time (ten times), however it was clear that the learners know the coach (observed three times) and she is perceived to be a familiar face. • The coach does not usually look at learner books to understand progress in overall lesson plan–she was not observed doing this during ten of the 13 lessons. During the three lessons that she did review, she perused a sample of learner books as the lesson was in progress. • The teacher did not share his/her lesson plan for the day, nor was the lesson plan discussed with the teacher, as the researcher did not observe her doing either of these during any of the 13 lessons. She did read the lesson plan on her own from the file, but no discussion was held. 3. Classroom Observation • The coach always checks that the LTSM is available–sometimes looking or walking around to review it. However, she often fails to check that the LTSM is relevant (not observed 11 times) or if it is well-organized (not observed 12 times). • She is not consistent in checking if the teacher understands the content of the lesson. The researcher does not observe her checking this during five lessons, while she partially checked this four times. However, when she did check the teacher’s understanding (four times), she expressed her pleasure and satisfaction with the teacher’s preparedness and understanding. • The coach usually checks that the routine for DAR/DEAR is well established (fully observed nine times), and partial observation (three times) confirms that there are usually reading corners or small allocated areas in all classes. Learners are usually given an opportunity to read during assembly time once or twice a week (observed ten times). It is noted that two classes were “Too packed to have a reading corner.” • The coach can improve on checking if learners are working in well organized groups, as this was not observed during seven of the 13 lessons. Twice, however, the researcher noted that walking around in class is difficult due to limited space. When organized groups were observed, it was usually according to ability (observed six times). • She usually checks that the introduction acknowledges learners’ experiences and prior knowledge (fully observed 11 times)‒once it was noted that the teacher starts from known to unknown. She also observes if LTSM is properly used, most of the time (observed nine times). She expressed her pleasure with the teacher printing objects that relate to the vocabulary words of the day during one lesson, and notes the use of flashcards during another. Once she notes that the LTSM is not used properly, as sign and vocabulary words were not cut, and the big book was not well used. • She does not always check that the lesson introduction is creative and within the context of the planned learning content, as this was observed during seven of the 13 lessons–usually via song. She was not usually observed in making notes on whether the outcomes of the lesson are clear (not observed 11 times). Once, the researcher notices that she did not pay much attention to the outcome of the lesson. 248 • The coach checks if the routines are adhered to almost all of the time (observed 12 out of 13 times). For example, she kept referring to and studying the lesson plan as the lesson progressed, and showed satisfaction that the teacher is able to follow the lesson and weekly routines as protocol. • She does not always check if core methodologies for reading are used to make the new content clear to learners, as this was only fully observed five times. Once, she observes that core methodologies for reading are not used to make the content clear to the learners. She did note satisfaction on how group guided reading was conducted and applauded the teacher’s efforts. However, this step was partially observed most of the time (during eight lessons) as it was not always clear whether she fully checked that the core methodologies for reading are followed to make new content clear to learners. Although the coach was impressed with the teacher’s teaching of group guided reading, it is not clear if the core methodologies are used to make the new content clear to learners. • The coach usually checks if reading strategies and processes are used during lessons (fully observed nine times). Once, she picked up that the teacher was not using the reading strategies and processes and took over the lesson and assisted the teacher. She also clearly outlined the process that the teacher used in her reading lesson at another occasion. Twice, it was not observed at all. • Most of the time, it was not clear if the reading strategies and processes are observed (nine times), as it was fully observed only four times. Only once during those lessons did the coach notice this aspect as the teacher took a group of learners to the reading corner. During another lesson, she noticed that skills are not taught through focused activities as reading was done randomly in class–moving from one group to the other. • She does not always check if focused activities are integrated within components of language, as this was only fully observed during four out of the 13 lessons. Once, she applauded the teacher for using new words and checking if learners can recognize before reading. During another lesson, she notices that focused activities are not integrated within components of language. However, during two lessons this was not observed at all, and most of the time (seven lessons), it was partially executed, as it was not clear that the activities were fully integrated. Neither does she regularly check if learners with learning with barriers are catered for, as this was not observed during seven of the 13 lessons. • The coach did identify that learners were grouped according to abilities once and given differentiated tasks and/or special attention during five of the lessons. Once, she checked this partially, by being more focused on observing reading instead of other learners who are engaged in occupational activities. • She does check if opportunities for application of new knowledge are created for learners, as this was observed during 11 lessons. For example, learners spelled out words, wrote words in the air, flagged new words, and so forth. Once, she observed that the task given to the learners did not give the learners opportunity to apply new knowledge. • She often is not observed checking if learners are continuously assessed and given developmental feedback (not observed 11 times), mostly because she did not go through learner books. Furthermore, she often does not check whether the planned assessment activities assess learner development and skills (not observed nine times, partially observed three times) although there were instances where an assessment was given to the learners at that particular lesson. Almost half the time (seven observations) she does not check if learners are assisted to reflect on new knowledge, skills, attitudes and values–although when checked, she observed that 249 learners are not assisted to reflect on this. Sometimes, she only observed skills and values, but not knowledge and attitude (three times). 4. School based post-observation discussion • She fails to ask teachers to identify what went well during lessons, before providing her own observations; and she does not guide teachers in identifying action steps. Both of these steps were not observed during all of the 13 lessons. The researcher noted that she usually starts by giving feedback and then asking how the teacher thinks he/she did thereafter. • The coach usually does not ask teachers to identify things they would do differently in future; or pose questions to prompt clarification and reflection on areas of improvement–both steps were observed only once during 13 lessons. Interestingly, this was during a group feedback session, where the coach gave the teachers an opportunity to identify things they would do differently, and the researcher noticed that it was done well. However, she usually does not guide teachers to identify solutions for problem areas through action steps (not observed ten times). Once, she advised the teacher on innovative ways to improve her lesson offering (although action steps were not clear). • She successfully offers observations of things that went well–this was noticed during every lesson (13 times). The researcher notes that most of the time, she provides positive and negative feedback (nine times). Once, she congratulated the teacher as she could not fault her at all. She manages to almost always offer suggestions (observed 12 times). Once, the researcher noted that there were no areas for improvement, thus no suggestions provided. • Most of the time, the coach takes the teacher through the lesson observation document, as this was observed during 12 lessons. Once it was noted that she did this section with diligence, including providing comments on the observation tool. • With regards to discussing topics under the lesson observation document: curricula (during all 13 lessons) and classroom management (during 12 lessons) were well covered. She acknowledged that teachers were behind schedule six times, and on schedule five times. Once, she suggested a recovery plan because the teacher was four weeks behind schedule. She mostly applauded teachers on classroom management, although some classes were heavily overcrowded. On another occasion, she notes that classroom management was a challenge to the teacher. Other less well-covered topics included the: following of weekly routine (observed seven times), and compliance to core methodologies (observed six times). Once, she observed that the lesson did not comply with core methodologies. • The coach does well to offer suggestions that are appropriate and reasonable in scope (observed 12 times)–once, no suggestions were offered because she was so satisfied with the lesson’s content. She usually offers a rationale for each suggestion, as this was observed during 11 of the 13 lessons. She can improve with regards to helping recipients identify solutions to potential barriers to implementation (not observed ten times) and actions steps (not observed 13 times). Interestingly, the only time that she successfully guided teachers to identify solutions to barriers, were in the group feedback setting. Twice, she only partially performed this task, as she identified solutions for the teacher, rather than prompting the teacher to do so. • During the 13 lessons, she provided examples of correct implementation with suggestions five times, but this was not observed eight times. Once, she took over the lesson after an hour, and demonstrated to the teacher how to teach group guided reading. 250 • The coach demonstrates good communication skills, as she offers suggestions for improvement constructively (observed 11 times). Twice, she did not offer any suggestions, but when she did, she never offered suggestions in a manner that the researcher deemed as not constructive. She always avoids judgement or bias when providing observations and suggestions (observed 13 times). • The coach did not use questions to prompt reflection rather than telling the teacher their problem areas (not observed 12 times). She tends to speak most of the time, rather than prompting teachers with questions or allowing the sharing of experiences. She did not set concrete goals for the next visit with the teacher (not observed 13 times). 5. After school workshops • The coach facilitated one after school workshop during the coach shadowing period. She had not planned to conduct any after school needs-based workshops. • One was on the request of two teachers for help on differentiated reading for Grades 1 and 2. The researcher notes that other teachers from the Foundation Phase could have been invited to this training as only the two teachers attended, and he feels other teachers have similar issues. • The coach has not provided any catch-up LTSM training for teachers who might have missed the initial training, because she believes no teachers missed the initial training. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach noted the following challenges in providing coaching services to teachers in the schools: • The coach feels that in terms of content, her developmental needs are met. But she states that coaches require more developmental coaching, facilitation and/or report writing training–that her other training needs are not met and she is not adequately supported (see below). She also suggests performance developmental reviews for the coaches. • She notes that finding her way to schools has been one of the most challenging aspects of being a coach. She explains that some of the schools are in deep rural areas and the GPS is unable to pick up the school location. She requested attendance registers for telephone numbers and called teachers asking for directions. She receives support from the Logistics Manager and having vehicles with petrol cards and tablets with data to contact stakeholders, help to overcome these challenges. • She feels that the RSP program’s communication needs to improve, as the provincial office does not frequently communicate. She further notes that it seems like the head office is taking charge of the provinces, but that there should be autonomy in the provinces and not constant reliance on the national office. The researcher noted the following challenges: 251 • The researcher observes class size/overcrowding as a definite concern. For example, during two lessons, there were 64 and 65 learners in the class respectively, and that this class size makes it difficult to implement RSP and conduct group guided reading. There is no reading corner due to space limitation, and learners have to read at their tables. During these instances, it was very difficult for the teacher to walk around in class between groups, and she could only use the walking space in the front of the class. • Further development of the coach to provide feedback was noted by the researcher as she rarely allows the teacher to evaluate their own teaching or suggest solutions or explain what they think went well. It was noted that the time of feedback after each lesson is not used effectively, as she tends to rush over feedback. Once, she had limited time for undivided feedback, and she arranged a group feedback after school. On this day, the coach discussed general challenges first, and then delved into individual challenges and where teachers did well, with the group. The researcher notes that this was problematic, as teachers did not seem comfortable to discuss their own “weaknesses” in front of others. • She often does not complete pre-observation before class, thus she needs to improve her document perusal processes (reviewing learner books, etc.). • The researcher comments that it seems that the instruments used by the coaches are limiting, thus observations shared in writing, was scarce–yet detailed in giving verbal feedback to the teachers. • Although it seems that the WhatsApp group platform improves the distribution of schedules and communication between the coach and teachers, it was noted that the coach circulated the approved school visit schedule for the month of September to teachers/schools only two working days before the beginning of the month–which is very short notice and could cause problems. This coach noted the following success in providing coaching services to teachers in the schools: • Amicable interaction and good relationships with teachers are important. COACHING SUPPORT The coach confirms that the Head Coach accompanies her to schools and she does receive feedback on her monthly coaching schedules. However, she is not clear on what the Head Coach’s role is, especially regarding what they should be supporting coaches on ‒thus she is not sure what to expect from the Head coach. She does not feel adequately supported. She indicated that she needs support in terms of her role and responsibility, and Head Coaches are better acquainted with coaching skills. She feels that the Head Coaches are in the position to assist more on coaching practice. The coach feels the Head Coaches should be more visible, as she does not feel her Head Coach’s presence. She feels that she is adequately supported by the SMTs, as she can communicate openly with them and they convey message to teachers. She feels welcomed whenever she visits schools, and is able to make copies and use other resources. 252 EVALUATION OBSERVATIONS As the researcher pointed out, this coach is knowledgeable about early grade reading, and she is highly educated and experienced. She is effective in pointing out areas of development while building positive relationships with teachers and schools. However, based on the evidence, it is clear that she requires more training and support. The coach acknowledged this, and the researcher emphasized that the coach needs assistance or more training on coaching and writing reports, especially in completing the lesson observation instrument. The researcher elaborated that she needs to understand that her role is “More developmental” than merely observing/indicating if all went well or not, and therefore her approach to providing feedback to teachers needs attention, for example, probing teachers to share their opinions on the lesson first, asking questions, setting action steps, and so forth. __________________________________________________________ Data sources: coaching Observation Tool Part C; coaching Observation Tool Part B Lesson Observations; researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the coaching Design; RSP coach Job Description 253 COACH CASE STUDY 14 PROFILE Qualifications Editing and Proofreading Certificate (2018) Bachelor of Arts (BA) degree (Minor in Setswana) (1992) Primary Teacher’s Certificate (1980) Experience of teaching Teacher, Setswana (Grades 8 and 9), Secondary School (2015) Provincial Manager, Via Afrika Publishers (2013‒2015) Provincial Manager (North West Province), Heinemann Publishers (2001‒2002) Teacher, Setswana and Afrikaans, High School (1988–1992) Teacher, Primary School (1981–1985) Experience and training in coaching Coach at National Education Collaboration Trust (NECT) (2015 – 2018) RSP training provided by OUP and Molteno on the lessons plans and components of language. Coach did not receive training on classroom libraries. Setswana language proficiency Fluent in reading, writing and speaking Ratio of coach to RSP schools 1:8 Ratio of coach to RSP teachers 1: 68 Self-reported frequency of RSP coaching a teacher Once per term. “There are some schools that I can go thrice in a month. That is the maximum that I can go to school in a month and the minimum is twice.” 254 Data Source The coach was shadowed in the week of September 9, 2019 during which he was interviewed and observed at three schools working with 12 individual teachers. Duration of practicing as an RSP coach 5 months Coach’s understanding of their role as a coach: “My role as a coach is to work with teachers, basically, in order to help them to be able to interpret and to implement the lesson plans that are given to them so that they can be able to use them profitably, and to help them in terms of the workshops to sharpen their skills in teaching.” Coaching Practice The nature of interaction between the coach and the teachers, Principal/SMT and officials The coach feels that he receives very good cooperation from most teachers. He notes that when he started in this role, he made teachers aware that his role is to help them, especially regarding interpreting material that might be new‒rather than pinpointing their mistakes. He regards teachers as his colleagues, and asked them to regard him similarly. The researcher felt that the coaching relationship is mostly good. The researcher further notes that the coach is extremely good and knows what he is doing, and that he has a very professional outlook. The researcher further noted that he “Lacks the personal touch” and during two of the 12 lessons, reported that there was not a good relationship between the coach and teacher, and the relationship seemed “Questionable”, these were both female teachers, however, so are all the other teachers that the researcher observed the coach interact with. Coaching Standards THE FIVE STAGES OF COACHING PRACTICE This section combines the coach’s reflections with the researcher’s observations. 1. Planning and preparation • The coach considers which teachers he has not supported and then plans to observe those teachers in that particular language, while focusing on potential challenges with that school or teacher. • He circulates the coaching schedule beforehand. For the month of September 2019, he sent it to all 68 teachers allocated to him, on August 31 via WhatsApp group chat. This was followed by an email to all Principals of schools participating in the program. He does not call every teacher to confirm or re-confirm appointments or visits. He assumes that the teachers have received the plan and are aware of the visits once the schedule is sent through WhatsApp and email. 255 • The coach also shares the schedule with the Head Coach on the same day as it is circulated to the schools. He does not circulate the schedule to the Subject Advisors, as his understanding is that it is the Head Coach’s role to communicate with Subject Advisors. • Although he used to in the past, he does not call each teacher to check curriculum coverage‒he performs this task during school visits. • The coach did not create/review teacher profiles for each teacher before going to the school for the month of September. • He makes sure that he always has a coaching tool with him. He feels it is not appropriate to prepare for a specific week only, and then find out that a teacher might be behind. He feels it is important to be familiar with the lesson plans and core-methodologies because that is how coaches support teachers. 2. Pre-observation discussion • During ten of the 12 observed lessons, the teacher briefly shows the coach which lesson is going to be presented from the lesson plan. Out of these ten lessons, two lessons are behind schedule, while three classes already completed all lessons for the term. For the two lessons that this did not occur, one teacher was not following the RSP lesson plan (no lesson plan prepared) and the other teacher was busy with assessments when the coach and the researcher arrived. • The teacher introduces their coach to the class during only three out of the 12 lessons. Only during one of these three lessons do the learners seem to be familiar with the coach. During another lesson, the researcher observes that the teacher did not seem welcoming and appreciative of the coach’s presence–she was busy with a reading assessment and she continued without “Taking the coach on board.” • Most of the time, the coach takes time to discuss the lesson plan with the teacher (observed eight out of 12 lessons). Once, the researcher observes that the teacher shares, but does not discuss the lesson plan. Once, the teacher was not prepared for the lesson, and she did not know where she was in terms of the lesson plan. • The researcher notes that the coach looked at the group guided reading books once, but he did not seem to thoroughly check learner books for an understanding of progress during any of the 12 lessons. • The coach allows the teacher to share the lesson plan with him most of the time–in this case, during ten of the 12 observations. However, during three of those, the teacher only showed the plan and did not seem to fully share it. The researcher notes in this case, the teacher seemed too occupied with an assessment to take her guests through the lesson plan. On the other occasion, it seems like the teacher does not follow the RSP lesson planning at all. 3. Classroom Observation • The coach always observes that LTSM is available (12 times), and usually checks that it is well-organized (ten times), but his observations around LTSM’s relevance, are less noticeable. During the 12 lessons, LTSM’s relevance is not observed eight times, partially observed three times and fully observed only once. • He checks to ensure that the teacher understands the content of the lesson, was observed during only three lessons (out of 12). Once, the researcher notes that the coach paid attention to the understanding and content knowledge of the teacher, and provided corrections as the lesson progressed. Overall, the coach did not check the teacher’s understanding of the lesson’s content 75 percent of the time. 256 • With regards to classroom organization, he thoroughly checks that the routine for DAR/DEAR is well established during only three lessons (out of 12). Once, he notes that the Grade 3 timetable does not cater for DAR/DEAR as reading except where lesson-based reading is done. During the other nine lessons, the researcher notes partial observation five times. Once, the teacher did not have the timetable in her possession and claimed that the timetable was at the office. • The coach checks whether learners are working in groups during almost all of the lessons (11 times), but the researcher notes that he does not seem to check whether these groups are well-organized groups. For example, it is observed that learners are sitting in their ability groups but he did not check if they are working well. He often notes (five times), that learners are grouped according to ability. The researcher notes that one Grade 1 classroom was arranged in a formal, old￾fashioned fashion, rather than in groups. • Half the time (six of 12 lessons), he observes if LTSM are properly used. The researcher notes that he does not always observe the coach checking this (five times), and once, it was partially checked. Once, the coach noted that the teacher does not use the LTSM correctly. • Regarding lesson implementation, the researcher observes that the coach does not always completely check the relevant aspects of the introduction. During five lessons, he did not check that the introduction acknowledges learners’ experiences and prior knowledge (this was fully observed three times, partially observed four times). Furthermore, during eight lessons, he did not check that the introduction is creative and within the context of the planned learning (fully observed twice, partially observed twice). Once, the coach noticed that the teacher did not introduce her lesson at all. • The coach sometimes observes and makes notes on whether the outcomes of the lesson are clear–this was observed three times. Twice, however, the researcher notes that this was checked, but the teacher did not clarify the outcomes. During five lessons, this step was partially completed‒and noted that the outcomes were outlined‒but not completely clear. During four lessons, the researcher did not observe the coach making sure outcomes of the lessons are clear. • The researcher notes that most of the time (during eight lessons), the coach checks the lesson plan‒but during those lessons, it is not always clear if he observed whether routine is indeed adhered to (five times). Three times, the coach fully checked that routines are adhered to, and once, he did not check at all. Once, he kept the lesson plan open next to him and followed it continuously, observing that routines are adhered to very well. • With regards to teaching and learning, the coach usually checks if the core methodologies for reading focus are used to make content clear to learners (fully checked seven times, partially checked twice). The researcher notes three times that the coach checked and observed that core methodologies are not followed as indicated. He also usually checks if reading strategies and processes are used during lessons–in this case, it was noticed nine times. During the four lessons, he observed that strategies and processes are not used during the lesson. The researcher noted that sometimes, he would demonstrate to the teacher exactly what to do when reading to help develop the teacher’s own understanding of reading pedagogy and strategies. • The coach does not always check if reading skills are taught through focused activities–during the 12 lessons, this was not done five times. Once, it was noted that he checked if all the punctuations marks are observed and words are correctly 257 spelled based on his observation, and once, reading skills were not taught through focus activities. • The coach is passionate about contributing to teachers being able to implement their own lesson plans, because the researcher notes that the coach kept on stepping in where he felt the teacher needed assistance (be it classroom management, or explanation of concepts, or the correct use of LTSM). He would ask teachers if he could add, or “Do it for them (if he felt it was not properly done).” He would then take over the class for some period so the teacher can observe, and then hand the class back to the teacher. “So it was like a baton, taking back from you.” The researcher felt that using the demonstration method worked, as learners won’t necessarily see that the teacher is being coached. • Just over half the time (seven out of 12 classes), the researcher noted that the coach checks if focused activities are integrated within components of language. For example, once he corrected the punctuation of learners’ reading and helped with word recognition. Twice he noticed that focused activities are not totally integrated within language components. • Most of the time, the coach does not check if learners with learning barriers are catered for through differentiated activities; nor that opportunities for application of new knowledge are created for learners (eight times for both steps). Once, during the other four classes where it was checked, he noticed that learners with learning barriers are not catered for, as learners were not arranged in groups, but rows. Therefore, it was difficult to identify a group with learning barriers. Once, during break, a teacher informed the coach that she has four learners with barriers. • The coach does not always perform the required checks around assessments. During the 12 lessons, he did not check eight times if learners are continuously assessed and given developmental feedback; nor five times whether the planned assessment activities assess learner development and skills; and seven times if learners are assisted to reflect on new knowledge, skills, attitudes and values. He did note during three lessons that no planned assessment(s) were observed. 4. School based post-observation discussion • The coach usually does not ask the teacher to identify the things they felt went well, before providing his own observations (ten times), and he fails to ask teachers to identify things they would do differently in the future (12 times). Furthermore, during none of the 12 lessons did he ask questions to elicit clarification and reflection on improvement areas, or guided teachers to identify solutions for problem areas via action steps (12 times). The researcher notes that the coach does take time in providing feedback, but he tends to provide feedback first. It was suggested that he could start with probing questions and build from there. • On the other hand, it was well-observed that he always offers suggestions and provide the teacher with positive and negative feedback during every observation (12 times), and he always takes the teacher through the lesson observation document (12 times). • The coach often does not observe classroom management (not observed eight times), the extent to which the teacher follows weekly routine (not observed eight times), nor the extent to which the lesson complied with the core methodology (not observed seven times). The researcher notes that: - During the two lessons that classroom management was covered properly, the coach applauded the teacher for managing the classroom well (once), and although he did not comment on the tool, he indicated that the teacher is able to manage classroom (once). 258 - During the four lessons that weekly routine was covered, the step was partially executed twice, and fully executed twice. Once, he noted that although the teacher was behind, she followed a weekly routine. - During the lessons that compliance to core methodology was checked (five lessons), the coach noted that the teacher did not comply (once). • The coach covered curriculum coverage more often during the observations (eight times). During which time he noticed that teachers were behind schedule (four times) and performing revisions due to completing the lesson plan (twice). • The coach does not help recipients identify solutions to potential barriers to implementation (not observed 12 times), he does not provide examples of correct implementation with suggestions (not observed 12 times) and he does not guide the teacher to successfully identify action steps (not observed 12 times). However, he offers suggestions that are appropriate and reasonable in scope (12 times) and always provides a rationale for each suggestion (12 times). • The coach has good communication skills, as he always offers suggestions for improvement constructively (12 lessons) and avoids judgement or bias when providing observations and suggestions (12 lessons). However, he does not use questions to prompt reflection (not observed 12 times), rather he tells the teacher what their problem areas are. The researcher notes that he does most of the talking. He is usually not successful in setting goals with the teacher for the next visit (observed three times only). When completed, these goals include working on vocabulary, planning and group guided reading. 5. After school workshops • The coach facilitated one needs-based cluster training during the coach shadowing period, although he had planned to facilitate four. The researcher notes that the coach’s workshop focused on group guided reading, and that all ten Foundation Phase teachers, including the Head of Department, Deputy Principal and Principal attended the workshop. • The researcher noted that it is very helpful when the Heads of Department are also class teachers as once training has been completed, they know what to look out for. • He does perform additional one-on-one catch up training with individual teachers when needed, working through the lesson observation and providing feedback, and focusing in on potential developmental areas. • The coach indicates that he has not come across schools or teachers that did not attend the initial workshop on LTSM. CONTEXT FACTORS THAT AFFECT COACHING PRACTICES The coach noted the following challenges in providing coaching services to teachers in the schools: • Sometimes it is not possible for him to visit all teachers as planned. He explains that these include contextual factors such as a teacher falling ill, or sport, music or other activities that causes him to be unable to see all the teachers on a particular day. Also, he indicates that he has a lot of teachers to support. • He feels that not finding the teachers he is supposed to train because they are absent/unavailable is one of his biggest challenges, and he cannot do anything 259 about it. He does perform one-on-one sessions with teachers to catch up on what he/she has missed, but he does not have immediate support for that. He says coaches need support such as Subject Advisors, or someone with authority, to ensure they are doing their work. • He also notes that he has not been able to provide a lot of virtual/remote support for teachers, but does not explain why. He elaborates that although he has indicated on the WhatsApp group that challenges that may arise, can be discussed “But as you know, it is very seldom that they would use the platform. It is very low, as I'm saying, it's very low.” • He feels that the observation tools can be improved, as it does not really speak to or provide a sense of the needs in the school. He notes that he does not think managers of the program “Get what they want from [the] school because it can be interpreted differently and we can’t give it to the teacher and say this is what [we] have observed.” • Sufficient time to perform administrative duties is also a challenge, as coaches tend to be very busy throughout the week and time is also required for reflection. He suggests that one day a week is allocated to administration, without school visits, and allow for reflection on the week and for the future. • He mentions a transport challenge which impacts coaches and Head Coaches’ ability to visit schools as required/needed–further elaboration provided below. The researcher noted the following challenges: • With regards to classrooms: twice, the researcher observed that there is no clearly marked reading corner. Once, he noticed that there were shelves with Vula Bula reading books, and boxes used as shelves to house other reading books. • Twice, the researcher shares that the class is built out of corrugated iron, therefore could not hold on the posters. In one of these classrooms, it was noticed that there were no flash cards mounted on the wall due to heat that is absorbed by the walls of the classroom. However, the teacher used the chalkboard to display. In total, it was observed that three of the 12 classrooms are not print-rich. • Although it seems that the WhatsApp group platform improves the distribution of schedules and communication between the coach and teachers, it is noted that the coach circulated the approved school visit schedule for the month of September to teachers/schools during the weekend before the beginning of the month–which is very short notice. This coach noted the following successes in providing coaching services to teachers in the schools: • He feels that he receives very good cooperation from most teachers and feels adequately supported by the SMT, as he always interacts with them. • The researcher noted that the Principal and Deputy Principal of the school also attended the workshop he presented. He notes that this shows support from the management side. The researcher suggested that Deputies and Principals should be attending these workshops after school. 260 COACHING SUPPORT The coach mentions that he receives visits from the Head Coach once a month or every two months. He confirms that he does receive feedback/approval on his schedule from the Head Coach, in this case, the day after receipt. He received a message to confirm that he may proceed, but no physical signature, hard copy message or report. He feels that the support from the Head Coach is not adequate, seeing that there are so many coaches. The researcher notes that some improvement is required from the Head Coaches, and that the roles of the Head Coaches should be outlined to the coaches, as it seems they are not clear. The researcher feels that the Head Coaches should be empowered further. The coach also indicates that transport is additional challenge: that Head Coaches do not always have transport to visit schools. For example, his Head Coach needs to arrange transport with another coach, and often the coach will need to take the Head Coach back to the provincial office. He is of the opinion that this lack of transportation impacts the progress of the support he receives, as coaches and Head Coaches are unable to come out as often as required and/or needed. EVALUATION OBSERVATIONS This coach is an experienced teacher and trainer with previous coaching experience for NECT and other organizations. The researcher noted that he “Knows what he is doing,” and is effective especially with workshop facilitation, when he presents himself with authority. However, to the researcher, it seems that he can come across as impersonal (“Lack the personal touch”), and he seems to need to develop his skills to allow the coaching process to be a more collaborative process. For example, that he should use probing questions more and allow teachers to provide feedback (before he shares his own). The coach indicated to the researcher that he does not feel that his development needs are met, as he has not received any formal/developmental coaching training. He stated that he needs training on coaching, that he “Would love that to be made available to us,” as that would make their work better. The researcher suggested that the DBE should arrange continuous workshops around coaching, because the world and coaching is evolving: “We cannot have a coach and give them a crash program and expect them to perform miracles. They need to have ongoing training programs.” ______________________________________________________________________ Data Sources: coaching Observation Tool Part C; Coaching Observation Tool Part B Lesson Observations; Researcher debriefing interview September 2019 Valuing Criteria based on: RSP Standard Operating Procedures; Design Report (August 2019) Table 7: Assessing the coaching Design; RSP Coach Job Description 261 COACHING CASE STUDY RUBRIC Key Word Description Extensive Met the criteria / all elements AND the researcher noted additional activities, for example tasks, level of engagement with teacher or school Fully Consistently met the criteria / all elements of the coach observation tool were noted by the researcher to have been completed by the coach Partial The criteria / element / activity was partially completed (met some of the criteria, or was not implemented to the desired standard, or implementation was not consistent) Inadequate Did not meet the criteria/element or activity, for example did not do the activity or did not do it to the desired standard Not reviewed This element was not included in the field work observation or interview tool Not observed The researcher noted that this element was not observed during the coach observations visits Not reported This element was not documented by the researcher Unclear It was unclear despite the researcher trying to gather evidence (for example, it was unclear if the head coach did provide feedback to the coach on the schedule or not) () Reason for rating explained or further explanation to inform evaluation or RSP Criteria from Coaches’ Job Descriptions Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 Coach 7 Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 Coach 13 Coach 14 Qualified as teacher Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Experience in teaching Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Extensive Ratio coach to schools 1: 10 9 8 8 7 8 10 12 14 10 7 11 8 8 Ratio coach to teachers 1: 56 43 55 46 50 40 52 61 57 55 57 56 57 68 Setswana language proficiency Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Fluent Develop monthly plan and share Develop monthly visit schedules Fully Fully Fully Fully Inadequate Fully Fully Partial Fully Fully Fully Fully Fully Fully 262 Criteria from Coaches’ Job Descriptions Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 Coach 7 Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 Coach 13 Coach 14 Monthly Plan submitted to Head Coach Fully Fully Fully Fully Inadequate Fully Fully Fully Fully Fully Fully Unclear Fully Fully Head Coach Approval Partial Partial Partial Fully Inadequate Fully Inadequate Inadequate Inadequate Inadequate Inadequate Inadequate Fully Fully Coach tells teacher about planned visit Fully Fully Fully Fully Fully Fully Fully Fully Fully Partial Partial (tells SMT) Unavailable Fully Fully Coach sets up direct communication channel with teacher (not through school) Fully Fully Fully Fully Fully Fully Fully Fully Fully Partial (no airtime) Inadequate Inadequate Fully Fully Coach re-confirms visit Fully Fully Fully Fully Fully Partial Fully Fully Fully Partial Inadequate Inadequate Fully (no airtime) Inadequate Prepares for the visit by reviewing notes, looking at WhatsApp messages, etc. Partial Partial Fully Fully Partial Fully Fully Fully Inadequate Fully Inadequate Inadequate Fully Inadequate Conducting classroom visits and lesson observations Follows steps of Class Observation Fully Fully Fully Fully Partial Partial Fully Fully Fully Fully Partial Partial Partial Partial Monitors progress against lesson plans for the whole lesson Fully Fully Fully Fully Inadequate Partial Fully Fully Fully Fully Fully Partial Fully Partial Looks at learners’ books Fully Fully Fully Fully Inadequate Partial Fully Fully Fully Partial Fully Partial Inadequate Inadequate Monitors teachers’ assessment processes Fully Fully Fully Fully Partial Partial Fully Fully Fully Fully Partial Inadequate Inadequate Partial Provide in-class and in-school coaching Provides feedback to teachers Fully Fully Fully Fully Fully Fully Partial (but not appropriate) Fully Fully Fully Partial Inadequate Fully Fully Teachers reflect on their practice Partial Inadequate Partial Fully Partial Fully Partial Fully Fully Partial Partial Inadequate Partial Partial Conducts after school workshops / Facilitate PLCs for teachers to reflect on opportunities for improvement Conducts needs￾based workshops in the afternoons 2 of 4 0 of 0 1 of 4 2 of 4 0 of 4 0 of 4 3 of 4 2 of 3 4 of 4 2 of 4 1 of 0 (2 schools attended) 0 of 4 1 of 0 1 of 4 Sets up PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed 263 Criteria from Coaches’ Job Descriptions Coach 1 Coach 2 Coach 3 Coach 4 Coach 5 Coach 6 Coach 7 Coach 8 Coach 9 Coach 10 Coach 11 Coach 12 Coach 13 Coach 14 Facilitates PLC Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Teachers reflect on their practice Not observed Not observed Not observed Fully Not observed Not observed Inadequate Fully Fully Partial Inadequate Inadequate Inadequate Inadequate Complete and file prescribed coaching and reporting tools Completes in-class coaching tools Fully Partial Fully Fully Partial Fully Fully Fully Fully Fully Fully Fully Fully Fully Completes After School Workshop / PLC tools Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Not Reviewed Completes training tools Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Not observed Participate in train-the-trainer sessions to enhance own professional growth and development Participates in training sessions 16 days 16 days 16 days 16 days 16 days 16 days 16 days 16 days 11 days 6 days 16 days 16 days 11 days 11 days Head coach support (as reported by coach) Head coach visits the coach at school (since January 2019) Once Once Three times None None Yes Not reported Once Occasional None Once None Yes Three times Head coach provides support to coach Inadequate Inadequate Inadequate (only admin support) Inadequate Inadequate Adequate Adequate Inadequate Partial Inadequate Inadequate Inadequate Inadequate Partial 264 ANNEX 6: EVALUATION TOOLS RSP Coach Administrative Checklist Page 1 of 3 RSP IMPLEMENTATION EVALUATION COACH ADMINISTRATIVE CHECKLIST OBSERVATION DETAILS RESEARCHER: DISTRICT SCHOOL NAME SCHOOL EMIS NUMBER COACH NAME: DATE: Questions Response options Did the school (or head coach) share the school timetable with the literacy coach? Yes No Has each teacher in the school accommodated / scheduled the coach visit into their activities for the day? Yes, it is evident that teachers are prepared to accommodate the coach visit No, it is evident that teachers are not prepared to accommodate the coach visit Do the coaches have a coaching plan? Yes No Is it a year plan, quarterly plan, or monthly plan? (Please select all that apply) Daily Weekly Monthly Quarterly Yearly Does the coach identify and document support required for each individual teacher? Yes, for all teachers Partially, for some teachers No Does the coach complete a lesson observation tool while at the school? Yes, for all lessons Partially, for some lessons No If yes, how thoroughly is this filled in? Thoroughly filled in with a lot of details Filled in, but with short answers and not much detail Only a few questions filled in, and many questions left blank All questions left blank Please take a photo of the lesson plan observation tool Does the coach complete a school visit log form? Yes No RSP Coach Administrative Checklist Page 2 of 3 If yes, how thoroughly is this filled in? Thoroughly filled in with a lot of details Filled in, but with short answers and not much detail Only a few questions filled in, and many questions left blank All questions left blank Please take a photo of the school visit log form Does the coach complete any other tools while at the school? Yes No Please list what other tools the coach completes during the school support visit Does the coach complete the tools electronically or on paper-based format? (Please select one) Electronically Paper-based Both Does the coach share the lesson observation plan with the teacher before the class? Yes No Does the coach provide feedback to the principal and HOD? Yes No Does the coach provide reflective feedback to individual teachers after each coaching session? Yes, coaching report signed by teacher Yes, coaching report not signed by teacher No How often does the coach submit reports? Daily Weekly Monthly Quarterly Does the coach use a reporting template? Yes No Who does the coach submit reports to? (Select all that apply) Teacher Principal Head Coach District official Other (please specify): Does the coach get feedback on reports? Yes, after submitting all reports Yes, but only sometimes Not at all Has the coach attended any training to help write and submit coaching reports? Yes No Has the coach established a Professional Learning Community (PLC) at the school to facilitate peer learning between teachers? Yes No RSP Coach Administrative Checklist Page 3 of 3 Did the coach offer or administer a catch up training session to teachers? Yes No NA – all teachers at the school attended training Did the coach provide the Head Coach with feedback on rescheduling of visits (when and why) and why visits did not take place at all? Yes No NA Does the coach attend monthly reflection sessions with their Head Coach and if not, why not? Yes No Does the Head Coach monitor and support the coach through monthly classroom visits and if not, why not? Yes No Do Oxford University Press (OUP) and Molteno provide quarterly technical support to coaches on coaching and lesson plans? Yes No To what extent does the coach feel adequately equipped for your administrative work? To a large extent To a limited extent Not at all Does the coach have a set day to do the administrative work? Yes No If yes, which day of the week? Monday Tuesday Wednesday Thursday Friday Weekends How many days a month? How much time does coach spend doing admin in a week? (specify number of hours) 1 RSP TRAINING OBSERVATION: COACH DRY RUN Coach Dry Runs: 27 September, 2019 The following aspects must be looked for when observing the coach dry runs for the Reading Support Project (RSP): 1. How confident the coaches are in the dry run in terms of delivering this training (confidence in the materials and confidence in delivering training, as evidenced in the training style, ability to answer questions, pacing through the training curriculum, etc.) 2. The degree of fidelity to the content of the RSP training materials Please note that you must complete an information sheet and a rating sheet for each coach (14 coaches in total) during the dry run sessions. 2 Coach 1 Information Name of the coach (Coach 1) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 1 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 3 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 4 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 5 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 6 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 7 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 8 Coach 2 Information Name of the coach (Coach 2) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 2 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 9 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 10 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 11 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 12 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 13 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 14 Coach 3 Information Name of the coach (Coach 3) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 3 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 15 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 16 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 17 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 18 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 19 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 20 Coach 4 Information Name of the coach (Coach 4) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 4 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 21 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 22 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 23 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 24 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 25 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 26 Coach 5 Information Name of the coach (Coach 5) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 5 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 27 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 28 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 29 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 30 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 31 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 32 Coach 6 Information Name of the coach (Coach 6) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 6 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 33 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 34 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 35 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 36 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 37 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 38 Coach 7 Information Name of the coach (Coach 7) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 7 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 39 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 40 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 41 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 42 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 43 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 44 Coach 8 Information Name of the coach (Coach 8) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 8 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 45 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 46 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 47 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 48 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 49 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 50 Coach 9 Information Name of the coach (Coach 9) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 9 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 51 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 52 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 53 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 54 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 55 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 56 Coach 10 Information Name of the coach (Coach 10) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 10 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 57 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 58 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 59 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 60 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 61 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 62 Coach 11 Information Name of the coach (Coach 11) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 11 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 63 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 64 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 65 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 66 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 67 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 68 Coach 12 Information Name of the coach (Coach 12) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 12 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 69 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 70 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 71 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 72 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 73 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 74 Coach 13 Information Name of the coach (Coach 13) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 13 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 75 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 76 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 77 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 78 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 79 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 80 Coach 14 Information Name of the coach (Coach 14) Grade(s) covered during the dry run – Grades 1 and/or 2 and/or 3 Subject(s) covered during the dry run – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by coaches during the dry run – Setswana and/or English Are any Setswana and/or EFAL Curriculum Advisors present during the dry run? Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please make detailed notes on each coach Coach 14 Dry Run Observation Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 81 Question Rating Score Comment 2 Does the coach appear familiar with the lesson plans and LTSM, including how they should be used in class? Make detailed notes 3 Is there evidence of the coach adhering to the facilitator’s manual that explains to the coach how to facilitate the teachertraining workshop? Ask for a copy of the manual ahead of the observation sessions. 82 Question Rating Score Comment 4 Does the dry run address teachers’ content knowledge and knowledge of literacy concepts (i.e. in HL and EFAL literacy? – note examples. E.g. training addresses phonics, letter sounds, etc. Make detailed notes 5 Does the dry run address teachers’ pedagogy (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples. E.g. does the coach suggest different ways in which a literacy concept can be taught? Make detailed notes 83 Question Rating Score Comment 6 Does the coach demonstrate understanding of the five different reading methodologies (read aloud, shared reading, group guided reading, paired reading and independent) and how they work together? Make detailed notes 7 Is the style of the dry run participatory? – Does the coach solicit questions and engage the audience? 8 Is the coaches’ dry run presentation well assembled and put together? 9 Is there variety in how the presentation is made? i.e. the delivery of the dry run is not monotonous 84 Question Rating Score Comment 10 Is there a high level of interest from the dry run observers? How is this shown, or not? Are observers bored? 11 Does the coach make the purpose and structure of the training clear to observers as part of the dry run? 12 Does the coach receive feedback from the master trainers, other coaches, and/or curriculum advisors after the dry run? Make detailed notes 85 Question Rating Score Comment 13 Additional comments and observations? Make detailed notes 1 RSP TRAINING OBSERVATION: COACH TRAINING Training of Coaches (25 September – 27 September 2019) The following aspects (points 1 to 6 below) must be looked for when observing the training of teachers for the Reading Support Project (RSP): 1. Whether the master trainers are training coaches effectively on subject knowledge (EFAL and HL literacy for Grades 1, 2 and 3) and pedagogical knowledge (i.e. how to teach literacy) 2. How confident the coaches are in delivering this training (confidence in the materials and confidence in delivering training, evidenced training style, ability to answer questions, pacing through the training curriculum, etc.) 3. The degree of fidelity to the content of the RSP training materials as the train-the-trainer model is rolled out 4. Whether teachers demonstrate more knowledge over the course of the training (e.g. asking good questions, engaging with the content) 5. To what extent the integrity of the RSP is maintained through scale up (adhering to materials, approaches, etc.) 6. What factors constrain or enhance the scale up of the RSP training approach (e.g. language, number of people being trained, training environment, and other training practices) 2 Information Name(s) of the master trainer(s) conducting the training Date of observation Venue of observation Number of coaches in the room during observation Grade(s) covered during the training – Grades 1 and/or 2 and/or 3 Subject(s) covered during the training observation – Home Language (HL) and/or English First Additional Language (EFAL) Language(s) used by trainers during the observation – Setswana and/or English Start time of observation End time of observation Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please note that you must complete the general impressions on day one, as well as one training observation sheet per session observed (if applicable), per day. 3 General Impressions Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 2 Does the trainer allow coaches to introduce themselves? 3 Is an agenda for each day made available to coaches? 4 Are name tags are given out to coaches? 5 Is an attendance register made available and signed by the coaches? 6 Is an evaluation form, and/or pre- and post-tests available at the end of each day/end of training? 4 Question Rating Score Comment 7 Are there enough copies of the training materials for all the coaches? 8 Are the training materials in a file or folder or stapled? 9 Is there a facilitator’s manual that explains to the coach how to facilitate the teacher training workshop? 10 Are the training sessions building the necessary skills, according to the purpose and objectives of the training? – note examples 11 Are the trainers building coaches’ content knowledge (i.e. in HL and EFAL literacy? – note examples 12 Are the trainers building coaches’ pedagogy (i.e. the practice of teaching literacy and teaching methods)? – note examples 13 Is the style of training participatory? – look for examples of style and approach and how participants react to this. Note examples 5 Question Rating Score Comment 14 Has anyone stayed away from the training if it is holiday time? Get a copy of the attendance register and note absentees versus required attendance. 15 Are presentations well assembled and put together? 16 Is there variety in how presentations are made? i.e. the delivery of training is not monotonous 17 Is there a high level of interest and excitement in the room? How is this shown, or not? Are participants bored? 18 Is the purpose and structure of the training made clear at the beginning of the training, or early on? 6 Day One (Wednesday 25 September 2019) Please note: you may need to observe more than one session. Please fill this out for each session observed and indicate the time, venue, master trainer and description of the session. Start time: End time: Venue: Master trainer: Description of the session: Question Rating Score Comment 1 Is there a warm up activity/exercise? 2 Is the purpose of the day articulated/ written etc.? 3 Is the trainer confident about what he/she is doing? Not nervous; speaking clearly; not rushing through the module etc. 4 Is the pace of training right for learning? 5 Do all participants get a chance to give their thoughts and opinions on matters? – i.e. is there a balance between input, discussion and response? 6 Does the facilitator synthesise all inputs from participants at the end of a section? (very important skill) – i.e. participants know what the trainer is aiming for, in terms of the 7 Question Rating Score Comment development of an activity towards the expected outcomes. Note examples 7 Is the coaches’ knowledge being ‘scaffolded’, that is built-up in incremental blocks? i.e. learning not scattered, exercises are linked to previous learning, exercises discussed sufficiently 8 Is the timing of exercises appropriate to the level and depth of responses needed from participants? 9 Are the hand-outs appropriate to the learning activities and given out when needed? 10 If there are videos, are these well received and are the learning points clear? 11 If there are videos, are the participants asked about what they saw and heard? 12 If there are videos, is sufficient time given to checking on what was learnt from the videos? 13 Is the level of questioning by trainers becoming deeper and more reflective as the day progresses? - give examples; make a general comment and judgement. 8 Day Two (Thursday 26 September 2019) Please note: you may need to observe more than one session. Please fill this out for each session observed and indicate the time, venue, master trainer and description of the session. Start time: End time: Venue: Master trainer: Description of the session: Question Rating Score Comment 1 Is there a warm up activity/exercise? 2 Is the purpose of the day articulated/ written etc.? 3 Is there a link with the previous day? 4 Is the trainer confident about what he/she is doing? Not nervous; speaking clearly; not rushing through the module etc. 5 Is the pace of training right for learning? 6 Do all participants get a chance to give their thoughts and opinions on matters? – i.e. is there a balance between input, discussion and response? 9 Question Rating Score Comment 7 Does the facilitator synthesise all inputs from participants at the end of a section? (very important skill) – i.e. participants know what the trainer is aiming for, in terms of the development of an activity towards the expected outcomes. Note examples 8 Is the coaches’ knowledge being ‘scaffolded’, that is built-up in incremental blocks? i.e. learning not scattered, exercises linked to previous learning, exercises discussed sufficiently 9 Is the timing of exercises appropriate to the level and depth of responses needed from participants? 10 Are the hand-outs appropriate to the learning activities and given out when needed? 11 If there are videos, are these well received and are the learning points clear? 12 If there are videos, are the participants asked about what they saw and heard? 13 If there are videos, is sufficient time given to checking on what was learnt from the videos? 14 Is the level of questioning by trainers becoming deeper and more reflective as the day progresses? - give an example; make a general comment and judgement. RSP Head Coach Questionnaire Page 1 of 3 RSP HEAD LITERACY COACH QUESTIONNAIRE 1. Identifying information: 1.1. Name of Head Coach: ________________________________________________ 1.2. Names of Coaches Supervised by Head Coach: 1.2.1.________________________________________________________________ 1.2.2.________________________________________________________________ 1.2.3.________________________________________________________________ 1.2.4.________________________________________________________________ 1.2.5.________________________________________________________________ 1.2.6.________________________________________________________________ 1.2.7.________________________________________________________________ 1.2.8.________________________________________________________________ 1.2.9.________________________________________________________________ 1.2.10. __________________________________________________________ 2. Training on Standard Operating Procedures 2.1 Did you receive training from the RSP Monitoring and Evaluation advisor on the Standard Operating Procedures (SOPs) for recording, collation, reporting, storage, and management of the Reading Support Project data? 2.2 If yes, did this training help you understand the objectives of the SOP and other inter-related activities for accountability purposes? 3. Coaching Oversight 3.1. Do you collect lesson observation forms from your coaches? 3.1.1. If yes, how often do you collect them (per term)? ___________________ 3.1.2. If yes, do you review the contents for quality assurance and come up with comments for the literacy coach, where applicable? Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 Yes 1 No 2 RSP classroom walk-through checklist Page 2 of 3 4. INTERVIEW GUIDE 1.1. What is your role as a head coach? 1.2. How do you support the coaches? What works and what does not? 1.3. Whose role is it to communicate the coach visits to schools? 1.4. What is the quality of coaching? What needs to be improved? 1.5. What is the quality of the coaches reporting? 1.6. Are the coaches adequately reporting against the criteria of the lesson plans etc.? 1.7. What makes a ‘good’ coach? If you look at the coach criteria sheet do these adequately cover the coach roles? What should be added or changed for the head coaches? RSP classroom walk-through checklist Page 3 of 3 1.8. One of the challenges the coaches face is meeting with the teacher before the lesson to plan, what is your view on this and how can it be addressed? 1.9. Another challenge the coaches seem to be facing is holding the afternoon workshops – why is this? What can be done to address this? 1.10. Relationships between coach, head coach and SMT? Coaches are asking for more support and engagement from the Head Coaches, in your opinion, is this request valid? What support can you provide and how? 1.11. Challenges and successes of role of Head Coach? 1.12 If the RSP went to all schools in the district and province, what are the critical elements that must be planned when it comes to the coaching element? (prompt: recruitment and selection of coaches, training, supervision, conducting coaching sessions, how many head coaches to coaches etc.) Any further comments? Semi-Structured Coaching Observation Tool: Part 1 - School level data Page 1 of 5 RSP IMPLEMENTATION EVALUATION Semi-Structured Coaching Observation Tool OBSERVATION DETAILS RESEARCHER NAME: SCHOOL: COACH NAME: SCHOOL EMIS NUMBER: DATE OF VISIT: Please complete this tool with information collected at each school you visited with the coach. A. PLANNING AND PREPARATION 1. Did the coach circulate their schedule for the month of September 2019 to the school principal? a. Yes i. If yes, when?_______(insert date) b. No – they did not circulate their schedule to the principals c. No – they did not develop a schedule for the month of September 2019 Any comments? Semi-Structured Coaching Observation Tool: Part 1 - School level data Page 2 of 5 B. TIMELINE OF DAILY ACTIVTIES 1. Please take note of the time the coach leaves the school___________ 2. Please note if you observed the following steps: Yes No NA (fieldworker did not observe) Time The coach arrives at the school before the school starts The coach reports to the office on arrival at the school The coach introduces the guests with him/her to the principal and explain the purpose of the visit The coach/principal introduces the guest to the teacher and explain the purpose of the visit Semi-Structured Coaching Observation Tool: Part 1 - School level data Page 3 of 5 1. How many teachers did the coach PLAN to see during the daily visit? ___________________ 2. How many teachers did the coach ACTUALLY see during the daily visit? _________________ C. NEEDS-BASED WORKSHOP Comments NEEDS-BASED WORKSHOP Yes/No Did the coach conduct a needs-based workshop in the afternoon? If yes, proceed with following questions Start Time:___________ Please take a photo of the coach conducting the needs￾based workshop with teachers What was the content of the workshop? (please provide a brief description of what was covered) Semi-Structured Coaching Observation Tool: Part 1 - School level data Page 4 of 5 Comments Was it a Grade intervention or Phase intervention? (Please select one): □ Grade □ Phase If Grade, please select Grade: □ 1 □ 2 □ 3 How did the coach facilitate the needs-based workshop? Did all the teachers concerned attend? □ Yes □ No Please take photo of the attendance register Semi-Structured Coaching Observation Tool: Part 1 - School level data Page 5 of 5 Comments If there was no workshop in the afternoon, what were the reasons for not having it? (please provide brief description based on what the coach says) Were there any plans made for another workshop? If so, please state when it was scheduled for. Semi-Structured Coaching Observation Tool Page 1 of 15 RSP IMPLEMENTATION EVALUATION Semi-Structured Coaching Observation Tool Part 2: Lesson Observation 1. SCHOOL DETAILS SCHOOL NAME: SCHOOL EMIS NUMBER: TEACHER NAME TEACHER NATIONAL ID/ PASSPORT NUMBER LESSON OBSERVED (select) □ Home Language □ EFAL GRADE: (select) □ 1 □ 2 □ 3 2. PRE-LESSON ACTIVITIES 1. Did the coach circulate their schedule for the month of September 2019 to the teachers? a. Yes i. If yes, when? __________(insert date) b. No – they did not circulate their schedule to the teachers c. No – they did not develop a schedule for the month of September 2019 Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 2 of 15 Any comments? 2. Does the teacher briefly show the coach which lesson is going to be presented from the lesson plan? a. Yes, ____________(insert time) b. No c. N/A – fieldworker did not observe Any comments? 3. Does the teacher introduce the guest (coach) to the class and start with the lesson promptly? a. Yes, ___________(insert time) b. No c. N/A – fieldworker did not observe Any comments? Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 3 of 15 4. LESSON OBSERVATION While you observe the coach’s routine in the classroom, please mark the appropriate option and make relevant notes. Please take note of the start and end times of each of the key sessions below. Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments 1. PRE-OBSERVATION DISCUSSION Start Time:____________ The coach looks at learner books to understand progress in overall lesson plan The coach discusses the lesson plan with the teacher The coach allows the teacher to share his/her lesson plan for the day Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 4 of 15 End Time: _____________ Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments 2. LESSON OBSERVATION Start Time:_______________ LESSON PREPARATION The coach observes if LTSM is available The coach observes if LTSM is relevant The coach observes if LTSM is well organized Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 5 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach checks if the teacher understands the content of the lesson CLASSROOM ORGANISATION The coach checks if the routine for Drop all and read (DAR)/ Drop everything and read (DEAR) is well established For fieldworker: check if DAR/DEAR is on timetable (on the wall) and verify that the coach does the same. The coach checks if learners are working in well organized groups LESSON IMPLEMENTATION The coach observes if LTSM is properly used Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 6 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach checks that the introduction acknowledges learners’ experiences and prior knowledge The coach checks that the introduction is creative and within the context of the planned learning content The coach observes and makes notes on whether the outcomes of the lesson are clear TEACHING AND LEARNING The coach checks if the routines are adhered to For fieldworker: check if coach looks at the lesson plan to follow the lesson Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 7 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach checks if the core methodologies for reading focus (Shared Reading, Group Guided Reading, Paired/Independent Reading and Phonics) are used to make the new content clear to learners The coach checks if reading strategies and processes are used during lesson (pre, during and after reading) The coach checks if reading skills are taught through focused activities The coach checks if focused activities are integrated within components of language (vocabulary, word recognition, fluency etc) Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 8 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach checks if learners learning with barriers are catered for – differentiated activities The coach checks if opportunities for application of new knowledge are created for learners ASSESSMENT The coach checks if learners are continuously assessed and given developmental feedback The coach checks whether the planned assessment activities assess learner development and skills The coach checks if learners are assisted to reflect on new knowledge, skills, attitudes and values End Time:___________ Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 9 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments 3. POST OBSERVATION DISCUSSION Start Time: _______________ FEEDBACK PROCESS Please take a photo of the coach providing feedback to teacher (use ODK on tablet) Before providing his/her own observations, the coach asks the teacher to identify the things they felt went well The coach offers his/her observations of things that went well The coach asks the teacher to identify things they would do differently in the future Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 10 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach asks questions to elicit clarification and prompt reflection regarding areas of improvement The coach offers suggestions The coach guides the teacher to identify solutions for problem areas in the form of action steps The teacher successfully identifies action steps Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 11 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach takes the teacher through the lesson observation document The coach covers the following topics under the lesson observation document: Classroom management Curriculum coverage The extent to which the teacher follows the weekly routine Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 12 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The extent to which the lesson complied with the core methodology CONTENT The coach helps recipients identify solutions to potential barriers to implementation The coach offers suggestions that are appropriate and reasonable in scope The coach provides examples of correct implementation with suggestions Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 13 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach provides a rationale for each suggestion The teacher successfully identifies action steps COMMUNICATION The coach offers suggestions for improvement in a constructive way The coach avoids judgement or bias when providing observations and suggestions Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 14 of 15 Not observed Partially or unsuccessfully executed Completely and successfully executed Coach notes (from documents completed by coach) Researcher’s comments The coach uses questions to prompt reflection rather than telling the teacher their problem areas The coach and the teacher set goals for the next visit For fieldworker: Please note the goals discussed in the comment box End time:_________________________ EFFICIENCY Do the teachers being coached appear to be open to implementing the suggestions/action steps? Unreceptive 1 2 3 4 5 6 7 8 9 10 Highly Receptive N/A Overall, I would rate the coaching as: Not effective 1 2 3 4 5 6 7 8 9 10 Highly Effective N/A Collect form used by coach after lesson is complete and check what has been completed. Use this to cross check with what you observed and make any relevant comments. Semi-Structured Coaching Observation Tool – Part 2 Lesson Observation Page 15 of 15 ADDITIONAL COMMENTS Semi-Structured Coaching Observation Tool Page 1 of 19 RSP IMPLEMENTATION EVALUATION Semi-Structured Coaching Observation Tool Part 3: Coach Interview OBSERVATION DETAILS RESEARCHER: COACH NAME: SECTION A: COACH EXPERIENCE In this section, you will ask questions to the coach and write down the answers in this document. Before starting the interview, please turn on Otter on your tablet to record the discussion. Questions Responses Coaching Experience 1. How long have you been working as a literacy coach? Please describe your educational and professional background. Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 2 of 19 2. How would you describe your role as a coach? (e.g., work with teachers on new strategies, observing teaching practice, work with students, data analysis, co￾teaching, lesson plan, demonstration lessons, work with individual teachers and grade/phase level teams) School and teacher support 3. How many schools do you support? 4. How many teachers do you support? 5. How do you identify and record the needs of the teachers? 6. Have you rated the schools and teachers for differentiated support? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 3 of 19 7. If yes, what criteria do you use to rate the schools and teachers? 8. How many times do you support a teacher per term? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 4 of 19 9. Do you visit some teachers more than others, and if so why? 10. Do you manage to visit all teachers you plan to visit per month? If no, explain what challenges you face. Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 5 of 19 11. Have you provided any catch up training to teachers who missed the training on LTSM? If yes, how long has this training been and how have you organised and delivered it? If no, why not? 12. How do you build a trust relationship with the teachers? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 6 of 19 13. Do you share your classroom observations with the principal? 14. If yes, how does that make the teachers feel? If no, why not? 15. Do you provide virtual/remote support for teachers? If yes, how? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 7 of 19 Training and Support to Coaches 16. What kind of support do you get from the Head Coach? 17. Do you feel that you are adequately supported by the Head Coach? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 8 of 19 18. Do you feel that you are adequately supported by the School Management Team? 19. Have you attended any developmental training? 20. If yes, what was the content? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 9 of 19 21. Do you feel that your developmental needs are met? If no, why not? 22. Did you receive training on the classroom libraries that have been distributed to some RSP schools? (Probe: If yes, was it useful? If no, why not?) Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 10 of 19 23. What has been the most difficult/challenging aspect of your role as a coach? 24. How do you deal with these difficulties/challenges? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 11 of 19 25. Is support readily available to deal with challenges that are beyond your power? 26. Is there a system in place to ensure that coaches do get to the schools? If yes, how does it work? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 12 of 19 27. If you could change one thing about the RSP coaching programme, what would that be and why? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 13 of 19 SECTION B: SUPPORT VISITS PLANNING AND PREPARATION The Standard Operating Procedures (SOPs) for coaching under the Reading Support Project (RSP) outline the procedures to be followed by coaches before, during, and after each classroom support visit. Please check with the coach the below: 1. What informs the coach’s monthly/weekly and daily plans? (i.e., reports, teacher profiles, feedback sessions, etc.) 2. How does the coach prepare for each visit? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 14 of 19 3. Did the coach draw up a schedule for the month of September 2019? a. Yes. Started on ____________ (insert date) and submitted on _______________ (insert date) a. No, the coach did not draw up a schedule for September 2019 Any comments? 4. Did the coach send their schedule for the month of September 2019 to the head coach? b. Yes. Started on ____________ (insert date) and submitted on _______________ (insert date) c. No, the coach did not send their schedule for the month of September 2019 to the head coach d. NA – The coach did not draw up a schedule for the month of September 2019 e. NA – The coach does not recall Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 15 of 19 Any comments? 5. Did the coach receive feedback on their schedule for the month of September 2019 from their head coach? a. Yes a. If yes,__________________ (insert date) b. No, the coach did not receive feedback from the head coach on their schedule for the month of September 2019 c. NA, the coach either did not develop a schedule or did not send their schedule for the month of September 2019 to the head coach Any comments? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 16 of 19 6. Did the head coach sign off the coach’s schedule for the month of September 2019? a. Yes a. Please provide when received feedback____________ (insert date) b. No, the head coach did not sign off on the coach’s schedule for the month of September 2019 c. NA, the coach either did not develop a schedule or did not send their schedule for the month of September 2019 to the head coach for signature Any comments? 7. Did the coach circulate their schedule for the month of September 2019 to the Subject Advisors? a. Yes _______(insert date) b. No – they did not circulate their schedule to the Subject Advisors c. No – they did not develop a schedule for the month of September 2019 Any comments? 8. In the month of September, did the coach call each teacher to confirm each visit to each school? Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 17 of 19 a. Yes, called all teachers b. Yes, but only some teachers c. No Any comments? 9. In the month of September, did the coach call each teacher to check curriculum coverage (progress to date)? a. Yes, all teachers b. Yes, but only some teachers c. No Any comments? 10. In the month of September, did the coach call each teacher to re-confirm his or her appointment? a. Yes, called all teachers b. Yes, but only some teachers c. No Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 18 of 19 Any comments? 11. In the month of September, did the coach create/review the teacher profiles for each teacher before going to the school? a. Yes, and researcher was able to observe it b. Yes, but only some teachers c. No Any comments? E. EVIDENCE OF COACH OBSERVATION 1. Please take a photo of the completed Coach Lesson Observation Plan Semi-Structured Coaching Observation Tool: Part 3 – Coach Interview Page 19 of 19 F. COMMENTS Information Sheet and Consent Form for Coaches Page 1 of 3 READING SUPPORT PROJECT Information Sheet and Consent Form for Coaches WHO WE ARE Hello! I am ________________________. I am working on an evaluation undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be used to improve the programme in 2020. Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. As a key role in the RSP, all RSP coaches will be involved in the evaluation. YOUR PARTICIPATION We are asking you, as a coach, to be open to have an experienced education researcher follow you for four days as you carry out your routine support visits to schools. We call this a “coach shadowing” process. This shadowing will allow the researcher to understand your experience as a coach, and how you provide support to teachers, through lesson observations Information Sheet and Consent Form for Coaches Page 2 of 3 and feedback sessions with the teachers you support. The researcher will also conduct an interview with you to get a better sense of your experience with the RSP, what works well, and what can be improved. If you have any concerns regarding your participation, please contact Khulisa (see: Who to Contact If You Have Been Harmed or Have Any Concerns). CONFIDENTIALITY We undertake to handle all your information confidentially and for the purposes of this evaluation only. Records that identify you will be available only to people working on the evaluation study and RSP programme. The identity or names of coaches, schools, principals, SMT members, and teachers will never be attached to any of the information released through official reports or articles. RISKS/DISCOMFORTS While risks associated with participation are not expected, there are general risks associated with this type of research. These include: Information risks (e.g., loss of privacy and/or breach of confidentiality). We will ensure that all researchers are briefed on the importance of your privacy and confidentiality. In addition, all data will be protected in accordance with the Protection of Personal Information (POPI) Act. Psychological or emotional risks (e.g., fear, stress, confusion). We will ensure that all researchers are briefed on the importance of your psychological wellbeing. Please contact Khulisa (see: Who to Contact If You Have Been Harmed or Have Any Concerns) if you experience any psychological or emotional harm. BENEFITS The key purpose of the RSP is to improve learners’ reading proficiency through teacher interventions in the project schools, and your participation in this evaluation will assist with improving the project going forward. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Katharine Tjasink at Tel: 011 447 6464 Ext 3207, or e-mail at ktjasink@khulisa.com. You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. Information Sheet and Consent Form for Coaches Page 3 of 3 CONSENT I _________________________________ hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I also understand that I can contact Khulisa should I not want to continue. I understand that my participation will remain confidential to anyone outside of the evaluation study and RSP programme. I understand that the information that I provide will be stored in accordance with the POPI Act and that evaluation findings will be communicated to senior DBE managers, the programme funders, and through articles in academic journals without making known my identity or that of the schools, principals, SMT members, and teachers I support. _______________________ Signature of Coach ________________ Date 1 RSP EVALUATION PROVINCIAL & DISTRICT OFFICIAL INTERVIEW QUESTIONNAIRE This guide is to be used for the Reading Support Project Implementation Evaluation by the Khulisa Management Services Evaluation Team. Khulisa Management Services (Khulisa) – a Monitoring and Evaluation (M&E) firm based in Johannesburg. Khulisa Management Services (Pty) Ltd (hereinafter referred to as “Khulisa”) was awarded the Practical Education Research for Optimal Reading and Management: Analyse, Collaborate, Evaluate (PERFORMANCE) Indefinite Delivery Indefinite Quantity (IDIQ) contract to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from the PERFORM interventions. We thank you for taking the time for this interview. Your feedback and personal information will be kept strictly confidential and you do not need to share any sensitive or confidential information with me. Your perspective will be consolidated when reported. Do I have your consent to continue with the interview? Yes ________ No _________ Do I have your consent to audio record this interview? Yes ________ No _________ PERSONAL DETAILS Interview date Day Month Year Name of Interviewer Name of Interviewee (s) Interviewee Organisatio n / Department Interviewee Title / Designation QUESTIONS 2 1. Are you aware of the EGRS and did you have any experience with the EGRS study? 2. How have you been involved in the RSP (if at all)? What is your role? And how do the roles of coaches, Subject Advisors and district officials work together? 3. In your experience what are the key similarities between the EGRS and the RSP? …”In your experience, what are the key similarities and differences between the EGRS and the RSP?” 4. Has the RSP been integrated into the provincial structures and processes and, if so, how? 5. Are you seeing any positive changes in the schools as a result of the RSP? If so, what are these and why you think they occurred? 6. Do you feel that the RSP literacy coaches have a unique role in supporting teacher literacy practices and, if so, how is this role different to the support provided by HODs, SA’s, and other district or provincial support structures?” 3 7. Could the management and implementation structures be improved? If so how? If not, what is working well in how the program is being managed? 8. Do you receive reports on the RSP? If so, what type of reports and when do you receive them? And from whom? 9. Could the monitoring and reporting system be improved? Please explain 10. If the RSP went to scale in the district or province, how could this be achieved in terms of: a. RSP teacher training b. RSP coaching c. RSP materials d. How to involve the district and provincial officials e. Quality f. Languages 4 11. Is the RSP a cost effective project? Why or why not? 12. What value (for money) does the RSP bring to early grade reading? Any other comments? Observations: 1 RSP Curriculum Advisor Questionnaire Date of interview: Name of interviewer: 1. Identifying information: Role of advisor: Curriculum or Subject? 1.1. Name of Advisor: 1.2. Grade(s) covered by Curriculum/Subject Advisor: 1.2.1 Setswana or EFAL? 1.3. District(s) in which Curriculum/ Subject Advisor works: 1.4 Informed consent and permission to record interview? 2 1.5 How familiar are you with the RSP programme? 1.6 Did you receive a tablet loaded with RSP materials? CURRICULUM ADVISORS) 2. Training on Standard Operating Procedures 2.1 Did you receive training from the RSP Monitoring and Evaluation advisor on the Standard Operating Procedures (SOPs) for recording, collation, reporting, storage, and management of the Reading Support Project data? Yes =1 No =2 Comments 2.2 If yes, did this training help you understand the objectives of the SOP and other inter -related activities for accountability purposes? Yes = 1 No= 2 Comments 3. Attendance at RSP Trainings 3.1 RSP planned a number of training sessions for Advisors this year. 3 3.1.1 How many did you attend? Number 3.1.2 How many days in total? Number Comments: Why attend / why not attended? 3.2 RSP conducted a number of Reference Group meetings this year. 3.2.1 How many did you attend? Number 3.2.2 How many days in total? Number Comments: Why attend / why not attended? 3.2 RSP conducted a number of training sessions with coaches. 3.2.1 How many did you attend? Number 3.2.2 How many days in total? Number Comments: Why attend / why not attended? 4 3.3 RSP conducted a number of training sessions with teachers. 3.3.1 How many did you attend / monitor / observe? Number 3.3.2 How many days in total? Number Comments: Why attend / why not attended? 3.4 How many other meetings on RSP did you attend between Jan and October ? 3.4.1 How many did you attend? Number 3.4.2 Average length of meetings? Minutes 3.4.3 Total time spent (minutes) Comments: Why attend / why not attended? 4. Perception of the RSP 5 4.1 What is your role in the RSP? 4.1.1. What is your role in relation to Subject Advisor / Curriculum Advisor / Coach / Head Coach / Teacher and SMT? 4.2 What is the role of the RSP coaches? 4.3 In your view, how effective are the coaches? Please explain your answer 4.4 What are the main challenges to improving early reading in the schools you support? 6 4.5 What is working well in improving early grade reading? Please explain your answer 4.6 Is there evidence of RSP practices being implemented in class by teachers? Please explain your answer (evidence for this statement?) 4.7 Has RSP LTSM been provided to all the participating schools? Please explain / expand (are teachers using it? Have they seen the classroom libraries? Is it quality material in their opinion) 4.8 How could the RSP be made more efficient? 4.9 Do you think the RSP is a cost efficient programme? What, if anything, do you regard as good value for money in the RSP? 7 4.10 If necessary, how could management and implementation structures / processes be improved? 4.11 Do you receive reports about the RSP? If so from whom do you receive them? Does the content meet your needs? Could the monitoring and reporting system be improved? Please explain. 4.12 If the RSP went to every Foundation Phase in the District / Province, what would be the critical elements that the program MUST take into consideration for this to be achieved? (examples: how train teachers, how ensure quality, how will materials be distributed etc.) 4.13 How should district and provincial officials be involved if the project went to all schools (scale)? 4.14 What are the key considerations for taking the coaching element of the RSP to scale? 8 4.15 What has worked well in the past for taking education (teaching and learning) initiatives to all schools in the district and province? Any further comments? RSP Classroom Observation Tool Page 1 of 9 RSP CLASSROOM OBSERVATION TOOL 1. Demographics 1. Name of school Write name 2. School EMIS number Write EMIS number 3. District 1= Dr Kenneth Kaunda 2= Ngaka Modiri Molema 4. Name of fieldworker responsible for data collection Write initials and surname 5. Today’s date __ / __ / 2019 Write date 6. Day of the week Write the day of the week 7. Name of teacher Write initial and surname Note if no classroom observation 8. Teacher’s Date of birth Write the teacher’s Date of Birth (DD/MM/YYYY) 9. Teacher’s PERSAL number Write the teacher’s PERSAL number 10. Is the teacher observed the usual class teacher? 1= Yes 2= No – she is substituting 99= Don’t know 11. If no, where is the usual class teacher? 1= Absent due to illness 2= Attending workshop 3= Other, specify/explain 98= N/A. This is usual teacher 99= Don’t know 12. Name the specific Grade 1 or 3 class observed (or grades if combined grades) e.g. Grade 1a Write grade/s 13. Subject observed 1= Home language 2= EFAL 14. If you experienced any problems gaining access to the classroom to observe or were unable to observe a FULL Setswana Home Language / EFAL lesson, note details below. 0=No problems 15. Time lesson was supposed to start Write time. Use 24 hour clock e.g.09h45 98=N/A. No observation 16. Time lesson actually started Write time. Use 24 hour clock 98=N/A. No observation 17. Time lesson was supposed to end Write time. Use 24 hour clock e.g.09h45 98=N/A. No observation 18. Time lesson actually ended Write time. Use 24 hour clock 98=N/A. No observation RSP Classroom Observation Tool Page 2 of 9 2. Lesson Observation Schedule \ Section A: Teaching and learning environment 19. How many learners are actually present in class? (start of the lesson) Boys: Girls: Write the number 99= Don’t know 20. Is the classroom clean? 1= Yes 2= No 98= N/A. Outdoor lesson Please use your tablet to take a picture from one corner of the room, showing the layout of the room Once you complete this form on ODK, please upload the photo taken when requested on the form. 21. Can the teacher move around the room comfortably and walk easily between all the desks/tables? 1= Yes 2= No 98= N/A. Outdoor lesson. 21.1 If no to the question above, please explain 22. Is there a reading carpet, mat (or similar ‘shared reading space’) evident in the classroom? (e.g. for story time) Note: This is not provided by the RSP 1= Yes 2= No 98= N/A. Outdoor lesson. 22.1 If no to the question above, please explain 23. Is there a Home Language (HL) word wall in the classroom? 1= Yes 2= No 98= N/A. Outdoor lesson. 23.1 If there is a HL word wall, in the classroom, are the words systematically organised (i.e. are words/flash cards organised and grouped together on the wall because of shared features e.g. phonetically) 1= Yes 2= No 98= N/A. There is no word wall 24. Is there an English First Additional Language (EFAL) word wall in the classroom? 1= Yes 2= No 98= N/A. Outdoor lesson. 24.1 If there is an EFAL word wall, in the classroom, are the words systematically organised (i.e. are words/flash cards organised and grouped together on the wall because of shared features e.g. phonetically) 1= Yes 2= No 98= N/A. There is no word wall RSP Classroom Observation Tool Page 3 of 9 25. How many different types of learner’s recent work or other recent learner-made objects are on display in the classroom? (Note: This will be observed during Life Skills. LPs do not demand learners to do objects. Learner’ shared writing can be displayed) . Write one code only 0=None 1=1 or 2 types 2=More than 3 types 98= N/A. Outdoor lesson 26. To what extent do the following activities slow down teaching and learning at the beginning or during the lesson: 1. handing out workbooks/readers/exercise books or other material/apparatus/equipment to learners? Write one code for each 0= Not at all 1= Hardly at all 2= To some extent 3= To a large extent 2. teacher consulting her notes or lesson plan? 3. learners re-arranging the furniture, moving chairs/ desks, etc.? 4. learners who do not have necessary pens, pencils, crayons, erasers, rulers and other necessary stationery moving around the class borrowing from classmates? 27. During the lesson, are there ‘outside’ interruptions to teaching in the classroom (e.g. intercom announcements, other teachers entering the room to talk to the teacher; learners bringing messages)? Write one code 0= None 1= 1 - 2 2= 3 - 5 3= More than 5 28. When the class is busy with independent written work or reading tasks (whether in groups or alone), does the teacher … 1. monitor all learners and check that they are doing what they are supposed to be doing? Write one code for each 0= Never 1= Sometimes 2= Mostly 3= Always 98= Not applicable – no independent tasks/classwork given to the children 99= Don’t know – unable to ascertain this 2. supervise quite closely less capable learners as they work? 3. give additional tasks to learners who have completed their work/tasks and are coping well? (Note: Additional work can be given to the fast workers) . 29. To what extent do learner appear to be engaged with independent written work? 0 = None 1 = Some learners 2 = Most learners 3 = All learners 98= Not applicable – no independent tasks/classwork given to the children 99= Don’t know – unable to ascertain this 30. To what extent do learner appear to be engaged with reading tasks? 0 = None 1 = Some learners 2 = Most learners 3 = All learners 98= Not applicable – no independent tasks/classwork given to the children 99= Don’t know – unable to ascertain this 31. What do learners who finish their work ahead of other children generally do? 1. They take a book from the book corner to read Write a code for each category 1= Yes 2= No 3= No learner finishes ahead of others 98= N/A – no independent tasks/ classwork given 2. The teacher uses workbooks as a source or means for giving learners who finish classwork ahead of others opportunities to do more exercises 3. They help other learners with their work 4. They sit idle 5. Other, specify RSP Classroom Observation Tool Page 4 of 9 Section B: Discipline 32. Does the teacher refer to individual learners by their names? (or, for example, does the teacher just point at learners when asking questions) 1= Yes 2= No 33. In your observation, how many learners are: Number of boys Number of girls Write the number 0=None 99=Don’t’ know a) Not concentrating or paying attention b) Undisciplined or disruptive Section C: Listening and speaking 34. When the teacher provides front of class explanations, instruction or in class discussions, to what extent do learners answer in unison or repeat responses after or with the teacher, or after or with other learners, rather than actually coming up with their own responses? Write one code only 0= Never 1= Seldom 2= Sometimes 3= Mostly 4= Always 35. Does the teacher try to make sure that a wide variety of learners have a chance to answer questions by actively trying to involve those learners who do not voluntarily participate in teacher-learner interactions? Write one code only 0= Never 1= Seldom 2= Sometimes 3= Mostly 4= Always 98= N/A – teacher does not ask questions. 36. If the teacher explicitly develops learners’ concepts about print and printed material during the course of the lesson, does he/she do any of the following: 1. Identify front/cover of a book Tick relevant boxes 2. Identify the spine of a book 3. Identify the title / author / illustrator 4. Identify headings/sub-headings 5. Identify the beginning/end of sentences 6. Show that print moves from left to right 7. Identify full stops/other punctuation 8. Identify capitals 9. Identify bottom/top of page or picture 10. Identify page numbers 11. Show that pages turn from right to left 37. When the teacher provides front of class explanations, instruction or in class discussion, to what extent does the teacher model the correct terminology in English (even if explanations include code-switching)? Write one code only 0= Not at all 1= Hardly at all 2= To some extent 3= To a large extent 98= N/A – teacher does not provide front of class instructions. 38. Does the teacher highlight or foreground the English terms for the concept (e.g. background, make meaning, illustrations)? Write one code only 0= Never 1= Seldom 2= Sometimes 3= Mostly 4= Always 98= N/A – teacher does not ask questions. RSP Classroom Observation Tool Page 5 of 9 39. Does the teacher encourage learners to use English terms when responding? Write one code only 0= Never 1= Seldom 2= Sometimes 3= Mostly 4= Always 98= N/A – teacher does not ask questions. Section C: Literacy and Language Development 40. When learners are reading extended text aloud (sentences or paragraphs including in textbook text or stories), does the teacher provide learners with any of the following strategies for self-correcting: 1. sound out a word 1= Yes 2= No 98= N/A learners do not read text aloud 99= Don’t know 2. guess a word 3. use words similar to words that are already known, to work out what the word could be 4. skip or read past difficult words and then go back and use the context/meaning in which a word is embedded to work out what the word could be 41. When any extended texts (sentences or paragraphs including in textbook text or stories) are read or told, are learners encouraged and required to …. 1. Retell, act out or summarise what they have read or what has been read to them (e.g. the story or plot)? For example, repeat a sequence of events in the story correctly. Write a code for each 1= Yes 2= No 98= N/A - no extended texts read 99= Don’t know. Unable to tell 2. Discuss or respond to pictures or illustrations? 3. Use pictures or illustrations as clues to aid understanding the text? 4. Identify and/or re-state the main theme/idea/principle? 5. Answer open-ended questions about what they have read or what has been read to them and give reasons for their answers (e.g. ‘Do you think she was right to…?’ and ‘Why do you think that?’) 6. Answer predictive/inferential questions (e.g., questions about what will happen next before the next part of the text is read; or ‘what would have happened if…? writing their own version of how they think a story will end, etc.)? 7. Answer literal, evaluative, appreciative, or critical questions? Section D: Vocabulary and spelling development 42. Does any vocabulary development (learning new words and meanings, learning synonyms and antonyms) take place during the observation period? (e.g. learners making lists or personal ‘dictionaries’ or ‘word banks’ with new words) 1= Yes 2= No 99= Don’t know 43. Is there any evidence of spelling development or testing of spelling during the lesson observation? 1= Yes 2= No 99= Don’t know Section E: Opportunities to write 44. Do learners complete any writing/written tasks (not drawing) at all during the observation period? Write one code 0= None of the learners 1= Less than a quarter of the class 2= Less than half but at least a quarter of the class 3= About half the class 4= More than half the class 5= All/most learners RSP Classroom Observation Tool Page 6 of 9 45. Does the teacher check that learners are … a) holding their pens/pencils and positioning their writing material 1= Yes 2= No 98= N/A – learners do not write during the observation 99= Don’t’ know. Can’t tell b) taking any corrective measures when learners are not doing this correctly? 2. writing work in exercise books/jotters? 3. writing work in pre-printed workbooks? 46. Which of the following writing demands are made on all or most learners individually in the lesson? 1. Colouring in Tick relevant boxes 2. Drawing (own creation) 3. Copying a drawing/diagram (e.g. from the chalkboard, overhead, textbooks, etc.) 4. Drawing patterns in preparation for joined script or cursive writing 5. Copying vowels/letters/syllables 6. Writing vowels/letters/syllables (learners’ own productions without copying) 47. (cont.) 7. Copying words(Tick one only) a) one or two individual words b) three to ten individual words Tick relevant box c) more than ten individual words 8. Writing words (Tick one only) a) one or two individual words b) three to ten individual words Tick relevant box c) more than ten individual words 9. Copying sentences (Tick one only) a) one or two sentences b) three to five sentences Tick relevant box c) more than five sentences (but not paragraphs) 10. Writing sentences (Tick one only) a) one or two sentences b) three to five sentences Tick relevant box c) more than five sentences (but not paragraphs) 11. Copying paragraphs Tick relevant boxes 12. Writing paragraphs dictated by the teacher 13. Writing their own paragraphs (each unique – i.e. creating their own text) 48. How many different types of tasks/exercises involving writing text (can be letters, individual words, sentences, paragraphs but not drawing) does the teacher give learners to do during the lesson (including homework)? Write the number 0= None 98= N/A – no writing tasks provided 99= Don’t know – unable to ascertain this 49. Are all learners involved in writing any cursive text during the lesson? Write relevant codes 0= No 1= Write letters 2= Write words 3= Write phrases (less than 6 words) 4= Write sentences (More than 5 words) 5= Write a paragraph 99= Don’t know RSP Classroom Observation Tool Page 7 of 9 Section F: Opportunities to read 50. To what extent does the following range of reading opportunities occur for learners during the course of the observation (text can be letters, individual words, sentences, paragraphs)? (Includes text written on the chalkboard) 1. Teacher reading aloud to the class (class not following as they do not have or cannot see the text) Write one code for each 0= Never 1= Sometimes 2= Mostly 3= Always 2. Teacher leading reading with class following silently (they have or can see the text) 3. Whole class reading aloud together with the teacher 4. Whole class reading aloud together without the teacher 5. Learners reading aloud together in groups or pairs 6. Learners reading individually aloud to the class 7. Learners reading individually silently independently 8. Individual guided reading practise (where a learner gets a turn to read aloud so that the teacher can monitor their reading, give comment, make corrections and ask questions) 51. When learners are reading aloud, do they seem to be 1. … repeating the text off by heart (without actually engaging in reading)? Write a code for each 0= None of the learners 1= Less than a quarter of the class 2= Less than half but at least a quarter of the class 3= About half the class 4= More than half the class 5= All of the learners 98= N/A. No reading aloud 2. … simply repeating what is read by the teacher or by other learners who can read (without actually engaging in reading)? 52. If learners are split into groups for reading, do different groups read graded readers of different ability levels? 1= Yes 2= No 98= N/A. Learners are not split into groups 99= Don’t know 53. If learners are grouped for reading during the lesson a) Are there roughly the same number of learners in each group? 1= Yes 2= No 98= N/A. Learners are not grouped for reading b) If yes, how many learners are in each group on average Write the number 98= N/A. Learners are not grouped for reading c) If no, how many learners are in the largest group? Write the number 98= N/A. Learners are not grouped for reading 54. Per the lesson plan instructions, does the teacher ask the after reading questions (during shared reading)? Write one code 1= Yes 2= No 98= N/A. No shared reading Section F: Assessment 55. When learners are given written tasks or exercises, is work marked or checked individually (including individual group work) by … 1. the teacher Write one code for each 0= Never 1= Hardly ever 2= Sometimes 3= Mostly 4= Always 98= N/A – no/hardly any written tasks given 2. learners mark their own work? 3. learners mark each other’s work? (Note: Peer and self-assessment is allowed as long as the assessment criteria is explained clearly to learners) RSP Classroom Observation Tool Page 8 of 9 Section G: Homework 56. Is there any evidence of the teacher … 1. giving the class any reading homework? Write one code for each 1= Yes 2= No 2. giving the class any writing homework? (i.e. writing letters, words, phrases or sentences/ paragraphs, not drawing)? 3. giving the class any spelling homework? 4. checking whether homework was completed? 5. collecting and keeping homework to mark later? Section H: Classroom Library Write one code for each 1 = Yes 2 = No 98 = N/A. No classroom book collection/ box library available 99 = Don’t know/unable to ascertain this 57. Is there any evidence of the teacher … 1. Referring to the book collection/reading corner/box library or equivalent? 2. Administering books to learners (i.e. checking out books) 3. Retrieving books from learners (i.e. checking in books) 4. Using the “five finger rule” to determine what level book the learner should be reading 58. Fieldworker: Compare the work actually covered in the lesson observed with the work outlined in the work plan for this time of the term/year a) Was the content of the lesson/s observed in line with the work planned or indicated for the particular week in the term/year plan or series of lesson plans? 1= Yes 2= No 98= N/A. No plan made available 99= Don’t know. Unable to tell e.g. days not indicated on plan or plan ‘too general’ b) Was the content of the lesson/s observed in line with the work planned or indicated for the particular day in the term/year plan or series of lesson plans? 1= Yes 2= No 98= N/A. No plan made available 99= Don’t know. Unable to tell e.g. days not indicated on plan or plan ‘too general’ c) Did the teacher cover the entire lesson plan? Note: Please check lesson plan 1= Yes 2= No 98= N/A. No plan made available d) Did the teacher use the correct texts for the lesson? Note: Please check lesson plan 1= Yes 2= No 98= N/A. No plan made available RSP Classroom Observation Tool Page 9 of 9 Further comments If you have any additional comments you feel you should make about the lesson observed or about the observation schedule, write them in the space below RSP classroom walk-through checklist Page 1 of 4 RSP CLASSROOM WALK-THROUGH CHECKLIST 1. Identifying information: 1.1. District name: ________________________________________________ 1.2. School name: ________________________________________________ 1.3. School EMIS number: __________________________________________ 1.4. Teacher’s name: ___________________________________________________ 1.5. Teacher’s surname:____________________________ 1.6. Teacher’s PERSAL number:______________________ 1.7. Grade in which classroom walk-through observed: Grade 1 □ Grade 2 □ Grade 3 □ 2. Observation of classroom facilities and materials 2.1. Please rate the existence, sufficiency, and quality of the classroom infrastructure, facilities, and materials as indicated in the table. (Data collector inserts an “X” in each row. The rating key appears in the table.) Facility Key         Classroom reading corner (i.e. corner where books are stored/accessible for learners) No reading corner Small reading corner, not many books Average reading corner, but quantity and quality of books could be improved Classroom reading corner exists and is well stocked 1 2 3 4 Please take photo of reading corner RSP Big Books EFAL* None 0-10 10-20 20+ 1 2 3 4 Graded Readers EFAL (New Heights / AWEH! Readers) Note: Count all graded readers in the teacher’s possession None 0-10 10-20 20+ 1 2 3 4 Graded Readers HL (Vula Bula) Note: Count all graded readers in the teacher’s possession None 0-10 10-20 20+ 1 2 3 4 RSP classroom walk-through checklist Page 2 of 4 RSP posters (EFAL) None Poster(s) not on display Posters on display outdated/not relevant Posters on display up-to-date, relevant 1 2 3 4 RSP posters (HL) None Poster(s) not on display Posters on display, outdated/not relevant Posters on display up-to-date, relevant 1 2 3 4 Visibility of RSP posters NA Few learners can see them well About half the learners can see them well All learners can see them well 1 2 3 4 Please take photo of posters RSP Theme Vocabulary Flashcards displayed (e.g. on classroom walls) None Flash cards not on display Flashcards on display, outdated/not relevant Flashcards on display up-to-date, relevant 1 2 3 4 RSP Sight Words Flashcards displayed (e.g. on classroom walls) EFAL None Flash cards not on display Flashcards on display, outdated/not relevant Flashcards on display up-to-date, relevant 1 2 3 4 RSP Sight Words Flashcards displayed (e.g. on classroom walls) HL None Flash cards not on display Flashcards on display, outdated/not relevant Flashcards on display up-to-date, relevant 1 2 3 4 RSP Consolidated Words Flashcards displayed (e.g. on classroom walls) EFAL None Flash cards not on display Flashcards on display, outdated/not relevant Flashcards on display up-to-date, relevant 1 2 3 4 RSP Consolidated Words Flashcards displayed (e.g. on classroom walls) HL None Flash cards not on display Flashcards on display, outdated/not relevant Flashcards on display up-to-date, relevant 1 2 3 4 Visibility of flash cards NA Few learners can see them well About half the learners can see them well All learners can see them well 1 2 3 4 Please take photo of flash cards Assessment Record Book (CAPS & SA￾SAMS compliant assessment record tables) Note: these are termly records so records should be filled in up to the date of this visit None Not filled in Partially filled in with formal assessments of some learners recorded based on checklists/ rubrics in lesson plans Fully filled in with formal and informal assessments of all learners recorded based on checklists/ rubrics in lesson plans 1 2 3 4 RSP classroom walk-through checklist Page 3 of 4 Besides the above, Teachers are expected to have checklists with learners names and record learners’ performance continuously. It might be good to assess these rather than the SA SAMS. Teacher Management and Storage Files None Files do not contain any resources Files contain few resources Files contain many resources 1 2 3 4 Alphabet Frieze (EFAL)* None Alphabet frieze not on display N/A Alphabet frieze on display 1 2 3 4 Please take photo of the alphabet frieze Please check the items below in the Termly Lesson Plans RSP Lesson plans EFAL No lesson plan available Have a lesson plan, but not RSP lesson plans RSP lesson plan but outdated/not relevant RSP lesson plan, up-to-date, relevant 1 2 3 4 RSP Lesson plans HL No lesson plan available Have a lesson plan, but not RSP lesson plans RSP lesson plan but outdated/not relevant RSP lesson plan, up-to-date, relevant 1 2 3 4 Assessment tasks with rubrics/ mark sheet available Note: review the lesson plan from the week prior to this visit None Not filled in Partially filled in with some ratings for learner tasks Fully filled in with all ratings for learner tasks 1 2 3 4 Weekly reflection and weekly tracker None Not filled in Partially filled in - teacher records some lessons completed / some notes on curriculum coverage / some catch up plans Fully filled in - teacher records all lessons completed, records curriculum coverage, and develops catch up plans 1 2 3 4 Timetable incorporating weekly routine None Present but not RSP weekly routine Present but outdated/ not relevant Present, up-to-date, relevant 1 2 3 4 RSP classroom walk-through checklist Page 4 of 4 Curriculum Tracker None Curriculum tracker in the lesson plan, but clearly not used N/A Curriculum tracker in the lesson plan and clearly used 1 2 3 4 Please take photo of the lesson plans Please take photos of the four walls: Wall 1 Wall 2 Wall 3 *Not required for Grade 3 Any further comments? RSP Learner Workbook and Exercise Book Assessment Page 1 of 2 RSP LEARNER WORKBOOK AND EXERCISE BOOK(S) ASSESSMENT 1. Instructions and identifying information Instructions:  Fieldworkers must get the DBE workbook and the Learner Exercise Book/s: HL EFAL DBE Workbook Exercise Books DBE Workbooks Exercise books Grade 1 1 All (+/- 3) 1 All (+/- 3) Grade 3 1 All (+/- 3) 1 All (+/- 3)  For the DBE Workbook: the period under review is Beginning Term 3 (9 July 2019) up to 20 September 2019  For the Learner Exercise books the period under review Beginning of the year (9 January 2019) to 20 September 2019 (all exercise books needed except the test book) Ask the teacher to select one of the most proficient learners in her own class. Then request to see that learner’s DBE workbook/s, which will have all the work that the learner has done from the beginning of Term 3 up to 20 September 2019 in them, and all of the learner’s exercise books, which will have the work that the learner has done since the beginning of the year to 20 September 2019 (all exercise books needed except the test book). 1.1 District:__________________ 1.2 School Name:______________________ 1.3 School EMIS number:_______________________ 1.4 Grade 1.5. Teacher name and surname:_____________________________ 1.6. Teacher PERSAL number:_____________________________________ 1.8 Subject 1.9 The name and surname of this learner is _______________________________________ 2. DBE workbook assessment. Please count and record the following work that appears in this learner’s DBE workbook from the beginning of Term 3 until 30 August 2019. Grade 1 1 Grade 3 2 Home Language 1 EFAL 2 Learner Workbook and Exercise Book Assessment Page 2 of 2 Type of content counted Nr (i) The overall number of pages on which the learner completed any exercises (ii) The overall number of pages students had marked their own exercises (in pencil) (iii) The overall number of pages the teacher had marked any exercises (usually in pen) (iv) The number of pages involving writing* exercises completed by the learner (v) The number of pages involving exercises comprising the writing of at least one full sentence (vi) The number of pages involving exercises comprising the writing of at least one paragraph * This is when the learner wrote at least one letter in an exercise. 3. Learner exercise book(s) assessment Please count and record the following work that appears in this learner’s exercise book/s from the beginning of the year until 30 August 2019. Please ask for all books in case the learner has started working in a new book sometime during the year. Use the dates provided in the books as a reference for the number of days. Type of content counted Nr (i) The overall number of days on which the learner completed any exercises (ii) The overall number of days students had marked their own exercises (in pencil) (iii) The overall number of days the teacher had marked any exercises (usually in pen) (iv) The number of days involving writing* exercises completed by the learner (v) The number of days involving exercises comprising the writing of at least one full sentence. (vi) The number of days involving exercises comprising the writing of at least one paragraph. * This is when the learner wrote at least one letter in an exercise School DQA Tool Page 1 of 2 RSP SCHOOL DATA QUALITY ASSESSMENT (DQA) TOOL 1. Identifying information: 1.1. District name: ________________________________________________ 1.2. School name: ________________________________________________ 1.3. School EMIS number: _________________________________________ 2. Teacher Training and LTSM NOTE: ASK THE SCHOOL PRINCIPAL THE FOLLOWING: Question Answer 2.1 How many grade 1, 2, and 3 teachers does the school currently have? 2.1.1 Number Grade 1 teachers: __________ 2.1.2 Number Grade 2 teachers: __________ 2.1.3 Number Grade 3 teachers: __________ 2.2 How many grade 1, 2, and 3 vacancies does the school currently have? 2.2.1 Number Grade 1 vacancies: __________ 2.2.2 Number Grade 2 vacancies: __________ 2.2.3 Number Grade 3 vacancies: __________ 2.3 Did the Provincial department communicate the training dates to your school via a circular? 2.3.1 Yes 1 No 2 2.4 Were the RSP teacher training dates for the year shared with Principals (via SMS and/or email) and teachers during the first school term? 2.4.1 Yes 1 No 2 2.5 Between Jan and Oct 2019, how many teacher training sessions did the schools’ Foundation Phase teachers attend? 2.5.1 Number of training sessions attended by Foundation Phase teachers: ___________ 2.6 Where teachers did not attend RSP teacher training was any catch-up training and LTSM delivered to these teachers? 2.6.1 Yes 1 No 2 2.7 Between Jan and Oct 2019, how many times did a coach visit the school? 2.7.1 Number of coach visits: ___________ School DQA Tool Page 2 of 2 2.8 For the fourth term, are there any Foundation Phase teachers at the school who did not get their RSP LTSM and Lesson plans? If yes, how many? 2.8.1 Number Foundation Phase teachers without LTSM for Term 4: ___________ 2.9 Between Jan and Oct 2019, if LTSM was not delivered to the schools at any point, did the school arrange for LTSM to be picked up from the warehouse? 2.9.1 Yes 1 No 2 Not Applicable 3 2.10 Between Jan and Oct 2019, how many school-based afternoon workshops were held by an RSP coach? 2.10.1 Number of school based afternoon workshops: ___________ 2.11 Did any teachers at the school receive a classroom library from RSP? If yes, how many? 2.11.1 Number Grade 1 teachers: __________ 2.11.2 Number Grade 2 teachers: __________ 2.11.3 Number Grade 3 teachers: __________ 3. Data Verification 3.1 Please ask to see the school asset register. How many RSP tablets are recorded (if the SMT received tablets, these should be recorded on the register)? 3.1.1 Number of RSP tablets recorded on the asset register: _____________________________ 3.2 For schools that have RSP Literacy Coaches, please ask the principal for the name of the Literacy Coach. Please ask to see the school logbook. Between Jan and Oct 2019, how many RSP Literacy Coach visits are recorded? 3.2.1 Number of Literacy Coach visits recorded in the school log: __________________________ 3.3 For schools that have RSP coaches, please ask the principal for the name of the Head Literacy Coach to whom the Literacy Coach reports. Please ask to see the school logbook. Between Jan and Oct 2019, how many Head Literacy Coach visits are recorded? 3.3.1 Number of Head Literacy Coach visits recorded in the school log: ______________________ Any further comments? RSP SMT Interview Guide Page 1 of 12 RSP SCHOOL MANAGEMENT TEAM INTERVIEW GUIDE Demographics Researcher Name: School name: SMT member(s) present: Date: Total number of participants (excl. researcher) Position(s) (e.g. Principal, Deputy Principal, HOD) Instructions 1. Introduce yourself: Hello! I am ________________________. I am working on a data collection project undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). 2. Circulate register and informed consent form with SMT participant(s) 3. Read out below consent information: We are asking you, as SMT members involved in the Reading Support Project, to provide your honest feedback on your experience of the project so far. This discussion will take around 30 minutes. Please understand that your participation is voluntary and that you are not being forced to take part. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the discussion at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. Do you agree to participate in this discussion? Yes/No For note taking purposes, this group discussion will be recorded. Everything you share will be kept confidential and will not be used against you. Do you consent to audio recording this interview? Yes/No Thank you, we will now begin. BEFORE YOU BEGIN 1. Set Otter.ai for recording and start recording RSP SMT Interview Guide Page 2 of 12 2. Follow the discussion questions on this guide 3. At the end, thank participants:  That concludes our discussion. Thank you so much for coming and sharing your thoughts and opinions with us. Your contribution will help us improve the intervention going forward and understand how the intervention is being implemented. 4. Collect register and consent forms. Before leaving, ensure that everyone has signed both documents. QUESTIONS When answering each of these questions, please pay attention to: 1. Whether there was consensus, partial agreement, widely divergent opinions 2. Any other notable points 1. SMT Training QUESTION PROBE 1.1 Has the School Management Team received any in-service training on school leadership and / or school management in 2019? If so, how valuable was the content of the training workshop(s)? - Who provided the training? - Who from the SMT attended? - Did they learn anything new from the workshop(s)? - Quantity of the training (enough sessions? Long enough? Too many / too long?) - Quality of the training (trainers well prepared? Adequacy of the content?) Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: RSP SMT Interview Guide Page 3 of 12 QUESTION PROBE ( Note: ask only If the SMT received any external training on school leadership and/or school management - not just RSP training) 1.2 Is the SMT doing anything differently in the school as a result? If, so, what are you doing differently? - Does the SMT encourage reporting on SA-SAMS in the school? - Do teachers in the school receive supervisory support from the SMT? - Does the SMT identify and respond to bottlenecks and other issues? - Does the SMT hold regular meetings/updates in the school to increase accountability? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: RSP SMT Interview Guide Page 4 of 12 QUESTION PROBE 1.3. Has the School Management Team received any in-service training on supporting literacy development in the school in 2019? If so, how valuable was the content of the training workshops - Who offered the training? - Who from the SMT attended? - Did they learn anything new from the workshop(s)? - Quantity of the training (enough sessions? Long enough? Too many / too long?) - Quality of the training (trainers well prepared? Adequacy of the content?) Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) RSP SMT Interview Guide Page 5 of 12 Any other notable points? (please tick one) Yes No If yes, comments: QUESTION PROBE ( Note: ask only If the SMT received any external training on supporting literacy development in the school – not just RSP training) 1.3 Is the SMT doing anything differently in the school as a result? If, so, what are you doing differently? - Does the SMT support book clubs at the school? If not, why not? - Does the SMT support a culture of reading in the school? If so, how? - Is the SMT more likely to observe HL/ EFAL lessons as a result of the training? - Does the SMT meet more regularly to prepare for language lessons? - Is the SMT more likely to get support on the aspects of the language lessons that they struggle with? - Is the SMT more likely to ask questions to their SMT members if they struggle? Notes: RSP SMT Interview Guide Page 6 of 12 Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: 2. Professional Learning Communities QUESTION PROBE 2.1 Have you established a Professional Learning Community at your school? If so, how does it work? - How often do they meet? - What topics do they cover? - Is it virtual (E.g. WhatsApp) or face-to￾face? - Is the support that the SMT receive useful? - When did you establish it and why? Notes: RSP SMT Interview Guide Page 7 of 12 Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: 3. Coaching* * Only ask in RSP coaching support schools. For other schools, note “not applicable” in the notes. If the SMT is unaware of the RSP coaches in the school, note this in the notes section. QUESTION PROBE 3.1 Do you feel Foundation Phase teaching practice has changed as a result of teachers’ interactions with their literacy coach and in what ways? - Quantity of coaching visits? (more than enough, enough, need more?) - Quality of coaching visits? - Quality and relevance of the feedback coaches provide to teachers? - Are teachers doing anything differently in the classroom and, if so, what? - Would you recommend literacy coaches in schools and why / why not? - Do they have a good relationship with the coach allocated to their school? RSP SMT Interview Guide Page 8 of 12 - Do they consider the coach to be knowledgeable? - Do they trust the coach? - Are the coaches reliable – i.e. they come when they said they will come. Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: QUESTION PROBE 3.2 Has the RSP literacy coach held any school-based workshops in the afternoons that you are aware of, do you feel these are valuable and why/why not? - Relevance? Comprehensiveness? - Topics covered? - Aimed at individual Grades or cross￾Grade? RSP SMT Interview Guide Page 9 of 12 Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: RSP SMT Interview Guide Page 10 of 12 QUESTION PROBE 3.3 If you could change one thing about the RSP literacy coaching, what would that be and why? - Content? Quantity? Quality? Other? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: RSP SMT Interview Guide Page 11 of 12 4. LTSM** ** Ask if at least one SMT member received training through the RSP QUESTION PROBE Did you receive tablets at the RSP training session with the training programme and RSP Learning and Teaching Support Materials pre-loaded? If so, do you use these LTSM and in what ways? Note: This will be specific to DHs because Principals did not receive LTSM in their tablets - Do they access and use the online / offline learning platform and, if so, how? - Has the L TSM for Term 4 been uploaded on their tablets? - Do they use it for anything else? - How regularly do they use the tablets? - What has been most useful about having the tablet? - What has been problematic about using the tablet? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group N/A (there was only one respondent) Any other notable points? (please tick one) Yes No If yes, comments: RSP SMT Interview Guide Page 12 of 12 5. QUESTIONS AT THE END OF THE INTERVIEW (ONLY ANSWER IF SMT RECEIVED RSP TRAINING) 5.1 Did the SMT receive notice of the SMT training activity at least two weeks prior to actual training date either via SMS and/or emails? 5.2 Did a Voluntary Services Organization (VSO) volunteer contact the SMT member by phone, or WhatsApp to set up an appointment? 5.3 From January to October 2019, how many school support visits were conducted by RSP? ( on￾site visit to schools conducted by volunteers for participants of the SMT for one-on-one capacity building) 5.3.1 Number of school support visits: __________ Please provide feedback on the support provided during these visits in the questions below: 5.4 What is the content and topic of the support provided? 5.5 What is the quality of the support provided? 5.6 Did the volunteer understand and relate to the SA education context in South Africa? Any other notes on the support provided by volunteers? Yes 1 No 2 Yes 1 No 2 Good 1 Adequate 2 Poor 3 Yes 1 No 2 Information Sheet and Consent Form for School Principal / SMT member Page 1 of 3 READING SUPPORT PROJECT Information Sheet and Consent Form for School Management Team WHO WE ARE Hello! I am ________________________. I am working on an evaluation undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. YOUR PARTICIPATION We are asking you as a school principal/ SMT member to provide some background information that may underpin the teaching of reading in your school by participating in a group interview. The Information Sheet and Consent Form for SMT member Page 2 of 3 interview should not take longer than 20 minutes to complete. The questions are mainly about conditions and activities that may enhance or hinder the teaching of reading and language at the school and your experience participating in the RSP. We will also administer the intended evaluation instruments with some teachers from your school, including observing a HL and EFAL classroom in Grade 1 and in Grade 3. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. CONFIDENTIALITY Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office, and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. The records from your participation that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. RISKS/DISCOMFORTS We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not to other Department officials at the circuit, district, provincial, or national offices. BENEFITS The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members, and teachers will never be attached to any of the information that you provide or the information released through reports or articles. WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Katharine Tjasink at Tel: 011 447 6464 Ext 3207, or e-mail at ktjasink@khulisa.com. You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. Information Sheet and Consent Form for School Principal / SMT member Page 3 of 3 Instruction for researcher: Please hand out this sheet when conducting SMT group interviews. Collect at the end of interview. By signing this sheet I hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I understand that I am participating freely and without being forced in any way to do so. I also understand that I can stop participating at any point should I not want to continue and that this decision will not in any way affect me negatively. I understand that my participation will remain confidential. School name:__________________________________________________________ (researcher to write down school name) No. Name Surname ID number / Passport number Position Signature 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Information Sheet and Consent Form for School Principal / SMT member Page 1 of 3 READING SUPPORT PROJECT Information Sheet and Consent Form for School Principal WHO WE ARE Hello! I am ________________________. I am working on an evaluation undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. YOUR PARTICIPATION We are asking you as a school principal to provide some background information that may underpin the teaching of reading in your school by participating in a group interview. The interview should not take longer than 20 minutes to complete. The questions are mainly about conditions and activities that may enhance or hinder the teaching of reading and language at the school and your experience participating in the RSP. With your permission, we will also administer the intended evaluation Information Sheet and Consent Form for School Principal / SMT member Page 2 of 3 instruments with some teachers from your school, including observing a HL and EFAL classroom in Grade 1 and in Grade 3. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. CONFIDENTIALITY Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office, and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. The records from your participation that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. RISKS/DISCOMFORTS We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not to other Department officials at the circuit, district, provincial, or national offices. BENEFITS The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members, and teachers will never be attached to any of the information that you provide or the information released through reports or articles. WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Katharine Tjasink at Tel: 011 447 6464 Ext 3207, or e-mail at ktjasink@khulisa.com. You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. Information Sheet and Consent Form for School Principal / SMT member Page 3 of 3 CONSENT I ________________________________ hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I understand that I am participating freely and without being forced in any way to do so. I also understand that I can stop participating at any point should I not want to continue and that this decision will not in any way affect me negatively. I understand that my participation will remain confidential. I hereby acknowledge that teachers and SMT members from my school will be approached, as explained to me, to participate in evaluating the RSP. I understand that the information that I provide will be stored electronically and that evaluation findings will be communicated to senior DBE managers and through reports and academic articles without making known my identity or that of the school, its teachers, and SMT members. ____________________________ Signature of School Principal ____________________________ Date ____________________________ Additional signature, if required ____________________________ Date (School Management Team member or Personal Assistant) (if required, specify other designations) ____________________________________________________________________________ Semi-Structured Coaching Observation Tool Page 1 of 9 RSP TEACHER STRUCTURED GROUP DISCUSSION GUIDE Group Discussion Details RESEARCHER: SCHOOL NAME: NO. OF TEACHERS PRESENT GRADES REPRESENTED (select all that apply): 1, 2, 3 DATE: Instructions 1. Introduce yourself: Hello! I am ________________________. I am working on a data collection project undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). 2. Circulate register and informed consent form with teachers 3. Read out below consent information: You have been selected to participate in this group discussion because your school is participating in the Reading Support Project. As part of this intervention, you have received LTSM, attended training, and may have also received visits from a literacy coach. Khulisa has been contracted to evaluate the implementation of the project. The evaluation will tell us how the project is being implemented, what works well, and what needs improvement. We are asking you, as Foundation Phase literacy teachers involved in the Reading Support Project, to provide your honest feedback on your experience of the project so far. This group discussion will take around 1 hour. We would encourage everyone to participate and will ensure that you are given a chance to speak. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the discussion at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. Do you agree to participate in this group discussion? Yes/No For note taking purposes, this group discussion will be recorded. Everything you share will be kept confidential and will not be used against you. Do you consent to audio recording this interview? Yes/No Teacher Structured Group Discussion Guide Page 2 of 9 A few ground rules before we start: 1. Stay with the group and please don’t have side conversations 2. Turn off cell phones if possible 3. Please keep questions to the end of the discussion – we have set aside time to address your questions. Thank you, we will now begin. BEFORE YOU BEGIN 1. Set Otter.ai for recording and start recording 2. Follow the discussion questions on this guide 3. At the end, thank group discussion participants:  That concludes our group discussion. Thank you so much for coming and sharing your thoughts and opinions with us. Your contribution will help us improve the intervention going forward and understand how the intervention is being implemented. 4. Collect register and consent forms. Before leaving, ensure that everyone has signed both documents. QUESTIONS When answering each of these questions, please pay attention to: 1. Whether there was consensus, partial agreement, widely divergent opinions 2. Differences by Grade 3. Differences by Age 4. Any other notable points QUESTION PROBE 1. What do you like most and what do you like least about being part of the Reading Support Project? - The responses may be positive or negative. Probe for what the group thinks are the key challenge(s) or enablers Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 3 of 9 Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: LTSM QUESTION PROBE 2. Do the Learning and Teaching Support Materials provided by the RSP (Big Books, Lesson Plans, etc.) help you in your literacy teaching and are there any issues with these materials that you would like to note? - Did they receive all the materials? - Do they receive materials in time? - What is the quality of the materials? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 4 of 9 Teacher training QUESTION PROBE 3. Do you feel you received appropriate training to use the RSP LTSM and if there is anything you could change, what would that be? - Usefulness of the training sessions? - Quantity of the training (enough sessions? Long enough?) - Quality of the training (coaches well prepared? Adequacy of the content?) Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 5 of 9 Coaching* * Only ask in coaching support schools. For other schools, not “not applicable” in the notes QUESTION PROBE 4. Have your personal (i.e. one-on-one) interactions with the RSP literacy coach been helpful? Why or why not? - Quantity of coaching visits? (more than enough, enough, need more?) - Quality of coaching visits? - Primary focus of interactions with the coach? - Quality and relevance of the feedback they provide? - Would you recommend literacy coaches in schools and why? - What is most helpful/valuable in terms of: o Pre-observation discussion o Lesson Observation o Feedback provided by coach in post-observation session o School-based Workshops Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 6 of 9 QUESTION PROBE 5. Has your coach held any school-based workshops in the afternoons, are these valuable and why/why not? - Quantity of workshops? - Quality of workshops? (Relevance? Comprehensiveness?) - Topics covered? - Aimed at individual Grades or cross￾Grade? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: QUESTION PROBE 6. What is your relationship with the coach like – do you feel comfortable discussing your strengths and weaknesses? - What do you like the most about your coach? - How are your interactions with the coach the same/different than interactions you typically have with the principal? With other teachers in your grade/phase? With Subject Advisors? - What is your role when working with the coach? Notes: Teacher Structured Group Discussion Guide Page 7 of 9 Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: QUESTION PROBE 7. Has your teaching practice changed as a result of your interactions with your coach and in what ways? - Do you do anything differently in your classroom? - What aspect of coaching has had the greatest impact on your teaching practice? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 8 of 9 QUESTION PROBE 8. If you could change one thing about the RSP coaching, what would that be and why? - Quantity of coaching visits? - Quality of coaching visits? - Relevance of the feedback they provide? Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Teacher Structured Group Discussion Guide Page 9 of 9 School Management Team QUESTION PROBE 9. Do you receive support from your School Management Team, in general, and in reading and literacy in particular? - Does the SMT encourage reporting on SA-SAMS in the school? - Does the SMT identify and respond to bottlenecks and other issues? - Does the SMT support book clubs at the school? - Does the SMT support a culture of reading in the school? - Does the teacher receive supervisory support from the SMT? - Does the SMT support teachers in the coaching process and ensure they have the space and time available for observations, individual feedback and afternoon workshops (only if applicable) Notes: Was there (please tick one): Consensus in the group Partial agreement in the group Widely divergent opinions in the group Were there differences by Grade? (please tick one) Yes No If yes, comments: Were there differences by Age? (please tick one) Yes No If yes, comments: Any other notable points? (please tick one) Yes No If yes, comments: Information Sheet and Consent Form for Teachers Page 1 of 4 READING SUPPORT PROJECT Information Sheet and Consent Form for Teacher Group Discussions WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. YOUR PARTICIPATION We are asking you, as a language teacher, to participate in a group discussion where you will provide your feedback on your experience with the RSP. The questions mainly cover teaching practices and your experience with the intervention. The group discussion will take no longer than an hour. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you Information Sheet and Consent Form for Teachers Page 2 of 4 choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. CONFIDENTIALITY Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. Records that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. RISKS/DISCOMFORTS We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially and anonymously, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not other Department officials at the circuit, district, provincial, or national offices. BENEFITS The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members and teachers will never be attached to any of the information that you provide or the information released through reports or articles. WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Katharine Tjasink at Tel: 011 447 6464 Ext 3207, or e-mail at ktjasink@khulisa.com. You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. Information Sheet and Consent Form for Teachers Page 3 of 4 By signing this sheet, I hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I understand that I am participating freely and without being forced in any way to do so. I also understand that I can stop participating at any point should I not want to continue and that this decision will not in any way affect me negatively. I understand that my participation will remain confidential. School name:__________________________________________________________ (researcher to write down school name) No. Name Surname ID number / Passport number Grade Language taught (HL or EFAL or Both) Signature Information Sheet and Consent Form for Teachers Page 4 of 4 No. Name Surname ID number / Passport number Grade Language taught (HL or EFAL or Both) Signature RSP Teacher Survey Page 1 of 12 RSP TEACHER SURVEY GRADE 1-3 For Researcher only Researcher name and surname: ________________________________ 1. Identifying information 1.1. School name: ________________________________________________ 1.2. School EMIS number:___________________ 1.3. District: _____________________________ 1.4. Teacher’s name: ___________________________________________________ 1.5. Teacher’s surname: ___________________________________________________ 1.6. Teacher PERSAL number:______________________________ 1.7. Teacher date of birth:_________ 1.8. Are you the Foundation Phase Head of Department (HOD)? Yes 1 No 2 1.9. What Grade do you teach? Grade 1 1 Grade 2 2 Grade 3 3 2. RSP Learning and Teaching Support Materials (LTSM) 2.1. RSP planned for each Foundation Phase teacher to get quarterly packs of Learning and Teaching Support Materials (LTSM) and Lesson Plans. 2.1.1.Did you receive these? 2.1.1.1. If No, please explain/specify 2.1.1.2. If yes, did you receive the RSP LTSM prior at the start of the current school term? Yes 1 No 2 Yes 1 No 2 RSP Teacher Survey Page 2 of 12 2.1.1.3. If yes, which of the following LTSM did you receive between January and October 2019? (Please select ALL RECEIVED) with an “X”) Scripted lesson plans (Termly Lesson Plans) HL - plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Scripted lesson plans (Termly Lesson Plans) EFAL- plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (HL) Theme Vocabulary Flashcards (EFAL) Sight Words Flashcards (HL) 1 2 3 4 5 Sight Words Flashcards (EFAL) RSP Big Books (EFAL) RSP Graded Readers (EFAL) Handwriting poster Posters - Listening and Speaking 6 7 8 9 10 Assessment Record Book Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term Teacher Management and Storage files USB Stick with Songs Used in Lesson Plans Classroom Library 11 12 13 14 15 RSP Teacher Survey Page 3 of 12 2.1.2.How often do you use EACH of the LTSM outlined below? (Please mark one with an “X”) Not Applicab le (I did not receive this LTSM) Never Approximately once a term Approximately once a month Approximately once a week Nearly every day Every day Scripted lesson plans (Termly Lesson Plans) HL 0 1 2 3 4 5 6 Scripted lesson plans (Termly Lesson Plans) EFAL 0 1 2 3 4 5 6 Theme Vocabulary Flashcards (HL) 0 1 2 3 4 5 6 Theme Vocabulary Flashcards (EFAL) 0 1 2 3 4 5 6 Sight Words Flashcards (HL) 0 1 2 3 4 5 6 Sight Words Flashcards (EFAL) 0 1 2 3 4 5 6 RSP Big Books (EFAL) 0 1 2 3 4 5 6 RSP Graded Readers (EFAL) 0 1 2 3 4 5 6 Handwriting poster 0 1 2 3 4 5 6 Posters - Listening and Speaking 0 1 2 3 4 5 6 Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term 0 1 2 3 4 5 6 USB Stick with Songs Used in Lesson Plans 0 1 2 3 4 5 6 RSP Teacher Survey Page 4 of 12 2.1.3.Which LTSM do you find the MOST USEFUL for Home Language (HL)? (Please select ONE with an “X”) Scripted lesson plans (Termly Lesson Plans) HL - plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (HL) Sight Words Flashcards (HL) 1 2 3 2.1.4.Which LTSM do you find the MOST USEFUL for English First Additional Language (EFAL)? (Please select ONE with an “X”) Scripted lesson plans (Termly Lesson Plans) EFAL￾plans for each day with core methodologies sections, Weekly routine, Weekly reflection section & Tracker, Assessment Tasks & task sheet Theme Vocabulary Flashcards (EFAL) Sight Words Flashcards (EFAL) RSP Big Books (EFAL) RSP Graded Readers (EFAL) 1 2 3 4 5 2.1.5.Which LTSM do you find the MOST USEFUL? (Please select ONE with an “X”) 2.2. If you have not received the LTSM, what are the reasons? Handwriting poster Posters - Listening and Speaking Assessment Record Book Curriculum Tracker/ Teacher Monitoring and Support Tool for the Term Teacher Management and Storage files 1 2 3 4 5 USB Stick with Songs Used in Lesson Plans Classroom Library 6 7 8 9 10 3. RSP Training 3.1. Between January and October 2019, did you attend training workshops on how to use the RSP lesson plans and other LTSM? Yes 1 No 2 RSP Teacher Survey Page 5 of 12 3.1.1. If yes, which RSP training workshops have you attended (Select all you attended)? *please note that Term 2 and Term 3 training were residential Term 1 Term 2 Term 3 Term 4 1 2 3 4 3.1.2 If you missed one or more of these training workshops, did you receive catch up training from your RSP coach? 3.1.2.1 Please select for which training workshops you received RSP coach catch up training: Term 1 Term 2 Term 3 Term 4 1 2 3 4 3.1.2.2 If yes, what was the quality of this catch up training? Yes 1 No 2 N/A 3 Good 1 Adequate 2 Poor 3 3.1.2 If you missed one or more of the training workshops, what was the reason? 3.1.3 Which RSP training workshop did you find most useful (Select one)? Term 1 Term 2 Term 3 Term 4 1 2 3 4 3.1.3.1 Why did you prefer this workshop over the others you attended? 3.1.3 How valuable did you find the content of the training workshops? Extremely valuable Valuable Somewhat valuable Not at all valuable 4 3 2 1 RSP Teacher Survey Page 6 of 12 3.1.4 Please rate how useful the following training workshop content elements were for Home Language (Please mark one with an “X”): Very useful, I learned a lot Somewhat useful, learned some new things Not useful to me. I already knew the content. But well delivered Not useful to me and needs improvement Weekly Routine 4 3 2 1 Phonics Lessons 4 3 2 1 Reading Lessons 4 3 2 1 Handwriting Lessons 4 3 2 1 Writing Lessons 4 3 2 1 Listening and Speaking Lessons 4 3 2 1 Assessment Tasks 4 3 2 1 3.1.5 Please rate how useful the following training workshop content elements were for EFAL: (Please mark one with an “X”) Very useful, I learned a lot Somewhat useful, learned some new things Not useful to me, but well delivered Not useful to me and needs improvement Weekly Routine 4 3 2 1 Phonics Lessons 4 3 2 1 Reading Lessons 4 3 2 1 Handwriting Lessons 4 3 2 1 Writing Lessons 4 3 2 1 Listening and Speaking Lessons 4 3 2 1 Assessment Tasks 4 3 2 1 3.1. Did you receive the RSP LTSM at the training? Yes 1 No 2 N/A (did not attend training) 3 RSP Teacher Survey Page 7 of 12 3.2.1 If no, how did you receive your LTSM? I had the LTSM delivered to my school My coach brought the LTSM for me when s/he came for a school support visit I received the LTSM from other teachers in my school who attended the training I did not receive any LTSM 1 2 3 4 Other (please specify):_____________________________ 3.2.2 If no, when did you receive the LTSM? A week after the training Two weeks after the training A month after the training Only at the next training session Never 1 2 3 4 5 Other (please specify):_____________________________ 3.2. Did you receive the LTSM in time for you to use in the classroom? Yes 1 No 2 4. RSP Coaching 4.1. Do you receive visits from an RSP reading coach? If yes, please respond to all questions on this next section. If no, skip to Section 5. RSP Classroom Library Yes 1 No 2 RSP Teacher Survey Page 8 of 12 4.1.1 If yes, please rate the statements below related to your experience with the coach (Select one option in each row): (Please mark one with an “X”) To a very large extent To a large extent Neutral/ Not Applicable To a limited extent Not at all My coach communicates the planned coaching visits with me 5 4 3 2 1 My coach communicates the planned coaching visits with me sufficiently in advance such that I am able to accommodate coaching visits in my schedule 5 4 3 2 1 My coach’s observations and feedback have helped me grow as a teacher 5 4 3 2 1 I have gained helpful resources from my coach 5 4 3 2 1 My coach communicates regularly with me 5 4 3 2 1 My coach maintains open, two-way communication with me 5 4 3 2 1 My coach helps me overcome barriers to teaching and learning 5 4 3 2 1 My coach holds reflective feedback sessions with me 5 4 3 2 1 My coach sets action plans after each feedback session 5 4 3 2 1 4.1.2 If no, please explain 4.2. How many visits have you received from your coach: 4.2.6.Since the beginning of the year:________ 4.2.7.In term 3 only:_________ RSP Teacher Survey Page 9 of 12 4.3. How many times did your coach observe your Home Language lessons: 4.3.6.Since the beginning of the year:________ 4.3.7.In term 3 only:_________ 4.4. How many times did your coach observe your English First Additional Language lessons: 4.4.6.Since the beginning of the year:________ 4.4.7.In term 3 only:_________ 4.5. How many times did your coach give you feedback on your lessons: 4.5.6.Since the beginning of the year:________ 4.5.7.In term 3 only:_________ 4.6. Have you received any written feedback forms from your coach? 4.6.6.If yes, how many? 4.6.6.1. Since the beginning of the year: __________ 4.6.6.2. In term 3 only: ___________________________ 4.7. Have you participated in any school-based afternoon workshops organised by your coach? Yes 1 No 2 Yes 1 No 2 4.6.1. If yes, how many have you attended: 4.7.6.1. Since the beginning of the year:________ 4.7.6.2. In term 3 only:_________ 4.6.2. If yes, do you feel you would benefit from more workshops like these? Yes 1 No 2 4.6.3. If yes, what is the content of these workshops? 4.7 Do you interact with your coach for support outside of school visits? Yes 1 No 2 N/A 3 RSP Teacher Survey Page 10 of 12 4.7.1 If yes, what platform do you mainly use to collaborate? If Other, specify_________________________________ 4.8 Were you and your school part of the Early Grade Reading Study (EGRS) previously? 4.8.1 If yes, did you also receive coaching support from coaches affiliated with the Early Grade Reading Study (2015 – 2017)? WhatsApp 1 Calls or text messages 2 Face to Face 3 O th er 4 Yes 1 No 2 I don’t know 3 Yes 1 No 2 4.8.2. If yes, how does your experience with the RSP reading coach compare with your experience with the EGRS coach? RSP literacy coach support is of a better quality than support provided through the EGRS coach RSP literacy coach support is of the same or similar quality to support provided through the EGRS coach RSP literacy coach support is of a poorer quality than support provided through the EGRS coach 3 2 1 5 RSP Classroom Library 5.1 Did you receive an RSP Classroom Library kit/box? If yes, please complete the following questions on this section. If no, skip to Section 6. School Management Team Yes 1 No 2 5.7.1 Were you trained on the management and use of the classroom library? 5.7.2 Were you orientated on the library starter-kit and shown examples of how to use each component? Yes 1 No 2 Yes 1 No 2 5.7.3 Do you feel you received enough support to use the classroom library? Yes 1 No 2 5.7.4 Do you use the classroom library kit/box? Yes 1 No 2 5.7.4.1 If no, why not? RSP Teacher Survey Page 11 of 12 5.7.4.2 If yes, how often do you make use of the classroom library? (Please mark one with an “X”) Once a month Every 2 weeks Once a week 2-4 times a week Everyday 1 2 3 4 5 5.7.5 How helpful do you find having a classroom library? (Please mark one with an “X”) Not at all helpful Somewhat helpful Extremely helpful 1 2 3 5.7.6 You received an A5 plastic sleeve as part of your classroom library kit/box. How useful do you find this A5 plastic sleeve? (Please mark one with an “X”) Not at all useful Somewhat useful Extremely useful 1 2 3 5.7.7 Why do you find the A5 plastic sleeve useful/ not useful? 5.7.8 Have you appointed selected learners as classroom library monitors? 5.7.9 If yes, what duties have they fulfilled? (Please select all that apply with an “X”) Manage the books Book box and chart management Other 1 2 3 If Other, please specify:_______________________________________ Yes 1 No 2 5.7.10 Do learners take books home? Yes 1 No 2 5.7.10.1 If no, why not? 5.7.11 Are learners interested in taking books home? (Please mark one with an “X”) None Few learners Some learners Most learners All learners 1 2 3 4 5 6 School Management Team 6.1 Do you receive support from your School Management Team? Yes 1 No 2 RSP Teacher Survey Page 12 of 12 6.1.1 If yes, please rate the statements below related to your general experience with the SMT i.e. the Principal, Deputy Principal, Foundation Phase Head of Department or Other SMT member (Select one option in each row): Not at all To a limited extent Neutral/ Don’t know To a large extent To a very large extent My SMT encourages reporting on SA-SAMS in the school 1 2 3 4 5 My SMT identifies and responds to bottlenecks and other issues 1 2 3 4 5 My SMT supports book clubs at the school 1 2 3 4 5 My SMT supports a culture of reading in the school 1 2 3 4 5 I receive supervisory support from my SMT 1 2 3 4 5 My SMT holds regular meetings/updates in my school to increase accountability 1 2 3 4 5 My SMT supports teachers in the coaching process and ensure they have the space and time available for observations, individual feedback and afternoon workshops 1 2 3 4 5 My SMT observes my lessons 1 2 3 4 5 My SMT models any lessons I struggle to teach 1 2 3 4 5 My SMT has encouraged or praised me for a lesson well taught 1 2 3 4 5 7 RSP experience 7.1. On a scale from 1-10 (1 being least likely and 10 being most likely) how likely are you to recommend the RSP to a friend, colleague or others? 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 7.2 If you could change one thing about the RSP what would it be and why? Thank you for your time and effort! Information Sheet and Consent Form for Teachers Page 1 of 5 READING SUPPORT PROJECT Information Sheet and Consent Form for Teachers WHO WE ARE Hello! I am ________________________. I am working on an evaluation undertaken by Khulisa Management Services (Khulisa) and funded by the United States Agency for International Development (USAID) in partnership with the South African Department of Basic Education (DBE). WHAT WE ARE DOING To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified their existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned an implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). Khulisa is conducting the implementation evaluation of the RSP. The DBE has informed sampled schools of their role and participation in the evaluation. In sampled schools, the principal, SMT members, and a selection of Grades 1 - 3 teachers of Setswana Home Language (HL) and English First Additional Language (EFAL) will participate in the data collection. Information Sheet and Consent Form for Teachers Page 2 of 5 YOUR PARTICIPATION We are asking you, as a language teacher, to complete a questionnaire where you will provide some information on how you teach reading in your class, as well as your feedback on your experience with the RSP by completing a questionnaire. We may also require to observe your Home Language and English First Additional Language lessons, and conduct a classroom observation. The questions mainly cover teaching practices and your experience with the intervention. Completion of the whole questionnaire should not take longer than 20 - 30 minutes. Please understand that your participation is voluntary and that you are not being forced to take part in this evaluation. The choice of whether to participate or not is yours alone. If you choose not to take part, you will not be affected in any way whatsoever. If you agree to participate, you may stop participating in the evaluation at any time and tell me that you do not want to continue. If you do this, there will be no penalties and nothing will happen to you as a result of this decision. The same applies should there be specific questions you do not want to respond to. CONFIDENTIALITY Identifying information such as coded ID numbers and lists of names will be kept in a locked cabinet or office and/or stored on a secure server. It will not be available to others and will be kept confidential to the extent possible by law. Records that identify you will be available only to people working on the study, unless you give permission for other people to see the records. If needed, we will ask for your permission in the future at the appropriate time. RISKS/DISCOMFORTS We do not see large risks in your participation. However, you may feel embarrassed about the physical conditions of your school, or fear criticism about management or teaching practices at your school, or the knowledge or skills that you have. We undertake to handle all your information confidentially and anonymously, and for the purposes of this evaluation only. We will not share identifiable evaluation information with anyone else, especially not other Department officials at the circuit, district, provincial, or national offices. BENEFITS The main purpose of the RSP is to improve Grade 1, 2 and 3 learners’ reading proficiency through teacher interventions in the project schools. The aim is to help teachers enhance their subject knowledge and teaching skills. Such teacher changes are very likely to have a positive impact on learners’ reading proficiency and achievement. After completion of the full evaluation, the findings will be made available as soon as possible in evaluation reports to the DBE and USAID. The identity or names of schools, principals, SMT members and teachers will never be attached to any of the information that you provide or the information released through reports or articles. Information Sheet and Consent Form for Teachers Page 3 of 5 WHO TO CONTACT IF YOU HAVE BEEN HARMED OR HAVE ANY CONCERNS If you have concerns or questions about the evaluation, you may call the evaluation manager, Katharine Tjasink at Tel: 011 447 6464 Ext 3207, or e-mail at ktjasink@khulisa.com. You may also contact Nompumelelo Mohohlwane at Tel: 012 357 3200, or alternatively call center: 0800 202 933 or e-mail at Mohohlwane.N@dbe.gov.za. Information Sheet and Consent Form for Teachers Page 4 of 5 Instruction for researcher: Please give this sheet for teacher to sign after you read out consent form. Hand out teacher survey for the respective Grade. Collect the sheet and move to the next teacher. Repeat process. By signing this sheet I hereby agree to participate in the implementation evaluation by Khulisa Management Services with support from the Department of Basic Education and USAID. I understand that I am participating freely and without being forced in any way to do so. I also understand that I can stop participating at any point should I not want to continue and that this decision will not in any way affect me negatively. I understand that my participation will remain confidential. School name:__________________________________________________________ (researcher to write down school name) No. Name Surname PERSAL number Grade Signature Information Sheet and Consent Form for Teachers Page 5 of 5 1 RSP TRAINING OBSERVATION: CLUSTER TEACHER TRAINING 30 September – 2 October 2019 The following aspects (points 1 to 5 below) must be looked for when observing the cluster training of teachers for the Reading Support Project (RSP): 1. Whether the coaches are training teachers effectively on subject knowledge (EFAL and HL literacy for Grades 1, 2 and 3) and pedagogical knowledge (i.e. how to teach literacy in these languages) 2. How confident the coaches are in delivering this training (confidence in the materials and confidence in delivering training, evidenced training style, ability to answer questions, pacing through the training curriculum, etc.) 3. The degree of fidelity to the content of the RSP training materials as the train-the-trainer model is rolled out (i.e. adherence to the content as per the coach training) 4. Whether teachers demonstrate more knowledge over the course of the training (e.g. asking good questions, engaging with the content) 5. What factors constrain or enhance the scale up of the RSP training approach (e.g. language issues, numbers of people being trained, training environment, other training practices, etc.) Please rate your observations of the training against the questions below, as fully as possible. The rating options are as follows: 1 = No 2 = Somewhat 3 = Yes NA = Not applicable. I was unable to observe this Please fill in one sheet per cluster session observed. 2 Information: Cluster 1 Names of the coaches conducting the cluster training Name 1: Name 2: Date of observation Venue of observation Total number of teachers present Of those teachers, total number of Heads of Department present, if any? Total number of curriculum advisors present, if any? Total number of master trainers (e.g. Molteno, OUP) present, if any? Grade(s) covered during the training – Grades 1 and/or 2 and/or 3? Subject(s) covered during the training observation – Home Language (HL) and/or English First Additional Language (EFAL)? Language(s) used by coaches during the observation – Setswana, English, both? Any other notes on attendance? Start time of observation End time of observation 3 Observation: Cluster 1 A. Training Venue and Administration Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 2 Is an agenda for each day made available to teachers? 3 Are name tags are given out to teachers? 4 Is an attendance register made available and signed by the teachers? 5 Has anyone stayed away from the training if it is holiday time? Get a copy of the attendance register and note absentees versus required attendance. 6 Did the trainers/coaches fill out an attendance report form at the end of training? - similar fields to the hard copy attendance register, however it also has fields for coaches/data entry personnel to mark attendance per term. Ask the coach for this form. If it is not available, note the reason why 7 Did all teachers in the cluster receive their LTSM? If not, note why not. For example, if more teachers attended the training than were supposed to. 4 B. Coach Preparedness Question Rating Score Comment 8 Is it clear that the coaches have prepared all training facilitation materials beforehand, including lesson plans (and supporting LTSM)? 9 Are presentations well assembled and put together? 10 Are there enough copies of the training materials for all the teachers in the room? 11 Are the training materials in a file or folder or stapled? 12 Are the handouts appropriate to the learning activities and given out when needed? 13 During training, is there evidence that each coach has and is using a facilitator’s manual? This manual explains to the coach how to facilitate the teacher training workshop 5 Question Rating Score Comment 14 During training, do the coaches refer to the teacher-training manual, used to assist teachers? C. Coach Presentation Style Question Rating Score Comment 15 Is there a warm up activity/exercise? 16 Is the purpose and structure of the day made clear at the beginning of the training, or early on? Is there a link with the previous day? 17 Does the coach allow teachers to introduce themselves? 18 Is the style of training participatory? – look for examples of style and approach and how participants react to this. Note examples 6 Question Rating Score Comment 19 Have the coaches established good rapport with the teachers being trained? Please note your overall impressions as well as impressions per coach 20 Are the coaches good at listening? - A good coach is able to facilitate high-level discussion with adult learners. 21 Is there variety in how presentations are made? i.e. the delivery of training is not monotonous 22 Is there a high level of interest and excitement in the room? How is this shown, or not? Are participants bored? 23 Is the trainer confident about what he/she is doing? Not nervous; speaking clearly; not rushing through the module etc. 24 Is the pace of training right for learning? - do all participants get a chance to give their thoughts and opinions on matters? – is there a balance between input, discussion, and response? 25 Does the facilitator synthesise all inputs from participants at the end of a section? (very important skill) –participants know what the trainer is aiming for, in terms of the development of an activity towards the expected outcomes? 7 D. Fidelity to Content Question Rating Score Comment 26 Does the training content include all aspects of the lesson plan and use of the LTSM, including assessments? 27 Does the training address teachers’ reading needs to incorporate the “Big Five”: Phonemic Awareness, Phonics, Vocabulary, Fluency, and Comprehension? 28 Is the link between writing and reading addressed in the training? 29 Does the training address concepts and approaches related to “read aloud”. This is an essential part of children’s reading pleasure and vocabulary and conceptual development. 30 Does the training address concepts and approaches related to “paired reading” and “independent reading”? 31 Does the training address concepts and approaches related to group guided reading (GGR)? This is a very complicated methodology which requires that teachers know about baseline assessment, teaching GGR routines, setting up same ability 8 Question Rating Score Comment groups, conducting GGR and ensuring that the rest of the class is engaged in meaningful learning. 32 Does the training address concepts and approaches related to promoting reading engagement? (For example, role plays, puppet shows, oral presentations, retells, story, summaries and reader’s theatre to encourage engagement). Does the training address how teachers can use a range of question types with learners: literal, inferential, evaluative, appreciative, and critical? 33 Does the training address how teachers can generate rich discussion on books in the classroom? 34 During training, do the coaches encourage teachers to discuss their own practices and use examples from their own classroom? 35 Do the coaches encourage teachers to revisit vocabulary and readings in order to consolidate learnings? 36 Did the coach address the following topics during the training session: inclusive education, pacing, curriculum coverage, techniques for teaching literacy? Please give an overall rating and make notes for each topic. 9 E. Fidelity to Training Objectives and Approach Question Rating Score Comment 37 Overall, are the training sessions building the necessary skills, according to the purpose and objectives of the training? – note examples 38 Overall, are the coaches building teachers’ content knowledge and literacy concepts (i.e. in HL and EFAL literacy?) – note examples 39 Overall, are the coaches building teachers’ pedagogical ability (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples 40 Is the level of questioning by trainers becoming deeper and more reflective as the day progresses? Give an example; make a general comment and judgement. 41 Do the coaches provide time and space for practical applications in the form of role play or participant presentations? 42 Is there any evidence of the coaches differentiating training based on teacher needs and contexts? - e.g. when teachers note that they cannot implement a particular element for any reason, the coach suggests an alternate strategy 43 Is there a flow, in terms of knowledge development? Is knowledge being ‘scaffolded’, that is built-up in incremental 10 Question Rating Score Comment blocks? Or is the learning scattered and not focused with exercise not linked and discussed sufficiently? 44 Is the timing of exercises appropriate to the level and depth of responses needed from participants? 45 If videos are used, are videos well received and are the learning points clear? Are the participants asked about what they saw and heard? Is sufficient time given to checking on what was learnt from the videos? Only if relevant 46 Does the sequencing of HL and EFAL training complement teachers’ learning? Why/why not? F. Training Support and Feedback Question Rating Score Comment 47 Did all the teachers do a pre-test? In each of the sessions you attend each day, please ask to see the pre-tests and verify these against the registers for those sessions. This is a spot check exercise. 48 Did all the teachers do a post-test? In the last session that you attend each day, please ask to see the post-tests and verify these against the registers for those sessions. This is a spot check exercise. 49 Was an evaluation form made available to teachers at the end of training? 11 Question Rating Score Comment 50 Was a Focus Group Discussion conducted by data collectors with teachers at the cluster training venue? Note: the FGDs are supposed to provide qualitative in-depth feedback on the training as well as additional support required. These occur 6- monthly so they may not be carried out during the observation period. 51 Was a literacy head coach present to oversee and supervise the cluster training provided by coaches to teachers? Note: these are not the same as the master trainers. Please confirm at each session whether there is a head coach present – either via attendance register or during introductions 52 If a literacy head coach was present during the cluster training, did they provide support and/or feedback to the coaches? Did the coaches make any reference to classroom libraries and how to use them during the training? 53 Did any curriculum advisors attend the cluster training? 54 If any curriculum advisors attended the cluster training, did they provide any support and/or feedback to the coaches? 55 Were any of the original coach trainers (OUP & MOLTENO) present at the teacher training? 12 Information: Cluster 2 Names of the coaches conducting the cluster training Name 1: Name 2: Date of observation Venue of observation Total number of teachers present Of those teachers, total number of Heads of Department present, if any? Total number of curriculum advisors present, if any? Total number of master trainers (e.g. Molteno, OUP) present, if any? Grade(s) covered during the training – Grades 1 and/or 2 and/or 3? Subject(s) covered during the training observation – Home Language (HL) and/or English First Additional Language (EFAL). Language(s) used by coaches during the observation – Setswana, English, both? Any other notes on attendance? Start time of observation End time of observation 13 Observation: Cluster 2 G. Training Venue and Administration Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 2 Is an agenda for each day made available to teachers? 3 Are name tags are given out to teachers? 4 Is an attendance register made available and signed by the teachers? 5 Has anyone stayed away from the training if it is holiday time? Get a copy of the attendance register and note absentees versus required attendance. 6 Did the trainers/coaches fill out an attendance report form at the end of training? - similar fields to the hard copy attendance register, however it also has fields for coaches/data entry personnel to mark attendance per term. Ask the coach for this form. If it is not available, note the reason why 7 Did all teachers in the cluster receive their LTSM? If not, note why not. For example, if more teachers attended the training than were supposed to. 14 H. Coach Preparedness Question Rating Score Comment 8 Is it clear that the coaches have prepared all training facilitation materials beforehand, including lesson plans (and supporting LTSM)? 9 Are presentations well assembled and put together? 10 Are there enough copies of the training materials for all the teachers in the room? 11 Are the training materials in a file or folder or stapled? 12 Are the handouts appropriate to the learning activities and given out when needed? 13 During training, is there evidence that each coach has and is using a facilitator’s manual? This manual explains to the coach how to facilitate the teacher training workshop 15 Question Rating Score Comment 14 During training, do the coaches refer to the teacher-training manual, used to assist teachers? I. Coach Presentation Style Question Rating Score Comment 15 Is there a warm up activity/exercise? 16 Is the purpose and structure of the day made clear at the beginning of the training, or early on? Is there a link with the previous day? 17 Does the coach allow teachers to introduce themselves? 18 Is the style of training participatory? – look for examples of style and approach and how participants react to this. Note examples 16 Question Rating Score Comment 19 Have the coaches established good rapport with the teachers being trained? Please note your overall impressions as well as impressions per coach 20 Are the coaches good at listening? - A good coach is able to facilitate high-level discussion with adult learners. 21 Is there variety in how presentations are made? i.e. the delivery of training is not monotonous 22 Is there a high level of interest and excitement in the room? How is this shown, or not? Are participants bored? 23 Is the trainer confident about what he/she is doing? Not nervous; speaking clearly; not rushing through the module etc. 24 Is the pace of training right for learning? - do all participants get a chance to give their thoughts and opinions on matters? – is there a balance between input, discussion, and response? 25 Does the facilitator synthesise all inputs from participants at the end of a section? (very important skill) –participants know what the trainer is aiming for, in terms of the development of an activity towards the expected outcomes? 17 J. Fidelity to Content Question Rating Score Comment 26 Does the training content include all aspects of the lesson plan and use of the LTSM, including assessments? 27 Does the training address teachers’ reading needs to incorporate the “Big Five”: Phonemic Awareness, Phonics, Vocabulary, Fluency, and Comprehension? 28 Is the link between writing and reading addressed in the training? 29 Does the training address concepts and approaches related to “read aloud”. This is an essential part of children’s reading pleasure and vocabulary and conceptual development. 30 Does the training address concepts and approaches related to “paired reading” and “independent reading”? 31 Does the training address concepts and approaches related to group guided reading (GGR)? This is a very complicated methodology which requires that teachers know about baseline assessment, teaching GGR routines, setting up same ability 18 Question Rating Score Comment groups, conducting GGR and ensuring that the rest of the class is engaged in meaningful learning. 32 Does the training address concepts and approaches related to promoting reading engagement? (For example, role plays, puppet shows, oral presentations, retells, story, summaries and reader’s theatre to encourage engagement). Does the training address how teachers can use a range of question types with learners: literal, inferential, evaluative, appreciative, and critical? 33 Does the training address how teachers can generate rich discussion on books in the classroom? 34 During training, do the coaches encourage teachers to discuss their own practices and use examples from their own classroom? 35 Do the coaches encourage teachers to revisit vocabulary and readings in order to consolidate learnings? 36 Did the coach address the following topics during the training session: inclusive education, pacing, curriculum coverage, techniques for teaching literacy? Please give an overall rating and make notes for each topic. 19 K. Fidelity to Training Objectives and Approach Question Rating Score Comment 37 Overall, are the training sessions building the necessary skills, according to the purpose and objectives of the training? – note examples 38 Overall, are the coaches building teachers’ content knowledge and literacy concepts (i.e. in HL and EFAL literacy?) – note examples 39 Overall, are the coaches building teachers’ pedagogical ability (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples 40 Is the level of questioning by trainers becoming deeper and more reflective as the day progresses? Give an example; make a general comment and judgement. 41 Do the coaches provide time and space for practical applications in the form of role play or participant presentations? 42 Is there any evidence of the coaches differentiating training based on teacher needs and contexts? - e.g. when teachers note that they cannot implement a particular element for any reason, the coach suggests an alternate strategy 43 Is there a flow, in terms of knowledge development? Is knowledge being ‘scaffolded’, that is built-up in incremental 20 Question Rating Score Comment blocks? Or is the learning scattered and not focused with exercise not linked and discussed sufficiently? 44 Is the timing of exercises appropriate to the level and depth of responses needed from participants? 45 If videos are used, are videos well received and are the learning points clear? Are the participants asked about what they saw and heard? Is sufficient time given to checking on what was learnt from the videos? Only if relevant 46 Does the sequencing of HL and EFAL training complement teachers’ learning? Why/why not? L. Training Support and Feedback Question Rating Score Comment 47 Did all the teachers do a pre-test? In each of the sessions you attend each day, please ask to see the pre-tests and verify these against the registers for those sessions. This is a spot check exercise. 48 Did all the teachers do a post-test? In the last session that you attend each day, please ask to see the post-tests and verify these against the registers for those sessions. This is a spot check exercise. 49 Was an evaluation form made available to teachers at the end of training? 21 Question Rating Score Comment 50 Was a Focus Group Discussion conducted by data collectors with teachers at the cluster training venue? Note: the FGDs are supposed to provide qualitative in-depth feedback on the training as well as additional support required. These occur 6- monthly so they may not be carried out during the observation period. 51 Was a literacy head coach present to oversee and supervise the cluster training provided by coaches to teachers? Note: these are not the same as the master trainers. Please confirm at each session whether there is a head coach present – either via attendance register or during introductions 52 If a literacy head coach was present during the cluster training, did they provide support and/or feedback to the coaches? Did the coaches make any reference to classroom libraries and how to use them during the training? 53 Did any curriculum advisors attend the cluster training? 54 If any curriculum advisors attended the cluster training, did they provide any support and/or feedback to the coaches? 55 Were any of the original coach trainers (OUP & MOLTENO) present at the teacher training? 22 Information: Cluster 3 Names of the coaches conducting the cluster training Name 1: Name 2: Date of observation Venue of observation Total number of teachers present Of those teachers, total number of Heads of Department present, if any? Total number of curriculum advisors present, if any? Total number of master trainers (e.g. Molteno, OUP) present, if any? Grade(s) covered during the training – Grades 1 and/or 2 and/or 3? Subject(s) covered during the training observation – Home Language (HL) and/or English First Additional Language (EFAL)? Language(s) used by coaches during the observation – Setswana, English, both? Any other notes on attendance? Start time of observation End time of observation 23 Observation: Cluster 3 M. Training Venue and Administration Question Rating Score Comment 1 Is the training room set up so as to facilitate a good learning experience? – Not crowded; well ventilated; ability to see power point presentations; ability to hear the facilitator; ability to spread out the materials on a desk; ability to get up and move; etc. 2 Is an agenda for each day made available to teachers? 3 Are name tags are given out to teachers? 4 Is an attendance register made available and signed by the teachers? 5 Has anyone stayed away from the training if it is holiday time? Get a copy of the attendance register and note absentees versus required attendance. 6 Did the trainers/coaches fill out an attendance report form at the end of training? - similar fields to the hard copy attendance register, however it also has fields for coaches/data entry personnel to mark attendance per term. Ask the coach for this form. If it is not available, note the reason why 7 Did all teachers in the cluster receive their LTSM? If not, note why not. For example, if more teachers attended the training than were supposed to. 24 N. Coach Preparedness Question Rating Score Comment 8 Is it clear that the coaches have prepared all training facilitation materials beforehand, including lesson plans (and supporting LTSM)? 9 Are presentations well assembled and put together? 10 Are there enough copies of the training materials for all the teachers in the room? 11 Are the training materials in a file or folder or stapled? 12 Are the handouts appropriate to the learning activities and given out when needed? 13 During training, is there evidence that each coach has and is using a facilitator’s manual? This manual explains to the coach how to facilitate the teacher training workshop 25 Question Rating Score Comment 14 During training, do the coaches refer to the teacher-training manual, used to assist teachers? O. Coach Presentation Style Question Rating Score Comment 15 Is there a warm up activity/exercise? 16 Is the purpose and structure of the day made clear at the beginning of the training, or early on? Is there a link with the previous day? 17 Does the coach allow teachers to introduce themselves? 18 Is the style of training participatory? – look for examples of style and approach and how participants react to this. Note examples 26 Question Rating Score Comment 19 Have the coaches established good rapport with the teachers being trained? Please note your overall impressions as well as impressions per coach 20 Are the coaches good at listening? - A good coach is able to facilitate high-level discussion with adult learners. 21 Is there variety in how presentations are made? i.e. the delivery of training is not monotonous 22 Is there a high level of interest and excitement in the room? How is this shown, or not? Are participants bored? 23 Is the trainer confident about what he/she is doing? Not nervous; speaking clearly; not rushing through the module etc. 24 Is the pace of training right for learning? - do all participants get a chance to give their thoughts and opinions on matters? – is there a balance between input, discussion, and response? 25 Does the facilitator synthesise all inputs from participants at the end of a section? (very important skill) –participants know what the trainer is aiming for, in terms of the development of an activity towards the expected outcomes? 27 P. Fidelity to Content Question Rating Score Comment 26 Does the training content include all aspects of the lesson plan and use of the LTSM, including assessments? 27 Does the training address teachers’ reading needs to incorporate the “Big Five”: Phonemic Awareness, Phonics, Vocabulary, Fluency, and Comprehension? 28 Is the link between writing and reading addressed in the training? 29 Does the training address concepts and approaches related to “read aloud”. This is an essential part of children’s reading pleasure and vocabulary and conceptual development. 30 Does the training address concepts and approaches related to “paired reading” and “independent reading”? 31 Does the training address concepts and approaches related to group guided reading (GGR)? This is a very complicated methodology which requires that teachers know about baseline assessment, teaching GGR routines, setting up same ability 28 Question Rating Score Comment groups, conducting GGR and ensuring that the rest of the class is engaged in meaningful learning. 32 Does the training address concepts and approaches related to promoting reading engagement? (For example, role plays, puppet shows, oral presentations, retells, story, summaries and reader’s theatre to encourage engagement). Does the training address how teachers can use a range of question types with learners: literal, inferential, evaluative, appreciative, and critical? 33 Does the training address how teachers can generate rich discussion on books in the classroom? 34 During training, do the coaches encourage teachers to discuss their own practices and use examples from their own classroom? 35 Do the coaches encourage teachers to revisit vocabulary and readings in order to consolidate learnings? 36 Did the coach address the following topics during the training session: inclusive education, pacing, curriculum coverage, techniques for teaching literacy? Please give an overall rating and make notes for each topic. 29 Q. Fidelity to Training Objectives and Approach Question Rating Score Comment 37 Overall, are the training sessions building the necessary skills, according to the purpose and objectives of the training? – note examples 38 Overall, are the coaches building teachers’ content knowledge and literacy concepts (i.e. in HL and EFAL literacy?) – note examples 39 Overall, are the coaches building teachers’ pedagogical ability (i.e. the practice of teaching literacy and teaching methods in reading and writing instruction)? – note examples 40 Is the level of questioning by trainers becoming deeper and more reflective as the day progresses? Give an example; make a general comment and judgement. 41 Do the coaches provide time and space for practical applications in the form of role play or participant presentations? 42 Is there any evidence of the coaches differentiating training based on teacher needs and contexts? - e.g. when teachers note that they cannot implement a particular element for any reason, the coach suggests an alternate strategy 43 Is there a flow, in terms of knowledge development? Is knowledge being ‘scaffolded’, that is built-up in incremental 30 Question Rating Score Comment blocks? Or is the learning scattered and not focused with exercise not linked and discussed sufficiently? 44 Is the timing of exercises appropriate to the level and depth of responses needed from participants? 45 If videos are used, are videos well received and are the learning points clear? Are the participants asked about what they saw and heard? Is sufficient time given to checking on what was learnt from the videos? Only if relevant 46 Does the sequencing of HL and EFAL training complement teachers’ learning? Why/why not? R. Training Support and Feedback Question Rating Score Comment 47 Did all the teachers do a pre-test? In each of the sessions you attend each day, please ask to see the pre-tests and verify these against the registers for those sessions. This is a spot check exercise. 48 Did all the teachers do a post-test? In the last session that you attend each day, please ask to see the post-tests and verify these against the registers for those sessions. This is a spot check exercise. 49 Was an evaluation form made available to teachers at the end of training? 31 Question Rating Score Comment 50 Was a Focus Group Discussion conducted by data collectors with teachers at the cluster training venue? Note: the FGDs are supposed to provide qualitative in-depth feedback on the training as well as additional support required. These occur 6- monthly so they may not be carried out during the observation period. 51 Was a literacy head coach present to oversee and supervise the cluster training provided by coaches to teachers? Note: these are not the same as the master trainers. Please confirm at each session whether there is a head coach present – either via attendance register or during introductions 52 If a literacy head coach was present during the cluster training, did they provide support and/or feedback to the coaches? Did the coaches make any reference to classroom libraries and how to use them during the training? 53 Did any curriculum advisors attend the cluster training? 54 If any curriculum advisors attended the cluster training, did they provide any support and/or feedback to the coaches? 55 Were any of the original coach trainers (OUP & MOLTENO) present at the teacher training? 515 ANNEX 7: METHODOLOGY PLAN & STUDY PROTOCOL 516 METHODOLOGY PLAN AND STUDY PROTOCOL Design and Implementation Evaluation of the Reading Support Project 9 September 2019 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd in collaboration with the South African Department of Basic Education 517 METHODOLOGY PLAN AND STUDY PROTOCOL DESIGN AND IMPLEMENTATION EVALUATION OF THE READING SUPPORT PROJECT 9 September 2019 Contract Number: 72067418D00001, Order Number: 72067419F00015 Cover Photo: School Books, Photo Credit: StockSnap, available from Pixabay DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United 518 AUTHORS Jennifer Bisgard (Project Director) Katharine Tjasink (Senior Project Manager) Leticia Taimo (Evaluation Coordinator) CONTACT DETAILS Margaret Roper 26 7th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: mroper@khulisa.com Web Address: www.khulisa.com 519 TABLE OF CONTENTS INTRODUCTION AND BACKGROUND ...........................................................................522 INTRODUCTION..............................................................................................................522 BACKGROUND ................................................................................................................522 EVALUATION OVERVIEW .............................................................................................522 IDENTIFYING INFORMATION.......................................................................................524 RESEARCH QUESTIONS ....................................................................................................525 DESIGN .................................................................................................................................525 INTERVENTION DESIGN................................................................................................525 EVALUATION DESIGN....................................................................................................527 DESIGN EVALUATION (INCEPTION PHASE) .............................................................527 IMPLEMENTATION EVALUATION ...............................................................................528 STUDY POPULATION ........................................................................................................530 SAMPLING PLAN.................................................................................................................530 INSTRUMENTS ....................................................................................................................533 PHASE 1: COACH SHADOWING AND SCHOOL PILOTING.....................................533 PHASE 2: FIELDWORK AT SAMPLE SCHOOLS...........................................................534 DATA HANDLING AND COLLECTION............................................................................537 DESIGN EVALUATION (INCEPTION PHASE) .............................................................537 IMPLEMENTATION EVALUATION ...............................................................................537 Fieldworker Recruitment ..............................................................................................537 Training and preparation for fieldwork ........................................................................538 Fieldwork .......................................................................................................................539 Data preparation and reporting....................................................................................542 ETHICAL CONSIDERATIONS............................................................................................542 520 ACRONYMS AHL African Home Language CFR Code of Federal Regulations CSV Comma Separated Value DBE South African Department of Basic Education DQA Data Quality Assessment EFAL English First Additional Language EGRS Early Grade Reading Study GoSA Government of South Africa GPS Global Positioning System FPD Foundation for Professional Development HOD Head of Department ICT Information and Communications Technology IDIQ Indefinite Delivery/Indefinite Quantity ODK Open Data Kit LOE Level of Effort LTSM Learning and Teaching Support Materials PERFORM Practical Education Research for Optimal Reading and Management PERFORMAN CE Practical Education Research For Optimal Reading and Management: Analyze, Collaborate, Evaluate PIRLS Progress in Reading and Literacy Study QASP Quality Assurance Surveillance Plan RSP Reading Support Project SMT School Management Team SOP Standard Operating Procedures 521 TO Task Order USA United States of America USAID United States Agency for International Development VSO Voluntary Service Overseas LIST OF TABLES Table 1 Potential Sampling Techniques and Data Sources....................................................................532 Table 2 Education researcher profiles..............................................................................................538 Table 3 Illustrative one-day plan for data collection ............................................................................541 LIST OF FIGURES Figure 1 Proposed Coach Shadowing Process.....................................................................................540 522 INTRODUCTION AND BACKGROUND INTRODUCTION Khulisa Management Services Pty Ltd. (Khulisa), is pleased to present this Methodology Plan and Study Protocol to the United States Agency for International Development (USAID) for a Design and Implementation Evaluation of the Reading Support Project. BACKGROUND To address the challenge of children not learning to read for understanding, in 2015 the DBE initiated the Early Grade Reading Study (EGRS) in two districts in the North West province (districts of Ngaka Modiri Molema and Dr Kenneth Kaunda). The EGRS evaluated three Setswana Home Language interventions aimed at improving reading in the early grades: a teacher training intervention, an on-site teacher training and coaching intervention, as well as a parental intervention. The interventions ended in 2017 and EGRS impact results showed substantial impacts on learner results through training, coaching, and provision of learning materials. In 2019 and 2020, the DBE requested USAID’s support in proceeding with a second phase to the EGRS, specifically to scale up the coaching intervention, which showed the most impact. In response to this request, USAID and the Foundation for Professional Development (FPD) Consortium (comprising FPD, the Molteno Language Institute, Oxford University Press of South Africa, and Voluntary Services Oversees) modified the existing Reading Support Project (RSP) to include the selected EGRS components. USAID, along with its partner, the DBE, has commissioned a design and implementation evaluation of the RSP to evaluate the implementation of RSP as it scales up this component of the EGRS. The evaluation will provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. In 2019, the results of the evaluation will be formative (i.e. used to improve the programme in 2020) and, in 2020, the results will be summative (i.e. compared to the 2019 results). EVALUATION OVERVIEW The aim of the evaluation is to support USAID and the Government of South Africa (GoSA) in evaluating the extent to which the RSP is being implemented with fidelity, with a particular focus on whether it is being implemented in line with GoSA’s predecessor program, the Early Grade Reading Study (EGRS). The evaluation will: 1. Identify any risks to the successful implementation of the RSP 2. Provide early findings to strengthen the implementation of the RSP 3. Ascertain whether the allocated time and resources were sufficient to successfully implement the RSP 523 4. Determine whether the FPD Consortium is implementing its activities as planned 5. Determine whether impact-level change can realistically be expected with RSP’s theory of change, its design and within the timeframe of the RSP implementation 6. Explore whether and how the RSP theory of change could be strengthened to produce the intended outcomes 7. Validate the data collected through the RSP M&E system 8. Determine whether teacher training and coaching is leading to any early changes in teaching methods (e.g. group guided reading, phonics, use of chapters in lesson plans to document implementation) 9. Ascertain what dosage of support teachers are receiving from coaches (i.e. number of visits, time spent per visit) 10. Determine whether coaches are supporting teachers and reporting on their coaching visits 11. Determine the contribution of volunteering in generating change in school management 12. Determine whether the School Management Team (SMT) support can be linked to changed support practices in schools. Under this Task Order (TO), the evaluation team has a responsibility to provide a careful account of how the RSP was adapted to include EGRS, whether the adapted design is likely to lead to impact, whether the dosage and coverage of services is sufficient to support pathways to impact, and whether there is fidelity to the intended design in the actual implementation of the program. This is important for a number of reasons. First and foremost, this information is critical towards accurately attributing any impact to the program. If successful, the program can be repeated in another context as the elements of implementation will be known. If unsuccessful, the evaluation will provide some insight into whether this was due to design failure (i.e. poor plausibility of the pathways to impact) or to implementation failure (i.e. the design was plausible but the implementation lacked the robustness required to deliver the intended impact). Second, this information will assist the implementers of the RSP - the Foundation for Professional Development (FPD) Consortium – in understanding any gaps in implementation or identifying pockets of success. The data will be used in a formative manner to give the program the best possible chance to lead to impact. The design evaluation will entail a mini literature review, a review of project documents, surfacing the RSP Theory of Change and comparing it to the EGRS Theory of Change; interviews with key stakeholders and a design workshop with an expert panel hosted at the DBE in Pretoria. Findings from this design phase will be included in a Design/Inception Report. The implementation evaluation will be conducted in the North-West province of South Africa. The settings within which the research will take place include Quintile 1 – 3 schools,33 which 33 In South Africa, public schools are grouped into quintiles, which describe the wealth of the school, and therefore how much government funding they are entitled to. Quintile 5 schools are the wealthiest schools in the country, while Quintile 1 schools are the poorest. Quintile 1 – 3 schools do not charge school fees, and a large proportion of these schools participate in the National School Nutrition program where learners receive a meal at the school. These schools are legally not allowed to charge school fees, and are usually the worst performing in the system. 524 use Setswana as the language of learning and teaching in the Foundation Phase (Grade 1-3 classrooms). The population of interest includes: • Teachers in Grades 1 to 3 in the participating schools in the North-West province; • Coaches providing support to participating schools; • SMT members in participating schools; and • The DBE and North West Provincial and District officials in Kenneth Kaunda and Ngaka Modiri Molema districts. The study findings will be directly applicable to the program funders, implementers, the North￾West province Department of Education, as well as the DBE. The information generated through this evaluation will add to the evidence base for policy decisions about the most effective way to support the teaching of Home Language and English as First Additional Language (EFAL) in the Foundation Phase, in South Africa. Research authorization is not required as the DBE has an agreement with USAID. The DBE in turn is mandated by laws to have oversight on program implementation in schools. IDENTIFYING INFORMATION Dr Stephen Taylor, Director of Research Coordination, Monitoring and Evaluation from the DBE is the Principal Investigator for the EGRS I, supported in this role by Dr. Janeli Kotzé and Ms. Nompumelelo Mohohlwane. Khulisa is responsible for conducting the design and implementation evaluation. Ms. Jennifer Bisgard is the Project Director, supported in this role by Ms. Margaret Roper (Senior Project Manager) and Ms. Leticia Taimo (Evaluation Coordinator). The DBE is contactable at Tel: +2712 357 4156 | Call Centre: 0800 202 933 and Khulisa is contactable at Tel: +2711 447 6464. This activity is funded by USAID under Contract Number 72067418D00001 and Task Order number 72067418F00010 525 RESEARCH QUESTIONS Design Evaluation Implementation Evaluation 1. Is the RSP theory of change likely to lead to the anticipated results? 1. To what extent did the FPD consortium implement the intended intervention design with fidelity? 2. What evidence is there of RSP uptake by teachers in class? 3. Were there any major divergences from the assumptions in the theory of change? 4. How could the management and implementation structures be improved? 5. How could the M&E systems be improved? 6. What are the unique challenges of delivery at scale? 7. Is the RSP cost efficient? DESIGN INTERVENTION DESIGN34 The RSP’s strategy was revised taking into account EGRS evidence that the Coaching intervention showed a substantial positive impact after two years of intervention (end of Grade 2) and the DBE’s interest to expand the evidence based around in-school coaching model. In a written request to USAID, the DG specifically recommended that the RSP support the EGRS model in the Dr. Kenneth Kaunda and Ngaka Modiri Molema districts that were earmarked for scaling after the RSP pilot. The DG further expressed the view that building on the EGRS model would ensure continuity in the schools that were involved in the EGRS in these two districts. The revised overarching goal of the RSP is to improve the reading skills of primary grade learners in African Home Languages (AHLs), as well as in English as a first additional language (EFAL). The project objective is to improve language and literacy content knowledge and pedagogy of primary grade teachers in AHLs as well as in EFAL. By end of the four-years, the project aims to achieve the following planned results: • Result 1: Increased capacity of curriculum advisors (CA) • Heads of department (HODs) to support primary grade teachers in language and literacy in African Home Languages as well as English First Additional Language. (Component 1) • Result 2: Increased capacity of principals or deputy principals to support primary grade teachers in language and literacy in African Home Languages as well as English First Additional Language. 34 Information retrieved from Attachment 5 Revised Reading Support Activity 26 July 2019 526 • Result 3: Improved quality and use of LTSM for African Home Languages as well as English First Additional Language in the primary grades. (Component 3) [with inclusion of EGRS structured lesson plans] • Result 4: Increased competency and improved practice of primary grade teachers to teach in AHL and EFAL. [new Result area; primary focus area for revised strategy] The revised strategic approach is designed to improve teacher effectiveness and quality by means of improving capacity of teachers through a structured learning program and specialist on￾site coaching for better learner results in AHLs and EFAL. [Significant change with focus on the teacher and coaching.] The Project specifically aimed to improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of Learner Teacher Support Materials (LTSM); and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning. The [RSP] proposed strategic approach is designed to build teacher capacity to teach early grade reading in AHL and EFAL focusing on grades one to three [significant revision through direct engagement with teachers in school via coaching and pivot from focusing on capacitating Heads of Department (HoDs) to perform a coach-like role for teachers in their schools] and strengthen supervisory and management systems for AHL and EFAL literacy at the primary grade level. The revised RSP model is a direct capacity building model for teachers with supplemental systems strengthening activities to foster a more enabling environment for the coaching program and to promote sustainability of practices. Component 4 [new] is introduced and funded through: • Savings in limiting geographic expansion to two districts in North West; the project is no longer envisioned for Limpopo; • Savings in limiting the AHL material to Setswana (developed for pilot) by continued focus on North West; • Savings in offering a combined SMT (Principals and HODs) program (Components 1 and 2 in the RSP’s original version) • Implementation of a fundraising strategy to supplement available USAID and DBE funds. • Savings from reduced volunteer engagement in the SMT program • Savings resulting from the reallocation of vehicles from projects that closed down to RSP 527 EVALUATION DESIGN This study is divided into two parts: a design evaluation (July-August 2019) and an implementation evaluation (September 2019 – November 2020). DESIGN EVALUATION (INCEPTION PHASE) The assignment will begin with a mini literature review to explore extant reports, documents, performance indicators, and literature relevant to the subject area of the evaluation. The review will then examine the intervention itself and the data generated from the intervention thus far. This will help the evaluation team to understand the perspective of the implementers and the progress that the program has made to date. The review will also describe and assess the project’s intervention logic, including assumptions about how success will be achieved. It will help to inform tool development and will inform the evaluation team broadly about the intervention. Khulisa will conduct process mapping in 2019 to understand and outline the processes that are being followed to deliver the RSP. The process maps will unpack the process theory of change in more detail and indicate the activities and workflows relevant to key processes such as: • Recruitment, training, deployment, and supervision of trainers/coaches • Delivering training and coaching to teachers, SMTs and District Officials • Developing, printing, and delivering lesson plans and other Learning and Teaching Support Materials (LTSM) to schools and teachers. Khulisa will conduct a full design review in 2019 to document the RSP theory of change based on interviews, the process mapping, a design workshop, and the mini literature review. This will be compared to the EGRS theory of change. A panel of local and international literacy experts will review the RSP theory of change and assumptions and express an opinion on the feasibility of achieving the expected results. Approximately 30 stakeholders involved in the RSP will be invited to attend the workshop, which will be held at the DBE in Pretoria. The design review and process mapping will inform a fidelity review by comparing the RSP Standard Operating Procedures (SOPs) against actual implementation and will feed into the cost and time analysis, highlighting inefficiencies or redundancies that could be improved in 2020. A design evaluation report will be submitted in 2019 at the end of the inception phase of the assignment. The design review will document and contrast the EGRS and RSP impact theories of change, the process theories of change, and also draw comparisons based on a map of the systems in which the EGRS and RSP functions. This will help to identify if there is a material departure from the original outcome design, from the implementation model, or if the system within which the programs operate differ in complexity. The design review report will indicate how this information will be used in the implementation phase of the assignment. 528 IMPLEMENTATION EVALUATION As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling up the RSP. The purpose of the implementation evaluation is to determine the degree of fidelity of implementation. Khulisa will carry out a fidelity review drawing on secondary data provided through the FPD Consortium’s monitoring systems, supported by a data quality assessment (DQA) in 2019 and 2020 to determine the extent to which the data provided through the program monitoring systems is relevant, effective, timely, and efficient. The monitoring data (attendance data / visit logs / coach feedback / participant feedback forms, etc.) of the FPD consortium will be reviewed to determine if the teacher training / teacher coaching / SMT training / district training / volunteer support happened as planned – i.e. that the anticipated number of contact sessions were conducted for the specified duration (dosage), with the targeted number of participants (coverage), with quality of input, within the planned time frame. The DQA will be carried out on the monitoring data within each sampling arm (See Sampling Plan section). Based on the finalized RSP M&E Plan, and RSP M&E data review, key indicators will be developed and tools designed to ensure sufficient and appropriate evidence is collected by education researchers at sampled schools and with coaches to verify the M&E data. The DQA will document findings, draw conclusions, and make recommendations to improve data quality in 2020. In Khulisa’s prior DQA work, questions have scaled responses (i.e. “No-Not at all“, “Partly”, “Yes, completely”, “N/A”) with supporting narrative substantiating the response. Thus, the DQA tool will (i) streamline data synthesis across DQA tools, (ii) calculate performance scores at each level, (iii) produce graphs to describe strengths and weaknesses of the various indicators, and (iv) develop focused recommendations to improve program performance in specific domains. Case studies will be developed in 2019 and 2020 to document the typical week of a coach using monitoring data, coach observation, and interviews with the coach, teachers, supervisors, and district officials. The assessments will look for top performing coaches and their characteristics, middle and then potentially poor performing or low performing coaches and reasons why. The coaching case studies will comprise an assessment of all 14 coaches (case studies) to inform an early implementation report. More in-depth case studies will be developed and included as part of the formative implementation evaluation report. The evaluation team will survey teachers who previously had coaches under the EGRS as well as teachers who have never had a coach, and will compare and contrast the amount and quality of coaching received. These teachers will be drawn from the coaching intervention and coaching plus SMT samples (See Sampling Plan section). Assuming coaches also fulfil the role of trainers, interviews with all coaches/trainers will provide valuable feedback on their experience of training. In addition, two mid-level education researchers will conduct training observations (covering both districts) observing: 1) Subject Advisor Steering Committee meeting on 29-30 August, 2019 2) Coach training on 25-27 September where coaches are trained to train teachers for Term 4 and to support teachers during Term 4 529 3) Teacher training for Term 4 on 30 September to 2 October, conducted by the coaches trained in the previous week. At least one training observation researcher will be fluent in Setswana and therefore able to conduct observations during the Setswana sessions. Observations, triangulated with teacher group interviews, will allow for a better understanding of the quality of the training. The evaluation team will carry out interviews with the SMT with all arms to compare the support experienced by teachers in the SMT treatment group to those outside of this group (Section 2.5, Sampling). The SMT will provide insight into coaching (where relevant) and the role of the Voluntary Services Organization (VSO) in providing mentors (where relevant), and will be asked to describe any resultant changes within the school – both in terms of teacher practice and school management. Using a structured interview guide, education researchers will gather information across a range of assessment domains. The education researchers will conduct classroom observations in all schools within each sampling arm (See Sampling Plan section) to understand how the different treatment options are leading to possible changes in the classroom. The classroom observations will be carried out in two Grades, in Home Language (HL) and English First Additional Language (EFAL). Information from the teacher survey and teacher group discussions will be triangulated with the findings. Khulisa will triangulate the information gleaned from coach shadowing and coach interviews, data from the SMT interviews, and classroom observations, with a small sample of teacher group discussions (See Sampling Plan section). Teachers will help evaluate the coaching and SMT support received and its effect on their teaching practices. To supplement the teacher group discussions, the evaluation team will administer a short survey to teachers to rate the RSP along dimensions such as quality and quantity of services received. The results will be compared from 2019 to 2020 to gage improvements. In the sample schools that have been allocated a classroom library, as part of the classroom observations specialist education researchers will conduct an assessment of classroom libraries. In addition, researchers will conduct a learner workbook assessment from a sample of 4 learner workbooks (i.e. DBE workbooks and learner exercise books) per sample school. This will comprise a comprehensive review of the DBE workbooks (from the start of the third term until the date of the assessment, where available) as well as a review of learner exercise books (from the start of the school year until the date of assessment, where available). The learner workbook assessment will take place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks will be drawn will be the same classes in which the Classroom Observations were conducted. The teachers of those classes will be asked to provide the workbook of the best learner in the class. Khulisa will carry out a full cost and time analysis toward the end of 2019 and again in 2020. For the cost analysis, we will analyze the budget and expenditure figures of the RSP (compiled by the FPD Consortium) and compare it to the EGRS budget (compiled by the DBE). Where possible, the audited records of the FPD Consortium could be used. For the time analysis, the process map will be used to identify the main activities in the delivery of the RSP. Key stakeholders from FPD and the DBE will be asked to provide estimates of the time that is required to carry out the RSP. This will be contrasted with the Level of Effort (LOE) estimates in the budget, and where possible, verified through key informant interviews (KIIs) with stakeholders in the field 530 including the North West DBE. This will allow the evaluation to surface any misalignment between the planned time investment as per the budget and the actual time investment for project staff. The analysis will also reflect estimates of the time required of other role players to implement the program. This information will provide formative input on LOE during the planning for 2020, will identify potential cost efficiencies that could be achieved, and in 2020 will check whether the costing and time management options were actually implemented. STUDY POPULATION The RSP is being implemented in the North-West province of South Africa, in the districts of Dr. Kenneth Kaunda and Ngaka Modiri Molema. The implementation evaluation will be conducted in the two districts. The settings within which the research will take place include Quintile 1 – 3 schools,35 which use Setswana as the language of learning and teaching in the Foundation Phase (Grade 1-3 classrooms). Data will be collected across a sample of 60 treatment and control schools. The population of interest includes: • Teachers in Grades 1 to 3 in the participating schools in the North-West province; • Coaches providing support to participating schools; • SMT members in participating schools; and • North West Provincial officials and District officials in Kenneth Kaunda and Ngaka Modiri Molema districts. SAMPLING PLAN As a primarily qualitative evaluation, the aim is to enhance understanding of, and illuminate, the complexities associated with scaling up the RSP. The purpose of the implementation evaluation is to determine the degree of fidelity of implementation. Unlike the quantitative evaluation of the EGRS, the sample will not assess schools against extant characteristics such as whether they performed well or poorly or whether they had large learning gains in the EGRS. Since the RSP program is the larger scale up of the EGRS study, the sampling frame will also include schools that did not form part of the original sample of EGRS schools. For school-level sampling, the actual sampling strategy was defined in consultation with the DBE and USAID, to draw a sample of 60 schools36. In total, 14 schools were selected from Dr. Kenneth 35 In South Africa, public schools are grouped into quintiles, which describe the wealth of the school, and therefore how much government funding they are entitled to. Quintile 5 schools are the wealthiest schools in the country, while Quintile 1 schools are the poorest. Quintile 1 – 3 schools do not charge school fees, and a large proportion of these schools participate in the National School Nutrition program where learners receive a meal at the school. These schools are legally not allowed to charge school fees, and are usually the worst performing in the system. 36 Together with the pilot schools, the number of schools will total 66 531 Kaunda District and 44 from Ngaka Modiri Molema district to reflect the distribution of schools per district. The sample of 60 schools is comprised as follows: 1. 10 schools receiving coaching, who previously were control schools; 2. 10 schools receiving coaching & SMT training, who previously were control schools; 3. 10 schools receiving coaching, who previously were EGRS coaching schools; 4. 10 schools receiving coaching & SMT training, who previously were EGRS coaching schools; and 5. 20 schools currently receiving LTSM only, who were previously EGRS parental involvement schools)? It was also decided that these schools would be selected randomly from amongst all eligible schools matching the above criteria. The sample was drawn by Stephen Taylor (Director: Research Coordination, Monitoring and Evaluation, DBE) on 13 August 2019 using a statistical software called STATA. The list of 251 schools participating in RSP as at 13 August 2019 was used as a starting point. A number of different “strata” were classified in order to group all the schools into the 5 categories of school listed above, as well as to identify schools which did not fit into any of those categories and therefore would not be eligible for the sample. Schools not eligible for the sample included the 34 schools who had previously been part of the EGRS “Training” intervention, and the 85 RSP schools which were not previously part of EGRS and therefore have no relevant comparison group. Each school was then assigned a random number using STATA’s “runiform” command. Next, the list of schools was sorted in order first of strata and then in terms of the random number. Within each strata, therefore, the order of schools is strictly random. The first 10 schools within each of the above strata (except for the 5th strata where the first 20 schools were needed) were then selected for inclusion in the implementation evaluation sample. Finally, the last 4 schools within each of the five strata were also selected to serve as schools to be used by Khulisa for simulation and piloting of the survey instruments. These 20 schools were thus also randomly selected. It turned out that a fair number of schools who have received classroom libraries were included in the sample even though this was not a criterion upon which the sample was selected. The schools which have received classroom libraries were indicated in a list of sampled schools provided by the DBE to Khulisa. The sample is too small to be representative or to allow generalization, but should be big enough to allow for identification of consistent patterns within and between the groups. The evaluation team is adapting the tools and methods that the DBE used to gather data during the EGRS and, where relevant, will use the same or similar tools and approaches. Within each school, a list of sampling techniques is described below. 532 Table 61 Potential Sampling Techniques and Data Sources Data Source Sampling Method Coaching case studies Mid-level researchers will shadow all 14 coaches, which is the total population. The aim of these interviews is to gain qualitative insight into a “week in the life of a coach”. Coaching assessment and interviews Total population sampling of all 14 coaches Group discussion with teachers The evaluation team will identify Grade 1, 2 and 3 teachers to participate in a group discussion at 12 sample schools. The team will aim for maximum variation and consider variables such as gender, age, whether the respondent received previous EGRS interventions, subject (Home Language and EFAL) to ensure that the sample provides a range of views that are relevant to the evaluation questions. In schools with very few Grade 1, 2, and 3 teachers, all teachers will be included in the group discussions. KIIs with principals, SMT, district officials, and other identified stakeholders The evaluation team will use purposive sampling to engage key informants as the aim of these interviews is to gain qualitative insight into the implementing agency and other key informants’ perspective on the project and sampling for proportionality is not the main concern. The team will use a framework for selecting the final respondents that will include variables such as gender, role in the program, etc., to ensure that the sample can adequately address the evaluation questions. Where relevant, the evaluation team will use snowball sampling to identify other useful candidates for study. Classroom observations Teaching practices will be observed in one Grade 1 HL class and one Grade 1 EFAL class, as well as one Grade 3 HL class and 1 Grade 3 EFAL class per sample school (per the consultation with the DBE and USAID).Classes within those Grades/subjects will be randomly sampled. Workbook and Learner Exercise Book analysis The learner workbook assessment will take place in one Grade 1 EFAL class and one Grade 1 HL class, as well as one Grade 3 EFAL class and one Grade 3 HL class. The classes from which the workbooks will be drawn will be the same classes in which the Classroom Observations were conducted. This will make 4 learner workbook assessments per school and 240 in total across the 60 sample schools. Data quality assessment (DQA) School-level data collected through the FPD monitoring system will be cross-checked with the data available at the school. As part of the assessment, data about all coaches / trainers and the delivery of all materials will be assessed. Short survey with teachers All Grade 1 to 3 teachers at each of the subsample schools will be administered a short survey. Interviews with the principal and SMT All principals and available SMT members at each of the subsample schools will be interviewed. 533 INSTRUMENTS Data collection will occur in two phases: 1) coach shadowing and instrument piloting 2) fieldwork at sample schools. Training observation will be conducted between the two phases. The instruments required for each phase are described in the section below. The administration and completion of all the instruments administered throughout the evaluation will be exclusively in English. The bulk of these instruments will be administered using an electronic tablet and data capturing will therefore happen directly upon the administration of the questionnaires, or captured into electronic format on the same day as the school visit. PHASE 1: COACH SHADOWING AND SCHOOL PILOTING The tools to be used during the coach shadowing and piloting activities will be developed by Khulisa in collaboration with our Coaching Expert, and will be reviewed by the DBE before piloting and data collection. These include: 1. Coach consent form: This tool comprises an information sheet and consent form for coaches to participate in the coach shadowing data collection activity. 2. Coach administrative checklist: This tool comprises a data quality assessment of coaching administrative activities against the Standard Operating Procedures (SOPs). The tool will enable fieldworkers to capture what coaches do at the school, and whether this is in alignment with the SOPs. 3. Coaching observation tool: part A (school level data): This tool comprises a checklist for each school that the coaches visit over a four-day period. The tool observes coach arrival and exit times, whether they conduct an afternoon workshop with the teachers and other related activities, which will be compared against the SOPs. 4. Coaching observation tool: part B (lesson observation): This tool comprises a checklist for each lesson that the coaches observe. The checklist will check the coach’s routine in the lesson observation against the SOPs. 5. Coaching observation tool: part B (coach interview): This tool comprises a structured template for conducting and documenting an interview with each of the coaches. This template will capture the coaches’ experience of coaching, their experiences working with the schools, and the level of support that they receive from head coaches. 6. Teacher structured group discussion consent form: This tool comprises an information sheet and consent form for teachers to participate in a group discussion with the fieldworker. 7. Teacher structured group discussion guide: The teacher group discussion guide will be used to collect data from teachers to help evaluate the coaching and SMT support received and its effect on their teaching practices. Group discussions will be conducted with teachers from a sub-sample 12 of schools during the coach shadowing week. Items in the guide include: whether teachers are receiving adequate support from the coaches whether they're receiving afternoon, workshops, and what the issues are around receiving 534 support in different grades, and in HL and EFAL. The results will be collated and reported in the aggregate, but qualitative evidence will be used to support findings from other data collection points. The tool will also collect information on the level of support that the SMT provides in the school, as well as the quality of the RSP LTSM and training support. 8. Training observation tool: This tool will guide education researchers’ data collection at the coach training scheduled for 25 and 26 September 2019 followed by a dry-run with the coaches on 27 September 2019, the teacher training from 30 September – 2 October 2019 and the reference group scheduled for 29 and 30 August 2019. Data collected through this tool will be triangulated with teacher group discussions, allowing for better understanding the quality of the training. PHASE 2: FIELDWORK AT SAMPLE SCHOOLS Khulisa will work closely with the DBE to adapt existing EGRS tools that can be used for this evaluation. The following instruments will be administered across the sample of 60 schools: 1. SMT consent form; 2. Teacher consent form; 3. SMT interview guide; 4. Classroom observation guide; 5. Classroom walk-through checklist; 6. Learner workbook and exercise book assessment tool; 7. Survey with teachers; 8. Classroom library assessment; 9. School DQA tool. The tools are outlined as follows: 1. SMT consent form The SMT members (who are available for interview) will be given a consent form which will provide them with the full details of the study as well as capture consent. This form will allow the SMT to indicate whether they are willing to participate in a short 20-minute interview. The consent form provides information on the following before obtaining consent: a. Who we are b. What we are doing c. Your participation d. Confidentiality e. Risks/discomforts f. Benefits 535 g. Who to contact if you have been harmed or have any concerns h. Consent 2. Teacher consent form Each teacher asked to complete the teacher survey will be given a teacher consent form, which will provide them with the full details of the study. This form will allow the teacher to indicate whether she/ he is willing to complete the survey. In addition, the teacher consent form will be administered to those whose lessons will be observed by the fieldworker. The teacher consent form provides information on the following before obtaining consent: a. Who we are b. What we are doing c. Your participation d. Confidentiality e. Risks/discomforts f. Benefits g. Who to contact if you have been harmed or have any concerns h. Consent 3. SMT interview guide The SMT interview aims to gather information from all sampled schools to compare the support experienced by teachers in the SMT treatment group to those outside of this group. The SMT will provide insight into coaching (where relevant) and the role of the Voluntary Services Organization (VSO) in providing mentors (where relevant), and will be asked to describe any resultant changes within the school – both in terms of teacher practice and school management. Using a structured interview guide, education researchers will gather information across a range of assessment domains. The SMT interview guide collects information on the following areas: a. SMT training b. Establishment of Professional Learning Communities c. RSP Coaching d. RSP LTSM 536 4. Classroom observation guide The Classroom observation guide will be administered in all schools within each sampling arm to understand how the different treatment options are leading to possible changes in the classroom. The classroom observations will be carried out in two Grades, in Home Language (HL) and English First Additional Language (EFAL). This guide will draw heavily from the EGRS classroom observation instruments. The evaluation team will review the EGRS instruments and note where the EGRS observation instrument picked up differences between the control and intervention groups. The questions related to these observation items will be included in the tool. 5. Classroom walk-through checklist Junior researchers will conduct a classroom walk-through in all sampled schools to observe classroom facilities and materials. This instrument will be used to guide the classroom walkthrough, and comprises a checklist of items and their presence/ absence and quality/ quantity. 6. Learner workbook assessment tool This tool will provide a guideline/checklist on reviewing learner workbooks to gain a sense of the complexity and completeness of learner work completed during class. 7. Teacher survey To supplement the teacher group discussions, teachers will be required to complete a short survey to rate the RSP along dimensions such as quality and quantity of services received. The results will be compared from 2019 to 2020 to gage improvements. The tool captures teachers’ experience with the RSP LTSM, training, coaching, SMT support and classroom libraries (the latter where relevant) 8. Classroom library assessment This tool will comprise a structured template, containing information on: • Whether there is evidence that the classroom libraries are being used as intended; • Whether there is a working system for students to check out books; and • Whether the quality of the materials are durable enough for the classroom environment. The tool will be used to rate the quality of classroom libraries, where they exist. 9. School, District/Province and FPD DQA tools Each of these tools will be tailored and used with respective groups (teachers, SMT, district, province, and FPD consortium) to collect data related to the RSP implementation. The monitoring data (attendance data / visit logs / coach feedback / participant feedback forms, etc.) of the FPD consortium will be reviewed to determine if the teacher training / teacher coaching / SMT training / district training / volunteer support happened as planned – i.e. that the anticipated number of contact sessions were conducted for the specified duration (dosage), with the targeted number of participants (coverage), with quality of input, within the planned time frame. Questions will include 537 scaled responses (i.e. “No-Not at all”“, partly”“, Yes, completely”“, N/A”) with supporting narrative substantiating the response. 10. Cost & time analysis interview guide (structured) This tool with provide a structured guide with questions that gather information on estimates on the Level of Effort required to carry out RSP activities, time and processes taken for activities. The process maps developed during the design evaluation will serve as a foundation for developing key questions. The tool will be administered to the FPD consortium and stakeholders in the field including the North West DBE. DATA HANDLING AND COLLECTION Data collection, analysis, and reporting will be carried out in two evaluation processes (design/inception and implementation evaluation): DESIGN EVALUATION (INCEPTION PHASE) In the design evaluation, evaluation team members will collect data from primary and secondary sources: • Primary data collection includes interviews with key stakeholders and a design workshop with FPD consortium members, provincial/district representatives, the evaluation team, the DBE and USAID. • Secondary data sources include project documents provided by the DBE and FPD, as well as literature explored in the mini-literature review. The data collected will be consolidated and submitted in a design report submitted to the client. IMPLEMENTATION EVALUATION Fieldworker Recruitment Khulisa will use a mix of Gauteng and North-West Setswana speaking fieldworkers or education researchers (herewith used interchangeably) with the experience and qualifications necessary to successfully carry out this assignment. Minimum standards are required for the recruitment of high-quality fieldworkers with the experience and expertise necessary to conduct this assignment. These differ between the mid￾level education researchers and the junior education researchers, and were agreed upon with the DBE. 538 Table 62 Education researcher profiles Level Profile/Qualification Requirement Rationale Mid Minimum 5 Years’ experience with bachelor degree OR 10 years’ experience without degree with experience in the education sector, previous experience with conducting observations, familiarity with electronic data collection. Ideally university post-graduate candidates or recent post-graduates such as PhD students (education, languages, etc.). Fluency in Setswana and English. • Ensures an in-depth understanding of Foundation Phase teaching and classroom practices. • Conducting classroom observations and coach case studies requires a specialized approach. Junior Minimum 3 Years’ experience with bachelor degree OR 5 years’ experience without degree with experience in the education sector, familiarity with electronic data collection. Ideally university candidates/recent graduates (education, languages, other). Fluency in Setswana and English. • Ensures the ability to collect the data on workbooks, classroom libraries, classroom walk￾through and verify data as part of the DQA Fieldwork for 2019 will be conducted over two phases, as described in the Fieldwork section of this document. Mid-level education researchers will shadow all 14 coaches from the coaching intervention and coaching plus SMT intervention groups during phase 1, and phase 2 will entail school level data collection and contextual data collection (i.e. interviews district/provincial level and other key stakeholders). Schools will be mapped using Global Positioning System (GPS) coordinates provided by RSP and clustered according to their location. Education researchers will be sent out to conduct the fieldwork in and will be assigned to the clusters and reviewed based on ongoing fieldworker recruitment and assessment. Upon completion of fieldwork training, the fieldwork schedule will be finalized. The Project Manager will monitor adherence to the fieldwork schedule on a daily basis in order to be reactive to unexpected challenges in the field, adjusting the schedule accordingly. Education Researchers will report directly to the Fieldwork Manager and Fieldwork Coordinator who in turn will work in cooperation with the Evaluation Coordinator, and Project Manager. The Fieldwork Manager and the Fieldwork Coordinator will offer technical, logistical, and administrative support remotely to the Education Researchers. Khulisa has found that creating constant, open lines of communication between project staff and education researchers allows the team to quickly and effectively address challenges in the field. Khulisa will create a WhatsApp group for education researchers and project staff to discuss issues and solutions on a real-time basis. The Fieldwork Manager will monitor adherence to the fieldwork schedule on a daily basis in order to be reactive to unexpected challenges in the field, adjusting the schedule accordingly. Training and preparation for fieldwork Fieldwork for the implementation evaluation will take place in two phases, further described in the Fieldwork Section of this document. Below are the training and preparation requirements described per phase. 539 Phase 1 training: Coach shadowing and school piloting Khulisa will compile a preliminary fieldworker list prior to fieldworker training. In planning for fieldworker training, Khulisa will compile a fieldworker training manual which will include information on the project background, the data collection methodology, all instruments and tracking protocols, and the use of electronic devices and software for data collection. Khulisa will also obtain and configure all handheld devices. Khulisa will hold a pre-training meeting to ensure the trainers are familiar with the tools, processes, and procedures for the training workshop. Khulisa will host the five day training workshop at the Khulisa offices in Johannesburg. A group of 10-12 mid-level education researchers will be trained, from which the final 8 researchers will be selected based on their performance and a post-training evaluation. Three days of the workshop will be dedicated to the instruments, while the fourth day will entail fieldwork simulation, and the fifth will focus on administration and logistical arrangements. The last day of training will look at the degree to which the researchers are able to accurately assess learner workbooks, conduct classroom and training observations, and assess classroom libraries. This will ensure high-quality data collection. Most instruments will be adapted from instruments used by the DBE in EGRS. However, the coach shadowing tools used during phase 1 require development. Phase 2 training: Fieldwork at sample schools For phase 2, Khulisa will follow pre-workshop preparations followed in phase 1, as well as an inter-rater reliability quality check. Khulisa will host a five-day fieldworker training workshop at a training venue in the North-West province. The 8 mid-level education researchers (used during phase 1) as well as 8 junior level education researchers will attend the training, from which the final 12 fieldworkers (6 mid and 6 junior) will be selected based on their performance and a post-training evaluation. The training structure remains similar to phase 1 training however, this will include simulation data collection in schools as part of the training. The tools to be used during this phase will have been piloted prior to data collection. The use of electronic devices and digital software for data collection will be built into both training workshops. Fieldwork Fieldwork for the Implementation Evaluation will take place in two phases, described below. Phase 1: Coach Shadowing and school piloting (2 September 2019 to 13 September 2019) This includes Mid-level education researchers shadowing all 14 coaches and piloting the tools. In week one of coach shadowing, seven (7) education researchers will shadow one coach each over a period of four days. In week two of coach shadowing, seven (7) education researchers will shadow one coach each over a period of four days. The timing of this exercise is demonstrated below. 540 Figure 20 Proposed Coach Shadowing Process In addition to coach shadowing, seven mid-level education researchers will administer the evaluation tools and conduct all other school-level evaluation activities in one pilot school each during day 5 of week 1. The eighth fieldworker will conduct their pilot test during day 5 of week 2. This will allow for field-testing of the tools in 8 pilot schools in total. In week 2, six (6) researchers will conduct teacher group discussions during Day 5, i.e. one education researcher will go to two different schools on their Friday and conduct two different teacher group discussions. By the end of Week 2, the team would have collected data from 12 teacher group discussions. The 12 schools will be drawn from the coach shadowing schools. The sample for the pilot has been drawn by the DBE from schools which are outside of the main sample for fieldwork (i.e. pilot schools will be a subset of those schools that were not selected to be part of the main sample). At the same time, sampling will be purposive to include school location. This will ensure that education researchers are logistically able to get to the school on time to conduct the piloting and teacher group discussions. Phase 2: Fieldwork at sample schools (7 October – 18 October 2019) For the full fieldwork in the sample schools, 6 teams of education researchers (one mid-level paired with one junior-level researcher – i.e. 12 fieldworkers in total) will carry out the assignment. The research teams will be split into two districts. Khulisa understands that there may be more schools in Ngaka Modiri Molema district, so teams may not be distributed evenly across districts (e.g. currently allocated 3 teams per district). To increase efficiency, sampled schools will be mapped using Global Positioning System (GPS) coordinates clustered according to their location. Each team will be assigned to clusters of selected schools. All 12 researchers will conduct school visits over two weeks. In addition, one team (2 people) will spend one week observing teacher training during the school holidays. This will occur during the school holidays prior to fieldworker training. Another team (2 people) will carry out all provincial and district-level DQA activities as well as return to any schools that could not be accessed on an initial visit. This will occur the week after fieldwork ends. A final team (2 people) will spend one week interviewing head coaches and subject advisors. Below is an illustrative fieldwork plan for one-day data collection involving a junior and mid￾level education researcher at a school level. Week 1 Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5 Researcher 6 Researcher 7 Day 1 Day 2 Day 3 Day 4 Day 5 District 1, School 1 District 1, School 2 District 1, School 3 District 1, School 4 District 2, School 5 District 2, School 6 District 2, School 7 Week 2 Researcher 1 Researcher 2 Researcher 3 Researcher 4 Researcher 5 Researcher 6 Researcher 8 Day 1 Day 2 Day 3 Day 4 Day 5 District 1, School 9 District 1, School 10 District 1, School 11 District 2, School 12 District 2, School 13 District 2, School 14 District 2, School 8 District 1, Coach 1 District 1, Coach 2 District 1, Coach 3 District 1, Coach 4 District 2, Coach 5 District 2, Coach 6 District 2, Coach 7 District 1, Coach 9 District 1, Coach 10 District 1, Coach 11 District 2, Coach 12 District 2, Coach 13 District 2, Coach 14 District 2, Coach 8 541 Table 63 Illustrative one-day plan for data collection Time Activity Day before school visit 1. Confirm the venue of the school as well as the estimated travelling time. 2. Call the school to find out exactly where the school is from a landmark. 3. Charge the tablets overnight 4. Arrange all the documents and other things you will need - Classroom observation guide (including use of LTSM) - Classroom walk-through checklist - Teacher FGD guide covering o use of LTSM - all treatment arms plus the control arm o teacher training – teacher training and coach treatment arms o coach support – coach treatment arm - School DQA tool - Classroom library assessment tool (administered to all schools in the sample that have a classroom library - SMT interview guide - Rapid coaching / RSP quality survey - Teacher and SMT consent forms - Stationery 5. Ask accommodation for a packed breakfast if you have to leave before breakfast is served TBC – based on distance from school 6. Leave accommodation to arrive at the school by 7h00 This gives you enough time to complete introductions and arrange the day’s activities 7:00 7. Introduce yourself and purpose of visit to principal, Foundation Phase HOD, and teachers. 8. Obtain consent from teachers and the SMT to participate in the study 9. Confirm venues for interviews and observations 10. Ask the principal to show you where the venues are for the various assessments No later than 8:00 11. Begin school data collection Mid-level education researcher: Classroom Observation, HL Junior Researcher: DQA against SOPs and FPD monitoring data, administer teacher surveys – surveys include feedback on SMT and coaches (capture teacher surveys electronically) Around 10:00 Mid-level education researcher: Classroom Observation, EFAL By 12:00 Mid-level education researcher: SMT interview(s) Junior Researcher: conduct a classroom walk-through to determine visibility of LTSM, assess classroom libraries, assess 4 learner workbooks (capture classroom walk-through/library assessment & workbook analysis electronically) 13:00 11. Say goodbye to principal, teachers, HODs and sign log book 13:30 12. Travel to accommodation for the night 15:00 13. Capture notes electronically: teacher group discussions (mid), principal & SMT notes: (junior) 16:00 14. Update evaluation coordinator on the day’s events 16:15 15. Capture classroom observation electronically (mid), Capture DQA findings in the assessment template, including notes, electronically (junior) 17:00 16. Prepare for next day as per start of this schedule 542 *Teacher Group Discussions will be conducted during Phase 1 of fieldwork, in Week 2 of the Coach Shadowing. The tools/instruments will be administered using electronic devices, where relevant. Fieldworkers will have one laminated copy of the SMT and teacher questionnaire, which will be handed to the teacher/ SMT member while the interview is conducted to allow the teacher/ SMT member to follow along. Completed instruments will be automatically and instantly uploaded to a Khulisa-based server, allowing the Project Manager to monitor data completeness and quality in real-time. The tablets will be distributed to each fieldworker. One tablet will remain at Khulisa with the Project Manager to spot-check any relevant queries. Data preparation and reporting Before data collection, the Project Manager and Evaluation Coordinator will develop and follow a Quality Assurance Surveillance Plan (QASP), which will be developed in collaboration with USAID and the DBE. During fieldwork, the Project Manager will check, screen and clean the delivered data on an ongoing basis to ensure quality, accuracy, and completeness. This approach serves to 1) decrease the amount of time required post-fieldwork to prepare the data for submission and analysis, and 2) allow for ongoing quality control and real-time course correction. The data will be extracted in CSV format on a daily basis and shared with the DBE and USAID, on request. Education researchers will communicate daily with the PM, Evaluation Coordinator and Fieldwork Manager on the days’ activities. During Phase 2, at the end of every day, education researchers will capture notes electronically before submitting to the Fieldwork Manager. Within a timeframe agreed between the DBE and USAID, Khulisa will transmit the relevant data and technical materials to USAID, as required. ETHICAL CONSIDERATIONS In line with the Common Federal Policy for Protection of Human Subjects in research (the “Common Rule”) – 22 CFR 225, Khulisa’s practice is to uphold high ethical standards that are aligned with the type of respondents involved in an evaluation. Adult Participation The objectives of the study will be explained to adult participants (e.g. principals, teachers) before they make an informed decision on whether or not to participate. All participants will be expected to indicate their informed consent before taking part in the study. Participants will also have the freedom to answer or not to answer any question. Participants will be assured of confidentiality of information since no data for an individual respondent will be published and questionnaires will only be handled by members of the DBE and Khulisa, including trained fieldworkers. While the names of participants will be captured, this information will only be used to link data with a particular participant and for the purposes of returning to the correct participant to conduct follow-up research. In data sets that are submitted to USAID, all identifying information will be removed. Child Participation 543 Khulisa is sensitive to the fact that while no child will participate in this evaluation, the environments in which the evaluation will take place will have young children present. Photos taken will not focus on children’s faces, and Khulisa will ensure compliance with internationally recommended ethical standards on involvement of children in research. The ‘do no harm’ principles shall be applied and address the possible risks that children might encounter when they are participating in development projects. 544 ANNEX 8: CALCULATION OF PROJECT SCHOOLS 545 According to the RSP design specification of May 2019, there were supposed to be 263 participating schools in the RSP. A total of 140 of these are supposed to be coaching schools and a total of 65 of these are supposed to be schools that also received SMT support. 1. The DBE list of schools37 contains the names of 251 Schools. This includes a total of 136 coaching schools, and 115 LTSM only schools. Table 64: Number of RSP schools as per DBE source Intervention Condition Total Coaching 69 Coaching & SMT 67 LTSM only 115 Grand Total 251 The intervention condition is as per the DBE allocation on 12 March 2019. 2. The FPD list of schools contains the names of 270 Schools, but 14 of those schools had been removed because they were either multi-grade schools, had a LOLT other than Setswana, or they participated in another initiative. The number of “Active” schools in the FPD database, is therefore 256. When the Active FPD List and the DBE list is combined, a total of 258 schools are listed. When those schools who do not appear on either the DBE list or the FPD is removed, a total of 248 schools are listed. Table 65: Number of active RSP schools in a merged list of FPD and DBE schools STATUS Coaching Coaching & SMT LTSM only Grand Total Active 67 67 114 248 Active-Not Listed by DBE 1 7 8 Not Listed BY FPD 2 2 Grand Total 70 67 121 258 Note: Intervention condition as per FPD schools’ list 28 October 2019, except for two schools not listed by FPD 37 “List of all schools for Khulisa” provided by Stephen Taylor, November 22, 2019. This was the list which was used to draw the evaluation sample. 546 2.1 Two schools are listed by DBE, but not by FPD – These are allocated to the coaching condition by DBE. Eight schools are regarded as active by FPD, but these were not on the DBE list, only one of these schools are allocated to the coaching condition by FPD. 2.2 There was one school with a conflicting treatment condition. One Primary school in NMM is regarded by FPD as a coaching school, while the DBE regards it as an LTSM only school. The analysis shows that there are conflicting reports about the number of schools in the project. For all calculations in the Formative Evaluation Report, the evaluation team referenced the number of schools as per the May 2019 design specification - that is 263 schools. The DBE advised that the FPD database is the most up to date source for the actual number of schools in the program. The FPD "Revised list of schools for FPD October 28, 2019" reported being active in 256 schools. 547 ANNEX 9: CALCULATION OF COACHING DOSAGE 548 The DBE design specification assumes that it is possible to deliver up to 150 working days of coaching in 2019 and up to 170 working days of coaching in 2020. This means that with 14 coaches, up to 2,100 coaching days can be delivered in 2019 and up to 2,380 coaching days can be delivered in 2020. By end of September 2019, the design specification would have expected 80 percent or 1 680 coaching days to have been delivered. Given that the amount of actual teaching time in schools is likely far less than the 150 or 170 working days per annum (as school programs and disruptions may reduce the estimated number of teaching days significantly) and that the number of days for which the coaches are available for coaching is less than the 150 or 170 working days per annum (coaches deliver teacher training, and they take part in their own training and continuous professional development, spend time on administrative tasks), it would be more realistic to assume that 120 coaching days would be available for 2019 and 140 for 2020. With 14 coaches, it would be expected that 1,680 coaching days could be delivered in 2019 and 1,960 coaching days in 2020. By end of September 2019 approximately 80 percent of the year one visits would have had to be delivered – this is 1 344 coaching days. If three teachers are supported at each visit, 1 344 x 3 = 4 032 individual coaching sessions would have been expected by the end of September 2019. Altogether, 135-140 schools participate in the RSP coaching intervention. Therefore, it is possible to deliver eight to 12 days of coaching support to each school in 2019 and ten to 14 days of coaching support to each school in 2020. If three teachers were individually coached at each of these visits, one would expect four to six individual coaching sessions per teacher for 2019, and five to seven individual coaching sessions per teacher in 2020. By September 2019, 80 percent of the planned visits would have been conducted, meaning that, on average, each teacher would have 3.2 to 4.8 individual coaching sessions. The following table summarizes the described scenarios: 549 Table 66: Calculation of RSP coaching dosage Implementation Year School days per year Effective school days per year Schools per coach Support visits per school Number of teachers Days support to each teacher Individual coaching support sessions Group coaching support sessions Year 1 150 120 10 (may be as high as 14 per coach) 12 (may be as low as 8) 6 2/grade Two days 6 x 1 hour sessions (may be as low as 4) 12* Year 2 170 140 10 (may be as high as 14 per coach) 14 (may be as low as 10) 6 2/grade Two to three days 7 x 1 hour sessions (may be as low as 5) 14* Total 26 Five days 13 x 1 hour sessions 26* * It is assumed that all Foundation Phase teachers will attend all SBWs. It is possible that coaches will target some of the workshops to teachers in a specific grade only. This would result in a lower dosage. For 2019, four to six individual coaching visits per teachers would have been expected. By end of September 2019, an average or 3.2 to 4.8 individual coaching support visits would have been expected. 550 ANNEX 10: REPORTED RSP PARTICIPATION BY CURRICULUM / SUBJECT ADVISORS 551 A structured interview was conducted with six Curriculum Advisors which provided information about their actual participation in the RSP activities directed at them. Details are provided below. Reference Group Meetings: Four reference group sessions took place in 2019, one per term. Reportedly, three of the six interviewed CAs participated in only three of these meetings. Most of them reported attending most of the sessions. Table 67: Number of Reference Group meetings attended by interviewed Curriculum Advisors Curriculum Advisors Max. Number Reference group meetings Number of Reference group meetings attended Percentage of Reference group meetings attended Average number Reference group meetings attended Curriculum Advisor 1 (HL) 4 3 75% 3.5 meetings Curriculum Advisor 2 (HL) 4 4 100% Curriculum Advisor 3 (EFAL) 4 3 75% Curriculum Advisor 4 (EFAL) 4 3 75% Curriculum Advisor 5 (EFAL) 4 4 100% Curriculum Advisor 6 (HL) 4 4 100% Coaches training sessions for 2019 Up and until October 2019, four coach training sessions were completed. Two of the six interviewed Curriculum Advisors claimed to have attended all four sessions, with one not attending any of the coaching sessions. Table 68: Number of Coach Training sessions attended by interviewed Curriculum Advisors Curriculum Advisors Max. Number of coach training sessions that could be attended Number of coach training sessions attended % of coach training sessions attended Average number Reference group meetings attended Curriculum Advisor 1 (HL) 4 1 25% 2 training sessions Curriculum Advisor 2 (HL) 4 0 0% Curriculum Advisor 3 (EFAL) 4 1 25% Curriculum Advisor 4 (EFAL) 4 2 50% Curriculum Advisor 5 (EFAL) 4 4 100% Curriculum Advisor 6 (HL) 4 4 100% However, interview data suggests that most CAs only drop into the coach training sessions, and do not stay for the full time period. Data suggest that one of the EFAL CAs attended also some of the HL training. 552 Table 69: Reported attendance of Curriculum Advisors at coach training sessions Curriculum Advisors Max. Number could be attended Number of days of Coaching sessions attended % attended Average Curriculum Advisor 1 (HL) 12 3 25% 3.7 days Curriculum Advisor 2 (HL) 12 0 0% Curriculum Advisor 3 (EFAL) 8 1 13% Curriculum Advisor 4 (EFAL) 8 2 25% Curriculum Advisor 5 (EFAL) 8 12 150% Curriculum Advisor 6 (HL) 12 4 33% Coach monitoring and support of Training sessions with teachers in 2019 Of the six curriculum advisors that were interviewed, half (3) attended four training sessions with teachers. One attended three sessions, and another attended two sessions. One indicated attending but did not specify the number of sessions attended. Table 70: Number of teacher training sessions attended by interviewed Curriculum Advisors Curriculum Advisors Max. Number could be attended Number of Teacher sessions attended % attended Average Curriculum Advisor 1 (HL) 4 not specified 2.8 training sessions Curriculum Advisor 2 (HL) 4 2 50% Curriculum Advisor 3 (EFAL) 4 3 75% Curriculum Advisor 4 (EFAL) 4 4 100% Curriculum Advisor 5 (EFAL) 4 4 100% Curriculum Advisor 6 (HL) 4 4 100% One curriculum advisor was not sure of the number of days attended, whilst the average number of training days attended by a curriculum advisor was 7.2 days. Two curriculum advisors attended 12 days for the four training sessions, which is double than what took place. Table 71: Attendance at teacher training sessions by interviewed Curriculum Advisors Curriculum Advisors Max. Number of days could be attended Number of days of Teacher sessions attended % attended Average Curriculum Advisor 1 (HL) 4 days x 3 grades = 12 5 42% 7.2 Curriculum Advisor 2 (HL) 4 days x 3 grades = 12 not sure Curriculum Advisor 3 (EFAL) 2 days x 3 grades = 6 6 100% Curriculum Advisor 4 (EFAL) 2 days x 3 grades = 6 12 200% Curriculum Advisor 5 (EFAL) 2 days x 3 grades = 6 12 200% Curriculum Advisor 6 (HL) 4 days x 3 grades = 12 8 133% Data 553 Of the four training sessions, three took place over a three-day period and one took place over five days. The maximum number of days which could be attended for the coaching training sessions is 14 days. Curriculum Advisors had attended 26 percent of the training days of the four training with some attending either one day of a training or up to three days. The average number of days attended by the six interviewed Curriculum Advisors across all four trainings is 4.4 days. 554 ANNEX 11: STATEMENT OF DIFFERENCES There were no statements of differences at the time of writing this Formative Implementation Evaluation report. 555 ANNEX 12: TEACHER TRAINING MATRIX 556 1 2 3 4 5 6 7 8 Date 30-Sep-19 30-Sep-19 30-Sep-19 30-Sep-19 30-Sep-19 01-Oct-19 02-Oct-19 02-Oct-19 Venue Seasons (room 4) Seasons (room 1) Seasons (room 5) Seasons (room 2) Seasons (room 3) NG Kerk Lichtenburg (Room 1) Stilfontein, Ngwenya Hotel (Room 1) Stilfontein, Ngwenya Hotel (Room 4) Grades Grade 1 Grade 1 Grade 1 Grade 1 Grade 1 Grade 2 Grade 3 Grade 3 Subjects HL & EFAL HL & EFAL HL & EFAL HL & EFAL HL & EFAL HL & EFAL HL & EFAL HL & EFAL Total Teachers Expected 44 28 30 52 56 35 47 52 Total Teachers Present 32 20 24 27 42 32 36 41 Total HODs Present 4 4 2 2 2 5 3 3 Total SA's Present 2 1 1 1 2 2 0 0 Head Coach Present 2 2 2 2 2 1 0 0 Master Trainers Present 0 0 0 0 0 1 0 0 Languages used by coaches during training English & Setswana English & Setswana English & Setswana English & Setswana English & Setswana English & Setswana English & Setswana English & Setswana Session Observed Paired and independent reading Overview of highlights for term 4 lesson plan: lesson preparation & time table for HL & EFAL, Overview of reading methodologies (Generic HL & EFAL) Overview of reading methodology (Generic, HL & EFAL) Welcome and Introduction, Distribution of the material, Overview of highlights for term 4 lesson plan: lesson preparation & time table for HL & EFAL Paired and independent reading Full day of observation. All sessions were observed. Welcome and Introduction & Reflection on Term 3 & Logistics for distributing the Term 4 Lesson Plans & Resource Packs [HL & FAL]. Overview of highlights for term 4 Lesson Plans: Lesson Preparation & Timetable [HL & FAL]. SBA/ Lesson Preparation/ Model Books/ Informal Assessment/ Final Summative Assessment Overview of Reading Methodologies [Generic, HL & FAL]. Application Activity: Use lesson plan, prepare demo reading lesson [HL: Group Guided]/ [FAL: Pair & Independent] – clear instruction: simulate a life in the day of a coach supporting a teacher [Group Work]. Application Activity: Bi-Literacy Teaching Skills Quiz Game [Groups showing knowledge, HL &FAL questions mixed. What if? Supplying ideas & tips for needs based workshop [Handout]. Conclusion Duration of observation 1h10 1h40 55m 1h18 1h30 8h 3h40 3h40 Training venue set up 3 2 2 2 2 3 3 3 557 1 2 3 4 5 6 7 8 Agenda made available NA NA NA 3 NA 3 3 3 Name tags given out 1 1 1 3 1 1 1 1 Attendance register signed 3 3 3 3 3 3 3 3 Absentees 2 2 2 2 2 2 2 2 Teachers received LTSM? 3 3 3 3 3 2 3 3 Coach prepared 3 3 3 2 3 3 3 3 Presentations well prepared 3 3 3 3 3 3 3 3 Copies of training materials available 3 3 3 3 3 2 3 3 Training materials in a file/folder/stapled 2 2 2 2 2 2 2 2 Handouts appropriate 3 3 NA NA 3 3 3 3 Use of facilitators manual NA NA NA NA NA NA NA NA Reference to teacher training manual NA NA NA NA NA NA NA NA Warm -up activities conducted NA NA NA 1 3 1 2 2 Purpose and structure of training made clear NA NA NA 3 NA 3 NA NA Teachers introduce themselves NA NA NA 2 NA 1 NA NA Style of training participatory 3 3 3 2 2 3 3 3 Coach establishes rapport with teachers NA NA NA NA NA 2 NA NA Coach listening skills 3 3 2 NA 3 3 2 2 Variety in how presentations are made (not monotonous) 3 3 3 3 3 3 3 3 558 1 2 3 4 5 6 7 8 Interest and excitement in the room 3 3 3 3 3 3 3 3 Trainer confident 3 3 3 3 3 3 2 2 Pace of training right for learning 3 2 3 2 3 3 3 3 Synthesis of inputs at the end of a section 3 3 3 NA 3 3 3 3 Training content includes all aspects of lesson plan and use of LTSM 3 NA 3 NA 3 3 2 2 Addresses teachers' reading needs in Phonemic awareness, Phonics, Vocab, Fluency, Comprehension 2 3 3 NA 2 3 NA NA Link between reading and writing addressed NA 2 1 NA NA 1 NA NA Addresses concepts and approaches to "read aloud" NA NA 1 NA NA 1 3 3 Addresses concepts and approaches to paired and independent reading 3 NA 3 NA 3 2 3 3 Addresses concepts and approaches to group guided reading 3 2 3 NA 3 1 3 3 Addresses concepts and approaches to promoting reading engagement NA NA 2 NA NA NA NA NA Addresses how teachers can use a range of 1 NA NA NA NA 1 2 2 559 1 2 3 4 5 6 7 8 question types with learners Addresses how teachers can generate rich discussion on books 1 NA 1 NA 1 1 NA NA Encourage teachers to discuss their own practices and use examples 1 1 1 NA 1 1 1 1 Encourage teachers to revisit vocab and readings to consolidate learning NA NA NA NA 3 NA 1 1 Addresses inclusive education, pacing, curriculum coverage, techniques for teaching literacy NA 1 1 NA NA 2 2 2 Builds skills according to objectives of training 3 3 3 3 3 3 NA NA Builds teachers' content knowledge and literacy concepts 3 3 3 3 3 3 NA NA Builds teachers' pedagogical ability 3 2 3 NA 3 3 NA NA Questioning by trainers becomes deeper and more reflective as day progresses 3 NA 3 NA 3 3 1 1 Provide time and space for practical application (role play / presentations etc.) 3 NA 2 NA 2 3 3 3 560 1 2 3 4 5 6 7 8 Differentiates training based on teacher needs and contexts NA NA 3 NA NA 3 NA NA Knowledge is scaffolded - built up in incremental blocks 3 3 3 2 3 3 NA NA Activities approprately timed 3 2 3 NA NA 3 3 3 If videos used, well received? NA NA NA NA NA NA NA NA Sequencing of HL and EFAL complement teachers' learning 3 3 3 NA 3 3 3 3 Evaluation form made available to teachers NA NA NA NA NA NA NA NA Did literacy head coach provide support and/or feedback 2 2 2 NA 2 3 NA NA Sas/Cas provide support and/or feedback 3 3 3 3 3 3 NA NA