This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Khulisa Management Services, (Pty) Ltd. FINAL EVALUATION REPORT DESIGN EVALUATION OF THE READING SUPPORT PROJECT November 2019 Contract No: 72067418D00001, Order Number: 72067419F00015 BRANDON BARNARD PHOTOGRAPHY Contract No: 72067418D00001, Order Number: 72067419F00015 iii READING SUPPORT PROJECT: FINAL DESIGN EVALUATION REPORT DESIGN EVALUATION OF THE READING SUPPORT PROJECT November 2019 Photo Credit: Brandon Barnard Photography AUTHORS Benita Williams (Lead Technical Evaluator) Jennifer Bisgard (Project Director) Katharine Tjasink (Senior Project Manager) Leticia Taimo (Evaluation Coordinator) Margaret Roper (Senior Evaluator) Jacqui Dornbrack (Reading Expert) Janet Orr (International Reading Expert) Nombulelo Baba (Coaching Expert) CONTACT DETAILS Katharine Tjasink 26 7th Avenue Parktown North Johannesburg, 2196 Telephone: 011-447-6464 Email: ktjasink@khulisa.com Web Address: www.khulisa.com DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Contract No: 72067418D00001, Order Number: 72067419F00015 iv ABSTRACT From July 2019 to October 2019, Khulisa Management Services (Pty) Ltd conducted a Design Evaluation of the USAID-funded Reading Support Project (RSP). The key evaluation question answered in this report is: Is the Reading Support Project Theory of Change likely to lead to the anticipated results? Khulisa employed a qualitative research design. Findings include: • RSP and Early Grade Reading Study (EGRS) have similar understandings of reading acquisition and how teachers’ capacity to teach effectively can be enhanced. There are differences between these programs, including length and frequency of training, and the type and weighting of support provided. • While the RSP Theory of Change is based on evidence about reading acquisition and effective teacher development strategies, the quantum of desired change is not specified, neither is teachers’ experience, urban or rural schools differentiated, nor the inclusion of previously participating EGRS teachers considered. There are other implementation risks for the RSP that must be addressed for sustainability. • The success of the RSP Theory of Change depends on the uptake of the lesson plans, materials, and classroom libraries. The program assumes good quality training, materials are delivered, and that teachers have an incentive to implement. The quality, nature, and dosage of coaching may influence the degree to which the RSP is able to replicate or exceed the EGRS. Key words: South Africa; Education; Early Grade Reading; Design Evaluation; Theory of Change; Process Maps Contract No: 72067418D00001, Order Number: 72067419F00015 v CONTENTS EXECUTIVE SUMMARY 1 SECTION 1: INTRODUCTION 6 SECTION 2: BACKGROUND OF THE EARLY GRADE READING STUDY I AND THE RSP 7 2.1 EGRS I 7 2.2 RSP 8 SECTION 3: EVALUATION METHODOLOGY 11 3.1 PURPOSE AND QUESTIONS 11 3.2 EVALUATION APPROACH AND DESIGN 11 3.3 DATA COLLECTION 12 3.4 DATA ANALYSIS AND VALUING 14 3.5 LIMITATIONS 14 3.6 BIAS AND CONFLICT OF INTEREST 15 SECTION 4: REVIEW OF READING LITERACY IN SOUTH AFRICA 16 4.1 STATUS OF LEARNER LITERACY IN SOUTH AFRICA 16 4.2 EARLY GRADE READING CHALLENGES IN SOUTH AFRICA 19 4.3 RESPONSE TO IMPROVING LITERACY IN SOUTH AFRICA 23 4.4 PROGRAM RESPONSE TO THE LITERACY CRISIS IN SOUTH AFRICA 26 4.4.1 GAUTENG PRIMARY LITERACY AND MATHEMATICS STRATEGY (GPLMS) 26 4.1.2 NATIONAL EDUCATION COLLABORATION TRUST (NECT) 28 4.1.3 EARLY GRADE READING STUDY (EGRS I) 2015–2017 29 4.1.4 PRIMARY SCHOOL READING IMPROVEMENT PROJECT (PSRIP) 2016 - 2017 33 4.1.5 EARLY GRADE READING STUDY (EGRS II) 2017 – 2019 34 4.1.6 READING SUPPORT PROGRAM (RSP) 2019 – 2020 35 4.5 INTERNATIONAL EARLY GRADE READING IMPROVEMENT INITIATIVES 35 4.5.1 TEACHER TRAINING 36 Contract No: 72067418D00001, Order Number: 72067419F00015 vi 4.5.2 LTS MATERIALS 37 4.5.3 COACHING 37 4.5.4 SCHOOL MANAGEMENT TRAINING 37 4.6 SUMMARY 38 SECTION 5: DESIGN EVALUATION FINDINGS 40 5.1 EGRS I AND RSP: COMPARING THE THEORIES OF CHANGE 40 5.1.1 EGRS I 40 5.1.2 THE RSP AND EGRS I IMPACT THEORIES OF CHANGE 41 5.1.3 RSP PROCESS THEORY OF CHANGE AND KEY ASSUMPTIONS 56 5.1.4 HOW THE RSP DIFFERS FROM EGRS I 65 5.1.5 HOW THE RSP ALIGNS WITH THE EGRS I 67 5.1.6 IS THE RSP TOC COHERENT AND COMPREHENSIVE? 68 5.1.7 IS THE RSP TOC LIKELY TO LEAD TO THE ANTICIPATED OUTCOMES? 72 5.2 MAPPING OF THE KEY PROCESSES IN THE RSP 89 5.2.1 PROCESS MAP 1: DEVELOP, PRINT AND DELIVER LESSON PLANS AND OTHER LTSM 90 5.2.2 PROCESS MAP 2: RECRUITMENT AND TRAINING OF COACHES 93 5.2.3 PROCESS MAP 3: TEACHERS TRAINING 95 5.2.4 PROCESS MAP 4: COACHING TEACHERS 98 5.2.5 PROCESS MAP 5: PROFESSIONAL DEVELOPMENT OF SMTS 101 5.2.6 PROCESS MAP 6: PROFESSIONAL DEVELOPMENT OF CURRICULUM ADVISORS (CAS) 103 5.2.7 PROCESS MAP 7: CLASSROOM LIBRARIES 105 SECTION 6: CONCLUSIONS 107 SECTION 7: RECOMMENDATIONS 114 Contract No: 72067418D00001, Order Number: 72067419F00015 vii TABLE OF FIGURES FIGURE 1 STATUS OF LEARNER LITERACY IN SOUTH AFRICA 17 FIGURE 2: THE READING ACQUISITION THEORY OF CHANGE. SOURCE: DBE TECHNICAL REPORT, JUNE 12, 2018 41 FIGURE 3: THE EGRS I THEORY OF CHANGE (TOC) 42 FIGURE 4: THE RSP THEORY OF CHANGE (TOC) 43 FIGURE 5: RSP THEORY OF ACTION 52 FIGURE 6: PROCESS MAP 1 90 FIGURE 7: PROCESS MAP 2 93 FIGURE 8: PROCESS MAP 3 95 FIGURE 9: PROCESS MAP 4 98 FIGURE 10: PROCESS MAP 5 101 FIGURE 11: PROCESS MAP 6 103 FIGURE 12: PROCESS MAP 7 105 TABLE OF TABLES TABLE 1: CALCULATION OF RSP COACHING DOSAGE 54 TABLE 2: RSP PROCESS THEORY OF CHANGE (TOC) 56 TABLE 3: RSP ASSUMPTIONS 61 TABLE 4: ASSESSMENT OF RSP TOC COHERENCE AND COMPREHENSIVENESS 69 TABLE 5: ASSESSING LESSON PLANS 73 TABLE 6: ASSESSING TRAINING PLANS 75 TABLE 7: ASSESSING THE COACHING DESIGN 84 Contract No: 72067418D00001, Order Number: 72067419F00015 viii ACRONYMS AND ABBREVIATIONS AHL African Home Language ANA Annual National Assessment BWE Bernita Williams Evaluation CAs Curriculum Advisors CAPS Curriculum and Assessment Policy Statement COP Communities of Practice COS Circle of Services CPTD Continuous Professional Training and Development DBE Department of Basic Education DDD Data Driven Districts DHET Department of Higher Education DIP District Improvement Program ECF Education Collaboration Framework EFAL English First Additional Language EGRA Early Grade Reading Assessment or Activity EGRS Early Grade Reading Study ETDP SETA Education Training and Development Practices Sector Education and Training Authority FPD Foundation for Professional Development GDE Gauteng Department of Education GGR Group Guided Reading GoSA Government of South Africa GPLMS Gauteng Primary Literacy and Mathematics Strategy HOD Head of Department HSRC Human Sciences Research Council ICT Information and Communications Technology IDIQ Indefinite Delivery Indefinite Quantity ILO International Labor Office Khulisa Khulisa Management Services (Pty) Ltd JIT Just in Time LTSM Learning and Teaching Support Material M&E Monitoring and Evaluation MEL Monitoring, Evaluation and Learning Molteno Molteno Language Institute Contract No: 72067418D00001, Order Number: 72067419F00015 ix NCS National Curriculum Statement NDP National Development Plan NECT National Education Collaboration Trust NEEDU National Education Evaluation and Development Unit NRC National Reading Coalition OECD Organization for Economic Cooperation and Development OUP Oxford University Press of South Africa PERFORM Practical Education Research For Optimal Reading and Management PERFORMANCE Practical Education Research For Optimal Reading and Management: Analyze, Collaborate, Evaluate PIRLS Progress in International Reading Literacy Study PISA Program for International Student Assessment PLC Professional Learning Community PSRIP Primary School Reading Improvement Project RCT Randomized Control Trial RSP Reading Support Project RTI Research Triangle Institute SACE South African Council for Educators SA-SAMS South Africa-School Administration and Management System SLP Structured Learning Program SMT School Management Team SA South Africa TIMSS Trends in International Mathematics and Science Study TOC Theory of Change UFE Utilization Focused Evaluation UNICEF United Nations Children's Emergency Fund USAID United States Agency for International Development VSO Voluntary Services Overseas ZAR South African Rand Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |1 EXECUTIVE SUMMARY Introduction Khulisa Management Services (Pty) Ltd. (Khulisa) is pleased to present this Design Evaluation of the Reading Support Project (RSP) (RFA-674-16-000005) to the United States Agency for International Development (USAID) and the South African Department of Basic Education (DBE). RSP is focused on improving the reading teaching practices of Grade 1 to 3 teachers in Setswana and English First Additional Language (EFAL) with the ultimate outcome to improve Foundation Phase learners’ reading skills. It is implemented by the Foundation for Professional Development (FPD) with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP SA) and Voluntary Services Oversees (VSO). This Design Evaluation Report is the first step towards an Implementation Evaluation, which will review actual practice against the Design Evaluation Report findings. The audience for this report includes the implementers, education officials (at national, provincial and district level), USAID and other education stakeholders interested in improving Early Grade Reading. EGRS 1 In 2015, the DBE initiated the Early Grade Reading Study (EGRS I) to assess the effectiveness of three Setswana Home Language interventions that aimed to improve reading in the early grades. It was set up as a randomized control trial involving 230 schools. The EGRS I offered structured Setswana Home Language lesson plans together with one of the following (1) an off-site teacher training intervention, (2) an on-site teacher training and coaching intervention and (3) a parental intervention. The EGRS I was implemented in Grade 1 in 2015, Grade 2 in 2016 and Grade 3 in 2017 which was implemented in Quintile 1-3 schools in the North West Province in South Africa. The EGRS I impact evaluation (Taylor, Cilliers, Prinsloo, Fleisch, & Reddy, 2017) found, after two years of interventions, that structured learning programs aligned with the National Curriculum Statement (NCS), together with high quality reading support materials (graded reading books, flash cards, posters), can make a significant difference to learning outcomes, 1 The Early Grade Reading Study ran between 2015 and 2017 in North West Province. Subsequently, a similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |2 particularly if complemented with effective and carefully monitored coach support to teachers. There were two key EGRS I lessons. First, modelling lessons in a safe space, using lesson plans that can help teachers teach learners to read, is critical to the intervention’s success. Modelling lesson plans in a safe space refers to the teacher practicing in a safe environment, for example the training classroom, before using the lesson plans with children in a live classroom environment. Second, direct in-service training is likely better than “training of trainer” models, where a master trainer trains trainers who then train teachers. The EGRS I Sustainability Study, finalized in 2019, found that Grade 3 teachers in the original ‘training and coaching’ and ‘teacher training’ schools were almost three times more likely, compared to the control group, to report using external lesson plans and the reading series. Second, teachers in the intervention schools were significantly more likely to say that they conducted Group Guided Reading on a daily basis, were also more likely to follow the correct routines for Group Guided Reading and creative writing. RSP Implemented in 263 schools2 in two educational districts in the North West Province, the RSP is intended as a scale-up of the DBE-sponsored Early Grade Reading Study (EGRS). RSP has six focus areas. Since it is set up as a randomized control trial, not all schools benefit from all six focus areas: 1. The professional development of curriculum advisors (CA) in the two participating districts 2. The development of the leadership capacity of principals/deputies and Head of Departments (HODs) to promote a culture of reading in their schools (in 65 schools) 3. Quarterly ‘just in time’ training for teachers (from 263 schools) on the implementations of Curriculum and Assessment Policy Statement (CAPS) and lesson plans 4. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers 5. The provision of Learning and Teaching Support Material (LTSM) packages to 263 schools 6. Through the DBE, Classroom libraries are provided to 100 schools Based on these six areas, in 2019, the FPD Consortium started implementing the RSP in two educational districts in the North West Province (the same as the EGRS: Dr Kenneth 2 The number of schools was not finalized at the time of writing this report. This figure is based on the initial number of schools expected to participate in the RSP. The evaluation team understands that this number is now reduced. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |3 Kaunda and Ngaka Modiri Molema). The schools are divided into three groups, each receiving a different combination of interventions, which include: • Group 1: 123 schools (LTSM and Teacher Training only) • Group 2: 140 schools (Coaching, LTSM and Teacher Training schools) • Group 3 (a subset of Group 2): 65 schools also receiving an SMT program RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of LTSM; and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning (RSP, Attachment 5 Revised). Evaluation Methodology The Design Evaluation assesses the feasibility of the RSP reaching its intended outcomes by comparing it to, firstly, the EGRS I and secondly, good practices identified in the literature. The design evaluation will further explore the issues that arose from up-scaling EGRS I to the RSP and provide recommendations. The evaluation sets out to answer: Is the RSP theory of change likely to lead to the anticipated results? The evaluation approach is 1) a Utilization Focused Evaluation (Patton, 2008) and 2) Practical Participatory Evaluation (Weaver & Cousins, 2004). These two approaches ensure that multiple stakeholder perspectives are gathered, the evaluation process leads to credible evaluation findings from various data points, and the process and findings are useful and lead to informed decision making about the program and are used to clarify the program for educative purposes. Overall, the design seeks to inform program improvement. Stakeholders included the implementing consortium members, the National, Provincial, and District Department of Basic Education and donor agencies. Stakeholders informed the evaluation design and questions and participated in the evaluation. The evaluation team gathered primary and secondary data to address the evaluation questions. This included a mini literature review including program documentation, Key Informant Interviews, a participatory workshop to discuss and validate the theories of change and the process maps (which included review by a panel of experts), and process mapping using a basic flow-chart to document the high-level processes. Data was analyzed against the key evaluation questions. Qualitative data collected was analyzed using content analysis, pattern, and thematic analysis, as appropriate. The analysis process involved valuing the evidence against an evaluative rubric the evaluation team constructed. The rubric outlines the criteria and performance standards used to value the evidence. The rubric was based on Early Grade Reading literature and the team’s experience drawn from their relevant early grade reading projects. Findings The RSP TOC is based on evidence about reading acquisition and effective teacher development strategies, and clearly identifies the skills, knowledge, behaviors, and relationships that the RSP would need to impact. However, the quantum of desired change in the skills and behavior of teachers, SMTs, CAs, and the learners is not specified. RSP Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |4 TOC is built on the EGRS TOC, which often states actions rather than strategies such as “Coach corrects and supports teacher” rather than “Coach analyses, evaluates and guides teacher to adjust instruction”. The TOC is not differentiated for more experienced and less experienced teachers, for urban or rural schools, and also does not consider how teachers who were previously part of the EGRS, and are now part of the RSP, should be approached. There are differences between these programs, including length and frequency of training, and the type and weighting of support provided. For example, the length of training for the coaching interventions differ, as does the coaching to teacher ratio, the amount and type of support to SMTs, and the CA intervention was not part of the EGRS I. The report lays out the theory of action and the process maps for each of the main components of the project and identifies potential barriers, challenges, and pathways to enable the achievement of the outcomes. The findings from the EGRS sustainability study indicate that the results are sustainable after project exit. Critical to the continued use of materials was ongoing support to teachers. Therefore, building capacity across the system is important to embed support beyond the project. A number of critical elements are highlighted which, if implemented, can contribute to the sustainability of the results. These include firstly, that teacher training contributes to the professional development of teachers through alignment to CPTD and achievement of points; secondly that the coaches provide afternoon workshops as PLCs; and thirdly that the project has systemic impact. System impact requires working at, and involving, multiple levels (e.g., schools, district, province, national) and by building relationships and strengthening linkages between role-players (e.g., teachers, HOD, Principal, coach, CA). Ultimately, the success of the RSP TOC depends on the uptake of the lesson plans, LTSM, and classroom libraries3 in the classroom. The program assumes that good quality of training and lesson materials are delivered and that teachers will have an incentive to try these out in class. For the schools in the coaching intervention the quality of the coaches, the nature and dosage of coaching may influence the degree to which it is able to replicate or exceed the success of the EGRS I. Recommendations The evaluation team recommends that the design of the RSP be strengthened to reach the intended outcomes as follows: 3 The Classroom Libraries are provided through a different Service Provider (University of the Witwatersrand School of Education) to the FPD Consortium. However, they are part of the overarching Theory of Change of the RSP program and are, as such, included in this Design Evaluation. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |5 • Ensure that the RSP maximizes the linkages with the DBE Professional Teacher Development Frameworks. • Integrate Classroom libraries into the lesson plans. • Clearly articulate the differences between coaches and CA roles and responsibilities to officials, teachers, and other school actors. • Further develop the relationship between the Coach, CA, and School Management Team (SMT) members to effectively monitor and support teaching in the classroom. • Emphasize during coach and teacher training the use of rich classroom talk and well-facilitated discussions around texts particularly in shared and guided reading (to accommodate and encourage discussion). • Promote repeated readings of a text in both training and materials, as reading skills are built when children are given the opportunity to engage repeatedly with the text. • Support, during training and coaching, teachers’ engagement with different question types and ways of engaging with texts. Coaches (and teachers) should be encouraged to ask a range of questions and elaborate on learners’ responses. • Expand training to teach effective Group Guided Reading (GGR). Teachers need to be encouraged and supported in identifying reading abilities, setting up same￾ability groups, teaching and maintaining routines around GGR, ensuring that the other children are meaningfully engaged while one group is busy with the teacher, choosing relevant Graded readers. Teachers should know how to monitor progress among learners, and how to move children between groups depending on progress. • Continue to develop coaches’ skills. This includes just-in-time training, ongoing professional development workshops, enhancing support from Head Coaches, and practicing the delivery of training. Initiatives to train coaches on formal coaching methodologies should be a priority. • Consider coaching continuity. Coach turnover during the course of implementing the RSP is likely and needs to be addressed. Strategies for when coaches are ill need to be devised in order to maintain dosage. • Provide guidance to coaches about how to prioritize their support. For example, newer teachers and untrained teachers should receive more support. • Reiterate with coaches that afternoon workshops should happen during all school visits. The content of the workshops may be based on assessment of the needs of the school, but the assumption is that all schools can benefit from these workshops. • Compile a pack of possible training items for the coaches’ afternoon workshops, which is designed to address topics such as promoting comprehension, classroom management, teaching routines, types of questions, and encouraging rich talk in the classroom. • Encourage coaches to facilitate reflective practice, rather than duplicating the compliance driven support provided by CAs. Coaches may need to be trained so that they know how to facilitate reflection in individual and group settings. • Consider using the SMT component to promote parental support. The RSP should help SMTs plan for engaging parents and the community in a campaign to promote reading. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |6 SECTION 1: INTRODUCTION Khulisa Management Services (Pty) Ltd. (Khulisa) is pleased to present this Design Evaluation of the Reading Support Project (Award Number: RFA-674-16-000005) to the United States Agency for International Development (USAID) under Indefinite Delivery Indefinite Quantity (IDIQ) Contract Number: 72067418D00001, Order Number: 72067419F00015. This report is the second deliverable for Task Order Three (Annex 1) under the PERFORMANCE IDIQ contract: Design and Implementation Evaluation of the Reading Support Project (RSP) (RFA-674-16-000005). To support the Government of South Africa (GoSA), USAID is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve the reading skills of primary grade learners. Khulisa was awarded the Practical Education Research for Optimal Reading and Management: Analyze, Collaborate, Evaluate (PERFORMANCE) IDIQ contract. The contract aims to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from the PERFORM interventions. For the period 2019 to 2020, USAID commissioned Khulisa to conduct a design and Implementation Evaluation of the RSP. The Design Evaluation was commissioned to ensure that the design of the RSP, as it has evolved, is well understood prior to conducting an Implementation Evaluation. The Design Evaluation documents the agreed design, compares it to the design of the Early Grade Reading Study (EGRS), and notes some issues that need to be tracked during the Implementation Evaluation. It is also supplemented by a mini-literature review which is used to inform the evaluation. The RSP is an educational program implemented in 2634 schools in two educational districts in the North West Province (i.e., Dr Kenneth Kaunda and Ngaka Modiri Molema) over the period November 1, 2016 – September 30, 2020. It focuses on improving the reading teaching practices of grade one to three teachers in Setswana and English First Additional Language (EFAL) with the ultimate outcome to improve reading skills of Foundation Phase learners. The Foundation for Professional Development (FPD) implements the RSP in partnership with the Molteno Language Institute (Molteno), Oxford University Press of South Africa (OUP), and one international partner, Voluntary Services Overseas (VSO). The project is intended as a scale-up of EGRS which was implemented in 230 Quintile one to three schools in the North West Province in South Africa. The Design Evaluation was carried out by an independent evaluation team (profiles of the team are included in Annex 2 and statements of conflict of interest are included in Annex 3). 4 This figure is based on school numbers provided to the evaluation team at the start of the evaluation. The number of schools has subsequently changed. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |7 The team consists of evaluation specialists from Khulisa Management Services and Benita Williams Evaluation (BWE), as well as subject matter experts - two South African consultants with experience in designing and implementing Early Grade Reading interventions in South Africa, and an international Early Grade Reading specialist involved in the design, implementation and evaluation of Early Grade Reading projects in Africa, the Middle East and Asia. The Design Evaluation begins with a short introductory overview of the EGRS and RSP, followed by a mini literature review. The literature review explores reports, documents, and literature relevant to the evaluation and includes an examination of the two interventions and the data generated to date. The purpose of the review is to provide the evaluation team with an understanding of the perspectives of the implementers and the progress the program has made. Thereafter, the Design Evaluation documents the comparisons of the two interventions theories of change, theories of action, mapping of the key processes, and provides recommendations. SECTION 2: BACKGROUND OF THE EARLY GRADE READING STUDY I AND THE RSP One of the biggest developmental challenges facing South Africa is the high number of children who do not learn to read for meaning in the early years of school (PIRLS, 2016). Reading for meaning is the foundational skill upon which all others build and has therefore become a leading priority for the South African Department of Basic Education (DBE). 2.1 EGRS I5 In 2015, the DBE initiated EGRS I to assess the effectiveness of Setswana Home Language interventions that aimed to improve reading in the early grades. It was set up as a randomized control trial (RCT) involving 230 schools located in two districts in South Africa’s North West Province. The EGRS I offered structured Setswana Home Language lesson plans together with one or a combination of the following (1) an off-site teacher training intervention, (2) an on-site teacher training and coaching intervention and (3) a parental intervention. The EGRS I design is graphically depicted below: 5 The EGRS ran between 2015 and 2017 in North West Province. Subsequently, another similar intervention called the EGRS II was implemented in Mpumalanga. For this reason, this report refers to EGRS I throughout when referring to the initial EGRS study. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |8 The EGRS I was implemented in Grade 1 in 2015, Grade 2 in 2016 and Grade 3 in 2017. When first evaluated and tested in an RCT, the EGRS I showed promising findings with regards to the importance and usefulness of training teachers off-site and coaching them on￾site and the improved reading for meaning. 2.2 RSP In 2016, USAID awarded RSP to FPD (award RFA-674-16-000005 between November 1, 2016–September 30, 2020). Funded through a USAID Cooperative Agreement, the RSP aimed to support the DBE in its effort to improve the reading skills of Foundation Phase grade learners. RSP’s overarching goal was to improve the reading skills of primary grade learners in African Home Languages (AHLs), as well as in English as a first additional language (EFAL). In 2016, three events occurred that influenced the RSP’s mandate and strategy. First, the Progress in International Reading Literacy Study (PIRLS) released a report that had damning findings. Essentially, the report stated that the majority of Grade 4 learners in the South African education system were unable to read with meaning. Second, the on-site teacher training and coaching intervention of the EGRS I had shown promising results which encouraged USAID to further their understanding on how coaching teachers contributes to improved reading for meaning. Third, the DBE prioritized teacher coaching interventions that focused on improving reading for meaning. These combined factors resulted in USAID and the DBE significantly revising the initial RSP mandate to align better with the EGRS I. Some of the RSP components – like School Management Team (SMT) training and involvement of Curriculum Advisors (CAs) were maintained, but the design of the lesson plans, offsite teacher training and on-site coaching and support was meant to be aligned with the EGRS I. While the RSP is still designed to Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |9 improve teacher effectiveness and quality to support better learner results in AHLs and EFAL, the strategy to achieve that goal has changed. The revised RSP is designed around six focus areas. Since it is set up as an RCT, not all schools benefit from all six focus areas: 1. The professional development of CAs in the two participating districts 2. The development of the leadership capacity of principals/deputies and Head of Departments (HODs) to promote a culture of reading in their schools – in 65 schools 3. Quarterly ‘just in time’ (JIT) training for teachers (from 263 schools) on the implementations of Curriculum and Assessment Policy Statement (CAPS) and lesson plans 4. The provision of 14 literacy coaches to 140 schools to offer classroom-based support to Foundation Phase teachers 5. The provision of Learning and Teaching Support Material (LTSM) packages to 263 schools 6. Through the DBE, Classroom libraries are provided to 100 schools The RSP design is graphically depicted below6 : 6 As of June 26, 2019 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |10 Based on these six areas, in 2019, FPD started implementing the RSP in 263 schools in two educational districts, Dr Kenneth Kaunda and Ngaka Modiri Molema, in the North West Province. Further information on the EGRS and RSP is provided later in the report. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |11 SECTION 3: EVALUATION METHODOLOGY 3.1 PURPOSE AND QUESTIONS The Design Evaluation assesses the feasibility of RSP reaching its intended outcomes by comparing it to (1) the EGRS I and (2) good practices identified in the literature. The evaluation also explores the issues that arose from up-scaling EGRS I to the RSP and provides recommendations. The report is intended to bring about discrete decisions about the program and the evaluation, and to clarify the program for educative purposes. The evaluation sets out to answer: Is the RSP theory of change (TOC) likely to lead to the anticipated results? To address this question, the following sub-questions were used to guide the evaluation • What is the RSP’s TOC? • What are the intended changes and the expected causal pathways? − Is the coaching design likely to lead to the anticipated impact? − Is the training design likely to lead to the anticipated impact? − Are the lesson plans and other learning materials likely to support the achievement of the anticipated impact? • Is the TOC coherent, comprehensive, and scalable? • Does the RSP design depart in material ways from the EGRS I? • What are the inputs, activities, outputs and expected short term outcomes? − What are the steps in the main delivery processes? − Where are the main implementation risks in the processes? The evaluation team proposed the evaluation questions after an initial Design Evaluation Workshop was held with stakeholders from the DBE, USAID, the implementing consortium (August 2, 2019). The questions were further refined based on a close examination of the proposed TOC and a review of an initial design document submitted to USAID in March 2019. 3.2 EVALUATION APPROACH AND DESIGN Evaluations are (or should be) based on a good theory; otherwise they are just research with an opinion attached, or at worst, a haphazard process. Theory informs the methods chosen, the decisions made in the field, how data are analyzed and, importantly, how an intervention is valued. The evaluation team gathered empirical data from multiple sources at different levels, triangulated that data to answer each key question, and then synthesized the findings to present a comprehensive evaluative narrative. The evaluation approach is firmly grounded in evaluation theory appropriate to this assignment. Two evaluation theories guided the evaluation: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |12 • Utilization Focused Evaluation (UFE) guided the team’s overall process and decision￾making. In UFE, the team places a high value on ensuring that the evaluation process as well as the findings are useful to those involved (actionable). – An evaluation is considered good if it is used as intended by the intended users. • Practical Participatory Evaluation informed the approach on how to engage key stakeholders. In this approach, the evaluation team works with major stakeholders at key steps in the evaluation process to enhance evaluation use. These two approaches supported the evaluation team to address the evaluation’s objectives: (1) UFE ensured that the process and findings are useful, and (2) the participatory approach ensured that the team gathered multiple stakeholder perspectives. Combined the two approaches supported a process that led to credible evaluation findings by engaging a diverse group of stakeholders and gathering data from various data points (e.g., literature, documents, interviews). Specifically, the evaluators worked with key stakeholders to implement an evaluation that sought to inform program improvement. Stakeholder involvement The stakeholders consulted included: implementing consortium members, the National, Provincial and District Departments of Basic Education, and the donor agency, USAID. The evaluation team selected stakeholders in consultation with USAID and Department of Education. The stakeholders were involved at various times in the evaluation. The stakeholders informed the evaluation design and the evaluation questions and participated in the evaluation (refer to Annex 4 for stakeholder roles). 3.3 DATA COLLECTION The evaluation team gathered primary and secondary qualitative data to address the evaluation questions. The data collection and analysis tools are included in Annex 5. Document and literature review The document review provided background information and descriptive data that informed the evaluation design and to the extent possible, addressed evaluation questions. The team reviewed all accessible EGRS I and RSP program documents. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |13 For the literature review, the team read published and grey literature7 with a focus on understanding the typical scope, core elements, and context of early grade reading initiatives in South Africa and a summary of findings from research on early grade programs in other developing countries. The review identified key examples through a key word search for documents published in English, between 2002 and 2019. The key search words and phrases included: • Early grade reading in South Africa • International Early Grade Reading Improvement Projects • Teacher uptake in class and factors that hinder or enable uptake • Coaching teachers • Scalability of early reading programs in education The bibliography and list of project document reviewed is in Annex 6. Key Informant Interviews The evaluation team selected key informants in collaboration with the donor, the DBE, and the implementing partner. Key informants were selected based on their ability to provide insight into the EGRS I, RSP, early grade reading and/or the South African education sector. Findings based on these data are integrated throughout the Design Evaluation report. A list of key informant interviews are included in Annex 7. Participatory Workshop The theories of change and the systems map were discussed and validated in a participatory workshop with a range of stakeholders (refer to Annex 8). A panel of experts reviewed the RSP TOC, assumptions, and systems map, and expressed an opinion on the feasibility of achieving the expected results. The results of this workshop were used to revise and update the process maps, and to determine the valuing criteria for each element of the RSP. Process Mapping The evaluation team conducted Process Mapping using a basic flow-chart to document the high-level processes. The process maps unpacked the process TOC in more detail and indicated activities and workflows relevant to key processes, including: • Developing, printing and delivering lesson plans and other LTSM 7 Grey literature refers to research and materials developed outside of academic publishing, usually produced by organizations. Examples would be unpublished evaluation reports and government reports. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |14 • Recruiting and training of coaches • Teacher training • Coaching of teachers • Professionally developing SMTs • Professionally developing CAs • Providing Classroom libraries The process maps were based on a review of project documents and interviews with key stakeholders. They are presented in the evaluation report just before the conclusion section. 3.4 DATA ANALYSIS AND VALUING Data was analyzed against the key evaluation questions. Qualitative data collected was analyzed using content analysis, pattern, and thematic analysis, as appropriate. The analysis process is a standard approach to analyzing qualitative data. An evaluation provides and evaluative judgment8 . To provide that, a clear approach to valuing the evidence is needed. The next paragraph describes that approach. Valuing the evidence The evaluation team constructed an evaluative rubric. The rubric outlined the criteria and performance standards used to value the evidence. An evaluative rubric helps to surface the values that are used to judge the merit and worth of a program and its components, and provides a transparent, systematic way to arrive at valid and credible judgments. The evaluation team recognizes that multiple stakeholders could potentially have competing values at stake. To engage with and make sense of those opposing views, the evaluation team developed the rubric based on Early Grade Reading literature and the team’s experience drawn from their relevant early reading education projects. These rubrics are extensively described in this Design Evaluation. 3.5 LIMITATIONS There are a few limitations in the methodology, which stem from two factors: 1. Khulisa proposed to conduct a materials review but this was removed from the proposal as it was to be addressed outside of this evaluation. Unfortunately, this 8 Evaluation is distinct from applied research, in that valuing results against criteria is an integral part assessing the findings Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |15 materials review by another entity has not been completed yet, as was expected, and therefore leaves a gap in the evaluation. 2. The evaluation team interviewed a range of stakeholders for the design evaluation. However, this was a sample of the population of people involved in the RSP and mainly from the perspective of the implementers. 3. This evaluation judges the design of the RSP against criteria and performance standards defined by a panel of experts. These criteria were informed by the literature and the experts' previous experience. The evaluation team recognises that other criteria could have been used, that the weighting of the criteria could have been applied differently, and that there may not be a consensus about the performance standards defined by the panel of experts. The evaluation did, however, attempt to make the criteria and performance standards used in this evaluation transparent in order to allow the reader an opportunity to interrogate the analysis. 4. At the time of writing the report, the final activity description had not yet been agreed between FPD and USAID. For the evaluation, the team used a design specification developed by the DBE (May 2019) together with interview data as the basis for the evaluation. 5. The evaluation depicts the process maps and theories of change based on information about how the processes and program are designed to work. It is possible that the actual implementation may differ from the documented design. This will be checked in the implementation evaluation component of the programme. These limitations have affected the evaluation in the following ways: • There is an underlying assumption in the RSP Theory of Change that the LTSM are of good quality and are sufficiently well organised for teachers to implement. The evaluation team is unable to test this assumption given the removal of the expert materials review. However, the evaluation team used the pedagogy expert to conduct a cursory review of the EFAL materials and this expert, in collaboration with a Setswana junior researcher from BWE, conducted a cursory review of the Setswana materials. • The evidence presented in this report is from the perspective of those interviewed. The evaluation team notes this as a limitation of the methodology. 3.6 BIAS AND CONFLICT OF INTEREST The evaluation team was able to work freely and without interference and there were known or unresolvable differences of opinion or conflicts of interest either within the evaluation team or between the evaluation team, the DBE and USAID. The evaluation team attempted to reduce the sampling bias inherent in qualitative research by ensuring that a range of stakeholders with different perspectives were interviewed, that the team provided a space for alternate views, and that the data was triangulated in an open and transparent manner (both data triangulation and investigator triangulation) using a set of valuing criteria. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |16 SECTION 4: REVIEW OF READING LITERACY IN SOUTH AFRICA 4.1 STATUS OF LEARNER LITERACY IN SOUTH AFRICA The South African (SA) basic education system consistently performs poorly on international ratings compared to their economic standing. In a 2015 Organization for Economic Cooperation and Development (OECD)9 Program for International Student Assessment (PISA) report, SA was rated 75th out of 76 countries in a comparison of 15-year old learners’ mathematics and science performance. In 2016, The Progress in International Reading and Literacy Study (PIRLS, 2016) found that 78 percent of South African Grade 4 learners cannot read for meaning. To have a deeper understanding of the above statistic, consider it when compared with other middle-income countries, such as Iran (35 percent) or Chile (13 percent). When compared with high income countries such as the United States (four percent) or England (three percent), South Africa’s attaining 78 percent is even more alarming (Spaull, 2017; PIRLS, 2016). GoSA has acknowledged the poor ratings and challenges faced in improving the rating (DBE, December 2017). 9 OECD is an inter-government organization focusing on economic development, progress and world trade. It currently has 36 members. South Africa is one of the OECD “five Key Partners since 2007 contributing to OECD’s work in a sustained and comprehensive manner”. South Africa and the OECD started a joint work program in 2018 to “promote stronger and more inclusive growth” (http://www.oecd.org/southafrica/south-africa-and-oecd.htm). The OECD conducts PISA in mathematics, reading and science every three years to gauge students mastery of subjects to prepare them for real-life situations (http://www.oecd.org/pisa/). Although South Africa has not participated in PISA, the mathematics and science rankings use international assessments including the Trends in International Mathematics and Science Study (TIMSS) which includes South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |17 Figure 1 Status of learner literacy in South Africa The poor reading ability of South African learners may be related to the fact that almost half of South African children have never had a parent read to them and that two thirds of adults are not active readers (PIRLS, 2016). Alarmingly, the inability to read for meaning also impacts learners’ ability in mathematics: 61percent of children cannot do basic mathematics in Grade 5 (TIMSS, 2015). Diving deeper into the statistics The South African findings for the PIRLS 2016, is based on a locally administered version of the PIRLS literacy assessment. The Centre for Evaluation and Assessment (CEA) at Pretoria University conducted a nationally representative study that tested 12,810 Grade 4 students from 293 schools across the country (Howie et al 2016, pg. 309). Students were tested in whatever language was used in that school in Grades 1-3, i.e., all eleven official languages were tested, and children were generally tested in the language with which they were most familiar. Nic Spaull (2017) identifies key insights from PIRLS: • South African learners cannot read for meaning. 78 percent of learners could not attain the Low International PIRLS Benchmark in reading. This means that eight of ten South African learners cannot read for meaning. For example, these learners could not locate and retrieve explicitly stated information or make straightforward inferences about events and reasons for actions (PIRLS, pg. 55). • The decline. A trend shows a declining number of SA learners reaching high levels of reading achievement. In 2011, three percent of SA Grade 4 students reached the High International Benchmark. In 2016 only two percent reached this same benchmark (PIRLS, 2016, pg. 58). • The gender gap. The South African gender gap in reading is the second highest in the world. Girls score much higher than boys in reading across the board. In Grade 4, girls are on average a full year of learning ahead of boys. This gender gap is the second largest among all 50 countries that participated in PIRLS, coming second to Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |18 Saudi Arabia (girls score higher than boys in Saudi Arabia) (PIRLS, 2016, pg. 36; Howie et al, 2016). There is also a trend that the gap in South Africa is increasing, as the gap between boys and girls was larger in 2016 than in 2011 (PIRLS, pg. 43), mainly due to SA boys’ scores declining between 2011 and 2016. While these highlights provide a more descriptive picture of the literacy challenges faced in South Africa, when interrogating these results more closely, PIRLS provides findings that describe literacy differences among the 9 provinces, and within different home languages. For example, there is a large difference between provinces regarding the percentage of Grade 4 learners who can read. In Limpopo, 91percent of Grade 4 children cannot read for meaning. There are similarly high percentages in the Eastern Cape (85 percent), and Mpumalanga (83 percent), with a lower percentage in Gauteng (69 percent), and the lowest found in the Western Cape (55 percent). Another interesting difference is found when comparing literacy in the different languages, where a high percentage of learners could not read for meaning. In Sepedi, 93 percent of Grade 4 learners tested could not read for meaning with similarly large percentages in Setswana (90 percent), Tshivenda (89 percent), isiXhosa (88 percent), Xitsonga (88 percent), isiZulu (87 percent), isiNdebele (87 percent), Sesotho (82 percent) and siSwati (83 percent) learners (Howie et al, 2016, pg. 5). By comparison, in Afrikaans 56 percent of learners could not read for meaning and English (57 percent) (Howie et al, 2016). What these numbers mean The literature identified that 78 percent of learners cannot read for meaning, which is likely to have effects on these children when they become adults, and potentially a long-term effect on South Africa’s economy. For example, South Africa is ranked 85th in the Global Competitive Index (2017/2018) stating that the skill set of graduates in 2018 is regarded as “inadequate for the progress of a successful economy” in the Fourth Industrial Revolution. (Department of Higher Education [DHET], 2018, pg. 4) Research recognizes that poor education is linked to poor economic growth. A correlation between the quality of schooling and future country wealth means that poor schooling condemns countries to a perpetual state of partial economic recession (Hanushek & Wößmann, 2007). Relatedly, literacy and learning competency are identified as worker survival skills given that in the United States 47 percent of current jobs are destined for redundancy due to technological change. The World Bank (2018) estimates that due to technology disruption, two-thirds of jobs will become redundant in the developing world (International Labor Office [ILO], 2019 pg. 9). While some jobs will become obsolete, there is potential for new work practices and jobs (ILO, 2019 pg.14). The next generation of workers would need to continuously retrain and develop cross-professional skills to remain competitive (ILO, 2019 pg. 14), placing emphasis on targeted vocational training, promoting lifetime learning, and educational reform for new industries originating from new technologies. In addition, a forward-looking education system, particularly in the area of science, technology and research will be required to address the disconnect between education systems and the needs of employers (ILO, 2019 pg. 17). This suggests the need to improve skills required by businesses, particularly skills such as computer skills, digital reading and basic coding (DHET, 2018, pg. 6). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |19 4.2 EARLY GRADE READING CHALLENGES IN SOUTH AFRICA The literature identified several nonexclusive factors that contribute to early grade reading challenges. The challenges include insufficient ‘opportunity to learn’ due to poor time management resulting in patchy curriculum coverage, weak or absent instructional leadership, lack of coordination between national and provincial efforts, inefficiencies around resources and provisioning of materials, poor sustainability and scaling of promising pilot projects from civil society reading improvement programs (Mohangi et al. 2016; van der Berg et al, 2011; National Education and Development Unit, 2018). Compounding these challenges has been a failure to realize that a poorly prepared teaching force requires considerable scaffolding (i.e., detailed lesson plans) and substantial institutional support from curricular advisory and management staff who require training in mentorship and adult education skills. A number of these noted challenges are expounded on next. • Ineffective English and mother tongue language teaching practices Poor academic performance of SA learners from Grade 4 onwards has been attributed to weak literacy skills due to ineffective English and mother tongue language teaching practices in the Foundation Phase (Reeves et al 2008). The SA policy of mother tongue instruction and additive bilingualism is evidence-based, however it faces serious implementation challenges, such as lack of skilled educators, insufficient funding, and resources (Stoop, 2017). Numerous systemic efforts (for example teacher training and the provision of learner materials) to address the poor literacy achievement demonstrated in systemic tests (i.e., Southern and Eastern Africa Consortium for Monitoring Educational Quality (SEACMEQ), PIRLS) have been implemented. • Head of Department (HOD) and school leadership The difficulties do not lie at the level of the individual teachers alone. HODs and school leadership seldom engage in classroom monitoring or monitoring of learner work, although this is part of their role (as outlined in the South African Personal Administrative Measures for Education). Commonly cited challenges include poor relationships with teachers; lack of trust in the monitoring process; lack of time for monitoring due to their own teaching loads; shortages of educators in schools; and poor understanding of HOD roles (RSP, Attachment 5 Revised). Curriculum Advisors (CAs) in particular are meant to provide both a platform for accountability and a vehicle for in-service professional development to the teachers (including HODs) under their jurisdiction through workshops, school support visits and other initiatives. However, a central weakness in the SA education system has been identified as the inability of district officials to provide support to Principals, HODs and teachers at school level. As a result, few HODs engage in meaningful curriculum management, classroom monitoring, or professional development of the teachers they are meant to support. In the absence of structured guidance and accountability, ultimately learner performance suffers (Ibid). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |20 • Curriculum coverage Additionally, while emphasis is placed on curriculum coverage, the depth of coverage is often deficient. Learners are not assigned enough practice in some areas of the curriculum, and this is not revealed through current monitoring practices of district officials and HODs; nor in teaching common scripted lesson plans to ensure that learners are exposed to the full curriculum as opposed to the current practices whereby curriculum coverage is compromised (Ibid). The issues described above focus on the school system, yet there are other challenges that lie outside those boundaries. Reading is the foundation upon which all other learning takes place. Learners cannot excel in mathematics and other subjects without being able to comprehend, with meaning, the underlying text. Teachers who do not have the pedagogical content knowledge and materials required to teach the foundational skills of reading cannot be expected to teach learners to ‘read to learn’. Further, access to school libraries, supportive parenting, school attendance, and class size influence literacy scores. Howie et al (2017) describe four factors that influence learners’ performance: 1. School libraries Almost two-thirds of PIRLS Grade 4 learners are in schools (62 percent) which do not have school libraries. Learners who did not have school libraries scored significantly lower than those learners in schools with libraries. 2. Class size Despite the policy stipulating 40 learners in a class, South African Grade 4 class sizes average 45 learners. Class size is negatively correlated with learner’s achievement. 3. School attendance Nearly one in three learners are absent from school at least once a week. These children had reading literacy scores lower than those children who were rarely (or never) missed from school. 4. Parental role Children tended to have higher literacy scores when their parents reported enjoying reading and engaged in literacy activities with their child: “In all of the languages, a positive association between parents liking reading and learner achievement exists. Nationally, there was a 52-point difference in learner achievement if parents enjoyed reading compared to those who did not. The variation across languages was considerable for Tshivenda (71 points) and Xitsonga (66 points) but less pronounced for Afrikaans (32 points) and isiXhosa (28 points).” (PIRLS 2016, pg. 156) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |21 There are different theoretical approaches to language literacy and comprehension. Not all linguistic experts take the same approach, specifically pertaining to the assessment of children’s reading ability. For example, experts disagree on EGRA as a sound measure of reading comprehension verses the PIRLS approach to assessing reading comprehension (refer to PIRLS, 2016). Therefore, the measurement of reading comprehension is influenced by theories of measurement, theories of how children learn to read and the contextual factors (as discussed above). Reading is learned rather than acquired. There is extensive research (Shin & Crandall, 2014; Snow et al, 1998) on how children learn languages, but essentially all children, whether first or second-language readers, go through the same five initial literacy steps10: 1. Awareness and exploration 2. Experimenting with reading and writing 3. Early reading and writing 4. Transitional reading and writing 5. Conventional reading and writing Children who learn literacy in their first language develop broad areas of knowledge that they can access in English (or any other second/foreign language). These are: • Visual knowledge: about print and text direction • Phonological knowledge: about sounds represented by symbols • Lexical knowledge about words and collocations • Syntactic knowledge: about meaning construction and making sense of words • Semantic knowledge: about social use of language as discourse If there is sufficient language development, many of the first language skills and strategies in reading and writing in will transfer to another language. A further challenge is the bi-modal education system in South Africa, due to deeply entrenched inequality. Elliott (2016) states that approximately 83 percent of schools are classified as ‘dysfunctional’ and consequently learners are not being equipped for meaningful futures. Learners who attend functional schools are, in general, able to attain 10 International Reading Association and the National Association for the Education of Young Children, 1998 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |22 better employment, consequently the wealthy continue to receive better education (Elliott, 2016). A final point of importance is whether poor literacy is fundamentally about the lack of funding. Churr (2015) argued that the South African education system could be categorized as a high participation, high cost, but poorly performing system. There has been a large investment by both government and civil society into basic education. The GoSA basic education budget (2016) was 204 billion South African Rand (ZAR). Trialogue11 estimates that in 2014 the bulk of Corporate Social Investment (ZAR 8.2 billion) went to education. The United Nations Children's Emergency Fund (UNICEF) South Africa indicated that the budget for school children remained at 17 percent of total government expenditure in the 2017/2018 budget. UNICEF further state that “there is growing evidence that the government is making progress to tackle financial barriers to poor children’s education” (UNICEF South Africa, 2017, pg.2). However, as the UNICEF report states, there are schools in the two of the poorest quintiles in the Western Cape and Eastern Cape “that have a large percentage of their learners achieving university-endorsed passes”. Therefore, some schools perform above the general expectation despite quality remaining an issue. Consequently, budget allocation to education on its own will not address the literacy challenges in South Africa, but rather a focus on investing in areas that will bring about the greatest impact. The findings of the EGRS I impact and sustainability evaluations described in the next section provide insight into the cost-effectiveness of the intervention. In addition, research recommendations to address literacy in South Africa from the PIRLS SA Review (Howie et al, 2017) provided eleven recommendations that link to the challenges identified. These are presented below: 1. Strengthen teaching of reading literacy and training of pedagogical content knowledge of teachers across all languages in the Foundation Phase and especially African languages. 2. Increase proportion of time spent on reading in Foundation and Intermediate phases in the curriculum as well as encourage extra-mural reading and reading habits. 3. Initiate Pre-Primary Campaigns for parents and teachers and emphasize importance of Early Literacy activities and training at pre-primary level. 4. Urgently reduce class sizes to policy stipulations and stop the “class size creep” that is occurring across all schools and provinces. 5. Increase efforts to attract younger quality candidates into teaching to address attrition. 11 https://www.trialogueknowledgehub.co.za/index.php/education-overview Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |23 6. Target interventions for high-risk populations including boys, learners living in remote rural areas, townships. Limpopo, the Eastern Cape, and Northern Cape provinces need additional support. 7. Provide and increase school resources such as school libraries and classroom libraries, especially in areas where performance is poor. 8. Review interventions on Information and Communications Technology (ICT) provision in primary schools and increase effective and sustainable access to ICT and utilization thereof in education. 9. Intervene to reduce teacher and learner absenteeism at primary schools. 10. Increase and implement programs addressing bullying at schools. 11. Campaign for greater parental involvement in school and learner activities. These recommendations are by and large addressed through the policies and programmatic responses. 12 There are a number of responses aimed at improving literacy in South Africa. The next section explores the support the US government provides to South Africa, the response by the GoSA, and program responses. For the focus of this review, the focus is on US government support, however it is acknowledged that there a many other agencies supporting the improvement of education in South Africa. 4.3 RESPONSE TO IMPROVING LITERACY IN SOUTH AFRICA US Government Support in South Africa To support GoSA, USAID is implementing the PERFORM project. The overall goal of PERFORM is to improve the reading skills of primary grade learners. USAID’s investment is in line with the US Government Basic Education Strategy (2019-2023), and with USAID’s Education Policy (November 2018) (see below). The program aligns with USAID’s commitment to “…increase the percentage of students who attain a minimum proficiency in reading …encourage schools to… teach in a language children speak and understand; assist schools in the acquisition of high-quality learning materials and textbooks; provide teachers with quality training and ongoing support; [and] encourage the use of assessments to support instruction.” (USAID Government Education Strategy, 2019, p.32) The goal of the US Government Strategy on International Basic Education (2019 to 2023) “is to achieve a world where education systems in partner countries enable all individuals to 12 The Design and Implementation Evaluations do not address recommendations 5; 8 and 10. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |24 acquire the education and skills needed to be productive members of society.” (pg. 4) To accomplish this goal, the US Government has two key objectives: 1. Improve learning outcomes; and 2. Expand access to quality basic education for all, particularly marginalized and vulnerable populations. The strategy endorses a comprehensive approach to international basic education: “Through a comprehensive approach to international basic education, the US Government uses research, data, and evidence to help empower partner countries to better respond to their unique contexts, more holistically address their education needs, and build systems that can be fully supported by country resources, paving a way toward graduation from assistance. The US Government’s approach is guided by the principles of prioritizing country ownership; engaging holistically with education systems to strengthen their capacity and performance; partnering and leveraging resources; and responding to country needs and opportunities.” (pg. 4-5) One of the core areas that the US strategy focuses on is “Working with partner countries to reform policies, improve curricula, strengthen data systems, train teachers, and help ensure students have the books and materials necessary to provide an environment conducive to learning.” (pg. 16) The focus on literacy and reading falls within the strategic priority of ensuring that children and youth gain literacy, numeracy, and social-emotional skills that are foundational to future learning and success. Furthermore, for vulnerable children, efforts focus on increasing access to quality education that is safe, relevant, promotes social well-being, and contributes to their skills to secure better jobs as adults (USAID Government Education Policy, 2019). Reading and literacy are promoted as the foundation for success in school and life. The approach is not only about supportive policies to teach literacy at a national scale, but also on building capacity for scaling and sustaining successful literacy programs based on data on current achievement levels. The policy further identifies approaches for improving reading and literacy outcomes. These include • Quality teacher instruction and ensuring instructional time • High-quality textbooks and supplementary materials • Employing a language of instruction that students use and understand, and • Using assessments to support instruction In addition to the US policy focus, the World Development Report of 2018 (World Bank, 2018) focuses on the international education crisis. In the foreword to the report, the authors argue that the focus should not be on schooling, but on learning. As stated, But providing education is not enough. What is important, and what generates a real return on investment, is learning and acquiring skills (pg. xi). The statement suggests that the focus is not only on access to education, but a renewed investment in the process of learning and teaching is required. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |25 South African Government Response In response to the findings of national and international reading assessments, in 2006 the DBE embarked on efforts to promote a culture of reading in schools. The following initiatives were implemented: • Drop All and Read Campaign • 100 Story book project • National Reading Strategy • Foundations for Learning Campaign • Pilots of Early Grade Reading Assessment (EGRA) • DBE Workbook provision • Annual National Assessment (ANA) and • National Catalogue for Grades 1 to 12 Subsequently, after the release of the National Education and Evaluation Unit (NEEDU) report (2013) on the State of Literacy Teaching in Foundation Phase, the Ministerial Reading Audit Report (2013) and the 2011 Progress in International Reading and Literacy Study (PIRLS), the Minister of Education, Angie Motshekga, declared Reading Promotion and the Library and Information Services a national priority13. The following reading initiatives were put in place: • Read to Lead Campaign • Resuscitation of the “Drop All and Read” program. In the early grades “this program is better known as the “Read me a book” campaign • The development of DBE Reading Series which is modelled along the same lines as the DBE Workbook provision • 1,000 schools offering Grade 1-3 implemented the EGRA (In June 2015) • The CAPS give high weighting to reading and writing skills in Grades R to 12 • The National Reading Plan prescribes the implementation of reading norms for Grades R-12, and • The establishment of 1000 fully functional school libraries commenced in 2015 One of the South African Government’s National Development Plan (NDP) goals is to ensure that 90 percent of Grades 3, 6 and 9 learners, “achieve 50 percent or more in the Annual 13 https://www.gov.za/speeches/remarks-minister-basic-education-mrs-angie-motshekga￾and-mp-1st-reading-roundtable Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |26 National Assessment in literacy, numeracy/ mathematics and science”. In the DBE’s Action Plan to 2019: Towards the Realization of Schooling 2030, improvement of learner performance is a clear priority area. Goal 1 of the action plan is to “Increase the number of learners in Grade 3 who, by the end of the year, have mastered the minimum language and numeracy competencies for Grade 3”. The National Reading Strategy (2008) gave rise to the Foundations for Learning Campaign and various provincial specific initiatives in the Western Cape (2006), Gauteng (2010) and recently in KwaZulu-Natal (2017). These initiatives use a ‘systemic and whole school obstacles’ approach to improving reading achievement. The DBE leads the Read to Lead Campaign and supports a range of programs and interventions to address the challenges related to literacy in South Africa. These are presented in the following section. 4.4 PROGRAM RESPONSE TO THE LITERACY CRISIS IN SOUTH AFRICA 4.4.1 GAUTENG PRIMARY LITERACY AND MATHEMATICS STRATEGY (GPLMS) The Gauteng Department of Education (GDE) developed the GPLMS14 to improve learner performance in Literacy and Mathematics in underperforming primary schools in Gauteng. GDE developed the strategy to address the literacy challenges identified in the Systemic Evaluation for Grade 3 Literacy results (2008) where 792 out of 1,347 primary schools performed at or below 40 percent15. The strategy had two aims: (1) to raise the overall performance of the province in reading and mathematics and to (2) close the gap between the historically advantaged schools and the historically disadvantaged schools (Fleisch, 2014). Based on this strategy, two implementation phases were undertaken. Phase 1 (2010–2014) The GDE, in collaboration with Non-Governmental Organizations (NGOs) delivered the GPLMS intervention / program in 792 underperforming schools (which constitute about 65 14 http://www.bridge.org.za/wp-content/uploads/2015/08/The-Basics-of-the-GPLMS-for-Bridge￾August-2015.pdf; http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S0256- 01002014000300001 15 https://www.zenexfoundation.org.za/programmes/systemic-programme/item/226-gauteng-primary￾literacy-and-mathematics-strategy Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |27 percent of all public schools). Phase I concentrated on improving teaching practice by introducing scripted daily lesson plans which explicitly mapped out teaching content, teaching methods, curriculum coverage, sequencing and remediation. All the language and mathematics teachers and learners in project schools were supported by the project. Teachers were provided with lesson plans to use in conjunction with supplementary tools and resources (e.g., textbooks and graded readers). The GPLMS employed instructional coaches to work with the teachers on a one-to-one basis in their classes in order to establish and encourage the new practices. In Year 1, the GPLMS started by providing resources (literacy only) and coaches. In Year 2, GPLMS provided lesson plans and introduced the mathematics component. In Year 3 GPLMS began providing DVDs showing best practice. A quantitative evaluation16 identified that the GPLMS approach that includes changing teacher practice through scripted lessons, instructional coaching and appropriate resources, had a positive impact on learner performance. The report then made two suggestions. First, institutionalizing the GPLMS approach within the GDE structures would likely sustain the gains identified. Second that the program should be extended for an additional five years. A qualitative study led by Hartell, Steyn and Chetty (2015), supported these findings. The study identified improved literacy levels of learners and found that coaching was the most important factor contributing to the identified results. The study also identified challenges for GPLMS, such as the fast pace of the program, large class size, marking and lack of parental involvement. The report concluded by stating that the GPLMS provided more equitable teaching and learning opportunities (Hartell, et.al, 2015). Phase II (2015–2019) The Zenex Foundation is currently supporting a second phase to the GPLMS. The two￾pronged approach used in Phase II involves: • Institutionalizing the GPLMS in order to scale the program more widely in the province. This includes a change management program to embed a differentiated GPLMS program in Gauteng. • Improving the Phase I resources, training, and support for transforming teacher practice from Phase I. Twelve projects were identified to improve resources, training, and support for transforming teacher practice, which included a focus on revising lesson plans, distributing a variety of graded readers, training school management, and district officials for teacher development and curriculum management. 16 Fleisch, Scho¨er, Roberts, & Thornton (2016) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |28 4.1.2 NATIONAL EDUCATION COLLABORATION TRUST (NECT) In 2013, the NDP called for increased collaboration among stakeholders to improve educational outcomes. In response, the Education Collaboration Framework (ECF) was produced in early 2013 through consultation among key role players in the basic education sector. The ECF has the full support of the DBE. In addition, NECT17 was established by key role players. NECT is dedicated to strengthening partnerships within civil society and between civil society and government in order to achieve the national goals for basic education in South Africa. It aims to support and influence the agenda for reform of education. Consequently, NECT is informed by the six themes of the ECF. These are: 1. Professionalizing of the teaching service 2. Supporting courageous leadership 3. Improving government capacity to deliver 4. Improving the resourcing of education 5. Involving parents and communities in education, and 6. Enhancing support for learners and promoting their wellbeing. The NECT delivers on the above six themes through focusing on five programmatic areas, one of which is the District Improvement Program (DIP). The NECT DIP promotes a structured learning approach to provide support to teachers in low resource and low capability contexts. The program provides scripted lesson plans, training, and coaching in eight districts across Eastern Cape, KwaZulu-Natal, Limpopo, North West, and Mpumalanga. The DIP provides teacher support in the form of detailed lessons plans for each curriculum component and provides coaching and training for effective implementation of the lesson plans. The SLP is its most successful innovation. The SLP was designed based on the Gauteng Primary Literacy and Mathematics Strategy (GPLMS) (discussed later in the report) and informed by the EGRS. The NECT’s work through the SLP has included the following: • Revising the lesson plans initially designed for the GPLMS drawing on feedback from districts, NECT monitoring data, and consultations with teachers and experts. The NECT has revised lessons plans for mathematics, EFAL and the following AHLs; IsiZulu, Setswana, Sepedi, Xitsonga, Tshivenda and isiXhosa. • Developing training and coaching for HODs, subject advisors and teachers and implemented it in 14,769 schools across the country. 17 www.nect.org.za Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |29 • Developing instruments to track and monitor coverage of the CAPS curriculum. In addition to this, the NECT and DBE’s18 Read to Lead Campaign established the National Reading Coalition (NRC): a self-sustaining, agile ecosystem of reading initiatives across the country. The aim is to improve coordination amongst various stakeholders to reduce inefficient overlaps and gaps in the numerous forms of support for reading, promote the adoption of approaches that have evidence of success and increase the opportunities for economies of scale19. The NECT has identified five areas as critical to improve reading20: 1. Initial teacher preparation; 2. Access to relevant resources; 3. Continuing professional development; 4. Community support; 5. Policy, Research, and evaluation. The interventions will be based on a value chain model highlighting the above-mentioned focus areas. 4.1.3 EARLY GRADE READING STUDY (EGRS I) 2015–2017 In 2015, the DBE initiated EGRS in two districts in the North West Province of South Africa. The EGRS was implemented by Class Act Educational Services and the first three waves of quantitative data collection (start of Grade 1, end of Grade 1, end of Grade 2) were conducted by the Human Sciences Research Council (HSRC). Qualitative data collection and analyses (case studies) were conducted by the University of the Witwatersrand, and the DBE analyzed the quantitative data in collaboration with academics at the University of the Witwatersrand, HSRC, and Georgetown University (United States of America). EGRS was designed as an RCT that aimed to determine which (if any) interventions improve early grade reading outcomes in home language (Setswana) in 230 Quintile21 1-3 schools. The 18 DBE Annual Report 2017/2018: https://www.gov.za/sites/default/files/gcis_document/201810/dbe￾201718-annual-report.pdf 19 ww.nect.org.za 20 ww.nect.org.za 21 South African schools are divided into five categories called quintiles based on the socio-economic status of the community in which the school is situated. Quintile 1 schools are the poorest, while quintile 5 schools are the least poor. In South Africa, the socio-economic status of the school is correlated with school functionality. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |30 study was implemented in two districts (i.e., Dr Kenneth Kaunda and Ngaka Modiri Molema) in the North West Province in South Africa. The core of the EGRS project was a comparison of the cost-effectiveness of three promising interventions to improve reading outcomes in learners’ home language (Setswana22). Each intervention was implemented in a separate group of 50 schools with a further 80 control schools. The RCT was complemented by a 60- classroom observation study and eight detailed case studies. Combined, these data enabled the researchers to estimate the impact of each intervention and to understand where, how and why different elements of the intervention models worked or not. The research team implemented one of the three EGRS interventions at a cluster of 50 schools, such that each cluster had only one unique intervention. The interventions were: • A teacher training intervention. The first intervention provided teachers with lesson plans aligned to the National Curriculum Statement (NCS) Grades R-12 and the CAPS. Additional quality reading materials were provided for teachers to use, and training of the teachers occurred twice a year. • An on-site teacher training and coaching intervention. This intervention provided teachers with the same set of lesson plans and reading materials as the first intervention. Additionally, the intervention provided ongoing support to teachers through specialist on-site coaching and small cluster training sessions. • A parental intervention. The third intervention involved weekly meetings with parents to discuss the importance of learning to read in the early grades and to empower parents with the knowledge and tools to enable them to become more involved in their child’s literacy development. Class Act Educational Services implemented these three interventions with the teachers of a cohort of learners in Grade 1 in 2015, the teachers of the same cohort of learners in Grade 2 in 2016, and the first two interventions were extended to the teachers of the same learners in Grade 3 in 2017. The parental intervention was discontinued based on the evaluation findings (discussed below). The EGRS I evaluation identified several categories of findings, as summarized below (DBE, 2019). EGRS I results The evaluation found that structured learning programs (SLPs), aligned with NCS, together with high quality reading support materials (graded reading books, flash cards, posters) can make significant difference to learning outcomes, especially if accompanied by effective and carefully monitored support by coaches. At the end of the second year of intervention (when learners were in Grade 2), coaching was found to have a statistically significant impact on aggregate reading proficiency of 0.232 standard deviations. The impacts were smaller and 22 Setswana is the common home language in the North West Province of South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |31 statistically insignificant for the Training (0.095) and Parent (0.104) interventions. Overall, learners in the classes of teachers who received two years of the coaching intervention were approximately 40 percent of a year of learning ahead of learners in schools that received no intervention. . Additional findings with regards to coaching were that: • Girls performed substantially better than boys (reading 10 words per minute more than boys), however, boys caught up to girls to some extent as the gap is smaller in the coaching group than in the control group • Middle to top performing learners in the achievement distribution benefitted the most. • Large classes benefitted most because the intervention helped teachers to provide better instruction in a challenging setting up to a threshold of 50 learners in a class. • Further, across all three arms of the study, the highest impact was concentrated in urban schools, but no measurable impact on rural settings. This suggests a different intervention is required for deep rural schools. Cost-effectiveness of EGRS I The cost-effectiveness23 analysis considered the impacts and costs of the three implementation interventions. The analysis used the Year 3 budget as the inception challenges had been resolved and fixed costs paid (for example the materials development). However, as the parent program was not implemented in Year 3 the Year 2 budget was used. Results indicate that the coaching is cost-effective in producing improvement in the comprehension test. A learner is 12.3 percent more likely to pass the comprehension test per R1000 spent, compared to the parent (6.6 percent), and training (3.3 percent) programs (DBE, August 2017, Summary Report). Factors outside of EGRS I control The study identified two factors outside of the EGRS I control that influences the intervention. First, large class sizes of 38-45 learners (in both the training and coaching intervention groups) had the largest impacts. Second, the low ratio of Subject Advisors per school makes it impossible for them to fulfil the role of reading coaches. Lessons learned The evaluation identified two key lessons. First, modelling lessons in a safe space, through lesson plans for teaching learners to read, is critical to the intervention’s success. Modelling 23 A Cost Effectiveness Analysis “…compares the relative costs to the outcomes (effects) of two or more courses of action.” It is a useful technique when benefits (such as improvement on a comprehension test) cannot be monetised. A Cost Effectiveness equation calculates the cost “per unit of effectiveness” (Better Evaluation, https://www.betterevaluation.org/en/evaluation-options/CostEffectivenessAnalysis). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |32 lesson plans in a safe space refers to the teacher practicing the language in a safe environment, for example the training classroom, before using the new language in a public space or with children in a classroom. In public, students and learners often shy away from trying the new language because they are unsure or intimidated by first language speakers. Second, direct in-service training is better than “training of trainer” models, where a master trainer is trained who then trains trainers. Need for additional research Finally, the study identified an area for further investigation: policy makers and researchers should continue to investigate how to shift parental involvement at scale. These findings suggest that EGRS 1 is one potential intervention that could influence poor literacy rates in South Africa24. Identifying sustainability In 2018 Khulisa collected data for the DBE to assess the sustainability of EGRS I. The study focused on the EGRS first cohort of learners and assessed two types of sustainability: • Whether the results from the original cohort of learners who received the intervention were sustained into Grade 4 in 2018, one year after the learners had the benefit of being taught by teachers who had received the EGRS I training and coaching interventions. • Whether a new cohort of learners in Grade 3 in 2018, whose teachers had received the EGRS I interventions a year earlier, would benefit from a sustained change in teacher instructional practices. The study identified several sustained results. First, Grade 3 learners in coaching schools had a 0.1 to 0.15 standard deviation advantage in Setswana reading and literacy (as expressed in a composite score) over their peers in control schools. Second, teachers in the Coaching and Training intervention schools were almost three times more likely, compared to the control group, to report using external lesson plans and the Vula Bula reading series. This indicates a sustained use of the materials provided through EGRS I or, at the very least, a sustained awareness of the materials they should be using. Second, teachers in the intervention schools were significantly more likely to say that they conducted Group Guided Reading (GGR) on a daily basis, were also more likely to follow the correct routines for GGR and creative writing. Thus, the study suggested that effective early interventions in reading might have benefits that last beyond the intervention and can contribute to long-term improvements in 24 https://nicspaull.com/2017/08/16/egrs-probably-the-most-important-education-researchintervention-post￾apartheid/ Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |33 educational outcomes. A core factor attributed to the continued use of materials was the provision of support to teachers through a SLP with integrated materials. In the absence of rigorous impact evidence of what makes a positive impact in South Africa’s primary schools, this is an important finding. It also confirms that DBE’s initiatives, such as the Primary School Reading Improvement Project (PSRIP) and the work done by the National Education Collaboration Trust (NECT), which makes use of similar structured lesson plans, are likely on the right track25. 4.1.4 PRIMARY SCHOOL READING IMPROVEMENT PROJECT (PSRIP) 2016 - 2017 PSRIP provides a range of support interventions to improve the quality of teaching of Home Language literacy as well as English as a First Additional Language26. In support of the sector skills plan of the Education Training and Development Practices Sector Education and Training Authority (ETDP SETA), and the strategies of the DBE, the Minister and the Director General encouraged the DBE to design a national reading program with the NECT. The ETDP SETA provided the funding for the initiative, which is implemented by the DBE The PSRIP, which began in 2016, aims to improve the quality of teaching and learning in public schools. It is a fast-paced, high-impact reading development and support program, endorsed by the South African Council for Educators (SACE). It aims to strengthen reading in EFAL in the Foundation Phase by providing structured training, materials, and classroom support to teachers. By 2018, the program had provided 11,721 Foundation Phase Teachers and 263 subject advisors with three rounds of training and support materials with the aim of improving teaching practices (PSRIP Booklet). Foundation Phase EFAL Subject Advisers are critical to the PSRIP. Subject Advisors work under the coordination of Provincial Coordinators (two per province) and, on average, support 40 teachers each on the delivery of the program. Following Subject Advisor training, Subject Advisors train the teachers at district level. Learners are then tested using the EGRA and the data are submitted to the DBE via the participating districts. It is too early in the program to ascertain the impact, however initial monitoring and evaluation activities indicate that: 1. Learners are benefitting from reading anthologies and big books provided as part of the program resources 2. There is evidence of increased curriculum coverage in the classrooms 25 https://www.wits.ac.za/egrs/about-egrs/ 26 http://nect.org.za/publications/case-studies/psrip_booklet.pdf/view Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |34 3. Learners have more written work in their workbooks and that these activities often develop out of reading exercises and activities EGRA Phase 2 in the Language of Learning and Teaching (LoLT) was incorporated into the PSRIP and rolled out in 1,670 schools. The DBE distributed the EGRA Toolkit (teacher guide, assessment charts, and stop watches) to 1,670 schools in August 2017. The aim was for approximately 300,000 learners to benefit (PSRIP Booklet). According to the DBE annual report 2017/2018, an evaluation of the PSRIP suggested an encouraging shift in the classroom practice of teachers. The evaluation found that most teachers continue to practice new EFAL methodologies and routines in their classes. Also, all learners in the study recorded improvements in the EGRA. 4.1.5 EARLY GRADE READING STUDY (EGRS II) 2017 – 2019 Building on previous successes in the North West Province that focused on the teaching of reading in home language Setswana, EGRS II focuses on supporting teachers with English as First Additional Language (EFAL) in Mpumalanga province. The assumption that is being tested in the EGRS II is that increased support on EFAL teaching will help prepare learners for the transition to English as language of learning and teaching which occurs in Grade 4. The project works with 180 quintile 1-3 schools, mostly rural, in Mpumalanga in Gert Sibande and Ehlanzeni districts. The schools have isiZulu and siSwati as their language of learning and teaching. In 2017, the EGRS II focused on providing support to Grade 1 teachers, in 2018 supported Grade 2 teachers, and in 2019 it supported Grade 3 teachers. The study was set up as an RCT, where the 180 schools were divided into two different intervention and a control group: • Intervention 1: 50 schools receive face to face coaching, printed lesson plans, and integrated LTSM • Intervention 2: 50 schools receive virtual coaching, lesson plans on tablet and integrated LTSM. • Control group: 80 control schools that continue with normal teaching. The EGRS II project, therefore, experiments with two alternate ways to support teachers: this includes providing a face-to-face reading coach or a virtual coach who contacts the teachers via call, text, and instant messaging. The aim is to understand whether the interventions are successful or not, and why. An impact evaluation for EGRS II is underway. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |35 4.1.6 READING SUPPORT PROGRAM (RSP) 2019 – 2020 In November 2016, USAID launched the Reading Support Project27 which focused on improving Foundation Phase learners’ reading skills in AHLs, particularly in Setswana, as well as in English as a First Additional Language (EFAL). The first phase of implementation entailed a pilot study in North West’s Ruth Mompati District. This pilot was concluded in June 2018 and handed over to the district on August 16, 2018. Concurrent with the implementation of the pilot, ongoing collaboration efforts were undertaken with the DBE with a view to finding better alignment with their strategic goals particularly in two districts earmarked by DBE for scaling up. The synergy between the RSP and the EGRS I led to the revision of the RSP to take to scale the coaching benefits identified in EGRS I. As a result, from 2019 onwards, the RSP program was to run in 263 schools in two districts (Dr Kenneth Kaunda and Ngaka Modiri Molema) in the North West Province. The schools are divided into three groups, each receiving a different combination of interventions, which include: • Group 1: 123 schools (LTSM and Teacher Training only) • Group 2: 140 schools (Coaching, LTSM and Teacher Training schools) • Group 3 (a subset of Group 2): 65 schools also receiving an SMT program RSP specifically aims to: improve subject matter knowledge; promote more effective pedagogic practices; improve in-class time management; increase effective use of Learner Teacher Support Materials (LTSM); and foster a school environment to support teachers’ ability to implement the full curriculum and facilitate successful teaching and learning (RSP, Attachment 5 Revised). 4.5 INTERNATIONAL EARLY GRADE READING IMPROVEMENT INITIATIVES Following are notable findings from international Early Grade Reading programs in areas that RSP has identified as target areas effecting program improvement. 27 https://www.foundation.co.za/reading and RSP Attachment 5 Revised Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |36 The USAID’s Early Grade Reading Barometer28 indicates that Early Grade Reading initiatives have been implemented in over 65 countries, with 25 of those receiving USAID implementation support. Widespread adaptations of initiatives such as those in the USAID Barometer, have increased the availability of results that are used to monitor effective system-level changes as new initiatives develop program specific theories of change. New Early Grade Reading initiatives build on lessons learned from evaluations and practices implemented in other USAID programs. In addition, the goals, operating structure and strengths of local education systems are used to build unique programs suited to the local environment and culture. 4.5.1 TEACHER TRAINING A USAID-funded synthesis of Early Grade Reading projects implemented from 2013 to 2016 found that “activities that channeled a portion of resources into two or three complementary training components seemed to have a greater chance of success than those that focus exclusively on a single type of training intervention.” (USAID, 2018, p. 19) In-service programs were the most popular, complemented by in-school mentoring. District-level coaching and pre-service training occurred in only one third of the evaluations reviewed. “Cascade training models tended to result in lower training quality and limited impact on learning.” (USAID, 2018, p. 20) The report does not provide any recommendation regarding the amount of training time / dosage, and notes that little documentation of this kind appears in Early Grade Reading evaluation reports. However, it does point out that particular attention should be paid to teacher attrition, since it was a common issue. It also points out that cascade training is likely to be less effective. The Evaluation Team International Early Grade Reading Expert (Orr29) notes that teacher training in the Jamaica’s Basic Education Project (BEP) (Social Impact, 2013), Nigeria’s Northern Education Initiative (NEI Plus) (DevTech, 2017), and Malawi’s Early Grade Reading Activity (EGRA) (Khulisa, 2015) all include reading components associated with skills that would be included in student assessments like EGRA (administered through Tangerine)30. 28 https://earlygradereadingbarometer.org 29 Orr, J. (2019) Input received for this literature review. 30 Tangerine, developed by Research Triangle International (RTI), is electronic data collection software that includes a range of subtest modules found in EGRA instruments, such as Letter Name Knowledge, Familiar word identification, Invented word decoding, Oral passage reading, and comprehension. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |37 Typically, such training focuses on the use of instructional guides or scripted lessons introduced through new textbooks; strategies to increase student time on task; oral reading fluency; letter recognition; phonemic awareness; vocabulary and reading comprehension. 4.5.2 LTS MATERIALS Most Early Grade Reading programs include the addition of new reading materials. Many initiatives collaborate with respective Ministries of Education to produce, distribute, and train teachers to use textbooks and scripted/guided lesson plans. Often the delivery of those materials to schools is delayed or not available at the time of training. Other programs establish class libraries that encourage taking the books home to practice reading. Conducting mid-term program evaluations in Nigeria and Malawi, Orr (Ibid) observed a lack of reading materials in the home. This leads to the conclusion that young readers have insufficient opportunities to practice reading at home. 4.5.3 COACHING A March 2018 USAID report, Coaching in Early Grade Reading Programs: Evidence, Experiences, and Recommendations reviewed experiences from ten Early Grade Reading programs that utilize coaching to support the improvement of instruction. The ten coaching programs in recent and on-going USAID Early Grade Reading programs in Ghana, Ethiopia, Kenya (two programs), Liberia, Madagascar, Nigeria, Pakistan, Philippines, and Tanzania highlight a variety of programmatic models for inclusion in new programs. The report recommended that new coaching programs should pay attention to: 1. Preparation and support of coaches through formal training with a focus on reading content and instruction and coaching methods. Clearly communicate coach roles and responsibilities. 2. Monitoring and evaluation of effective coaching to identify effective practices. 3. Scaling up and sustainability by providing key data to administrators. Scaffold, stagger, and adapt coach training. 4. Allow sufficient time for design, implementation, and modification of coaching programs as it may take more time to show effectiveness. Some U.S. studies (Bean et al, 2010) are concluding that the frequency and duration of coaching may influence student reading gains, but no studies have been conducted internationally at this point. 4.5.4 SCHOOL MANAGEMENT TRAINING Effective training of school management to support Early Grade Reading programs builds the capacity of managers at all levels of education programming to strengthen and sustain program goals. School management can be made up of only hired administrators, i.e., Head Teachers. It can also consist of a committee composed of parents, civil society members, school administrators, and teachers. A study of school leadership and Early Grade Reading programs in Zambia (Pouezevara, 2018, p. 100) concluded a shift from school improvement planning to learner performance improvement planning was identified as producing a clear Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |38 vision for school improvement. A second finding in that study identified Head Teachers in high performing schools as those who were more active in communicating expectations and supporting curriculum changes in the Early Grade Reading program. 4.6 SUMMARY The good practices or key factors identified in the literature include: a) Challenges (Mohangi et al. 2016; van der Berg et al, 2011; National Education and Development Unit, 2018; Reeves et al 2008) to improving early grade reading include • Insufficient time to practice reading in due to poor time management and resulting in patchy curriculum coverage; • Lack of coordination between national and provincial efforts; • Inefficiencies in the provisioning of resources; • Poor sustainability and scaling of promising projects; • Low ratio of subject advisors per school make it impossible for them to fulfil the role of reading coaches; • Requirement for rapid pacing in mandated lesson plans (GPLMS). b) Factors that influence learner’s performance include access to school libraries, class size, school attendance, and parental enjoyment of reading and engaging with their child in literacy activities (Howie et al., 2017). The findings of the PSRIP indicate that learners benefit from reading anthologies and big books and learners have more written work developed out of reading exercises and activities. The program further contributes to increase curriculum coverage. The distribution of LTSM is often challenging. c) Components that address the issue of poorly prepared or unskilled teachers includes scaffolding through detailed lesson plans, substantial intuitional support from curricular advisors, training that strengthens English and mother tongue teaching practices in the Foundation Phase, and coaching/mentoring support for behavior change. d) Educational management should be provided training in mentorship and adult education skills in particular, HODs, and school leadership engaging in classroom support, modelling good practices, and providing support to teachers. e) Key findings from the EGRS I evaluation highlight that structured programs, aligned to NCS with high quality reading support materials, can make a significant difference to learning outcomes. A key factor is if this is accompanied by effective and carefully monitored support by coaches. These findings were corroborated by findings from the GPLMS studies where changing teacher practice occurs through scripted lessons, instructional coaching, and appropriate resources (Hartell, Steyn, and Chetty, 2015). Coaching was the most important factor contributing to the identified results. International reviews recommend that the preparation and support of coaches through formal training focusing on reading content, instruction and coaching methods is critical. Additionally, the monitoring and evaluation of effective coaching to identify effective practices; scaling up and sustainability by providing key data to administrators; scaffolding, staggering and adapting coach training; and Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |39 allowing sufficient time for design, implementation and modification of coaching programs to show effectiveness are all critical factors for effective coaching to improve teacher instruction (USAID, March 2018). • Furthermore, it is important for teachers to model lessons in a safe space, and direct in-service training is better than the “training of trainers” models. The EGRS I sustainability study suggests that effective early interventions in reading may have benefits that last beyond the intervention. A core factor was the provision of support to teachers through a structured program with integrated materials. f) The EGRS I results indicate that coaching is cost-effective in producing improvement in learner comprehension tests (DBE, August 2017, Summary Report). International studies indicate that channeling resources on a few complementary training programs have a greater chance of success; cascade training models result in lower quality and limited impact on learning (USAID, 2018). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |40 SECTION 5: DESIGN EVALUATION FINDINGS 5.1 EGRS I AND RSP: COMPARING THE THEORIES OF CHANGE The RSP and EGRS I aim to improve reading outcomes for learners in Grades one to three. The programs have similar components and activities that focus on improving reading instruction in the classroom. Yet differences also exist, both in the theory and the action (i.e., activities). This section explores the TOC and theory of action for the EGRS I and RSP and then compares the similarities and differences. 5.1.1 EGRS I The EGRS I and RSP draw on the same reading acquisition theory which is articulated by the DBE as follows: An effective reader is one who reads with rich comprehension and engagement with the substance of the text. Reading comprehension is the product of two components: vocabulary and decoding. To a great extent vocabulary (and language acquisition in general) comes naturally through hearing others speaking and then emulating this. Through speaking and hearing others speaking, phonological awareness also develops; this involves sound segmentation and recall of sound patterns. This phonological awareness is important for children to learn to decode since written symbols are associated with particular sounds. Decoding thus consists of letter recognition and phonemic awareness. Unlike learning to speak, decoding does not come naturally; it is a method that must be taught systematically. It is important to emphasize that reading is produced by the product of vocabulary and decoding: If one has a perfect vocabulary but has not been taught the method of decoding one will not be able to read at all. Letter recognition and phonemic awareness are mastered through systematic teaching and consistent practice. This leads to the next stage of reading acquisition: word recognition. Through practice and appropriate progression from simpler sounds and words to more complex ones, word recognition becomes established, leading to the next phase of reading acquisition: fluency. It is only once decoding and word recognition have become fluent, even to the point where it becomes automatic and unconscious, that it is possible to reach the ultimate goal of reading comprehension. The strong empirical relationship between oral reading fluency and comprehension demonstrates this point. In order to learn the basics of decoding, a child requires a teacher who is present, capable, and motivated to deliver systematic reading instruction. In order for decoding to become fluent a child requires suitable graded materials and the discipline (perhaps imposed) to practice a lot. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |41 The figure presents a theoretical diagram illustrating how reading acquisition occurs, what supportive conditions need to be in place and how each of the interventions being evaluated in the EGRS I address key stages in the development of reading acquisition (DBE design specifications for the Reading Support Programme, 2019). Figure 2: The Reading Acquisition Theory of Change. Source: DBE Technical Report, June 12, 2018 5.1.2 THE RSP AND EGRS I IMPACT THEORIES OF CHANGE The RSP and the EGRS I impact theories of change are depicted below. The RSP and EGRS I visual graphics are based on the DBE design specification (DBE design specifications for the Reading Support Programme, 2019). The diagrams are presented one-by-one to facilitate easy comparison between EGRS I and RSP, and then supported by a narrative comparison. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |42 Figure 3: The EGRS I Theory of Change (TOC) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |43 Figure 4: The RSP Theory of Change (TOC) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |44 RSP TOC EGRS I TOC Key Differences (Materials and LTSM) IF (all) teachers31 receive CAPS-aligned lesson plans, quality controlled by a reference group, and a complete package of quality LTSM for both EFAL and Setswana HL IF teachers receive lesson plans, AND they receive integrated LTSM Similar (Classroom libraries)32 AND IF teachers (in 100 schools) receive HL and English readers from Vula Bula and African Storybook Project in the form of classroom libraries AND IF teachers (in 100 schools) receive training on selection of books matched to learners’ reading level and the management and tracking of the library (NA) Similar 31 The RSP lesson plan and training intervention predicts that there will be 6 teachers from 263 schools, i.e., a total of 1578 teachers (although the total number of teachers is unknown). 32 The RSP classroom library intervention involves 6 teachers from 100 schools, i.e., a total of 600 teachers. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |45 RSP TOC EGRS I TOC Key Differences (Teacher Training) AND IF (all) teachers attend four centralized training sessions on the use of lesson plans and LTSM per annum, which equates to a total 12 days of training spread over two years (six days in HL and six days in EFAL) IF teachers also attend centralized training sessions twice a year (which equates to four days) RSP has more frequent training sessions, and higher dosage of training, however, this is split across two languages (Teachers Coaching / Support) AND IF (some33) teachers receive at least 9-13 individual coaching34 sessions from a literacy coach spread over two years (equating to 9 – 13 hours) AND (these) teachers attend at least 18-26 targeted School Based Workshops facilitated by the coach spread over two years AND IF (some) teachers also receive ten individual coaching sessions support (equating to 20 hours) AND IF (some) teachers attend occasional group meetings with a coach and a small cluster of teachers, AND a trusting relationship develops with coaches who will effectively correct and support the teachers RSP has a lower dosage of individual coaching (9- 13 one hour visits in RSP compared to at least 10 two hour visits in EGRS I), but it is spread over two years. The RSP has a higher dosage of school-based workshops (18-26 in RSP 33 140 schools receive the coaching intervention. With the assumption of 6 teachers per school, this involves 840 teachers. 34 The dosages figures are based on the evaluation teams’ calculations based on the details in the design specification. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |46 RSP TOC EGRS I TOC Key Differences AND (these) develop a trusting and open relationship with the coach compared to 10 in EGRS I) (Support from School Management (Principal or Deputy principal and Foundation Phase HOD35) AND IF (some schools’) SMT members36 attend five to eight training sessions to improve leadership and management capabilities, AND IF these same SMT members receive tablets which will give them access to both online and offline LTSM resources consisting of graded readers and other supplementary reading materials, links to online resources and good practice videos (Not explicitly stated) RSP has a greater focus on School Managers 35 The SMT component proposed in the DBE design specifications for the Reading Support Programme, April 15, 2019, and the design described in the FPD revised technical application of July 26, 2019 differs significantly. The original design was based on the existing SMT training in the RSP and was more deliberately geared towards leadership for promoting reading. The revised SMT design focuses on more generic leadership competencies with open ended activities that depends on the emerging needs of SMTs as identified in the course of implementation. For the purposes of this report, the detail as per the FPD technical application is included, since this was confirmed as more correct during the RSP Clarificatory Workshop on August 2, 2019. 36 65 schools participate in the SMT training intervention. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |47 RSP TOC EGRS I TOC Key Differences AND IF these same SMT members receive one support visit from a VSO volunteer (changes in school managers knowledge and skills) THEN improve their understanding of their role as curriculum managers in their schools, and gain relevant skills, knowledge, values and attitudes relevant to school management and pedagogic literacy (changes in school Management practice) THEN SMT’s provide more effective development, support, and supervision of teachers in delivery of the AHL and EFAL curriculum. AND THEN these SMTs ensure an enabling and supporting environment to develop a reading culture AND THEN these SMTs can unlock systemic change and performance systems (Buy in from Curriculum Advisors) IF key District Officials receive a once-off orientation training on the project, (Not explicitly stated) RSP has a greater focus on CAs Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |48 RSP TOC EGRS I TOC Key Differences THEN they will support and encourage participation (of project beneficiaries) in project activities, thus promoting sustainability and buy-in (Support from Curriculum Advisors) IF (all) Curriculum Advisors responsible for supporting Grade 1–3 teachers in project schools attend the training of coaches and the training of teachers AND IF these Curriculum Advisors receive tablets which will give them access to both online and offline LTSM resources consisting of graded readers and other supplementary reading materials, links to online resources and good practice videos AND these Curriculum Advisors participate in the Subject Advisors’ Reference Group to provide quality assurance of lesson plans on a quarterly basis (Curriculum Advisors’ Knowledge skills and motivation) THEN these Curriculum Advisors will be motivated to fulfil their roles AND their knowledge of approaches for effectively fostering bi￾literacy outcomes would increase (Not explicitly stated) RSP has a greater focus on CAs Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |49 RSP TOC EGRS I TOC Key Differences AND their capacity to support Foundation Phase teachers and HODs in language and literacy EFAL Setswana HL will improve (Curriculum Advisors’ practice) AND THEN Curriculum Advisors will provide better support to teachers and HODs in the schools AND conduct better classroom monitoring AND conduct better monitoring of curriculum coverage (Teacher knowledge and skills changes) THEN all teachers’ skills and pedagogical knowledge will improve AND THEN they will be sufficiently prepared to teach via a SLP THEN their knowledge would be updated, AND teachers could be motivated to implement more effective teaching strategies, AND could be sufficiently prepared and motivated to teach according to lesson plans and use LTSM in their lessons Essentially similar (Teacher behavior and classroom practice changes) THEN Teachers improve the way in which they implement a SLP in their classrooms AND THEN improve curriculum coverage and motivation for their teachers THEN they will change their practices, Essentially similar Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |50 RSP TOC EGRS I TOC Key Differences AND THEN Teachers will use more effective teaching strategies AND THEN Teachers’ instruction will become more effective AND THEN teachers will provide more effective instruction to learners AND THEN they will more effectively cover the curriculum, promote individualized reading and adopt more effective teaching strategies (Parent Intervention not included) (Parent intervention) OR IF parents attend weekly meetings with teachers, (Parent knowledge and attitude changes) THEN parent knowledge and attitudes will change, (Parent support changes) AND THEN parents will change their support practices towards their children The RSP does not have a parent component (Learner exposure to reading changes) AND THEN Learners will do more individual reading (Learner exposure to reading changes) AND THEN Learners will do more individual reading Similar Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |51 RSP TOC EGRS I TOC Key Differences Learners will do more extended individual reading Learners will do more extended individual reading (Learner reading proficiency changes) AND THEN ULTIMATELY Learners’ vocabulary, letter recognition, phonemic awareness, word recognition, fluency and comprehension will improve (Learner behavior changes) AND Learners will develop the habit of reading (Learner reading proficiency changes) AND THEN ULTIMATELY Learners’ vocabulary, letter recognition, phonemic awareness, word recognition, fluency and comprehension will improve (Learner behavior changes) AND Learners will develop the habit of reading Similar The information provided in the table above is depicted in the diagram below and provides more detail about each RSP activity. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |52 Figure 5: RSP Theory of Action Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |53 Interrogating the RSP Theory of Action: Coaching days available Altogether, 140 RSP schools benefit from on-site coaching support. Based on the evaluation team’s calculations (detailed description follows), the teachers in the RSP coaching intervention are likely to receive between nine and 13 individual coaching sessions over two years, and between 18 and 26 afternoon group engagements. The DBE design specification assumes that it is possible to deliver up to 150 working days of coaching in 2019 and up to 170 working days of coaching in 2020. This means that with 14 coaches, up to 2,100 coaching days can be delivered in 2019 and up to 2,380 coaching days can be delivered in 2020. With 140 schools participating in the RSP coaching intervention, it is possible to deliver up to 14 days of coaching support to each school in 2019 and up to 17 days of coaching support to each school in 2020. Given that the actual teaching time in schools is likely far less than the 150 or 170 working days per annum (as school programs and disruptions may reduce the estimated number of teaching days significantly) and that the number of days for which the coaches are available for coaching is less than the 150 or 170 working days per annum (coaches deliver teacher training, and they take part in their own training and continuous professional development, spend time on administrative tasks), it would be more accurate to assume that in Year 1 around 8-12 school visits can be carried out by each coach and in Year 2, 10-14 school visits can be carried out (i.e., 26 school visits are feasible in a two year period). The following table summarizes the described scenarios: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |54 Table 1: Calculation of RSP coaching dosage Implementation Year School days per year Effective school days per year Schools per coach Support visits per school Number of teachers Days support to each teacher Individual coaching support sessions Group coaching support sessions Year 1 150 120 10 (may be as high as 14 per coach) 12 (may be as low as eight) 6 2 per grade 2 days 6 x 1 hour sessions (may be as low as four) 12* Year 2 170 140 10 (may be as high as 14 per coach) 14 (may be as low as 10) 6 Two per grade 2-3 days 7 x one-hour sessions (may be as low as five) 14* Total 26 Five days 13 x one-hour sessions 26* * It is assumed that all Foundation Phase teachers will attend all afternoon sessions. It is possible that coaches may target some of the workshops to teachers in a specific grade only. This would result in a lower dosage. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |55 Distance between schools supported by coaches The way schools are allocated to coaches may either enhance or detract from the actual number of coaching days per school. A ratio of one coach to ten schools (1:10) can be achieved if 140 schools participate and 14 coaches are available for the full implementation period. When the geographic spread of schools is considered, however, some coaches would only be able to support up to eight schools, which means that other coaches may have to support up to 14 schools. The coaches that visit 14 schools would then provide teachers with fewer than 12 to 14 visits per school per year. Teachers per school and grade Despite best efforts, neither RSP nor the DBE could provide the evaluation team with accurate figures on how many teachers there are per school or per grade. Therefore, the report had to rely on the following assumptions: The team assumed that there are at least two teachers per grade (though this could be higher), and if a coach carries out 8-12 school visits in Year 1 and 10-14 schools visits in Year 2, each teacher will receive four to five teaching days of support from a coach over the two years split across HL and EFAL. The individual teacher support time could be scheduled so that four to five teaching days of support could be spread over multiple occasions (such as four to six one-hour support sessions in Year 1 and five to seven one￾hour support sessions in Year 2). This assumes that the coach supports three teachers for at least one hour per teacher in one school day. Coaches could use their own discretion to decide which schools and teachers require more support and adapt their own program to suit this. Coaching visits include workshops The coaching support is not limited to individual teacher contact. According to the RSP TOC, the coach is expected to facilitate an afternoon workshop with the Foundation Phase teachers at every school visit. The content and structure of these workshops are not prescribed, and the assumption is that the coach will be able to tailor the workshops to the needs of the teachers. If a coach is able to conduct 8-12 school visits in Year 1 and 10-14 school visits in Year 2, this means that the teachers can rely on approximately 18 to 26 hours of additional support over two years. This assumes that the afternoon workshops are one hour long, that all teachers participate in all meetings (not just one grade at a time), and that the workshops are conducted during every school visit. A skilled coach would be able to use this time to provide expert input and draw on the strengths in the school to promote good reading practices. The time may also be optimally used if the coach workshops are integrated into a well-structured program of school based professional learning community (PLC) meetings. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |56 5.1.3 RSP PROCESS THEORY OF CHANGE AND KEY ASSUMPTIONS The RSP process TOC is described in the table below (Draft FPD Monitoring, Evaluation and Learning plan, March 2019). Table 2: RSP Process Theory of Change (TOC) Aims Hierarchy Summary Indicators Means of Verification/ Source Documents Focus Area: SMT Training Impact Improved learner performance in AHL and EFAL • Percentage change in learner performance • Percentage learners achieving at least 50 percent in AHL and EFAL Annual school results DBE standard assessments Outcomes Improved school management and instructional leadership • Percentage curriculum coverage • Improvements in school reading culture* (strengthened reading systems) (e.g., Drop all and Read DBE campaign; Read Aloud, reading corners; classroom and school libraries) • Management reports/documents in place (for example meeting minutes, School Improvement Plans, policies etc.) • Project reports Outputs SMT with knowledge and understanding of their roles and responsibilities in the provision of instructional leadership and management support for teachers • Schools with evidence of strengthening reading systems (testing reading outcomes, monitoring performance of teachers, supervision etc.) • Schools with reading programs in place • Feedback forms • Training assessment forms • Project reports • School support reports • Classroom monitoring reports • Curriculum management/ follow￾up • Minutes of meetings in schools • Certificates handed out Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |57 Aims Hierarchy Summary Indicators Means of Verification/ Source Documents Activities • Conduct training for HODs and Principals in leadership and management • Issue tablets loaded with course content • Provide supplementary reading / handouts where necessary • Train SMT in basic ICT • Time spent in training • Number of people trained in each session • Number of presentations/ handouts given • Attendance registers • Training manuals; SMT and ICT • Training agenda FOCUS AREA: Just in Time (JIT) Teacher Training Impact Improved learner performance in AHL and EFAL • Percentage change in learner performance • Percentage learners achieving at least 50 percent in AHL and EFAL • Annual school results • DBE standard assessments* Outcomes Improved teacher effectiveness Improved quality in teaching AHL and EFAL Improved curriculum coverage in schools • Classrooms with evidence of using scripted lesson plans and LTSM • Number of learners reached • Lesson observation forms • Curriculum tracking reports • Project reports • Additional support by SMT* Outputs Teachers with knowledge on implementation of scripted lesson plans and supporting LTSM • Number of teachers satisfied with training • LTSM (including lesson plans) distributed • Feedback forms • Training assessment forms • Training reports Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |58 Aims Hierarchy Summary Indicators Means of Verification/ Source Documents Activities • Conduct training of Foundation Phase teachers (by Grade) • Issue scripted lesson plans • Issue supporting LTSM • Time spent in training • Number of people trained in each session • Number of presentations/ handouts given • Number of LTSM packages issued, per language • Attendance registers • Training agendas • LTSM distribution forms • Standardized program FOCUS AREA: Literacy Coaching Impact Improved learner performance in AHL and EFAL • Percentage change in learner performance • Percentage learners achieving at least 50 percent in AHL and EFAL • Annual school results • DBE standard assessments* Outcomes Improvements in skills level of teachers via skills transfer by coaches • Percentage curriculum coverage (per month/term) • Percentage lesson plan coverage (per month/term) • Teacher feedback reports • Lesson Observation forms • Reporting to South Africa School Administration and Management System (SA-SAMS) • Periodic Survey (teacher satisfaction) Outputs Coaches have increased knowledge, (technical and soft skills), to coach teachers • Teachers are satisfied with coaching • Teachers and coaches establish and maintain positive working relationships • Time allocated to reading • Time allocated to teaching AHL and EFAL • Feedback forms (teacher satisfaction survey) • Lesson observation forms • Follow-up visits reports • School-based PLC reports • Project reports Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |59 Aims Hierarchy Summary Indicators Means of Verification/ Source Documents Activities Conduct coaching sessions in selected schools and Foundation Phase Grades • Time spent by coaches in lesson observations • Number of teachers coached • Foundation Phase Grades (coached) • Number of schools that receive coaching (per month/term) • Written observation • Teacher assessments • Travel logbooks • Lesson observation guidelines • Training database Focus Area: Training of Curriculum Advisor (CAs)+ Impact Improved learner performance in AHL and EFAL • Percentage change in learner performance • Percentage learners achieving at least 50 percent in AHL and EFAL • North West Provincial Department of Education • Annual school results • DBE standard assessments* Outcomes Increased capacity of CAs to support primary grade teachers in language and literacy in AHL and EFAL • Change in performance of teachers who received support from CA • CAs conduct classroom monitoring of teachers • CAs monitor and manage curriculum • Monitoring tools • School visit reports Outputs • CAs with knowledge of their roles and responsibilities • CAs with knowledge of literacy coaching* • CAs understand their roles as instructional leaders • Number of CAs satisfied with course • Number of CAs who graduate from course (Certificates given) • Assessment/feedback forms • Project reports Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |60 Aims Hierarchy Summary Indicators Means of Verification/ Source Documents Activities • Conduct training for CAs • Provide catering* • Issue out tablets loaded with course content • Provide supplementary handouts/reading material • Number of tablets distributed • Time spent in training • Number of CAs in each session • Attendance register • Training agenda • Training manual/ curriculum The RSP assumptions are documented in the draft RSP Monitoring, Evaluation, and Learning (MEL) plan for 2019. The evaluation team elicited additional assumptions in the RSP Clarificatory Workshop (August 2, 2019). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |61 Table 3: RSP Assumptions Component Assumptions Materials Activities: • All teachers receive necessary LTSM prior to the start of school term Outputs: • Teachers have the LTSM in the classrooms • Lesson plans checked and verified by CAs • The EFAL and HL content is adequately paced, aligned with the curriculum, and integrates the LTSM Outcomes • It is possible to implement the strategies in the lesson plans with large classes37 Classroom libraries Activities • All teachers receive classroom libraries in time • All teachers receive training on how to select titles for learners, and how to manage the classroom libraries Outputs • Classroom libraries contain a range of appropriate titles applicable to different reading levels • Classroom libraries are accessible and well-managed (using the book management system) to make sure all learners benefit Outcomes 37 According to the OECD, smaller size classes are often seen as beneficial because they allow teachers to focus more on the needs of individual students, and reduce the amount of class time needed to deal with disruptions. According to the DBE Norms and Standards: a primary school classroom should not have more than 35 learners per educator. The Annual Performance Plan (2018) for the North West Education Department, however, indicates that 45 percent of learners are in classes with more than 45 learners. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |62 Component Assumptions • The library management system works, and learners are able to take books home and return them in time • Teachers are able to incorporate the classroom libraries without explicit mention in the lesson plans and without compromising the use of other LTSM • Learners have the ability to read on their own Teacher Training Activities • Competent coaches deliver training with fidelity • Training venues are adequate • Teachers attend all four training sessions • Relevant teachers attend specific trainings (by Grade) • Adequate catch up sessions are implemented for those who missed training • Time allocated for training is enough to cover content, • Sequencing HL and EFAL training is optimal, and complements teachers’ learning • The sequencing of longer and shorter training sessions is optimal Outputs • Teachers are empowered and motivated to deliver learning via scripted lesson plans • Teachers refer to LTSM and noted exercises at specified periods per lesson plans • Teachers understand how to implement lesson plans and reference LTSM • Teachers spend the allocated amounts of time in teaching EFAL and AHL • Teachers understand the technical aspects of teaching • Teachers’ SACE continuous professional training and development (CPTD) points are allocated Outcomes • Teachers implement as trained • Inclusive education, pacing, curriculum coverage, and all related aspects being implemented fully Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |63 Component Assumptions Teacher Coaching Activities • Coaches have the necessary competency and skills to coach (has appropriate teaching, reading and coaching experience, familiar with the lesson plans and LTSM, able to establish good rapport) • Coaches receive the necessary supervisory support from Head Coaches/technical project staff • Travel distances between schools / coach residence are manageable • Coaches have the necessary work equipment to carry out duties • Coaches have adequate transport • Coach to school and coach to teacher ratio is manageable • Teachers can schedule time for coaches, and lesson observation takes place (minimum disruptions occur) • Teachers present positive/learning attitudes and understand how the coach’s role is different from a CA • SMT in the school are supportive of coaches • The coach is able to have individual coaching sessions with each teacher at least five times per year • The coach conducts need based school workshops every time he / she visits a school • The coaching is adequately balanced between observing, giving feedback, demonstrating and problem solving and not just a duplication of what CAs should be doing Outputs Teachers make the necessary time to allow for classroom lesson observation • LTSM is used as and how prescribed • Adequate communication between teachers and coaches • Additional support is provided by coaches where necessary (school￾based PLCs) • Reporting of term results in SA-SAMS • Identified issues are attended to in time Outcomes • Teachers show improvement in lesson coverage, pacing, curriculum coverage Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |64 Component Assumptions Elements targeted at SMT members (Foundation Phase HOD and Principal or Deputy) Activities • Competent facilitators to conduct training • Support from District to encourage participation • Foundation Phase HOD and Principal or Deputy attend all training(s) [Minimized clashes with District activities] • Clustering exercise adequately done • Course responds to identified needs of SMT and incorporates practical strategies for them to fulfil their supervisory, administrative, literacy teaching in schools • Participants apply themselves to practical exercises Outputs • SMT are motivated and apply their knowledge in their respective schools • SMT have the necessary resources to carry out their tasks • SMT receive the necessary support from international volunteers • SMT encourage reporting on SA-SAMS in their schools • SMT receive supervisory visits from volunteers and maintain a positive working relationship Outcomes • Noted reading culture in schools • Teachers receive supervisory support from SMT • Regular meetings/updates in school to increase accountability (Reporting on SA-SAMS – evidence of using data for decision making) • Schools sustain gains Elements targeted at Curriculum Advisors (CAs) Activities • Attending teacher training and coach training aligns with CAs skills development needs • CAs pay attention to training • Clashes with District activities minimized to allow for maximum attendance • CAs receive tablets loaded with relevant and useful content Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |65 Component Assumptions • CAs participate in a (virtual) Community of Practice (COP) in Year 2 Outputs • CAs use the tablets and content for supporting reading in schools • CAs understand the teacher training course and its practicality • CAs perceive the course as critical • Trained CAs are empowered to apply their knowledge in literacy/pedagogy • CAs gain the expertise to enhance instructional leadership in schools • CAs have the necessary resources to implement their roles and responsibilities Outcomes • CAs apply their new knowledge • Teachers receive capacity building on curriculum, and support from CAs Overarching • There is sufficient integration across components and actors involved in different components to work together towards improving literacy outcomes • The combination of components delivered to each school are sufficient to lead to improved learner outcomes in all classes • Participants are willing to participate in RSP activities, and see the need for RSP 5.1.4 HOW THE RSP DIFFERS FROM EGRS I The RSP differs from EGRS I in three main ways. First, the RSP is implemented on a larger scale than the EGRS I – The RSP involves 263 schools, of which 140 schools benefit from the more intense coaching intervention. A total of 230 schools participated in the EGRS I, but only 50 schools benefited from the coaching intervention. Second, the RSP focuses on both Setswana Home Language and EFAL while the EGRS I only focused on Setswana (HL). Third, the RSP involves Grade 1-3 teachers simultaneously, while the EGRS I had a Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |66 phased implementation approach38. The RSP therefore engages with more teachers at one time and addresses the teaching of reading more comprehensively. The section breaks down how these three factors play out in a practical comparison. • Coaching Support—Individual versus workshop-based The RSP has fourteen coaches, and is a larger scale program when compared to EGRS I. Therefore, RSP’s individual coaching support to teachers is less than the EGRS I coaching support. In the EGRS I four coaches conducted ten school visits per year and were required to support only one grades teachers to teach HL (e.g., only Grade 1 teachers were supported). In EGRS I, each teacher received around 20 hours of individual coaching support and received about ten hours of afternoon workshop support throughout the year. In the RSP, teachers receive 9-13 hours of individual coaching support over two years. Teachers are likely to receive 18 to 26 hours of coaching support in afternoon workshops over two years. In the RSP, the weighting of the coaching support is slanted towards school-based workshops rather than the EGRS I approach of individual support. Key point. Comparing the effectiveness of the EGRS I approach with more individual support to fewer teachers over a one year period, with the RSP approach of group support for more teachers over a two year period is not possible at the moment, but is worthwhile to investigate in an evaluation of the differential impact of the RSP. Table 4: Total number of hours support provided by coaches over full period of implementation Individual coaching support Coaching support in afternoon workshops EGRS I 10 x 2 hours = 20 hours 10 hours RSP 9-13 x 1 hours = 9–13 hours 18-26 hours • School personnel and community members engaged (narrow versus broad) The RSP is designed to engage more departmental actors than the EGRS I. The RSP has a component that targets the Foundation Phase HOD, one additional SMT member in some schools, and a component that involves CAs. The EGRS I did not have these components. 38 Phases included Year 1 interaction with grade 1 teachers, Year 2 interaction with grade 2 teachers and then in Year 3 interaction with grade 3 teachers. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |67 Key point. Through engagement with different actors at the school level, the RSP has the potential to make the RSP more effective to influence the schools. • Lesson plans that integrated LTSM Like the EGRS I, the RSP has integrated the LTSM with the lesson plans. Key point. Participant interview data suggests that the RSP’s approach to delivering lesson plans that integrate the provided LTSM is a potential strength. It is possible that this approach minimizes the effort that is required from teachers. The integration of lesson plans and user-friendly LTSM may improve uptake by teachers. • Classroom libraries The RSP delivered classroom libraries together with a one and a half-hour training on how to select materials that match learners’ reading levels and how to manage a classroom library system. In the RSP, this is delivered close to the start of the intervention. Key point. The use of Classroom libraries is not integrated into the RSP lesson plans. Consequently, teachers are not trained on how to use libraries effectively together with the RSP lesson plans. It may be possible that library integration is covered in the coaching support. • Direct training versus cascading model The RSP relies on a cascade training model (i.e., the training designers train the coaches who then in turn provide the same training to teachers) while in EGRS I the service providers who developed the training materials offered direct training to the teachers. Key point. The literature indicates that while cascade training is likely to be more cost￾efficient, it is likely that efficiency gains may be offset against losses in terms of effectiveness. DBE appears to have this concern, as the Department requested that the OUP and Molteno training material developers attend all teacher training sessions to monitor and support the coaches in delivering the training. • Parental involvement The EGRS I had a component on parental involvement, while the RSP does not include this component. Key point. The EGRS I included an initiative to boost parental support for reading, but as a standalone component it was not shown to have an impact on learner outcomes. Literature points to the fact that parental involvement is crucial in helping to foster reading habits and may work together with other interventions. This is not investigated in the RSP. 5.1.5 HOW THE RSP ALIGNS WITH THE EGRS I The RSP was required to incorporate elements of the EGRS towards a combined program. There are similarities with minor deviations which are discussed below. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |68 • Lesson plans The RSP and the EGRS I offer teachers lesson plans and LTSM. However, the RSP involves CAs to review the lesson plans. Key point. The inclusion of CAs to review the lesson plans may have several benefits 1) enhances alignment with CAPs and 2) builds the ability of CAs to support the implementation of lesson plans. • Support to teachers to implement the lesson plan The RSP, like the EGRS I, provides training to teachers to support the implementation of the lesson plans. The RSP training dosage is higher than the EGRS I training dosage, but the training is required to prepare teachers to work with both the HL and EFAL lesson plans. RSP provides six days per year (12 days over two years) while EGRS I provided four days. Key point. HL training builds upon EFAL reading strategies and it is possible that the additional dosage could reinforce classroom application in both subjects. • Coaching ratios The RSP and EGRS I have a coaching component for some schools. However, the RSP coach is required to support far more teachers, more grades, and more schools than the EGRS I coach. This likely means RSP coaches may face more difficulty in building trusting relationships with individual teachers. The extra time that the RSP coaches have with teachers during afternoon workshops may be helpful in building trusting relationships. Key point. Given the critical role of coaches in producing positive learning outcomes in the EGRS I, the RSP decision to have coaches support more teachers and provide less individual coaching should be closely monitored. 5.1.6 IS THE RSP TOC COHERENT AND COMPREHENSIVE? The evaluation team’s analytical framework, used to assess the RSP TOC coherence and comprehensiveness, is based on criteria drawn from a checklist compiled by the Theory of Change Academy39. The analysis is presented below. 39 TOC academy (n.d.) https://changeroo.com/TOC-academy/posts/expert-TOC-quality-audit￾academy. The checklist draws on work of various leading evaluators including: S.C. Funnell & P.J. Rogers (2011) Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass. Hivos TOC Guidelines (n.d.). Theory of Change Thinking in Practice https://knowledge.hivos.org/theory-change-guidelines. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |69 Table 4: Assessment of RSP TOC Coherence and Comprehensiveness Assessment Criteria Analysis Adequate Is the desired vision of success clear and detailed enough? Does it present the issue(s) to change in a clear way, making the RSP’s objective clear? The RSP TOC is based on an evidence-based TOC about reading acquisition. The RSP program goals are articulated in terms of changes they hope to see in the values, skills, and knowledge of various actors. The quantum of desired change is not specified i.e., the TOC indicated that there is an expectation of improved curriculum coverage – but the level of improvement aimed for is unclear – Is full curriculum coverage expected or only a slight improvement? No Does the TOC reflect a solid consideration of the range of different aspects that need to change in order to make the desired change possible: relationships, capabilities, values, attitudes, behaviors? The TOC outlines the skills, knowledge, behaviors, and relationships that the RSP would need to impact. Yes Is the specification of who needs to do what differently to make the desired change possible, up to date with the present context and stakeholders? The content of the training is not well specified. No Does the TOC include all stakeholders important to the change process? The RSP includes a wide range of key actors in the education system. Yes Davies, R. (2013.) Planning evaluability assessments: A synthesis of the literature with recommendations. Working Paper 40. London: UK Department for International Development. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |70 Assessment Criteria Analysis Adequate Are change strategies explicit? The RSP TOC is built on the EGRS TOC, which often states actions rather than strategies i.e., “Coach corrects and supports teacher” rather than “Coach analyses, evaluates and guides teacher to adjust instruction”. No Are the strategic priorities and the strategies themselves still fitting given the experiences to date? The TOC does not differentiate between the needs of those teachers that participated in the EGRS I before, and the general cohort. No Are the underlying assumptions appropriate? The assumptions regarding the quality of coaches, the nature of the coaching, the dosage of coaching, and the level of teachers’ participation in the RSP may not be met and may not be appropriate. No Does TOC incorporate a variety of contexts and audiences? No mention of rural vs. urban context or gender considerations. The TOC is not differentiated for novice teachers or for teachers that previously were part of the EGRS I. No Is the TOC useful to track and analyze implementation progress? Some information about dosage is specified, but the content of training and the nature of coaching is not well designed. Somewhat Is the TOC built upon current research findings? RSP builds upon the Results of EGRS I Impact Evaluation findings and USAID supported Early Grade Reading compiled research (Kim et al, 2016). Yes Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |71 Assessment Criteria Analysis Adequate Is the TOC up to date with implementation? The TOC depicted in this report is not based on a finalized agreement between the FPD consortium, the DBE and USAID and may still change40. No 40 At the time of writing this report, there was a request for modification that had not been signed off by USAID. The evaluation team expects that this will be completed imminently, in which case this criterion will be rated adequate. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |72 5.1.7 IS THE RSP TOC LIKELY TO LEAD TO THE ANTICIPATED OUTCOMES? The RSP’s success is critically dependent on providing appropriate lesson plans and materials combined with correctly pitched training to the targeted teachers. For a sub-set of the schools, coaching is provided. The coaching component’s success is critically dependent on having adequately prepared coaches providing the right kind of support at adequate levels of dosage. The available information about the lesson plans, training design and coaching are explored in this section. Are the lesson plans likely to support the achievement of the anticipated impact? Our analysis of the RSP lesson plans41 is based on a review of the HL and EFAL lesson plans for Grades 1-3 provided by FPD and Molteno. The criteria for evaluating the lesson plans are drawn from literature42 and based on the TOC for reading acquisition described earlier. 41 The evaluation did not include a formal review of the lesson plans, as this was outside the scope of the evaluation. However, our pedagogy expert, along with a junior Setswana researcher, conducted a broad review against a narrow set of criteria. Additionally, this assessment does not include the five components of reading as criteria. 42 Including the National Reading Panel Report (National Institute of Child Health and Human Development [NICHD], 2000) ; Developing early literacy: report of the national early literacy panel, Lonigan and Shanahan, 2009 CAPS CURRICULUM Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |73 Table 5: Assessing Lesson Plans Assessment Criteria Performance Standard Analysis Adequate? Maximizes reading time Reading time is maximized as per CAPS. Teachers need to incorporate reading aloud every day, but this may require integration across the curriculum. For example, Read Aloud could be incorporated as part of the life skills curriculum using stories and picture books that deal with life skill issues (e.g., friends, pollution, and respect). The lesson plans cover shared reading and Group Guided Reading (GGR) quite well. Read Aloud is not specifically timetabled, and this would be important to add. Paired reading is not specifically mentioned and should be in Grade 2 and 3 HL plans. Mostly Sufficient time for talking about texts Children need to have space to talk about what they are reading; they should be able to formulate questions and share opinions and views with each other The lesson plan is tight and while question time and discussion time is allocated, it is recommended that more time is allocated and that teachers are encouraged to facilitate rich talk. No Time for all modes of reading Lesson plans should include paired, independent, extended reading Paired and independent reading should have more focus in lesson plans and training. No Incorporates the provided LTSM Lesson Plans need to specify which texts to use and when to use them The lesson plans are based on, and incorporates the LTSM and DBE Workbooks effectively Yes Works with the classroom libraries To incorporate the classroom libraries effectively, children should have access to interesting books, talk about these books should be timetabled, and children should be allowed to take books home. The lesson plans do not deliberately interface with the classroom libraries, and therefore some teachers may need support from a coach to use the libraries effectively. Limited interview data suggests that coaches have not been trained on using the libraries. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |74 Assessment Criteria Performance Standard Analysis Adequate? Encourages repeated reading of text Children need to be encouraged by teachers and parents to reread books (not only in GGR but also during paired reading and practice at home too). Not clear from a review of the lesson plans. An issue to follow up during the Implementation Evaluation. N/A Is the training design likely to lead to the anticipated impact? The evaluation team reviewed project documents and limited key informant interview data to assess the RSP training design. The criteria for evaluating the training design are drawn from the South African and international literature on successful teacher training Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |75 Table 6: Assessing Training Plans Assessment Criteria Performance Standard Analysis Adequate? The degree to which the delivery model is fit for the context and cost effective. Trainers are deployed efficiently and effectively Cascade training is cost effective but could be done really badly. Effective cascade training includes: • Good quality instruction • Sensitivity to cultural and contextual factors • Continuing support from the original trainers (Hayes, 2000, Sri Lanka reading project) Original trainers (OUP and Molteno) may not be present at the second layer of training43. The program involves CAs, and this helps with contextual knowledge and relationships. No Teachers are grouped optimally Cluster centralized training of teachers is cost and time effective and builds communities of teachers within a geographical space that lends itself to sharing across schools The RSP makes use of this clustering strategy. Yes Training design is contextualized Training design should align with CAPS requirements, link to the guidance that CAs can give. The RSP has a reference group as the mechanism for facilitating communication between coaches, and CAs. Records of Reference Group shows good attendance by CAs. Yes 43 This has been requested by the DBE, but was still being negotiated at the time of compiling this report Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |76 Assessment Criteria Performance Standard Analysis Adequate? Schools are differentiated Coaches differentiate training based on data, particularly on reading results. The RSP does not have access to EGRS I school based data on EGRA. No Training delivery model is appropriate Most Early Grade Reading projects (EGRS, FUNDA WANDE, and PRATHAM in India) use JIT training. JIT training is adequate to prepare teachers for content they need to teach and resources they need to use in the coming term. However, because training often needs to cover a term’s work in a few days, little time is left for the deeper level issues of pedagogy and understanding coherence between the reading methods. Ideally, there should be JIT training as well as a few immersion sessions for more complex issues. The RSP applies JIT training and techniques but focuses on just the term’s content. No immersion sessions are provided. No Missed training is addressed Teachers being absent from training needs to be quickly addressed. The records show that coaches are tasked to deliver a “catch up” training with teachers who have not attended, however it is questionable how effective this catch up can be given that two days training needs to be covered in a few hours. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |77 Assessment Criteria Performance Standard Analysis Adequate? Realistic volume of content is covered in the training period 1. Training content must include all aspects of the lesson plan and use of the LTSM, including assessments. 2. Training methods should demonstrate communicative approaches and encourage participation and engagement. 3. Training should provide time and space for practical applications in the form of role play and participant presentations. PRATHAM (India Project) also stresses the need to encourage teachers to revisit vocabulary and readings in order to consolidate learnings RSP uses JIT training and thus was unable to provide the evaluators with a full training curriculum against which these elements could be assessed. The evaluators will observe training sessions for the coaches and for the teachers and thus, reserves judgment Unclear Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |78 Assessment Criteria Performance Standard Analysis Adequate? Preparation of trainers for training is adequate Preparation of trainers covers early reading content / concepts and reading and writing instruction. Ideally the content should be targeted to fill gaps in the trainers understanding of • General Early Grade reading experience. • Familiarity with the RSP LTSM and how it should be used in class. • Understand five different reading methodologies (read aloud, shared reading, GGR, paired reading, and independent) and how they work together. • Facilitation skills. • Understand the knowledge and skills gaps of teachers. • Understand how to address gaps during the school-based training. • Understand the need to keep detailed records of training. The content covered in coach preparation for EFAL and HL is good. Coaches are supposed to do “dry runs” before they present training to the teachers. Per notes from the OUP training programs and records. Coaches receive written feedback on their dry runs. Yes Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |79 Assessment Criteria Performance Standard Analysis Adequate? Addresses the practical preparation of trainers Ideally, a trainer has experience delivering the content. Videoing dry runs and then discussing them is a very useful way of training (Done in Lesson Study Japan with teachers44). Dry runs have been included in the training. Both HL and EFAL offer written feedback on coaches’ dry runs. This will be checked in the Implementation Evaluation. Yes Trainers receive clear guidelines and structured training agendas. These are provided in the form of annotated PowerPoint slide decks. Yes Provides differentiated support for trainers Some trainers are more advanced than others. Ideally, the RSP provides additional support for newer or weaker trainers. EFAL reports contain suggestions that stronger coaches are paired with those who are not as strong for the teacher training. Coaches receive further support from the Head Coaches Yes Adequately prepared to offer training to teachers A coach has experience of delivering the training before, or is prepared through practical training to deliver the material Coaches are provided with detailed training programs, annotated PowerPoint slides, and have “dry runs’ to demonstrate before offering training to teachers. Adequately prepared to offer training to teachers 44 https://www.tandfonline.com/doi/full/10.1080/19415257.2018.1430050 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |80 Assessment Criteria Performance Standard Analysis Adequate? Adequate monitoring of training A highly knowledgeable expert observes the training and gives feedback. The program collects feedback from teachers and CAs. Program includes ways of collecting this monitoring data. Head coaches and CAs observe training. It was suggested that OUP and Molteno attend all training, but this was still being negotiated as part of the design finalization Yes LTSM and Lesson plans are available on time LTSM including lesson plans are available at the training sessions. Providing materials at the training is an incentive for attendance Yes Training Content HL and EFAL Covers full range of reading practices across the Grades and across HL and EFAL Teaching reading needs to incorporate the “Big Five”: Phonemic Awareness, Phonics, Vocabulary, Fluency, and Comprehension. Additionally, there should be the various reading methods and writing and reading needs to be closely linked. Term 1–4 should cover: Shared Reading, comprehension, paired and independent reading (mentioned but not in any detail), GGR, vocabulary development, speaking and listening, songs, rhymes, phonics and phonemic awareness, and sight words. There appears to be good alignment with the LP and CAPS and content coverage is good. What is missing is “read aloud” (mentioned in CAPS but not expanded on). This is an essential part of children’s reading pleasure and vocabulary and conceptual development. Another reading type that is not explicit is “Paired reading”. Likewise, independent reading is present in documents but may need more attention as it is the ultimate goal of reading; for children to be able to read with comprehension on their own. HL: Not extensively reviewed EFAL: The documents show good content and effective teaching methods for Additional Language. They use Yes Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |81 Assessment Criteria Performance Standard Analysis Adequate? appropriate additional language methodologies such as songs, rhymes, puppets, and Total Physical response. They could focus more on explicit comprehension and fluency as well as explicitly linking HL and EFAL reading methods i.e., Coaches should assist teachers to make these connections visible and spoken about in the classroom. Pays special attention to Group Guided Reading (GGR) GGR is regarded as a very complicated methodology which requires that teachers know about baseline assessment, teaching GGR routines, setting up same ability groups, conducting GGR and ensuring that the rest of the class is engaged in meaningful learning. The EGRS I showed that teachers struggle with GGR. It is covered but not sufficiently. RSP Project Manager also commented on how the teachers are struggling with GGR. This aspect needs far more training Also, the techniques suggested in lesson plans for keeping rest of the class learning, is very limited and works mainly with the DBE books. It would be advisable to provide teachers with more activities (especially more challenging ones) for the learners to do while the teacher is busy doing GGR. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |82 Assessment Criteria Performance Standard Analysis Adequate? Promotes rich classroom talk in all classes. Children should be given opportunities to generate meaningful talk (especially around books). Hoadley’s overview of research in Foundation Phase classes shows limited opportunities for children to engage meaningfully. Teachers need to ask more open￾ended questions, encourage children to ask questions and set authentic tasks where the talk is meaningful. Can only be assessed inside the classroom. Not mentioned in any training documents, in training on question types or on how to generate rich discussion on books No Encourages reading enjoyment Key to reading frequency is reading engagement and pleasure (Wigfield & Guthrie et al, 2004; Verhoeven and Snow, 2001) A larger body of research indicates that reading motivation is key to reading success. Reading motivation and reading enjoyment is largely absent from this project. A larger body of research on reading motivation suggests that this key to reading success. No Promotes reading engagement There needs to be explicit training on promoting reading engagement (Nal’ibali includes role plays, puppet shows, oral presentations, retells, story, summaries and reader’s theatre to encourage engagement). Teachers are encouraged to Interact with the text they are using (e.g., Big Book during shared reading). They should ask children meaningful questions and activate children’s prior knowledge and generate discussion on the topics raised in the book. There are some activities encouraged during shared reading including retells, role plays, etc., but these could be extended, especially for Home language No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |83 Assessment Criteria Performance Standard Analysis Adequate? Addresses question types and ways of engaging with texts A range of question types should be used: literal, inferential, evaluative, appreciative, critical Some mention in EFAL is given to inference. This is key to comprehension and children need to be explicitly taught Not assessed for HL No for EFAL Not sure for HL Degree to which the Pedagogy is adequately structured LTSM and training intervention aligns with coaching and other RSP interventions Lesson plans, training, LTSM and coaching fits well together These appears to be a good alignment with CAPS, LP and coaching (judging from EFAL training agendas) Yes Supports changing practices Complexity of changing practices are pertinently addressed None of the available documents has highlighted changing practices. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |84 Is the coaching design likely to lead to the anticipated impact? Table 7: Assessing the Coaching Design Criteria Performance Standard Analysis Adequate? Coach selection criteria and training are likely to lead to the deployment of coaches which can be regarded as “experts” Appropriate qualifications Ideally, a literacy coach has a teaching degree in Foundation phase (Preferably with an appropriate higher research degree or post graduate certificate) which would allow the coach to understand the theory of how children learn to read and be able to discuss the reasons why certain practices work while others do not. The Job Description indicates that a teaching degree/diploma in Foundation Phase (or equivalent) is required. No specialization in reading is required. Skills needed to teach reading are very specific and requires training in specific reading methodology. These skills are not covered in all teaching courses, so criteria beyond teaching qualification is required. No Experience of teaching Ideally, the coach has more than five years of teaching experience teaching Grade 1-3 in Setswana (and EFAL). Job Description requires seven years which is a good level of experience. Senior teachers at schools may regard coaches with more experience in contexts similar to their own as experienced enough to provide support in their schools. Yes Knowledge about CAPS Ideally, a coach has knowledge about CAPS and experience teaching using CAPS. It is assumed that coaches are familiar with CAPS, which might not be the case if the RSP selects teachers that retired prior to 2011. No Good language proficiency in Setswana and in English Ideally, the coach speaks both the Home Language and EFAL proficiently Home Language and English proficiency have been listed in the job description as requirements, but this is not tested. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |85 Criteria Performance Standard Analysis Adequate? It is unlikely that coaches would be able to teach teachers how to read and write in a language that they are not proficient. Reports from OUP indicate that there is some concern about Coaches’ English proficiency. Experience of training adult learners Ideally, a coach has experience of training (not just teaching). Other reading programs (i.e., Funda Wande) explicitly train coaches on coaching and training methods. This is not a recruitment criterion in the job description and is not covered in depth in any of the training material reviewed by the team. Explicit training on how to facilitate and lead discussions would be valuable to include in coach training. No Appropriate communication skills An effective coach is good at listening with empathy and as a critical friend. A good coach is able to facilitate high level discussion with adult learners. This is not included as a formal recruitment criterion and is not covered in depth in any of the training material reviewed by the team. No Appropriate classroom observation skills An effective coach knows how to conduct classroom observations, how to capture the classroom practices accurately, and how to give feedback. This is not included as a formal recruitment criterion and is not covered in depth in any of the training material reviewed by the team. The difference between the role of the coach and that of a CA is not clearly articulated in any of the training material reviewed by the team. No Skills to facilitate A coach needs to be able to encourage teachers to discuss their own practices This is not included as a formal recruitment criterion and is not covered in depth in any of the training material reviewed by the team. No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |86 Criteria Performance Standard Analysis Adequate? appropriate reflection and use examples from their own classroom. Able to promote reflective practice A coach needs to be able to facilitate reflection for professional development. Reflection notes are included in the lesson plans. However, they tend to be quite procedural. Deep reflection requires an ability to be critical of one’s own practice. Somewhat Encouraged to be readers themselves Coaches should read themselves and be knowledgeable about children’s literature and relevant reading material. Interview data reveal that the coaches are encouraged to read all the LTSM and are questioned on the content of the readers at sessions. However, they would benefit from richer exposure to books other than the readers. Yes Development of Coaches Support from Head Coaches In other reading programs, Head Coaches observe and debrief examples of coaching, co-learn and co-plan. Weekly reflection meetings are held, and on-site support is provided to coaches as needed. Interview data suggests that support visits take place once or twice per term because of transport challenges. Reportedly, Head Coaches largely rely on reports submitted by the coaches. It is likely that coaches would be unaware of the areas in which they need development. No Ongoing professional development The program identifies training needs, and addresses them through a program of ongoing professional development. According to interview data, the developmental training takes place monthly and is needs￾based. Training has been done on MS Office packages over the past few months. While training to comply with the administrative tasks in the RSP is important, the focus of Somewhat Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |87 Criteria Performance Standard Analysis Adequate? professional development should be on in￾classroom coaching skills, identifying common issues, and running appropriate needs-based workshops. Coaching design Ratio of coach: teachers is manageable and conducive to building rapport A study on coaching in Kenya showed that coaching ratios of 1:10 delivered better results than 1:15. In other literacy programs in South Africa (READ Learning for Living, GPLMS, EGRS I, NECT) a ratio of not greater than 1:30 is common. The coaching ratio may affect a coach’s ability to build rapport. However, the coaching dosage, and how it is spread over time, rather than the coaching ratio is likely to have an impact on outcomes. The coach ratio in the RSP is around 1:60 or more. In the RSP, the school-based workshops provide an additional opportunity to build rapport with teachers. This aspect should be further investigated in the Implementation Evaluation. Not clear Dosage of coaching is likely to lead to results EGRS I managed gains with 10 individual visits of two hours per teacher, combined with 10 school based support workshops. Coaches only supported HL teaching. In the RSP, teachers are projected to receive 9 to 13 hours of individual coaching support over two years. Teachers should also receive 18 to 26 hours of coaching support in afternoon workshops over two years. In the RSP, the weighting of the coaching support is slanted towards school-based workshops rather than the EGRS I approach of individual support. Not clear Duration of the coaching relationship In the EGRS I coaches built a relationship with teachers through monthly visits spread over one year. The RSP coach support is spread over two years, unless a coach makes the decision to Not clear Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |88 Criteria Performance Standard Analysis Adequate? coach selected teachers in Year 1, and the rest of the teachers in Year 2. Coaching Structure Focus on structured feedback after individual coaching Individual support includes structured feedback. This can be a session at the end of the lesson when the learners are engaged in writing and may include going through the observation tool which the teacher has to sign. In-depth feedback session can be scheduled for breaks or after school. Not clear how much feedback is emphasized in RSP coaching. This will be investigated in the Implementation Evaluation. Not clear Focus on structured feedback in school based workshops The needs-based workshop is used to discuss and solve common problem – drawing on the experience at the school. Not clear how much feedback is emphasized in RSP coaching. This will be investigated in the Implementation Evaluation. Not clear Coach conducts appropriate coaching activities Successful coaches visit classrooms for lesson observation and provide feedback. They model the teaching strategies – they support teachers by helping with ‘how’ of teaching. They assist teachers with pacing of lessons, the development of differentiated lessons, and the selection of best practices to meet the needs of their students. Interview data indicate that coaches are expected to train the teachers, monitor implementation of the lesson plans, monitor curriculum coverage and assessment, ensure that resources are used, observe lessons, and give feedback. This will be investigated in the Implementation Evaluation. It is not clear that coaches understand that the focus of their support should extend beyond monitoring compliance. Not clear Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |89 5.2 MAPPING OF THE KEY PROCESSES IN THE RSP Process maps were compiled to (1) understand the RSP delivery steps, and (2) identify areas that must be carefully tracked during the Implementation Evaluation. The evaluation team mapped the processes based on project documents/data and key informant interviews. Participants at the RSP Clarificatory Workshop (August 2, 2019) verified the process maps and identified areas in which implementation challenges should be expected. These process maps are presented below and should be read from top to bottom, following the direction of the arrows. A narrative description of each process map follows the process map diagrams, and some implementation issues are noted in these narratives. The process maps are coded as follows: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |90 5.2.1 PROCESS MAP 1: DEVELOP, PRINT AND DELIVER LESSON PLANS AND OTHER LTSM Figure 6: Process Map 1 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |91 The first process map illustrates the process for developing, printing, delivering and distributing the Setswana HL and EFAL Lesson Plans and LTSM for Grades 1, 2 and 3 to participating teachers. Developing Material Once material requirements for a school quarter are finalized, OUP and Molteno develops the required materials, the materials are aligned to EGRS I and adapted where necessary. The Reference Group, which includes the Setswana HL and EFAL CAs and experts from Molteno and OUP, reviews the materials. The materials are then revised (if necessary). The Reference Group has limited time to engage with materials. This might affect the review process and ultimately quality of materials. The service providers, Molteno and OUP, have limited time to finalize Home Language lesson plans after the Reference Group meetings. Printing of material Once development and revisions are complete, all materials are finalized and printed. Materials are delivered to the FPD provincial warehouse and stored until materials are transferred to the training venues for distribution at the quarterly teacher training sessions. At the same time, materials are finalized and uploaded to the tablets used by SMT managers and CAs. The amount of materials printed is based on an estimate of six teachers in 263 schools with a contingency of three to ten percent. The contingency material can be used where estimates are below actual numbers or if new teachers join the program. It is likely that many schools do have more than six teachers per school and currently there is no mechanism for confirming teacher numbers45. The process for ensuring current materials are uploaded to the SMT and CA tablets, is not well developed. Distributing of materials to teachers Materials are distributed to teachers at the training sessions. If teachers do not attend, the materials are either be placed in the care of another teacher from the same school, or if the teacher is from a coaching school, the coach distributes the materials during their next school visit. Materials can also be distributed at a catch-up training session if it is held. 45 Classroom libraries were delivered towards the beginning of the RSP implementation period. These numbers may be useful to consider. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |92 Multiple processes are in place to ensure that teachers receive the required materials, as this is an important aspect that forms the basis of the intervention. However even with a variety of alternative distribution processes in place, it is possible that not all teachers from the selected schools receive the materials in time or at all. No process exists to check if teachers have received materials in schools where coaching does not take place. No process exists to check if teachers who miss training receive materials in schools where coaching does not take place. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |93 5.2.2 PROCESS MAP 2: RECRUITMENT AND TRAINING OF COACHES Figure 7: Process Map 2 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |94 Recruitment of coaches. The recruitment of coaches commences with the development of job requirements (requires specialized knowledge and experience in Foundation phase didactics and pedagogy and extensive experience in the teaching of literacy in Setswana and EFAL) and job descriptions for both Literacy Coach and the Head Coach positions. The recruitment process involves advertising, conducting interviews, selecting and contracting. Recruitment of adequately qualified coaches with a solid grasp of early reading pedagogy, fluency in Setswana and the familiarity with coaching strategies proves difficult. Induction. During the induction process, coaches are orientated about their roles and responsibilities in the RSP. An initial five-day training session includes coach training as well as a train-the-trainer process to prepare coaches for their role as facilitators in the first teachers training sessions. A training session on the data management in the project is also conducted. Quarterly training. For the remainder of the project, coaches are trained with a three-day, quarterly training. The first two days focus on how to train teachers to implement the lesson plans. On day three there are supposed to be dry run sessions. CAs, Head Coaches, and project team members provide feedback and a report on the performance of coaches feeds back into the performance appraisal system. The feedback informs future rounds of coach training. In 2019, formal training on coaching and reflection strategies is not provided. In 2020, the coaches will attend an accredited literacy coach training program which ideally will take place during school holidays (so not to lose any coaching days). Ongoing Professional Development Once a month, the coaches participate in a reflection session, during which their professional development needs are identified and addressed to the extent possible. The Head Coach accompanies each coach on a coaching visit once a month, to provide additional support Variable levels of support may be required depending on the skills and experience of the coaches. Head Coaches may not have the transport needed to conduct monitoring visits. Compilation of consolidated coach training materials Following a full year’s implementation, the materials used in the coach training will be compiled. This may include the coach training programs, the coaching material (compiled into a manual) and a box of material to use during coaching. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |95 5.2.3 PROCESS MAP 3: TEACHERS TRAINING Figure 8: Process Map 3 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |96 Invitations to Training. Foundation Phase teachers from selected schools are invited to attend the quarterly teacher training sessions. These are held over one (Term 2 and 4) or two days (Term 1 and 3). Half of the training time is devoted to Setswana HL and the other half to EFAL training. Three different mechanisms are used to invite teachers to the quarterly teacher training sessions: 1. The Provincial department communicates the training dates to schools via a circular. 2. FPD invites teachers who attended the previous training sessions via SMS (using data from the attendance registers.) 3. Coaches invite the teachers to the training sessions; however, this mechanism only applies to about 50 percent of schools where coaching is included in the project (140 schools). No system is in place to accurately determine how many teachers will attend training, since no mechanism for responding to invitations are in place. Training. Literacy coaches facilitate the teacher training, with support from Head Coaches, FPD, and CAs. At the training, teachers receive all the lesson plans and LTSM pertaining to the next term. Once training is concluded, the attendance registers are submitted to SACE for allocation of continuous professional training and development (CPTD) points. The training details are entered into TraiNet46. Accurate data on how many teachers will attend at each venue are not available, estimates are used to determine accommodation, materials needed, and travel arrangements. This may result in periodic shortages or wasted expenditure. Catch up Training. Coaches are responsible to conduct catch up training with the teachers of coaching schools who did not attend, while additional catch up sessions should be organized for teachers at schools where the coaching component is not implemented. The process for identifying the need for catch up training, and to send out invitations for such training is not well specified – especially for schools that do not participate in the coaching intervention. 46 TraiNet Desktop is USAID's official training data management system that is installed on a local computer or local area network. It is the entry point for data about training programs and participants in their country of origin, a third country, or for potential exchange visitors who will come to the United States on a USAID J-1 visa. You must enter data into TraiNet for any program fully or partially funded by USAID. https://trainethelp.usaid.gov/FAQs/UseTraiNet/index.htm#169e5477_9177_4637_b6ec_0ec3bbfa767 6 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |97 Facilitator’s guide and manual. A teacher training manual to assist teachers as well as a facilitators guide are developed to be used by coaches at the training sessions. The program has coaches with variable levels of experience. Stronger coaches are paired with others to address any weaknesses. Additional support is available from the Head Coaches, Molteno, OUP, and the CAs. The standardization in the quality of training across all clusters, grades, and languages remains a challenge, because of the variable levels of coach/trainer expertise. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |98 5.2.4 PROCESS MAP 4: COACHING TEACHERS Figure 9: Process Map 4 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |99 Introducing coaches to schools. FPD assigns literacy coaches and introduces them to school principals. The coach to school and coach to teacher ratio is variable – Some coaches may support up to 14 schools. While it is assumed that schools will have six Grade 1–3 teachers, this may not always be the case. Consequently, the amount of time that a coach can spend with each teacher may be constrained. Scheduling visits. Schools submit their timetables to coaches, which the coaches use to schedule coaching visits. Once the Head Coach approves the schedule it is shared with the schools and teachers. It is expected that SMT members (the Principal and Foundation Phase HOD) should support teachers in the coaching process and ensure they have the space and time available for observations, individual feedback, and afternoon workshops. School visits. During a school visit, the coach conducts in class coaching and provides appropriate feedback, facilitates reflection and provides assistance. The coach documents the individual coaching sessions and the document is signed by both the coach and the teacher. For a coach to engage with three Foundation Phase teachers on an individual basis, the timetable may have to be adjusted so that the coach is able to observe either HL or EFAL language instruction. Foundation Phase classrooms follow a similar time table – to support three teachers with reading instruction it may be necessary that some of the teachers change around the sequence of HL, EFAL, numeracy and life orientation content to ensure that HL or EFAL content is covered during the time that the coach is in class. After school workshops. At the end of each school visit, Coaches facilitate after school workshops. While the workshop’s requirement and content are not clearly documented, there are two suggested purposes. First, the training workshops are needs based, serve as PLCs, and should support peer learning. Second, they can be used for catch-up training. Further, the workshops should involve at least one grade of teachers but could be attended by teachers across grades.47 Due to the limited time available, multiple afternoon sessions would be required to catch up on the one- or two-day training sessions. Administrative duties. Although coaches have administrative duties as well as coaching responsibilities, the administrative duties and their own training should not constitute the bulk of their workload. 47 DBE Design Specifications of the Reading Support Program (April 15, 2019) page 17 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |100 New approach to coaching considered. A different approach to coaching is suggested based on the ability of the teachers. The better performing teachers will be engaged to support the struggling teachers in their schools or clusters. This will support skills transfer and sustainability.48 Support to the coaching process. A variety of support processes are built into the coaching of teachers, with the Head Coach providing ongoing support, while at the same time monitoring the implementation of the coaching component by each coach. Molteno and OUP provides technical support. Coaches are responsible for collecting monitoring data for both the individual coaching as well as the afternoon workshops. Towards the end of the training, the attendance registers can be submitted to SACE for CPTD points. This CPTD process is a critical element of sustainability and it is not clear if RSP adequately supports and monitors this process. 48 DBE Design Specifications of the Reading Support Program (April 15, 2019) page 17 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |101 5.2.5 PROCESS MAP 5: PROFESSIONAL DEVELOPMENT OF SMTS Figure 10: Process Map 5 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |102 SMT professional development training. About half the schools selected for coaching are also selected to participate in the RSP’s SMT professional development component. The Principal or Deputy Principal as well as the Foundation Phase HOD of the SMT selected school (65 schools) participate. The training focuses on leadership and managerial skills, with additional training in using ICT systems such as SA-SAMS and the Data Driven Districts (DDD) Dashboard. On completion of the full course, SMT members receive certificates as well as CPTD points if the attendance registers are submitted and the course is accredited with SACE. The late selection of schools delayed the start of the SMT professional development implementation. Foundation phase HOD’s will be expected to attend six days of teacher training and four days of SMT training in 2019. A total of ten training days in 2019 may interfere with their ability to teach. Materials in electronic form SMT’s are issued with tablets containing a complete workshop manual compiled with all the necessary SMT support reading/study material. Some additional material referred to in the manual is provided either separately on the tablet or referred to through a link. It is not clear how SMT behavior change will be monitored by RSP There is an implicit assumption that SMTs developing an action plan to improve literacy in the school will be actioned, without sufficient support The process for ensuring materials are uploaded to the SMT tablets, is not well developed. International volunteers. VSO recruits international volunteers who are trained to support SMTs. Volunteers support schools with a once off school visit The focus of these visits are not well defined. The component rests heavily on the experience and skills of international volunteers. These volunteers might not be able to understand and relate to the SA education context in SA and as such might be less effective in their support of SMTs than expected. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |103 5.2.6 PROCESS MAP 6: PROFESSIONAL DEVELOPMENT OF CURRICULUM ADVISORS (CAS) Figure 11: Process Map 6 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |104 RSP Reference Group The CAs serve as members of the Reference Group that quality assures the lesson plans and assists in planning the teacher training sessions. Prior to the first reference group meeting, an orientation meeting is held to explain the purpose of the reference group. The CAs support the implementation of the program in this way, but also benefit from engaging with the materials. If they become more familiar with the materials, they would be more likely to support teachers on the use of these materials. Participating in the Training of Coaches Setswana and EFAL CAs responsible for the selected schools are invited to sit in as participants in the training of coaches, and to observe the dry runs. In this way, the CAs’ knowledge about the use of the RSP-specific LTSM and lesson plans is enhanced. Supporting the Training of Teachers Following the coach training, the CAs attend the teacher training sessions. They observe the training, but also assist with the training in some ways. Targeted ICT Training for Curriculum Advisors The purpose of the ICT training sessions is to improve the school management competency in using relevant data management dashboards such as SA-SAMS and the DDD Dashboard. Making LTSM available to Curriculum Advisors In Year 2, CAs receive tablets pre-loaded with lesson plans and other relevant software. This may enhance their ability to support teachers since they would have access to the newest versions of the RSP materials. A PLC for Curriculum Advisors Based on the experience in the first year of the RSP, CAs are able to identify priority development needs. These needs will be addressed in PLC meetings facilitated for the CAs. CAs attend the sessions and observe the dry runs before the coaches facilitate teacher training sessions. CAs have an important role to play in the RSP as they are involved in the quality assurance of lesson plans, training of coaches, and training of teachers. The commitment and time demands may distract them from other tasks. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT |105 5.2.7 PROCESS MAP 7: CLASSROOM LIBRARIES Figure 12: Process Map 7 Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 106 The main goal of the classroom libraries is to enable individual reading. Contents of a classroom library. A classroom library contains appropriate reading material for the specific grade. Grade 1 libraries include 20 titles form the Vula Bula Home language range, and the African Storybook Project. Grade 2 and 3 libraries contain 40 titles. A team of field trainers delivers these materials to schools. Training on using the library. The field trainers train the teachers on how to use the libraries in the class. They focus on teaching teachers how to select titles for different reading levels, and on how to manage the classroom libraries. RSP coaches and CAs are trained in the use and extension of the libraries49. The deliberate integration of this component with the other components such as teacher training and coaching is not clear. As such, the library might not be used to its full potential. 49 Some interview data indicates that this was not done. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 107 SECTION 6: CONCLUSIONS The RSP is a combination of interventions that aim to improve early grade reading outcomes in 263 schools across two districts in the North West Province. All Schools (264) In a subset of schools Also including Curriculum Advisors (CAs) Lesson plans and LTSM + Coaching (140 Schools) + CA Training + Classroom libraries (100 schools) + Central Teacher Training + SMT Training (65 Schools) Before providing an overall conclusive assessment on the RSP’s coherence, the evaluation team describes the main components of the RSP TOC and points out the major differences between the RSP and EGRS I. It shares some observations about the RSP design and highlights possible implementation risks. All EGRS I and RSP schools receive lesson plans + LTSM Justification for the intervention The RSP and EGRS I theories of change rely on similar understandings of how reading acquisition takes place. They also rely on similar understandings of how teachers’ capacity to teach effectively can be enhanced. A key component is structured lesson plans that guides teachers to pay attention to all important parts of reading acquisition, and effectively integrates a package of LTSM that supports the implementation of the lesson plans. Lesson plans set the performance expectations in terms of what should be covered and indicates the pace at which the content should be covered. A package of LTSM makes it possible for teachers to implement the lesson plans. In the RSP, an extended package of LTSM is delivered in the form of classroom libraries. The combination of lesson plans and LTSM is thought to enhance curriculum coverage and help teachers to expand the reading strategies that they employ in class. With improved curriculum coverage and more effective teaching strategies, it is thought that more opportunities for reading will be created in class – and with the material available in the classroom libraries, it should be possible to encourage more individual reading. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 108 Differences between the RSP and EGRS I HL lesson plans are redeveloped for the RSP, and EGRS II EFAL lesson plans are revised by OUP. There are no significant differences between the EGRS I and the RSP, except that the EGRS I schools did not receive classroom libraries. An expert review of this component shows that • Lesson plans are based on and incorporates the LTSM and DBE books effectively, however classroom libraries are not deliberately incorporated into the lesson plans. • Lesson plans are quite demanding in terms of what it requires teachers to cover in each lesson. Paired and individual reading time, reading aloud and opportunities to promote rich talk are not adequately incorporated into the current lesson plans. • Lesson plans are customized based on feedback from a reference group that also involves CAs. This has the potential to enhance the uptake of materials. If CAs are familiar with the materials, they are more able to support their use in class. The implementation risks to monitor include • In the materials development process, the time for reviewing materials and adapting them is limited. The Reference Group’s ability to quality assure materials may be constrained in such a short period of time. • Without accurate teacher numbers and an effective process for determining which teachers missed training, delivering adequate materials to all teachers on time is challenging. • Teachers and coaches (and others) may identify errors in LTSM (e.g., Lesson plans) and it is unclear if there is a feedback process so that errors can be corrected. All EGRS I and RSP schools receive centralized training to mediate the lesson plans and LTSM Justification for the intervention There is an inherent incentive for teachers to adopt lesson plans and LTSM: Structured and detailed lesson plans which integrate with a package of LTSM makes it easier for the teacher to prepare and deliver lessons. However, both the EGRS I and the RSP recognize that providing the material requires some mediation to ensure that teachers understand what they need to do with the lesson plans and LTSM, and why they need to do it. Just-in-time centralized training is delivered that helps teachers understand the requirements. The assumption is that with the new materials, and some mediation, teachers would be able to change their practices and deliver more effective instruction. All teachers in the EGRS I and RSP receive the lesson plans + LTSM + training intervention. Differences between the RSP and EGRS I There are some differences between the EGRS I and the RSP. The EGRS I training only focused on HL while the RSP focuses on both HL and EFAL. The EGRS I delivered four days of training in the course of one year. The RSP delivers six days of HL training and six days of EFAL training over two years. The training in the RSP is more frequent four Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 109 sessions per year compared to two sessions per year in the EGRS I, and the training dosage is slightly higher in the RSP. An expert review of this components shows that • The project relies on cascade training where the trainers are trained by reading experts to deliver the training. This is a cost effective and necessary strategy when implementing at a larger scale, but it may result in variable quality of training. • The project has concerns about variable levels of training expertise in the program, and have implemented measures such as training of trainers, providing annotated training material, implementing dry-runs, pairing of trainers, and providing oversight and support from Head Coaches to address areas of weakness. • There is not a clearly defined feasible strategy for helping teachers who have missed training; this is especially true of teachers who are not part of the coaching intervention. • The training content aligns well with CAPS and the lesson plans. • The training content covers appropriate topics and effective teaching methods. It could be more explicit on promoting comprehension and fluency as well as explicitly linking HL and EFAL reading methods. • The content promotes reading engagement through some suggested activities, but these could be extended. • The training content does not pay enough attention to teaching the very complicated methodology of GGR. The implementation risks to monitor include: • Invitations to training are issued through multiple channels, but no system is in place to accurately determine how many teachers will attend training since no mechanism is in place for responding to invitations. • The process for identifying the need for catch up training, and to send out invitations for such training is not well specified – especially for schools that do not participate in the coaching intervention. Some EGRS I and RSP schools receive onsite support: Individual Coaching + Group Coaching Justification for the intervention Emerging learning about changing teacher practice recognizes that there may be a flaw in assuming that training leads to understanding which leads to change. Guskey’s (2002) model suggests support needs to be provided to encourage teachers to try out the new approaches in class, and to help teachers recognize the gains that the new approach can bring over the short term. Only once teachers have evaluated, or reflected upon the outcomes of trying something new, and only if these outcomes are found to be sufficiently positive to justify the continued effort to change practice, will actual sustainable change take place in the classroom. With this understanding, the EGRS I tested a coaching intervention Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 110 that showed to be effective in improving learning outcomes. A similar coaching intervention is implemented with 140 schools in the RSP. Coaches are expected to spend one-on-one time with teachers in their classrooms and to develop a trusting relationship with teachers. The coaches motivate teachers to try out new practices. Within a trusting relationship, the coach observes, facilitates reflection, corrects and supports teachers and co-develops solutions to implementation challenges. The individual coaching support is supplemented by afternoon workshops with groups of teachers. In the afternoon workshops, coaches are able to draw on a variety of observed sessions and on the strengths at schools to facilitate reflection on good practices and ways to overcome challenges. Differences between the RSP and EGRS I The RSP and EGRS I coaching interventions differ in the weighting of individual coaching and school-based support workshops. The RSP has a lower amount of individual coaching (9-13 one-hour visits in RSP compared to at least ten two-hour visits in the EGRS I). The RSP has a higher dosage of school-based workshops (18-26 in RSP compared to ten in EGRS I). The RSP support is spread over two years, while the EGRS I provided only one year of support. The coaching ratios also differ significantly – EGRS I coaches supported ten schools and 30 teachers in HL only, while the RSP coaches support 8-14 schools and at least 60 teachers in EFAL and HL. An expert review of this components shows that It is challenging to recruit experienced coaches with Foundation Phase and Early Grade Reading expertise, as well as experience of facilitating training with adult learners and the ability to effectively coach. Ongoing professional development of coaches is important and needs to focus on improving their ability to facilitate individual and group reflection after structured classroom observations. Coaches need to be familiar with the lesson plans, and LTSM. They should also be able to demonstrate and support the teaching strategies. The project has mechanisms for monitoring and support of coaches, but more intensive support may be required for less experienced coaches. Head coaches are supposed to accompany coaches to schools once a month to build the skills of the coaches. There may be a focus on compliance related tasks during the individual coaching sessions, instead of developmental support that helps teachers to try out the new teaching methods covered in the lesson plans and training. A Head Coach’s description of RSP coach roles points out that coaches must provide on-site training, monitor implementation, ensure that resources are used, ensure that the curriculum is covered and conduct need-based workshops and PLCs. Such a role description seems to duplicate what a CA would be doing at school and does not make the most of the support possibilities of coaching. The degree to which coaches are able to effectively observe, facilitate reflection and give supporting feedback may be constrained if the coach is not experienced at carrying out this role. The degree to which coaches are able to support teachers may be negatively affected if a trusting relationship does not exist. With the high coach to teacher ratio in the RSP, the Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 111 trusting relationship may take longer to develop if the contact opportunity during afternoon workshops is not optimized. Coaches use their discretion to decide what will be covered in school-based workshops. An experienced coach may be able to maximize the opportunity, but more structure may be required for coaches with less coaching experience. The coaching processes are susceptible to the following risks • The coach to school and coach to teacher ratio is variable – Some coaches may support up to 14 schools and this will affect the coaching dosage and ultimately the coaching success • The RSP assumes that there are six Foundation Phase teachers at each school. It is possible that a significant number of schools have more than six teachers. This may negatively affect the coaching dosage. • For a coach to engage with three Foundation Phase teachers on an individual basis during a single visit, the timetable may have to be adjusted so that the coach is able to observe either HL or EFAL language instruction. • The RSP intends for catch up training to also take place during afternoon workshops. Due to the limited time available, multiple afternoon sessions would be required to catch up on the one- or two-day training sessions. Some RSP schools receive SMT training Justification for the intervention The RSP provides four training workshops to the Principal, Deputy Principal, and the Foundation Phase HODs to support reading literacy improvement at their school. The SMT members receive tablets, which give them access to both online and offline resources consisting of a complete workshop manual compiled with all the necessary SMT support reading/study material. Some additional material referred to in the manual is provided either separately on the tablet or referred to through a link. SMT members receive one support visit from a volunteer who facilitates SA-SAMS and DDD Dashboard use, exposes SMT members to Circle of Services (COS) and COP and facilitate the SMTs to take leadership in ensuring their own digital training. The RSP anticipates that the SMT support will enhance SMTs’ ability to monitor and support teachers in teaching the EFAL curriculum. Differences between the RSP and EGRS I: The SMT intervention was not part of the EGRS I. The implementation risks to monitor include: • The late selection of schools delayed the start of the SMT professional development implementation. The planned number of training session, eight sessions in two years, might be unattainable. • It is not clear how SMT behavior change will be monitored by RSP Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 112 • There is an implicit assumption that SMTs developing an action plan to improve literacy in the school will be actioned, without sufficient support. • Foundation phase HOD’s will be expected to attend six days of teacher training and four days of SMT training. A total of ten training days in 2019 may interfere with their ability to teach. • The process for ensuring materials are uploaded to the SMT tablets, is not well developed. • The international volunteers might not be able to understand and relate to the SA education context in SA and as such might be less effective in their support of the SMT than expected. RSP Curriculum Advisors are trained Justification for the intervention RSP CAs are oriented towards the purpose of the RSP to make sure they support and motivate teachers in their schools to participate actively in the RSP. CAs participate in a Reference Group that reviews lesson plans, LTSM, and training content. This is thought to enhance their understanding of the RSP content and to help them to align their support with the messages from the RSP training. CAs also participate in the coach training and receive tablets with the RSP materials. This support is intended to motivate and capacitate CAs to provide more effective support to teachers in RSP schools. Differences between the RSP and EGRS I The CA intervention was not part of the EGRS I. The implementation risks to monitor include: • CAs have an important role to play in the RSP as they are involved in the quality assurance of lesson plans, training of coaches, and training of teachers. The commitment and time demand of the RSP may distract them from other tasks. Final Observations The RSP TOC is based on evidence about reading acquisition and effective teacher development strategies, and clearly identifies the skills, knowledge, behaviors, and relationships that the RSP would need to impact. However, the quantum of desired change in the skills and behavior of teachers, SMTs, CAs, and the learners is not specified. The RSP TOC is built on the EGRS TOC, which often states actions rather than strategies such as “coach corrects and supports teacher” rather than “Coach analyses, evaluates and guides teacher to adjust instruction”. The TOC is not differentiated for more experienced and less experienced teachers, for urban or rural schools, and does not consider how teachers who were previously part of the RSP should be approached. The findings from the EGRS sustainability study indicate that the results are sustainable after project exit. Critical to the continued use of materials was ongoing support to teachers. Therefore, building capacity across the system is important to embed support beyond the Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 113 project. A number of critical elements are highlighted which, if implemented, can contribute to the sustainability of the results. These include firstly, that teacher training contributes to the professional development of teachers through alignment to CPTD and achievement of points; secondly that the coaches provide afternoon workshops as PLCs; and thirdly that the project has systemic impact. System impact requires working at, and involving, multiple levels (for example, schools, district, province and national) and by building relationships and strengthening linkages between role-players (such as teachers, HOD, Principal, coach, CA etc.). Ultimately, the success of the RSP TOC depends on the uptake of the lesson plans, LTSM, and classroom libraries in the classroom. The program assumes that good quality of training and lesson materials are delivered, and that teachers will have an incentive to try these out in class. For the schools in the coaching intervention the quality of the coaches, the nature and dosage of coaching may influence the degree to which it is able to replicate or exceed the success of the EGRS I. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 114 SECTION 7: RECOMMENDATIONS The evaluation team recommends that the design of the RSP be strengthened as follow: Ensure that the RSP maximizes the linkages with the Professional Teacher Development Frameworks of the DBE Afternoon workshops should be used to establish and strengthen of school-based PLCs The RSP should help teachers to claim CPTD points professional development – both for the central training, and for school-based workshops. The integration of Classroom libraries could be strengthened by updating the lesson plans. When lesson plans are updated in the future, it would be worthwhile to integrate the use of classroom libraries. In the meantime, coaches could help teachers embed the use of the classroom libraries. This would require that all coaches should be familiar with the titles in the classroom libraries and the training that was provided to teachers about the selection of titles and managing the classroom library. The complementarity in different actors’ roles should be exploited to ensure that expectations for teaching reading, and an understanding of what effective practice entails is spread throughout the education system. The way in which the coach role and the CA roles differ and complement each other should be clearly articulated to coaches and teachers. The program should find opportunities to further develop the relationship between the Coach, CA, and SMT members in order to effectively monitor and support teaching in the classroom. The design of the lesson plans and training are adequate, but could be strengthened in some ways: The training should more strongly promote the use of rich classroom talk and well￾facilitated discussions around texts particularly in shared and guided reading (to accommodate and encourage discussion). The training and materials should promote repeated readings of a text. While one text is read in Shared Reading during the week, there are not opportunities allocated for children to engage repeatedly with the text. This should be included during paired reading, during listening and speaking and during GGR. The training should cover question types and ways of engaging with texts. Coaches (and teachers) should be encouraged to ask a range of questions and should be trained in elaborating on learners’ responses. There should a rich talk around texts, especially during reading Aloud and Shared Reading and GGR. The training should be expanded to teach effective GGR. Teachers need to be encouraged and supported in identifying reading abilities, setting up same-ability Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 115 groups, teaching and maintaining routines around GGR, ensuring that the other children are meaningfully engaged while one group is busy with the teacher, choosing relevant Graded readers. Teachers should know how to monitor progress among learners, and how to move children between groups depending on their progress through the year. The coaching component is critically dependent on the quality of coaching that can be delivered. The evaluation team recommends: That strategies for coach development should be sustained. This includes just-in￾time training, ongoing professional development workshops, support from Head Coaches and practicing the delivery of training. Initiatives to train coaches on formal coaching methodologies should be investigated50, and online training courses could be considered as a way of building the pedagogic content knowledge of coaches. The RSP made a significant investment into the development of their coaches – many of whom have not coached before. This reduces the time available for coaching in school, but is a worthwhile trade-off should it yield better quality coaching. Coach turnover during the course of implementing the RSP is a likelihood. Strategies for building in redundancy – like promoting some of the current coaches to Head Coaches and recruiting new coaches for the next year would be worthwhile to investigate. Since the coach to teacher ratio and available coaching time differs across schools, it may be useful to provide some guidance to coaches about how to prioritize their support. For example, newer teachers and untrained teachers should receive more support. The RSP needs to reiterate that afternoon workshops should happen during all school visits. The content of the workshops may be based on assessment of the needs of the school, but the assumption is that all schools can benefit from these workshops. To ensure that the potential value of afternoon workshops can be realized, even by inexperienced coaches, it would be helpful to compile a pack of possible training items, which is designed to address topics such as promoting comprehension, classroom management, teaching routines, types of questions and encouraging rich talk in the classroom. Other reading initiatives in South Africa have examples that can be drawn from. 50 There are recorded plans for coaches to complete a newly developed UNICEF-sponsored coaching course. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 116 Coaches should be encouraged to focus more on facilitating reflective practice, rather than duplicating the compliance driven support provided by CAs. Coaches may need to be trained so that they know how to facilitate reflection in individual and group settings. The RSP does not replicate the parental support intervention of the EGRS I. The RSP could consider using the SMT component to promote parental support. The RSP should help SMTs plan for engaging parents and the community in a campaign to promote reading. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 117 ANNEX 1: SCOPE OF WORK SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK C.1 INTRODUCTION Despite the government of South Africa’s (GoSA) large investment in basic education – ZAR 228.8 billion (approximately $15.25 billion) in 2016/17 – roughly 16% of the national budget – South Africa continues to face challenges providing a quality education in the majority of the country’s schools and its education indicators continue to lag behind that of its peers. In international comparative reading tests South Africa consistently performs at the bottom with nearly 80% of Grade 4 students unable to read with comprehension in the language of their choice (PIRLS 2016). The GoSA considers education to be one of its highest domestic priorities and one of the greatest long-term challenges facing the country, as is evident in the National Development Plan of which the number one outcome is improving the quality of basic education (DBE, 2013). To support the GoSA, USAID/SA is implementing the Practical Education Research for Optimal Reading and Management (PERFORM) project. The overall goal of PERFORM is to improve reading skills of primary grade learners which, at the time of publication was in line with the continuation of Goal 1 of USAID’s Global Education Strategy, aiming for 100 million children worldwide with improved reading skills. The newly published US Government Basic Education Strategy (2019-2023) also prioritizes improved quality of instruction in basic education, and USAID’s new Education Policy (November 2018) continues to prioritize improved early grade reading outcomes. PERFORM will contribute to these shared goals through implementing education interventions in support of three objectives: ● Improved primary grade reading instruction; ● Increased quality of educational administration and support; and ● GoSA support for reading initiatives built at district, provincial and/or national level. Interventions will be designed and implemented in partnership with GoSA to focus on building teacher knowledge and skills that can improve literacy pedagogy, strengthen and improve education management for better reading outcomes, and involve community members in supporting reading outcomes. PERFORM will use a demonstration effect and rigorous evaluation to support GoSA in bringing proven education solutions to scale, thus increasing the impact and value of national investments in the education sector. This will be done by piloting and testing local solutions which show promise to improve the reading skills of primary grade learners. Close collaboration with GoSA will aim to support buy-in for new reading interventions and to ensure GoSA has the data necessary to decide how best to take successful, cost-effective literacy improvement interventions to scale across the country. In order to deliver rigorous evaluations under PERFORM, the PERFORMANCE Indefinite Delivery Indefinite Quantity (IDIQ) single holder contract was awarded to Khulisa Management Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 118 Services to provide technical, analytical, advisory, monitoring, evaluation and related support services to assist USAID/SA in effectively diagnosing needs, and planning, designing, monitoring, evaluating and learning from interventions. The contractor will also be responsible for relaying this information to GoSA, the education research community, and other education sector stakeholders. This dedicated education sector evaluation IDIQ contract will provide cohesion across all evaluations of PERFORM activities, as well as a meta-analysis of outcomes across the project. It will also allow for the flexibility to adapt to changes in PERFORM activities and to address additional reading related research and analysis requests from USAID that develop during the course of PERFORM’s implementation, both in response to the needs of DBE and otherwise. PERFORMANCE will help to fill a critical research gap by providing rigorous analysis in target areas related to improving the quality of language and literacy skills of primary grade learners in South Africa and the region. USAID/SA found there is little data available on the impact of teacher training programs on student learning outcomes, including literacy. There is also little rigorous analysis available on the effectiveness of school principals, district authorities, communities, and parental engagement to increase student learning. Furthermore, research studies on the costeffectiveness of quality teacher training and support, school management and administration is even more limited, especially that which estimates the cost-effectiveness of various types of programming in developing countries. In general, the quality and methodological rigor of the research that is available varies so significantly that it does not offer a sound empirical base for program design in South Africa. Rigorous research that does exist is so context-specific that it is not necessarily generalizable. PERFORMANCE aims to fill this crucial research gap with rigorous evaluations, studies and assessments. The IDIQ contract has three distinct objectives: ● Objective 1: Design and conduct evaluations, surveys, studies and sector assessments to inform and improve education sector activities. ● Objective 2: Make recommendations for viable future education sector activities. ● Objective 3: Strengthen the community of practice surrounding education research, monitoring, evaluation, and learning for languages and literacies. This task order under PERFORMANCE will require the contractor to propose design concepts for pertinent evaluation and assessment activities and to deliver such design reports to USAID. This Task Order will also require the management and reporting deliverables as discussed in the IDIQ Contract and laid out below in Section C.3. C.2 BACKGROUND On November 8, 2017, the PERFORMANCE single holder IDIQ contract was awarded for a performance period of five years. To accomplish the objectives of the PERFORMANCE IDIQ as listed above in section C.1,, the contractor will provide design and delivery services to USAID through conducting desk reviews, stakeholder dialogues and other scoping and research activities on specific topics upon written request by the TOCOR. The findings, conclusions and recommendations from these reviews and engagements will be presented to USAID as design reports. Based on the design reports, new task orders may or may not be solicited by USAID under the IDIQ contract. Furthermore, the IDIQ contract has core support administrative and reporting requirements that will be met through this task order. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 119 C.3 OBJECTIVES AND RESULTS The objective of this task order contract is twofold: (1) to administer core support services for the PERFORMANCE IDIQ as laid out below; and (2) to produce, upon request from USAID, design reports to be used for the development of future task orders for research and/ or community strengthening activities through a process of desk reviews, stakeholder dialogues and other scoping and research activities. To achieve these objectives, the contractor must produce the following intended results: 1. Five design reports presenting the findings, conclusions and recommendations based on desk reviews, stakeholder dialogues and other scoping and research activities on specific topics, upon written request by the TOCOR. 2. Reports and deliverables or outputs as specified by the IDIQ Contract in Section F.5, and incorporated here: REPORTS AND DELIVERABLES OR OUTPUTS In addition to the requirements set forth for submission of reports in Section I and in AIDAR clause 752.242- 70, “Periodic Progress Reports,” under the first task order issued to this IDIQ, the Contractor shall submit the following deliverables or outputs to the COR specified in accordance with Section I: (a) Contents Of Periodic Progress Reports (1) Performance Monitoring Reports (PMRs). The Contractor shall submit performance reports at intervals agreed upon with the COR (Assume the COR requests annual reports), summarizing progress of the major activities in process during the period of implementation of the contract, indicating any problems encountered, and proposing remedial actions as appropriate. In addition, the Contractor shall include a section which discusses any salient programmatic trends that can be distilled from major activities that are in-process or recently completed; and highlight unresolved or ongoing administrative/ bureaucratic constraints to the Contractor's optimal performance; an update of which will be used as part of future award evaluations. (2) Contract Financial Report. The contractor shall submit a contract financial report including the following: (i) Notification of New Task Order Report Content: (1) Task Order number, (2) Mission/Bureau contracting the task order, (3) Period of Performance, (4) CO, (5) COR, (6) Ceiling Price, and (7) Initial Obligated Amount. (ii) Modifications to Existing Task Orders Report Content, to include a short description of any of the following changes to existing task orders: Incremental funding, (1) Time extensions, (2) Change of CO and/or COR, and (4) Completion of work. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 120 The above reports shall be submitted within 30 days of the three-month periods (calendar quarters) ending on March 31, June 30, September 30, and December 31. The contractor shall submit the reports to the COR identified in Section G of the Schedule, who will forward the reports to the responsible Contracting Officer. The contractor shall promptly notify the Contracting Officer and COR of any problems, delays, or adverse conditions which materially impair the contractor's ability to meet the requirements of the contract. (b) Accrual Reports The contractor shall submit an estimated accrual report to the cognizant technical office for the task order: (1) Total amount obligated, (2) Total amount invoiced for, (3) Total amount expended but not yet invoiced for, (4) Remaining unexpended funds. The contractor shall submit these reports to the COR identified in Section G of the Contract on March 10, June 10, September 10, and December 10. (c) Language of Reports and Other Deliverables All reports and other deliverables shall be in American English, unless otherwise specified by the task order. (d) Meetings The contractor shall schedule and attend semi-annual meetings with the COR. (e) Reports (1) The cover page of all deliverables required hereunder shall include the USAID Identity (or the name of the Agency written out) prominently displayed, the contract number (see the cover page of this contract), Contractor name, name of the USAID project office (USAID/Southern Africa/Regional Environment, Education and Democracy Office), the publication or issuance date of the document, document title, author name(s), and activity name (PERFORMANCE). Descriptive information is required whether Contractor furnished products are submitted in paper or electronic form. All materials shall include the name, organization, address, telephone number and email address of the person submitting the materials. (2) Hard copy reports shall be prepared on non-glossy paper (preferably recycled and white or off white) using black print. Elaborate art work, multi-color printing, and expensive bindings are not to be used. Whenever possible, pages shall be printed on both sides (see also Section E of this contract). (3) Electronic formats shall be submitted with the following descriptive information: (i) Operating system format, e.g., Windows or Macintosh compatible; (ii) Name of application software used to create the files, e.g., Microsoft Word 2010; (iii) The format for any graphic and/or image files included, e.g., TIFF compatible; and (iv) Any other necessary information, e.g., special backup or data compression routines/software used for storing/retrieving submitted data. (f) Distribution to COR and Briefing Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 121 Distribution of Task Orders to Basic Contract COR: One copy of each new task order issued under this IDIQ contract must be forwarded to the IDIQ COR within one week of receipt by the Contractor. C.4 CROSS CUTTING THEMES The following guidance is provided with respect to alignment with the US Government Basic Education Strategy 2019-2023, USAID’s Education Policy (November 2018) and Education Evaluation Policy, using local systems, sub-awards and incorporating gender considerations in evaluation activities. Where applicable and feasible for this task order the Contractor must consider all these themes in achieving contract goals and objectives and apply them to the overall performance of the Contract. C.4.1 Alignment with the USG Basic Education Strategy (2019-2023), the USAID Education Policy (2018) and USAID’s Education Evaluation Policy It is anticipated that the contract will be financed 100% from Basic Education (BE) funds. Funding for BE activities must meet all statutory requirements and align with the USG Basic Education Strategy available at https://www.usaid.gov/education/usg-strategy and USAID’s Education Policy (2018) and corresponding implementation guidance, including USAID’s Education Evaluation Policy. In particular, direct BE funding must address the six Key Principles of the USAID Education Policy as well as Priorities 1 and 2. Key Principles: ● Prioritize country-focus and ownership ● Focus and concentrate investments on sustainable results ● Strengthen systems and develop capacity ● Work in partnership and leverage resources ● Drive decision-making and investments using evidence and data ● Promote equity and inclusion Relevant priorities: ● Children and youth, particularly the most marginalized and vulnerable, have increased access to quality education that is safe, relevant, and promotes social well-being. ● Children and youth gain literacy, numeracy, and social-emotional skills that are foundational to future learning and success. C.4.2 Using Local Systems The Contractor must engage local institutions in every step of implementation when feasible, building technical and management capacity within targeted institutions and communities where possible, working with or through host country systems when appropriate, and transferring managerial and activity implementation responsibility to local institutions where feasible. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 122 C.4.3 Sub-awards A means of establishing partnerships with local institutions is through the use of sub￾awards. The Contractor may subcontract as outlined and referenced in the PERFORMANCE IDIQ base contract. Any other subcontracts must be pre-approved by the CO. C.4.4 Incorporating Gender in Design Activities The contractor shall ensure that relevant gender issues are explored through the design process, as appropriate. The contractor shall further ensure that all evaluation topics or research questions that result from the design process are sensitive to gender. All proposed topics must require that the data be disaggregated by sex to enable analysis on relevant gender issues in language education and outcomes and to answer other relevant gender￾based evaluation questions. C.5 GENERAL MANAGEMENT AND ADMINISTRATIVE APPROACH The contractor must provide all general management and administrative support necessary to perform the contract and achieve the above results. This includes, but is not limited to, the following: 1. The contractor will provide overall management and administration of the contract, including home office support and administrative services. The contractor will provide both the key personnel specified in the contract and additional personnel, long￾term and short-term, necessary to meet recurring general management and administrative support needs under the contract. 2. The contractor will procure or lease facilities, supplies and services as necessary to perform the contract. 3. The contractor will provide the planning necessary for performance of the contract. 4. The contractor will get the necessary ethical clearance and permission from authorities to conduct research in government sites, particularly where students will be involved. Requirements for ethical clearance may vary by province and target provinces are not yet determined. The contractor will be responsible for investigating and fulfilling the requirements for such clearance. 5. The contractor will provide oversight, quality control, and general technical support of all services and deliverables provided pursuant to the contract. This includes the provision of copywriters/editors that are familiar with US American English and US American report writing standards. 6. The contractor will provide and assure the proper, efficient, and uniform use of modern management and accounting practices, information technology (IT), communications, reporting, human resource management, property control, security, records, and other administrative processes and systems required under the contract. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 123 C.6 STANDARDS OF PERFORMANCE This section defines the performance requirements to which the contractor shall be held, establishes the performance levels or standards, and defines how these performance standards will be measured and verified. The contractor will be continually evaluated against these standards. Key Performance Indicators and Assessment Plan CONTRACTOR EXPECTATIONS KEY MEASURES OF PERFORMANCE PLAN FOR VERIFYING PERFORMANCE i. Technical Quality of Service The Contractor will deliver professional, high quality services that responds to the contract requirements. Consistency and accuracy of the Contractor’s work is demonstrated at all times throughout the period of performance of the contract, including quality of deliverables. Reports and presentations to USAID and other counterparts reflect professional quality standards in writing, data collection, and analysis. Official acceptance of submitted reports, and other deliverables by the COR via written correspondence, copies saved in COR file. ii. Schedule The Contractor’s Results and deliverables are performed and/or delivered to USAID according to the specified timeline. The Contractor will provide timely answers and feedback on all requests, issues and or questions raised by the COR and/ or the CO. Deliverables and task order results are completed by dates identified in Section F. No more than fifteen calendar days transpires between submission of comments, questions, or issues to Contractor and a response received by USAID. Date of receipt of submitted reports and presentations via physical delivery or electronic submission, copies saved in COR file. iii. Cost Control The Contractor’s work plans and budget are adequate and result in the completion of all deliverables and tasks as outlined in Section C. The Contractor must provide the deliverables or outputs described in Section C and F and comply with all contract requirements, performing to the highest standards under the terms of the Contract. Completion of Contract without any modifications or amendments related to task order price due to contractor performance. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 124 iv. Business Relations Sound working relationship between the Contractor and USAID/South Africa Avenues of communication are clear; Effective communication practices and team management are evident and there are no documented instances of problems arising due to management of key personnel or the team; Logistical aspects pertaining to Section C tasks and deliverables are clearly designed, well thought out processes, are organized and implemented so that tasks and deliverables are efficiently completed. No documented problems or issues arise due to Key personnel management or communication issues; No documented problems or issues arise due to logistical issues (within the control of the Contractor) as documented by the COR. Performance Evaluation Criteria, Ratings and Standards 1. Exceptional Performance meets contractual requirements and exceeds many to USAID/South Africa’s benefit. The contractual performance of the required results was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective. 2. Very Good Performance meets contractual requirements and exceeds some to USAID/South Africa’s benefit. The contractual performance of the required results were accomplished with some minor problems for which corrective actions taken by the Contractor were effective. 3. Satisfactory Performance meets contractual requirements. The contractual performance of the required results contains some minor problems for which corrective actions taken by the Contractor appear or were satisfactory. 4. Marginal Performance does not meet some contractual requirements. The contractual performance of the required results reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor’s proposed actions appear only marginally effective or were not fully implemented. 5. Unsatisfactory Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 125 Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the required results contains a serious problem(s) for which the Contractor’s corrective actions appear or were ineffective. C.7 PROVIDE DATA, TECHNICAL MATERIALS, AND OTHER INFORMATION The Contractor will provide USAID with data, technical materials, and other relevant materials produced in the execution of this contract in line with USAID’s Open Data Policy as outlined in ADS 579 http://www.usaid.gov/sites/default/files/documents/1868/579.pdf. This includes pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives, as well as data and information needed for reporting under the relevant foreign assistance objectives, areas and elements. Pedagogical Materials and Technical Inputs When applicable the Contractor must provide pedagogical materials and other technical inputs developed to support early grade reading outcomes and other contract objectives. Examples of technical inputs to be provided to USAID include scripted lesson plans, supplementary readers, assessment instruments, observation tools, training guides, workshop reports, radio programs, assessment tools, sampling frames, photographs, videos, and other recordings. The Contractor must transmit technical materials to the relevant TOCOR and submit them to the USAID Development Experience Clearinghouse (https://dec.usaid.gov/). C.8 MATERIALS Materials developed under this contract are subject to FAR 52.227-14 (RIGHTS IN DATA – GENERAL), and the Government of South Africa and others will be granted a paid-up, nonexclusive, irrevocable, worldwide license (under the Creative Commons By “CC BY” http://creativecommons.org or otherwise as USAID may determine) to reproduce, prepare derivative works and distribute copies to the public. Any material not first produced in the performance of this contract is subject to clause FAR 52.227-14(c)(2). [END OF SECTION C] Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 126 ANNEX 2: EVALUATION TEAM PROFILES Jennifer Bisgard, Project Director Ms. Jennifer Bisgard co-founded Khulisa Management Services in 1993. An expert in M&E and organizational development, she leads evaluations and capacity building assignments in the education, and democracy and governance sectors. She has 20+ years of experience leading evaluations and research assignments for USAID, including impact evaluations, performance assessments, program/project design/management and data analysis, including leading evaluations, such as the USAID/SA Evaluation IDIQ and serving as Project Director on the previous PERFORMANCE Task Orders. Jennifer has co-authored a chapter of “Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned” published by Sage Publishers in August 2018. The book is edited by Kylie Hutchinson, with a forward by Michael Quinn Paton. The book features 22 case studies of evaluation failures, including ours which is based in South Africa. Prior to establishing Khulisa, Jennifer was the Senior Education Specialist at USAID/Pretoria from 1988 to 1993. She has served on boards for the: African Evaluation Association (AfrEA), International Organization for Cooperation in Evaluation (IOCE) and South African Monitoring and Evaluation Association (SAMEA). She has a Master’s Degree in Social Change and Development from Johns Hopkins University. Katharine Tjasink, Senior Project Manager Ms. Katharine Tjasink joined Khulisa in 2012, as Regional Technical Coordinator on the Farmer Voice Radio project. In April 2014, she took up the position of Senior Associate in the Education and Social Development Division, and currently serves as Associate Director. She is an experienced M&E and research professional with 10+ years’ experience conducting and managing M&E and research assignments for a number of clients, including USAID. Katharine has successfully led multiple evaluations and data collection projects, including evaluations of early grade language and literacy projects in South Africa. She has in-depth knowledge of USAID’s activity reporting, financial and project management requirements and evaluation policy. Prior to joining Khulisa, Ms. Tjasink worked with a radio and Information and Communication Technology (ICT)-based agricultural program funded by the Bill and Melinda Gates Foundation. In 2013, she was selected as one of four award recipients, from a pool of 60 applicants from 25 developing countries, to attend and present her unique “meta-story” evaluation methodology at the 27th annual American Evaluation Association (AEA) conference in Washington D.C. She holds a Master’s Degree in Research Psychology from the University of the Witwatersrand. She is also ISO 9001:2008 certified in Data Quality Management Systems Assessment. Leticia Taimo, Evaluation Coordinator (Mid) Ms. Leticia Taimo joined Khulisa in 2015 as an intern, and is now a Senior Associate with 5+ years’ evaluation, project management and coordination experience. She has successfully participated and coordinated multiple education evaluation, research and Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 127 assessment plus data collection projects in in South Africa for a variety of stakeholders (private sector, NGOs, government, and international donors). In particular, she is developing a strong expertise in Early Childhood Development. She has presented on how to use technology to monitor ECD at the SAMEA Conference 2017, presenting evaluation findings at the “CSI that Works” 2017, facilitating an interactive session on M&E at the Play Conference 2017 and introducing M&E concepts at BRIDGE’s Early Childhood Development Community of Practice in 2018. Leticia has an MSc Development Studies from the School of Oriental and African Studies, University of London, and is ISO 9001:2008 certified in Data Quality Management Systems Assessment. Leticia was awarded the Mandela Rhodes Scholarship in 2013 and the Commonwealth Scholarship in 2014 as recognition of her commitment to social change in the African continent. Margaret Roper, Senior Evaluator Ms. Margaret Roper fulfils the position of Deputy Director and senior MEL Specialist. She has extensive experience in program development, M&E, and knowledge sharing in social policy and practice, social justice, social protection, child protection, school and community health and safety, educational development, and behavior change. Since 1993, Margaret has worked in the non-profit, government, donor, and business sectors in Southern Africa. She has held positions in the Secretariat for Safety and Security, the South African Police Service, Health and Development Africa, Mott MacDonald, Inkanyezi Initiative and consulted for a range of organizations including Oxfam Australia, UNICEF and government departments such as Basic Education, Social Development, Correctional Services, and Gauteng Department of Community Safety. Margaret belongs to SAMEA, AfrEA and to the International Society for the Prevention of Child Abuse and Neglect (ISPICAN). She has presented at ISPICAN conferences in Turkey (2012) and India (2011); and Pathways to Resilience IV, at an international conference held in Cape Town, 2017, on South Africa’s Response to Enabling Child Resilience Through Expanding Community-based Services She has a Masters in Social Research from the University of Sheffield (United Kingdom), and a Bachelors of Primary Education and Education (Honors) from the University of the Witwatersrand. Benita Williams, Senior Evaluation Specialist Benita Williams is a South African evaluator and current director of Benita Williams Evaluation. Over the past eighteen years, she has conducted various evaluations of education, youth development, income generation, and health initiatives in Southern Africa for corporate donors, government departments, and community-based NGOs. She has a keen interest in evaluation methods and evaluation design. She has experience in the sophisticated qualitative and statistical analysis of education data, voluntary counselling and testing/ health behaviour survey data and skills-audits. Recently her focus has been on the evaluation of education support initiatives related to early childhood education, whole school development in public schools, and the training of teachers in maths, physical science, and English subject areas. She is team lead for an evaluation of a Zenex Foundation Secondary School development project across three provinces, and the evaluation of the Anglo American Education programme that targets schools and ECD centres in eight areas across the country. She was part of a team that evaluated an early -childhood focused social-franchise movement Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 128 and also collaborated with RESEP at the University of Stellenbosch on a Public Expenditure Tracking Study in ECD. Ms. Williams frequently works as Developmental Evaluator and has done evaluations informed by Utilization Focused Evaluation, Outcome Mapping, Realistic Evaluation, and Systems Evaluation theories. Ms. Williams has academic training in the field of Research Psychology at Master’s level (University of Pretoria – Dissertation not completed) and is currently pursuing a Master’s in Development Studies at the University of Johannesburg. In 2009-2010, Benita served as executive secretary for the African Evaluation Association, a network of evaluation associations and evaluators across Africa. Benita was the founding treasurer for the South African Monitoring and Evaluation Association (SAMEA) a voluntary association of evaluators and evaluation users in SA, and served until September 2009. Benita has published in peer-reviewed evaluation publications. Jacqui Dornbrack, Panel Expert Jacqui Dornbrack holds a PhD (Applied Linguistics) and a Master’s in Education (Language and Literacy). She is an Independent Literacy Consultant and has consulted for both Zenex and Funda Wande this year. In 2018, she was the Content Director for the Funda Wande, responsible for the development, writing, and design of course material for teachers as well as video scripts to capture good classroom practice. The Funda Wande Course entails 13 modules on teaching reading and writing in the Foundation Phase in both home language and EFAL. Before she joined Funda Wande, Jacqui worked as a Foundation Phase literacy coach for Pearson (Western Cape Department Of Education [WCED] 100 Schools Improvement project) in five schools in Delft and Kuilsriver and as the Programme Advisor for Shine Literacy (an Non-Profit Organization (NPO) who works in schools to support reading and writing in Grades 2 and 3). Before this, she worked in academia as a teacher educator for University of Cape Town (UCT) (Postgraduate Certificate in Education [PGCE] Intermediate Phase) and University of the Western Cape (UWC) (Intermediate Phase) and has supervised 12 post graduate students in the field of language education. Jacqui has also been a school teacher and has presented at numerous local and international conferences and has eight peer reviewed articles and is on the review board of two education journals: Reading and Writing (South African) and English Practice and Critique (New Zealand). Janet Orr, Panel Expert Janet Orr is a highly experienced educator providing quality basic education program services for youth in both the United States and developing countries in Africa, the Middle East, and Asia. She served as the English as a Second Language lecturer when Ohio University/USAID began to establish the Department of Primary Education at the University of Botswana. Janet’s M.Ed. specialization in teaching primary school reading and English language learning was earned at the University of Illinois. She expanded her knowledge into program services and evaluation as the Associate Director of the Center for Equity and Excellence in Education at the George Washington University. She applied those skills in an overseas context when she served as USAID’s Education Specialist in Tanzania and Sri Lanka. Most recently, she is focused on consulting services to design effective instructional programs using research and data gained through program evaluation. Janet served as an Education/Reading Specialist in three Early Grade Reading Mid-Term Evaluations: Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 129 Jamaica’s Education Transformation Project (ETP), Malawi EGRA, and Nigeria’s Northern Education Initiative Plus. Nombulelo Baba, Panel Expert Nombulelo Baba holds a BA (Hons) degree in Development Studies from the University of the Western Cape. She is currently affiliated with the National Collaboration Education Trust (NECT), working as NRC Coordinator. Nombulelo Baba has worked as a teacher educator at READ Educational Trust and Matthew Goniwe School of Leadership and Governance. She has also served as Subject Advisors Coach at the NECT during which time she was instrumental in reviewing and improving the design of the coaching models. She has reviewed coaching tools to ensure effectiveness, evaluated coaching models and TOC, plus supervised EFAL coaches practicing in schools. As part of developing coaches and Subject Advisors, she was involved in developing and training coaches on case studies that are relevant to their context. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 130 ANNEX 3: CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 131 ANNEX 4: STAKEHOLDERS Stakeholder Name Interest or Perspective Role in the Evaluation USAID Donor − Participate in the design workshop − Liaise regularly with the evaluation team − Help to refine and focus the evaluation questions and methodology − Review the evaluation tools − Participate in evaluation debriefing meetings − Review and provide feedback on evaluation reports, and provide general feedback and guidance − Participate in Key Informant Interviews (KIIs) to provide important information which will inform the evaluation Department of Basic Education (DBE) Key national stakeholder − Provide permission to conduct the evaluation − Review the evaluation timeline to ensure minimal disruptions to learning and teaching − Facilitate access to relevant government departments − Inform the provincial department of the evaluation − Review and provide feedback on the evaluation questions and methodology − Participate in evaluation debriefing meetings − Participate in Key Informant Interviews (KIIs) to provide important information which will inform the evaluation Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 132 Stakeholder Name Interest or Perspective Role in the Evaluation North West provincial Department of Education Key provincial stakeholder − Review the evaluation timeline to ensure minimal disruptions to learning and teaching − Facilitate access to schools − Participate in Key Informant Interviews (KIIs) to provide important information which will inform the evaluation Foundation for Professional Development Consortium (including Molteno Language Institute, Oxford University Press and Voluntary Services Overseas Project implementer/s − Review evaluation timeline to ensure feasibility with planned RSP activities − Help to refine and focus the evaluation questions and methodology − Review the evaluation tools − Participate in Key Informant Interviews (KIIs) to provide important information which will inform the evaluation Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 133 ANNEX 5: EVALUATION AND DATA ANALYSIS TOOLS Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 134 SEMI-STRUCTURED KEY INFORMANT INTERVIEW INSTRUMENT TO INFORM THE RSP PROCESS MAPPING INTRODUCTION • My name is ______________. I am working with Khulisa Management Services, to conduct an independent evaluation of the design of the RSP. • I am here today to ask some questions about the way in which the RSP conducts . • Our aim is to understand the steps and potential implementation risks in • We have [60 - 90 minutes] for our time together. Are you available to respond to some questions during this time? • (Consent) This interview is entirely voluntary and you may choose not to participate. If you agree to participate, you can choose to stop at any time or to skip any questions you do not want to answer. Your answers and your participation in this interview are completely confidential. We will not share any information that identifies you with anyone outside of the evaluation team. • Please feel free to stop this interview at any time to ask questions you may have about this consent or anything else. Do I have your consent to proceed? (Transition) Do you have any questions for me before we start? Process Map Title: Interviewer(s) Location of interview/Type of interview (phone, Skype, in person, group) Interviewee(s) Organization/Role Date of interview Consent Given Yes / No Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 135 INTRODUCE THE PROCESS MAP (Show diagram) This is a draft process map that we compiled based on an analysis of documents received from FPD/ USAID / DBE. It is a flow diagram which is read from left to right and from top to bottom. The main document(s) we used to compile this draft process map is . PROCESS MAP QUESTIONS 1. Are the documents we used to compile the draft process map the best documents to use for the process mapping, or are there other more detailed or more up-to-date information that we should access? 2. Are the steps indicated on the process map accurate? If not, please indicate what should change. 3. Are the roles of the key role players adequately distinguished? If not, please indicate what should be changed. 4. Are the steps indicated on the process maps exhaustive? If not, please indicate what should be added. 5. Does this process map link to any of the other key RSP processes? If so, please indicate how? 6. What are the key implementation risks that you are aware of? CONCLUSION OF INTERVIEW Thank you for your time. This concludes the interview. We are going to use the information that you provided to us, to try to update the process maps. The process maps will be discussed and further refined in a workshop with all RSP stakeholders. Before I go, 7. Do you have anything else you would like to add, or you think we should know before we leave? 8. Who else do you think I should talk to that can provide a different viewpoint? 9. Do you have any questions for me? Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 136 ANALYSIS TOOLS RUBRIC TO ASSESS THE RSP LESSON PLANS Use the criteria and performance standards in the rubric below, to rate the design of the RSP lesson plans. Draw on evidence from the literature, the document review, and key informant interviews. Write a short comment in the analysis column which indicates in which ways the RSP meets the ideal performance standard, or where it falls short. Indicate in the right most column whether the RSP design can be considered adequate on the particular criterion. Adjust or add to the assessment criteria and performance standards as necessary. Assessment Criteria Performance Standard Analysis Adequate? Maximizes reading time Teachers need to incorporate reading aloud every day, but this may require integration across the curriculum. E.g. Read aloud could be incorporated as part of the life skills curriculum using stories and picture books that deal with life skill issues (e.g. friends, pollution, respect). Frequency of reading is optimal Reading time is maximized as per CAPS. Sufficient time for talking about texts Children need to have spaces to talk about what they are reading, they should be able to formulate questions and share opinions and views with each other Time for all modes of reading Lesson plans should include paired, independent, extended reading Incorporates the provided LTSM Lesson Plans need to specify which texts to use and when to use them Works with the Classroom libraries To incorporate the classroom libraries effectively, children should have access to interesting books, talk about these books should be timetabled and children should be allowed to take books home. Encourages repeated reading of text Children need to be encouraged by teachers and parents to reread books (not only in group guided reading but also during paired reading and get to practice at home too). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 137 RUBRIC TO ASSESS THE DESIGN OF THE RSP TEACHER TRAINING Use the criteria and performance standards in the rubric below, to rate the design of the RSP teacher training. Draw on evidence from the literature, the document review, and key informant interviews. Write a short comment in the analysis column which indicates in which ways the RSP meets the ideal performance standard, or where it falls short. Indicate in the right most column whether the RSP design can be considered adequate on the particular criterion. Adjust or add to the assessment criteria and performance standards as necessary. Assessment Criteria Performance Standard Analysis Adequate? The degree to which the delivery model is fit for the context and cost effective. Trainers are deployed efficiently and effectively Cascade training is cost effective but could be done really badly. Effective cascade training includes: Good quality instruction Sensitivity to cultural and contextual factors Continuing support from the original trainers (Hayes, 2000- Sri Lanka reading project) Teachers are grouped optimally Cluster centralized training of teachers is cost and time effective and builds communities of teachers within a geographical space that lends itself to sharing across schools Training design is contextualized Training design should align with CAPS requirements, link to the guidance that Curriculum Advisors can give. Schools are differentiated Coaches differentiate training based on data, particularly on reading results Training delivery model is appropriate Most Early Grade reading projects (EGRS, FUNDA WANDE, and PRATHAM￾India) use Just in Time training. JIT training Is adequate to prepare teachers for content they need to teach and resources they need to use in the coming term However, because training often needs to cover a term’s work in Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 138 Assessment Criteria Performance Standard Analysis Adequate? a few days, little time is left for the deeper level issues of pedagogy and understanding coherence between the reading methods. Ideally there should be JIT training as well as a few immersion sessions for more complex issues Missed training is addressed Teachers being absent from training needs to be quickly addressed. Realistic volume of content is covered in the training period 1. Training content must include all aspects of the lesson plan and use of the LTSM, including assessments. 2. Training methods should demonstrate communicative approaches and encourage participation and engagement. 3. Training should provide time and space for practical applications in the form of role play and participant presentations. PRATHAM (India Project) also stresses the need to encourage teachers to revisit vocabulary and readings in order to consolidate learnings Preparation of trainers for training is adequate Preparation of trainers covers early reading content / concepts and reading and writing instruction. Ideally the content should be targeted to fill gaps in the trainers understanding of • General Early Grade reading experience • Familiarity with the RSP LTSM and how it should be used in class, • Understand five different reading methodologies ( read aloud, shared reading, group guided reading, paired reading and independent) and how they work together Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 139 Assessment Criteria Performance Standard Analysis Adequate? • Facilitation skills • Understand the knowledge and skills gaps of teachers • Understand how to address gaps during the school based training. • Understand the need to keep detailed records of training. Addresses the practical preparation of trainers Ideally a trainer has experience delivering the content. Videoing dry runs and then discussing them is a very useful way of training (Done in Lesson Study Japan with teachers) Trainers receive clear guidelines and structured training agendas Provides differentiated support for trainers Some trainers are more advanced than others. Ideally, the RSP provides additional support for newer or weaker trainers. Adequately prepared to offer training to teachers A coach has experience of delivering the training before, or is prepared through practical training to deliver the material Adequate monitoring of training A highly knowledgeable expert observes the training and gives feedback. The program collects feedback from teachers and Curriculum Advisors. Program includes ways of collecting this monitoring data. LTSM and Lesson plans are available on time LTSM including lesson plans are available at the training sessions. Training Content – HL and EFAL Covers full range of reading practices across the Grades and Teaching reading needs to incorporate the “Big Five”: Phonemic Awareness, Phonics, Vocabulary, Fluency and Comprehension. Additionally Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 140 Assessment Criteria Performance Standard Analysis Adequate? Across HL and EFAL there should be the various reading methods and writing and reading needs to be closely linked. Term 1 – 4 should cover: Shared reading, comprehension, paired and independent reading (mentioned but not in any detail), group guided reading, vocabulary development, speaking and listening, songs, rhymes, phonics and phonemic awareness, and sight words. Pays special attention to group guided Reading Group guided reading is regarded as a very complicated methodology which requires that teachers know about baseline assessment, teaching GGR routines, setting up same ability groups, conducting GGR and ensuring that the rest of the class is engaged in meaningful learning. The EGRS showed that teachers struggle with GGR. Promotes rich classroom talk in all classes. Children should be given opportunities to generate meaningful talk (especially around books). Hoadley’s overview of research in FP classes shows limited opportunities for children to engage meaningfully- Teachers need to ask more open ended questions, encourage children to ask questions and set authentic tasks where the talk is meaningful. Encourages reading enjoyment Key to reading frequency is reading engagement and pleasure (Wigfield & Guthrie et al, 2004; Verhoeven and Snow, 2001) Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 141 Assessment Criteria Performance Standard Analysis Adequate? A larger body of research indicates that reading motivation is key to reading success. Promotes reading engagement There needs to be explicit training on promoting reading engagement (Nal’ibali includes role plays, puppet shows, oral presentations, retells, story, summaries and reader’s theatre to encourage engagement). Teachers are encouraged to Interact with the text they are using (e.g. Big Book during shared reading). They should ask children meaningful questions and activate children’s prior knowledge and generate discussion on the topics raised in the book. Addresses question types and ways of engaging with texts A range of question types should be used: literal, inferential, evaluative, appreciative, critical Degree to which the Pedagogy is adequately structured LTSM and training intervention aligns with coaching and other RSP interventions Lesson plans, training, LTSM and coaching fits well together Supports changing practices Complexity of changing practices are pertinently addressed Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 142 Criteria Performance Standard Analysis Adequate? Coach selection criteria and training are likely to lead to the deployment of coaches which can be regarded as “experts” Appropriate qualifications Ideally, a literacy coach has a teaching degree in Foundation phase-(Preferably with an appropriate higher research degree or post graduate certificate) which would allow the coach to understand the theory of how children learn to read and be able to discuss the reasons why certain practices work while others do not. Experience of teaching Ideally, the coach has more than five years of teaching experience teaching Grade 1 - 3 in Setswana (and EFAL). Knowledge about CAPS Ideally, a coach has knowledge about CAPS and experience teaching using CAPS. Good language proficiency in Setswana and in English Ideally, the coach speaks both the Home Language and EFAL proficiently Experience of training adult learners Ideally, a coach has experience of training (not just teaching). Other reading programs like Funda Wande explicitly trains coaches on coaching and training methods. Appropriate communication skills An effective coach is good at listening- with empathy and as a critical friend. A good coach is able to facilitate high level discussion with adult learners. Appropriate classroom observation skills An effective coach knows how to conduct classroom observations, how to capture the classroom practices accurately, and how to give feedback Skills to facilitate appropriate reflection A coach needs to be able to encourage teachers to discuss their own practices and use examples from their own classroom. Able to promote reflective practice A coach needs to be able to facilitate reflection for professional development Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 143 Criteria Performance Standard Analysis Adequate? Encouraged to be readers themselves Coaches should read themselves and be knowledgeable about children’s literature and relevant reading material Development of Coaches Support from Head Coaches In other reading programs, head coaches observe and debrief examples of coaching, co-learn and co-plan. Weekly reflection meetings are held and on-site support is provided to coaches as needed. Ongoing professional development The program identifies training needs, and addresses them through a program of ongoing professional development Coaching design Ratio of coach: teachers is manageable and conducive to building rapport A study on coaching in Kenya showed that coaching ratios of 1:10 delivered better results than 1:15. In other literacy programs in South Africa (READ Learning for Living, GPLMS, EGRS, NECT) a ratio of not greater than 1:30 is common. The coaching ratio may affect a coach’s ability to build rapport. However, the coaching dosage, and how it is spread over time, rather than the coaching ratio is likely to have an impact on outcomes. Dosage of coaching is likely to lead to results EGRS managed gains with 10 individual visits of two hours per teacher, combined with 10 school based support workshops. Coaches only supported HL teaching. Duration of the coaching relationship In the EGRS I coaches built a relationship with teachers through monthly visits spread over one year. Coaching Structure Focus on structured feedback after individual coaching Individual support includes structured feedback. This can be a session at the end of the lesson when the learners are engaged in writing, and may include going through the observation tool with the teacher has to sign. In-depth Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 144 Criteria Performance Standard Analysis Adequate? feedback session can be scheduled for breaks or after school. Focus on structured feedback in school based workshops The needs-based workshop is used to discuss and solve common problem – drawing on the experience at the school Coach conducts appropriate coaching activities Successful coaches visit classrooms for lesson observation and provide feedback. They model the teaching strategies – they support teachers by helping with ‘how’ of teaching. They assist teachers with pacing of lessons, the development of differentiated lessons, and the selection of best practices to meet the needs of their students. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 145 IS THE TOC COHERENT AND COMPREHENSIVE? Use the criteria and performance standards in the rubric below, to rate the coherence and comprehensiveness of the RSP Theory of Change. Draw on evidence from the literature, the document review, and key informant interviews. Write a short comment in the analysis column which indicates in which ways the RSP meets the ideal performance standard, or where it falls short. Indicate in the right most column whether the RSP design can be considered adequate on the particular criterion. Adjust or add to the assessment criteria and performance standards as necessary. Assessment Criteria Analysis Adequate Is the desired vision of success clear and detailed enough? Does it present the issue(s) to change in a clear way, making the RSP’s objective clear? Does the ToC reflect a solid consideration of the range of different aspects that need to change in order to make the desired change possible: relationships, capabilities, values, attitudes, behaviors? Is the specification of who needs to do what differently to make the desired change possible, up to date with the present context and stakeholders? Does the ToC include all stakeholders important to the change process? Are change strategies explicit? Are the strategic priorities and the strategies themselves still fitting given the experiences to date? Are the underlying assumptions appropriate? Does ToC incorporate a variety of contexts and audiences? Is the ToC useful to track and analyze implementation progress? Is the ToC built upon current research findings? Is the ToC up to date with implementation? Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 146 ANNEX 6: BIBLIOGRAPHY Baker, L. Breher, MJ. & Guthrie, JT. (Eds) (2000) Engaging Young Readers: promoting achievement and Motivation. Guilford Press. Bean, R. M., Draper, J., Hall, V., Vandermolen, J., & Zigmond, N. (2010) Coaches and coaching in Reading First Schools: A reality check. The Elementary School Journal, 111(1), 87-114. Bold, T., Kymani, M., Mwiru, G., Ng’ang’a, A., & Sandefur, J. (2013) Scaling up what works: Experimental evidence on external validity in Kenyan education. Working paper no. 321, Center for Global 318 Journal of Educational Change (2018) 19:293–3211 3. Development, Washington, DC. Retrieved June 18, 2018, https://www.cgdev.org/publication Churr, C. (2015) Realization of a child's right to a basic education in the South African school system: Some lessons from Germany, PER Volume 18 number 7 http://dx.doi.org/10.4314/pelj.v18i7.01 Available from: http://www.scielo.org.za/scielo.php?script=sci_arttext&pid=S1727-37812015000700002 Clarke, J., & Dede, C. (2009) Design for scalability: A case study of the River City curriculum. Journal of Science Education and Technology 18(4), 353-365. Coburn C. E. (2003) Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher 32(6), 3-12. Coenders, F., & Verhoef, N. (2019) Lesson Study: professional development (PD) for beginning and experienced teachers, Professional Development in Education, 45:2, 217- 230, DOI: 10.1080/19415257.2018.1430050 Davies, R. (2013) Planning evaluability assessments: A synthesis of the literature with recommendations. Working Paper 40. London: UK Department for International Development. Deacon, R. (2016) The initial teacher education research project: Final report Johannesburg: JET Education Services: www.jet.org.za Department of Basic Education (12 June 2018) Authors: Taylor, S (Department of Basic Education), Cilliers, J (Georgetown University), Prinsloo, C (Human Sciences Research Council), Fleisch, B (University of the Witwatersrand) & Reddy, V (Human Sciences Research Council). Technical Report: The Early Grade Reading Study: Impact evaluation after two years of interventions. Pretoria, South Africa. Department of Basic Education (August 2017) Summary Report Results of Year 2 Impact Evaluation: The Early Grade Reading Study (EGRS). South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 147 Department of Basic Education, Annual Report 2017/2018. Available from: https://www.gov.za/sites/default/files/gcis_document/201810/dbe-201718-annual-report.pdf Department of Basic Education. (December 2017) Response to release of Progress in International Reading Study 2016 results. South Africa. Available from: https://www.education.gov.za/Newsroom/MediaReleases/tabid/347/ctl/Details/mid/5986/ItemID/5522 /Default.aspx Department of Higher Education (5 December 2018) 2018 Global Competitiveness Fact Sheet. South Africa. DevTech Systems, Inc. (2017) Northern Education Initiative Plus Mid-term Evaluation. https://pdf.usaid.gov/pdf_docs/PA00N894.pdf Dubeck. M.M. & Gove. A. (2015) The Early Grade Reading Assessment (EGRA): Its theoretical foundation, purpose, and limitations. International Journal of Educational Development 40 p.315–322. Elish-Piper, L., & L'Allier, S. K. (2011) Examining the relationship between literacy coaching and student reading gains in Grades K to 3. The Elementary School Journal, 112(1), 83e106. http://dx.doi.org/10.1086/660685. Elliott, T.A. (2016) An overview of education and drama in South Africa, in Research in Drama Education: The Journal of Applied Theatre and Performance, 21:1. Pp.57- 59, DOI: 10.1080/13569783.2015.1127150 Fleisch, B. (2017). The development of reading change in education research and practice. Fleisch, B. & Schöer, V. (2014) Large-scale instructional reform in the Global South: insights from the mid-point evaluation of the Gauteng Primary Language and Mathematics Strategy. SA Journal of Education, Vol 34, No 3. Fleisch, B., Schöer, V., Roberts, G., & Thornton, A. (2016) System-wide improvement of early-grade mathematics: New evidence from the Gauteng primary language and mathematics strategy. International of Educational Development, 49, 157–174. https://doi.org/10.1016/j.ijedudev.2016.02.006 Fleisch, B. & Dixon, K. (Forthcoming) Identifying mechanisms of change in the early Grade Reading Study in South Africa in South African Journal of Education. Funnell, S.C. & Rogers P.J. (2011) Purposeful program theory: Effective use of theories of change and logic models. San Francisco: Jossey-Bass. Green, Parker, Deacon and Hall. (2011) South African Journal of Childhood Education 2011 1(1): 109-122. Guskey, T. R. (2002). Professional development and teacher change. Teachers and teaching, 8(3), 381-391. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 148 Guthrie, J. T., Wigfield, A., Barbosa, P., Perencevich, K. C., Taboada, A., Davis, M. H., & Tonks, S. (2004. Increasing reading comprehension and engagement through concept￾oriented reading instruction. Journal of educational psychology, 96(3), 403. Hanushek, E.A. & Wößmann, L. 2007. “The Role of Education Quality in Economic Growth.” Policy Research Working Paper 4122, World Bank, Washington, D.C. http://www￾wds.worldbank.org/servlet/ WDSContentServer/WDSP/IB/2007/01/29/000016406_20070129113447/Rendered/PDF/wp s4122.pdf. Hartell, C. G., Steyn, M. G., & Chetty, M. (2015) Towards Equality and Equity in Education: Assessing an Initiative to Strengthen Teacher Professional Development in South Africa (SPECIAL ISSUE: Equality and Equity in Education). 17(2), 73-93. Hayes, D. (2000) Cascade training and teachers' professional development. ELT Journal Volume 54/2: Oxford University Press. Hivos TOC Guidelines (n.d.). Theory of Change Thinking in Practice https://knowledge.hivos.org/theory-change-guidelines. Hoadley, U. (2012) What do we know about teaching and learning in South African primary schools? Education as Change, Taylor & Francis. Howie, S.J., Combrinck, C., Roux, K., Tshele, M., Mokoena, G.M., and McLeod Palane, N. (2017) PIRLS Literacy 2016: South African Highlights Report. The Centre for Evaluation and Assessment, Faculty of Education, University of Pretoria http://www.saep.org/wp￾content/uploads/2017/12/pirls-literacy-2016-hl-report-3.zp136320.pdf ILO / Ballester, T. & Esheikhi, A. 2019. ILO Working Paper Number 29: The Future of Work: A Literature Review. Joyce, B. & Showers, B. (1982) The coaching of teaching. Educational Leadership, 40(1):4-8. Kasch J., van Rosmalen P., Kalz M (2009). A Framework towards Educational Scalability of Open Online Courses. Journal of Universal Computer Science (23)9, 845-867. Khulisa Management Services (Pty) Ltd. (2015) Performance Evaluation of USAID/Malawi Early Grade Reading Activity. https://pdf.usaid.gov/pdf_docs/PA00KBNS.pdf Kim, Y.-S. G., Boyle, H. N., Zuilkowski, S. S., & Nakamura, P. (2016). Landscape Report on Early Grade Literacy. Washington, D.C.: USAID. Lynch School of Education International Study Center. (2017) What Makes A Good Reader: PIRLS 2016 International Results in Reading. Downloaded from https://nicspaull.files.wordpress.com/2017/12/p16-pirls-international-results-in-reading.pdf McIlwraith, H. (ed.) (2013) The Cape Town Language and Development Conference: Looking beyond 2015; Language Rich Africa: Policy dialogue https://www.britishcouncil.org.za/sites/default/files/language_rich_africa_policy_dialogue_brit ish_council.pdf Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 149 McLean, R & Gargani, J. (2019) Scaling Impact: Innovation for the public good. Routledge & International Development Research Centre. Mohangi, K., Krog, S., Stephens, O. & Nel, N. (2016). Contextual challenges in Early Literacy Teaching and Learning in Grade R Rural Schools in South Africa, in Per Linguam: A Journal for Language Learning, 32(1); 71-87. University of South Africa. Murray, S. (2002) Language issues in South African education: An overview. In R. Mesthrie (Ed.), Language in South Africa, Cambridge University Press: Cambridge. National Education and Development Unit. (February 2018) NEEDU Policy Brief: Habits of Highly Effective Instructional and Facilitative Leaders: Leadership Practices That Support Teaching and Learning. Available from: https://www.education.gov.za/Portals/0/Documents/Publications/NEEDU%20POLICY%20B RIEF%20SERIES/Policy%20Brief%208-Instructional%20leadership_.pdf?ver=2018-07-02- 103215-710 National Institute of Child Health and Human Development. (2000) National Reading Panel Report: Developing early literacy: report of the national early literacy panel, Lonigan & Shanahan. Oliveira, C., Lopes, J. & Spear-Swerling, L. (2019) Teachers’ academic training for literacy instruction. European Journal of Teacher Education, Volume 42, 3. Organization for Economic Cooperation and Development (OECD) (2016) PISA Results. Organization for Economic Cooperation and Development (OECD) (July 2015) OECD Economic Surveys: South Africa Overview. Patton, M.Q. (2008) Utilization-focused evaluation. Thousand Oaks, CA: Sage publications. Pflepsen, A. (2018) Coaching in Early Grade Reading Programs: Evidence, experiences and recommendations. A Global Reading Network Resource. Prepared by University Research Co., LLC. (URC) under the Reading within REACH initiative for USAID’s Building Evidence and Supporting Innovation to Improve Primary Grade Assistance for the Office of Education (E3/ED). Available at www.globalreadingnetwork.net Piper, B. & Zuikowski, S.S (2015) Teacher Coaching in Kenya: Examining instructional support in public and non-formal schools. Piper, B., Destefano, J., Kinyanjul, E.M & Ong’ele, S. (2018) Scaling up successfully: Lessons from Kenya’s Tusome National Literacy program. Journal of Educational Change 19, 293-321. Pouezevara, S. R. (Ed.) (2018) Cultivating dynamic educators: Case studies in teacher behavior change in Africa and Asia. (RTI Press Publication No. BK-0022-1809). Research Triangle Park, NC: RTI Press. https://doi.org/10.3768/rtipress.2018.bk.0022.1809. Primary School Reading Improvement Project (PSRIP). http://nect.org.za/publications/case￾studies/psrip_booklet.pdf/view. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 150 Progress in International Reading and Literacy Study (PIRLS) (2016) http://pirls2016.org/pirls/summary/ Reeves, C., Heugh, K., Prinsloo, C. H., Macdonald, C., Netshitangani, T., Alidou, H., & Herbst, D. (2008) Evaluation of literacy teaching in the primary schools of the Limpopo Province. Pretoria, South Africa: Human Sciences Research Council in Association with the Department of Language Education at the University of Limpopo. Rutgers L. (2012) Coaching Foundation Phase Literacy Teachers as leaders in a school in the Western Cape Province: A Professional Development Strategy http://scholar.sun.ac.za Sailors, M., Hoffman, J. V., Matthee, B., Pearson, P. D., & Beretvas, N. S. (2007) Evaluating the impact of the “Learning for Living” project on the development of literacy skills of learners in low-income communities in South Africa. The enterprise of education, 121- 143. Shin, J.K. & Crandall, J. (2014). Teaching Young Learners English: from Theory to Practice. Boston, MA: National Geographic Learning & Heinle Cengage Learning. Snow, C.E.; Burns, M.S. & Griffin, P. (1998). Preventing Reading Difficulties in Young Children. Washington D.C.: National Research Council. Social Impact, Inc. (2013) Midterm performance evaluation of the USAID/Jamaica basic education project: in support of the Jamaica Education Transformation Project (ETP). https://pdf.usaid.gov/pdf_docs/PDACX310.pdf Sonmark, K., Revai, N., Gottshalk, F., Deligiannidi, K. & Burns, T. (2017) Understanding teachers’ pedagogical knowledge: report on an international pilot study. OECD Education Working Papers, No. 159, OECD. Paris. South African Book Development Council (SABDC) & the DG Murray Trust. (2016) National survey into the reading and book reading behavior of adult South Africans. TNS, South Africa. Available from: http://sabookcouncil.co.za/wp-content/uploads/Final-Report-NRS￾2016.pdf Spaull, N. (2017) The unfolding reading crisis: The new PIRLS 2016 results. Available from: https://nicspaull.com/2017/12/05/the-unfolding-reading-crisis-the-new-pirls-2016-results/ Posted on December 5, 2017 Stanovich, Keith E. (1986) Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 22, 360-407. Stoop, Chrizell. (2017). Children's rights to mother-tongue education in a multilingual world: a comparative analysis between South Africa and Germany. PER: Potchefstroomse Elektroniese Regsblad, 20(1), 1-35. https://dx.doi.org/10.17159/1727-3781/2017/v20n0a820 Synthesis of Findings and Lessons Learned From USAID-Funded Evaluations (March 2, 2018) Education Sector, 2013-2016. https://pdf.usaid.gov/pdf_docs/PA00T1HD.pdf Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 151 The University of Florida Lastinger Centre for Learning, Learning Forward & Public Impact. Coaching for Impact: Six Pillars to Create Coaching Roles That achieve their potential to improve Teaching and Learning. The William & Flora Hewlett Foundation. (2014). Learning to improve Learning: Lessons from Early Primary Interventions and Evaluations in India and Sub-Saharan Africa. https://www.hewlett.org/wp-content/uploads/2016/08/2014-02- 14_Learning%20to%20Improve%20Learning%20Synthesis% 20for%20Publishing_Edited_0.pdf Theory of Change Academy (n.d.) https://changeroo.com/TOC-academy/posts/expert-TOC￾quality-audit-academy. Trialogue South Africa. https://www.trialogueknowledgehub.co.za/index.php/education￾overview United Nations Children Fund (UNICEF) South Africa. Education Budget South Africa 2017/2018. Available from: www.unicef.org/southafrica United States (2019). US Government Education Policy. United States. (2018). Draft USAID Education Policy. USAID (2018) Report: Coaching in Early Grade Reading Programs: Evidence, Experiences and Recommendations. USAID (2019) Early Grade Reading Barometer. Available from: https://earlygradereadingbarometer.org/ USAID (March 2, 2018) Synthesis of Findings and Lessons Learned from USAID-Funded Evaluations: Education Sector, 2013-2016... https://pdf.usaid.gov/pdf_docs/PA00T1HD.pdf USAID. 2013. Midterm performance evaluation of the USAID/Jamaica basic education project: in support of the Jamaica Education Transformation Project (ETP). Social Impact, Inc. https://pdf.usaid.gov/pdf_docs/PDACX310.pdf Van der Berg, S, Burger, C, Burger, R, de Vos, M, du Rand, G, Gustafsson, M, Moses, E, Shepherd, D, Spaull, N, Taylor ,S, van Broekhuizen, H & von Fintel, D (2011) Low Quality Education as a Poverty Trap. Stellenbosch: University of Stellenbosch. Verhoeven, L. & Snow, C. (2001) Literacy and Motivation. Lawrence Erlbaum Association Publishers, London. Weaver, L. & Cousins, J.B. (2004) Unpacking the participatory process. Journal of Multidisciplinary evaluation, 1(1), pp.19-40. World Bank. (2007) Refer to Hanushek & Wößmann. World Bank. (2018) World Development Report 2018: Learning to Realize Education’s Promise. Washington, DC: World Bank. doi:10.1596/978-1-4648-1096-1. License: Creative Commons Attribution CC BY 3.0 IGO Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 152 PROJECT DOCUMENTATION Alsofrom, K. (n.d.). EGRS II Qualitative Report 2018: Coaching Mechanism (draft 2). South Africa. Reading Support Project. (2019). Attachment 1_Revised Technical Application FINAL. South Africa. Reading Support Project. (2019). Attachment 2_Revised Reading Support Activity. South Africa. Reading Support Project. (2019) Attachment 5_Revised. South Africa. Adjusted Reading Support Project. Reading Support Project. (2019). Coaching Norms and Standards: Recruitment and Employment. South Africa. Department of Basic Education. (2017). Policy Summary: Results of Year 2 Impact Evaluation (EGRS). South Africa. Department of Basic Education design specifications for the Reading Support Project. Dated 15 April 2019, but finalized on 24 May 2019 As modified in the FPD Revised Technical Application to USAID dated 26 July 2019 Foundation for Professional Development. (2018) RSP Learning and Teaching Support Materials (LTSM) Handbook. South Africa. Department of Planning, Monitoring and Evaluation / Department of Basic Education. (2018). Improvement Plan for the Impact Evaluation of the Early Grade Reading Study. South Africa. Foundation for Professional Development. (2018). RSP Learning and Teaching Support Materials (LTSM) Handbook. South Africa. Foundation for Professional Development. (2018). Reading Support Project: Quarter 1 Report: 1 Oct to 31 Dec 2018. South Africa. Foundation for Professional Development. (2018). Standard Operating Procedures Draft. South Africa. Foundation for Professional Development. (March 2019). Draft Monitoring and Evaluation Plan 2019-2020. South Africa. Foundation for Professional Development. (2019). Reading Support Project Final Indicators. South Africa. Foundation for Professional Development. (2019). Reading Support Project: Annual Work Plan October 1, 2018 to September 30 2019. South Africa. Foundation for Professional Development. (2019). Research Design Presentation. South Africa. Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 153 Foundation for Professional Development. (2019). Specifications for the Reading Support Programme, April 15, 2019 Consolidated. South Africa. Foundation for Professional Development. (2019). Standard Operating Procedures. South Africa. Foundation for Professional Development. (2019). Strategic Indicators_revision_DBEinput_11012019. South Africa. Foundation for Professional Development. (2019). USAID Quarter 2 Indicators. South Africa. Foundation for Professional Development. (2019). Reading Support Project: Quarterly Report: Quarter 2 – JAN 1 to MAR 30, 2019. South Africa. South Africa. Mothiba, M. (2019) RSP Term 2 Coaches Monthly Plans. South Africa. Pretorius, M. (2018) Inception Report: EGRS 1 Wave 4 Test Development. South Africa. Reading Support Project. (2019). Rapid Review: Coaching Programme and Materials. South Africa. Reading Support Project. (2019). Reading Support Project: Activity Monitoring, Evaluation and Learning Plan. South Africa. Taylor, S., Cilliers, J., Prinsloo, C., Fleisch, B., & Reddy, V. (2017). The Early Grade Reading Study: Impact evaluation after two years of interventions. Retrieved online https://www.jet.org.za/clearinghouse/projects/primted/resources/language-and-literacy￾resources-repository/egrs-technical-report-13-oct-2017.pdf UNICEF. (2019) Request for Proposal for Services: Development of a Accredited Coaching Programme for Teaching and Reading in the Foundation Phase (Gr 1-3). Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 154 ANNEX 7: KEY INFORMANT INTERVIEWS Interviewer Person Interviewed Purpose Nombulelo Baba FPD Head Coach Interviewed on Coaches’: • Qualifications • Roles and Responsibilities • Monthly Plans • Number of schools and teachers supported • Development Training and • Needs based workshops Jacqui Dornbrack Oxford University Press: Master trainer/writer (EFAL) • Content of coach training, • training approaches and materials • EFAL methodology • concerns and problems regarding training, • time allocation and sequencing of training • Communication with Subject Advisors • Lesson plans-contents, sequencing, pacing and progress. • Coaches feedback/engagement with training • Concerns/issues regarding lesson plans • Recommendations on training Jacqui Dornbrack Molteno: Master Trainer/writer (HL) • Content of coach training, • training approaches and materials • EFAL methodology • concerns and problems regarding training, • time allocation and sequencing of training • Communication with Subject Advisors • Lesson plans-contents, sequencing, pacing and progress. • Coaches feedback/engagement with training • Concerns/issues regarding lesson plans • Recommendations on training • Feedback on dry runs • Opinions on training contents • Preparation of coaches for delivering training to teachers • Training and focus on helping teachers teach routines (esp. for GGR) • Text types in Vula Bula • Time allocation in LP of reading methodologies in HL Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 155 • Recommendations for both training and Lesson Plans. Jacqui Dornbrack Project manager of RSP • Opinions on training offered and concerns • Clarification of training given to coaches • Areas that coaches might be struggling with • Response from coaches-in terms of current training and areas that their teachers might be struggling with. • Training on reading corners and integration with LP • Role of Head Coaches in training Daleen Botha Project manager of RSP Activities and links between activities for process maps: Training of coaches and teachers Daleen Botha Molteno: Literacy Specialist and RSP Project Manager Activities and links between activities for process maps: LTSM development, printing, delivery Daleen Botha FPD: M&E Advisor RSP Activities and links between activities for process maps: Coach selection and Monitoring and Evaluation (M&E) processes Daleen Botha FPD Activities and links between activities for process maps: RSP Coaching and Districts Daleen Botha SMT Expert Via email Activities and links between activities for process maps: SMT component Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 156 ANNEX 8: PARTICIPANT WORKSHOP Agenda: Design Evaluation Workshop August 2, 2019 08:30 Introduction and objectives for the Workshop 08:45 Present: Impact TOC for EGRS I and RSP 9:15 Work session and Feedback Groups engage with the TOC and indicate inaccuracies in the depictions, and expand on assumptions. Facilitators give short feedback to wrap up the working session 10:00 Tea 10:30 Present: Process TOC for EGRS I and RSP 11:00 Work session and Feedback Groups engage with the TOC and indicate inaccuracies in the depictions, and expand on assumptions. Each group will have a different aspect of the TOC to focus on e.g., training, coaching, materials etc. Facilitators give short feedback to wrap up the working session 12:00 Lunch 12:30 Work session to interrogate Process Maps * Recruitment, training, deployment, and supervision of trainers/ coaches * Delivering training to teachers * Delivering coaching to teachers * Delivering training to SMTs * Delivering training/ working with District officials. *Developing, printing, and delivering lesson plans and other LTSM *Monitoring and reporting processes 13:30 Feedback from process map work (gallery walk) 14:00 Reflection from panelists Coaching Expert Pedagogy Expert International Reading Expert Comments, Questions and Answers 15:30 Reflection on the day 16:00 Closure Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 157 Attendance Register: August 2, 2019 USAID Department of Basic Education North West Provincial Department of Basic Education Implementers Foundation for Professional Development Volunteer Services Overseas Molteno Language Insitute Evaluators Khulisa Management Services Bernita Williams Evaluation Experts Contract No: 72067418D00001, Order Number: 72067419F00015 READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 158 ANNEX 9: STATEMENT OF DIFFERENCES 159 FPDs STATEMENT OF DIFFERENCE ON THE KHULISA EVALUATION REPORTS (Formative and Design) INTRODUCTORY REMARKS The Khulisa Evaluations took place against the following backdrop: Since its inception, RSP has been in a state of perpetual flux occasioned by: • The incorporation of EGRS into an existing RSP Agreement with predetermined deliverables and funding regulations. • Contestations around the Theory of Change for the revised RSP Activity. • The added design features/activities proposed by DBE in February 2019, post the approved October 2018 RSP design activities. • The inclusion of Classroom Libraries beyond the approved RSP activities. • Uncertainties regarding the finalisation of project schools leading to data credibility concerns, and with serious implications for: - The finite details of project participants, since the list of participating schools, as provided by DBE, kept changing. - Irregular distribution of LTSM i.e underserving schools received LTSM - Venue and catering arrangements - Allocation of coaches to schools • The delayed finalisation of the MEL Plan beyond February 2019, and pending the submission and approval of the Modification Request to USAID • The delayed finalisation of the RSP Standard Operating Procedures for the reasons highlighted above • The reliance of the evaluators on information over and above the approved 2018 project plan which formed the basis for the Agreement • The inherent tension between a research study (EGRS) and an intervention (RSP) • The prominent role played by the Project Management Team chaired by DBE in ensuring that the objectives of the study (EGRS) are not compromised. FPD reviewed the Khulisa reports and concurs with many of their findings. However, given the complex history of this project, FPD deems it appropriate to provide clarity on some of the findings that could have been arrived at based on inadequate background information that informed the current RSP design. It is important that Khulisa, as external evaluators, are fully appraised of the issues that dictated the form and content of the RSP Theory of Change and the overall design of the RSP. The approach adopted below is intended to focus attention on some of the key areas of difference. It is our considered view that the issues highlighted below will add substantial value to the work of Khulisa as they finalise the Design Evaluation Report. READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 160 IDENTIFIED INACCURACIES IN THE DRAFT FORMATIVE REPORT 1.Khulisa Report: Implementation in Ngaka Modiri Molema started in Nov 2016. FPDs Response: The RSP Award done on 1 Nov 2016 for the implementation of the pilot project in Dr Ruth Segomotsi Mompati education district in the North West Province. Approval for the Modification of RSP/EGRS Activity was granted on 24 October 2018 for Implementation in Ngaka Modiri Molema and Dr Kenneth Kaunda Districts in January 2019. Implementation in the two districts therefore started in January 2019 and not in 2016. 2.Khulisa Finding: …researchers noted that there were not enough books for all the children” FPDs Response: The material distributed to schools was not based on the number of learners. The package given to teachers was based on an average of 6 teachers per school i.e 2 teachers per grade. 3.Khulisa Finding: “Less than 45% of classrooms have reading mats…” FPDs Response Reading mats are not part of RSP intervention. Equipment and infrastructure in schools do not form part of project scope. Teachers can be taught how to try and manage this and other contextual factors. 4.Khulisa Finding 16 CAs did not receive tablets FPDs Response CAs have not received Tablets to date. The distribution of Tablets is linked to their specific training because it is planned that the Tablets will be loaded with training content. The training of CAs was deferred to year 2 (2020) because of their involvement in Reference Group meetings and centralised training of teachers. Further, some of the materials, including lesson plans have not been finalised to upload on the Tablets for CAs. 5.Khulisa Finding READING SUPPORT PROJECT: DESIGN EVALUATION REPORT 161 Feedback on the MEL Plan FPDs Response The plan was initially submitted to USAID for comments in Jan 2019, based on the project approved in October 2018, in-line with the M&E system which had been developed and was ready for testing at the time. Conversation around DBE Design specification started in February 2019 and warranted a lot of changes to be made to the project processes, M&E system, re￾defining indicators and targets and the tools used to collect data. This is still an on-going process. The new design has not been approved by USAID to date, and therefore specifications have not been updated and incorporated into the Plan. In fact, no comprehensive updates have been made post-Feb 2019 and this was communicated to the DBE’s PMT in the presence of USAID, as well as Khulisa evaluators several times. To evaluate a plan that has not been updated after having received information to that effect is simply unfair. The Plan does not inform the project; it is informed by the project; the MEL plan will be revised as soon as the modifications have been approved and accordingly funded. This also goes for the SOP and data collection tools. The final plan and processes will be supported by RedCap to ensure timely and accurate data. It is anticipated that the Database will be implemented as from February 2020. MEL plan will be updated after approval of the modification incorporating DBE design specifications has been received. 6.Khulisa Finding The Theory of Change (TOC) as it is currently documented is not finally agreed, but the actors in the RSP are working towards a common understanding of what the RSP entails FPD Response The Agreement with USAID is based on the TOC as currently documented. The RSP design is based on the approved TOC. RSP is not intending to review the theory of change. FPD is currently focused on strengthening implementation and not reviewing the TOC. The above points notwithstanding, the Khulisa report provided us with invaluable insight into implementation challenges. Following this report, FPD developed an implementation plan to address the shortcomings/ recommendations of the report. Duly signed by, Dr Abe Seakamela: RSP Chief of Party 15 April 2020 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Jennifer Bisgard Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 8BE05F45-C69C-4489-9B58-CD6BAA840A66 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Katharine Tjasink Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 72D195A6-07EC-40EE-8576-5E8360EA306A READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Leticia Taimo Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 782E2174-DE83-4807-8E1C-F9632BE8ACB0 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Margaret Roper Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 423FBF3E-1BB0-498F-802D-0D6520E06B34 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Benita Williams Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 2E324AEB-673B-4DB8-8931-3CE53635C1A1 READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Jacqui Dornbrack Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: 3AD406BF-F716-422B-A911-F47AFCB66A2E READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Janet Orr Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: E6D722DB-DC15-4D44-B58A-C6B53768238A READING SUPPORT PROJECT: DISCLOSURE OF CONFLICT OF INTEREST Contract No: 72067418D00001, Order Number: 72067419F00015 Proposed evaluation team members are required to sign the below statement attesting to a lack of conflict of interest or describing an existing or potential conflict of interest relative to the program being evaluated that could lead reasonable third parties to conclude that the evaluator or evaluation team member is not able to maintain independence and, thus, is not capable of exercising objective and impartial judgment on all issues associated with conducting and reporting the work. Real or potential conflicts of interest may include, but are not limited to: 1. Immediate family or close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant/ material though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant/material though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. ✔ I declare no conflict of interest □ I declare the following potential conflict of interest: N/A Name Nombulelo Baba Signature ___________________________________ Date October 11, 2019 DocuSign Envelope ID: B4A3525B-BC70-4D10-BF48-0627C2F8BF1B