TUSOME EXTERNAL EVALUATION ENDLINE REPORT July 2020 Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00010, Tasking N 7617.018.01 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by NORC at the University of Chicago through Erika Keaveney, Carlos Fierros, Alexander Rigaux, and Dr. Alicia Menendez with support from Varuni Dayaratna and Charles Munene. This publication was made possible by the support of the American people through USAID. PHOTO SOURCE: ALEX KAMWERU/TUSOME USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | i USAID READING AND ACCESS TUSOME EXTERNAL EVALUATION ENDLINE REPORT (JULY 2020) Prepared under Contract No.: GS-10F-0033M/AID-OAA-M-13-00010, Tasking N 7617.018.01 Revised July 27, 2021 Submitted to: USAID/Kenya and East Africa Submitted by: Erika Keaveney, Carlos Fierros, Alexander Rigaux, and Dr. Alicia Menendez Contractor: NORC at the University of Chicago 4350 East West Highway, 8th Floor Bethesda, MD 20814 Attention: Varuni Dayaratna Tel: 301-634-9414; E-mail: Dayaratna-varuni@norc.org DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | ii CONTENTS TABLES IV FIGURES V ACRONYMS VI EXECUTIVE SUMMARY 1 EVALUATION PURPOSE AND AUDIENCE 1 EVALUATION QUESTIONS 1 EVALUATION METHODS 2 EVALUATION STRENGTHS AND LIMITATIONS 2 FINDINGS 4 CONCLUSIONS 13 RECOMMENDATIONS 14 1. EVALUATION PURPOSE AND QUESTIONS 17 EVALUATION PURPOSE 17 EVALUATION QUESTIONS 18 2. PROJECT BACKGROUND 19 3. EVALUATION METHODS AND LIMITATIONS 21 EVALUATION TEAM 21 DATA COLLECTION TOOLS 22 EARLY GRADE READING ASSESSMENT 22 CLASSROOM OBSERVATION 23 CURRICULUM SUPPORT OFFICER INTERVIEW 23 HEAD TEACHER INTERVIEW 23 TEACHER INTERVIEW 24 PUPIL INTERVIEW 24 YOUTH FUND LEADERSHIP SURVEY 24 QUALITATIVE DATA COLLECTION TOOLS 24 SAMPLING 24 QUANTITATIVE SAMPLING 24 QUALITATIVE SAMPLING 26 DATA COLLECTION 27 SUPERVISOR AND ENUMERATOR TRAINING 27 DATA MANAGEMENT 27 DATA QUALITY ASSURANCE 28 RESEARCH ETHICS AND STUDY AUTHORIZATION 28 DATA ANALYSIS 28 QUANTITATIVE DATA ANALYSIS 28 QUALITATIVE DATA ANALYSIS 29 LIMITATIONS 29 LACK OF COUNTERFACTUAL 29 TIMING OF BASELINE 29 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | iii RESPONSE BIAS 30 LIMITATIONS OF REGRESSION ANALYSIS 30 EXTERNAL VALIDITY 30 DISCONTINUITY BETWEEN SURVEY ROUNDS 31 REPRESENTATIVENESS OF SAMPLE 31 RISK OF DIVERGENT FINDINGS 31 COVID-19 RELATED DELAYS 31 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 32 FINDINGS 32 EVALUATION QUESTION 1 32 EVALUATION QUESTION 3 46 EVALUATION QUESTION 4 51 EVALUATION QUESTION 5 63 CONCLUSIONS 70 RECOMMENDATIONS 72 ANNEXES 74 ANNEX I: ADDITIONAL EGRA RESULTS 74 ANNEX II: SCHOOL SAMPLING METHODOLOGY 81 ANNEX III: EVALUATION STATEMENT OF WORK 83 ANNEX IV: EVALUATION DESIGN MATRIX 90 ANNEX V: DATA COLLECTION INSTRUMENTS 98 ANNEX VI: DISCLOSURE OF CONFLICTS OF INTEREST 184 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | iv TABLES Table 1: English EGRA Raw Subtask Scores................................................................................................................ 6 Table 2: Kiswahili EGRA Raw Subtask Scores............................................................................................................ 7 Table 3: Average Number of Tusome Lessons Completed in October 2016 and 2019 ................................. 9 Table 4: Reading Skills Evaluated by the Kenya Early Grade Reading Assessment..........................................22 Table 5: Planned v. Actual Sample for Quantitative Data Collection .................................................................25 Table 6: Planned v. Actual Sample for Qualitative Data Collection ....................................................................26 Table 7: Endline ORF Performance Categories for English and Kiswahili, by Grade......................................33 Table 8: English Oral Reading Fluency Performance Categories .........................................................................34 Table 9: English Endline ORF Scores by Performance Category, Grade, and School Type ..........................35 Table 10: English Endline ORF Scores by Performance Category, Grade, and Gender.................................35 Table 11: Kiswahili Oral Reading Fluency Performance Categories ...................................................................36 Table 12: Kiswahili Endline ORF Scores by Performance Category, Grade, and School Type ....................37 Table 13: Kiswahili Endline ORF Scores by Performance Category, Grade, and Gender.............................38 Table 14: English EGRA Raw Subtask Scores...........................................................................................................40 Table 15: Kiswahili EGRA Raw Subtask Scores........................................................................................................44 Table 16: Independent Variables Used in Final Regression Models.....................................................................47 Table 17: OLS Regression Results for English ORF ................................................................................................48 Table 18: OLS Regression Results for Kiswahili ORF.............................................................................................49 Table 19: Average Number of Tusome Lessons Completed in October 2016 and 2019.............................55 Table 20: English EGRA Raw Subtask Scores including ML-EL differences.......................................................74 Table 21: Kiswahili EGRA Raw Subtask Scores including ML-EL differences....................................................75 Table 22: Grade 1 English EGRA Raw Subtask Scores, by School Type............................................................76 Table 23: Grade 2 English EGRA Raw Subtask Scores, by School Type............................................................77 Table 24: Grade 1 Kiswahili EGRA Raw Subtask Scores, by School Type ........................................................77 Table 25: Grade 2 Kiswahili EGRA Raw Subtask Scores, by School Type ........................................................78 Table 26: Grade 1 English EGRA Raw Subtask Scores, by Gender.....................................................................78 Table 27: Grade 2 English EGRA Raw Subtask Scores, by Gender.....................................................................79 Table 28: Grade 1 Kiswahili EGRA Raw Subtask Scores, by Gender.................................................................80 Table 29: Grade 2 Kiswahili EGRA Raw Subtask Scores, by Gender.................................................................80 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | v FIGURES Figure 1: English Reading Performance Categories, By Round............................................................................... 4 Figure 2: Kiswahili Reading Performance Categories, by Round............................................................................ 5 Figure 3: Teacher Challenges Implementing Tusome at Endline (Teacher-Reported).....................................9 Figure 4: Youth Fund Grants Program Impact (Grantee-Reported)...................................................................10 Figure 5: Tusome Results Framework........................................................................................................................20 Figure 6: English Reading Performance Categories, By Round.............................................................................34 Figure 7: Kiswahili Reading Performance Categories, by Round..........................................................................37 Figure 8: English Grade 1 Percent Correct EGRA Scores.....................................................................................41 Figure 9: English Grade 2 Percent Correct Reading Scores..................................................................................42 Figure 10: Reading Comprehension A........................................................................................................................43 Figure 11: Reading Comprehension B ........................................................................................................................43 Figure 12: Kiswahili Grade 1 Percent Reading Scores............................................................................................45 Figure 13: Kiswahili Grade 2 Percent Reading Scores............................................................................................45 Figure 14: Teacher Training Participation and Subjective Assessment (Teacher-Reported) ........................52 Figure 15: Number of Days (of 5) the Following Activities Took Place (Teacher-Reported)......................53 Figure 16: Teacher Classroom Practices (Observed).............................................................................................53 Figure 17: Teacher Beliefs on Phonics v. Whole Language Approach to Literacy Acquisition.....................54 Figure 18: Teacher Beliefs on Phonics v. Whole Language Approach to Literacy Acquisition.....................54 Figure 19: Place in Tusome Teacher’s Guide at Time of Data Collection (Observed)..................................55 Figure 20: Teacher Adherence to Tusome Teacher Guides (Observed)..........................................................56 Figure 21: Teacher Challenges Implementing Tusome (Teacher-Reported) ....................................................57 Figure 22: Frequency and Helpfulness of CSO and Head Teacher Observations (Teacher-Reported).....58 Figure 23: Youth Fund Grants Program Impact (Grantee-Reported).................................................................60 Figure 24: CSO Beliefs on Continuation of Tusome at County-Level ...............................................................68 Figure 25: Baseline to Endline Effect Sizes, English Subtests .................................................................................75 Figure 26: Baseline to Endline Effect Sizes, Kiswahili Subtests..............................................................................76 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | vi ACRONYMS APBET Alternative Provision of Basic Education and Training BL Baseline CBC Competency-Based Curriculum CDE County Directors of Education CSO Curriculum Support Officer CWPM Correct Words/Items per Minute DIM Direct Instruction Model EDY Education and Youth Office EGRA Early Grade Reading Assessment EL Endline FGD Focus Group Discussion GoK Government of Kenya IC Instructional Coach IR Intermediate Result IRB Institutional Review Board IRR Inter-Rater Reliability KEA Kenya and East Africa KEMI Kenya Education Management Institute KICD Kenya Institute for Curriculum Development KII Key Informant Interview KNEC Kenya National Examinations Council KSH Kenyan Shillings MoE Ministry of Education MSI Management Systems International NACOSTI National Commission for Science, Technology and Innovation NESP National Education Sector Implementation Plan NTT Tusome National Technical Team ODK Open Data Kit OLS Ordinary Least Squares ORF Oral Reading Fluency PCA Principal Component Analysis PRIMR Primary Math and Reading Initiative PTTC Primary Teacher Training College QASO Quality Assurance and Standards Officer QCO Quality Control Officer R&A Reading & Access RTI Research Triangle Institute SAGA Semi-Autonomous Government Agencies TaRL Teaching at the Right Level TLM Teaching and Learning Materials ToT Training of Trainers TSC Teachers Service Commission TSC-CD Teacher Service Commission County Director USAID United States Agency for International Development USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND AUDIENCE The Tusome endline evaluation is a mixed-methods performance evaluation of the seven-year (2014– 2021) Tusome (“Let’s Read” in Kiswahili) programme. The goal of the evaluation is to address descriptive and normative questions about Tusome, as well as measure change in early grade reading performance following the activity’s national rollout. This will inform USAID and the Government of Kenya (GoK) on designing and implementing sustainable education programming in support of Kenya’s Journey to Self-Reliance. The specific evaluation objectives are to: ● Document results of Tusome during the project period by comparing baseline, midline, and endline results; ● Assess the effectiveness of the design and implementation of Tusome; and ● Document lessons learned and recommendations for future implementation. The intended audience for this evaluation is USAID/Kenya and East Africa (KEA), USAID/Washington, and other USAID education offices around the world; the GoK including the Ministry of Education (MoE), relevant Semi-Autonomous Government Agencies (SAGAs), the Teachers Service Commission (TSC), and county governments; education sector development partners, the private sector, international researchers, and other stakeholders in and outside of Kenya; and the Kenyan public. Tusome ranks among USAID’s first experiences partnering with a host-government to take a piloted literacy program to national scale. As such, this evaluation offers an important and timely case study for translating USAID-funded pilot programs into large-scale national educational reforms. More broadly, results of this evaluation can offer key insights on strategies for transitioning donor-supported educational activities to partner governments, with the ultimate goal of ensuring program sustainability and reducing dependence on foreign assistance. EVALUATION QUESTIONS The evaluation questions for the Tusome endline—developed by USAID/KEA and refined through a stakeholder validation workshop—are as follows: 1. What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? 2. What are the skill levels of grade 1 and 2 pupils on reading subtasks? 3. What school-level and institutional factors influence reading outcomes when implementing at scale, and how? 4. How effective was implementation of Tusome with regard to: a. Teacher buy-in and implementation; b. Community awareness and engagement in reading through Youth Fund grants; and c. Government ownership and buy-in? 5. What are the policy implications of Tusome? a. What GoK procedures, policies, or guidelines have been established as a result of or in support of Tusome? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 2 i. What role did Tusome activities play in the formulation and adoption of such procedures, policies, or guidelines? ii. To what extent are these procedures, policies, or guidelines being implemented? What are the catalysts and barriers to effective implementation? iii. What are the early effects of such procedures, policies, or guidelines? b. Which existing GoK procedures, policies, or guidelines are critical for the long-term sustainability of the program? c. Are there additional procedures, policies, or guidelines that would further enhance Tusome’s sustainability? d. What other key lessons have been learned throughout the policy-making process? EVALUATION METHODS The Tusome endline evaluation is a mixed-methods performance evaluation. Given the diverse goals of the evaluation, several methodologies were used including: pre-post outcomes assessment, which focused on outcomes at the school-level using quantitative data collected at baseline, midline, and endline (evaluation questions 1 and 2); summative evaluation using a cross-section of quantitative and qualitative data to retrospectively assess program effectiveness, stakeholder beliefs and practices, and lessons learned from a variety of perspectives (evaluation questions 3, 4, and 5a); and formative evaluation to prospectively identify procedures, policies, and guidelines that can enhance the sustainability of Tusome-related activities over the long term (evaluation questions 5b-5d). Outcomes-related questions were primarily addressed through a school-level reading assessment, structured classroom observation, and surveys with Curriculum Support Officers (CSOs), Head Teachers, teachers, and pupils. The Early Grade Reading Assessment (EGRA) tool was developed during baseline and includes eight subtasks in English and six subtasks in Kiswahili. Survey instruments were modeled on those used in prior rounds of data collection, however several new questions were added in order to address endline-specific evaluation questions. In addition, new tools including Key Informant Interview (KII) protocols, Focus Group Discussion (FGD) guides, and a Youth Fund leadership web survey were developed for the endline. Primary respondents sampled for this evaluation included the aforementioned school-level respondents; MoE, TSC, and SAGA government officials at the county, regional, and national levels; personnel of USAID/KEA and Tusome implementing partners; and members of youth networks/bunges receiving grant support. For the endline, the evaluation team assessed pupils from the same 204 schools sampled at baseline by Management Systems International (MSI). Using a three-stage cluster sampling procedure from a sampling frame of 22,154 formal public schools and 1,500 non-formal (or Alternative Provision of Basic Education and Training – APBET) schools, the baseline evaluation team drew a clustered, random sample, resulting in a target of 4,896 total pupils comprising 2,448 boys and 2,448 girls divided equally between grades 1 and 2. EVALUATION STRENGTHS AND LIMITATIONS The evaluation methodology and implementation resulted in valid, reliable endline data. The data collection tools and analyses were sufficient for answering the endline evaluation questions. The tools covered a variety of aspects of the Tusome reading intervention by collecting data from pupils, teachers, USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 3 Head Teachers, CSOs, and education officials. Several important limitations should be taken into consideration when interpreting the evaluation findings, a few of which are highlighted here. First, the evaluation used a non-experimental pre-post design, and thus is not able to causally attribute changes in reading performance to Tusome. This is because Tusome was rolled out at a national scale, which made the use of a comparison group infeasible. Instead, the pre-post design examined progress toward Tusome’s stated objectives while relying upon earlier experimental research on USAID’s Primary Math and Reading (PRIMR) initiative to infer that observed improvements can be attributed to Tusome. Second, it is a general best practice (and USAID recommendation) to collect baseline data before the start of an intervention. To ensure comparability over time, midline and endline data should then be collected at the same time point as baseline data during subsequent school year(s). However, due to various issues encountered by MSI, baseline data were collected in June and July 2015, shortly after Tusome started working with grade 1 teachers (but before it started working with grade 2 teachers). It was therefore decided that midline and endline data would be collected during September and October, somewhat after the same time point in the school year as baseline, thus providing slightly more time for learning during the school year.1 Third, the use of regression analysis to address evaluation question 3 is designed to overcome some of the inferential challenges associated with simple cross-tabulations or correlations. Despite this, the risk of omitted variable bias—i.e., the omission of an independent variable that is a determinant of the dependent variable and correlated with one or more independent variables in the model—remains. As such, readers should be cautious in assuming causal relationships between independent variables and pupil reading scores. Furthermore, lack of a statistical relationship between the tested variables and reading performance does not mean that such a relationship does not exist. Fourth, outcomes-level assessment results reflect the period of implementation under USAID support, which involved considerable financial and human resource investments. Consequently, some observed results may not be sustained if/when USAID funding comes to an end, limiting the external validity of findings. In addition, results are not generalizable to private schools or mobile schools serving nomadic communities, as these schools were not targeted by the Tusome intervention and therefore were not included in the original sampling frame of public and APBET schools. While schools serving special needs learners were targeted by Tusome, this external independent EGRA assessment did not cover special needs schools. Finally, the global COVID-19 pandemic led to several delays in validating and finalizing this evaluation report. Consequently, some stakeholders may express concerns that the 2019 data presented herein are dated at the time of publication. It is important to note, however, that face-to-face schooling was deferred from March 2020 to January 2021 due to COVID-19. As such, grade 1 and 2 pupils and teachers spent just three months in schools between the time of data collection and the start of the current school year, meaning findings still reflect relatively recent school conditions. 1 Despite this, there remains a risk of either 1) overestimating changes in reading scores since pupils assessed in September/October have had more classroom exposure than those assessed in June/July or 2) underestimating changes in reading scores, since pupils assessed at baseline had already been exposed to Tusome methods/materials. Given the timing of the program rollout, the risk of (1) is higher for grade 2 while the risk of (2) is higher for grade 1. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 4 FINDINGS Evaluation question 1: What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? To answer evaluation question 1, EGRA performance is reported against MoE-established reading fluency benchmarks. The grade 1 English reading fluency benchmark is 35 or more correct words per minute (CWPM) and the intermediate skill range for emergent readers is 20 to 34 CWPM. The grade 2 English reading fluency benchmark is 65 or more CWPM and the intermediate skill range for emergent readers is 30 to 64 CWPM. At endline, 14 percent of grade 1 pupils were emergent readers and 13.5 percent were fluent readers in English, and 35 percent of grade 2 pupils were emergent readers and 18 percent were fluent readers in English. As shown in Figure 1, the number of non-readers (or zero readers) dropped considerably between baseline (2015) and endline (2019), while the number of fluent and emergent readers increased for both grade levels. The proportion of grade 1 pupils that did not read a single word correctly decreased by 18.4 percentage points, from over half of pupils at baseline to just over a third at endline. The proportion of grade 2 non-readers decreased by around 24 percentage points, from 38 percent at baseline to 14 percent at endline. However, the large drop in non-readers (or zero readers) observed between baseline and midline (2016) was partially reversed by endline for both grades, as was the increase in fluent and emergent readers. Grade 2 pupils showed greater improvements from baseline to endline, having retained the gains observed at midline more so than their grade 1 counterparts. Figure 1: English Reading Performance Categories, By Round For Kiswahili, the grade 1 reading fluency benchmark is 30 or more CWPM and the intermediate skill range for emergent readers is ten to 29 CWPM. The grade 2 Kiswahili reading fluency benchmark is 45 or more CWPM and the intermediate skill range for emergent readers is 17 to 44 CWPM. At endline, 53% 23% 34% 38% 12% 14% 26% 33% 38% 28% 23% 32% 11% 18% 14% 22% 38% 35% 10% 26% 14% 12% 27% 18% Baseline Midline Grade 1 Endline Baseline Midline Grade 2 Endline Zero Reader Beginning Reader Emergent Reader Fluent Reader USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 5 22 percent of grade 1 pupils were emergent readers and four percent were fluent readers in Kiswahili, and 46 percent of grade 2 pupils were emergent readers and eight percent were fluent readers in Kiswahili. In general, Kiswahili reading benchmark performance improved from baseline to endline, but to a lesser degree than English benchmark performance. Notably, the Kiswahili gains observed at midline dropped by endline, particularly for grade 1 pupils. Figure 2: Kiswahili Reading Performance Categories, by Round Evaluation question 2: What are the skill levels of grade 1 and 2 pupils on reading subtasks? From baseline to endline, pupils have shown statistically significant improvements on all EGRA subtasks in both English and Kiswahili. Of particular note, grade 2 pupils’ English improved considerably on higher order skills such as passage reading and comprehension. Indeed, grade 2 English reading gains observed from baseline to endline are nearly equal to gains from an additional full year of schooling.2 Relative to midline, however, endline scores have declined for most subtasks and in a few cases reverted to baseline levels.3 These declines are more pronounced for grade 1 pupils. The raw EGRA scores are shown below in Table 1 for English and Table 2 for Kiswahili. 2 To estimate, we divide the baseline-endline difference in English reading fluency for grade 2 (12.4 CWPM) by the baseline difference in English reading fluency for grades 1 and 2 (13.2 CWPM) which gives 0.94. 3 See Table 20 and Table 21 in Annex I for statistical comparisons between midline and endline. 70% 45% 62% 43% 19% 29% 10% 10% 11% 19% 15% 17% 16% 30% 22% 33% 54% 46% 4% 15% 4% 4% 12% 8% Baseline Midline Grade 1 Endline Baseline Midline Grade 2 Endline Zero Reader Beginning Reader Emergent Reader Fluent Reader USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 6 Table 1: English EGRA Raw Subtask Scores Subtask Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 1.1 3.8 2.8 1.6 *** 0.6 5.0 4.1 3.6 *** Letter Sound Knowledge 15.1 26.1 21.4 6.3 *** 10.2 32.6 26.3 16.1 *** Invented Word Decoding 5.7 10.4 7.7 2.0 ** 10.4 18.4 15.5 5.1 *** English Vocabulary 5.9 7.8 7.1 1.3 *** 8.2 10.2 9.9 1.6 *** Passage Reading (A) 10.6 22.3 13.9 3.3 ** 23.8 43.6 36.2 12.4 *** Reading Comp. (A) 0.2 0.5 0.3 0.1 * 0.5 1.0 1.0 0.5 *** Passage Reading (B) 9.6 22.0 13.4 3.8 ** 21.8 44.2 35.8 14.0 *** Reading Comp. (B) 0.2 0.8 0.5 0.3 *** 0.6 1.7 1.8 1.2 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 7 Table 2: Kiswahili EGRA Raw Subtask Scores Subtask Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 16.5 29.2 22.8 6.3 *** 16.2 39.5 33.8 17.5 *** Syllable Fluency 11.0 21.3 13.7 2.6 * 20.9 36.5 29.7 8.8 *** Invented Word Decoding 4.7 8.1 5.0 0.3 - 10.3 16.1 12.5 2.2 ** Passage Reading 4.9 12.2 6.2 1.3 * 13.5 24.5 19.1 5.6 *** Reading Comprehension 0.4 0.9 0.4 0.1 - 1.1 2.0 1.5 0.4 *** Listening Comprehension 1.2 2.0 2.1 0.9 *** 1.9 2.8 3.1 1.2 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. In order to assess progress on a common scale, we also analyzed average percent correct for each subtask by grade. In English, grade 1 pupils displayed the highest percent correct scores on the vocabulary and phoneme segmentation subtasks, followed by letter sound knowledge. Grade 2 pupils displayed the highest percent correct scores on the passage reading subtasks, followed by vocabulary and phoneme segmentation. In Kiswahili, both grades received the highest percent correct scores on the listening comprehension subtask. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 8 Evaluation question 3: What school-level and institutional factors influence reading outcomes when implementing at scale, and how? To answer evaluation question 3, we conducted multiple regression analysis using a single cross-section of data to assess factors that are associated with reading performance at endline. At the pupil-level, grade, age, gender, language used at school, reading stories at home, and practicing reading aloud at school were all statistically correlated with reading fluency. Pupils who report reading stories at home and/or reading aloud at school score 3-6 CWPM higher than those who don’t, holding all else constant. Reading fluency was also statistically correlated with regularly wearing shoes to school, which is presumed to be a proxy for household socio-economic status. At the classroom level, class size, frequency of pupils sounding out unfamiliar words, Tusome lesson unit number/progress in the Tusome teacher’s guide, use of local language during lessons, use of English and Kiswahili during lessons, and whether a teacher is trained were all statistically correlated with reading fluency. In addition, the availability of tables and chairs/benches for children and the classroom having the timetable posted on the wall were correlated with reading fluency, possibly because they serve as proxies for school leadership and/or school resources. Of particular note, local language use during English instruction is strongly negatively associated with both English and Kiswahili reading fluency, with pupils from such classrooms scoring 7-12 CWPM lower on average. This is consistent with open-ended feedback/suggestions from 6 percent of interviewed teachers who say restricting language use at school to English and Kiswahili only—thus reducing or eliminating the use of other languages at school—is the best way to improve teaching and learning of English and Kiswahili. Regression results show that higher implementation of Tusome in the classroom positively correlates with English reading fluency: each 10-unit advance in the Tusome teacher’s guide is associated with an increase of one CWPM in English reading fluency. Similarly, teacher-reported frequency of pupils sounding out unfamiliar words is positively associated with English reading performance. When teachers use both English and Kiswahili while teaching Kiswahili—i.e., use "code switching"—pupils score 3.19 words per minute higher in Kiswahili. Evaluation question 4.a: How effective was implementation of Tusome with regard to teacher buy-in and implementation? To address question 4a, the evaluation team drew upon multiple data sources to explore three broad categories of program activities including improved reading instruction methods and delivery, use of teaching and learning materials, and teacher supervision and support from tutors/coaches. Overall, teachers generally demonstrated moderate to high levels of support for the Tusome trainings and teaching methodologies. Those who attended one or more Tusome trainings universally reported a positive experience, with 99 percent of teachers finding the trainings both useful and relevant. Most teachers also expressed a preference for structured teacher’s guides over developing their own lesson plans. During data collection, field teams observed and assessed teachers’ overall adherence to the Tusome teacher’s guide as well as their coverage of the unit(s) taught that day. Over 90 percent of teachers who were observed using the guide followed the instructions/script and skipped minimal content, if any. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 9 While teachers seem comfortable using Tusome materials in the classroom, they have faced significant challenges after the midline in covering all of the approved content following the 40 percent reduction in instructional hours allocated to English and Kiswahili under the new Competency-Based Curriculum (CBC) timetable implemented in 2019. That is, English instruction was reduced from five to three periods per week and Kiswahili instruction was similarly reduced from five to three periods per week under the initial CBC timetable. Endline results found on average that teachers were around 30 (of 150 total) lessons behind midline levels (see Table 3). Table 3: Average Number of Tusome Lessons Completed in October 2016 and 2019 Subject Grade 1 Grade 2 Midline Endline Difference Midline Endline Difference English language activities 122.2 90.6 -31.6 125.3 95.2 -30.1 Kiswahili language activities 122.3 89.6 -32.7 127 90.4 -36.6 As shown in Figure 3, insufficient time was the main Tusome implementation challenge teachers reported at endline, with 39 percent of teachers who face challenges saying they lack sufficient time to cover the content and 23 percent claiming that Tusome lesson pacing is too fast. In addition, 31 percent of teachers report “lack of materials” as a challenge in implementing Tusome, particularly supplementary reading and learning materials for pupils. Figure 3: Teacher Challenges Implementing Tusome at Endline (Teacher-Reported) Yes, 54% No, 46% Teachers have challenges implementing Tusome 39% 31% 23% 12% 9% 6% 5% 4% 3% 3% 1% 0% 10% 20% 30% 40% Not enough time to cover Not enough materials Lesson pacing too fast Misalignment w/ work schemes Materials old/damaged Misalisgnment w/ CBC Errors in printing Pupil language barrier Lesson pacing too slow Not effective Inadequate training USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 10 In terms of teacher supervision and support, teachers find coaching and supervision from Curriculum Support Officers (CSOs)/Instructional Coaches (ICs)4 and Head Teachers helpful; however the frequency of such visits has declined since midline, with a 20 percentage point drop in the number of teachers who reported being visited by a CSO at least once per term. Qualitative data show that county-level respondents believe CSO and Quality Assurance and Standards Officer (QASO) visits have declined recently for various reasons, including: heavy workloads with multiple schools/zones to visit, lack of motivation due to their heavy workload, and lack of mobility. Evaluation question 4.b: How effective was implementation of Tusome with regard to community awareness and engagement in reading through Youth Fund grants? As part of the Tusome intervention, USAID’s implementation partner RTI established Youth Fund grants to raise community awareness and engagement around the importance of early grade reading at the grassroots level. To this end, the Youth Fund provided grants to existing youth groups to carry out reading activities in their communities. The Tusome Youth Fund grant program was able to use over US$690,000 to reach 66,512 direct beneficiaries in 526 schools, as reported by the youth groups themselves. Figure 4: Youth Fund Grants Program Impact (Grantee-Reported) 66,512 526 226,167 69,176,968 DIRECT BENEFICIARIES SCHOOLS REACHED INDIRECT BENEFICIARIES FUNDING RECEIVED (KSH) The overall implementation of the Youth Funds grants was successful. In total, 23 youth groups received grants to implement activities in their communities, and after implementation of the first round of activities each youth group received a second round of funding to implement activities in a second location. Activities sponsored by these grants include mobile libraries, reading exhibitions, reading clubs, theater groups, murals, and parental sensitization campaigns. When asked, Youth Fund grant administrators on average gave their programs an efficacy rating of 3.85 out of four for both raising community awareness on the importance of reading and engaging children in reading.5 However, while the grants program was considered successful, the administration of the program presented some challenges. Two respondents reported difficulty overseeing 23 grantees under the larger Tusome intervention since the grants required high levels of financial scrutiny and administrative oversight, at times requiring technical staff to support the Youth Fund grants program beyond what was originally anticipated. Respondents explained that the Tusome team was prepared to implement the 4 Instructional Coaches (ICs) play the equivalent role of CSOs vis-à-vis Tusome implementation for APBET institutions. For the purpose of simplicity, the term CSO is intended to refer to both CSOs and ICs for the remainder of this report. 5 Grantees were asked via web survey to score the effectiveness of their grant programs from one to four, with one being very ineffective and four being very effective. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 11 Youth Fund grants, however they found that the Kenyan youth groups engaged in the Tusome activity had lower capacity than expected and required greater support throughout implementation than originally planned. The evaluation found that the youth groups required substantial support identifying the types of activities that would qualify for the grants as well as training on how to manage the activities, how to properly report activity progress, and how to ensure the activities adhered to USAID’s branding requirements. While this level of training and capacity building was not originally planned for, it was a positive unintended outcome of the grants program. Qualitative data also show the capacity of these organizations increased over time. Evaluation question 4.c: How effective was implementation of Tusome with regard to government ownership and buy-in? Overall, Tusome was successful in securing ownership and buy-in from the GoK at different levels of government and at different points in the implementation process. Qualitative data show that government ownership and buy-in were priorities from the outset of Tusome, and GoK respondents were personally supportive of the methods and materials developed under Tusome and hoped to see them continue in the future. While Tusome was broadly successful in securing ownership and buy-in from the GoK, there were moments throughout implementation where USAID and RTI had to work harder to secure support for the Tusome methods, including during the CBC development process. In spite of these accounts of broad support from the GoK, it is still seen as important to continue building deeper government ownership and buy-in to fully transition the Tusome methods and materials to the GoK for long-term implementation. In particular, USAID, RTI, and GoK respondents identified the need for the GoK to take more financial ownership of Tusome, specifically integrating different costs of Tusome into the national budget when it ends. Evaluation question 5.a: What GoK procedures, policies, or guidelines have been established as a result of or in support of Tusome? Through qualitative interviews, three key policies that were established as a result or in support of Tusome emerged. First, with Tusome’s support, the GoK was able to successfully implement a centralized book procurement procedure. The new centralized procedure led to economies of scale, enabling the GoK to purchase textbooks at a significantly reduced rate. The cost savings allowed the GoK to purchase textbooks at a 1:1 student-to-textbook ratio, thereby operationalizing for the first time an existing policy for providing each student with his or her own textbook. Second, the timing of the CBC reform overlapped with the implementation of the Tusome program, which presented an opportunity to integrate Tusome methods and materials into the new national curriculum. Grade 3 Tusome student textbooks and teacher guides were aligned to the CBC in 2017, while grade 1 and 2 Tusome teaching and learning materials were aligned to the CBC in 2018. However, it should be noted that while the Tusome method is formally integrated into the CBC, there was confusion at the time of data collection among some teachers on how this translates into teaching in the classroom, particularly as it relates to the use of learning materials and the amount of time dedicated to each subject.6 Third, Tusome was able to leverage civic responsibility and funding to motivate existing GoK personnel to 6 Since the collection of endline data in 2019, the Ministry of Education issued a circular in early 2020 to address the confusion among teachers identified by the evaluation. To help teachers operationalize the circular, the MoE intended to further clarify the time dedicated to classroom instruction for English and Kiswahili using Tusome materials through teacher training that unfortunately had to be postponed due to COVID-19. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 12 implement more frequent instructional support through classroom-based observations and teacher coaching. These classroom observations capture data on teacher and pupil performance, which are uploaded to a central dashboard created by the Tusome project and accessible by county, regional, and national stakeholders. The classroom observations are considered a success, and respondents believe policies and systems should be put in place to keep CSOs accountable for observing teachers after Tusome implementation ends. National and county-level stakeholders specifically mentioned using the Tusome dashboard, a tool primarily used to track teacher and learner performance, as a possible tool to track classroom observations and keep CSOs accountable for their classroom observations. Evaluation question 5.b: Which existing GoK procedures, policies, or guidelines are critical for the long-term sustainability of the program? Qualitative findings indicate that the most critical GoK procedures, policies, or guidelines for the long￾term sustainability of the program are the continuation of the centralized book procurement procedure and the continuation of more frequent classroom observations and teacher coaching sessions by CSOs, as described under question 5.a. The qualitative findings indicate the centralized book procedure is critical to the long-term sustainability of the program. It is important to note that the centralized book procedure has also led to significant reductions in the cost of textbooks for the GoK. This has enabled the GoK to achieve its policy of providing textbooks to students at a 1:1 ratio, which is key to improving student learning. Evaluation question 5.c: Are there additional procedures, policies, or guidelines that would further enhance Tusome’s sustainability? The qualitative findings in regard to what additional procedures, policies, or guidelines would enhance Tusome’s sustainability revolve primarily around how to reinforce the newly rolled out CBC and ensure its implementation is uniform across schools. The most common recommendation was the need for clarification of key points of the CBC. Respondents reported confusion around both the amount of time that should be dedicated to English, Kiswahili, and literacy lessons and the materials that should be used for these lessons. This was likely due to the fact that under the initial CBC timetable, there was a 40 percent reduction in instructional hours allocated to English and Kiswahili (from five to three periods per week per language), and other teaching and learning materials beyond those developed under Tusome were made available to schools. Subsequent to data collection in 2019, the Kenyan Ministry of Education released a 2020 circular aimed at clarifying questions related to the time dedicated to classroom instruction for English and Kiswahili using Tusome materials. According to USAID, the circular increased the number of periods allocated to Tusome English back to five per week and the MoE had planned to help teachers operationalize the circular through national teacher training. However due to COVID-19, the MoE shut down schools nationwide, and plans for teacher training had to be postponed indefinitely. Evaluation question 5.d: What other key lessons have been learned throughout the policy-making process? The key lesson learned based on qualitative findings, which has not been discussed in the evaluation questions above, is the importance of collaboration between USAID, RTI, and the GoK in the design and implementation of Tusome. This sentiment was expressed by each stakeholder group, in one form or another, most of whom believe that Tusome’s success is due in large part to the program’s ability to engage government officials at all levels and foster buy-in from them. Some GoK respondents reported USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 13 moments of animosity between different GoK stakeholders at the start of implementation, indicating they felt the intervention was being imposed on them without an opportunity to provide their inputs. These respondents argue there should have been more dialogue in the initial stages of the Tusome design in order to allow the MoE, TSC, and SAGAs to fully understand the activities that were being implemented and give them the space to provide feedback for improvements to the program. Nevertheless, the majority of respondents applauded the collaboration between USAID, RTI, and the GoK through the design and implementation of Tusome. CONCLUSIONS Overall, independent evaluation results provide strong evidence for the efficacy of the Tusome model of instruction. From baseline to endline, pupils have shown statistically significant improvements on all EGRA subtasks in both English and Kiswahili. Of particular note, grade 2 English reading gains observed from baseline to endline are roughly equal to gains from an additional full year of schooling. Furthermore, over the life of the evaluation exposure to Tusome has consistently corresponded with reading performance: increased exposure to Tusome between baseline and midline/endline corresponded with an increase in pupil reading performance. Similarly, reduced exposure to Tusome between midline and endline corresponded with a decrease in pupil reading performance. Statistical models at endline also show that better implementation of Tusome in the classroom positively correlates with reading fluency. Tusome lesson plan progress is positively associated with English reading performance: each 10-unit advance in the Tusome teacher’s guide is associated with an increase of one correct word per minute in English reading fluency. Similarly, teacher-reported frequency of pupils sounding out unfamiliar words is positively associated with English reading performance. When teachers use both English and Kiswahili while teaching Kiswahili—i.e., use “code switching"—pupils score 3.19 words per minute higher in Kiswahili. Despite marked progress since baseline, the vast majority of early grade learners in Kenya are unable to read at a grade-appropriate level. Across the board, fewer than eight percent of grade 1 and 2 pupils are able to read Kiswahili at a grade-appropriate level. In English, just 14 percent of grade 1 pupils and 18 percent of grade 2 pupils are meeting Kenya’s reading benchmarks. Furthermore, there is a concerningly high proportion of pupils who are unable to read entirely: nearly two-thirds of grade 1 pupils and one-third of grade 2 pupils cannot read a single word in Kiswahili. For English, the number of non-readers is 34 and 14 percent for grades 1 and 2, respectively. The reduction in English and Kiswahili instructional hours under the new CBC timetable appears to be a driver of post-midline declines in reading performance. While several implementation factors changed since midline, teachers were on average around 30 (of 150 total) Tusome lessons behind midline levels due to the reduction in English and Kiswahili instructional time. That grade 1 pupils saw the sharpest performance losses since midline fits with this hypothesis, as grade 2 pupils were still under the pre-CBC timetable when they were in grade 1 and thus had a higher “dosage” of English and Kiswahili using the Tusome approach compared to their grade 1 counterparts. Overall, reduced exposure to Tusome under the new CBC timetable appears to have negatively impacted children’s English and Kiswahili skills. Regular reading practice and classroom resources are positively correlated with reading fluency while larger class sizes and the use of local language at school are negatively correlated with reading fluency. Pupils who report reading stories at home and/or reading aloud at school score 3-6 correct words per minute higher than USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 14 those who don’t, holding all else constant. In addition, smaller class sizes, the availability of tables and chairs/benches for children, and the classroom having the timetable posted on the wall were correlated with reading fluency, possibly because they serve as proxies for school leadership or school resources. On the other hand, local language use during English instruction is strongly negatively associated with both English and Kiswahili reading fluency, with pupils from such classrooms scoring 7-12 correct words per minute lower on average (no such correlations were found for local language use during Kiswahili instruction). Teachers generally demonstrate moderate to high levels of support for Tusome, yet face a number of implementation challenges. While teachers broadly support Tusome, they have struggled to keep pace with the instructional approach from the midpoint of the program onward. Insufficient time was the main implementation challenge teachers reported at endline, with 39 percent of teachers who face challenges saying they lack sufficient time to cover the content and 23 percent claiming that Tusome lesson pacing is too fast, likely owing to the CBC timetable reduction in instructional time from five periods to three periods per week for English and five periods to three periods per week for Kiswahili. In addition, 31 percent of teachers report “lack of materials” as a challenge in implementing Tusome, particularly materials for pupils such as story books and homework books. Other frequently cited implementation challenges include misalignment with work schemes (12 percent), old or damaged materials (9 percent), and misalignment with CBC (6 percent). Tusome was largely successful in securing ownership and buy-in from the GoK at different levels of government and at different points in the implementation process, but more work is needed to ensure long-term sustainability after USAID funding comes to an end. Most GoK respondents were personally supportive of the methods and materials developed under Tusome and hoped to see them continue in the future. However, long￾term sustainability of Tusome will depend on upholding existing procedures, policies, and guidelines— particularly the centralized book procurement procedure, with its facilitation of the 1:1 textbook to student ratio, and CSO observations/coaching. RECOMMENDATIONS Ensure that Kenya’s learners achieve the English and Kiswahili reading goals of the CBC by allocating sufficient instructional time to cover the content developed under Tusome. The reduction in lessons from five to three per week during the 2019 school year led to gaps in coverage of Tusome instructional content. As a result, pupil reading performance actively worsened, moving away from GoK reading benchmarks. The GoK should thus consider increasing the instructional time for both English and Kiswahili back to five lessons per week to better align with the Tusome materials and Kenya’s reading goals under the CBC. Likewise, the GoK should formally sensitize educators across the Kenyan system to the change through trainings and other communication channels. Implement remedial literacy programming for pupils in the “zero” and “beginner” reader categories. While there has been marked progress since baseline, the great majority of pupils in Kenya are still unable to read with comprehension at a grade-appropriate level. Remedial programming for struggling readers could involve supplementing core reading lessons with additional English and Kiswahili instruction and guided practice time tailored to the learning levels and needs of pupils or extracurricular reading support by tutors, volunteers, teachers, and/or through education technology. Cross-cultural research on “teaching at the right level” (TaRL) shows that grouping children based on learning levels—for example, their EGRA benchmark categories—rather than age or grade and tailoring instruction and USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 15 practice based on what they know can lead to large, cost effective gains in learning outcomes for struggling pupils. While the TaRL approach should not replace instruction using Tusome materials, the value and importance of remedial reading instruction is especially salient given the extended school closures during the COVID-19 pandemic, which now require learners to “catch up” in order to acquire expected foundational skills. Ensure pupils have the time and resources to regularly practice reading, both inside and outside of the classroom. Given the strong, statistically robust relationship between pupil reading performance and regular reading practice, ensure that children are able to practice reading in a variety of formats, including practicing reading silently/aloud as well as at home/in the classroom. This may involve ensuring access to English and Kiswahili decodable and leveled story books and work books for home use or supporting extracurricular reading clubs, especially for children whose home environments are less conducive for reading practice. Work with appropriate GoK actors to put in place systems to ensure classroom-based observations continue after USAID funding ends. This might include creating accountability systems to ensure CSO/QASO school visits take place under the current GoK transport facilitation scheme, additional school-based support to complement CSO/QASO observations, or alternatives to face-to-face training and coaching in situations where face-to-face interactions are not feasible. The Tusome dashboard can be transferred to the Ministry of Education and serve as an accountability system to ensure CSOs/QASOs are conducting school-based observations, in addition to its primary purpose of providing valuable school-level data for national, regional, and local stakeholders. Additional school-based support can be found in Head Teachers and Deputy Head Teachers who can conduct classroom observations in their schools, reducing the burden on CSOs/QASOs and the current GoK transportation facilitation scheme. Finally, alternatives to face-to-face interaction may include using technology to implement virtual CSO/QASO observations and coaching. Implementing a remote CSO/QASO system would require an initial investment in the technology, for example tablets and virtual meeting software, as well as training CSOs/QASOs and teachers on how to use the new technology, but would cut the recurring CSO/QASO transport facilitation costs. A move to virtual observations and coaching can capitalize on the current expansion of virtual meeting technology that the world experienced as a result of the COVID-19 pandemic. Continue working with the GoK including MoE, TSC, and SAGAs to build renewed and deeper buy-in for key Tusome policies and practices as well as future USAID-funded programming. USAID, RTI, and implementers of future education interventions should continue participatory engagement with GoK stakeholders— across all seniority levels within the MoE, TSC, and SAGAs—to build buy-in for donor-supported initiatives and ensure they are aligned with government policies and priorities. Deeper buy-in across the GoK will foster a greater sense of ownership as well as insulate key programs from leadership transitions over the longer-term. Developing a clear strategy for fully transitioning Tusome to government ownership—including ensuring timely and sufficient budget allocations from MoE and GoK—is also critical for long-term sustainability. Work with the Ministry of Education to implement a revised textbook policy that includes centralized procurement. Given the importance of the centralized book procurement procedure implemented by Tusome, and the resulting 1:1 student to textbook ratio, USAID, RTI, and implementers of future education interventions should work with the GoK to implement a revised textbook policy solidifying USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 16 the centralized procurement procedure. A revised textbook policy will ensure students have access to critical learning materials while reducing the costs of textbook procurement for the GoK, and ensure the long-term sustainability of a core aspect of the Tusome intervention. For future interventions, the design of Youth Fund grants programs should take into account the existing capacity of local youth organizations, and ensure enough funding/staff to allow for sufficient administrative and managerial oversight. Given the level of effort needed to administer and manage the Youth Fund grants under the Tusome project, future interventions should familiarize themselves with the existing capacity of local organizations, including their capacity for implementing activities as well as their capacity for reporting on those activities. This will allow future interventions to secure enough funding and staffing for the administrative and managerial needs of the grants. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 17 1. EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE The USAID/Kenya and East Africa (KEA) Office of Education and Youth (EDY), in partnership with the Kenya Ministry of Education, Science and Technology (MoE), is implementing an $88.8 million basic education initiative to improve the reading skills of approximately 7.8 million Kenyan children who began primary school during the 2014-2021 school years. The project, Tusome (Too-SOH-meh; “Let’s Read” in Kiswahili), will continue through 2021, and has integrated options for transition to government ownership during the out years. In response to the COVID-19 pandemic, the project also began working with MoE and other partners to develop distance learning modules and catch-up instructional content. Tusome builds on research-based reading initiatives to create a sustainable and affordable national reading program in Kenya. It seeks to improve the English and Kiswahili skills of grade 1, 2, and 3 children in 24,038 Kenyan primary schools (22,538 public schools and 1,500 APBET schools) across the country. Tusome also assists the Government of Kenya (GoK) at the technical and policy levels to sustainably improve reading skills beyond the life of the project. To this end, Tusome works to build the capacity of the GoK, specifically MoE, the Teachers Service Commission (TSC), and relevant Semi￾Autonomous Government Agencies (SAGAs) to implement literacy activities. Tusome seeks to improve children’s reading skills on a nationwide scale through evidence-based programming. In line with USAID policy, the testing of innovative activities is built into the Tusome design and the previous Primary Math and Reading (PRIMR) initiative, which developed and tested the methodology that lies at the heart of Tusome. The overarching purpose of this evaluation is to: • Highlight the achievements of Tusome project goals; • Highlight the program’s impact on Kenya’s education policies; and • Document lessons learned during its implementation. This will inform USAID and the GoK on designing and implementing sustainable education programming in support of Kenya’s Journey to Self-Reliance. The specific evaluation objectives are to: • Document results of Tusome during the project period by comparing baseline, midline, and endline results; • Assess the effectiveness of the design and implementation of Tusome; and • Document lessons learned and recommendations for future implementation. The intended audience for this evaluation is USAID/Kenya and East Africa (KEA), USAID/Washington, and other USAID education offices around the world; the GoK including the Ministry of Education (MoE), relevant Semi-Autonomous Government Agencies (SAGAs), the Teachers Service Commission (TSC), and county governments; education sector development partners, the private sector, international researchers, and other stakeholders in and outside of Kenya; and the Kenyan public. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 18 EVALUATION QUESTIONS The evaluation questions for the Tusome endline—developed by USAID/KEA and refined during a stakeholder validation workshop held August 28, 2019—are as follows: EVALUATION QUESTION 1 What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? EVALUATION QUESTION 2 What are the skill levels of grade 1 and 2 pupils on reading subtasks? EVALUATION QUESTION 3 What school-level and institutional factors influence reading outcomes when implementing at scale, and how? EVALUATION QUESTION 4 How effective was implementation of Tusome with regard to: • Sub-question 4.a: Teacher buy-in and implementation; • Sub-question 4.b: Community awareness and engagement in reading through Youth Fund grants; and • Sub-question 4.c: Government ownership and buy-in? EVALUATION QUESTION 5 What are the policy implications of Tusome? • Sub-question 5.a: What GoK procedures, policies, or guidelines have been established as a result of or in support of Tusome? i. What role did Tusome activities play in the formulation and adoption of such procedures, policies, or guidelines? ii. To what extent are these procedures, policies, or guidelines being implemented? What are the catalysts and barriers to effective implementation? iii. What are the early effects of such procedures, policies, or guidelines? • Sub-question 5.b: Which existing GoK procedures, policies, or guidelines are critical for the long-term sustainability of the program? • Sub-question 5.c: Are there additional procedures, policies, or guidelines that would further enhance Tusome’s sustainability? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 19 • Sub-question 5.d: What other key lessons have been learned throughout the policy-making process? 2. PROJECT BACKGROUND7 As noted in Kenya’s 2013-2018 National Education Sector Implementation Plan (NESP), GoK’s passage of a reform package in 2003 guaranteeing free primary education precipitated drastic improvements in education access and equity throughout Kenya. While encouraging, enrollment increases were not accompanied by a corresponding increase in resources, straining the education system and leading to sharp declines in the quality of instruction and pupil reading performance. Reading is the foundational skill upon which other educational milestones depend. As a result, poor reading performance in the early grades produces negative ripple effects throughout the education system including increasing costs due to grade retention/repetition and dropouts and undermining the achievement of Kenya’s Vision 2030 goals. The NESP highlights raising literacy and numeracy levels among its core focuses. In line with GoK priorities and USAID’s strategic focus on early grade reading in its Education Strategy, Tusome addresses the need to improve learning outcomes for young children in all Kenyan public schools and 1,500 low-cost institutions, all of which teach the content in the approved Kenya Institute for Curriculum Development (KICD) syllabi. Starting in 2007, USAID/KEA and MoE ran a one-year randomized controlled trial in 40 schools in the Malindi district. Building on these findings, USAID/KEA funded a three-year applied research programme—the Primary Math and Reading Initiative (PRIMR). According to PRIMR reports, this activity reached 56,036 pupils across 547 formal public schools and APBET institutions in Nairobi, Kiambu, Nakuru, and Kisumu and led to dramatic improvements in pupils’ ability to read. In formal schools, pupils were between 2.9 and 27.9 times more likely to read than in the non-PRIMR schools right next door. In non-formal schools, pupils were between 1.9 and 3.3 times more likely to read than those in non-PRIMR schools. Following the success of PRIMR, MoE requested a national expansion of the PRIMR model. USAID/KEA awarded Tusome to RTI International in 2014 in order to support scale up of the intervention nationally. This seven-year, $88.8 million basic education initiative was designed to improve the reading skills of the approximately 7.8 million individual Kenyan children who began primary school during the 2014-2021 school years. The Tusome activity is intended to 1) scale up the previous (2011-2014) PRIMR pilot activity and 2) increase the capacity of GoK to deliver and administer early grade reading programmes nationwide. Tusome will continue through 2021 and has integrated options for transition to government ownership. In response to the COVID-19 pandemic, the project also began working with MoE and other partners to develop distance learning modules and catch-up instructional content. As was initially detailed in Tusome’s Performance Management Plan, the main program objective is to “sustainably improve reading skills of grade 1 and 2 learners.” Grade 3 was also included later in the implementation process. The means for achieving this objective are outlined in two intermediate results: 1) improved supervision, support, and delivery of reading instruction to target students and 2) improved 7 Adapted from Tusome Statement of Work and Midline Evaluation Report. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 20 government capacity, in target departments, to sustainably improve reading outcomes. The Tusome activity results framework is presented below in Figure 5. Figure 5: Tusome Results Framework Program Objective Sustainably improve reading skills of Standard 1 and 2 students Intermediate Result (IR) 1 Improved supervision, support, and delivery of reading instruction to target students Sub-IR 1.1 Increased availability and use of appropriate textbooks and supplementary materials that support reading Sub-IR 1.2 Improved methods of reading instruction delivery Sub-IR 1.3 Supervision support provided to teachers by TAC Tutors and Head Teachers in teaching and assessing reading IR 2 Improved government capacity, in target departments, to sustainably improve reading outcomes Sub-IR 2.1 Increased capacity of GoK to design, implement, and assess programs to increase reading skills Sub-IR 2.2 Improved policy environment that promotes reading skills USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 21 3. EVALUATION METHODS AND LIMITATIONS The Tusome endline evaluation is a mixed-methods performance evaluation. The goal of the evaluation is to address descriptive and normative questions about Tusome, as well as measure change in early grade reading performance following the activity’s national rollout. Given the diverse goals of the evaluation, several methodologies were used including: pre-post outcomes assessment, which focused on outcomes at the school-level using quantitative data collected at baseline, midline, and endline (evaluation questions 1 and 2); summative evaluation using a cross-section of quantitative and qualitative data to retrospectively assess program effectiveness, stakeholder beliefs and practices, and lessons learned from a variety of perspectives (evaluation questions 3, 4, and 5a); and formative evaluation to prospectively identify procedures, policies, and guidelines that can enhance the sustainability of Tusome￾related activities over the long term (evaluation questions 5b-5d).8 NORC elucidated and refined the overarching analytical strategy through an evaluation design report and design matrix (see Annex VI). The evaluation design report underwent several rounds of review by USAID/KEA and RTI, culminating in an Evaluation Design Validation Workshop held on August 28, 2019 in Nairobi which was attended by key stakeholders across the Kenyan education system. EVALUATION TEAM The Tusome endline evaluation was conducted by NORC at the University of Chicago under USAID’s Reading & Access (R&A) Evaluations contracting mechanism. The NORC team consisted of national and international experts in education and social sciences research, and was supported by local data collection subcontractor Dalberg Research. Evaluation team members included: • Evaluation Team Lead – Erika Keaveney • Local Technical Expert – Charles Munene • Qualitative Lead and Trainer – Carlos Fierros • Research Analyst – Alexander Rigaux • R&A Principal Investigator – Dr. Alicia Menendez • R&A Project Director – Varuni Dayaratna • Quality Control Officers, Field Supervisors, and Enumerators The evaluation was also supported by Management Systems International (MSI), which managed both baseline and midline data collection and provided transitional support to the NORC team at endline. 8 For more information on proposed methodologies, see USAID (n.d.). Performance Evaluation Designs. Retrieved from https://www.usaid.gov/project-starter/program-cycle/project-monitor-evaluation-plan/monitor-evaluation-plan-evaluation￾component/performance-evaluation-designs. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 22 DATA COLLECTION TOOLS Primary data collection instruments were selected based on their ability to triangulate the evaluation questions. Outcomes-related questions were primarily addressed through school-level reading assessments, structured classroom observations, and survey tools. These tools were modeled on those used at midline, however several new questions were added in order to address endline-specific evaluation questions. In addition, new tools including a web survey, KII protocols, and FGD guides were developed by NORC and reviewed/revised by stakeholders at USAID/KEA, RTI, and MoE. EARLY GRADE READING ASSESSMENT The Early Grade Reading Assessment (EGRA) was developed by USAID in response to a growing demand for an open source and easy-to-use tool that assesses learner abilities along a reading continuum, from pre-reading skills to reading fluency and comprehension. EGRA has been adapted and used in over 70 countries including Kenya, which conducted national EGRA assessments in 2009, 2015, and 2016. In Kenya, the EGRA tool was piloted, adapted, and validated in both English and Kiswahili. Specific subtests included in the EGRA used for this evaluation, along with their associated reading stages, are described in Table 4 below. Table 4: Reading Skills Evaluated by the Kenya Early Grade Reading Assessment Reading Stage Subtask Description Language(s) Pre-reading Listening comprehension A simple story is read out loud by an enumerator. Pupils are then asked five comprehension questions based on the story. Subtest is scored as both number of items correct (raw score) and percent correct. Kiswahili Vocabulary Pupils are given 20 vocabulary items verbally, one after the other, and asked to point to/show/demonstrate the item. Subtest is scored as both the number of items correct (raw score) and percent correct. English Initial reading Letter sound knowledge Pupils are given a stimulus sheet containing 100 letters and asked to produce sounds for as many of the letters as they can in 60 seconds. Subtest is scored as correct letters produced per minute and percent correct. English, Kiswahili Non-word decoding Pupils are given a stimulus sheet containing 50 unfamiliar written words (non-words) and asked to read as many non-words as they can in 60 seconds. Subtest is scored as correct non-words produced per minute and percent correct. English, Kiswahili Phoneme segmentation Pupils are given ten words verbally, one after the other, and asked to produce the sounds they hear in the word: e.g., cat = /k//a//t/. Subtest is scored as the number of correct phonemes produced (raw score) and percent correct. English USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 23 Reading Stage Subtask Description Language(s) Syllable fluency Pupils are given a stimulus sheet containing 100 syllables and asked to read as many of the syllables as they can in 60 seconds. Subtest is scored as correct syllables produced per minute and percent correct. Kiswahili Fluency and comprehension Passage reading Pupils are given a short passage (60 to 70 words) to read within 60 seconds. Subtest is scored as correct words produced per minute and percent correct. English, Kiswahili Oral reading comprehension Pupils are verbally asked up to six comprehension questions about the passage they read in the previous subtest. Subtest is scored as both the number of items correct (raw score) and percent correct. English, Kiswahili CLASSROOM OBSERVATION The classroom observation checklist was developed in collaboration with MSI, USAID/KEA, MoE, and the Tusome team. The classroom observation tool was administered by a trained enumerator who sits in the back of the classroom during a Kiswahili or English lesson9 and takes periodic “snapshots” of classroom activities and records information related to teacher focus, instructional content, teacher actions, pupil actions, and the materials in use. In addition, the observation tool captured an inventory of classroom materials and equipment as well as the enumerator’s overall assessment of the teacher with respect to feedback offered to pupils, lesson pacing, language use, and adherence to the Tusome lesson plan and instructional model. CURRICULUM SUPPORT OFFICER INTERVIEW The Curriculum Supporting Officer (CSO) interview captured CSOs’ self-reported frequency of school visits and beliefs on the efficacy of the Tusome instructional approach. At endline, the CSO interview was appended to include additional questions related to localized sustainability of Tusome after USAID funding ends. HEAD TEACHER INTERVIEW The Head Teacher survey captured school-level information about instructional practices, curriculum, and pupil and teacher assessment. In addition, the instrument captured school characteristics such as availability and quality of school facilities and management structures. At endline, the Head Teacher survey was expanded to include a school climate protocol in order to better understand mediators and moderators of program effectiveness at the school-level. 9 Whether English or Kiswahili is observed for a given school was randomly determined, based on an anonymized and randomly assigned school ID number; school IDs ending in odd numbers were assigned Kiswahili whereas school IDs ending in even numbers were assigned English. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 24 TEACHER INTERVIEW The teacher interview captured information on the teacher’s qualifications; pupil reading habits inside and outside of the classroom; support/coaching received from Head Teachers and CSOs; participation in, and perceived value of, Tusome training; and self-reported teaching practices related to the Tusome model of instruction. PUPIL INTERVIEW The pupil interview was a brief survey completed immediately after administration of the EGRA. The pupil interview asked questions on the language spoken in the home environment, availability of reading materials at home, reading practices both inside and outside of the classroom, and household assets. YOUTH FUND LEADERSHIP SURVEY A web-based survey module was developed and administered to leaders/representatives from Tusome Youth Fund grant recipient organizations. The survey contained questions on the types of activities undertaken with awarded funds; awareness and level of engagement of children, parents, and the broader community in the activities; and perceptions on the efficacy of the grant program in improving reading outcomes. QUALITATIVE DATA COLLECTION TOOLS NORC developed separate Key Informant Interview (KII) protocols for Tusome stakeholders including county-level officials, national government officials, USAID/KEA, and RTI. In addition, a Focus Group Discussion (FGD) protocol was developed for the Tusome National Technical Team (NTT). Qualitative instruments included questions, sub-questions, and probes related to government ownership and buy-in; procedures, policies, and guidelines established in support of Tusome; and long-term sustainability of the program. Copies of all data collection tools used at the endline are included in Annex VII. SAMPLING Primary respondents sampled for this evaluation included school-level respondents; MoE, TSC, and SAGA government officials at the county, regional, and national levels; personnel of USAID/KEA and Tusome implementing partners; and members of youth networks/bunges receiving grant support. Details on sampling/selection for each of these groups are presented below. QUANTITATIVE SAMPLING The sample size for school-based data collection was prescribed by USAID/KEA and included the same panel of 204 schools sampled for baseline and midline. MSI sampled schools at baseline using a multi￾stage clustered approach whereby 26 counties—covering all eight of the (former) provinces of Kenya— were randomly selected in the first stage and 204 schools were randomly selected in the second stage, proportionally by county. The 204 schools were drawn from a sampling frame of 22,154 formal public USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 25 schools and 1,000 non-formal APBET schools, yielding a final sample of 174 public schools and 30 APBET institutions.10 Within each sampled school, a cross-section of 12 grade 1 and 12 grade 2 pupils—equally split between boys and girls—was randomly selected, for a total pupil target sample of 4,896 per wave. In addition, each school’s Head Teacher and CSO or Instructional Coach (IC)11 was interviewed as well as the grade 1 and grade 2 teachers, who were also observed teaching a reading lesson. Finally, a contact list of Youth Fund grantees was provided by RTI, and all listed representatives were contacted by NORC to complete a web-based leadership survey. Table 5 reports the realized/actual sample against the target for all quantitative respondents. Where the sample was below target was due to schools having fewer than the anticipated sample of children, combined classrooms for grades 1 and 2, and in the case of CSOs and Youth Fund grantees, non￾response. The school sample was slightly above target because field teams determined upon arrival to a community that a school previously thought to be inactive was still functioning, yet its replacement had already been successfully enumerated.12 Table 5: Planned v. Actual Sample for Quantitative Data Collection Sampling Unit Realized Sample Target Sample % of Target Schools 205 204 100.5% Pupils (EGRAs, interviews) 4,850 4,896 99.1% Grade 1 2,418 2,448 98.8% Grade 2 2,432 2,448 99.3% Teachers (interviews) 391 408 95.8% Grade 1 194 204 95.1% Grade 2 197 204 96.6% Classrooms (observations) 393 408 96.3% Grade 1 195 204 95.6% Grade 2 198 204 97.1% 10 For a full description of baseline sampling procedures, see Freudenberger, E., & Davis, J. (2017). Tusome external evaluation— Midline report. Washington, DC: Management Sciences International, a Tetra Tech company. http://pdf. usaid.gov/pdf_docs/PA00MS6J.pdf. 11 Instructional Coaches (ICs) play the equivalent role of CSOs vis-à-vis Tusome implementation for APBET institutions. For the purpose of simplicity, the term CSO is intended to refer to both CSOs and ICs for the remainder of this report. 12 The 205th school was selected by MSI as a replacement school using the same rigorous random sampling procedures as other schools in the sample. It is important to note that schools in the final analysis are weighted according to their probability of selection, therefore the inclusion of this extra school meant that the weights of all other schools in the strata were adjusted downward since they all had a higher chance of being selected. The net effect is an increase in sample size which improves statistical precision without biasing the results. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 26 Sampling Unit Realized Sample Target Sample % of Target Head Teachers (interviews) 205 204 100.5% CSOs (interviews)13 146 157 93.0% Youth Fund grantees 17 23 73.9% QUALITATIVE SAMPLING Initial sampling for the KIIs was done in consultation with USAID/KEA and RTI and reviewed and finalized with key stakeholders at the Evaluation Design Validation Workshop. As part of the purposive selection process, the team first identified key stakeholder groups, including USAID/KEA, GoK, Kenyan SAGAs, TSC, NTT, and RTI. The team then differentiated between national GoK stakeholders and county-level GoK stakeholders and identified specific individuals within each stakeholder group to be interviewed. The FGDs were born out of a desire to interview all members of the NTT, while still being conscious of the need to collect data as efficiently as possible. In light of this, NORC and USAID/KEA agreed to interview the NTT together through a series of FGDs. For county-level qualitative data collection, NORC consulted with MoE and RTI to get anecdotal information on which counties had high and low levels of Tusome uptake.14 We combined the feedback from MoE and RTI and compiled the final suggested sample of counties, which was approved by USAID. County-level respondents included County Directors of Education (CDEs) and Teacher Service Commission County Directors (TSC-CDs) or their deputies. In total, NORC and USAID/KEA identified 49 KII respondents and ten FGD participants for inclusion in the evaluation. The FGD participants represented all NTT members minus those who were selected for individual KIIs. Table 6 details each key stakeholder group, the target number of interviews for each group, and the number of successful interviews completed. Table 6: Planned v. Actual Sample for Qualitative Data Collection Sampling Unit (Personnel) Realized Sample Target Sample % of Target USAID/Kenya East Africa 3 3 100.0% Government of Kenya – National 10 15 66.7% Tusome National Technical Team 4 10 40.0% Government of Kenya – County 17 20 85.0% RTI 9 11 81.8% 13 In cases where a CSO oversees more than one school in the sample, s/he was interviewed only once. 14 The evaluation team planned to use CSO data collected as part of the Tusome intervention, aggregated at the county level, to allow for purposive selection of counties based on variation in Tusome implementation fidelity. However, upon reviewing available data there was not enough variation to do a purposeful selection. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 27 The primary reason the number of actual interviews/respondents fell short of the target for GoK respondents was the timing of data collection. Qualitative data collection was launched in October 2019, shortly after enumerator training ended. When reaching out to GoK respondents to schedule interviews, the evaluation team was told respondents were in the field observing national examinations. While the evaluation team stayed in contact with the proposed respondents, we were not able to secure interviews prior to the December holidays at which point we had to conclude data collection to begin data cleaning and analysis. Although we did not reach the target for national GoK respondents, respondents from the MoE as well as major SAGAs, including KICD, TSC, the Kenya Education Management Institute (KEMI), and the Kenya National Examinations Council (KNEC) were all included. DATA COLLECTION For school-based data collection, NORC subcontracted with Dalberg Research (formerly Research Solutions Africa), an established Kenya-based data collection firm which conducted both baseline and midline data collection for the Tusome evaluation led by MSI. Primary qualitative data collection was conducted by NORC’s Qualitative Lead and Local Technical Expert. SUPERVISOR AND ENUMERATOR TRAINING Enumerator training occurred in two phases. First, NORC directly trained QCOs to serve as Master Trainers as well as accompany and observe each team for the duration of field work. Twenty-four QCOs were exclusively recruited among a pool of qualified researchers that served as QCOs during baseline and/or midline. This QCO “Training of Trainers” (ToT) took place in Nairobi from September 5-10 and was focused on re-orienting QCOs to the study, data collection procedures, sampling, logistics, and administration of the tools as well as preparing them to lead breakout sessions during the main enumerator training. Immediately following the ToT, a 6-day enumerator training was held in Nairobi from September 11-16. Ninety-five enumerators were selected to participate in the main training from a pool of over 10,000 applicants. The main enumerator training included a combination of plenary sessions (led by the NORC team) and breakout review and practice sessions (led by QCOs) to orient enumerators on field procedures and instruments. The final days of the training consisted of a pilot exercise and debrief at nearby schools to ensure enumerators had adequate practice prior to launch. Following the main training, 85 enumerators and supervisors were selected to participate in field work. Selection was made based on training attendance and participation, pilot performance, inter-rater reliability (IRR) testing, and written exams. Overall, EGRA IRR from the main training—measured as the raw rate of agreement with a “gold standard”—was 97.1 percent. The main training event was attended by EDY office members at USAID/KEA, members of the Tusome NTT, and key representatives from MoE and GoK. DATA MANAGEMENT Quantitative data collection was tablet-based, utilizing the Tangerine software for EGRA assessments and SurveyCTO/Open Data Kit (ODK) for all of other survey tools. Survey and assessment programming was conducted in-house by NORC and data collection platforms/servers were centrally managed by the evaluation team. All tablets and servers were encrypted to ensure maximum data USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 28 security. Data uploads were completed on a daily basis (connectivity permitting) to allow for real-time data quality reviews. Qualitative data was captured via detailed field notes recorded during the KIIs or FGDs. FGDs were conducted using two evaluators so that one could facilitate the discussion and the other could serve as notetaker. Field notes were typed up as soon as possible to capture details from the interview while still fresh. Notetakers typed complete notes in English, listening to audio recordings as needed to supplement or clarify field notes. The Qualitative Lead regularly reviewed these electronic notes for quality and completeness—including reviewing a subset of notes against full audio recordings—as well as to track progress against the field work schedule. Qualitative data was transferred and stored on a secured, password-protected server. DATA QUALITY ASSURANCE To ensure high quality data throughout the field period, NORC employed a number of quality assurance protocols and strategies including supervisor accompaniments (“sit-ins”), QCO co-enumeration for real￾time IRR monitoring, school re-visits (“back checks”), weekly field reporting and data reconciliation, and real-time data quality reviews. Over the course of data collection, NORC flagged 193 data quality review issues to Dalberg through a cloud-based issues log, all of which were quickly and satisfactorily addressed. In addition, all electronic data was fully reconciled with weekly field reports; all back-checked schools confirmed the assessments took place and random sampling procedures were correctly followed; and accompaniment data show strong adherence to survey administration protocols. With respect to inter-rater reliability, the overall EGRA IRR for the field work period was 93.1 percent. RESEARCH ETHICS AND STUDY AUTHORIZATION The evaluation was conducted in line with human subjects research guidelines in both the United States and Kenya. NORC received Institutional Review Board (IRB) exemption from its internal IRB, which follows a formal process for ensuring all research projects are conducted in accordance with the U.S. Federal Policy for the Protection of Human Subjects. NORC also obtained IRB approval from Strathmore University, a local IRB accredited by Kenya’s National Commission for Science, Technology and Innovation (NACOSTI). In accordance with section 17(1) of Kenya’s Science, Technology and Innovation Act of 2013, NORC applied to NACOSTI for a Research Permit which was issued on August 15, 2019. NORC also worked with USAID/KEA to obtain permission letters from MoE authorizing data collection teams to enter schools and complete planned activities. DATA ANALYSIS QUANTITATIVE DATA ANALYSIS Analysis of quantitative assessment and survey data includes summary statistics of performance indicators and other key outcomes of interest, disaggregated by sociodemographic characteristics such as gender and school type. EGRA performance is reported against MoE-established reading benchmarks, and change between baseline, midline, and endline is presented for each of the EGRA subtests. In USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 29 addition, multiple regression analysis was conducted to assess the extent to which pupil-, teacher-, classroom-, and school-level factors are associated with reading performance at endline. As in prior rounds of data collection, the evaluation team analyzed the psychometric properties of endline EGRA data using Pearson correlation and Cronbach’s Alpha to assess the consistency of responses across subtests as well as internal consistency within subtests (see Annex IV). For sampling weights, the calculations of the final weights was based on the inverse of the overall probabilities of selection. Consistent with baseline and midline, weight calculations took into consideration the stratification (counties, public/APBET) and the number of students per school. Primary quantitative analysis was conducted using the Stata SE/15.1 statistical software package. QUALITATIVE DATA ANALYSIS NORC implemented a multi-step approach for qualitative analysis using Dedoose, a qualitative data analysis program. First, we conducted a phenomenological analysis to identify key themes emerging from the KIIs and FGDs. Each key theme aligned with the outlined evaluation questions. The phenomenological approach helped to identify topics that emerged most frequently in the qualitative interviews, as well as topics that were important to respondents, but not necessarily included in the protocols. The key themes that emerged were then used to develop a code frame, which anchored the qualitative analysis, in which each code represented a pre-determined area of interest. Next, we used a grounded theory approach to triangulate data across sources, identify explanations for the phenomena observed, and explore the relationships between these phenomena. To ensure the quality of the analysis and reduce the risk of bias, multiple coders were involved in the formation of the code frame and the coding process. A team of three began with a draft code frame based on evaluation questions and the qualitative instruments. Next, the team reviewed a sample transcript from each beneficiary type. Upon review, the team added to the draft code frame, and selected a subset of transcripts to test the code frame and establish inter-rater reliability. The team then finalized the code frame, added code definitions, and used the final code frame to serve as the basis for organizing the data. Throughout analysis, the team identified additional coding gaps and made necessary additions and refinements to the codebook. LIMITATIONS LACK OF COUNTERFACTUAL Because Tusome was rolled out on a national scale, the evaluation was not able to causally attribute changes in reading performance to the project. Instead, the pre-post design examined progress toward Tusome’s stated objectives while relying upon earlier experimental research on USAID’s Primary Math and Reading (PRIMR) initiative to infer that observed improvements can be attributed to Tusome. TIMING OF BASELINE It is a general best practice (and USAID recommendation) to collect baseline data before the start of an intervention. To ensure comparability over time, midline and endline data should then be collected at the same time point as baseline data during subsequent school year(s). However, due to various issues USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 30 encountered by MSI, baseline data were collected in June and July 2015, shortly after Tusome started working with grade 1 teachers (but before it started working with grade 2 teachers). It was therefore decided that midline and endline data would be collected during September and October, somewhat after the same time point in the school year as baseline, thus providing slightly more time for learning during the school year (even taking into consideration the school break in August). Despite this, there remains a risk of either 1) overestimating changes in reading scores since pupils assessed in September/October have had more classroom exposure than those assessed in June/July or 2) underestimating changes in reading scores, since pupils assessed at baseline had already been exposed to Tusome methods/materials. Given the timing of the program rollout, the risk of (1) is higher for grade 1 while the risk of (2) is higher for grade 2. RESPONSE BIAS Response bias encompasses a range of tendencies among respondents to answer in a way that is not truthful. For this evaluation, the risk of response bias comes primarily from recall bias (inability to recall facts or past events) and social desirability bias (tendency to answer in a way that will be seen as favorable versus answering truthfully). While it is difficult to overcome this risk in social sciences research, the team worked to minimize it where possible through question framing, shortened recall periods, and preambles to sensitive questions reminding respondents of both the strict confidentiality of their responses and the importance of accuracy in research. Related to social desirability bias is the observer or Hawthorne effect, whereby respondents alter their ordinary behavior in response to the presence of an observer. The evaluation team believes that this bias could be pervasive with respect to classroom observations, with teachers changing the way they conduct reading lessons to impress the observer. As a consequence, classroom observation data may not fully capture the true day-to-day practices of teachers. To help mitigate this risk, during the consent proceedings enumerators reviewed a script with teachers that 1) affirmed we were not assessing the teacher’s performance but rather trying to understand what happens in Kenyan classrooms on a typical school day, 2) explained that the value of the research is maximized if they teach as they would in an ordinary day, and 3) assured teachers that the data captured will only be reported in combination with anonymized observations from hundreds of other classrooms. LIMITATIONS OF REGRESSION ANALYSIS The use of regression analysis to address evaluation question 3 is designed to overcome some of the inferential challenges associated with simple cross-tabulations or correlations. Despite this, the risk of omitted variable bias—i.e., the omission of an independent variable that is a determinant of the dependent variable and correlated with one or more independent variables in the model—remains. As such, readers should be cautious in assuming causal relationships between independent variables and pupil reading scores. Furthermore, lack of a statistical relationship between the tested variables and reading performance does not mean that such a relationship does not exist. Factors such as sample size, measurement accuracy/precision, and omitted variable bias (among others) may lead to a Type II error, or failure to detect a statistically meaningful relationship even if one exists. EXTERNAL VALIDITY Outcomes-level assessment results reflect the period of implementation under USAID support, which involved considerable financial and human resource investments. Consequently, some observed results USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 31 may not be sustained if/when USAID funding comes to an end, limiting the external validity of findings. In addition, results are not generalizable to private schools or mobile schools serving nomadic communities, as these schools were not targeted by the Tusome intervention and therefore were not included in the original sampling frame of public and APBET schools. Finally, while schools serving special needs learners were targeted by Tusome, they were not covered by this external independent EGRA assessment. DISCONTINUITY BETWEEN SURVEY ROUNDS Baseline and midline data collection were conducted by a separate team of evaluators at MSI, which could potentially undermine comparability between assessment rounds if procedures and protocols changed with the evaluation team. To minimize this risk, NORC consulted extensively with the original MSI evaluation team starting in 2018 to ensure comprehensive handover of project documents and transfer of all relevant background information/knowledge. In addition, NORC ensured continuity between baseline/midline and endline through the Local Technical Expert, QCOs, and subcontracted data collection firm, all of whom participated in prior rounds of data collection for Tusome including co￾facilitating enumerator trainings.15 The data collection subcontractor also was required to prioritize enumerators that had participated in prior rounds of Tusome. Finally, the proposed evaluation team brings decades of combined experience conducting education research throughout sub-Saharan Africa and is highly skilled in following standardized USAID norms and protocols in conducting EGRA-based studies, including those set forth in USAID’s EGRA 2.0 Toolkit. REPRESENTATIVENESS OF SAMPLE Respondents for the qualitative component of the evaluation were not randomly selected and may not be representative of the full range of experiences among Tusome stakeholders. The small sample of qualitative respondents may not reflect the perspective of the groups they were selected to represent, and systematic sampling error or bias might be present. RISK OF DIVERGENT FINDINGS Different individual respondents and groups of respondents may have different views on program effectiveness and sustainability. While responses are triangulated, determining whose information is correct in case of conflict is challenging. In addition, qualitative and quantitative findings may diverge in some cases. As appropriate, evaluators acknowledge such diverse perspectives in this report. COVID-19 RELATED DELAYS The global COVID-19 pandemic led to several delays in validating and finalizing this evaluation report. Consequently, some stakeholders may express concerns that the 2019 data presented herein are dated at the time of publication. It is important to note, however, that face-to-face schooling was deferred from March 2020 to January 2021 due to COVID-19. As such, grade 1 and 2 pupils and teachers spent just three months in schools between the time of data collection and the start of the current school year, meaning findings still reflect relatively recent school conditions. 15 The evaluation team also utilized the Tusome Papaya app available on Google Play to demonstrate proper letter sound pronunciation in English and Kiswahili during training. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 32 4. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS FINDINGS EVALUATION QUESTION 1 What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? KEY FINDINGS For English, The grade 1 reading fluency benchmark is 35 or more correct words per minute (CWPM) and the intermediate skill range for emergent readers is 20 to 34 CWPM. The grade 2 English reading fluency benchmark is 65 or more CWPM and the intermediate skill range for emergent readers is 30 to 64 CWPM. • At endline, 14 percent of grade 1 pupils were emergent readers and 13.5 percent were fluent readers in English. • At endline, 35 percent of grade 2 pupils were emergent readers and 18 percent were fluent readers in English. For Kiswahili, the grade 1 reading fluency benchmark is 30 or more CWPM and the intermediate skill range for emergent readers is ten to 29 CWPM. The grade 2 Kiswahili reading fluency benchmark is 45 or more CWPM and the intermediate skill range for emergent readers is 17 to 44 CWPM. • At endline, 22 percent of grade 1 pupils were emergent readers and four percent were fluent readers in Kiswahili. • At endline, 46 percent of grade 2 pupils were emergent readers and eight percent were fluent readers in Kiswahili. The metrics used to address evaluation question 1 are the oral reading fluency (ORF) performance categories established by GoK and assessed using the Early Grade Reading Assessment (EGRA). The MoE and Kenya National Examinations Council (KNEC) are responsible for setting separate ORF performance benchmarks for English and Kiswahili. These benchmarks are used to categorize pupils based on their ORF score to determine the extent to which pupils are reading at grade-level, in line with Kenya’s curricular goals. Benchmarks for grade 2 pupils were first set by the MoE and KNEC in 2012, and remain unchanged since their initial approval. Distinct benchmarks for grade 1 pupils were established and approved in 2019. Results presented in this report reflect the current benchmarks for both grades. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 33 Table 7 shows the current English and Kiswahili ORF benchmarks for grades 1 and 2.16 The benchmarks assign pupils to one of four basic reading categories – zero, beginning, emergent, and fluent readers – based on the number of correct words per minute (CWPM) scored on the oral passage reading subtasks in each language. For grade 1 pupils, the English fluency benchmark is 35 CWPM and the Kiswahili fluency benchmark is 30 CWPM. For grade 2 pupils, the English fluency benchmark is 65 CWPM and the Kiswahili benchmark is 45 CWPM. Table 7: Endline ORF Performance Categories for English and Kiswahili, by Grade English CWPM Kiswahili CWPM ORF Category Grade 1 Grade 2 Grade 1 Grade 2 Fluent reader 35+ 65+ 30+ 45+ Emergent reader 20-34 30-64 10-29 17-44 Beginner reader 1-19 1-29 1-9 1-16 Zero reader 0 0 0 0 English Benchmark Results For English ORF, pupils are assigned to performance categories based on their CWPM scores for the English Passage Reading (A) subtask. Table 8 shows the distribution across English ORF performance categories by grade for each wave of data collection, as well as the percentage point change between baseline and endline. As shown in Table 8, the number of non-readers (or zero readers) dropped considerably between baseline and endline, while the number of fluent and emergent readers increased for both grade levels. The proportion of grade 1 pupils that did not read a single word correctly decreased by 18.4 percentage points, from over half of pupils at baseline to just over a third at endline. The proportion of grade 2 non￾readers decreased by around 24 percentage points, from 38 percent at baseline to 14 percent at endline. However, the large drop in non-readers (or zero readers) observed between baseline and midline was partially reversed by endline for both grades, as was the increase in fluent and emergent readers. Grade 2 pupils showed greater improvements from baseline to endline, having retained the gains observed at midline more so than their grade 1 counterparts. 16 Because distinct grade 1 benchmarks did not exist prior to 2019, MSI applied grade 2 benchmarks to grade 1 pupils at baseline and midline. To reflect the most up-to-date ORF categories and to allow for comparison across rounds, NORC re￾generated the baseline and midline ORF categories using the 2019 benchmarks. For this reason, grade 1 ORF benchmark figures in this report will not match those from the baseline or midline reports. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 34 Table 8: English Oral Reading Fluency Performance Categories ORF Category Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff.*** Baseline Midline Endline BL-EL Diff.*** Fluent Reader 10.0% 25.7% 13.5% 3.5% 11.6% 26.5% 18.0% 6.4% Emergent Reader 11.2% 18.0% 14.1% 2.9% 22.2% 38.3% 35.3% 13.1% Beginning Reader 26.2% 33.4% 38.1% 12.0% 28.3% 23.3% 32.4% 4.2% Zero Reader 52.7% 22.9% 34.3% -18.4% 37.9% 11.8% 14.2% -23.7% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic. Any discrepancies between reported figures and BL-EL differences are due to rounding. Figure 6 shows the same English ORF category breakdowns in the form of a bar chart. This chart helps to visualize the decrease in the proportion of non-readers and increase in other performance categories from baseline to endline, as well as the differences in performance by grade over time. In particular, these charts help to highlight that grade 2 pupils retained more of their ORF improvements from midline than did grade 1 pupils, especially in the non-reader category. Figure 6: English Reading Performance Categories, By Round In addition to comparing trends in ORF categories across rounds, we also disaggregate endline results by school type and gender to analyze potential differences within these groups. Consistent with midline 53% 23% 34% 38% 12% 14% 26% 33% 38% 28% 23% 32% 11% 18% 14% 22% 38% 35% 10% 26% 14% 12% 27% 18% Baseline Midline Grade 1 Endline Baseline Midline Grade 2 Endline Zero Reader Beginning Reader Emergent Reader Fluent Reader USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 35 findings, pupils in APBET institutions scored far higher than pupils from public schools, and girls scored slightly higher than boys. As shown in Table 9, pupils in APBET institutions displayed higher English ORF scores across every category compared to those in public schools, with a higher proportion of fluent and emergent readers and a lower proportion of zero and beginner readers in both grades. While over a third of grade 1 pupils and 14 percent of grade 2 pupils in public schools were not able to read any English words correctly at endline, only around three percent of APBET pupils were non-readers at endline. The majority of APBET pupils were either fluent or emergent readers (78 percent of grade 1 and 86 percent of grade 2) compared to a much smaller proportion from public schools (27 percent of grade 1 and 53 percent of grade 2). Table 9: English Endline ORF Scores by Performance Category, Grade, and School Type Class School Type Zero Beginner Emergent Fluent Grade 1*** Public 34.7% 38.4% 13.9% 12.9% APBET 3.7% 18.1% 25.3% 52.9% Grade 2*** Public 14.4% 32.7% 35.4% 17.6% APBET 2.5% 11.3% 30.9% 55.3% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic. Any discrepancies between reported figures and BL-EL differences are due to rounding. Table 10 displays the same English ORF categories disaggregated by gender. Overall, girls displayed slightly higher ORF scores than boys at endline, with a lower share of zero and beginner readers and a higher share of fluent and emergent readers. The proportion of girls who were non-readers was six percentage points lower than the proportion of boys in both grades, while the proportion of fluent girls was around five percentage points greater than the proportion of boys. Table 10: English Endline ORF Scores by Performance Category, Grade, and Gender Class Gender Zero Beginner Emergent Fluent Grade 1** Male 37.1% 38.3% 13.7% 10.9% Female 31.3% 38.0% 14.5% 16.2% Grade 2*** Male 17.1% 36.4% 30.8% 15.8% Female 11.3% 28.4% 40.0% 20.4% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 36 Kiswahili Benchmark Results For Kiswahili ORF, pupils are assigned to performance categories based on their CWPM scores for the Kiswahili Oral Passage Reading subtask. Table 11 shows the distribution across these categories by grade for each wave of data collection, as well as the percentage point change between baseline and endline. In general, Kiswahili reading benchmark performance improved from baseline to endline, but to a lesser degree than English benchmark performance. Notably, the Kiswahili gains observed at midline dropped by endline, particularly for grade 1 pupils. From baseline to endline, the proportion of non-readers decreased for pupils in both grade levels, though the decrease is more pronounced among grade 2 pupils. The proportion of grade 1 pupils that scored zero CWPM decreased 7.4 percentage from baseline to endline, with the difference only marginally significant at the 0.10 level. The proportion of non-reader grade 2 pupils decreased by 14 percentage points, from 43 percent at baseline to 29 percent at endline. In both grade levels, the large performance gains observed at midline had been partially reversed by endline. As with English ORF, the proportion of fluent and emergent Kiswahili readers increased from baseline to endline, though grade 2 pupils showed greater improvements. Table 11: Kiswahili Oral Reading Fluency Performance Categories ORF Category Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff.* Baseline Midline Endline BL-EL Diff.*** Fluent Reader 4.2% 14.7% 4.4% 0.2% 4.3% 11.8% 7.6% 3.2% Emergent Reader 16.0% 30.1% 21.8% 5.7% 33.3% 54.4% 46.1% 12.7% Beginning Reader 9.9% 10.4% 11.4% 1.5% 19.0% 14.9% 17.4% -1.7% Zero Reader 69.9% 44.7% 62.4% -7.4% 43.3% 18.9% 29.0% -14.3% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic. Any discrepancies between reported figures and BL-EL differences are due to rounding. Figure 7 shows the same Kiswahili ORF category breakdowns in the form of a bar graphs. Through a visual inspection of these graphs, it is apparent that endline results look more similar to baseline than midline, again highlighting that improvements observed at midline were not fully sustained. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 37 Figure 7: Kiswahili Reading Performance Categories, by Round As with English ORF, we disaggregate endline Kiswahili ORF results by school type and gender. Generally, APBET institutions scored much higher than public schools and girls scored slightly higher than boys, which mirror earlier findings for English ORF as well as the midline results. As shown in Table 12, pupils in APBET institutions displayed higher average Kiswahili ORF scores compared to those in public schools, with a significantly higher proportion of fluent and emergent readers and a significantly lower proportion of non-readers. In both grades, the proportion of non￾readers from public schools is more than three times greater than the proportion from APBET schools, while the proportion of fluent readers is around three times lower.17 Table 12: Kiswahili Endline ORF Scores by Performance Category, Grade, and School Type Class School Type Zero Beginner Emergent Fluent Grade 1*** Public 63.0% 11.3% 21.4% 4.3% APBET 20.4% 16.6% 49.1% 13.9% Grade 2*** Public 29.3% 17.4% 45.9% 7.4% APBET 7.9% 11.9% 58.1% 22.1% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic 17 This finding is consistent with a growing body of research that suggests that informal and/or contract teachers may be higher performing than traditional civil service teachers due to factors such as motivation, reduced absenteeism, willingness to try new tasks and methods, etc. (see e.g., Simmons Zuilkowski, S., Piper, B., & Ong'ele, S. A. (2020). Are low-cost-private schools worth the investment? Evidence on literacy and mathematics gains in Nairobi primary schools. Teachers College Record, 122(1) or Hanushek, E. A., Machin, S. J., & Woessmann, L. (Eds.). (2016). Handbook of the Economics of Education. Elsevier). 70% 45% 62% 43% 19% 29% 10% 10% 11% 19% 15% 17% 16% 30% 22% 33% 54% 46% 4% 15% 4% 4% 12% 8% Baseline Midline Grade 1 Endline Baseline Midline Grade 2 Endline Zero Reader Beginning Reader Emergent Reader Fluent Reader USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 38 Table 13 disaggregates Kiswahili ORF categories by gender. As with English ORF, girls generally displayed higher ORF scores, with a lower share of non-readers and a greater share of fluent and emergent readers. The proportion of girls who were non-readers was around ten percentage points lower than the proportion of boys who were non-readers in both grades, while the proportion of fluent girls was 2-3 percentage points greater than the proportion of fluent boys. Table 13: Kiswahili Endline ORF Scores by Performance Category, Grade, and Gender Class Gender Zero Beginner Emergent Fluent Grade 1*** Male 67.5% 9.5% 19.6% 3.4% Female 57.3% 13.3% 24.0% 5.5% Grade 2*** Male 33.5% 18.8% 41.5% 6.2% Female 24.4% 15.9% 50.8% 8.9% *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10 for design-based F-statistic USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 39 EVALUATION QUESTION 2 What are the skill levels of grade 1 and 2 pupils on reading subtasks? KEY FINDINGS • Pupils have shown improvement from baseline to endline on all EGRA subtasks in both English and Kiswahili. • Relative to midline, endline scores have declined for most subtasks and in a few cases reverted to baseline levels. Notably, however, English reading comprehension gains observed at midline were sustained by Grade 2 pupils at endline. • In English, grade 1 pupils displayed the highest percent correct scores on the vocabulary and phoneme segmentation subtasks, followed by letter sound knowledge. Grade 2 pupils displayed the highest percent correct scores on the passage reading subtasks, followed by vocabulary and phoneme segmentation. • In Kiswahili, both grades received the highest percent correct scores on the listening comprehension subtask, which is the only subtask that shows improvements since midline. Relative to the other Kiswahili subtasks, pupils also displayed higher percent correct scores on letter sound knowledge. • In both languages, scores for reading comprehension – considered to be the most difficult subtask – improved, but remained among the lowest performing subtasks (when scored as percent correct). • Reading comprehension gains between baseline and endline were greater when the subtask included a silent reading component, which may suggest a positive relationship between the practice of independent or silent reading and comprehension. The EGRA used for the Tusome evaluation consists of 14 subtasks, including eight English and six Kiswahili subtasks. These subtasks are designed to assess pupils’ skills along a reading continuum, from pre-reading skills—such as listening comprehension and initial sound identification—to reading fluency and comprehension. The primary indicators of interest for evaluation question 2 are raw EGRA subtask scores. To facilitate comparisons across time and subtests, NORC analyzed changes in both the values of the raw scores as well as the percent correct (of total items) for each subtask. Results by grade are explored separately for English and Kiswahili. For English, a separate discussion of the two reading comprehension subtasks is also provided. The calculation of raw scores varies for untimed and timed subtasks. Raw scores for untimed subtasks are presented as the number of items the pupil got correct for that subtask. Scores for timed subtasks are expressed as correct words or items per minute, which adjust scores for those who attempt all of the items before the 60 seconds have lapsed. We also explore the magnitude and statistical significance of the change in each subtask score from baseline to endline. In addition, we visualize scores expressed as the percent correct (i.e., number of correct items over the number of total items) to allow for easy comparison of relative performance across subtasks and over time. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 40 Full results disaggregated by school type (public and APBET) and pupil gender (male and female) are included in Annex I. English EGRA Results As shown in Table 14, pupils exhibited improvements in every subtask from the beginning to the end of the program. The change in English raw reading scores from baseline to endline is universally positive and statistically significant for all subtasks. Average scores in phoneme segmentation and reading comprehension B subtasks saw notable relative increases, with scores more than doubling in both grades from baseline to endline. While there were overall gains since baseline, it is important to note performance peaked at midline and subsequently dropped at endline in most cases.18 Between baseline and endline, grade 2 scores increased more than grade 1 scores—in both nominal and relative terms—for all subtasks except vocabulary. In addition, grade 2 scores remained closer to midline levels than grade 1 scores, though gains observed at midline diminished across grades and subtasks. The one English subtask that showed improvement since midline was reading comprehension B for grade 2, though the change is not statistically significant. Table 14: English EGRA Raw Subtask Scores Subtask Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 1.1 3.8 2.8 1.6 *** 0.6 5.0 4.1 3.6 *** Letter Sound Knowledge 15.1 26.1 21.4 6.3 *** 10.2 32.6 26.3 16.1 *** Invented Word Decoding 5.7 10.4 7.7 2.0 ** 10.4 18.4 15.5 5.1 *** English Vocabulary 5.9 7.8 7.1 1.3 *** 8.2 10.2 9.9 1.6 *** Passage Reading (A) 10.6 22.3 13.9 3.3 ** 23.8 43.6 36.2 12.4 *** Reading Comp. (A) 0.2 0.5 0.3 0.1 * 0.5 1.0 1.0 0.5 *** Passage Reading (B) 9.6 22.0 13.4 3.8 ** 21.8 44.2 35.8 14.0 *** Reading Comp. (B) 0.2 0.8 0.5 0.3 *** 0.6 1.7 1.8 1.2 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. 18 See Annex I for tables depicting English Raw Scores including Midline-Endline differences and significance levels. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 41 In order to assess progress on a common scale, we analyze average percent correct for each subtask by grade. As shown in Figure 8,19 the average percent correct scores for grade 1 pupils increased across all subtasks from baseline to endline. Grade 1 pupils scored highest on vocabulary (36 percent correct) and phoneme segmentation (28 percent correct) at endline, though phoneme segmentation averages dropped significantly since midline. Consistent with other rounds, the two subtasks with the lowest average percent correct scores are the reading comprehension subtasks, though the reading comprehension B saw the greatest relative increase, tripling from three percent at baseline to nine percent at endline. Despite large declines since midline, phoneme segmentation average correct scores saw the next highest relative increase, more than doubling from 11 percent at baseline to 28 percent at endline. While oral passage reading scores improved between baseline and endline, these subtasks displayed the largest relative declines since midline with average percent correct score gains dropping by more than half. Figure 8: English Grade 1 Percent Correct EGRA Scores As shown in Figure 9, percent correct scores for grade 2 pupils similarly increased across the board for every subtask, though grade 2 pupils appear to have retained their midline gains more than grade 1 pupils on average. Of particular note, the reading passage A and reading comprehension A subtasks barely changed from midline levels for grade 2, signifying that baseline improvements were well-retained 19 In each chart, subtasks are sorted by endline performance such that the subtasks with the highest percent correct at endline appear first. 29% 36% 39% 11% 28% 38% 15% 21% 26% 15% 20% 30% 13% 19% 30% 11% 15% 20% 3% 9% 13% 4% 5% 8% Phoneme Segmentation Letter Sound Knowledge Oral Passage Reading (A) Vocabulary Oral Passage Reading (B) Invented Word Decoding Reading Comp. (B) Reading Comp. (A) Baseline Endline Midline USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 42 for these subtasks relative to other English subtasks. Further discussion of the reading comprehension subtasks is presented later in this section. At endline, grade 2 pupils performed best on the oral passage reading A and B subtasks, scoring an average of 49 and 47 percent correct, respectively. Interestingly, while the passage reading B subtask averages dropped nearly ten percentage points from midline, passage reading A averages remained relatively steady. Grade 2 pupils also performed relatively well on vocabulary (46 percent correct) and phoneme segmentation (41 percent correct) at endline. From baseline to endline, grade 2 pupils showed considerable improvement on the subtasks that had the lowest average percent correct scores at baseline. Phoneme segmentation exhibited the most sizable improvement, with the average percent correct score for that subtask increasing nearly sevenfold from six percent at baseline to 41 percent at endline. The reading comprehension subtasks likewise improved, with the reading comprehension A percent correct scores doubling and the reading comprehension B more than tripling from baseline to endline. Letter sound knowledge also increased substantially from ten percent average at baseline to 26 percent at endline. Figure 9: English Grade 2 Percent Correct Reading Scores In order to explain the differences in performance and improvement in the reading comprehension subtasks, it is important to discuss the differences between these subtasks and the rationale behind their design. 32% 47% 56% 29% 46% 56% 41% 49% 51% 6% 41% 50% 21% 31% 36% 10% 26% 33% 9% 29% 29% 8% 16% 17% Baseline Endline Midline Oral Passage Reading (A) Oral Passage Reading (B) Vocabulary Phoneme Segment. Invented Word Decoding Letter Sound Knowledge Reading Comp. (B) Reading Comp. (A) USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 43 Unlike the Kiswahili EGRA, the English EGRA includes two distinct reading comprehension subtasks that measure slightly different skillsets. As shown in Figure 10, reading comprehension A is the standard EGRA subtask, during which the pupil reads a passage aloud then answers questions from memory about the passage, without being able to reference the text. Figure 10: Reading Comprehension A In contrast, reading comprehension B is a custom subtask first developed by MSI and RTI at baseline at the request of MoE. As illustrated in Figure 11, this subtask differs from the standard comprehension subtask in two important ways. First, pupils are provided with an additional minute to silently re-read the passage they read aloud during passage reading B. Second, pupils are allowed to refer to the passage when they answer the comprehension questions. With this in mind, it is possible to better understand the differences in reading comprehension scores discussed previously. Figure 11: Reading Comprehension B Despite starting at similar levels at baseline for comprehension A and B, pupils showed a greater improvement in reading comprehension B than reading comprehension A. While reading comprehension A raw scores did increase—from 0.2 (out of 6) at baseline to 0.3 at endline for grade 1 and 0.5 to 1.0 for grade 2—reading comprehension B gains were more than double these amounts. These findings suggest a positive relationship between the Tusome model’s emphasis on independent or silent reading practice and pupils’ ability to better comprehend printed text. Kiswahili EGRA Results Kiswahili EGRA raw scores are presented in Table 15. When analyzing changes in Kiswahili EGRA scores across rounds, it is evident that pupils exhibited improvements over the course of the evaluation. As with the English EGRA, however, performance on most subtasks peaked at midline and subsequently declined. The exception is listening comprehension, which actually saw improvements from midline to endline in both grades. Generally, scores for grade 2 pupils increased more from baseline to endline in percentage terms than did scores for grade 1 pupils. It should be noted that the statistical significance of observed differences between baseline and endline varies across grades and subtasks. For grade 1, changes in Kiswahili invented word decoding and reading comprehension are not statistically significant while changes in syllable fluency and passage reading are only marginally significant. For grade 2, all differences are significant at the 0.01 level except for invented word decoding, which is significant at the 0.05 level. Pupil has 60 seconds to read the passage out loud Enumerator removes the passage Enumerator asks six comprehension questions Pupil has 60 seconds to read the passage out loud Pupil has 60 additional seconds to read the passage silently Pupil keeps the passage in front of him/her Enumerator asks six comprehension questions USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 44 Table 15: Kiswahili EGRA Raw Subtask Scores Subtask Grade 1 Grade 2 Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 16.5 29.2 22.8 6.3 *** 16.2 39.5 33.8 17.5 *** Syllable Fluency 11.0 21.3 13.7 2.6 * 20.9 36.5 29.7 8.8 *** Invented Word Decoding 4.7 8.1 5.0 0.3 - 10.3 16.1 12.5 2.2 ** Passage Reading 4.9 12.2 6.2 1.3 * 13.5 24.5 19.1 5.6 *** Reading Comprehension 0.4 0.9 0.4 0.1 - 1.1 2.0 1.5 0.4 *** Listening Comprehension 1.2 2.0 2.1 0.9 *** 1.9 2.8 3.1 1.2 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. As with the English subtasks, we present the Kiswahili subtask scores in terms of percentage of correct items, with subtasks sorted by endline performance. Kiswahili percent correct scores for grade 1 and 2 are presented in Figures 12 and 13. Across each round of data collection, pupils performed best on the listening comprehension subtask, followed by letter sound knowledge and syllable fluency. Despite already high percent scores relative to other subtasks at baseline, the “pre-reading” skill of listening comprehension saw the greatest improvement across rounds in both grades. For grade 2, letter sound knowledge also exhibited a relatively large improvement from baseline to endline. For grade 1, the large gains seen in several subtasks from baseline to midline have been significantly reduced by endline, with only minor increases observed in reading comprehension, passage reading, invented word decoding, and syllable fluency from baseline to endline. For grade 2, the gains from midline were retained to a larger extent than grade 1 yet still declined across most subtasks.20 20 See Annex II for tables depicting Kiswahili Raw Scores including Midline-Endline differences and significance levels. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 45 Figure 12: Kiswahili Grade 1 Percent Reading Scores Figure 13: Kiswahili Grade 2 Percent Reading Scores 25% 40% 42% 16% 23% 29% 11% 14% 21% 9% 10% 16% 7% 9% 18% 6% 7% 15% Letter Sound Knowledge Syllable Fluency Invented Word Decoding Listening Comprehension Oral Passage Reading Reading Comp. Baseline Midline Endline 38% 55% 62% 16% 34% 39% 21% 30% 37% 20% 28% 36% 18% 25% 34% 21% 25% 32% Letter Sound Knowledge Syllable Fluency Oral Passage Reading Listening Comprehension Reading Comprehension Invented Word Decoding Baseline Midline Endline USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 46 EVALUATION QUESTION 3 What school-level and institutional factors influence reading outcomes when implementing at scale, and how? KEY FINDINGS • Local language use during English instruction is strongly negatively associated with both English and Kiswahili ORF, with pupils from such classrooms scoring 7-12 CWPM lower on average. This is consistent open-ended feedback/suggestions from 6 percent of interviewed teachers who say reducing or eliminating the use of other languages at school is the best way to improve teaching and learning of English and Kiswahili. • Pupils who report reading stories at home and/or reading aloud at school score 3-6 CWPM higher than those who don’t, holding all else constant. • Statistical models at endline show that higher implementation of Tusome in the classroom positively correlates with reading fluency. To answer evaluation question 3, we use multiple regression analysis to assess factors that are associated with reading performance at endline. Two separate Ordinary Least Squares (OLS) regression models were estimated using English and Kiswahili ORF as dependent variables. A wide range of student-, classroom-, and school-level independent variables that were theorized by the evaluation team to be potential predictors of reading performance were included in the initial models. Each model was then built up iteratively through a series of individual, step-wise regressions in order to identify variables which significantly predict reading performance. Variables that were not predictive of reading performance at any stage of the process (i.e. not statistically significant at the five percent level) were excluded from subsequent analyses. The independent variables included in the final regression models are summarized in Table 16. The majority of variables were predictive of both English and Kiswahili reading performance, however—as denoted by asterisks—some variables are included in the model for one language only because they were found to be insignificant for the other language. Index variables for pupil household assets and school infrastructure/facilities were also created through separate principal component analyses (PCA). PCA is a variable reduction technique that decomposes a set of correlated variables into another set of linearly unrelated components which are expected to represent an underlying or latent concept that the variables have in common. Despite the inclusion of these indices as control variables in the final models, it is important to acknowledge that other confounders may remain so causal relationships between predictors and ORF should not be assumed. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 47 Table 16: Independent Variables Used in Final Regression Models Pupil-level Classroom-level School-level • Grade • Age* • Gender • Language used at school • Regularly wears shoes to school21 • Reads stories at home • Practices reading aloud at school • Pupil household assets (index) • Class size • Frequency pupils sound out unfamiliar words* • Tusome lesson unit number/progress in Tusome teacher’s guide* • Classroom has tables and chairs/benches for children* • Classroom has timetable posted on the wall* • Use of local language during lessons • Use of English and Kiswahili during lessons** • Untrained teacher** • School infrastructure/facilities (index) *=English only; **= Kiswahili only Predictors of English ORF Regression results for the English ORF model are presented in Table 17 below. Coefficient magnitudes and p-values are presented for each of the 14 predictor variables and for the PCA indices. It should be noted that interpretation of coefficients varies depending on the type of variable: for dichotomous variables such as “female,” the coefficient represents the change in ORF associated with that characteristic, holding all else constant. For continuous variables such as class size, the coefficient represents the change in ORF associated with a one-unit increase in that variable, holding all else constant. While the full results are presented in While pupil household assets index was statistically correlated with reading performance, the school resources index was not. Table 17, some policy-relevant observations include: • Local language use during English instruction is strongly negatively associated with English ORF, with pupils from such classrooms scoring 12 CWPM lower on average. This is consistent with open-ended feedback/suggestions from 6 percent of interviewed teachers who say reducing or eliminating the use of other languages at school is the best way to improve teaching and learning of English and Kiswahili. • Practicing reading aloud at school has a strong, positive association with English ORF performance. Holding all else constant, pupils who report reading aloud in school scored six CWPM higher than pupils who did not. • Similarly, reading at home is associated with a 5.5 CWPM increase in English ORF, which is consistent with findings from similar studies. 21 Whether or not a pupil wears shoes to school was not included in the pupil household asset index because none of the variation was explained by the principal components meeting the eigenvalue threshold. As such, it was included as a standalone variable in the final regression models. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 48 • Tusome lesson plan progress—measured as the lesson number at the time of the classroom observation—is positively associated with English ORF: each 10-unit advance in the Tusome teacher’s guide is associated with an increase of around one CWPM in English reading fluency. • Teacher-reported frequency of pupils sounding out unfamiliar words is positively associated with English ORF. For every day in a given week that teachers practice this skill, ORF increases by one CWPM. • Class size is negatively associated with reading performance: for every ten additional children in a classroom, English ORF drops by one CWPM. • While pupil household assets index was statistically correlated with reading performance, the school resources index was not. Table 17: OLS Regression Results for English ORF Predictor Coefficient (CWPM) P-Value Pupil is in grade 2 21.83 0.000 Pupil age -1.23 0.019 Pupil is female 4.87 0.000 Pupil reports speaking Kiswahili at school 4.99 0.002 Pupil reports speaking English at school 9.12 0.000 Pupil reports usually wearing shoes to school 4.51 0.024 Teachers frequently use local language while teaching English -12.40 0.000 Pupil reports reading stories at home 5.51 0.000 Pupil reports practicing reading aloud at school 6.24 0.000 Number of children in pupil’s classroom -0.13 0.005 Number of days/week pupils are required to sound out unfamiliar words 1.18 0.021 Lesson progress (unit number) in Tusome teacher's guide 0.09 0.009 Classroom has tables/benches for children 7.71 0.001 Classroom has timetable posted on the wall 3.20 0.036 Index for pupil household assets (component 1) 1.24 0.005 Index for pupil household assets (component 2) -1.43 0.029 Index for pupil household assets (component 3) -0.28 0.650 Index for school resources (component 1) 0.29 0.632 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 49 Predictor Coefficient (CWPM) P-Value Index for school resources (component 2) -0.96 0.389 Index for school resources (component 3) 0.43 0.730 Constant -36.1856 0.000 Observations 3,785 R-Squared 0.2987 * p < 0.10, ** p < 0.05, *** p < 0.01 Predictors of Kiswahili ORF Regression results for the Kiswahili ORF model are presented in Table 18. Coefficient magnitudes and p-values are presented for 11 predictor variables and for the PCA indices. In contrast to the English model, the relationship between ORF and sounding out unfamiliar words and Tusome lesson progress is not significant. However, results on local language use and class size are similar to those presented in the English model, albeit the magnitude of the ORF coefficients are lower for Kiswahili. Some further policy-relevant observations include: • As with English, reading stories at home and reading aloud at school are both positively associated with Kiswahili ORF, with coefficients of 3.49 and 4.21 respectively. • When teachers use both English and Kiswahili while teaching Kiswahili—i.e., use “code switching"—pupils score 3.19 CWPM higher in Kiswahili. Table 18: OLS Regression Results for Kiswahili ORF Predictor Coefficient (CWPM) P-Value Pupil is in grade 2 12.42 0.000 Pupil is female 2.82 0.000 Pupil reports speaking Kiswahili at school 2.15 0.004 Pupil reports speaking English at school 4.63 0.000 Pupil reports usually wearing shoes to school 2.63 0.014 Pupil reports reading stories at home 3.49 0.000 Pupil reports practicing reading aloud at school 4.21 0.000 Teachers frequently use local language while teaching English -6.83 0.000 Teachers use both English and Kiswahili when teaching Kiswahili 3.19 0.001 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 50 Predictor Coefficient (CWPM) P-Value Teacher is untrained (no professional qualification) 11.44 0.000 Number of children in pupil’s classroom -0.08 0.003 Index for pupil household assets (component 1) 0.25 0.256 Index for pupil household assets (component 2) -0.56 0.134 Index for pupil household assets (component 3) -0.11 0.748 Index for school resources (component 1) 0.15 0.640 Index for school resources (component 2) -0.16 0.795 Index for school resources (component 3) 0.06 0.921 Constant -16.428*** 0.000 Observations 3,785 R-Squared 0.2987 * p < 0.10, ** p < 0.05, *** p < 0.01 These regressions revealed relationships between a number of other pupil-level factors and ORF that are commonly seen in the literature. For example, over-aging is negatively associated with ORF while higher grade level, being female, wearing shoes to school, and speaking the language of instruction at school are positively associated with ORF. Interestingly, a teacher being “untrained” was strongly positively associated with Kiswahili ORF—even when controlling for higher-performing APBET school status. While seemingly counter-intuitive, some research suggests that informal and/or contract teachers may be higher performing than traditional civil service teachers due to factors such as motivation, reduced absenteeism, willingness to try new tasks and methods, etc.22 Other factors positively associated with ORF include the presence of tables/benches for children and the timetable being posted to the classroom wall, possibly driven by latent variables such as school leadership and school resources that have not been accounted for in the model. 22 See e.g., Simmons Zuilkowski, S., Piper, B., & Ong'ele, S. A. (2020). Are low-cost-private schools worth the investment? Evidence on literacy and mathematics gains in Nairobi primary schools. Teachers College Record, 122(1) or Hanushek, E. A., Machin, S. J., & Woessmann, L. (Eds.). (2016). Handbook of the Economics of Education. Elsevier. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 51 EVALUATION QUESTION 4 4.a How effective was implementation of Tusome with regard to teacher buy-in and implementation? KEY FINDINGS • Teachers generally demonstrate moderate to high levels of support for the Tusome trainings and teaching methodologies. • While teachers seem comfortable using Tusome materials in the classroom, they have faced significant challenges after the midline in covering all of the approved content following the 40 percent reduction in instructional hours allocated to English and Kiswahili under the new Competency-Based Curriculum (CBC) timetable implemented in 2019—on average, they were around 30 (of 150 total) lessons behind midline levels. • Overall, teachers find coaching and supervision from CSOs/ICs and Head Teachers helpful; however teachers report that the frequency of such visits has declined since midline, with a 20 percentage point drop in the number of teachers who are visited by a CSO at least once per term. Teacher buy-in and implementation at endline is examined against Tusome’s Intermediate Result (IR) 1: improved supervision, support, and delivery of reading instruction to target pupils. Specifically, we draw upon multiple data sources to explore three broad categories of program activities including improved reading instruction methods and delivery, use of teaching and learning materials (TLM), and teacher supervision and support from tutors/coaches. Given the drop in pupil reading performance observed since midline, we also make some comparisons between midline and endline vis-à-vis fidelity of implementation in order to explore possible mechanisms for these declines. Improved Reading Instruction Methods and Delivery Teachers were asked to rate the frequency, relevance, quality, and usefulness of any Tusome trainings they attended in the preceding 12 months, the results of which are presented in Figure 14. The majority of interviewed teachers reported participating in at least one Tusome training, with only 12 percent saying they did not partake in any training in the past year. Those who attended one or more trainings universally reported a positive experience, with nearly all teachers finding the trainings both useful and relevant. The quality of training was likewise rated highly, with 85 percent saying the trainings were of high or very high quality. It is noteworthy that not a single teacher we spoke with gave a negative rating of the quality, relevance, or utility of Tusome trainings. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 52 Figure 14: Teacher Training Participation and Subjective Assessment (Teacher-Reported) Teachers were also asked to report the frequency with which they engaged in classroom activities aligned to the Tusome pedagogical approach and curriculum, as outlined in Figure 15. Teachers reported doing the majority of activities with their classes at least three times per week, with the exception of having pupils re-tell stories they read or hear, which around ten percent of teachers said they never do and 30 percent report doing once per week. Overall, the frequency of these activities decreased relative to midline, with a notable shift in the number of teachers who report doing a given activity five versus three days per week. For example, over half of teachers at midline reported asking pupils comprehension questions about a read text five days a week; at endline, only 20 percent reported doing so. This is presumably due a change to the timetable in 2019—now followed by 70 percent of teachers—which reduced the number of English and Kiswahili lessons from five to three per week (or from 2.5 to 1.5 instructional hours per week). 47 11 17 12 12 Number of Tusome training sessions More than 6 5-6 sessions 3-4 sessions 1-2 sessions None 36 50 14 0% 20% 40% 60% 80% 100% Quality of Tusome teacher training Very high quality High quality Moderate quality 52 47 Relevance of Tusome teacher training Very relevant Relevant Somewhat rel. 71 28 0% 20% 40% 60% 80% 100% Usefulness of Tusome teacher training Very useful Useful Somewhat useful USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 53 Figure 15: Number of Days (of 5) the Following Activities Took Place (Teacher-Reported) In addition to conducting interviews, data collectors observed teachers leading an English or Kiswahili lesson and recorded real-time and post-observation data on the lesson. As shown in Figure 16, over 80 percent of observed teachers used the Direct Instruction Model (DIM) of “I do, we do, you do” at endline. DIM is a widely used instructional strategy in which a new skill is scaffolded by gradually releasing responsibility from the teacher to the pupils, and represents a core component of the Tusome pedagogical approach. Figure 16: Teacher Classroom Practices (Observed) Other practices such as appropriate lesson pacing and giving positive or constructive feedback were similarly demonstrated by teachers around 90 percent of the time. As can be seen in Figure 16, however, there was an across-the-board decline in these pedagogical practices relative to midline; 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Pupils retold story they read Pupils retold story I read Pupils answered comp. questions about text they read Pupils answered comp. questions about text I read Pupils sounded out unfamiliar words Pupils assigned reading to do at home Pupils assigned reading during school time Choral repetition/reading Pupils learned meaning of new words Pupils read aloud to others None Pupils copied text 1 Day 2 Days 3 Days 4 Days 5 Days 90% 94% 92% 94% 93% 84% 89% 89% 89% 88% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Generally followed DIM Used adequate pacing Gave encouraging feedback Praised pupils for effort Provided remediation to wrong/no answer Midline Endline USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 54 despite this, endline values remain high and the changes since midline are likely not substantial enough to be driving drops in reading performance. Tusome emphasizes the use of synthetic phonics, an evidence-based approach to literacy acquisition in which children are taught letters, letter sounds, and blending to produce words. Teachers’ self-reported beliefs on phonics versus a more traditional “whole language” approach were mixed. During the interviews, teachers were randomly presented with one of two statements—one positively framed towards whole language or one positively framed towards phonics—and asked to state whether they agree or disagree with that statement. While teachers were inclined to agree with either statement due to framing bias, they were overall more likely to agree with the pro-phonics statement, and more likely to disagree with the pro-whole language statement. Figure 17: Teacher Beliefs on Phonics v. Whole Language Approach to Literacy Acquisition A similar question was posed vis-à-vis lesson planning. Teachers were asked to state whether they agree or disagree with one of two randomly presented statements regarding the use of structured guides versus developing their own lesson plans. While framing bias is likewise evident here, teachers were overall more likely to favor structured guides than custom lesson plans. More specifically, 85 percent of teachers agreed or strongly agreed with the positively framed statement on structured teachers’ guides—nearly double the amount of teachers reporting the same for teacher-customized lesson plans. Figure 18: Teacher Beliefs on Phonics v. Whole Language Approach to Literacy Acquisition 53% 13% 40% 45% 4% 32% 2% 9% Pupils best learn to read new words by first understanding the letters and sounds that make up the word Pupils best learn to read new words by visually memorizing them as whole units Strongly agree Agree Neutral Disagree Strongly disagree 35% 17% 50% 27% 11% 43% 4% 11% When it comes to teaching reading, it is better for teachers to work from structured teachers’ guides than to develop their own lessons When it comes to teaching reading, it is better for teachers to develop their own lessons than to work from structured teachers’ guides Strongly agree Agree Neutral Disagree Strongly disagree USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 55 Teaching and Learning Materials To facilitate improved reading instruction methods and delivery, Tusome provides teachers with structured/scripted instructional guides for English and Kiswahili language activities. Overall, 97 percent of teachers reported having a Tusome teacher’s guide in their classroom at endline, steady from results at midline. However, 11 percent of teachers did not use these Tusome teacher’s guides at all during the English or Kiswahili lesson observation. For the 89 percent who did use the Tusome guide, lesson observers were asked to record the unit number of the lesson being taught that day, the distribution of which is shown in Figure 19. Figure 19: Place in Tusome Teacher’s Guide at Time of Data Collection (Observed) There are 150 lessons or “units” in the Tusome teacher’s guide for a given grade level, which are designed to be completed during the academic year so that pupils enter the next grade prepared for more advanced content. Despite it being near the close of the school year at the time of endline data collection, the majority of teachers had completed less than two-thirds of these 150 units. This is in stark contrast to the same time point at midline, when teachers had completed over 80 percent of lessons in the teacher’s guides.23 Table 19: Average Number of Tusome Lessons Completed in October 2016 and 2019 Subject Grade 1 Grade 2 Midline Endline Difference Midline Endline Difference English language activities 122.2 90.6 -31.6 125.3 95.2 -30.1 Kiswahili language activities 122.3 89.6 -32.7 127 90.4 -36.6 23 Piper, B., Destefano, J., Kinyanjui, E. M., & Ong’ele, S. (2018). Scaling up successfully: Lessons from Kenya’s Tusome national literacy program. Journal of Educational Change, 19(3), 293-321. Yes, 89% No, 11% Teacher taught lesson using Tusome guide 3% 1% 1% 1% 1% 3% 8% 9% 18% 13% 16% 11% 9% 5% 2% 0% 5% 10% 15% 20% 25% Units 1-10 Units 11-20 Units 21-30 Units 31-40 Units 41-50 Units 51-60 Units 61-70 Units 71-80 Units 81-90 Units 91-100 Units 101-110 Units 111-120 Units 121-130 Units 131-140 Units 141-150 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 56 As shown above in Table 19, teachers at endline were considerably behind midline in terms of progress in the Tusome teacher’s guides. Compounding this, many teachers reported that they had not fully covered the content prior to the unit they were teaching at the time of the observation, with around 17 percent of teachers covering one half or fewer of the earlier units. This is presumed to be an outcome of MoE’s 2019 Competency-Based Curriculum (CBC) reform, which revised the grade 1 and 2 timetables including reducing the classroom time available to cover the content in the Tusome guides. While the Tusome guides were qualitatively updated to reflect CBC curricular changes, the number of lessons (150) and the length of the guides did not change with the revision. As such, they remained prima facie aligned to the earlier timetable. In cases where the Tusome teacher’s guide was used, lesson observers followed along with the teachers using an electronic copy of the guide pre-loaded to a tablet. At the end of the lesson, observers then assessed teachers’ overall adherence to the guide script as well as their coverage of sub-sections of the unit(s) taught that day. As shown in Figure 20, teachers who used the guide mostly followed the instructions/script and skipped minimal content, if any. This suggests that most teachers are able to use the guide comfortably and successfully. Figure 20: Teacher Adherence to Tusome Teacher Guides (Observed) That most teachers appear comfortable using the guide but lack adequate time to cover its content is supported by feedback from teachers themselves. As shown in Figure 21, over half of teachers reported challenges implementing Tusome, however only one percent of such teachers pointed to inadequate training as the source of these challenges. Furthermore, of all the teachers interviewed only four individuals cited Tusome’s effectiveness as an implementation challenge. Insufficient time was the main implementation challenge teachers reported at endline, with 39 percent of teachers who face challenges saying they lack sufficient time to cover the content and 23 percent claiming that Tusome lesson pacing is too fast. Per Figure 21, other commonly reported challenges include lack of materials and perceived misalignment with the new CBC work schemes. Followed very closely, 72% Followed closely, but not all the time, 24% Did not refer to guide at all, 2% Other, 3% How closely did teacher follow specific instructions in Tusome guide? All the content was covered, 77% Skipped some sections, 16% Skipped most sections, 2% Lesson not followed, 2% Other, 3% How well did teacher cover the units taught during observation? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 57 39% 31% 23% 12% 9% 6% 5% 4% 3% 3% 1% 0% 10% 20% 30% 40% Not enough time to cover Not enough materials Lesson pacing too fast Misalignment w/ work schemes Materials old/damaged Misalisgnment w/ CBC Errors in printing Pupil language barrier Lesson pacing too slow Not effective Inadequate training Figure 21: Teacher Challenges Implementing Tusome (Teacher-Reported) In terms of other TLMs provided by Tusome, most classrooms maintained a 1:1 student-to-book ratio even as the printing and distribution of books shifted to GoK ownership. Overall, 80 percent of classrooms—83 percent of grade 1 and 77 percent of grade 2 classrooms—had at least one Tusome pupil’s book per student, which is down from the 96 percent seen at midline. This finding is consistent with the 31 percent of teachers who reported “lack of materials” as a challenge in implementing Tusome. Supervision and Support Continuous, classroom-based monitoring, supervision, and coaching are a core pillar of the Tusome model. Under Tusome, supervisory staff including Head Teachers, Curriculum Support Officers (CSOs), and Instructional Coaches (for APBET schools) are tasked with regularly observing English and Kiswahili lessons and providing real-time feedback and coaching on teachers’ performance. Teachers were thus asked to report on the frequency and helpfulness of such visits, the outcomes of which are reported below in Figure 22. Yes, 54% No, 46% Teachers have challenges implementing Tusome USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 58 Figure 22: Frequency and Helpfulness of CSO and Head Teacher Observations (Teacher-Reported) In terms of frequency, 58 percent of grade 1 and 60 percent of grade 2 teachers reported being observed about once per term by a CSO, each down around 20 percentage points from midline. Ninety percent of grade 1 and 79 percent of grade 2 teachers reported being observed at least once per term by their Head Teachers, down six percentage points and 11 percentage points from midline, respectively. While this drop in teacher observations marks a reduction in program implementation since midline, the analysis conducted under evaluation question 3 failed to establish a statistically meaningful relationship between such visits and pupil reading performance at endline. Nevertheless, teachers found both types of observations to be helpful in improving their teaching of Kiswahili and English, although they were more likely to report CSO’s being as “very helpful” (48 percent) than Head Teachers (35 percent). Very few reported such visits as “unhelpful,” suggesting strong buy-in among teachers for the supervision and support components of Tusome. 1 13 48 18 20 Frequency of CSO observations Weekly Monthly Termly Yearly Never 48 50 2 Helpfulness of CSO observations Very helpful Helpful Somewhat helpful Not helpful 9 26 47 5 12 Frequency of HT observations Weekly Monthly Termly Yearly Never 35 60 4 1 0% 20% 40% 60% 80% 100% Helpfulness of HT observations Very helpful Helpful Somewhat helpful Not helpful USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 59 4.b How effective was implementation of Tusome with regard to community awareness and engagement in reading through Youth Fund grants? KEY FINDINGS • The Tusome Youth Fund grant program was able to reach 66,512 direct beneficiaries in 526 schools, as reported by the youth groups themselves. • Youth fund grant administrators consider their grant programs to be effective in raising community awareness on the importance of reading and engaging children in reading. • While stakeholders considered the grants program successful overall, the administration of the program presented some challenges. As part of the Tusome intervention, USAID’s implementation partner RTI established Youth Fund grants to raise community awareness and engagement around the importance of early grade reading at the grassroots level. The Youth Fund provided grants to existing youth groups to carry out activities in their communities including mobile libraries, reading exhibitions, reading clubs, theater groups, murals, and parental sensitization campaigns. To evaluate the effectiveness of the implementation of Youth Fund grants, NORC included a set of questions on Youth Fund grants in the RTI KII protocols and also fielded a short web survey for youth group leaders to share their experiences participating in the Youth Fund grants program. Based on the KII responses, the implementation of the Youth Funds grants was successful. The respondents reported that 23 youth groups received grants to implement activities in their communities, and after implementation of the first round of activities each Youth Group received a second round of funding to implement activities in a second location. These findings are supported by the Young Fund web survey which found that the Youth Fund grant activities reached 66,512 direct beneficiaries in 526 schools, as self-reported by the youth groups themselves. Data from the Youth Fund web survey also revealed that youth fund grant administrators considered their grant programs to be effective in raising community awareness on the importance of reading and engaging children in reading: when asked to score the effectiveness of their grant programs from one to four, with one being very ineffective and four being very effective, grant administrators on average gave their programs a rating of over 3.85. The reach of the Youth Fund grants, and their perceived effectiveness in raising community awareness on the importance of reading and engaging children in reading, is an important finding in light of quantitative findings from evaluation question 3 that show pupils who read stories at home score 3-6 CWPM higher than those that don’t. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 60 Figure 23: Youth Fund Grants Program Impact (Grantee-Reported) 66,512 526 226,167 69,176,968 DIRECT BENEFICIARIES SCHOOLS REACHED INDIRECT BENEFICIARIES FUNDING RECEIVED (KSH) However, while the grants program was considered successful, the administration of the program presented some challenges. Two respondents reported difficulty overseeing 23 grantees under the larger Tusome intervention since the grants required high levels of financial scrutiny and administrative oversight, at times requiring technical staff to support the Youth Fund grants program beyond what was originally anticipated. Respondents explained that the Tusome team was prepared to implement the Youth Fund grants, however they found that participating youth groups had lower capacity than expected and required greater support throughout implementation than originally planned. The evaluation found that the youth groups required substantial support identifying the types of activities that would qualify for the grants as well as training on how to manage the activities, how to properly report activity progress, and how to ensure the activities adhered to USAID’s branding requirements. While this level of training and capacity building was not originally planned for, it was a positive unintended outcome of the grants program. Qualitative data also show the capacity of these organizations increased over time. The data also show that turnover within the youth groups resulted in losses in institutional knowledge over the course of implementation. One respondent also noted that some of the youth groups were able to identify pupil needs outside of the realm of reading. Specifically, they were able to lobby support from the local communities to address issues such as hunger and sanitation in schools, which went above and beyond their mandate to establish reading-related activities in their communities. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 61 4.c How effective was implementation of Tusome with regard to government ownership and buy-in? KEY FINDINGS • Tusome was successful in securing ownership and buy-in from GoK at different levels of government and at different points in the implementation process. GoK respondents were personally supportive of the methods and materials developed under Tusome and hoped to see them continue in the future. • While Tusome was broadly successful in securing ownership and buy-in from the GoK, there were moments throughout implementation where USAID and RTI had to work harder to secure support for the Tusome methods, including during the CBC development process. Overall, Tusome was successful in securing ownership and buy-in from GoK at different levels of government and at different points in the implementation process. Qualitative data show that government ownership and buy-in were priorities from the outset of Tusome. USAID respondents reported that Tusome, by design, placed the MoE and SAGAs as leads of Tusome technical committees in order to secure ownership and buy-in. These findings are corroborated through RTI interviews, during which respondents emphasized that their role was not to simply get a rubber stamp of approval from the GoK for an intervention, but rather to collaborate with the GoK to harness existing will for quality education and build a curriculum using evidence-based decision making. This collaboration between USAID, RTI, and the GoK, and buy-in from all actors, can be seen through the passage of various GoK policies that mainstreamed the Tusome method. These policies are described in detail under evaluation question 5a. While GoK ownership and buy-in was a focus from the start of Tusome, there were moments throughout the implementation during which USAID and RTI had to work harder to secure buy-in and support for the Tusome methods. A large part of USAID’s role throughout the implementation, according to USAID respondents, was to act as interlocutors between USAID, Tusome, and the GoK to ensure the MoE and SAGAs each understood their roles and responsibilities in the delivery of Tusome, and to resolve disputes as they arose. This role was crucial during the development of the CBC, which required USAID and RTI to adapt to the GoK’s desire for a new curriculum while ensuring that the Tusome model was included in the new curriculum. Respondents reported that RTI was included in CBC design meetings from the beginning, however, there were moments of confusion on how Tusome would fit into the new curriculum as the CBC was evolving. Ultimately, the Tusome method was mainstreamed into the CBC though the new curriculum initially reduced instructional hours and the number of lessons per week dedicated to English and Kiswahili language activities. Results from GoK KIIs also found that individual GoK respondents bought into the Tusome model and were hopeful it would continue after USAID funding ends. Many of the respondents said they were personally supportive of the methods and materials developed under Tusome, both as professionals within the GoK and personally as Kenyan citizens. They also reported broad institutional support for the Tusome method. However, two respondents reported that while they were supportive of the Tusome methods and materials, they would like to see the method evolve to include other local languages in the future. Findings from the qualitative interviews also highlight the importance of buy-in and ownership of USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 62 the Tusome program at the county level. While basic education is still centralized in Kenya, many GoK respondents indicated that counties supported, and would continue to support, the Tusome method because their teachers and CSOs believe in the program. In spite of these accounts of broad support from the GoK, it is still seen as important to continue building deeper government ownership and buy-in to fully transition the Tusome methods and materials to the GoK for long-term implementation. In particular, USAID, RTI, and GoK respondents identified the need for the GoK to take more financial ownership of Tusome, specifically integrating different costs of Tusome into the national budget when it ends. Respondents also reported concerns with sustaining current ownership and buy-in, indicating they may erode over time as GoK personnel changes. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 63 EVALUATION QUESTION 5 5.a What GoK procedures, policies, or guidelines have been established as a result of or in support of Tusome? KEY FINDINGS The three key policies, procedures, or guidelines that were established or operationalized as a result of or in support of Tusome were: 1) GoK’s centralized book procurement procedure, 2) the national CBC reform, and 3) classroom observations and teacher coaching. • With Tusome’s support, the GoK was able to successfully implement its own centralized book procurement procedure which led to economies of scale, enabling the GoK to purchase textbooks at a significantly reduced rate. The cost savings allowed the GoK to purchase textbooks at a 1:1 student-to-textbook ratio, thereby operationalizing for the first time an existing policy for providing each student with his or her own textbook • The timing of the CBC reform overlapped with the implementation of the Tusome program, which presented an opportunity to integrate Tusome methods and materials into the formal national curriculum. While the Tusome method is formally integrated into the CBC, there remained confusion at the time of data collection among some teachers on how this translates into teaching in the classroom. • Tusome was able to leverage civic responsibility and funding to motivate existing GoK personnel to implement more frequent instructional support through classroom-based observations and teacher coaching. The classroom observations are considered a success, and respondents believe policies and systems should be put in place to keep CSOs accountable for observing teachers after Tusome implementation ends. Findings from qualitative interviews reveal several GoK policies that were established or operationalized as a result of or in support of Tusome, including a centralized book procurement procedure, the national CBC reform, and classroom observations and teacher coaching. The following sections will focus on these three key policy areas. In this section we also present findings for the three sub￾questions for evaluation question 5a, as outlined below: • 5.a.i What role did Tusome activities play in the formulation and adoption of such procedures, policies, or guidelines? • 5.a.ii To what extent are these procedures, policies, or guidelines being implemented? What are the catalysts and barriers to effective implementation? • 5.a.iii What are the early effects of such procedures, policies, or guidelines? In addition to the three key policy areas, the data also indicate that other policies—including the development of Tusome training modules for Primary Teacher Training Colleges (PTTCs), use of data for decision making, and use of technology in the classroom—were also established as a result of or in support of Tusome, and thus warrant further research. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 64 Centralized Book Procurement Procedure During implementation of the Tusome program, the GoK shifted from a decentralized book procurement procedure to a centralized book procurement procedure. Under the previous decentralized procurement and distribution system, each school in Kenya independently decided which books they would use for each subject, while staying within the pre-approved list of books outlined in the national Orange Book, and was responsible for working with vendors to purchase copies of the selected books for their school.24 According to qualitative findings, this model led to inflated book prices since each school was negotiating with vendors independently instead of banding together to create economies of scale through larger orders. The inflated pricing in turn resulted in a 3:1 student-to-book ratio in most schools. In January 2019, the GoK shifted to the centralized book procurement procedure, in which they mandated which book would be used for each subject across all schools and evaluated bids directly with vendors to procure these books for all Kenyan schools. This centralized policy allowed the GoK to take advantage of economies of scale and reduce the price of books enough to achieve a 1:1 student-to-book ratio in each school. The GoK’s decision to move to a centralized book procurement procedure came after the Tusome program successfully used a centralized procedure to procure enough Tusome teaching and learning materials to ensure a 1:1 student- and teacher-to-book ratio in each school. The qualitative findings indicate that once the GoK saw that a centralized system could reduce costs to a point that could ensure 1:1 ratios, they asked RTI for guidance on transitioning to a centralized procedure. RTI then supported the GoK in operationalizing their own centralized procurement procedure. The GoK successfully developed its own centralized book procurement procedure and used it to procure grade 1 Tusome learning materials across the country in 2019 with GoK funds. After the successful procurement of grade 1 materials, it is expected that GoK will continue to use the centralized book procurement procedure to buy Tusome learning materials in the future. The qualitative findings indicate that the primary catalyst for the centralized book procurement procedure was the political will to reduce costs and ensure a 1:1 student-to-book ratio. National Competency Based Curriculum Reform In 2017, the GoK began to pilot, and subsequently rolled out, the national CBC reform. The CBC was the culmination of a national needs assessment led by KICD in 2016 which determined the need for a shift to a competency-based curriculum, and a national stakeholders’ conference held in January 2017 which endorsed the framework for curriculum reforms. The new curriculum is intended to be learner￾centered; adapt to changing needs of students, teachers, and society; and emphasize what pupils are expected to do rather than what they are expected to know. The overall goal of the CBC is to ensure that pupils have the required knowledge, skills, values, and attitudes to solve situations they encounter in daily life, and can apply them in practice.25 The key changes to the curriculum, as is relevant to the 24 The Orange Book is a document provided by the KICD which provides a comprehensive list of course books and other instructional materials approved by the MoE for use in pre-primary and lower primary education. 25 David Njeng’ere Kabita, Lili Ji. (2017). The Why, What and How of Competency-Based Curriculum Reforms: The Kenyan Experience. Current and Critical Issues in Curriculum, Learning and Assessment, 11. https://unesdoc.unesco.org/ark:/48223/pf0000250431 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 65 Tusome program, were changes to the class timetable and learning materials. The revised grade 1-3 timetable now includes a new subject area—literacy—and reduced the number of instructional hours for English and Kiswahili language activities. While the announcement of the CBC reform was not anticipated during the initial Tusome design, it did present an opportunity for the Tusome methods and materials to be integrated into the new national curriculum. USAID and RTI worked with the GoK, specifically KICD, to align Tusome materials to the CBC. As a result, the Tusome methods, including in-service teacher training techniques and modules for PTTCs, have been aligned to the official curriculum. The nationwide roll out of the CBC in 2019, which attempted to include Tusome, faced some difficulties. The first among these, as it relates to the sustainability of the Tusome method, was the initial reduction of both English and Kiswahili language lessons from five lessons per week to three. The potential impact of the reduced English and Kiswahili lessons on pupil reading performance can be seen in the assessment findings of this evaluation. In addition, qualitative findings show that teachers were unsure how CBC and Tusome worked together, leading to uneven implementation of the new curriculum. Some respondents reported teachers being unsure how much time to dedicate to each subject, and others reported confusion amongst teachers on how to use the new learning materials designed under the CBC. In addition to English and Kiswahili learning materials, CBC introduced learning materials for a third subject area called literacy,26 which KII respondents reported were often confused with the English and Kiswahili learning materials. Concerns over the confusion were reported to the MoE, which planned to release a circular clarifying key points of the CBC.27 In addition to the circular, respondents recommend holding more PTTC lecture trainings to ensure new teachers are being adequately trained on CBC and, by extension, Tusome. Classroom Observations and Teacher Coaching Through the Tusome intervention, the GoK and RTI were able to motivate existing GoK personnel to implement a more effective method of instructional support through classroom-based observations and teacher coaching. For each observation, a CSO observes either an English, Kiswahili, or math lesson and fills in a tablet-based teacher observation form, making comments on what the teacher did well and what areas need improvement. After the lesson, the observer assesses three randomly selected pupils to estimate an average reading level for the classroom. Finally, the observer confers with the teacher to share feedback from the observation and reading assessment. This entire interaction is tracked through the tablet-based platform and allows CSOs to track progress of individual teachers over time and provide data on student performance to county, regional, and national stakeholders. Tusome was crucial in preparing CSOs to carry out the classroom observations and teacher coaching, and provided necessary resources for them to implement the observations on a regular schedule. According to the qualitative findings, the classroom observation and coaching activities were already part of CSOs responsibilities, however the observations were not taking place on a regular schedule 26 According to KICD’s Basic Education Curriculum Framework, literacy as a subject is to be taught in the first language of the learner. 27 Since the collection of endline data in 2019, the Ministry of Education issued a circular in early 2020 to address the confusion among teachers identified by the evaluation. To help teachers operationalize the circular, the MoE intended to further clarify the time dedicated to classroom instruction for English and Kiswahili using Tusome materials through teacher training that unfortunately had to be postponed due to COVID-19. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 66 prior to Tusome. The program addressed this by first training the CSOs on Tusome methods and then providing resources, in the form of travel reimbursements, for the CSOs to visit schools in their areas and conduct the observations. The qualitative findings suggest CSO observations now take place on a regular schedule, however results from the evaluation’s teacher survey differ. While the teacher survey found that 80 percent of teachers reported being observed by a CSO at least once in the past 12 months, the frequency of these visits vary from once a year to monthly. In spite of the variation in frequency of these visits, teachers generally find coaching and supervision from CSOs helpful. When discussing the catalysts for increasing classroom observation and teacher coaching sessions, respondents reported that Tusome motivated CSOs to conduct observations out of civic responsibility but that the visits were ultimately made possible through the allocation of resources to reimburse CSOs for traveling to schools. The classroom observations are considered a success, and respondents believe policies and systems should be put in place to keep CSOs accountable for observing teachers after Tusome implementation ends. Respondents had different views on how to ensure the continuation of classroom observations and coaching. Some respondents, primarily from within the GoK, suggested providing motorbikes to CSOs and reimbursing fuel costs to facilitate their travel to conduct classroom observations after USAID funding ends, while other respondents expressed skepticism that providing motorbikes would be sufficient as there are additional prohibitive costs for fueling and maintaining the motorbikes that would fall on CSOs. Other respondents mentioned using the Tusome dashboard, a tool primarily used to track teacher and learner performance, as a possible tool to track classroom observations and keep CSOs accountable for their classroom observations. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 67 5.b Which existing GoK procedures, policies, or guidelines are critical for the long-term sustainability of the program? KEY FINDINGS • A centralized book procurement procedure and the continuation of classroom observations and teacher coaching sessions by CSOs were identified as key procedures, policies, and guidelines for ensuring the long-term sustainability of the Tusome method. Qualitative findings indicate that the most critical GoK procedures, policies, and guidelines for the long￾term sustainability of the program are the continuation of the centralized book procurement procedure and the continuation of classroom observations and teacher coaching sessions by CSOs, as described in the preceding section. To understand Tusome’s sustainability from the standpoint of CSOs, the evaluation included a series of questions on sustainability in the CSO survey. As seen in Figure 24, the majority of CSOs (54 percent) believe their counties will continue with the Tusome approach after USAID funding is discontinued. Of those who do not believe their counties will continue with the Tusome approach after USAID funding is discontinued, the majority cited insufficient resources/budget for either training (73 percent) or monitoring (64 percent) as the reasons. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 68 Training 20% Sensitization 9% Ensure continued funding of activities 9% Establish funding at county-level 8% Government ownership 8% Support CSO school visits / transport 8% Develop policies / guidelines 6% Provide materials 6% Monitoring 6% Integration of Tusome and CBC 5% Efficient management / supervision 2% Other 13% What is needed to ensure Tusome will continue Figure 24: CSO Beliefs on Continuation of Tusome at County-Level When asked what, if anything, CSOs thought was needed to ensure their counties continued implementing Tusome after donor support was discontinued, answers varied more widely. The most common response was additional training (20 percent). However, expressions of financial need or support for continued CSO school visits and monitoring add up to 30 percent of responses when combined, as seen in Figure 24. Yes, 54% No, 38% Dont know/ no answer, 8% CSO believes Tusome will continue in county USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 69 5.c Are there additional procedures, policies, or guidelines that would further enhance Tusome’s sustainability? KEY FINDINGS • Clarification on how the Tusome method is formally embedded within the CBC is critical to the continued sustainability of Tusome. The qualitative findings in regard to what additional procedures, policies, or guidelines would enhance Tusome’s sustainability revolve primarily around how to reinforce the newly rolled out CBC and ensure its implementation is uniform across schools. The most common recommendation was the need for clarification of key points of the CBC, including how the Tusome method fits within the CBC. Respondents reported confusion around both the amount of time that should be dedicated to Tusome lessons, meaning English and Kiswahili language instruction, and the materials that should be used for these lessons. The CBC introduced a new subject called literacy, which is different from the English and Kiswahili lessons, and is supposed to have its own corresponding teaching and learning materials. These materials were often confused as those that should be used for English and Kiswahili lessons. The qualitative findings suggest the root of this confusion may be the terminology, as literacy can be seen as describing the English and Kiswahili lessons as well as the new subject created under the CBC. As mentioned previously, a circular dated January 3, 2020 was distributed by the Kenyan Ministry of Education clarifying the number of lessons per week that should be dedicated to each subject and which course books correspond to each subject. Another recommendation made by two GoK respondents was to make Tusome more flexible to adapt to the changing education climate in Kenya after USAID funding ends. Proponents of this recommendation indicated that the Direct Instructional Model (DIM), what has been referred to as the “Tusome method,” is too rigid and does not align with the CBC. They argue that the model is appropriate for some students who require high levels of support, but is not appropriate, and may hinder students who grasp content quickly. One respondent’s suggested solution to this perceived inflexibility is to allow teachers to assess student’s progress and determine which lessons they should focus on based on their performance, instead of following a pre-prescribed plan. However it is important to contrast these views with the quantitative findings, which show only one percent of teachers think the Tusome lesson pacing is too slow. Moreover, the consistent relationship between pupil reading performance and Tusome “dosage” suggests the model has been successful. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 70 5.d What other key lessons have been learned throughout the policy-making process? KEY FINDINGS • The strong collaboration between USAID, RTI, and the GoK in the design and implementation of Tusome was one of the greatest keys to Tusome’s success. • Lessons learned associated with the handover of Tusome implementation from RTI to the GoK are largely left unanswered at the time of writing, as the Tusome program received a one-year extension that will focus on the handover as well as COVID-19 response. The key lesson learned based on qualitative findings, which has not been discussed in the evaluation questions above, is the importance of collaboration between USAID, RTI, and the GoK in the design and implementation of Tusome. This sentiment was expressed by each stakeholder group, in one form or another, most of whom believe that Tusome’s success is due in large part to the program’s ability to engage government officials at all levels and foster buy-in from them. USAID and RTI respondents expressed how important this idea was from the beginning, indicating that it was an intentional decision to include the GoK from the beginning of the process as true collaborators and not just gate keepers from whom they needed approval to implement a project. GoK respondents also expressed the importance of being seen as collaborators; however, some reported moments of animosity between the Ministry and some SAGAs at the start of implementation, indicating they felt the intervention was being imposed on them without an opportunity to provide their inputs. These respondents argue there should have been more dialogue in the initial stages of Tusome design in order to allow some SAGAs to fully understand the activities that were being implemented and give them the space to provide feedback for improvements to the program. They believe this additional dialogue would have fostered more buy-in from the SAGAs and given them the tools needed to defend the program against the push back they received. Expanding on this idea of collaboration between Tusome and key stakeholders, respondents also noted the importance of fostering buy-in from people on the ground—including Head Teachers, teachers, pupils, parents, and community members—through sensitization on the impacts of the program within their communities. Fostering buy-in from these stakeholders and gaining their support would further incentivize the GoK to continue implementing the Tusome method. CONCLUSIONS Overall, independent evaluation results provide strong evidence for the efficacy of the Tusome model of instruction. From baseline to endline, pupils have shown statistically significant improvements on all EGRA subtasks in both English and Kiswahili. Of particular note, grade 2 English reading gains observed from baseline to endline are roughly equal to gains from an additional full year of schooling. Furthermore, over the life of the evaluation exposure to Tusome has consistently corresponded with reading performance: increased exposure to Tusome between baseline and midline/endline corresponded with an increase in pupil reading performance. Similarly, reduced exposure to Tusome between midline and endline corresponded with a decrease in pupil reading performance. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 71 Statistical models at endline also show that better implementation of Tusome in the classroom positively correlates with reading fluency. Tusome lesson plan progress is positively associated with English reading performance: each 10-unit advance in the Tusome teacher’s guide is associated with an increase of one correct word per minute in English reading fluency. Similarly, teacher-reported frequency of pupils sounding out unfamiliar words is positively associated with English reading performance. When teachers use both English and Kiswahili while teaching Kiswahili—i.e., use “code switching"—pupils score 3.19 words per minute higher in Kiswahili. Despite marked progress since baseline, the vast majority of early grade learners in Kenya are unable to read at a grade-appropriate level. Across the board, fewer than eight percent of grade 1 and 2 pupils are able to read Kiswahili at a grade-appropriate level. In English, just 14 percent of grade 1 pupils and 18 percent of grade 2 pupils are meeting Kenya’s reading benchmarks. Furthermore, there is a concerningly high proportion of pupils who are unable to read entirely: nearly two-thirds of grade 1 pupils and one-third of grade 2 pupils cannot read a single word in Kiswahili. For English, the number of non-readers is 34 and 14 percent for grades 1 and 2, respectively. The reduction in English and Kiswahili instructional hours under the new CBC timetable appears to be a driver of post-midline declines in reading performance. While several implementation factors changed since midline, teachers were on average around 30 (of 150 total) Tusome lessons behind midline levels due to the reduction in English and Kiswahili instructional time. That grade 1 pupils saw the sharpest performance losses since midline fits with this hypothesis, as grade 2 pupils were still under the pre-CBC timetable when they were in grade 1 and thus had a higher “dosage” of English and Kiswahili using the Tusome approach compared to their grade 1 counterparts. Overall, reduced exposure to Tusome under the new CBC timetable appears to have negatively impacted children’s English and Kiswahili skills. Regular reading practice and classroom resources are positively correlated with reading fluency while larger class sizes and the use of local language at school are negatively correlated with reading fluency. Pupils who report reading stories at home and/or reading aloud at school score 3-6 correct words per minute higher than those who don’t, holding all else constant. In addition, smaller class sizes, the availability of tables and chairs/benches for children, and the classroom having the timetable posted on the wall were correlated with reading fluency, possibly because they serve as proxies for school leadership or school resources. On the other hand, local language use during English instruction is strongly negatively associated with both English and Kiswahili reading fluency, with pupils from such classrooms scoring 7-12 correct words per minute lower on average (no such correlations were found for local language use during Kiswahili instruction). Teachers generally demonstrate moderate to high levels of support for Tusome, yet face a number of implementation challenges. While teachers broadly support Tusome, they have struggled to keep pace with the instructional approach from the midpoint of the program onward. Insufficient time was the main implementation challenge teachers reported at endline, with 39 percent of teachers who face challenges saying they lack sufficient time to cover the content and 23 percent claiming that Tusome lesson pacing is too fast, likely owing to the CBC timetable reduction in instructional time from five periods to three periods per week for English and five periods to three periods per week for Kiswahili. In addition, 31 percent of teachers report “lack of materials” as a challenge in implementing Tusome, particularly materials for pupils such as story books and homework books. Other frequently cited USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 72 implementation challenges include misalignment with work schemes (12 percent), old or damaged materials (9 percent), and misalignment with CBC (6 percent). Tusome was largely successful in securing ownership and buy-in from the GoK at different levels of government and at different points in the implementation process, but more work is needed to ensure long-term sustainability after USAID funding comes to an end. Most GoK respondents were personally supportive of the methods and materials developed under Tusome and hoped to see them continue in the future. However, long￾term sustainability of Tusome will depend on upholding existing procedures, policies, and guidelines— particularly the centralized book procurement procedure, with its facilitation of the 1:1 textbook to student ratio, and CSO observations/coaching. RECOMMENDATIONS Ensure that Kenya’s learners achieve the English and Kiswahili reading goals of the CBC by allocating sufficient instructional time to cover the content developed under Tusome. The reduction in lessons from five to three per week during the 2019 school year led to gaps in coverage of Tusome instructional content. As a result, pupil reading performance actively worsened, moving away from GoK reading benchmarks. The GoK should thus consider increasing the instructional time for both English and Kiswahili back to five lessons per week to better align with the Tusome materials and Kenya’s reading goals under the CBC. Likewise, the GoK should formally sensitize educators across the Kenyan system to the change through trainings and other communication channels. Implement remedial literacy programming for pupils in the “zero” and “beginner” reader categories. While there has been marked progress since baseline, the great majority of pupils in Kenya are still unable to read with comprehension at a grade-appropriate level. Remedial programming for struggling readers could involve supplementing core reading lessons with additional English and Kiswahili instruction and guided practice time tailored to the learning levels and needs of pupils or extracurricular reading support by tutors, volunteers, teachers, and/or through education technology. Cross-cultural research on “teaching at the right level” (TaRL) shows that grouping children based on learning levels—for example, their EGRA benchmark categories—rather than age or grade and tailoring instruction and practice based on what they know can lead to large, cost effective gains in learning outcomes for struggling pupils. While the TaRL approach should not replace instruction using Tusome materials, the value and importance of remedial reading instruction is especially salient given the extended school closures during the COVID-19 pandemic, which now require learners to “catch up” in order to acquire expected foundational skills. Ensure pupils have the time and resources to regularly practice reading, both inside and outside of the classroom. Given the strong, statistically robust relationship between pupil reading performance and regular reading practice, ensure that children are able to practice reading in a variety of formats, including practicing reading silently/aloud as well as at home/in the classroom. This may involve ensuring access to English and Kiswahili decodable and leveled story books and work books for home use or supporting extracurricular reading clubs, especially for children whose home environments are less conducive for reading practice. Work with appropriate GoK actors to put in place systems to ensure classroom-based observations continue after USAID funding ends. This might include creating accountability systems to ensure CSO/QASO school visits take place under the current GoK transport facilitation scheme, additional school-based support to USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 73 complement CSO/QASO observations, or alternatives to face-to-face training and coaching in situations where face-to-face interactions are not feasible. The Tusome dashboard can be transferred to the Ministry of Education and serve as an accountability system to ensure CSOs/QASOs are conducting school-based observations, in addition to its primary purpose of providing valuable school-level data for national, regional, and local stakeholders. Additional school-based support can be found in Head Teachers and Deputy Head Teachers who can conduct classroom observations in their schools, reducing the burden on CSOs/QASOs and the current GoK transportation facilitation scheme. Finally, alternatives to face-to-face interaction may include using technology to implement virtual CSO/QASO observations and coaching. Implementing a remote CSO/QASO system would require an initial investment in the technology, for example tablets and virtual meeting software, as well as training CSOs/QASOs and teachers on how to use the new technology, but would cut the recurring CSO/QASO transport facilitation costs. A move to virtual observations and coaching can capitalize on the current expansion of virtual meeting technology that the world experienced as a result of the COVID-19 pandemic. Continue working with the GoK including MoE, TSC, and SAGAs to build renewed and deeper buy-in for key Tusome policies and practices as well as future USAID-funded programming. USAID, RTI, and implementers of future education interventions should continue participatory engagement with GoK stakeholders— across all seniority levels within the MoE, TSC, and SAGAs—to build buy-in for donor-supported initiatives and ensure they are aligned with government policies and priorities. Deeper buy-in across the GoK will foster a greater sense of ownership as well as insulate key programs from leadership transitions over the longer-term. Developing a clear strategy for fully transitioning Tusome to government ownership—including ensuring timely and sufficient budget allocations from MoE and GoK—is also critical for long-term sustainability. Work with the Ministry of Education to implement a revised textbook policy that includes centralized procurement. Given the importance of the centralized book procurement procedure implemented by Tusome, and the resulting 1:1 student to textbook ratio, USAID, RTI, and implementers of future education interventions should work with the GoK to implement a revised textbook policy solidifying the centralized procurement procedure. A revised textbook policy will ensure students have access to critical learning materials while reducing the costs of textbook procurement for the GoK, and ensure the long-term sustainability of a core aspect of the Tusome intervention. For future interventions, the design of Youth Fund grants programs should take into account the existing capacity of local youth organizations, and ensure enough funding/staff to allow for sufficient administrative and managerial oversight. Given the level of effort needed to administer and manage the Youth Fund grants under the Tusome project, future interventions should familiarize themselves with the existing capacity of local organizations, including their capacity for implementing activities as well as their capacity for reporting on those activities. This will allow future interventions to secure enough funding and staffing for the administrative and managerial needs of the grants. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 74 ANNEXES ANNEX I: ADDITIONAL EGRA RESULTS Table 20: English EGRA Raw Subtask Scores including ML-EL differences Subtask Grade 1 Grade 2 Baseline Midline Endline ML-EL Diff. Sig. Baseline Midline Endline ML-EL Diff. Sig. Phoneme Segmentation 1.1 3.8 2.8 -1.0 ** .6 5.0 4.1 -.9 ** Letter Sound Knowledge 15.1 26.1 21.4 -4.8 *** 10.2 32.6 26.3 -6.3 *** Invented Word Decoding 5.7 10.4 7.7 -2.7 *** 10.4 18.4 15.5 -2.9 *** English Vocabulary 5.9 7.8 7.1 -.7 - 8.2 10.2 9.9 -.3 - Passage Reading (A) 10.6 22.3 13.9 -8.4 *** 23.8 43.6 36.2 -7.4 *** Reading Comp. (A) .2 .5 .3 -.2 *** .5 1.0 1.0 -.1 - Passage Reading (B) 9.6 22.0 13.4 -8.6 *** 21.8 44.2 35.8 -8.5 *** Reading Comp. (B) .2 .8 .5 -.2 ** .6 1.7 1.8 .0 - *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and ML-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 75 Table 21: Kiswahili EGRA Raw Subtask Scores including ML-EL differences Subtask Grade 1 Grade 2 Baseline Midline Endline ML-EL Diff. Sig. Baseline Midline Endline ML-EL Diff. Sig. Letter Sound Knowledge 16.5 29.2 22.8 -6.5 *** 16.2 39.5 33.8 -5.7 *** Syllable Fluency 11.0 21.3 13.7 -7.7 *** 20.9 36.5 29.7 -6.8 *** Invented Word Decoding 4.7 8.1 5.0 -3.1 *** 10.3 16.1 12.5 -3.6 *** Passage Reading 4.9 12.2 6.2 -6.0 *** 13.5 24.5 19.1 -5.4 *** Reading Comprehension .4 .9 .4 -.5 *** 1.1 2.0 1.5 -.5 *** Listening Comprehension 1.2 2.0 2.1 .1 - 1.9 2.8 3.1 .3 * *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and ML-EL differences are due to rounding. Figure 25: Baseline to Endline Effect Sizes, English Subtests 0.52 0.38 0.21 0.31 0.19 0.12 0.22 0.33 1.02 0.91 0.40 0.36 0.43 0.37 0.49 0.68 0.00 0.20 0.40 0.60 0.80 1.00 Phoneme Segmentation Letter Sound Knowledge Invented Word Decoding Vocabulary Passage Reading (A) Reading Comprehension (A) Passage Reading (B) Reading Comprehension (B) Grade 1 Grade 2 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 76 Figure 26: Baseline to Endline Effect Sizes, Kiswahili Subtests Table 22: Grade 1 English EGRA Raw Subtask Scores, by School Type Subtask Public APBET Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 1.1 3.7 2.7 1.7 *** 4.4 7.2 5.7 1.3 *** Letter Sound Knowledge 14.8 26.0 21.2 6.4 *** 31.7 39.1 34.9 3.1 ** Invented Word Decoding 5.5 10.2 7.5 2.0 *** 16.6 22.4 17.1 0.5 - English Vocabulary 5.8 7.7 7.0 1.3 *** 11.8 14.5 13.1 1.4 *** Passage Reading (A) 10.2 21.8 13.6 3.4 *** 38.0 58.2 39.4 1.4 - Reading Comp. (A) 0.2 0.5 0.3 0.1 *** 1.5 2.1 1.7 0.2 - Passage Reading (B) 9.2 21.6 13.1 3.9 *** 35.1 58.0 36.1 1.0 - Reading Comp. (B) 0.2 0.7 0.5 0.3 *** 1.7 3.0 2.7 0.9 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. 0.36 0.16 0.03 0.12 0.09 0.54 0.83 0.40 0.19 0.34 0.31 0.69 0.00 0.20 0.40 0.60 0.80 1.00 Letter Sound Knowledge Syllable Fluency Invented Word Decoding Passage Reading Reading Comprehension Listening Comprehension Grade 1 Grade 2 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 77 Table 23: Grade 2 English EGRA Raw Subtask Scores, by School Type Subtask Public APBET Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 0.5 5.0 4.1 3.6 *** 3.0 8.0 6.2 3.2 *** Letter Sound Knowledge 10.0 32.5 26.2 16.2 *** 23.8 40.7 35.4 11.7 *** Invented Word Decoding 10.1 18.2 15.3 5.2 *** 24.6 31.1 26.1 1.5 - English Vocabulary 8.1 10.1 9.8 1.7 *** 14.5 16.1 15.2 0.7 *** Passage Reading (A) 23.2 43.1 35.8 12.6 *** 61.9 81.1 68.1 6.3 ** Reading Comp. (A) 0.5 1.0 0.9 0.5 *** 2.8 3.2 3.2 0.4 ** Passage Reading (B) 21.2 43.7 35.4 14.1 *** 58.2 81.9 67.8 9.6 *** Reading Comp. (B) 0.5 1.7 1.7 1.2 *** 3.4 4.0 4.5 1.1 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. Table 24: Grade 1 Kiswahili EGRA Raw Subtask Scores, by School Type Subtask Public APBET Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 16.1 28.9 22.5 6.4 *** 38.9 52.2 39.9 1.0 - Syllable Fluency 10.7 21.1 13.4 2.7 *** 30.8 43.0 29.1 -1.7 - Invented Word Decoding 4.6 7.9 4.8 0.3 - 13.4 18.8 12.3 -1.0 - Passage Reading 4.7 12.0 6.1 1.4 ** 15.7 26.8 15.7 0.0 - Reading Comprehension 0.3 0.9 0.4 0.1 * 1.4 2.3 1.3 -0.1 - Listening Comprehension 1.2 2.0 2.1 0.9 *** 2.5 3.0 2.7 0.2 - *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 78 Table 25: Grade 2 Kiswahili EGRA Raw Subtask Scores, by School Type Subtask Public APBET Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 15.9 39.3 33.6 17.7 *** 40.4 55.0 49.4 9.0 *** Syllable Fluency 20.6 36.2 29.5 8.9 *** 41.3 54.2 44.5 3.2 * Invented Word Decoding 10.1 15.9 12.4 2.3 *** 21.7 26.2 20.6 -1.1 - Passage Reading 13.3 24.3 19.0 5.7 *** 29.6 39.0 30.6 1.0 - Reading Comprehension 1.0 2.0 1.5 0.5 *** 2.6 3.5 2.6 0.0 - Listening Comprehension 1.9 2.8 3.1 1.2 *** 3.2 3.6 3.5 0.4 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. Table 26: Grade 1 English EGRA Raw Subtask Scores, by Gender Subtask Male Female Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 1.0 3.5 2.6 1.6 *** 1.2 4.0 2.9 1.7 *** Letter Sound Knowledge 14.1 23.5 20.4 6.3 *** 16.0 28.7 22.4 6.4 *** Invented Word Decoding 5.1 9.3 7.1 2.0 ** 6.3 11.5 8.2 2.0 ** English Vocabulary 5.9 7.6 7.1 1.2 ** 5.8 8.0 7.2 1.3 *** Passage Reading (A) 9.4 20.1 12.4 3.0 ** 11.9 24.5 15.5 3.7 ** Reading Comp. (A) 0.2 0.5 0.3 0.1 * 0.2 0.5 0.3 0.1 - Passage Reading (B) 8.4 20.0 12.1 3.7 ** 10.8 24.1 14.8 3.9 ** Reading Comp. (B) 0.2 0.7 0.5 0.3 *** 0.2 0.8 0.5 0.3 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 79 Table 27: Grade 2 English EGRA Raw Subtask Scores, by Gender Subtask Male Female Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Phoneme Segmentation 0.6 4.9 3.8 3.2 *** 0.5 5.1 4.4 3.9 *** Letter Sound Knowledge 9.6 30.5 23.7 14.1 *** 10.8 34.7 28.9 18.1 *** Invented Word Decoding 9.6 17.3 14.2 4.5 *** 11.1 19.4 16.8 5.6 *** English Vocabulary 8.1 10.4 9.8 1.6 *** 8.3 10.0 10.0 1.7 *** Passage Reading (A) 21.7 41.0 32.7 11.1 *** 25.9 46.3 39.8 13.9 *** Reading Comp. (A) 0.5 1.0 0.9 0.4 *** 0.5 1.1 1.1 0.6 *** Passage Reading (B) 20.0 41.4 32.4 12.3 *** 23.5 47.0 39.2 15.7 *** Reading Comp. (B) 0.6 1.8 1.6 1.1 *** 0.6 1.7 1.9 1.3 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 80 Table 28: Grade 1 Kiswahili EGRA Raw Subtask Scores, by Gender Subtask Male Female Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 15.2 26.6 21.3 6.1 *** 17.8 31.9 24.4 6.5 *** Syllable Fluency 10.1 19.7 12.2 2.2 - 12.0 23.1 15.1 3.1 * Invented Word Decoding 4.2 7.4 4.5 0.3 - 5.2 8.8 5.4 0.2 - Passage Reading 4.2 11.0 5.4 1.2 - 5.7 13.5 7.1 1.5 - Reading Comprehension 0.3 0.8 0.4 0.0 - 0.4 1.0 0.5 0.1 - Listening Comprehension 1.3 2.1 2.2 0.9 *** 1.2 2.0 2.1 0.9 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. Table 29: Grade 2 Kiswahili EGRA Raw Subtask Scores, by Gender Subtask Male Female Baseline Midline Endline BL-EL Diff. Sig. Baseline Midline Endline BL-EL Diff. Sig. Letter Sound Knowledge 15.1 37.4 31.2 16.1 *** 17.4 41.6 36.4 19.0 *** Syllable Fluency 19.1 35.2 26.9 7.8 *** 22.7 37.7 32.5 9.7 *** Invented Word Decoding 9.4 15.2 11.3 2.0 * 11.2 16.9 13.7 2.5 ** Passage Reading 12.4 23.3 17.0 4.6 *** 14.7 25.7 21.3 6.6 *** Reading Comprehension 1.0 2.0 1.3 0.4 ** 1.1 2.1 1.7 0.5 *** Listening Comprehension 1.9 2.9 3.1 1.2 *** 1.9 2.7 3.1 1.2 *** *** indicates p<0.01, ** indicates p<0.05, and * indicates p<0.10. Any discrepancies between reported figures and BL-EL differences are due to rounding. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 81 ANNEX II: SCHOOL SAMPLING METHODOLOGY28 Through discussions with USAID, MOE, and RTI, the MSI evaluation team designed and implemented a sampling process during the 2015 baseline to determine the appropriate sample size and select the schools for the baseline. The objective was to produce a sample that would be nationally representative. The process involved six steps: • Step 1: Define the sampling frame using lists of public and APBET institutions • Step 2: Develop a set of design parameters to determine the sample size • Step 3: Enter the parameters into sampling software to calculate the sample size • Step 4: Select a nationally representative sample of schools equal to the sample size • Step 5: Check on the feasibility of the sample and verify the schools in the field • Step 6: Replace a limited number of schools (if needed) and finalize the sample The sampling frameworks, which were provided by RTI, included 22,154 public schools and 1,000 APBET (Alternative Provision of Basic Education and Training) schools. There was information on school name, administrative units (county, sub-county, and zone), school code, and number of pupils in class 1. It is important to ensure that the study is sufficiently powered to detect effects. In determining whether the statistical power is sufficient for the study, it is most critical to randomize an adequate number of groups (e.g., schools) – much more so than the number of individuals per group (e.g., pupils).29 Values for several parameters (listed below) were assumed to reach a level of minimum detectable effects (MDE) for the study. The MDE is the smallest true effect that has a good chance of having statistical significance. MDE is typically defined as the effect that has 80 percent power for a two-tailed test of statistical significance of 0.05 (alpha level) for all comparisons. A typical MDE target is 0.20 for randomized groups with approximately 10 to 15 individuals per group. MSI’s parameters below were set using typical values for statistical power and statistical significance, along with the number of counties that would be reasonable to reach within the time and resource constraints of the revised baseline. The design parameters were as follows: 1. Representative set of counties (K = 24 out of 47 total) 2. Number of pupils per class per school (n = 12) 3. Statistical power set to 0.80 4. Alpha (statistical significance) level set to 0.05 5. Intra-class correlation (rho) set at 0.23 (from the RTI PRIMR pilot results) Based on these design parameters, the MSI statistician used Optimal Design software to calculate the number of schools for the sample. They found that an average of 8.5 schools for each of the 24 clusters (counties) would result in an MDE = 0.20. This led to a total sample size of 204 schools in Kenya for the EGRA baseline, i.e., 8.5 x 24 = 204 schools, with 12 pupils per class per school. Out of the 204 schools, 174 were public schools and 30 were APBET institutions. Based on a desire for more representation in 28 Section adapted Freudenberger, E., & Davis, J. (2017). Tusome external evaluation—Midline report. Washington, DC: Management Sciences International, a Tetra Tech company. http://pdf. usaid.gov/pdf_docs/PA00MS6J.pdf. 29 Bloom, H. (2007). Sample design for group-randomized trials. Prepared for the U.S. Institute of Educational Sciences/National Center for Educational Research (IES/NCER) Summer Research Training Institute. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 82 some of the former provinces, MSI increased the number of counties (K = 26) for an average of 7.85 schools per county. Using a three-stage cluster sampling procedure with the frameworks, MSI drew random samples. The 204 schools were selected proportionally from each of the sampled counties, with independent samples for public and APBET institutions based on their respective sampling frames. School-level samples were 24 pupils, with 12 (6 boys and 6 girls) in each of Classes 1 and 2. The sampling plan resulted in a target of 4,896 total pupils with 2,448 boys and 2,448 girls, along with two teachers and the Head Teacher from each school. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 83 ANNEX III: EVALUATION STATEMENT OF WORK STATEMENT OF WORK Endline Performance Evaluation Project Name: Tusome (“Let’s Read” in Kiswahili) RFP/CA Number: AID-615-C-14-00007 Future Project COR/AOR: Lilian Gangla Life of Project: August 1, 2014 through December 31, 2019 Estimated Total Project Funding: $73.8M A.1. BACKGROUND The USAID/Kenya and East Africa (KEA) Office of Education and Youth (EDY), in partnership with the Kenya Ministry of Education (MoE), is implementing an $88.8 million basic education initiative to improve the reading skills of approximately 7.8 million individual Kenyan children who began primary school during the 2015-2019 school years30. The project, Tusome (Too-SOH-meh; “Let’s Read” in Kiswahili), will continue through December 2019, and has integrated options for transition to government ownership during the out years. Tusome builds on research-based reading initiatives to create a sustainable and affordable national reading program in Kenya. It seeks to improve the English and Kiswahili skills of Standard 1, 2 and 3 children in approximately 24,160 formal and non-formal31 public and low-cost private primary schools across the country. Tusome also assists the Government of Kenya (GOK) at the technical and policy levels to sustainably improve reading skills beyond the life of the project. To this end, the Tusome award works to build the capacity of the GOK, specifically the MoE and relevant Semi-Autonomous Government Agencies (SAGAs) to implement literacy activities. USAID/KEA anticipates that in year four, Tusome will transition to GOK implementation of activities. Two Tusome modalities of note are: the Tusome Partnership Fund (to leverage private sector support, particularly in the development, production, and distribution of materials) and the Tusome Youth Fund (to support youth engagement in improving literacy in the early grades). A.2. STATEMENT OF WORK A.2.1. Purpose of the Evaluation Tusome seeks to improve children’s reading skills on a nationwide scale through evidence-based programming. In line with USAID policy, the testing of innovative activities is built into the Tusome design and the previous Primary Math and Reading (PRIMR) initiative, which developed and tested the methodology that lies at the heart of Tusome. This evaluation serves to: 30 Approximately 1.3 – 1.4 million children enter Standard 1 each year. 31 Non-formal schools – or “low-cost private community schools” – are the predominant form of schooling for children in Kenya’s informal urban settlements (“slums”). The 2011 UWEZO report estimates 20 percent of Kenya’s school-age children attend private schools, with rates much higher in slums. These community private schools charge a nominal fee to educate thousands of underserved and poor children who would not otherwise have access to schooling despite a national free primary education policy. Tusome is working, at GOK request, in 1,500 of these schools, in addition to 23,600 public primary schools across all 47 counties. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 84 1) highlight the achievements of Tusome project goals; 2) highlight the program’s impact on Kenya’s education policies; and 3) document lessons learned during its implementation. This will inform USAID and the GOK on designing and implementing sustainable education programming, in support of Kenya’s journey to self-reliance - https://selfreliance.usaid.gov/ The specific evaluation objectives are to: 1. Document achievements of Tusome during the project period by comparing baseline and endline results. 2. Assess the effectiveness of the design and implementation of Tusome. 3. Document lessons learned and recommendations for future implementation. The audience for the evaluation is: a) USAID KEA Mission, USAID Washington, and other USAID education offices around the world; b) GOK, including MoE, relevant SAGAs32, the Teachers Service Commission, and county governments; c) education sector development partners, the private sector, international researchers, and other stakeholders in and outside of Kenya; and d) the Kenyan public. This will be a whole-of-project performance evaluation. The evaluation design must therefore meet the standard evaluation criteria. A mix of qualitative and quantitative methods applied in a systematic and structured way will be employed. The endline evaluation shall be designed in accordance with Agency guidance (Automated Directives System (ADS) 201 - https://www.usaid.gov/sites/default/files/documents/1870/201.pdf evaluation policy). Key Evaluation Questions The evaluation questions for this final evaluation are as follows: 1. What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? 2. What are the levels of grade 1 and 2 pupils on reading subtasks? 3. What school-level and institutional factors influence reading outcomes when implementing at scale, and how? 4. How effective was implementation of Tusome with regard to: a. teacher buy-in and implementation; b. community awareness and engagement in reading through Youth Fund grants; c. government ownership and buy-in? 5. What are the policy implications of Tusome? a. What policies have been established as a result of or in support of Tusome, e.g. benchmarks, textbook policies? b. Which existing policies are critical for the long-term sustainability of the program? c. Are there additional policies or guidelines that would further enhance Tusome’s sustainability?) A.2.2. Dissemination and Utilization Plan It is expected that the findings will be used to inform program implementation for early grade reading activities in and outside Kenya. The initial draft findings will be shared with the USAID/KEA EDY Office 32 SAGAs include the Kenya Institute of Curriculum Development (KICD), the Kenya National Examinations Council (KNEC), the Kenya Institute for Special Education (KISE), and the Kenya Education Management Institute (KEMI). USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 85 and the implementing partner for necessary review. After the report is finalized, the findings will be shared with relevant offices in USAID/KEA and USAID/Washington, as well as relevant education sector development partners and other stakeholders, including the private sector (e.g., publishing houses and media). For the GoK, the findings will be shared through national and county fora (several counties to be clustered into ten regions), mainly targeting the MoE and related SAGAs. Findings of the Early Grade Reading Assessments (EGRAs) will also be shared with the schools via county fora participants. The deliverables must include presentations to senior Ministry officials together with various presentations to other key stakeholders. In addition to the detailed report, a two page report will be required that can be presented to other key stakeholders such as head teachers, teachers, county officials, parents and School Boards of Management. The short reports are geared towards communication outreach and information sharing as opposed to the detailed report which will be vital for sector decision making. The evaluators will develop reporting mechanisms in consultation with USAID to ensure the needs of each stakeholder group are met. For example, a series of forums will be conducted at national and county level to disseminate findings to GoK stakeholders. The results of the EGRA will be shared with schools via fora participants; and reports will be prepared for USAID in a range of media. A.2.3. Identification of Intervention(s) to Be Examined Early Grade Reading Assessments (EGRAs) The Tusome endline evaluation must primarily focus on utilizing the EGRA tool in Kiswahili and English to measure students’ reading fluency and comprehension. External evaluations have measured “improved reading skills” using the guidance from USAID’s 2011 Education Strategy Technical Notes, revised April 201233, which defines them as increases in fluency and comprehension in reading grade level text (at grade 2). Fluency is the ability to read text accurately, quickly, and with good expression and is calculated based on words correct per minute read; while comprehension is understanding the meaning of what has been read. The recommended indicator to measure reading with comprehension is based on the point at which words correct per minute (wcpm) produces 80 percent reading comprehension. USAID seeks data related to the standard indicator, "percent of learners who demonstrate reading fluency and comprehension of grade level text at the end of grade 2 with U.S. government assistance." A.2.4. Evaluation Design The overall design for this evaluation will be a before-and-after performance evaluation. This approach entails a comparison of baseline and endline reading performance across a panel of 204 schools.34 Project impact will be theorized, but not proven, by comparing project indicators and early grade reading outcomes between and within each group. This approach will allow researchers to determine the extent of project implementation over the course of five years, detect and compare differential changes in early grade reading indicators. The evaluation design described below is based on the 33 Education Strategy Technical Notes: http://pdf.usaid.gov/pdf_docs/PDACT681.pdf; offerors may also want to review the Education Strategy Implementation Guidance: http://transition.usaid.gov/our_work/education_and_universities/pdfs/2012/ED_implementation_guidance_2011.pdf 34 Experimental or quasi-experimental methods are not possible at this time because the intervention was implemented nationwide in 2014 and thus will not include control schools. The 204 schools will form a panel across baseline, midline, and endline however the classroom and pupil sampling will be cross-sectional. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 86 assumption that the Tusome Project is similar in design to the PRIMR initiative. The report will be submitted in English. Methodology Endline groups will be drawn from the same schools and in the same manner and numbers as the baseline and midline groups. Reading fluencies and comprehension levels at endline will be compared to baseline to determine if any significant change has occurred (Research questions 1 and 2). School-level contextual data, classroom instructional practices, and Curriculum Support Officers (CSOs) and education official interviews at the endline will be compared to those at the baseline to document project implementation and account for any changes in enrollment, staffing, facilities, instructional resources, or relevant policies and procedures (Research questions 3, 4, and 5). Note on data collection time periods: Data collection should be completed by October 15, 2019. Data Collection Methods Seven different data collection methods will be used throughout this evaluation. Except for the desk review, each is designed to collect data from individual children, education officials, the school context, or the community context. The same set of EGRA that were used for the baseline and midline will be used for the endline survey. Table 1: Data collection methods for the eight evaluation questions Data Collection Methods Evaluation Questions Desk Review 1, 2, 3, 4, 5 Early Grade Reading Assessment (EGRA) 1, 2 Classroom Observation 1, 2, 3 Head Teacher Interview 3, 4, 5 School Data Protocol 3, 4, 5 Curriculum Support Offices/County/National Education Official35 Interview 3, 4, 5 Data Analysis Methods A number of analysis methods will be employed to check the validity and consistency of the data collected, describe the population from whom data was collected, examine the influence of certain variables on others, and track changes in variables over time. All methods will include at a minimum disaggregation by gender and geography. 35 This will also include ministers where possible. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 87 Table 2: Data analysis methods for the five evaluation questions Data Analysis Methods Evaluation Questions Pearson Correlation/Cronbach’s alpha 1, 2 Frequency distributions/Cross-tabulations 1, 2, 3, 4, 5 Descriptive statistics 1, 2, 3, 4, 5 Multivariate analysis 1, 2, 3, 4, 5 Trend analysis 1, 2, 3, 4, 5 A.2.5. Baseline Data Required An EGRA at the nationwide sample of schools used in the Tusome baseline evaluation, plus associated contextual information, is required. The Tusome endline evaluation must be appropriately linked to the Tusome baseline. A.2.6. Operating Considerations Because the implementation area is nationwide, the contractor may face security restrictions when seeking to collect endline information in certain areas, including the northeastern counties of Garissa, Mandera, and Wajir. Lead time will also be necessary in acquiring the research permit and other required documentation prior to start of the actual study. The Contractor must adequately plan and budget for these considerations. A.2.7. Participation GOK staff will be incorporated into the evaluation in collaboration with USAID to ensure skill and capacity transfer. The evaluation team must coordinate and work closely with the MoE Tusome Technical Team, Kenya National Examination Council and Kenya National Bureau of Statistics. The government officers from these departments will also participate in the data collection training, validation of tools and results, and dissemination. The contractor must propose any level of effort and cost implications of involving GOK staff. USAID will participate in this evaluation as observers. B.1 PERIOD AND PLACE OF PERFORMANCE This evaluation will take place in 204 primary schools across Kenya. The period of performance is May 2019 to June 2020. However, the field work must be completed by October 15, 2019 due to academic testing at school year end which will impede data collection. The contractor must obtain the research permit and other required documentation prior to start of the actual study. C.1 DELIVERABLES In addition to the evaluation report, requirements for which are outlined in the USAID ADS 201, all de￾identified raw data collected by the evaluation must be provided to USAID including the analytic code. This data shall be in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 88 Final, clean, documented data must be transmitted to the DDL in line with contractual requirements. Deliverable Date 1. Evaluation design report and work plan submitted to USAID June 7, 2019 2. Data collection tools including Intermediate Result (IR2) measurement tool submitted for stakeholder comments July 19, 2019 3. Revision and finalization of tools August 27, 2019 4. Weekly reports during data collection period Weekly, submitted on Fridays, after data collection begins. 5. A half-day preliminary results presentation and validation session (virtual) with partners including MOE and RTI November 22, 2019 6. Submission of draft evaluation report to USAID December 13, 2019 7. Revised draft report. 12 days after receiving USAID’s and stakeholders’ comments 8. Submission of Final Report, two-page summary of findings, and policy briefs 12 days after receiving final round of comments 9. Development Experience Clearinghouse (DEC) submission 5 days after approval of final report by USAID. 10. Submission of all raw data and electronic copies of all background documents on Compact Discs: 5 days after #8 11. Dissemination of evaluation findings to USAID and mission staff Within 30 days after approval of final report by USAID 12. Dissemination of evaluation findings and policy briefs to national government, private sector, other development partners, and county government fora Within 30 days after approval of final report by USAID D.1 KEY PERSONNEL The offeror shall propose the most effective team composition based on the proposed methodology. All team members must have relevant prior experience in Sub-Saharan Africa, familiarity with USAID’s objectives, approaches, and operations and prior evaluation/assessment experience. In addition, individual team members should have the technical qualifications identified for their respective positions to effectively conduct an education evaluation of reading outcomes. The following are the required key personnel: Evaluation Team Lead (TL): The TL is ultimately responsible for the overall management of the evaluation team and the final products. In addition, the TL is responsible for coordinating evaluation activities and ensuring the production and completion of all deliverables in conformance with this scope USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 89 of work and timelines. The TL will ensure data integrity, high quality analysis, written reports and report integration. S/he is also responsible for quality assurance and timeliness of all contract deliverables. S/he is responsible for the writing and finalizing of the final evaluation report. All team members report to the Team Leader. Required Qualifications: ● A minimum of a Master’s degree in evaluation, international development or a related technical field; preferably that includes coursework in qualitative and quantitative monitoring and evaluation approaches and methods or other technical training in M&E methods. The team leader should be a reading specialist and have experience in conducting EGRAs. ● At least five years of experience with leading the design and management of evaluations. ● Experience in leading evaluations in Sub-Saharan Africa. Experience working in Kenya is an added advantage. ● Labor Category: Monitoring and Evaluation or Research Specialist (Senior Expat) Local Consultant or Technical Expert: Together with the Team Leader, will finalize the evaluation methodology; develop the data collection strategy, instruments, and protocols; direct data collection and compilation; and conduct data analysis. Required Qualifications: ● A minimum of a Master’s degree in Education, Curricular Development, Policy Development or a related field. ● Should be a reading specialist and have experience in conducting EGRAs. ● Over five years of working experience in Education with specialist knowledge on assessing reading skills. ● Good understanding of working in the Kenyan Education System, with knowledge of working in the primary education sector in Kenya; ● Experience working in an evaluation team. ● Labor Category: Education Specialist (Senior Local) USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 90 ANNEX IV: EVALUATION DESIGN MATRIX # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods 1 What proportion of students demonstrate they can read and comprehend grade-level text (within Kenya’s curricular goals) by the end of grades 1 and 2? Pupils Random sample of grade 1 and 2 pupils in panel of 204 schools selected at baseline Early Grade Reading Assessment (EGRA) – oral reading fluency subtest Summary statistics on the proportion of children falling into each of the four reading benchmark categories (fluent reader, emergent reader, beginning reader, and zero reader) at baseline and endline, disaggregated by test language, grade, sex, and school type; significance of the overall relationship between benchmark category and baseline/endline status will be tested using chi-square 2 What are the levels of grade 1 and 2 pupils on reading subtasks? Pupils Random sample of grade 1 and 2 pupils in panel of 204 schools selected at baseline Early Grade Reading Assessment (EGRA) – all subtests Pre/post-test using multiple regression analysis— controlling for time invariant school characteristics—to assess the extent to which performance on all EGRA subtests has improved since baseline, disaggregated by grade and sex; findings will be reported through cross￾tabulations and data visualizations 3 What school￾level and institutional factors influence reading outcomes when implementing at scale, and how? Pupils Random sample of grade 1 and 2 pupils in panel of 204 schools selected at baseline Pupil survey Multiple regression analysis to identify pupil, teacher, classroom, and school characteristics that are associated with oral reading fluency and “zero reader” status at endline, disaggregated by grade; findings will be reported through cross-tabulations and data visualizations EGRA – oral reading fluency subtest Teachers Grade 1 and 2 teachers in panel of schools selected at baseline Teacher survey Classroom observation USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 91 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods Head Teachers Head Teachers in panel of schools selected at baseline Head Teacher survey 4a How effective was implementation of Tusome with regard to teacher buy-in and implementation? Teachers Grade 1 and 2 teachers in panel of schools selected at baseline Teacher survey Summary statistics on interview responses related to teacher buy-in and implementation; summary statistics on implementation fidelity, as measured by classroom observations Classroom observation Head Teachers Head Teachers in panel of schools selected at baseline Head Teacher survey Curriculum Support Officers (CSOs) CSOs for all schools selected at baseline; former Instructional Coaches (ICs) for APBET schools (where available) CSO survey 4b How effective was implementation of Tusome with regard to community awareness and engagement in reading through Youth Fund grants? Head Teachers All Head Teachers at sampled schools/zones where Youth Fund activities are implemented Semi￾structured interview Qualitative data and debriefing notes will be captured mainly through typed up FGD and KII field notes. Electronic field notes will be imported into Dedoose for first-order and second-order qualitative data analysis. This analysis will then be examined in relation to the research questions, desk review, and related quantitative data to develop key findings USAID implementing partner staff Tusome Grants Fund Manager; 3-4 RTI regional Technical Officers KII Pupils Random sample of grade 1 and 2 pupils in panel of 204 schools selected at baseline Pupil survey Pre- and post-analysis of pupils’ engagement in reading activities outside of a formal classroom setting in schools/communities targeted by grantee activities; comparison of pupils’ reading activities in communities targeted by grantee activities to pupils that were not targeted USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 92 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods Youth Fund Grantees Leaders/representatives from all 23 Youth Fund grant recipient organizations Youth Group leadership web survey Summary statistics on interview responses related to implementation effectiveness, challenges, and long-term sustainability Existing project data Existing Tusome project annual and quarterly reports; existing project evaluations Document review Content analysis to summarize implementing partner results related to research question 4c How effective was implementation of Tusome with regard to government ownership and buy-in? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE; 3- 4 RTI regional Technical Officers KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program implementation fidelity (aggregated at the county-level) KII 5a What official procedures, policies, or Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KII See qualitative data analysis methods for question 4b USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 93 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods guidelines have been established as a result of or in support of Tusome, e.g. benchmarks, textbook policies? KISE, KNEC, and/or MoE Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII Existing project data Existing Tusome project annual, quarterly, and special reports Document review Content analysis to summarize implementing partner results related to research question 5a.i What role did Tusome activities play in the formulation and adoption of such official procedures, policies, or guidelines? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KII USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 94 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods KEMI, KICD, KISE, KNEC, and/or MoE Existing project data Existing Tusome project annual, quarterly, and special reports Document review Content analysis to summarize implementing partner results related to research question 5a.ii To what extent are these official procedures, policies, or guidelines being implemented? What are the catalysts and barriers to effective implementation? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE; 3- 4 RTI regional Technical Officers KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program implementation fidelity (aggregated at the county-level) KII 5a.iii What are the early effects of such official procedures, Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 95 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods policies, or guidelines? Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE; 3- 4 RTI regional Technical Officers KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program implementation fidelity (aggregated at the county-level) KII 5b Which existing procedures, policies, or guidelines are critical for the long-term sustainability of the program? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 96 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE; 3- 4 RTI regional Technical Officers KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program implementation fidelity (aggregated at the county-level) KII 5c Are there additional procedures, policies, or guidelines that would further enhance Tusome’s sustainability? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE; 3- 4 RTI regional Technical Officers KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program KII USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 97 # Evaluation Question Data Source Sampling and Selection Criteria Data Collection Methods Data Analysis Methods implementation fidelity (aggregated at the county-level) Curriculum Support Officers (CSOs) CSOs for all schools selected at baseline; former Instructional Coaches (ICs) for APBET schools (where available) CSO survey Summary statistics on interview responses related to sustainability beyond the life of the activity 5d What other key lessons have been learned throughout the policy-making cycle/process? Government of Kenya Directors and/or technical officers at TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII See qualitative data analysis methods for question 4b Tusome National Technical Team All members of NTT, grouped by administrative level FGD USAID implementing partner staff COP and/or DCOPs; Senior Director, Africa Education; 2-3 policy￾oriented technical staff and/or primary liaison(s) with TSC, KEMI, KICD, KISE, KNEC, and/or MoE KII USAID/KEA Tusome Activity Manager/COR/AOR; outgoing EDY Office Director KII County￾based CDEs and TSC￾CDs 8-10 purposively selected from among 24 sampled counties, based on program implementation fidelity (aggregated at the county-level) KII Curriculum Support Officers (CSOs) CSOs for all schools selected at baseline; former Instructional Coaches (ICs) for APBET schools (where available) CSO survey USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 98 ANNEX V: DATA COLLECTION INSTRUMENTS USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 99 EARLY GRADE READING ASSESSMENT Did the child assent to participate in the study? (1) Yes (0) No (do not continue) A. Basic Information 1. Date of assessment: (a) Day: (b) Month: (c) Year: (d) Start time (HH:MM): : 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code: (f) School name: 3. Class information: (a) School shift: (b) Pupil grade: (1) Full day (1) Class 1 (2) Morning only (2) Class 2 (3) Afternoon only (c) Multigrade class: (d) Stream name: (1) Yes (0) No 4. Pupil information: (a) Unique pupil ID: (b) Pupil age (c) Pupil gender: (0) Male (1) Female General Instructions It is important to read aloud slowly and clearly ONLY the bold sections in the grey boxes. Always record the pupil’s response before moving on to the next instruction/exercise. It is important to establish a playful and relaxed environment with the children to be assessed using simple initial conversation among topics of interest to the pupil. The pupil should perceive the following assessment almost as a game to be enjoyed rather than an exam or severe situation. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 100 Phoneme segmentation (English)  No pupil stimuli Untimed If the pupil stops for 3 SECONDS. If the pupil does not give a correct response in the first 5 words, say “stop”. Remember, when you do the practice, say the “clipped” sounds /p/. DO NOT say “puh”. DO NOT say “pee”.  This is a listening activity. You know that words have sounds. For example, the sounds in the word “pot” are /p/ /o/ /t/. I will say the word twice. Then tell me the sounds in the word. 1. For example, tell me the sounds in the word “too” - “too” If the pupil responds correctly say: Very good, the sounds in the word “too” are /t//oo/ If the pupil does not respond correctly, say: the sounds in the word “too” are /t//oo/. Now it’s your turn. Tell me the sounds in the word “too” – “too”. [Wait 5 seconds for the pupil to respond]. 2. Let’s try another one: Tell me the sounds in the word “fish” - “fish” If the pupil responds correctly say: Very good, the sounds in the word “fish” are /f/ /i/ /sh/ If the pupil does not respond correctly, say: The sounds in the word “fish” are /f/ /i/ /sh/. Now it’s your turn. Tell me the sounds in the word “fish” – “fish”. [Wait 5 seconds for the pupil to respond]. Now I am going to tell you other words. I will say each word two times. Please listen carefully and tell me the sounds in the word. Tell me the sounds in the word “___, ___.” (Read each word twice) 1 in /i/ /n/ 2 see /s/ /ee/ 3 at /a/ /t/ 4 bake /b/ /ei/ /k/ 5 net /n/ /e/ /t/ 6 cup /k/ /u/ /p/ 7 dig /d/ /i/ /g/ 8 may /m/ /ei/ 9 rock /r/ /o/ /k/ 10 lice /l/ /ai/ /s/  Number of correct responses  Check if early stop   USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 101 Letter sound knowledge (English)  English Letter Sounds  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a letter for 3 SECONDS.  If the pupil does not give a single correct response on the first line, say “stop”.  Here is a page full of letters of the English alphabet. Please tell me the SOUNDS of as many letters as you can; not the NAMES of the letters, but the SOUNDS of the letters. 1. For example, the sound of this letter [point to N] is /n/ Now you try: Tell me the sound of this letter [point to a]: If the pupil responds correctly say: Good, the sound of this letter is /a/ If the pupil does not respond correctly, say: The sound of this letter is /a/ 2. Now try another one: Tell me the sound of this letter [point to d]: If the pupil responds correctly say: Good, the sound of this letter is /d/ If the child does not respond correctly, say: The sound of this letter is /d/ When I say “Begin,” please tell me the sound of the letters as best as you can. Tell me the sound of the letters, starting here and continuing this way. [Point to the first letter on the row after the example and draw your finger across the first line and then across the second line until the end]. Use your finger to show me the letters. I will keep quiet and listen to you. Ready? Begin. Examples: N a d 1 2 3 4 5 6 7 8 9 10 S o m i n t H k G e 10 E n h D s u a F R o 20 a b v p T f Z i s K 30 c w s E m N r b A i 40 l U r a g Q e o x M 50 o h L n e a d T g s 60 d A u r o y I E B w 70 R I y K w e t s i z 80 N t a O l C v j E r 90 h e p Z a O s H l t 100 Time remaining at completion (number of SECONDS)  Number of letters read correctly  Check if early stop  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 102 Invented/non-word decoding (English)  English Non-words  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a letter for 3 SECONDS.  If the pupil does not give a single correct response on the first line, say “stop”. Here are some made-up words. I would like you to read as many made-up words as you can. For example, this made-up word is: “gat”. 1. Now you try: Please read this word [point to lim] [If the pupil says “lim”, say]:“Very good: “lim” [If the pupil does not say “lim” correctly say]: This made-up word is “lim”. 2. Now try another one: Please read this word [point to rep] [If the pupil says “rep”, say]:“Very good: “rep” [If the pupil does not say “rep” correctly say]: This made-up word is “rep.” When I say “begin,” read the words as best as you can. Read the words by pointing with your finger, starting here and continuing this way. [Point to the first word on the row after the example and draw your finger across the first line]. I will keep quiet and listen to you. Ready? Begin. Examples: gat lim rep 1 2 3 4 5 mip pog bem jeb lal 5 laj het cur fov wim 10 reg gux wis yut roz 15 jol mof pim zin pug 20 mak heg zay tep gat 25 fik sab fem jif pef 30 reb pos daf lep wog 35 dix dap ruk vob mep 40 raz yot vap kom zil 45 mal bis lop pab kar 50 Time remaining at completion (number of SECONDS)  Number of words read correctly  Check if early stop  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 103 Vocabulary (English)  No pupil stimuli  Untimed  If the pupil stops for 3 SECONDS. Materials: a sheet of paper, pencil, rubber, desk  A. Body Parts: I'll say words in English. Show me what part of your body the word means. Let’s practice. “nose” [Point to your nose so that you model for the pupil] “head” [Wait for the pupil to gesture to his/her head]. Let’s start. 1 Arm Correct Incorrect No Response 2 Back Correct Incorrect No Response 3 Shoulder Correct Incorrect No Response 4 Chin Correct Incorrect No Response 5 Mouth Correct Incorrect No Response 6 Foot Correct Incorrect No Response 7 Knee Correct Incorrect No Response 8 Elbow Correct Incorrect No Response  B. Words from the environment: Now I will say other words and you will show me the object. 9 Desk Correct Incorrect No Response 10 Paper Correct Incorrect No Response 11 Pencil Correct Incorrect No Response 12 ground (floor) Correct Incorrect No Response 13 Shoes Correct Incorrect No Response 14 Rubber Correct Incorrect No Response USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 104  C. Spatial words: Place a pencil and sheet of paper side by side in front of the pupil. Take this pencil. [Hand the pencil to the pupil.] This is paper. [Point to the piece of paper] You will place the pencil where I tell you to put it. Put the pencil… 15 under the paper Correct Incorrect No Response 16 on the paper Correct Incorrect No Response 17 next to the paper Correct Incorrect No Response 18 On top of your head Correct Incorrect No Response 19 in front of your head Correct Incorrect No Response 20 behind your head Correct Incorrect No Response  Number of correct responses USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 105 Reading passage and comprehension - A (English) English Reading Passage A  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a word for 3 SECONDS.  If the pupil does not read a single word correctly on the first line, say “stop”. Here is a short story. I want you to read it aloud. When you finish, I will ask you some questions about what you have read. When I say “begin,” read the story as best as you can. Read the story by pointing with your finger, starting here and continuing this way. [Point to the first word on the row and draw your finger across the first line and continuing to the next line, etc.]. Ready? Begin. REMOVE the stimuli from in front of the pupil. Read the instructions to the pupil. Then read each question slowly and clearly. After you read each question, give the pupil 10 seconds to answer the question, mark the pupil’s response as correct or incorrect, and move to the next question. Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can. STORY: QUESTIONS ANSWERS Ben has a tin with a red lid. 8 What is the colour of the lid? [red] Correct Incorrect No Response He puts milk in the tin. Ben keeps the tin on the table. 21 What is in the tin? [milk] Correct Incorrect No Response He wants to take the milk to school tomorrow. 30 When will he take the milk to school? [tomorrow] Correct Incorrect No Response Ben wakes up to go to school. But the tin is not on the table. 45 What is not on the table? [the tin/the milk] Correct Incorrect No Response Then he sees his cat under the bed. 53 Where is the cat? [Under the bed] Correct Incorrect No Response The tin is near the cat. It has no lid and no milk. Ben is very sad. 70 Where did the milk go? [the cat drunk it/it poured] Correct Incorrect No Response Time remaining at completion (number of SECONDS)  Number of words read correctly  Check if early stop   Number of correct answers USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 106 Reading passage and comprehension - B (English) English Reading Passage B  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a word for 3 SECONDS.  If the pupil does not read a single word correctly on the first line, say “stop”. Here is a short story. I want you to read it aloud. When you finish, I will ask you some questions about what you have read. When I say “begin,” read the story as best as you can. Read the story by pointing with your finger, starting here and continuing this way. [Point to the first word on the row and draw your finger across the first line and continuing to the next line, etc.]. Ready? Begin. Start the timer to give the student one minute to read the passage again. Now I will give you 1 minute to read the story again as many times as you want. I will tell you when to stop. Ready? Begin. When the timer reaches 0, stop the student. Read the instructions to the pupil. Then read each question slowly and clearly. After you read each question, give the pupil 15 seconds to answer the question, mark the pupil’s response as correct or incorrect, and move to the next question. Ask the pupil all 6 questions.  Now I am going to ask you a few questions about the story you just read. Try to answer the questions as well as you can. You can use the text to find the answer if you need to. STORY: Tim, Linda and their Pets QUESTIONS ANSWERS Tim and Linda live in a big hut. They have pets. 11 Where do Tim and Linda live? [hut/big hut] Correct Incorrect No Response Tim has a fat dog. Linda has a black cat. 21 Who has a dog? [Tim] Correct Incorrect No Response Linda likes to play with Tim. One day they go to the lake with their pets. 37 Where do Linda and Tim go? [lake] Correct Incorrect No Response But the dog wants to bite the cat. 45 What does the dog want to bite? [cat] Correct Incorrect No Response The cat runs away. Linda is sad. Tim takes the dog home. Linda and the cat are happy. 63 Where does Tim take the dog? [home] Correct Incorrect No Response They play a game near the lake. 70 Who plays a game at the lake? [Linda and the cat] Correct Incorrect No Response Time remaining at completion (number of SECONDS)  Number of words read correctly  Check if early stop   Number of correct answers USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 107 Letter sound knowledge (Kiswahili)  Kiswahili Letter sounds  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a letter for 3 SECONDS.  If the pupil does not give a single correct response on the first line, say “stop”.  Karatasi hii ina herufi mbali mbali za Kiswahili. Tafadhali zitamke sauti za herufi zote unazozijua; tamka SAUTI za herufi lakini sio MAJINA. Kwa mfano, sauti ya herufi hii [kisha mwonyeshe herufi o] ni /o/ 1. Hebu tufanye mazoezi: Nitamkie sauti ya herufi hii [mwonyeshe herufi N]: Iwapo jawabu la mwanafunzi ni sahihi, sema : Vyema, sauti ya herufi hii ni /n/ Iwapo jawabu la mwanafunzi sio sahihi, sema: Sauti ya herufi hii ni /n/ 2. Sasa, hebu jaribu sauti nyingine za herufi: Hebu nitamkie sauti ya herufi hii [mwonyeshe herufi m]: Iwapo jawabu la mwanafunzi ni sahihi, sema: Vyema, sauti ya herufi hii ni /m/ Iwapo jawabu la mwanafunzi sio sahihi, sema: Sauti ya herufi hii ni /m/ Nikisema “Anza”, tafadhali zitamke sauti za herufi hizi haraka iwezekanavyo lakini kwa makini. Nitamkie sauti za herufi, kuanzia hapa kisha kuendelea hivi. [Elekeza kidole chako katika herufi ya kwanza katika mstari wa juu baada ya mfano kisha uendelee hadi mwisho wa mstari huo]. Tumia kidole chako kunionyesha herufi. Nitanyamaza nikusikilize. Uko tayari? Anza. Mifano: o N m 1 2 3 4 5 6 7 8 9 10 A f ny p T o D u b i 10 sh i R u h a V p t n 20 E U a n ch L K m I A 30 B n o I k R A gh z u 40 k z V dh n u i e s a 50 u O i e a gh z n w i 60 i W u k p n S ng' a ch 70 y p g M i A u O N f 80 t a h S M I N th d K 90 n i y A G m a e gh I 100 Time remaining at completion (number of SECONDS)  Number of letters read correctly  Check if early stop  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 108 Syllable Fluency (Kiswahili)  Kiswahili Syllables  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a letter for 3 SECONDS.  If the pupil does not give a single correct response on the first line, say “stop”. Karatasi hii ina silabi mbali mbali za Kiswahili. Tafadhali zitamke silabi zote unazozijua. Kwa mfano, silabi hii [kisha mwonyeshe silabi ya] ni “ya” 1. Hebu tufanye mazoezi: Nitamkie silabi hii [mwonyeshe silabi “si”]: Iwapo jawabu la mwanafunzi ni sahihi, sema : Vyema, silabi hii ni “si” Iwapo jawabu la mwanafunzi sio sahihi, sema: Silabi hii ni “si” 2. Sasa, hebu jaribu silabi nyingine: Hebu nitamkie silabi hii [mwonyeshe silabi “fu”]: Iwapo jawabu la mwanafunzi ni sahihi, sema: Vyema, silabi hii ni “fu” Iwapo jawabu la mwanafunzi sio sahihi, sema: Silabi hii ni “fu” Nikisema “Anza”, tafadhali zitamke silabi hizi haraka iwezekanavyo lakini kwa makini. Nitamkie silabi, kuanzia hapa kisha kuendelea hivi. [Elekeza kidole chako katika silabi ya kwanza katika mstari wa juu baada ya mfano kisha uendelee hadi mwisho wa mstari huo]. Tumia kidole chako kunionyesha silabi. Nitanyamaza nikusikilize. Uko tayari? Anza. Mifano: ya si fu 1 2 3 4 5 6 7 8 9 10 le fi mwe se ye nde ni fa zi ha 10 ja he nyu ba zo cho bu dho yu ngu 20 thi ii gu je wa pe ki nya pi go 30 fu nda shi ga ri sha au be to nga 40 na de ma di la vi ra zu we mo 50 do za ne bwa bi ho ngi ku su he 60 mba ti po wi mwa re la so mi du 70 nu ko li sa no ya si pa nzi che 80 sha ju ke da vu nye me te o ndi 90 tu yo shi mu cho ji wu hi ru ka 100 Time remaining at completion (number of SECONDS)  Number of syllables read correctly  Check if early stop  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 109 Invented/non-word decoding (Kiswahili)  Kiswahili Non-words  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a letter for 3 SECONDS.  If the pupil does not give a single correct response on the first line, say “stop”. Karatasi hii ina maneno yaliyobuniwa. Ningependa usome maneno yote unayoweza. Kwa mfano, neno hili la kubuni ni: “ buza” 1. Hebu fanya mazoezi: Tafadhali lisome neno hili [mwonyeshe neno “zefu”] [Iwapo mwanafunzi atasema “zefu ”, mwambie]: “Vizuri sana : “zefu” [Iwapo mwanafunzi hakusoma neno “zefu” vizuri, mwambie]: Neno hili la kubuni ni “zefu.” 2. Sasa, hebu jaribu neno lingine: Tafadhali soma hili neno mwonyeshe neno: “sharu”. [Iwapo mwanafunzi atasema “sharu ”, mwambie]: “Vizuri sana : “sharu” [Iwapo mwanafunzi hakusoma neno “sharu” vizuri, mwambie]: Neno hili la kubuni ni “sharu.” Nikisema “Anza”, yasome maneno haya haraka iwezekenavyo lakini kwa makini. Yasome maneno ukifuatiliza kwa kidole kuanzia hapa kisha kuendelea hivi. [Elekeza kidole chako katika neno la kwanza kwenye mstari wa juu baada ya mfano kisha uendelee hadi mwisho wa mstari huo]. Nitanyamaza nikusikilize. Uko tayari? Anza. Mifano: buza zefu sharu 1 2 3 4 5 kabe regu toko kine leye 5 mapa mtozo vicha kengu sharu 10 chuso nyuza gazu nziki mwela 15 ndise kenzi mtofi bwara ndaho 20 josa yota kuvi ngiso sine 25 ngute honzi vube gowe ndami 30 rubwa fipe riki nepu howe 35 ripi hungu mwate nzinga zefu 40 dusu msino chena rime mbeta 45 hefa choyu shifi ndweku vili 50 Time remaining at completion (number of SECONDS)  Number of words read correctly  Check if early stop  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 110 Reading passage and comprehension (Kiswahili) Kiswahili Reading passage  60 seconds  When timer reaches 0, say, “stop.”  If the pupil stops on a word for 3 SECONDS.  If the pupil does not read a single word correctly on the first line, say “stop”. Hii hapa ni hadithi fupi. Ningependa uisome kwa sauti. Ukimaliza kuisoma, nitakuuliza maswali kuhusu yale uliyosoma. Nikisema “Anza”, isome hadithi vizuri kadri ya uwezo wako. Isome hadithi ukifuatiliza kwa kidole chako kuanzia hapa kisha kuendelea hivi. [Elekeza kidole chako katika neno la kwanza kwenye mstari wa juu kisha uendelee hadi mstari unaofuata, na kadhalika]. Uko tayari? Anza. IONDOE hadithi kutoka mbele ya mwanafunzi. Soma maagizo kwa mwanafunzi. Kisha soma kila swali pole pole na kwa makini. Baada ya kusoma kila swali, mpe mwanafunzi hadi sekunde 10 alijibu swali; tia alama mwafaka kulingana na jibu lake, halafu uendelee katika swali linalofuata.  Sasa nitakuuliza maswali machache kuhusu hadithi uliyosoma. Jaribu kujibu maswali uwezavyo. STORY: QUESTIONS MAJIBU Sungura alikuwa mjanja sana. Alipenda kuwasumbua wanyama wengine. 8 Ni nani alikuwa mjanja? [sungura] Sahihi Kosa Hakujibu Siku moja alikula chakula cha kobe. Kisha akakimbia akapanda mti. 18 Chakula kilikuwa cha nani? [kobe] Sahihi Kosa Hakujibu Naye kobe akasimama chini ya mti akamwambia sungura ashuke. 27 Kobe alitaka sungura afanye nini? [ashuke] Sahihi Kosa Hakujibu Sungura akasema, ‘ukitaka nishuke kusanya mchanga chini ya mti. Nitaruka juu ya mchanga ili nisiumie’. 42 Sungura alitaka kuruka juu ya nini? [mchanga] Sahihi Kosa Hakujibu Kobe akakusanya mchanga. Halafu sungura akamwambia, ‘fungua macho sana uniangalie nikiruka ili nisitoroke’. 55 Ni nani aliyekusanya mchanga? [Kobe] Sahihi Kosa Hakujibu Sungura akaruka juu ya mchanga, akamrushia kobe mchanga machoni. Kobe hakumwona Sungura akitoroka. 68 Kwa nini kobe hakumwona sungura akirotoka? [alikuwa na mchanga machoni/mchanga machoni] Sahihi Kosa Hakujibu Time remaining at completion (number of SECONDS)  Number of words read correctly  Check if early stop   Number of correct answers USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 111 Listening Comprehension (Kiswahili)  No pupil stimuli Untimed  Nitasoma hadithi fupi kwa sauti. Nitaisoma mara moja tu. Halafu nitakuuliza maswali. Tafadhali sikiliza kwa makini kisha ujaribu kujibu maswali uwezavyo.  Shule ya Mikindani ina wanafunzi wengi. Pia ina shamba kubwa la matunda. Lakini shule hii haina uwanja wa kutosha. Mwalimu mkuu amemwomba Chifu na wanakijiji watetengeneze uwanja. Sasa wanafunzi wana furaha sana. Wametunga wimbo kumshukuru mwalimu mkuu. 1 Shambani kuna nini? [matunda] Sahihi Kosa Hakujibu 2 Shule ya Mikindani haina nini? [uwanja/uwanja wa kuchezea] Sahihi Kosa Hakujibu 3 Nani alimwomba chifu kutengeneza uwanja? [mwalimu/mwalimu mkuu] Sahihi Kosa Hakujibu 4 Ni kwa nini wanafunzi wana furaha? [uwanja/ watapata/watatengenezewa uwanja] Sahihi Kosa Hakujibu 5 Nani ametunga wimbo? [wanafunzi/watoto] Sahihi Kosa Hakujibu  Number of correct responses  USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 112 PUPIL INTERVIEW Did the child assent to participate in the study? (1) Yes (0) No (do not continue) B. Basic Information 1. Date of interview: (a) Day: (b) Month: (c) Year: 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code:36 3. Pupil information: (a) Unique pupil ID:1 (b) Pupil age (c) Pupil gender: (0) Male (1) Female 36 All identification numbers will be anonymized for respondent protection/privacy USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 113 B. Pupil Questionnaire Thank you very much. Now, I am going to ask you some questions about you and your reading habits. Asante sana. Sasa nitakuuliza maswali kukuhusu na pia kuhusu mtindo wako wa kusoma. Ask each question verbally to the pupil, as in an interview. Do not read the response options aloud. Wait for the pupil to respond then write this response in the space provided, or check the box of the option that corresponds to the pupil’s response. If there is no special instruction to the contrary, only one response is permitted. 1. What language does your family speak at home? Familia yako huongea lugha gani nyumbani? (Select all that apply) (1) Kiswahili (2) English (3) Gikuyu (4) Dholuo (5) Kalenjin (6) Luhya dialect (7) Somali (8) Kikamba (-888) Other (-999) Don’t know / no answer 2. What language do you speak at school? Wewe huongea lugha gani ukiwa shuleni? (Select all that apply) (1) Kiswahili (2) English (3) Gikuyu (4) Dholuo (5) Kalenjin (6) Luhya dialect (7) Somali (8) Kikamba (-888) Other (-999) Don’t know / no answer 3. Did you go to school before grade 1 (nursery, pre-unit, baby class)? Ulienda shule yoyote kabla ya kuanza darasa la kwanza? (Shule ya chekechea) (1) Yes (0) No (-999) Don’t know / no answer 4. Do you have English books or other English reading materials at your home? Una vitabu vya Kiingereza au nakala zingine za kiingereza zakusoma nyumbani? (1) Yes (0) No (-999) Don’t know / no answer 5. Do you have Kiswahili books or other Kiswahili reading materials at your home? Una vitabu vya Kiswahili au nakala zingine za kiswahili za kusoma nyumbani? (1) Yes (0) No (-999) Don’t know / no answer 5a. Do you have books or other reading materials in any other languages at your home? [Je, una vitabu au nakala nyingine za kusoma katika lugha nyingine nyumbani?…] (1) Yes (0) No (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 114 6. Does anyone read stories aloud to you at your home? Kuna mtu yeyote nyumbani kwenu ambaye hukusomea hadithi kwa sauti? (1) Yes (0) No (-999) Don’t know / no answer 7. Do you read stories at your home? Wewe husoma hadithi nyumbani kwenu? (1) Yes (0) No  skip to 8 (-999) Don’t know / no answer  skip to 8 7a. How often do you read at home? [Wewe husoma mara ngapi nyumbani?…] (1) Every day (2) Some days (3) Rarely (4) Never (-999) Don’t know / no answer 8. Do you ever practice reading aloud to your teacher or to other pupils? Wewe hufanya mazoezi ya kusoma kwa sauti kwa mwalimu au kwa wanafunzi wengine? (1) Yes (0) No (-999) Don’t know / no answer 9. Do you practice silent reading in school? Wewe hufanya mazoezi ya kusoma kimya shuleni? (1) Yes (0) No (-999) Don’t know / no answer 10. Does your teacher assign reading for you to do at your home? Je, mwalimu wako hukupa mazoezi ya kusoma ukiwa nyumbani? (1) Yes (0) No (-999) Don’t know / no answer 10a Have you gone to any reading or literacy events outside of school? [Umewahi kwenda katika hafla yoyote ya usomaji au ya kielimu nje ya shule?…] (1) Yes (0) No (-999) Don’t know / no answer 11. Do you have a lamp at home? Kuna taa nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 12. Is there electricity in your house? Kuna stima nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 13. Do you watch TV at your home? Je, wewe huangalia TV nyumbani kwenu? Je, huma unatazama runinga (TV) nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 14. Do you listen to the radio at your home? Je, wewe husikiliza redio nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 115 15. Is there a mobile phone or telephone at your home? Kuna simu ya mkono (mobile) au simu nyingine nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 16. Is there a computer or laptop or tablet or iPad at your home? Kuna kompyuta (laptop/tablet/ipad) nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 17. Is there a bicycle at your home? Kuna baiskeli nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 18. Is there a motorcycle at your home? Kuna pikipiki nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 19. Do you have many cows or goats or camels or donkeys or sheep at your home, such as more than 10? Kuna ng’ombe au mbuzi au ngamia au punda au kondoo wengi nymbani kwenu kama zadi ya kumi? (1) Yes (0) No (-999) Don’t know / no answer 20. Is there a car or truck or tractor or boat at your home? Kuna gari au lori au trakta au boat/boti nyumbani kwenu? (1) Yes (0) No (-999) Don’t know / no answer 21. Do you usually wear shoes to school? [Je, wewe huvaa viatu unapokwenda shuleni?…] (1) Yes (0) No (-999) Don’t know / no answer 22. Did you usually eat breakfast before coming to school? [Je, wewe hula kiamsha kinywa kabla ya kuja shuleni …] (1) Yes (0) No (-999) Don’t know / no answer 23. How many minutes does it usually take for you to get to school? [Wewe huchukua dakika ngapi kufika shuleni?…] (1) Yes (0) No (-999) Don’t know / no answer 24. How often are you absent from school? (Do not prompt, record relevant answer) [Ni mara ngapi wewe hukosa kufika shuleni?…] (1) I rarely or never miss school (2) I sometimes miss school (but the days I attend are more than the days I miss) (3) I regularly miss school (and the days I miss are more than the days I attend) (4) I rarely come to school (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 116 25. How often are you late to school? (Do not prompt, record relevant answer) [Ni mara ngapi wewe huchelewa kufika shuleni?…] (1) I am rarely or never late (2) I am sometimes late (but the days I am on time are more than the days that I am late) (3) I am regularly late to school (and the days I am late are more than the days I am on time) (4) I rarely come to school on time (-999) Don’t know / no answer 26. What grade were you in last year? [Ulikuwa gredi gani mwaka uliopita?…] (1) Grade 1 (2) Grade 2 (3) Grade 3 (4) Grade 4 or higher (5) Pre-primary (6) I was not in school (-999) Don’t know / no answer Thank you very much for your participation! Asante sana kwa kushiriki kwako! USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 117 TEACHER INTERVIEW Did the teacher consent to participate in the study? (1) Yes (0) No (do not continue) C. Background Information 1. Date of interview: (a) Day: (b) Month: (c) Year: 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code:37 (f) Unique teacher ID:1 3. Class information: (b) School shift: (b) Grade level: (1) Fully day (1) Grade 1 (2) Morning only (2) Grade 2 (3) Afternoon only (c) Boys enrolled: (d) Girls enrolled: 4. Teacher information: (a) Gender: (0) Male (1) Female D. Teacher Questionnaire 1. Which subjects do you teach in this grade? (Select all that apply) (1) Kiswahili activities (2) English activities (3) Literacy (-999) Don’t know / no answer 2. What is your highest professional qualification? (1) Untrained (2) P1 (3) Diploma/S1 (4) Bachelor of Education (5) Masters of Education (-888) Other (specify): ___________ (-999) Don’t know / no answer 37 All identification numbers will be anonymized for respondent protection/privacy USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 118 3. How long have you been teaching? (Enter -999 for don’t know/no answer) years months 3a. How long have you been teaching at this school? (Enter -999 for don’t know/no answer) years months 4. Which grades are you currently teaching? (Select all that apply) (1) Grade 1 (2) Grade 2 (3) Grade 3 (4) Grade 4 (-888) Other 5. Do you teach in multi-grade classes? (1) Yes (0) No (-999) Don’t know / no answer 6. Does your school or classroom have a functioning library? (0) No → skip to B9 (1) Yes, but only in school (2) Yes, but only in classroom (3) Yes, in both (-999) Don’t know / no answer → skip to B9 7. Do your pupils use the library? (1) Yes (0) No (-999) Don’t know / no answer 8. Do your pupils borrow books from the library to take home? (1) Yes (0) No (-999) Don’t know / no answer 9. Do you use books other than course books in your classroom? (1) Yes (0) No (-999) Don’t know / no answer 10. Do you give extra time or remediation to weak/struggling pupils? (1) Yes (0) No (-999) Don’t know / no answer 11. Over the last 12 months, how often has a Head Teacher observed you teaching Kiswahili or English language activities in your classroom? (0) Never → skip to B12 (1) About once a week (2) About once per month (3) About once per term (4) About once per year (-999) Don’t know / no answer → skip to B12 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 119 11b. How helpful were Head Teacher observations in improving your teaching of Kiswahili or English language activities: not helpful, somewhat helpful, helpful, or very helpful? (1) Not helpful (2) Somewhat helpful (3) Helpful (4) Very helpful (-999) Don’t know / no answer 12. Over the last 12 months, how often has a Curriculum Support Officer (CSO) (Instructional Coach for APBET) observed you teaching Kiswahili or English language activities in your classroom? (0) Never → skip to B12b (1) About once a week (2) About once per month (3) About once per term (4) About once per year (-999) Don’t know / no answer → skip to B12b 12a. How helpful were CSO observations in improving your teaching of Kiswahili or English language activities: not helpful, somewhat helpful, helpful, or very helpful? (1) Not helpful (2) Somewhat helpful (3) Helpful (4) Very helpful (-999) Don’t know / no answer 12b. Over the last 12 months, how often has a Quality Assurance and Standards Officer (QASO) observed you teaching Kiswahili or English language activities in your classroom? (0) Never → skip to B13 (1) About once a week (2) About once per month (3) About once per term (4) About once per year (-999) Don’t know / no answer → skip to B13 12c. How helpful were the QASO observations in improving your teaching of Kiswahili or English language activities: not helpful, somewhat helpful, helpful, or very helpful? (1) Not helpful (2) Somewhat helpful (3) Helpful (4) Very helpful (-999) Don’t know / no answer 13. Do you have teacher guides for teaching English language activities? Do you use these guides? (1) Yes, have and use (2) Yes, have but do not use (0) No, do not have (-999) Don’t know / no answer 14. Do you have teacher guides for teaching Kiswahili language activities? Do you use these guides? (1) Yes, have and use (2) Yes, have but do not use (0) No, do not have (-999) Don’t know / no answer 14a. [If B13=2 or B14=2] Why don’t you use the guides you have for teaching English and/or Kiswahili language activities? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 120 14b. Do you have separate teacher guides for teaching literacy? Do you use these guides? (1) Yes, have and use (2) Yes, have but do not use (0) No, do not have (-999) Don’t know / no answer 14c. [If B14b=2] Why don’t you use the guides you have for teaching literacy? 14d. How many lessons per week do you teach English language activities? (Enter -999 for don’t know/no answer) lessons 14e. How many lessons per week do you teach Kiswahili language activities? (Enter -999 for don’t know/no answer) lessons 14f. How many lessons per week do you teach literacy? (Enter -999 for don’t know/no answer) lessons 14g. In your view, how many lessons per week should you teach English language activities? (Enter -999 for don’t know/no answer) lessons 14h. In your view, how many lessons per week should you teach Kiswahili language activities? (Enter -999 for don’t know/no answer) lessons 14i. In your view, how many lessons per week should you teach literacy? (Enter -999 for don’t know/no answer) lessons 15. Which method do you use most often to measure your pupils’ progress during your classroom instruction of reading? (0) No measure used (1) Written assessment (2) Oral assessment (3) Check exercise books (4) Check homework (-888) Other (specify): ___________ (-999) Don’t know / no answer 16. How do you measure pupil achievement in reading at the end of the school year? (0) I do not measure (1) Oral assessment (2) Paper and pencil assessment (-999) Don’t know / no answer 17. Other than Tusome training, how many times did you receive in-service training in the past 2 years? (0) None → skip to B19 (1) 1-2 sessions (2) 3-4 sessions (3) 5-6 sessions (4) More than 6 sessions (-999) Don’t know / no answer → skip to B19 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 121 18. Were you trained on how to teach reading during these non-Tusome training(s)? (1) Yes (0) No (-999) Don’t know / no answer 19. How many Tusome teacher training sessions have you attended so far? (0) None → skip to B22c (1) 1-2 sessions (2) 3-4 sessions (3) 5-6 sessions (4) More than 6 sessions (-999) Don’t know / no answer → skip to B22c 20. Overall, how do you rate the quality of Tusome teacher training: very poor quality, poor quality, moderate quality, high quality, or very high quality? (1) Very poor quality (2) Poor quality (3) Moderate quality (4) High quality (5) Very high quality (-999) Don’t know / no answer 21. How would you rate the relevance of Tusome training: not relevant, somewhat relevant, relevant, or very relevant? (1) Not relevant (2) Somewhat relevant (3) Relevant (4) Very relevant (-999) Don’t know / no answer 22. How would you rate the usefulness of Tusome materials: not useful, somewhat useful, useful, or very useful? (1) Not useful (2) Somewhat useful (3) Useful (4) Very useful (-999) Don’t know / no answer 22a. Do you face any challenges in using the Tusome materials or methods in teaching? (1) Yes (0) No → skip to B22c (-999) Don’t know / no answer → skip t B22c USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 122 22b. What challenges do you face in using the Tusome materials? (Select all that apply) (1) Materials are old/damaged (2) There are not enough materials (3) Lesson pacing is too slow for my pupils (4) Lesson pacing is too fast for my pupils (5) Inadequate time to cover all the material (6) The content does not align with CBC (7) The content does not fit with schemes of work and lesson plans format/templates (8) Lessons are not effective (9) I am instructed not to use them (10) I am not adequately trained on how to use them (-888) Other (specify): ___________ (-999) Don’t know / no answer 22c. Are you currently implementing the Competency Based Curriculum (CBC)? (1) Yes (0) No → skip to B22e.iii (-999) Don’t know / no answer → skip t B22e.iii 22d. Do you face any challenges in implementing the CBC? (1) Yes (0) No → skip to B22e.i (-999) Don’t know / no answer → skip t B22e.i 22e. What challenges are you facing in implementing the CBC? 22e.i Would you change anything about how CBC is being implemented vis-à-vis English and Kiswahili language activities? (1) Yes (0) No → skip to B22e.iii (-999) Don’t know / no answer → skip t B22e.iii 22e.ii What would you change about how CBC is being implemented vis-a-vis English and Kiswahili language activities? 22e.iii Do you have any other suggestions for improving the teaching of English and/or Kiswahili language activities in your school? I will now read a few statements and ask you to state whether you strongly agree, agree, neither agree nor disagree, disagree, or strongly disagree with that statement. For these questions, we want to understand your personal perceptions on teaching reading effectively, even if those perceptions differ from current school practices or requirements. All of your responses are confidential. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 123 22f. When it comes to teaching reading, it is better for teachers to develop their own lessons than to work from structured teachers’ guides (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 22g. When it comes to teaching reading, it is better for teachers to work from structured teachers’ guides than to develop their own lessons (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 22h. Pupils best learn to read new words by visually memorizing them as whole units (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 22i. Pupils best learn to read new words by first understanding the letters and sounds that make up the word (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 22j. Regular observation and coaching by CSOs/ICs and Head Teachers usually helps teachers become more effective in their job (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 22k. Regular observation and coaching by CSOs/ICs and Head Teachers usually distracts teachers from doing their job effectively (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 124 Think about the last 5 days of school and indicate how often each of the following activities took place. Choose only one option per question: Activity / action Days per week 23. The whole class repeated sentences that you said/read first (choral repetition/reading) 0 1 2 3 4 5 24. Pupils copied text from the blackboard or textbook 0 1 2 3 4 5 25. Pupils retold a story that you read to them 0 1 2 3 4 5 26. Pupils retold a story that they read 0 1 2 3 4 5 27. Pupils sounded out unfamiliar words 0 1 2 3 4 5 28. Pupils learned meanings of new words 0 1 2 3 4 5 29. Pupils read aloud to teacher or to other pupils 0 1 2 3 4 5 30. Pupils answered comprehension questions based on the text you read to them 0 1 2 3 4 5 31. Pupils answered comprehension questions based on the text they read 0 1 2 3 4 5 32. Pupils were assigned reading to do on their own during school time 0 1 2 3 4 5 33. Pupils were assigned reading to do on their own at home 0 1 2 3 4 5 In what grade should pupils FIRST BE ABLE TO DEMONSTRATE the following skills? Choose only one option per question: Activity / action Before grade 1 Grade 1 Grade 2 Grade 3 Grade 4 or later 34. Recognize letters and say letter names 0 1 2 3 4 35. Recognize letters and say letter sounds 0 1 2 3 4 36. Sound out unfamiliar words 0 1 2 3 4 37. Read aloud a short passage with few mistakes 0 1 2 3 4 38. Understand stories they read 0 1 2 3 4 Thank you for your participation! You have been very helpful. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 125 HEAD TEACHER INTERVIEW Did the Head Teacher consent to participate in the study? (1) Yes (0) No (do not continue) E. Background Information 1. Date of interview: (a) Day: (b) Month: (c) Year: 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code:38 3. Class information: (c) School shift: (1) Fully day (2) Morning only (3) Afternoon only (b) Total grade 1 enrollments (c) Total grade 2 enrollments 4. Head Teacher information: (b) Gender: (0) Male (1) Female F. Head Teacher Questionnaire 1. What is your position at this school? (1) Head Teacher (2) Deputy Head Teacher (-888) Other (specify): ___________ 2. How long have you been in this position? years months 3. What is your highest professional qualification? (1) Untrained (2) P1 (3) Diploma/S1 (4) Bachelor of Education (5) Masters of Education (-888) Other (specify): ___________ (-999) Don’t know / no answer 38 All identification numbers will be anonymized for respondent protection/privacy USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 126 4. Have you received specialized training or taken courses in school management in the past 12 months? (1) Yes (0) No (-999) Don’t know / no answer 5. Have you received specialized training or taken courses in Kiswahili/English instruction in the past 2 years? (1) Yes (0) No → skip to C1 (-999) Don’t know / no answer → skip to C1 6. Overall, how would you rate the usefulness of training you received in Kiswahili/English instruction: not useful, somewhat useful, useful, or very useful? (1) Not useful (2) Somewhat useful (3) Useful (4) Very useful (-999) Don’t know / no answer G. School Instruction, Curriculum, and Assessment 1. Does your school timetable include periods for Kiswahili or English language activities (grades 1- 3)? (0) No → skip to C2b (1) Yes (2) There is no timetable → skip to C3 (-999) Don’t know / no answer → skip to C2b 2. In the timetable, how many periods in a week are there for teaching Kiswahili language activities? (Enter zero if none; -999 for don’t know/no answer) periods 2a. In the timetable, how many periods in a week are there for teaching English language activities? (Enter zero if none; -999 for don’t know/no answer) periods 2b. Does your school timetable include periods for literacy activities? (0) No → skip to C3 (1) Yes (-999) Don’t know / no answer → C3 2c. In the timetable, how many periods in a week are there for literacy activities? (Enter -999 for don’t know/no answer) periods 3. Have you directly supported teachers for grades 1 and 2 on how to teach Kiswahili or English? (1) Yes (0) No (-999) Don’t know / no answer 4. Are you satisfied with the pupils’ Kiswahili performance at the end of grades 1 and 2 in your school? (1) Yes (0) No (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 127 4a. Are you satisfied with the pupils’ English performance at the end of grades 1 and 2 in your school? (1) Yes (0) No (-999) Don’t know / no answer 5. How do you know whether your pupils are progressing in Kiswahili or English during the academic year? (Select all that apply) (1) Conduct classroom observations (2) Monitor pupils’ results on tests given by teachers (3) Evaluate pupils myself (4) Review pupils’ assignments or homework (5) Teachers provide me with progress reports (-888) Other (specify): ___________ (-999) Don’t know / no answer 6. Who is responsible for reviewing teachers’ lesson plans for Kiswahili or English? (Select all that apply) (0) No one → skip to C8 (1) Head Teacher (2) Deputy Head Teacher (3) Senior teacher (-888) Other (specify): ___________ (-999) Don’t know / no answer → skip to C8 7. How often are these Kiswahili or English lesson plans reviewed? (0) Never (1) Once per day (2) Every week (3) Once every 2 weeks (4) Once every month (5) Once every 2-3 months (6) Once per year (-999) Don’t know / no answer 8. In your school, who is responsible for observing teachers teaching Kiswahili or English in their classroom? (Select all that apply) (0) No one → skip to C10 (1) Head Teacher (2) Deputy Head Teacher (3) Senior teacher (4) Curriculum Support Officer (5) Quality Assurance and Standards Officer (QASO) (-888) Other (specify): ___________ (-999) Don’t know / no answer → skip to C10 9. [If C8=4] In one term, how often is a teacher observed teaching Kiswahili or English by CSOs? (0) Never → skip to C9c (1) One time (2) Two times (3) Three or more times (-999) Don’t know / no answer → skip to C9c USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 128 9a. [If C9>0] In your opinion, how helpful are these CSO observations in improving teaching of Kiswahili/English: not helpful, somewhat helpful, helpful, or very helpful? (1) Not helpful (2) Somewhat helpful (3) Helpful (4) Very helpful (-999) Don’t know / no answer 9b. [If C9a=1] Why do you think these CSO observations are not helpful? 9b.i [If C9a=1] What could be done to make them more helpful? 9c. [If C8=5] In one term, how often is a teacher observed teaching Kiswahili or English by Quality Assurance and Standards Officers (QASOs)? (0) Never → skip to C10 (1) One time (2) Two times (3) Three or more times (-999) Don’t know / no answer → skip to C10 9d. [If C9c>0] In your opinion, how helpful are these QASO observations in improving teaching of Kiswahili/English: not helpful, somewhat helpful, helpful, or very helpful? (1) Not helpful (2) Somewhat helpful (3) Helpful (4) Very helpful (-999) Don’t know / no answer 9e. [If C9d=1] Why do you think these QASO observations are not helpful? 9e. [If C9d=1] What could be done to make them more helpful? 10. Who provides textbooks for English and Kiswahili for grades 1 and 2? (Select all that apply) (1) Ministry of Education (2) County Government (3) School (via independent funds) (4) Parents (individually) (5) Board of management (6) NGO (-888) Other (specify): ___________ (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 129 10a. Are teachers at this school currently using the following resources? (Read list; select all that apply) (1) Tusome English Teacher’s Guide (1) (2) Tusome Kiswahili Teacher’s Guide (1) (3) Tusome English Teacher’s Guide (2) (4) Tusome Kiswahili Teacher’s Guide (2) (5) Tusome English Pupil Books (1) (6) Tusome Kiswahili Pupil Books (1) (7) Tusome English Pupil Books (2) (8) Tusome Kiswahili Pupil Books (2) (-999) Don’t know / no answer 10b. [If any of items 1-8 in C10a are not selected] Why aren’t teachers using these Tusome teacher guides and/or pupil books at this school? (Select all that apply) (1) We have never been provided this/ I do not know of Tusome (2) Materials are old/damaged (3) There are not enough materials (4) Lessons are not effective (5) We have been instructed to use different materials (6) We have chosen to use different materials (7) The teachers are not adequately trained on how to use them (8) Difficulties aligning content with the Competency Based Curriculum (CBC) (9) Difficulties aligning content with schemes of work and lesson plan format/templates (10) There is inadequate time to cover all the content (-888) Other (specify): ___________ (-999) Don’t know / no answer 10c. Is Tusome aligned with the recently rolled out Competency Based Curriculum (CBC)? (1) Yes (0) No (-999) Don’t know / no answer → skip to C10d 10c.i What is the relationship between Tusome and the CBC? 10d. Are you aware of any county-specific policies requiring use of the Tusome instructional approach? (1) Yes (0) No → skip to C11 (-999) Don’t know / no answer → skip to C11 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 130 10e. Please describe the county-specific policies requiring use of the Tusome instructional approach: 11. In your school, what language do grade 1 and 2 teachers use most often while teaching English as a subject? (1) English (2) Kiswahili (-888) Other (specify): ___________ (-999) Don’t know / no answer 12. In your school, what language do grade 1 and 2 teachers use most often while teaching Kiswahili as a subject? (1) English (2) Kiswahili (-888) Other (specify): ___________ (-999) Don’t know / no answer 13. How many pupils in this school speak either Kiswahili or English as their first language: none of them, some of them, most of them, or all of them? (1) None of them (2) Some of them (3) Most of them (4) All of them (-999) Don’t know / no answer H. Information about the School 1. Does the school have a functioning library? (1) Yes (0) No → skip to D5 (-999) Don’t know / no answer → skip to D5 2. Who uses the library? (Select all that apply) (1) Grade 1 pupils (2) Grade 2 pupils (3) Grade 3 pupils (4) Other pupils (5) Teachers (-999) Don’t know / no answer 3. Is there a scheduled library time for grades 1 and 2? (1) Yes (0) No (-999) Don’t know / no answer 4. Are grade 1 and 2 pupils allowed to take library books home? (1) Yes (0) No (-999) Don’t know / no answer 5. How many of the grade 1 and 2 teachers have received specific training on teaching Kiswahili/English skills? (1) None of them (2) Some of them (3) Most of them (4) All of them (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 131 7. How often did the school’s Board of Management (BOM) meet during the last school year? (0) Never (1) Once every week (2) Once every month (3) Once a term (6) Once per year (-777) N/A – do not have BOM → skip to D9 (-999) Don’t know / no answer → skip to D9 8. For what types of things does the BOM have decision making authority and/or responsibility? (Select all that apply) (1) Discuss school management problems (2) Discuss pupils’ problems and solutions (3) Review progress of school improvement efforts (4) Review financial situation (budgets) of the school (5) Manage school infrastructure and equipment (6) Discuss school curriculum (7) Raise funds (8) Manage procurement or distribution of textbooks (-888) Other (specify): ___________ (-999) Don’t know / no answer 9. Does the school have electricity? (1) Yes (0) No (-999) Don’t know / no answer 10. Does the school have a feeding program? (1) Yes (0) No (-999) Don’t know / no answer 11. Does the school have a computer room? (1) Yes (0) No (-999) Don’t know / no answer Thank you for your participation! You have been very helpful. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 132 I. School Climate Assessment The following checklist is to be completed by the enumerator at the end of the interview. Each of the following items should be directly observed by walking around the school grounds—they should not depend on respondent reporting. If a given item cannot be observed due to restricted access, ask the Head Teacher or another school official to show you; otherwise tick “could not observe” rather than “no.” 1. School playground (1) Yes (0) No (-999) Could not observe 2. Toilets for pupils (1) Yes (0) No (-999) Could not observe 3. Separate toilets for boys and girls (1) Yes (0) No (-999) Could not observe 4. Electricity (1) Yes (0) No (-999) Could not observe 5. Paved/sealed road leading up to school (1) Yes (0) No (-999) Could not observe 6. Pipe-borne water (1) Yes (0) No (-999) Could not observe 7. Other improved water source(s) (borehole, standpipe, other [non-surface] protected source) (1) Yes (0) No (-999) Could not observe 8. School library (1) Yes (0) No (-999) Could not observe 9. Properly demarcated and fenced grounds with a secure gate (1) Yes (0) No (-999) Could not observe 10. Physical structures are clean, well maintained, and safe (1) Yes (0) No (-999) Could not observe USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 133 CLASSROOM OBSERVATION TOOL Did the teacher consent to participate in the study? (1) Yes (0) No (do not continue) J. Background Information 1. Date of observation: (a) Day: (b) Month: (c) Year: 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code: 3. Class information: (d) Language lesson: (b) Grade level: (1) Kiswahili (1) Grade 1 (2) English (2) Grade 2 (c) Boys present: (d) Girls present: 4. Teacher information: Unique teacher ID: Before the lesson begins, ask the teacher if s/he will be using a teacher’s guide and/or lesson plan and go through the following questions with him/her. 5. Will you use a teacher’s guide today? (1) Yes (0) No → skip to A8a 6. What is the name of the teacher’s guide you will use? (Select all that apply) (1) Tusome English Teacher’s Guide (1) (2) Tusome Kiswahili Teacher’s Guide (1) (3) Tusome English Teacher’s Guide (2) (4) Tusome Kiswahili Teacher’s Guide (2) (-888) Other (specify): ________________ (-999) Don’t know / no answer → skip to A8a 7. What is the unit number (or week) being covered today? (Enter 0 if there is no numeric identifier or -999 if you are unable to determine) USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 134 8. How many units in the guide book have been covered prior to the unit being covered today? (1) All of them (2) Most of them (3) About half (4) Less than half (5) Barely any or none (-999) Don’t know / no answer 8a. Will you use pupil books during your lesson today? (1) Yes (0) No → skip to A9 (-999) Don’t know / no answer → skip to A9 8b. What is the name of the pupil book(s) you will use? (Select all that apply) (1) Tusome English Pupil’s Books (1) (2) Tusome Kiswahili Pupil’s Books (1) (3) Tusome English Pupil’s Books (2) (4) Tusome Kiswahili Pupil’s Books (2) (-888) Other (specify): ________________ (-999) Don’t know / no answer 9. Are you using a lesson plan? (1) Yes (0) No → skip to A11 (-999) Don’t know / no answer → skip to A11 10. Which of the following best describes the lesson plan? (Observe, do not read out) (1) It is entirely handwritten (2) It is handwritten using a pre￾prepared/printed template (3) It is entirely pre-prepared/printed (4) It is primarily copied from the teachers’ guide (-888) Other (specify): ________________ (-999) Could not determine 11. What is the anticipated duration of today’s lesson? minutes 12. Enter lesson start time (HH:MM; use 24 hour clock) : USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 135 K. Classroom Observation Form Every 3 minutes, scan the classroom and check all the activities that you observed and the materials that pupils and teachers are using. Note: Do not use all of the columns if the period is not 48 minutes. Use only those columns that correspond to the amount of minutes for the class period. For instance, if the period is 30 minutes, only go as far as the column with 30 minutes. If the teacher is using a Tusome Teacher’s Guide, open the appropriate guide on your tablet and navigate to the unit that s/he is covering today. When not taking snapshots, you should try to follow along in the guide with him/her so you can answer some questions at the end of the observation. Minutes 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 1. Teacher Focus (one response each period) Whole class Small group One individual pupil Not focusing on pupils Teacher not present/disengaged 2. Instructional Content (one response each period) Phonological awareness (sounds) Alphabetic principle (letters, including letter-sound correspondence) Fluency (speed, accuracy, and/or expression) Vocabulary (new words) Comprehension (reading or listening) Writing Teacher not present/disengaged Other 3. Teacher Action (one response each period) Reading Writing Lecturing/explaining Asking questions Listening to pupils Monitoring pupils USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 136 Minutes 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 Giving feedback Teacher not present/disengaged Other 4. Pupil Actions (one response each period) Choral reading (all h) Partner reading Individual reading out ld Silent reading Writing Listening to the h Repeating/recitation Off task/uninvolved 5. Materials Used (mark all being used) Blackboard Pocket chart Letter cards Pupil’s book Exercise books Teacher’s guide No material used Other USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 137 L. After Observation Form Immediately after the observed lesson ends, please answer the following questions. 0. Enter lesson finish time (HH:MM; use 24 hour clock) : 1. Indicate if the classroom has the following features and/or items: Classroom Inventory Yes No Child-sized tables and chairs/benches Timetable on the wall Decorations/materials on the walls Reading books for the children Exercise books for each pupil Pencils for each pupil Tusome pupil’s books Tusome teacher’s guide 1a. [If A8b=1, 2, 3, or 4] Approximately how many Tusome pupil books were there in the classroom? (1) More than enough for each child (2) Enough for each child (3) More than half (but less than all) (4) Less than half (but more than none) (0) None 1b. [If A8b=-888] Approximately how many other pupil books were there in the classroom? (1) More than enough for each child (2) Enough for each child (3) More than half (but less than all) (4) Less than half (but more than none) (0) None 2. Circle the number corresponding to the most common type of feedback provided to pupils by the teacher: 1 2 3 4 Gives no feedback at all. Gives feedback about incorrect responses only, in a manner that does not encourage further effort. Gives feedback about incorrect responses only, in a manner that encourages further effort. Gives feedback about correct and incorrect responses in a manner that encourages further effort. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 138 3. Did the teacher generally praise the pupils when they tried hard and/or gave the correct response? (1) Yes (0) No (-777) Not applicable 4. What did the teacher do when a pupil either gave the wrong response or did not respond at all? (Circle the number corresponding to the most common type of action by the teacher): 1 2 3 4 5 Criticized the pupil and then called on another pupil or otherwise moved on. Ignored the error and then called on another pupil or otherwise moved on. Provided remediation but then called on another pupil or otherwise moved on. Provided remediation and encouraged the pupil to try again. Not applicable / none 5. Which of the following components of the direct instructional model (DIM) did the teacher demonstrate during the lesson? (Select all that apply) (1) YI do (2) We do (3) NYou do 5a. [If 5=1, 2, and 3] Did the teacher generally follow the recommended order for DIM? I.e., first: I do, second: we do, third: you do? (1) Yes (0) No 6. [If A6=1, 2, 3, or 4] How well did the teacher cover the content of the Tusome unit(s) taught today? (1) All of the content was covered (2) S/he skipped some sections (3) S/he skipped most sections (4) The lesson was not followed for the most part (-888) Other (specify): ___________ 6a. [If A6=1, 2, 3, or 4] How closely did the teacher follow the material in the teacher’s guide? (1) Followed it very closely (2) Followed the lesson closely but did not refer to it all of the time (3) Did not refer to the teachers’ guide at all (-888) Other (specify): ___________ 87. Did the teacher generally use an adequate pace during instruction (i.e., not too fast or too slow)? (1) Yes (0) No USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 139 8. For English lessons only: Circle the number corresponding to the most common type of language usage by the teacher: 1 2 3 4 Used home language most of the time, with little integration of English and/or Kiswahili. Communicated in English – even when learners did not seem to understand – and discouraged use of Kiswahili. Used code switching (English-Kiswahili or vice versa) only when majority of the pupils did not seem to understand. Integrated English and Kiswahili as appropriate, i.e., depending on the level of understanding of the pupils. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 140 CURRICULUM SUPPORT OFFICER (CSO) INTERVIEW Did the CSO consent to participate in the study? (1) Yes (0) No (do not continue) M. Background Information 1. Date of interview: (a) Day: (b) Month: (c) Year: 2. School information: (a) Region: (b) County: (c) Sub-county: (d) Zone/cluster: (e) Unique school code(s):39 Enter codes for each sampled school that this CSO oversees 3. CSO information: (c) Gender: (0) Male (1) Female N. CSO Questionnaire 1. How long have you been a CSO? years months 2. How long have you been a CSO in this zone? years months 4. What is your highest professional training level in teaching? (1) Untrained (2) P1 (3) Diploma/S1 (4) Bachelor of Education (5) Masters of Education (-888) Other (specify): __________ (-999) Don’t know / no answer 39 All identification numbers will be anonymized for respondent protection/privacy USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 141 5. How many public primary schools are in your zone(s)? (Enter -999 for don’t know/no answer) schools 6. How many private and APBET primary schools are in your zone(s)? (Enter zero if none; -999 for don’t know/no answer) schools 7. In July 2019, how many days did you visit schools for classroom observations? (Enter zero if none; -999 for don’t know/no answer) days If zero/none → skip to B9a 8. When you visit a school, what classes do you observe? (Select all that apply) (1) Grade 1 (2) Grade 2 (3) Grade 3 (-888) Other 9. In July 2019, how many actual lessons did you observe? (Enter zero if none; -999 for don’t know/no answer) lessons 9a. [If B7<10 & B7>-1] Why didn’t you visit schools for classroom observations on at least 10 days during July? (Select all that apply) (1) It isn’t part of my current job description (2) I do not think the observations help teachers (3) I did not have means to travel to the schools (4) I did not have time because of other job responsibilities (5) I was on leave for some/all of the period (6) I do not have a tablet or my tablet is not working (7) I was told by my superior(s) not to do this (-888) Other (specify): ___________ 10. In July 2019, approximately what percentage of your time was spent on instructional support for Tusome? (Enter zero if none; -999 for don’t know/no answer) % USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 142 11. During the last 2 years, have you received training in providing instructional support in Kiswahili/English at lower primary level for Tusome? (1) Yes (0) No → skip to B15 (-999) Don’t know / no answer → skip to B15 12. What type of trainings were they? (Select all that apply) (1) School-based (2) Offsite in-service (-888) Other (specify): ___________ (-999) Don’t know / no answer 13. Who organized these trainings? (Select all that apply) (1) Ministry of Education (2) Tusome (3) Other project/donor (-888) Other (specify): ___________ (-999) Don’t know / no answer 14. What was the approximate total number of days of all of this type of training you have received in the last 2 years? (Enter -999 for don’t know/no answer) days 15. During the last 2 years, have you personally delivered Tusome in-service training for teachers at the zonal level? (1) Yes (0) No (-999) Don’t know / no answer 16. How do you assess the teachers’ performance in the schools? (Select all that apply) (1) Check KCPE results (2) Check zonal term exam results (3) Discuss with the Head Teacher (4) Conduct teacher observations (5) Assess pupils directly (6) No assessment procedure (-888) Other (specify): ___________ (-999) Don’t know / no answer 17. In July 2019, on how many days did you have to cancel a lesson observation in order to attend to other duties? (Enter zero if none; -999 for don’t know/no answer) Days USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 143 18. How effective do you think the current CSO classroom support system has been in improving Kiswahili and English outcomes: not effective, only a little effective, somewhat effective, or mostly effective? (1) Not effective (2) Only a little effective (3) Somewhat effective (4) Mostly effective (-999) Don’t know / no answer 19. How would you describe the current approach to early grade reading in your schools: not effective, only a little effective, somewhat effective, or mostly effective? (1) Not effective (2) Only a little effective (3) Somewhat effective (4) Mostly effective (-999) Don’t know / no answer 20. Are there, or have there been, any early grade reading initiatives in your zone over the last 2 years except for Tusome? (1) Yes (0) No → skip to B22 (-999) Don’t know / no answer → skip to B22 21. What early grade reading initiatives besides Tusome have been operating in your zone over the last 2 years? 22. Does the current timetable include periods for Kiswahili or English language activities (grades 1-3)? (0) No → skip to B25 (1) Yes (2) There is no timetable → skip to B27 (-999) Don’t know / no answer → skip to B25 23. In the timetable, how many lessons in a week are there for teaching Kiswahili language activities? (Enter zero if none; -999 for don’t know/no answer) periods 24. In the timetable, how many lessons in a week are there for teaching English language activities? (Enter zero if none; -999 for don’t know/no answer) periods 25. Does the current timetable include periods for literacy activities? (0) No → skip to B27 (1) Yes (-999) Don’t know / no answer → skip to B27 26. In the timetable, how many lessons in a week are there for literacy activities? (Enter -999 for don’t know/no answer) periods USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 144 27. In your view, how many lessons a week should teachers teach English language activities? (Enter zero if none; -999 for don’t know/no answer) periods 28. In your view, how many lessons a week should teachers teach Kiswahili language activities? (Enter zero if none; -999 for don’t know/no answer) periods 29. In your view, how many lessons a week should teachers teach literacy activities? (Enter zero if none; -999 for don’t know/no answer) periods 30. What instructional materials are grades 1-3 teachers in your zone(s) using to teach English language activities? 31. What instructional materials are grades 1-3 teachers in your zone(s) using to teach Kiswahili language activities? O. CSO Perceptions I will now read a few statements and ask you to state whether you strongly agree, agree, neither agree nor disagree, disagree, or strongly disagree with that statement. For these questions, we want to understand your personal perceptions on teaching reading effectively, even if those perceptions differ from current school practices or requirements. All of your responses are confidential. 1. When it comes to teaching reading, it is better for teachers to develop their own lessons than to work from structured teachers’ guides (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 2. When it comes to teaching reading, it is better for teachers to work from structured teachers’ guides than to develop their own lessons (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 145 3. Pupils best learn to read new words by visually memorizing them as whole units (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 4. Pupils best learn to read new words by first understanding the letters and sounds that make up the word (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 5. Regular observation and coaching by CSOs/ICs and Head Teachers usually helps teachers become more effective in their job (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer 6. Regular observation and coaching by CSOs/ICs and Head Teachers usually distracts teachers from doing their job effectively (1) Strongly agree (2) Agree (3) Neither agree nor disagree (4) Disagree (5) Strongly disagree (-999) Don’t know / no answer P. Tusome Ratings Use the scale on the right to rate various aspects of Tusome in improving reading in grades 1 and 2 in your schools. (Repeat scale on right for each item). Very poor Poor Fair Good Very good N/A Don’t know / refused 1. Quality of Tusome trainings for CSOs 1 2 3 4 5 -777 -999 2. Relevance of Tusome trainings for CSOs 1 2 3 4 5 -777 -999 2a . Frequency of Tusome trainings for CSOs 1 2 3 4 5 -777 -999 3. Duration of Tusome trainings for CSOs 1 2 3 4 5 -777 -999 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 146 Use the scale on the right to rate various aspects of Tusome in improving reading in grades 1 and 2 in your schools. (Repeat scale on right for each item). Very poor Poor Fair Good Very good N/A Don’t know / refused 4. Quality of Tusome teacher trainings delivered by CSOs 1 2 3 4 5 -777 -999 5. Relevance of Tusome teacher trainings delivered by CSOs 1 2 3 4 5 -777 -999 6. Quality of the content of Tusome teachers’ guides 1 2 3 4 5 -777 -999 7. Quality of the content of Tusome pupils’ books 1 2 3 4 5 -777 -999 8. Relevance of cluster/zonal termly meetings 1 2 3 4 5 -777 -999 9. Effectiveness of lesson observations by CSOs 1 2 3 4 5 -777 -999 10. Effectiveness of Tusome support to CSOs to undertake school visits 1 2 3 4 5 -777 -999 11. Effectiveness of Tusome approach: “I do”; “we do”; “you do” 1 2 3 4 5 -777 -999 What is your overall rating of the Tusome initiative from the following perspectives? (Repeat scale on right for each item). Not enough Almost enough Just right Too much N/A Don’t know / refused 12. Number of lessons in English language activities each week 1 2 3 4 -777 -999 13. Number of lessons in Kiswahili language activities each week 1 2 3 4 -777 -999 14. Time allowed to deliver a whole lesson in English language activities 1 2 3 4 -777 -999 15. Time allowed to deliver a whole lesson in Kiswahili language activities 1 2 3 4 -777 -999 16. Amount of homework for the pupil in English language activities 1 2 3 4 -777 -999 17. Amount of homework for the pupil in Kiswahili language activities 1 2 3 4 -777 -999 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 147 What is your overall rating of the Tusome initiative from the following perspectives? (Repeat scale on right for each item). Not enough Almost enough Just right Too much N/A Don’t know / refused 18. Overall amount of work for the pupil in a term 1 2 3 4 -777 -999 19. Overall amount of work for the teacher in a term 1 2 3 4 -777 -999 20. Overall amount of work for the CSO in a term 1 2 3 4 -777 -999 21. Overall amount of support from the Tusome team to the CSO in a year 1 2 3 4 -777 -999 21a . Number of lessons in Literacy activities each week 1 2 3 4 -777 -999 22. How do you keep track of pupils’ performance in reading in grades 1 and 2 in your schools? (Select all that apply) (1) Observe pupils in the classroom (2) Monitor pupils’ results on tests given by teachers (3) Review pupils’ assignments or homework (4) Collect progress reports from teachers (5) Directly assess the pupils myself (6) Do not keep track (-888) Other (specify): ___________ (-999) Don’t know / no answer 23. Are results from external Early Grade Reading Assessments or EGRAs (i.e., those not conducted by Tusome) communicated to you? (1) Yes (0) No (-999) Don’t know / no answer 24. In your opinion, what are the key strengths of Tusome? (Select all that apply) (1) Provision of teachers’ guide (2) Provision of pupils’ books (3) Trainings for teachers (4) Teacher observation and coaching (5) Pupil assessment (6) Monitoring dashboard (-888) Other (specify): ___________ (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 148 25. What challenges, if any, have you have faced in implementing Tusome? (Select all that apply) (1) Quantity of work (2) Time constraints given the number of activities (3) Short notice in communicating information (4) Availability of transport (5) Transport facilitation to widely spread schools (6) Inadequate training (7) Hardware/software issues (8) Difficulties aligning content with the Competency Based Curriculum (CBC) (9) Difficulties aligning content with schemes of work and lesson plan format/templates (-888) Other (specify):___________ (-999) Don’t know / no answer 25a. What challenges, if any, do schools in your zone face in implementing the new Competency Based Curriculum (CBC)? (Select all that apply) (1) Inadequate materials (2) Inadequate training of teachers (3) Lack of teacher buy-in (4) Difficulties aligning content with existing curricular materials (5) Difficulties aligning CBC content with existing schemes of work and lesson plan format/templates (-888) Other (specify):___________ (-999) Don’t know / no answer 26. What challenges, if any, do schools in your zone face in implementing Tusome? (Select all that apply) (1) Old/damaged materials (2) Inadequate training of teachers (3) Lack of teacher buy-in (4) Difficulties aligning content with the Competency Based Curriculum (CBC) (5) Difficulties aligning content with schemes of work and lesson plan format/templates (-888) Other (specify):___________ (-999) Don’t know / no answer USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 149 28. Is Tusome aligned with the recently rolled out Competency Based Curriculum (CBC)? (1) Yes (0) No (-999) Don’t know / no answer → skip to E1 29. What is the relationship between Tusome and the CBC? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 150 Q. Project Sustainability 1. Are you aware of any national policies requiring use of the Tusome instructional approach? (1) Yes (0) No (-999) Don’t know / no answer 2. Are you aware of any county-specific policies requiring use of the Tusome instructional approach? (1) Yes (0) No → skip to E4 (-999) Don’t know / no answer → skip to E4 3. Please describe the county-specific policies requiring use of the Tusome instructional approach: 4. Do you think your county will continue with the Tusome approach after donor support is discontinued? This includes all in-service trainings and oversight of school monitoring activities. (1) Yes → skip to E6 (0) No (-999) Don’t know / no answer → skip to E6 5. Why might your county not continue with the Tusome approach after donor support is discontinued? (Select all that apply) (1) Lack of motivation/interest (2) Not adequately trained to take over activities (3) Insufficient resources/budget to continue trainings (4) Insufficient resources/budget to continue school monitoring activities (5) Too much staff turnover (6) Inadequate succession planning/preparation (-888) Other (specify): ___________ (-999) Don’t know / no answer 6. What, if anything, do you think is needed to ensure that your county continues implementing Tusome after donor support is discontinued? 7. Do you have any other thoughts that you would like to share regarding the long-term sustainability of Tusome? Thank you for your participation! You have been very helpful. USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 151 YOUTH FUND LEADERSHIP INTERVIEW Thank you for your interest in taking the Youth Fund Leadership Survey. USAID/Kenya has partnered with NORC at the University of Chicago to sponsor an evaluation to assess the impact of Tusome on children’s reading abilities. The results of this study will be used by the Ministry of Education and USAID to inform future programs aimed at helping children in Kenya become good readers. As part of the evaluation, we are interviewing recipients of the Youth Fund Grants to better understand the scope of grant activities. Your decision to take part in this survey is voluntary. There are no foreseeable risks involved in participating in this survey other than those encountered in day-to-day life. Your identity will be kept private. If the results of this survey are written in a report or presented, your name will not be used. Responses from all participants will be presented in summary form only. The survey is expected to take approximately 15 minutes to complete. If you have any questions or technical difficulties while taking this survey, you may email Carlos Fierros with NORC at the University of Chicago at Fierros-Carlos@norc.org. Do you consent to participate in this survey? (1) Yes (0) No (do not continue) Thank you for consenting to participate in this survey. In answering the following questions, please consider both Round 1 and Round 2 grants received. 1. Please enter the date of the survey. (a) Day: (b) Month: (c) Year: 2. Youth group information: (a) Youth group name: (b) Province: (c) County: (f) Schools targeted (where applicable): (d) Number of schools directly targeted. Only include schools where activities actually took place: USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 152 3. Respondent information: (respondent_info) (a) Surname: (b) Given name: (c) Youth group title: (d) Telephone: (e) Age (f) Gender: (0) Male (1) Female 4. What types of activities did your organization undertake with support from the Youth Fund? (Select all that apply) (activity_type) (1) Community sensitization on the importance of reading activities (2) Home-based sensitization on the importance of reading activities (3) Community- and school-based reading events (4) Storage and care of book activities (5) Buying and distributing book activities (6) Artistic projects to promote literacy (7) Other (specify): 5. Approximately how many beneficiaries has your organization reached so far with the Youth Fund activities? Please include estimates for direct and indirect beneficiaries. (beneficiaries) ____Number of direct beneficiaries (individuals who receive a direct benefit from grant-supported activities) ____Number of indirect beneficiaries(individuals who receive an indirect benefit from grant-supported activities) (-999) Don’t know 5a. How do you define “direct beneficiaries”? (beneficiaries_direct_def) (Open response) 5b. How do you define “indirect beneficiaries”? (beneficiaries_indirect_def) (Open response) 6. Approximately how much funding did you receive to conduct these activities, in Kenyan Shillings? (funding_total) ____ KES (-999) Don’t know/ Refused 7. Over the past two years, have you observed changes in community awareness and engagement in reading in the counties where you conducted your activities? (community_awareness_cha nge) (1) Yes (1) No 99) Don’t know/ Refused PROG: SKIP TO QUESTION 9 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 153 8. How much of a role, if any, do you think the Youth Fund activities played in this change? (community_awareness_cha nge_open) (Open response) 9. PROG: PIPE IN ACTIVITY NAME TEXT, LOOP FOR AS MANY ANSWERS AS SELECTED IN QUESTION 4 For [activity X], please rate its effectiveness in raising community awareness on the importance of reading on a scale of 1 to 4, with 1 being very ineffective and 4 being very effective. (activity_effectiveness_co mmaware) (1) Very ineffective (2) Somewhat ineffective (3) Somewhat effective (4) Very effective (-999) Don’t know/ Refused 10. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 1 OR 2 IN QUESTION 9 [If answer to question 9 was 1 or 2] Why do you believe this activity was ineffective in raising community awareness on the importance of reading? (activity_ineffective_comm aware) (Open response) (-999) Don’t know/ Refused 11. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 3 or 4 IN QUESTION 9 If answer to question 9 was 3 or 4] Why do you believe this activity was effective in raising community awareness on the importance of reading? (activity_effective_comma ware) (Open response) (-999) Don’t know/ Refused USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 154 11a. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 3 or 4 IN QUESTION 9 If answer to question 9 was 3 or 4] Do you believe the effects [activity X] had on community awareness on the importance of reading will be long lasting or sustainable? Why or why not? (activity_effectiveness_co mmaware_lasting ) (Open response) (-999) Don’t know/ Refused 12. PROG: PIPE IN ACTIVITY NAME TEXT, LOOP FOR AS MANY ANSWERS AS SELECTED IN QUESTION 4 For [activity X], please rate its effectiveness in engaging children in reading on a scale of 1 to 4, with 1 being very ineffective and 4 being very effective. (activity_effectiveness_rea ding) (1) Very ineffective (2) Somewhat ineffective (3) Somewhat effective (4) Very effective (-999) Don’t know/ Refused 13. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 1 OR 2 IN QUESTION 12 [If answer to question 12 was 1 or 2] Why do you believe this activity was ineffective in engaging children in reading? (activity_ineffective_readin g) (Open response) (-999) Don’t know/ Refused 14. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 3 or 4 IN QUESTION 12 If answer to question 12 was 3 or 4] Why do you believe this activity was effective in engaging children in reading? (activity_effective_reading) (Open response) (-999) Don’t know/ Refused USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 155 14a. PROG: LOOP AS APPLICABLE FOR ANSWERS RATED 3 or 4 IN QUESTION 12 If answer to question 12 was 3 or 4] Do you believe the effects [activity X] had on engaging children in reading will be long lasting or sustainable? Why or why not? (activity_effectiveness_rea ding_lasting) (Open response) (-999) Don’t know/ Refused 15. Have you participated in a Youth Fund grants start-up induction workshop? (workshop_participate) (1) Yes (0) No PROG: SKIP TO QUESTION 26 (-999) Don’t know/ Refused 16 PROG: IF Q15 IS YES When did the most recent Youth Fund grants start-up induction workshop you attended take place? (workshop_date) [MONTH] [YEAR] (-999) Don’t know/ Refused 23. PROG: IF Q15 IS YES After this workshop, did you feel: (workshop_feeling) (1) More prepared to undertake round 2 activities (2) About the same amount of readiness to undertake round 2 activities (3) Less prepared to undertake round 2 activities (-999) Don’t know/Refused 24. PROG: IF Q15 IS YES After this workshop, did you feel like you have a better understanding of what is expected of your organization for round 2? (workshop_expectations) (1) Yes (2) Neither better nor worse understanding (0) No (-999) Don’t know/ Refused 25. After this workshop, were you able to identify which Curriculum Support Officers and Sub-County Quality Assurance and Standards Officers there are to support your organization? (workshop_CSO) (1) Yes (0) No (-999) Don’t know/ Refused 26. If you were to do your round 1 activity/activities again, what would you have done differently and why? (Open response) (-999) Don’t know/Refused USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 156 27. If there were no logistical or financial constraints, what do you think your organization could do to have the biggest possible impact on community awareness on the importance of reading in your county? (Open response) (-999) Don’t know/Refused 27a. If there were no logistical or financial constraints, what do you think your organization could do to have the biggest possible impact on engaging children in reading in your county? (Open response) (-999) Don’t know/Refused 28. Do you have any other comments or feedback on the Youth Fund program? (Open response) (-999) Don’t know/Refused Thank you very much for your participation! USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 157 USAID/Kenya East Africa KII Field Control Interview date: ___/____/_______ (DAY/MONTH/YEAR) Interviewer name: __________________ Section A: Introduction 1. Can you describe your role in the Tusome activity? a. When did you first become involved in Tusome? b. Has your role in the activity changed over time? If so, how? 2. One of Tusome's explicit objectives is to create an “improved policy environment that promotes reading skills” (Sub-IR 2.2). Have you played a direct role in influencing the policy environment in this manner? a. If so, what role have you played? What government departments/agencies have you coordinated with? b. Is this a part of your formal job description? c. What role, if any, are you likely to play in the future? 3. In your own words, can you briefly describe the Tusome approach to improving English and Kiswahili skills in the early grades? a. In what ways did Tusome change how English and Kiswahili skills are taught in Kenyan schools in the early grades? b. What are the main activities, inputs, and outputs of the project? Section B: Teaching and Learning Materials (TLM) 1. Now I would like to discuss changes in GoK procedures, policies, or guidelines related to teaching and learning materials (TLM). Can you describe any TLM- or curriculum-related changes in GoK procedures, policies, or guidelines that have been made as a result of, or in response to, the Tusome activity? a. How and when did these changes first come into existence? Are they still in effect today? b. Do these or other Tusome-related TLM policies apply to Alternative Provision of Basic Education and Training (APBET) schools? Special Needs Education (SNE)? c. Are there specific procedures/policies/guidelines related to pricing and procurement of TLM? d. Are these GoK procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to implement TLM-related procedures/policies/guidelines after the USAID activity closes? f. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 2. How important do you think these GoK procedures/policies/guidelines are for the long-term sustainability of Tusome? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 158 a. If they were revoked or not enforced, how would this effect the continued implementation of Tusome? 3. Do you feel these GoK procedures/policies/guidelines will remain in place over the long term? Why or why not? a. What are the biggest threats to their long-term sustainability? b. What more do you think can be done to ensure they remain in effect over the long term? 4. What additional procedures/policies/guidelines do you think are critical to ensuring the continued, long-term use of Tusome materials and methods? a. Procedures/policies/guidelines at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these procedures/policies/guidelines? c. What are the key barriers to the formulation and adoption of such procedures/policies/guidelines? Section C: Teacher Training and Professional Development 1. We will now talk about integration of the Tusome materials and methods into teacher training and professional development, starting with pre-service training (PRESET). Can you tell me about any changes in primary teacher education (PTE) that have been made as a result of, or in response to, the Tusome activity? a. Are these changes formal? Have they been formally adopted or mandated by MOE, TSC, SAGAs, and/or PTTCs? Why or why not? b. Are all Primary Teacher Training Colleges (PTTC) affected in the same manner? c. When did/will these changes go into effect? Are they still in effect today? d. Are these changes required/mandatory? i. [If so] What are the enforcement mechanisms? ii. [If not] What (if any) are the mechanisms for increasing and sustaining take-up? e. Do you know if sufficient human and/or financial resources have been put into place to implement these changes? f. Are any of these changes formally documented? Can this documentation be shared with the evaluation team? 2. In your opinion, how critical is PTE integration to the continued long-term use of Tusome materials and methods? Why? 3. What challenges have been or are being faced in integrating Tusome into PTE? How might these challenges be overcome? a. Are there unique challenges associated with different PTE actors? b. Are there challenges related to coordination between these multiple actors? 4. Do you think in-service teacher trainings (INSET) are critical to continued, long-term use of the materials and methods developed under Tusome? Why or why not? a. How important is the frequency of in-service trainings? b. What frequency is ideal in your opinion? 5. What has been or is currently being done to ensure the continuation of INSET on the materials and methods developed under Tusome after the USAID activity closes? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 159 a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to continuation of INSET on the materials and methods developed under Tusome? c. Do these or other Tusome-related INSET procedures/policies/guidelines apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to continue INSET on the materials and methods developed under Tusome? f. Are any of these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 6. Do you think formal INSET on the materials and methods developed under Tusome will continue over the longer term? Why or why not? a. Will this vary across counties, sub-counties, and schools? If so, in what ways? b. [If yes] Do you think the quality of INSET will remain the same? Why or why not? 7. What more could be done to ensure the continuation of formal INSET on the materials and methods developed under Tusome over the long term? a. At the national level? County-level? Sub-county level? School-level? 8. What additional procedures/policies/guidelines do you think are critical to ensuring teachers continue to get trained on Tusome methods and materials? a. Procedures/policies/guidelines at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these additional procedures/policies/guidelines? c. What are the key barriers to the formulation and adoption of these procedures/policies/guidelines? Section D: Monitoring, Supervision, and Assessment 1. We will now discuss the school-level monitoring and supervision components of Tusome, including tablet-centered observations, coaching, pupil assessment, and reporting. Do you think such monitoring activities are critical to the continued long-term use of Tusome materials and methods? Why or why not? a. What specific components are the most critical? The least critical? 2. What has been or is currently being done to ensure the continuation of Tusome-related monitoring and supervision activities after the USAID activity closes? a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to the continuation of observations, coaching, pupil assessments, and/or reporting? c. Do these or other Tusome-related monitoring policies apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to implement these procedures/policies/guidelines after the USAID activity closes? f. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 160 3. Do you believe that Tusome-related monitoring and supervision by CSOs, QASOs, and others will continue over the long-term? Why or why not? a. Are any of these components more likely than the others to continue? Which components and why? b. Do you think monitoring will continue with the same volume/frequency? Why or why not? c. Do you think monitoring and supervision will be of the same quality over the longer term? Why or why not? 4. In your opinion, what are the biggest challenges or threats to the continuation of Tusome-related monitoring and supervision activities? As a reminder, these activities include tablet-centered observations, coaching, pupil assessment, and reporting. a. At the national level? County-level? Sub-county level? School-level? b. Are there anticipated challenges related to continued training/capacitation of CSOs/QASOs to conduct observations/coaching/assessment? What do these challenges look like? c. Are there anticipated challenges related to long-term maintenance of equipment/hardware and software for monitoring? d. Are there anticipated challenges in ensuring monitors have adequate time and transportation for school-based monitoring? 5. What, if any, procedures/policies/guidelines specifically related to pupil assessment and pupil progress monitoring have been made as a result of, or in response to, the Tusome activity? a. What data are collected on pupils' reading progress? How are these data analyzed? b. How are these data used? How do they inform decision-making at the national, county, sub-county, school, or classroom levels? 6. Do you think these assessment policies are having an effect on Kenya's broader mandate to promote reading skills? Why or why not? 7. Is the proper use of materials and methods developed under Tusome formally factored in to teacher assessment and performance appraisal? If so, in what ways? a. For PRESET? INSET? b. Does this vary across counties, sub-counties, and schools? If so, in what ways? c. What, if any, enforcement mechanisms exist? d. [If yes] What happens when teachers are not properly using the Tusome materials and methods? Section E: General Sustainability Now I would like to discuss questions about the long-term sustainability of Tusome. These questions are targeted specifically to USAID and RTI respondents to better understand the future of Tusome. 1. Of all the changes in GoK procedures/policies/guidelines we have discussed or not discussed, which do you think are most critical to the long-term sustainability of the approaches and best practices established under the Tusome activity? 2. Beyond what has already been discussed, do you think the rollout of the new Competency Based Curriculum (CBC) will influence the continued long-term use of Tusome methods and materials? In what ways? a. What are the positive influences? Negative influences? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 161 b. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? c. [If negatively influencing sustainability] Could it have been avoided in your opinion? If so, how? d. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 3. Beyond what has already been discussed, do you think county-level authorities will influence the long-term sustainability of the approaches and best practices established under the Tusome activity? In what ways? a. What are the positive influences? Negative influences? b. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? c. [If negatively influencing sustainability] Could it have been avoided in your opinion? If so, how? d. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 4. Have there been any GoK procedures/policies/guidelines put forth that undermine the continued use of materials and methods developed under Tusome? a. What is the status of such procedures/policies/guidelines? Have they been adopted? Are they likely to be adopted in the future b. What is motivating these procedures/policies/guidelines? c. What do you think can be done to ensure they do not undermine the continued use of materials and methods developed under Tusome? 5. What do you think is the biggest threat to Tusome's sustainability after USAID funding ends? a. Was this threat known from the outset or did it only come to light after Tusome's design/implementation? b. Could this threat have been predicted or foreseen? c. Could it have been avoided? If so, how? d. How can it be minimized or avoided at this juncture? 6. What do you think is the best hope for Tusome's sustainability after USAID funding ends? 7. Besides what has already been discussed, are there any other things you would like to share about USAID and GoK’s partnership vis-à-vis Tusome? 8. Besides what has already been discussed, are there other key “lessons learned” during the Tusome activity that you would like to share? Section F: Closing 1. On a scale of 0 to 4, how successful do you think the Tusome activity was in improving government capacity to sustainably improve reading outcomes (IR 2)? Let's assume 0 is "not at all" and 4 is "to the fullest extent possible." a. Why did you choose this rating? b. [If below 4] What could have been done differently to rate the achievement of this objective as a "4"? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 162 c. [If below 4] What, if anything, do you think can still be done by December to improve this rating? 2. Would you personally like to see the materials and methods developed under Tusome continue after USAID funding ends? Why or why not? 3. Are there any additional comments that you would like to share with us? 4. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 163 RTI KII Field Control Interview date: ___/____/_______ (DAY/MONTH/YEAR) Interviewer name: __________________ Section A: Introduction 1. Can you describe your role in the Tusome activity? a. When did you first become involved in Tusome? b. Has your role in the activity changed over time? If so, how? 2. One of Tusome's explicit objectives is to create an “improved policy environment that promotes reading skills” (Sub-IR 2.2). Have you played a direct role in influencing the policy environment in this manner? a. If so, what role have you played? What government departments/agencies have you coordinated with? 3. In your own words, can you briefly describe the Tusome approach to improving English and Kiswahili skills in the early grades? a. In what ways did Tusome change how English and Kiswahili skills are taught in Kenyan schools in the early grades? b. What are the main activities, inputs, and outputs of the project? Section B: Teaching and Learning Materials (TLM) 1. Now I would like to discuss changes in GoK procedures, policies, or guidelines related to teaching and learning materials (TLM). Can you describe any TLM- or curriculum-related changes in GoK procedures, policies, or guidelines that have been made as a result of, or in response to, the Tusome activity? a. How and when did these changes first come into existence? Are they still in effect today? b. Do these or other Tusome-related TLM policies apply to Alternative Provision of Basic Education and Training (APBET) schools? Special Needs Education (SNE)? c. Are there specific procedures/policies/guidelines related to pricing and procurement of TLM? d. At what level(s) do these policies apply? The national level? County-level? Sub-county level? e. Are these GoK procedures/policies/guidelines enforceable? How are/will they be enforced? f. Have sufficient human and/or financial resources been put into place to implement TLM-related procedures/policies/guidelines after RTI's contract ends? g. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 2. Did RTI play a direct role in the formulation and/or adoption of these GoK procedures/policies/guideline changes? If so, what did this role look like? a. Who specifically at RTI was involved and at what stages (e.g., formulation, review, adoption)? What specifically did they do? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 164 b. Who were the non-RTI contributors? What was the scope of their contribution? c. How pivotal was RTI's role in these changes? I.e., if RTI staff had not been involved, would these change have occurred? 3. How important do you think these GoK procedures/policies/guidelines are for the long-term use of Tusome materials and methods? a. If they were revoked or not enforced, how would this effect the continued implementation of Tusome materials and methods? 4. Do you feel these GoK procedures/policies/guidelines will remain in place over the long term? Why or why not? a. What are the biggest threats to their long-term sustainability? b. What more do you think can be done before the end of RTI's contract to ensure they remain in effect over the long term? c. What do you think should be done (and by whom) after RTI's contract finishes to ensure they remain in effect over the long term? 5. What additional procedures/policies/guidelines do you think are critical to ensuring the continued, long-term use of Tusome materials and methods? a. Procedures/policies/guidelines at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these procedures/policies/guidelines? c. What are the key barriers to the formulation and adoption of such procedures/policies/guidelines? Section C: Teacher Training and Professional Development 1. We will now talk about integration of the Tusome materials and methods into teacher training and professional development, starting with pre-service training (PRESET). Can you tell me about any changes in primary teacher education (PTE) that have been made as a result of, or in response to, the Tusome activity? a. Are these changes formal? Have they been formally adopted or mandated by MOE, TSC, SAGAs, and/or PTTCs? Why or why not? b. Are all Primary Teacher Training Colleges (PTTC) affected in the same manner? c. When did/will these changes go into effect? Are they still in effect today? d. Are these changes required? i. [If so] What are the enforcement mechanisms? ii. [If not] What (if any) are the mechanisms for increasing and sustaining take-up? e. Do you know if sufficient human and/or financial resources have been put into place to implement these changes? f. Are any of these changes formally documented? Can this documentation be shared with the evaluation team? 2. In your opinion, how critical is PTE integration to the continued long-term use of Tusome materials and methods? Why? 3. What challenges have been or are being faced in integrating Tusome into PTE? How might these challenges be overcome? a. Are there unique challenges associated with different PTE actors? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 165 b. Are there challenges related to coordination between these multiple actors? 4. Do you think in-service teacher trainings (INSET) are critical to continued, long-term use of the materials and methods developed under Tusome? Why or why not? a. How important is the duration and frequency of in-service trainings? b. What duration and frequency is ideal in your opinion? 5. What has been or is currently being done to ensure the continuation of INSET on the materials and methods developed under Tusome after RTI's contract ends? a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to continuation of INSET on the materials and methods developed under Tusome? c. Do these or other Tusome-related INSET procedures/policies/guidelines apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to continue INSET on the materials and methods developed under Tusome? f. Are any of these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? g. Did RTI play a direct role in the formulation and/or adoption of these procedures/policies/guidelines? If so, what did this role look like? 6. Do you think formal INSET on the materials and methods developed under Tusome will continue over the longer term? Why or why not? a. Will this vary across counties, sub-counties, and schools? If so, in what ways? b. [If yes] Do you think the quality of INSET will remain the same? Why or why not? 7. What more could be done to ensure the continuation of formal INSET on the materials and methods developed under Tusome over the long term? a. At the national level? County-level? Sub-county level? School-level? b. What more do you think can be done before the end of RTI's contract to ensure they remain in effect over the long term? c. What do you think should be done (and by whom) after RTI's contract finishes to ensure they remain in effect over the long term? 8. What additional procedures/policies/guidelines do you think are critical to ensuring teachers continue to get trained on Tusome methods and materials? a. Procedures/policies/guidelines at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these additional procedures/policies/guidelines? c. What are the key barriers to the formulation and adoption of these procedures/policies/guidelines? Section D: Monitoring, Supervision, and Assessment 1. We will now discuss the school-level monitoring and supervision components of Tusome, including tablet-centered observations, coaching, pupil assessment, and reporting. Do you think such USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 166 monitoring activities are critical to the continued long-term use of Tusome materials and methods? Why or why not? a. What specific components are the most critical? The least critical? 2. What has been or is currently being done to ensure the continuation of Tusome related monitoring and supervision activities after RTI's contract ends? a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to the continuation of observations, coaching, pupil assessments, and/or reporting? c. Do these or other Tusome-related monitoring policies apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to implement these procedures/policies/guidelines after RTI's contract ends? f. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 3. Did RTI play a direct role in the formulation and/or adoption of these changes? If so, what did this role look like? a. Who specifically at RTI was involved and at what stages (e.g., procedures/policy/guideline formulation, review, adoption)? What specifically did they do? b. Who were the non-RTI contributors? What was the scope of their contribution? c. How pivotal was RTI's role in the procedure/policy/guideline change? I.e., if RTI staff had not been involved, would the change have occurred? 4. Do you believe that Tusome related monitoring and supervision by CSOs, QASOs, and others will continue over the long-term? Why or why not? a. Are any of these components more likely than the others to continue? Which components and why? b. Do you think monitoring will continue with the same volume/frequency? Why or why not? c. Do you think monitoring and supervision will be of the same quality over the longer term? Why or why not? 5. In your opinion, what are the biggest challenges or threats to the continuation of Tusome-specific monitoring and supervision activities? As a reminder, these activities include tablet-centered observations, coaching, pupil assessment, and reporting. a. At the national level? County-level? Sub-county level? School-level? b. Are there anticipated challenges related to continued training/capacitation of CSOs/QASOs to conduct observations/coaching/assessment? What do these challenges look like? c. Are there anticipated challenges related to long-term maintenance of equipment/hardware and software for monitoring? d. Are there anticipated challenges in ensuring monitors have adequate time and transportation for school-based monitoring? 6. What, if any, policies specifically related to pupil assessment and pupil progress monitoring have been made as a result of, or in response to, the Tusome activity? a. What specific data are collected on pupils' reading progress? How are these data analyzed and interpreted? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 167 b. How are these data used? How do they inform decision-making at the national, county, sub-county, school, or classroom levels? 7. Do you think these assessment policies are having an effect on Kenya's broader mandate to promote reading skills? Why or why not? 8. Is the proper use of materials and methods developed under Tusome formally factored in to teacher assessment and performance appraisal? If so, in what ways? a. For PRESET? INSET? b. Does this vary across counties, sub-counties, and schools? If so, in what ways? c. What, if any, enforcement mechanisms exist? d. [If yes] What happens when teachers are not properly using the Tusome materials and methods? 9. Did RTI play a direct role in the formulation and/or adoption of teacher assessment and performance appraisal policies? If so, what did this role look like? a. Who specifically at RTI was involved and at what stages (e.g., policy formulation, review, adoption)? What specifically did they do? b. Who were the non-RTI contributors? What was the scope of their contribution? c. How pivotal was RTI's role in the policy change? I.e., if RTI staff had not been involved, would the policy change have occurred? Section E: General Sustainability Now I would like to discuss questions about the long-term sustainability of Tusome. These questions are targeted specifically to USAID and RTI respondents to better understand the future of Tusome. 1. Of all the changes in GoK procedures/policies/guidelines we have discussed or not discussed, which do you think are most critical to the long-term sustainability of the approaches and best practices established under the Tusome activity? 2. Beyond what has already been discussed, do you think the rollout of the new Competency Based Curriculum (CBC) will influence the continued long-term use of Tusome materials and methods? In what ways? a. What are the positive influences? Negative influences? b. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? c. [If negatively influencing sustainability] Could it have been avoided in your opinion? If so, how? d. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 3. Beyond what has already been discussed, do you think county-level authorities will influence the long-term sustainability of the approaches and best practices established under the Tusome activity? In what ways? a. What are the positive influences? Negative influences? b. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? c. [If negatively influencing sustainability] Could it have been avoided in your opinion? If so, how? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 168 d. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 4. Have there been any GoK procedures/policies/guidelines put forth that undermine the continued use of materials and methods developed under Tusome? a. What is the status of such procedures/policies/guidelines? Have they been adopted? Are they likely to be adopted in the future? b. What is motivating these procedures/policies/guidelines? c. What do you think can be done to ensure they do not undermine the continued use of materials and methods developed under Tusome? 5. What do you think is the biggest threat to Tusome's sustainability after RTI's contract ends? a. Was this threat known from the outset or did it only come to light after Tusome's design/implementation? b. Could this threat have been predicted or foreseen? c. Could it have been avoided? If so, how? d. How can it be minimized or avoided at this juncture? 6. What do you think is the best hope for Tusome's sustainability after RTI's contract ends? 7. Besides what has already been discussed, are there other key “lessons learned” during the Tusome activity that you would like to share? Section F: Youth Fund Grants 1. In your own words, can you briefly describe the Youth Fund grants program? a. What is the main goal or objective of the Youth Fund? b. What are the main activities, inputs, and outputs of the Youth Fund? 2. Do you believe that grantee activities tangibly increased community awareness related to reading? Why or why not? a. In your opinion, what types of grantee activities were the most effective in increasing community awareness related to reading? b. What types of activities were the least effective in increasing community awareness? Why? c. Are any increases in community awareness likely to be sustained over time? Why or why not? 3. Do you believe that grantee activities tangibly increased children's engagement in reading? Why or why not? a. In your opinion, what types of grantee activities were the most effective in increasing children's engagement in reading? b. What types of activities were the least effective in increasing children's engagement in reading? Why? c. Are any increases in children's engagement in reading likely to be sustained over time? Why or why not? 4. Did the Youth Fund program have any other effects, intended or unintended? a. [If Yes] Please describe. Were they positive or negative? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 169 5. Do you believe that the Youth Fund is an effective approach for increasing community awareness and engagement in reading? a. Why or why not? b. Do you think there are other ways to use Youth Fund resources to better achieve the stated objectives? If so, please describe. 6. Did you face any significant challenges in implementing Youth Fund activities? If so, what challenges did you face and why? a. How did you overcome these challenges? b. Could these challenges have been anticipated or avoided? If so, how? 7. If the Youth Fund were to continue into the future, what would you like to see done differently? Section G: Closing 1. On a scale of 0 to 4, how successful do you think the Tusome activity was in improving government capacity to sustainably improve reading outcomes (IR 2)? Let's assume 0 is "not at all" and 4 is "to the fullest extent possible." a. Why did you choose this rating? b. [If below 4] What could have been done differently to rate the achievement of this objective as a "4"? c. [If below 4] What, if anything, do you think can still be done by December to improve this rating? 2. Are there any additional comments that you would like to share with us? 3. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 170 NATIONAL TECHNICAL TEAM FGD GUIDE Field Control Interview date: ___/____/_______ (DAY/MONTH/YEAR) Interviewer name: _______________ Questions Section A: Introduction 4. Can you describe the role of the National Technical Team (NTT) in the Tusome activity? 5. One of Tusome's explicit objectives is to create an improved the policy environment that promotes reading skills. Has NTT played a direct role in influencing the policy environment in this manner? a. If so, what role has NTT played? 6. In your own words, can you briefly describe the Tusome approach to improving English and Kiswahili skills in the early grades? a. In what ways did Tusome change how English and Kiswahili skills are taught in Kenyan schools in the early grades? Section B: Teaching and Learning Materials (TLM) 6. Now I would like to discuss changes in GoK procedures, policies, or guidelines related to teaching and learning materials (TLM). Can you describe any TLM-related changes in GoK procedures, policies, or guidelines that have been made as a result of, or in response to, the Tusome activity? a. How and when did these changes first come into existence? Are they still in effect today? b. Do these or other Tusome-related TLM policies apply to Alternative Provision of Basic Education and Training (APBET) schools? Special Needs Education (SNE)? c. Are there specific procedures/policies/guidelines related to pricing and procurement of TLM? d. At what level(s) do these policies apply? The national level? County-level? Sub-county level? e. Are these GoK procedures/policies/guidelines enforceable? How are/will they be enforced? f. Have sufficient human and/or financial resources been put into place to implement TLM-related procedures/policies/guidelines after USAID funding ends? i. [if yes] Can you describe the process and strategies for ensuring human/financial resources were put into place? ii. [If no] Why do you think they have not been put into place? Are they likely to be put into place in the future? Why or why not? iii. [If no] Going forward, what types of workable solutions do you recommend to ensure that sufficient human and/or financial resources are put into place? g. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 7. Did USAID’s partnership with the GoK play a direct role in the formulation and/or adoption of these procedure/policy/guideline changes? If so, what was their role? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 171 a. Who specifically from USAID (USAID education office, implementing partner, etc.) was involved and at what stages (e.g., formulation, review, adoption)? What specifically did they do? b. Who were the non-USAID contributors? What was the scope of their contribution? c. How pivotal was USAID’s partnership with GoK in these change? d. Do you have any other reflections on their involvement? 8. How important do you think these GoK procedures/policies/guidelines are for the continued long￾term use of the materials and methods developed under Tusome? a. If they were revoked or not enforced, how would this affect the continued implementation of Tusome? 9. Do you feel these GoK procedures/policies/guidelines will remain in place over the long term? Why or why not? a. What are the biggest threats to their long-term sustainability? b. What more do you think can be done before the end of the USAID funding to ensure they remain in effect over the long term? c. What do you think should be done (and by whom) after the USAID funding finishes to ensure they remain in effect over the long term? 10. What additional procedures/policies/guidelines do you think are critical to ensuring the continued, long-term use of the materials and methods developed under Tusome? a. Policies at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these policies? c. What are the key barriers to the formulation and adoption of such policies? Section C: Teacher Training and Professional Development 9. We will now talk about integration of the Tusome materials and methods into teacher training and professional development, starting with pre-service training (PRESET). Can you tell me about any changes in Primary Teacher Education (PTE) that have been made as a result of, or in response to, the Tusome activity? a. Are these changes formal? Have they been formally adopted or mandated by MOE, TSC, SAGAs, and/or PTTCs? Why or why not? b. Are all Primary Teacher Training Colleges (PTTCs) affected in the same manner? c. When did/will these changes go into effect? Are they still in effect today? d. Are these changes required? i. [If so] what are the enforcement mechanisms? ii. [If not] What (if any) are the mechanisms for increasing and sustaining take-up? e. Do you know if sufficient human and/or financial resources have been put into place to implement these changes? i. [If yes] Can you describe the process and strategies for ensuring human/financial resources were put into place? ii. [If no] Why do you think they have not been put into place? Are they likely to be put into place in the future? Why or why not? iii. [If no] Going forward, what types of workable solutions do you recommend to ensure that sufficient human and/or financial resources are put into place? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 172 f. Are any of these changes formally documented? Can this documentation be shared with the evaluation team? 10. In your opinion, how critical is PTE curriculum integration to continued, long-term use of the materials and methods developed under Tusome? Why? 11. What challenges have been or are being faced in integrating Tusome into PTE? How might these challenges be overcome? a. Are there unique challenges associated with different PTE actors? b. Are there challenges related to coordination between these multiple actors? 12. Do you think the in-service teacher trainings (INSET) are critical to continued, long-term use of the materials and methods developed under Tusome? Why or why not? a. How important is the duration and frequency of in-service trainings? b. What duration and frequency is ideal in your opinion? 13. What has been or is currently being done to ensure the continuation of INSET on the materials and methods developed under Tusome after USAID funding ends? a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to continuation of INSET on the materials and methods developed under Tusome? c. Do these or other Tusome-related INSET procedures/policies/guidelines apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to continue INSET on the materials and methods developed under Tusome? i. [if yes] Can you describe the process and strategies for ensuring human/financial resources were put into place? ii. [If no] Why do you think they have not been put into place? Are they likely to be put into place in the future? Why or why not? iii. [If no] Going forward, what types of workable solutions do you recommend to ensure that sufficient human and/or financial resources are put into place? f. Are any of these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? g. Did USAID’s partnership with the GoK play a direct role in the formulation and/or adoption of these procedures/policies/guidelines? If so, what was their role? 14. Do you think formal INSET on the materials and methods developed under Tusome will continue over the long term? Why or why not? a. Will this vary across counties, sub-counties, and schools? If so, in what ways? b. [If yes] Do you think the quality of INSET will remain the same? Why or why not? 15. What more could be done to ensure the continuation of formal INSET on the materials and methods developed under Tusome over the long term? a. At the national level? County-level? Sub-county level? School-level? b. What more do you think can be done before the end of USAID funding to ensure they remain in effect over the long term? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 173 c. What do you think should be done (and by whom) after USAID funding finishes to ensure they remain in effect over the long term? 16. What additional procedures/policies/guidelines do you think are critical to ensuring teachers continue to get trained on Tusome methods and materials? a. Procedures/policies/guidelines at the national level? County-level? Sub-county level? b. What, if anything, has been done to formulate or adopt these additional procedures/policies/guidelines? c. What are the key barriers to the formulation and adoption of these procedures/policies/guidelines? Section D: Monitoring, Supervision, and Assessment 10. We will now discuss the school-level monitoring and supervision components of Tusome, including tablet-centered observations, coaching, pupil assessment, and reporting. Do you think such monitoring activities are critical to the continued long-term use of Tusome materials and methods? Why or why not? a. What specific components are the most critical? The least critical? 11. What has been or is currently being done to ensure the continuation of Tusome-related monitoring and supervision activities after USAID funding ends? a. At the national level? County-level? Sub-county level? School-level? b. Are there any formal procedures/policies/guidelines in place related to the continuation of observations, coaching, pupil assessments, and/or reporting? c. Do these or other Tusome-related monitoring policies apply to APBET institutions? SNE? d. Are these procedures/policies/guidelines enforceable? How are/will they be enforced? e. Have sufficient human and/or financial resources been put into place to implement these procedures/policies/guidelines after USAID funding ends? i. [if yes] Can you describe the process and strategies for ensuring human/financial resources were put into place? ii. [If no] Why do you think they have not been put into place? Are they likely to be put into place in the future? Why or why not? iii. [If no] Going forward, what types of workable solutions do you recommend to ensure that sufficient human and/or financial resources are put into place? f. Are these procedures/policies/guidelines formally documented? Can this documentation be shared with the evaluation team? 12. Did USAID’s partnership with the GoK play a direct role in the formulation and/or adoption of these changes? If so, what was their role? a. Who specifically from USAID (USAID education office, implementing partner, etc.) was involved and at what stages (e.g., formulation, review, adoption)? What specifically did they do? b. Who were the non-USAID contributors? What was the scope of their contribution? c. How pivotal was USAID's partnership with the Gok in the procedure/policy/guidelines change? 13. Do you believe that Tusome-related monitoring and supervision by CSOs, QASOs, and others will continue over the long term? Why or why not? a. Are any of these components more likely than the others to continue? Which components and why? b. Do you think monitoring will continue with the same volume/frequency? Why or why not? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 174 c. Do you think monitoring and supervision will be of the same quality over the long term? Why or why not? 14. In your opinion, what are the biggest challenges to the continuation of Tusome-specific monitoring and supervision activities? As a reminder, these activities include tablet-centered observations, coaching, pupil assessment, and reporting. a. At the national level? County-level? Sub-county level? School-level? b. Are there anticipated challenges related to the continued training/capacity building of CSOs/QASOs to conduct observations/coaching/assessment? What are these challenges? c. Are there anticipated challenges related to long-term maintenance of equipment/hardware and software for monitoring? d. Are there anticipated challenges in ensuring monitors have adequate time and transportation for school-based monitoring? 15. What, if any, policies specifically related to pupil assessment and pupil progress monitoring have been made as a result of, or in response to, the Tusome activity? a. What specific data are collected on pupils' reading progress? How are these data analyzed and interpreted? b. How are these data used? How do they inform decision-making at the national, county, sub-county, school, or classroom levels? 16. Do you think these assessment policies are having an effect on Kenya's broader mandate to promote reading skills? Why or why not? 17. Is the proper use of materials and methods developed under Tusome formally factored into teacher assessment and performance appraisal? If so, in what ways? a. For PRESET? INSET? b. Does this vary across counties, sub-counties, and schools? If so, in what ways? c. What, if any, enforcement mechanisms exist? d. [If yes] What happens when teachers are not properly using the Tusome materials and methods? 18. Did USAID’s partnership with the GoK play a direct role in the formulation and/or adoption of teacher assessment and performance appraisal policies? If so, what was their role? a. Who specifically from USAID (USAID education office, implementing partner, etc.) was involved and at what stages (e.g., policy formulation, review, adoption)? What specifically did they do? b. Who were the non-USAID contributors? What was the scope of their contribution? c. How pivotal was USAID’s partnership with the GoK in the policy change? Section E: General Sustainability 8. Of all the changes in GoK procedures/policies/guidelines we have discussed or not discussed, which ones do you think are most critical to the long-term sustainability of the approaches and best practices established under the Tusome activity? 9. Beyond what has already been discussed, do you think the rollout of the new Competency Based Curriculum (CBC) will influence the continued long-term use of Tusome materials and methods? In what ways? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 175 a. What are the positive influences? Negative influences? b. [If positively influencing sustainability] In what ways has the rollout of CBC positively influenced the sustainability of Tusome? c. [If positively influencing sustainability] Do you think CBC improved the implementation of Tusome, as compared to before? Why or why not? d. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? e. [If negatively influencing sustainability] In your opinion, could it have been avoided? If so, how? f. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 10. Beyond what has already been discussed, do you think county-level authorities will influence the long-term sustainability of the approaches and best practices established under the Tusome activity? In what ways? a. What are the positive influences? Negative influences? b. [If positively influencing sustainability] In what ways have county-level authorities positively influenced the continued long-term use of Tusome materials and methods? c. [If negatively influencing sustainability] In what ways have county-level authorities negatively influenced the continued long-term use of Tusome materials and methods? d. [If negatively influencing sustainability] Could this negative influence have been predicted or foreseen? e. [If negatively influencing sustainability] In your opinion, could it have been avoided? If so, how? f. [If negatively influencing sustainability] How can it be minimized or avoided at this juncture? 11. Besides those already discussed, have there been any GoK procedures/policies/guidelines put forth that support the continued use of materials and methods developed under Tusome? a. What is the status of such procedures/policies/guidelines? Have they been adopted? Are they likely to be adopted in the future? b. What is informing these procedures/policies/guidelines? c. What do you think can be done to ensure they support the continued use of materials and methods developed under Tusome? 12. Besides those already discussed, have there been any GoK procedures/policies/guidelines put forth that undermine the continued use of materials and methods developed under Tusome? a. What is the status of such procedures/policies/guidelines? Have they been adopted? Are they likely to be adopted in the future? b. What is informing these procedures/policies/guidelines? c. What do you think can be done to ensure they do not undermine the continued use of materials and methods developed under Tusome? 13. What do you think is the best hope for the continued, long-term use of the materials and methods developed under Tusome after USAID funding ends? 14. What do you think is the biggest threat to the continued, long-term use of the materials and methods developed under Tusome after USAID funding ends? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 176 a. Was this threat known from the outset or did it only come to light after Tusome's design/implementation? b. Could this threat have been predicted or foreseen? c. Could it have been avoided? If so, how? d. How can it be minimized or avoided at this juncture? 15. Are you a member of the Tusome Sustainability and Transition Team? If so, can you briefly describe any activities the team has undertaken or plans to undertake to ensure the continuation of Tusome? Section F: Closing 4. Are there any additional comments that you would like to share with us? 5. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 177 NATIONAL GOVERNMENT KII GUIDE Field Control Interview date: ___/____/_______ (DAY/MONTH/YEAR) Interviewer name: _______________ Questions 1. Can you describe your role at [Department/Institute]? a. How long have you been in this role? b. [If less than 3 years] What were you doing prior to taking on this role? 2. What role--direct or indirect--does [Department/Institute] play in English and Kiswahili language instruction policy and planning in Kenya? 3. In your own words, can you briefly describe the Tusome early grade reading programme? 4. What role did [Department/Institute] play in the creation and launch of the Tusome early grade reading programme? 5. What role, if any, has [Department/Institute] played in the ongoing implementation of Tusome? a. Has this required significant contributions of [Department/Institute]'s own human and financial resources? If so, please describe. b. [If yes to a] Has this required a net increase in [Department/Agency's] human and financial resources? If so, please describe. c. [If yes to a] Has this required that resources be internally reallocated? What other department/agency activities were affected by this reallocation? Please briefly describe. 6. Since the launch of the programme in 2015, has [Department/Agency] established any new procedures/policies/guidelines in support of Tusome? If so, please describe them. a. How and when did these changes first come into existence? Are they still in effect today? b. Are these changes enforceable? How are/will these changes be enforced? c. Are these changes formally documented? Can this documentation be shared with the evaluation team? 7. Have sufficient human and/or financial resources been put into place to implement these procedures/policies/guidelines? Why or why not? a. Is the level of human and/or financial support for these procedures/policies/guidelines likely to change in the future? Why or why not? b. [If yes to a] How is it likely to change in the future? 8. Did USAID or its implementing partner (RTI) support [Department/Agency] in formulating these new procedures/policies/guidelines? If so, in what ways? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 178 a. Who at USAID or RTI was involved and at what stages (e.g., formulation, review, adoption)? b. What specifically did they do to support [Department/Agency] in formulating these procedures/policies/guidelines? c. How pivotal was USAID in implementing the changes? d. Do you have any other reflections on their involvement? 9. Did the Tusome National Technical Team (NTT) support [Department/Agency] in formulating these new procedures/policies/guidelines? If so, in what ways? 10. To what extent are these procedures/policies/guidelines being implemented today? a. What are the factors contributing to implementation? What people or activities are "making it happen"? b. If they are not being implemented at all, why not? c. If they are only being partially implemented, what sorts of things are preventing full implementation? d. Does implementation vary across counties, sub-counties, and schools? If so, in what ways? 11. What changes, if any, have you observed as a result of these new procedures/policies/guidelines? a. What positive effects have you observed? At the department/agency level? County-level? Sub-county level? School/pupil-level? b. What negative effects have you observed? c. Do these effects vary across counties, sub-counties, and schools? If so, in what ways? d. Has there been any resistance as a result to these procedures/policies/guidelines? If so, from whom and why? 12. Do you think these new procedures/policies/guidelines will remain in place over the long term? Why or why not? a. Will this vary across counties, sub-counties, and schools? If so, in what ways? b. What future challenges do you anticipate in implementing these changes? 13. How important do you think these procedures/policies/guidelines are for the continued long-term use of the materials and methods developed under Tusome? a. If they were revoked or not enforced, how would this affect the continued implementation of Tusome? 14. Is [Department/Institute] undertaking any other procedure/policy/guideline initiatives to ensure that the materials and methods developed under Tusome continue to be used after USAID funding finishes? If so, please describe. a. What do you think is the likelihood that these initiatives will go through? b. What do you think are the biggest challenges to approval of these initiatives? c. What will enforcement of these initiatives look like? What types of challenges will be encountered in their enforcement? d. Is implementation likely to vary across counties, sub-counties, and schools? If so, in what ways? e. What positive and/or negative effects do you think these initiatives will have, and why? 15. Have USAID and its implementing partner (RTI) supported [Department/Agency] in formulating these initiatives to ensure the continued use of Tusome materials and methods beyond 2019? If so, in what ways? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 179 a. Who at USAID or RTI have been involved and in what ways? b. How pivotal has USAID or RTI’s role been in the process? c. Do you have any other reflections on their involvement? 16. The Tusome programme involves a number of activities, including teacher training on Tusome methods, the production and distribution of Tusome teaching and learning materials, and school￾based monitoring and supervision of Tusome implementation. Do you believe that these activities will continue beyond 2019? Why or why not? a. In your opinion, which of these activities are the most likely to continue beyond 2019? Why? b. In your opinion, which of these activities are the least likely to continue beyond 2019? Why? c. Will continuation of these activities vary across counties, sub-counties, and schools? If so, how? d. Please describe any other challenges to the continuation of these activities beyond 2019. 17. Are there any other procedures/policies/guidelines that [Department/Agency] could undertake to ensure the continuation of these Tusome activities? If so, please describe. 18. Would you say there is broad institutional support for the continued use of the materials and methods developed under Tusome at [Department/Agency]? Why or why not? a. Does support vary by departments or staff? If so, how? b. Does support for the different project components vary? In what ways? c. What types of things could lead to changes in the current level of institutional support? How and why would these changes occur? d. [If yes] Will this support continue after USAID funding finishes in December 2019? 19. Are there any procedures/policies/guidelines that other government departments or agencies in Kenya could undertake to ensure the continuation of these Tusome activities? This can include both education and non-education departments/agencies. If so, please describe. a. Do you think these procedures/policies/guidelines are likely to be put forward? Why or why not? b. Would you say there is broad institutional support for the Tusome programme at these other departments and agencies? Why or why not? 20. What do you think is the biggest hope for the continued, long-term use of the materials and methods developed under Tusome after USAID funding finishes in December 2019? 21. What do you think is the biggest threat to the continued, long-term use of the materials and methods developed under Tusome after USAID funding finishes in December 2019? 22. Besides what has already been discussed, are there other key “lessons learned” during the Tusome activity that you would like to share? 23. Would you personally like to see the materials and methods developed under Tusome continue beyond 2019? Why or why not? a. Are there certain components that you would like to see continued more than others? Which components and why? b. Are there certain components that you would like to see discontinued? Which components and why? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 180 24. Are there any additional comments that you would like to share with us? 25. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 181 COUNTY LEVEL KII GUIDE Field Control Interview date: ___/____/_______ (DAY/MONTH/YEAR) Interviewer name: _______________ Questions 1. Can you briefly describe your main responsibilities as [CDE/TSC-DC]? 2. Are you familiar with the Tusome Early Grade reading programme? If so, please briefly describe it in your own words. 3. As [CDE/TSC-CD], what role(s) have you played in implementing the Tusome Early Grade reading programme in this county? a. Have you played any indirect roles (e.g., supervising persons tasked with implementation)? 4. Do you believe that the Tusome teaching methods and materials have helped to improve English and Kiswahili language skills of children in this county? Why or why not? a. How do you know whether they have helped or not? b. Are there specific components of the programme that are helping more than others? What are these? c. [If they have not helped] Why do you think Tusome isn't improving English and Kiswahili language skills? 5. Are you aware of any GoK procedures, policies, or guidelines related to use of the teaching methods and materials developed under Tusome? If so, please describe these policies. a. [If no] Are you aware of any other GoK procedures/policies/guidelines related to teaching of English and Kiswahili language activities? If so, what are these? b. [If yes] Do you believe these policies are being followed in your county? Why or why not? i. How do you know whether or not they are being followed? ii. [If not] Why are they not being followed? iii. What, if anything, can be done to ensure they are followed in the future? 6. In 2019, has your county supported, facilitated, or participated in any in-service teaching trainings (INSET) on the use of the teaching methods and materials developed under Tusome? a. If so, please describe (training content, facilitators, participants, volume/frequency, etc.). b. Is there anyone who should have participated in the training that did not? Why didn’t they participate? c. Did you receive help from persons, organizations, or institutes outside of the county? E.g., NGOs, MoE, KICD? Please describe, including listing relevant institutions. 7. Do you believe your county will support INSET on the materials and methods developed under Tusome in the future? Why or why not? a. If so, please describe (training content, facilitators, participants, volume/frequency, etc.). USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 182 b. Who do you think will participate in this training in the future? c. What, if any, outside support do you think will be needed to ensure the training goes well? 8. Are Curriculum Support Officers (CSOs) in this county required to observe and coach teachers on proper use of the teaching methods and materials developed under Tusome? a. What specific lessons are they required to observe? b. What specifically are they supposed to do before, during, and after these observations? c. How many times per term? d. Are they currently doing this? If not, why? 9. Do you believe that the CSO observations lead to improvements in teachers' ability to teach English and Kiswahili language activities in this county? Why or why not? a. How do you know whether they lead to improvements? b. [If not] why do you think they do not lead to improvements? c. What can be done to ensure these observations lead to teaching improvements in the future? 10. Do you think CSOs in this county are generally supportive of the methods and materials developed under Tusome today? Why or why not? a. Does support vary by position? By zone? If so, how? Why? b. Are there specific components that they support more or less than others? Please explain. c. [If no] Have CSOs in this county ever been supportive of the Tusome programme? If not, why? If so, what changed? 11. Are Quality Assurance and Standards Officers (QASOs) in this county required to observe and coach teachers on proper use of the materials and methods developed under Tusome? a. What specifically are they supposed to do before, during, and after these observations? b. How many times per term? c. Are they currently doing this? If not, why? 12. Do you believe that the QASO observations lead to improvements in teachers' ability to teach English and Kiswahili language activities in this county? Why or why not? a. How do you know whether they lead to improvements? b. [If not] why do you think they do not lead to improvements? c. What can be done to ensure these observations lead to teaching improvements in the future? 13. Do you think QASOs in this county are generally supportive of the methods and materials developed under Tusome today? Why or why not? a. Does support vary by position? By zone? If so, how? Why? b. Are there specific components that they support more or less than others? Please explain. c. [If no] Have QASOs in this county ever been supportive of the Tusome programme? If not, why? If so, what changed? 14. Are you currently receiving any data or reports on teacher observation and coaching related to Tusome? If so, please describe these data and where they came from. a. Have you ever accessed the Tusome dashboard? i. [If yes] How often do you access the Tusome dashboard? For what purpose? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 183 ii. [If no] Why not? 15. Are you currently receiving any data or reports on pupil reading performance at schools in your county? If so, please describe. 16. Do you use these data or reports for planning or decision-making? If so, how? a. In what ways do you think this data will be used for decision-making in the future? 17. The Tusome programme involves a number of activities, including teacher training on Tusome methods, the production and distribution of Tusome teaching and learning materials, and school￾based monitoring and coaching on Tusome implementation. Do you believe that these activities will continue in this county in the future? Why or why not? a. In your opinion, which of these activities are the most likely to continue? Why? b. In your opinion, which of these activities are the least likely to continue? Why? c. Will continuation of these activities vary across sub-counties and schools? If so, how? d. Please describe any other challenges to the continuation of these activities in the future. 18. Has your county engaged in joint work planning in 2018 or 2019? If so, please describe this process. a. When did it occur? Who was involved? What were the key outputs? b. Were there any major successes in this process? c. Were any major challenges encountered in this process? 19. [If yes to 18] Is implementation of the joint work plan being monitored? If so, please describe the process for monitoring joint work plan implementation. 20. [If yes to 18] In what ways, if any, does the joint work plan support continuation of specific Tusome activities over the long-term? 21. Are there any other procedures, policies, or guidelines that this county could undertake to ensure the continuation of Tusome activities? If so, please describe. 22. What are the main challenges your county faces in ensuring the continuation of Tusome activities? a. Do challenges vary by component? If so, how? b. How might each of these challenges be overcome? 23. Are you personally supportive of the methods and materials developed under Tusome? Why or why not? 24. Are there any additional comments that you would like to share with us? 25. Thank you very much for taking the time to answer our questions. Do you have any questions for our team? USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 184 ANNEX VI: DISCLOSURE OF CONFLICTS OF INTEREST Name Erika Keaveney Title Team Leader Organization NORC at the University of Chicago Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome External Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08/19/2021 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 185 Name Carlos Fierros Title Principal Research Analyst Organization NORC at the University of Chicago Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome External Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08/19/2021 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 186 Name Alexander Rigaux Title Principal Research Analyst Organization NORC at the University of Chicago Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome External Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08/19/2021 USAID.GOV TUSOME EXTERNAL EVALUATION, ENDLINE REPORT | 187 Name Munene Charles Kiura Title Education Evaluation Specialist Organization NORC at the University of Chicago Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) GS-10F-0033M/AID-OAA-M-13-00010 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Tusome External Evaluation I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 08/19/2021