1000 Thomas Jefferson Street NW Washington, DC 20007-3835 202.403.5000 www.air.org COPYRIGHT © 2017 AMERICAN INSTITUTES FOR RESEARCH. ALL RIGHTS RESERVED. READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT Final Performance Evaluation February 2018 Addis Ababa, Ethiopia READ Monitoring and Evaluation CREDIT: READ M&E FIELD OFFICE TEAM READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT Final Performance Evaluation February 2018 Addis Ababa, Ethiopia READ Monitoring and Evaluation Submitted to: U.S. Agency for International Development/Ethiopia Addis Yigzaw, COR Contracting Officer’s Representative Prepared by: American Institutes for Research 1000 Thomas Jefferson Street, NW Washington, DC 20007 Under Contract No. AID-663-C-15-00001 This report was made possible by the support of the American People through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of American Institutes for Research (AIR) and do not necessarily reflect the views of USAID or the United States Government. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | III ACRONYMS AND INITIALS AIR American Institutes for Research CAEB City Administration Education Bureau CTE College of Teacher Education ICT Information and communications technology IR Intermediate result MLCs Minimum Learning Competencies MoE Ministry of Education READ CO Read Ethiopia Achievement Developed Community Outreach READ II Read Ethiopia Achievement Developed Institutional Improvement READ M&E Read Ethiopia Achievement Developed Monitoring and Evaluation READ TA Read Ethiopia Achievement Developed Technical Assistance RSEB Regional State Education Bureau RTI Research Triangle Institute SNNPR Southern Nations, Nationalities, and People’s Region USAID United States Agency for International Development WEO Woreda Education Office ZED Zone Education Department USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | V CONTENTS ACRONYMS AND INITIALS III EXECUTIVE SUMMARY VII 1. INTRODUCTION 1 1.1. READ TA IMPLEMENTATION 2 1.2. THEORY OF CHANGE 3 1.3. EVALUATION QUESTIONS 5 2. METHODOLOGY 7 2.1. SAMPLING OF RESPONDENTS 7 2.2. DATA COLLECTION PROCEDURES 10 2.3. DATA ANALYSIS 10 2.4. ETHICS REVIEW 11 3. FINDINGS 11 3.1. IR1. READING AND WRITING MATERIALS APPROPRIATE FOR PRIMARY CLASSROOMS AND PRESERVICE AND IN-SERVICE TEACHER TRAINING DEVELOPED 11 3.2. IR2: LANGUAGE-SPECIFIC TEACHING AND LEARNING METHODOLOGIES AND STRATEGIES THAT FOCUS ON HELPING STUDENTS LEARN TO READ AND WRITE EFFECTIVELY ARE APPLIED 27 3.3. IR3: LANGUAGE TEACHING AND LEARNING SUPPORTED BY APPROPRIATE TECHNOLOGY AND TEACHER AIDS 33 3.4. IR4: TECHNICAL SUPPORT TO RSEBS AND MOE FOR READ INSTITUTIONAL IMPROVEMENT 36 4. LIMITATIONS AND CHALLENGES OF THE PROJECT 39 4.1. REGISTRATION 39 4.2. WORKING WITH MOE AND RSEBS 39 4.3. DELAY OF PLANNED ACTIVITIES 40 4.4. DIVISION OF ROLES AND RESPONSIBILITIES 40 4.5. DATA ACCURACY AND TIMING 41 4.6. MATERIAL DEVELOPMENT AND REVISION 41 4.7. TEACHER TRAINING 41 5. PROJECT SUSTAINABILITY AND LESSONS LEARNED 42 6. CONCLUSION 42 7. LESSONS LEARNED 44 ANNEX A. CLASSROOM OBSERVATION CHECKLIST A–1 ANNEX B. INTERVIEW GUIDE FOR PRINCIPALS AND SUPERVISORS B–1 ANNEX C. INTERVIEW GUIDE FOR CTE STAFF C–1 ANNEX D. INTERVIEW GUIDE FOR EXPERTS FROM MOE AND RSEBS D–1 VI | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV ANNEX E. INTERVIEW GUIDE FOR MATERIAL DEVELOPERS AND REVIEWERS E–1 ANNEX F. INTERVIEW GUIDE FOR TEACHERS F–1 ANNEX G. QUESTIONNAIRE FOR TEACHERS G–1 ANNEX H. INTERVIEW GUIDE FOR EXPERTS FROM READ TA H–1 LIST OF TABLES TABLE 1: EVALUATION QUESTIONS 5 TABLE 2: DATA SOURCES AND EVALUATION INSTRUMENTS 7 TABLE 3: NATIONAL-LEVEL DATA COLLECTION (COVERING MOE INTERVIEWS AND READ TA STAFF INTERVIEWS) 8 TABLE 4: REGION-LEVEL DATA COLLECTION (COVERING RSEB/ ZONE EXPERTS, MATERIAL DEVELOPMENT/REVIEW EXPERTS, READ TA STAFF, AND CTE STAFF) 8 TABLE 5: SAMPLE SIZE FOR SCHOOL-LEVEL DATA COLLECTION 9 TABLE 6: THE DISTRIBUTION OF GRADES 1–4 STUDENT-TO-TEXTBOOK RATIO BY LANGUAGE 22 TABLE 7: TEXTBOOK-TO-STUDENT RATIO BY GRADE AND LANGUAGE 22 LIST OF FIGURES FIGURE 1. READ TA THEORY OF CHANGE 4 FIGURE 2. THE DISTRIBUTION OF WHETHER EVERY STUDENT IN GRADES 1–4 OWN TEXTBOOK BY LANGUAGE 21 FIGURE 3. FREQUENCY OF BRINGING MOTHER-TONGUE TEXTBOOKS TO THE CLASS (GRADES 1–4) 23 FIGURE 4. FREQUENCY OF USE OF TEACHER’S GUIDE DURING TEACHING (GRADE 1–4) 24 FIGURE 5. OBSERVATION DATA FOR TEACHERS’ USE OF MOTHER TONGUE MATERIALS DURING INSTRUCTION (GRADES 1–4) 25 FIGURE 6. TEACHERS’ LEVEL OF SATISFACTION OF THE CONTENTS OF THE NEW MOTHER TONGUE MATERIALS (GRADES 1–4) 28 FIGURE 7. TEACHER’S OVERALL SATISFACTION ON THE LINK BETWEEN CONTENTS AND EMBEDDED METHODOLOGY BY LANGUAGE 29 FIGURE 8. TEACHERS’ OVERALL SATISFACTION OF THE QUALITY OF THE NEW MATERIALS BY LANGUAGE (GRADES 1–4) 30 USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | VII EXECUTIVE SUMMARY Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation (READ M&E) is a 5-year (January 2015–December 2019) United States Agency for International Development (USAID)- supported project implemented by American Institutes for Research (AIR). READ M&E is one of the four projects on early-grade reading launched by USAID Ethiopia. READ M&E implements Early Grade Reading Assessment (EGRA) in seven mother tongues, and conducts performance evaluations of the USAID READ projects, namely READ Technical Assistance [READ TA], READ Institutional Improvement [READ II], and READ Community Outreach [READ CO]. In this document, READ M&E presents the end-line performance evaluation of the READ Technical Assistance (READ TA) project. READ TA is a 5-year (January 2012–October 2017) project aimed at improving reading and writing in seven mother-tongue languages in collaboration with the Ministry of Education (MoE), the Regional State Education Bureaus (RSEBs), City Administration Education Bureaus (CAEBs), Colleges of Teacher Education (CTEs), and Woreda Education Offices (WEOs). The project supported the revision and implementation of curricula for Grades 1-8 in seven targeted mother-tongue languages (Amharic, Afaan Oromo, Af-Somali, Hadiyyisa, Sidaama Afoo, Tigrinya, and Wolayttatto); and revised the English language curriculum for Grades 1–8 with the goal of using English for academic purposes. Given the extremely low reading scores on the 2010 EGRA (USAID/RTI, 2010) READ TA’s mandate was to revise the existing mother-tongue curriculum; develop new evidence-based student textbooks and teacher guides to be more appropriate for developing students’ reading skills; conduct teacher professional development through a series of trainings; and support CTEs to develop teaching modules and technological assistance, such as use of tablets uploaded with teaching modules, aligned with the new curriculum and materials. This package of READ TA interventions was hypothesized to lead to improvement in children’s reading skills through improved teaching practices and resources. READ M&E conducted a final performance evaluation of the READ TA project to address the following evaluation objectives: • To assess READ TA’s performance and accomplishments in line with its implementation plan and deliverables, namely: o to develop appropriate primary school reading and writing materials and in-service teacher training materials; o to apply language-specific teaching and learning methods; o to support teaching and learning through appropriate technology and teacher aids; and o to improve technical support to RSEBs and the MoE. • To identify best practices and challenges encountered during project implementation. To address each objective listed above, we used both quantitative and qualitative data, as well as conducted a document review. The quantitative data came from classroom observations and teacher questionnaires. The qualitative data were drawn from in-depth interviews from a host of stakeholders, including teachers, principals, CTE staff, READ TA staff, and MOE education expert representatives. The desk review included a detailed analysis of READ TA quarterly and annual reports, Performance Monitoring Plan (PMP), and the English situation analysis report. VIII | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV For the quantitative data, we selected 8 zones from the Tigray, Amhara, Oromia, and Somali regional states, using simple random sampling technique. Three zones from the Southern Nations, Nationalities, and People’s Region (SNNPR) were included in the sample automatically since they (Sidama, Wolayta, and Hadiya) host the three languages of intervention. Altogether, 11 zones were selected from five regional states in this end-term performance evaluation. For the in-depth interviews, we purposively selected respondents from the MoE, RSEBs, zone education departments (ZEDs) (in SNNPR), READ TA project, CTE, and material developers. The selected respondents fulfil key roles in the development, performance, use, and ultimate sustainability of the READ TA intervention package. READ M&E recruited eighteen experienced qualitative and quantitative data collectors, who received a 3-day intensive training on the content and purpose of the data collection tools and procedures prior to data collection. The major findings from the performance evaluation are presented below, organized according to READ TA’s four intermediate results. IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Preservice and In-Service Teacher Training Developed Under this IR the evaluation team presents the mother-tongue material development process for Grades 5–8 in seven languages; the curricula and textbook revision process for English language for Grades 1–8; the level of alignment of the new mother-tongue materials for Grades 5–8 with the syllabi and minimum learning competencies (MLCs); the status of material distribution for Grades 5–8; and the support that READ TA made available for CTEs. • In-depth interviews with the material developers revealed that there were delays of planned activities, problems with division of labor, and difficulty working with MoE and RSEB staff during the development of mother-tongue materials for Grades 1–4. The material development team of each language anticipated these same set of challenges and incorporated mitigating strategies when developing mother-tongue student textbooks and teacher guides for Grades 5–8. As a result, the material development teams produced quality mother-tongue materials for Grades 5–8 that are appropriate in terms of exclusive use of the language, the use of illustrations, and addressing cultural, gender, and disability issues, in a shorter period of time. The development of mother-tongue materials for Grades 1-4 took approximately 12 months while material development for Grades 5-8 was completed in 6 months. • READ TA conducted a comprehensive and technically strong desk review of the English language textbooks to identify gaps in the old materials–for example, in terms of scope and sequence and levels–and addressed them. The textbooks were not ready for distribution at the time of data collection. • Interviews with experts from MoE, RSEBs, and READ TA, showed that the three organizations collaborated closely to develop–and provide trainings for–the new Grades 5–8 mother-tongue materials. This collaboration was at times strained by instances when the demands of some material developers’ full-time jobs affected the team’s ability to complete their material development on schedule. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | IX • The review of English textbooks for Grades 1–8 were delayed for more than a year due to MoE experts’ and higher officials’ inconsistent participation in meetings. Although the revisions were completed at the time of data collection, the textbooks were not published and were not ready for use. • According to mother-tongue material developers and MoE language experts, the new mother￾tongue materials for Grades 5–8 and the syllabus are well aligned with the MLCs because the team developed the materials at the same time in a parallel process, and also assigned a reviewer to ensure alignment. • Almost all schools received the new mother-tongue student textbooks and teacher guides for Grades 5–8. However, many schools have not yet distributed the books among teachers and students, and teachers have not yet used the new materials because they did not receive training. The respondents expressed a dire need for training the mother-tongue language teachers before they can use the materials. • 71.4% of mother-tongue teachers reported that every student in their class had a mother￾tongue student textbook. When examined by language, many students from Oromia and few students from other regions did not have their own textbooks. Reasons include a shortage of textbooks; improper text distribution (which was outside of READ TA’s responsibilities); difference between the number of students and the textbooks received; and the lack of a budget to distribute the materials. • READ TA provided trainings, material and technology support, including tablets with uploaded modules, LCD projectors, cameras, wireless internet connection, model videos, and modules, among other things with the goal of supporting mother-tongue teaching. However, criticism was leveled at the project’s inability to sustain technological support to mother-tongue instruction. IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are Applied The results in this IR focus on the extent to which appropriate language-specific teaching and learning methods are applied. • Quantitative and qualitative data show that the mother-tongue student textbooks and teacher guides are developed in such a way that is easy for teachers to follow the instructional routines, such as the “I do, we do, you do” methodology. READ TA oversaw planning, coordination, and grant management, whereas teacher training was the shared responsibility of all partners— READ TA, MoE, RSEBs, ZEO, WEO, and Save the Children. • READ TA organized capacity development trainings for the ministry, regions, and education leaders. Overall, respondents were satisfied with READ TA’s inputs, namely material development, selection and training of master and teacher trainers, grant management, and capacity building. From an input/output perspective, READ TA’s activities are indeed focused on helping students learn to read and write. That said, establishing a causal link between READ TA's outputs and improvement in students’ ability to learn and write requires an experimental or quasi-experimental design that is outside the scope of this performance evaluation. X | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids Under IR3, the evaluation reviewed READ TA’s work on the deployment and use of information and communications technology (ICT) packages; assistive technology, and multimedia teaching aids for Grades 1–8. • Nearly all CTE staff and education experts from RSEBs and ZEDs agree that READ TA’s ICT support to CTEs has been extremely invaluable as it has improved the delivery of pre-service training of mother-tongue teachers and made it more interactive and practical. However, READ TA was still piloting assistive technology in seven mother-tongue languages at the time of data collection. • READ TA worked with the Whiz Kids Workshop to develop and produce model teacher videos in seven mother-tongue languages. Even though READ TA produced multimedia learning materials and loaded them onto mother-tongue teachers’ tablets in the CTEs, some interviewees expressed concern about the limitation of a lack of follow up training on how to use the ICT. IR4: Technical Support to RSEBs and MoEs for the READ Institutional Improvement The two evaluation questions addressed under IR4 focus on READ TA’s progress toward developing pre-service teacher training modules and incorporating gender- and disability-sensitive approaches into the program. • CTE experts reported that READ TA played a leading role in developing and reviewing the training modules, and coordinating the development of preservice training modules. • The material developers confirmed that gender and disability issues were properly addressed during the material development processes, as the process involved inclusive education and gender experts in the material development team. For example, the mother-tongue materials were free from gender stereotypes and ensured that role models represented both sexes. What do the findings mean to the continuing improvement of READ in Ethiopia? Qualitative and quantitative data from diverse groups of READ TA stakeholders indicate that, overall, READ TA successfully implemented project activities per the contractual agreement. That being said, the challenges identified raise questions about the extent to which READ TA anticipated early implementation pitfalls that could have affected the deep implementation of project activities. Lessons from the implementation sciences literature help projects be more deliberate about anticipating such pitfalls at least at the project exploration and installation phases. Fixsen, Blasé, Metz, and Van Dyke (2015) divide the implementation continuum into four stages, exploration, installation, initial implementation, and full implementation. Exploration refers to the activities undertaken to consider the need for change, the availability of the necessary supports to satisfy the identified need(s), and the readiness of system leaders to own, initiate, and lead the process of improvement beyond initial program implementation. The installation stage refers to foundational provisions targeted at ensuring that READ TA was led/championed by project and government staff who have the power, interest, and the technical, relational, political, and administrative know-how necessary USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | XI for learning from early implementation dips (Fullan, 2014) to maximize deep implementation and sustainability of READ TA. This performance evaluation documented reports of delays of planned activities, problems with division of labor, difficulty working with MoE and RSEB staff, delays in the distribution of revised materials, challenges in securing fully dedicated staff to material development, technological interventions criticized as unsustainable, and uncertainty about the impact of READ TA outputs on student reading and writing outcomes. These challenges indicate that while trainings were provided, and good practices were implemented, questions arise on whether READ TA outputs will endure beyond the project funding period. What then? READ TA has put in motion important national innovations targeted at improving reading achievement in Ethiopia. Moving forward, the challenge is to turn these innovations into self-sustaining improvement processes that lead to the deep implementation of READ TA. Unlike initial implementation which is more fraught with risk because it involves disturbing the status quo, fundamental changes in existing structures, deep and full implementation of READ TA will be achieved when half or more of the intended users of the project outputs implement them with fidelity. At the heart of initial and full implementation are engaged leaders and persistent support (Marzano et al., 2005). As USAID and the MoE seek to create a self-sustaining improvement process, multi-tiered systems of support for reading achievement will have to be co-led by the Ministry of Education (MoE), the Regional State Education Bureaus (RSEBs), City Administration Education Bureaus (CAEBs), Colleges of Teacher Education (CTEs), and Woreda Education Offices (WEOs). In this regard, it is critical to identify the skilled implementation teams that were trained and coached by READ TA, and ensure that the improvement experience they accumulated is further leveraged to reach the tipping point of more than half intended users (administrators, supervisors, teachers) implementing READ TA practices with integrity and fidelity. In conclusion, from a performance evaluation perspective, the evaluation data collected about READ TA indicate satisfaction with the quality of technical assistance delivered to improve reading and writing. Moving forward, the challenge is to ensure that READ TA developed capacity is spearheaded by leaders who wield the necessary influence, interest, and the know-how (Eden & Ackermann, 1998) necessary for accelerating the improvement of early grade reading and writing outcomes for all children in Ethiopia. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 1 1. INTRODUCTION This document begins with a project overview and the evaluation objectives. The report then presents the performance evaluation methodology; the evaluation findings organized by IR and sub-IR; project limitations and findings; project sustainability and lessons learned; and the conclusion. READ TA is a 5-year (January 2012 to October 2017) project that works with the Ministry of Education (MoE), the Regional State Education Bureaus (RSEB), City Administration Education Bureaus (CAEBs), Colleges of Teacher Education (CTEs), and Woreda Education Offices (WEOs) to improve reading and writing in mother-tongue languages. The project supported the development and implementation of mother-tongue curricula for Grades 1–8 in seven targeted mother-tongue languages (Amharic, Afaan Oromo, Af-Somali, Hadiyyisa, Sidaama Afoo, Tigrinya, and Wolayttatto) and English as a medium of instruction. READ TA’s approach assumes that if the project revises the mother-tongue curriculum; develops student textbooks and teacher guides; and supports CTEs with teaching modules and technological assistance, then children’s reading skills will improve. READ TA’s in-service program use a training of trainers approach to build teachers’ professional development, while its preservice program addresses the training gap among mother-tongue language teachers by providing CTEs with technical and technological support. USAID provides funding for the READ TA project, and Research Triangle Institute (RTI) implements the project in partnership with Save the Children, Florida State University, SIL LEAD, Inveneo, Whiz Kids Workshop, and Books for Africa. READ TA’s main activities are categorized under four intermediate results (IRs): IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Preservice and In-Service Teacher Training Developed IR2: Language-Specific Teaching and Learning Methodologies and Strategies That Focus on Helping Students Learn to Read and Write Effectively Are Applied IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids IR4: Technical Support to RSEBs and MoE for the READ Institutional Improvement READ TA incorporates gender and inclusion as cross-cutting themes in its technical approach. READ M&E conducted a performance evaluation of the READ TA project during its final year of implementation. In this evaluation, READ M&E addresses the following objectives: • Assessing READ TA’s performance and accomplishments in accordance with its implementation plan and deliverables; • Identifying best practices and challenges encountered during project implementation; and • Using a proxy indicator to measure READ TA’s contribution to improving early-grade children’s reading and writing skills in their mother-tongue language. 2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 1.1. READ TA IMPLEMENTATION RTI and its consortium members implemented READ TA with the purpose of achieving four intermediate results. The following sections provide a detailed account of READ TA’s work. IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Preservice and In-Service Teacher Training Developed The READ TA consortium developed 112 student textbooks and teacher guides, covering seven mother-tongue languages in Grades 1–8. READ TA staff worked closely with the MoE, RSEBs, and local language and pedagogical experts to use materials prepared in three languages to develop and adapt 304 student textbooks and teacher guides in 12 language groups. When GQUIP’s tender process for printing Grades 1–4 textbooks and teacher guides took longer than planned, USAID granted READ TA additional funds to print about 2.5 million teacher guides and student textbooks in seven languages. READ TA confirmed that they developed textbooks for Grades 5–8 and handed over the camera-ready files to the MoE for printing and distribution. In addition, READ TA conducted a situational analysis and used the findings to prepare an action plan for reviewing the English language curriculum for Grades 1–8. At the MoE’s request, READ TA included kindergarten and Grades 9–12 in the curriculum revision. IR2: Language-Specific Teaching and Learning Methodologies and Strategies That Focus on Helping Students Learn to Read and Write Effectively Are Applied This intermediate result focuses on language-specific teaching and learning methodologies. READ TA integrated some activities related to the new “I do, you do, we do” teaching methodology into IR1 and supported READ II’s teacher training activities. READ TA collaborated with the MoE, RSEBs, and Save the Children to provide initial and refresher trainings to Master Trainers and Teacher Trainers in seven mother-tongue languages, who then trained Grades 1–4 teachers. READ TA also trained new teacher trainers for the Addis Ababa City Government and Amhara Regional State. In addition, the project developed and validated training manuals for Master Trainers and Teacher Trainers, who then trained mother-tongue language teachers in Grades 5–8 on the newly developed materials. Finally, READ TA developed and validated a mentoring framework and manuals for mother-tongue teachers. IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids Under IR3, READ TA must install ICT Packages for 36 CTEs; develop and upload model videos for Grades 1–4 and Grades 5–8 onto the server; and build initial capacity in the MoE, RSEBs, and CTEs to support and use the ICT package. In addition, READ TA is implementing the Assistive Technology Pilot Initiative in selected schools in five regions. IR4: Technical Support to RSEBs and MoE for the READ Institutional Improvement READ TA provided technical support to CTEs to improve and facilitate preservice teacher training. As part of this work, READ TA developed and adapted the national preservice mother-tongue teacher education curriculum into seven mother-tongue languages; developed and adapted CTE course modules into seven mother-tongue languages; developed training manuals for CTE teachers; and revised teacher training modules. READ TA developed seven modules, three of which have been finalized. Module 4 is USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 3 nearly finalized; READ TA staff developed and piloted the other three modules. They will revise the three modules based on the findings of the internal review process. READ TA built MoE and RSEB capacity to lead and monitor the implementation of the revised mother-tongue curriculum across the education system, and to sustain project efforts. 1.2. THEORY OF CHANGE READ M&E constructed a theory of change to ground the performance evaluation in the theoretical model that motivates the READ TA component of the READ program. The theory of change is based on our initial READ TA project document review, as well as conversations with various stakeholders, such as READ TA staff, the MoE and RSEBs. The theory of change considers a needs assessment initiative, state of the READ TA project, the long￾term change that the project seeks to support, and the sequence of change/input process/output anticipated to lead to the desired long-term outcomes. Figure 1 illustrates the READ TA theory of change. 4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV FIGURE 1. READ TA THEORY OF CHANGE USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 5 1.3. EVALUATION PURPOSE AND EVALUATION QUESTIONS This paper is an end-term performance evaluation of READ TA project. This evaluation assesses the achievements, successes, and challenges of project implementation in accordance with its implementation plan. The following evaluation questions guided READ M&E’s approach to the final performance evaluation. TABLE 1: EVALUATION QUESTIONS NO. EVALUATION QUESTION DATA TYPE DATA SOURCE DATA-GATHERING INSTRUMENT IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Preservice and In-Service Teacher Training Developed 1.1 A. What is the status of material development in mother-tongue languages for Grades 1–8? B. What is the status of the curricula and textbooks revision process for English language? Qualitative ● MoE ● RSEBs ● READ TA staff ● Document Analysis In-depth interview 1.2 What strategies has READ TA employed to coordinate: A. Material development for Grades 5–8 among material developers, reviewers and MoE and RSEB experts? B. Review of English textbooks Grades 1–8 between material developers, reviewers and MoE and RSEB experts? Qualitative ● MoE ● RSEBs ● READ TA staff ● Material developers ● Document Analysis In-depth interview 1.3 To what extent are the new mother-tongue materials and syllabi for Grades 5–8 aligned with the MLCs? Qualitative Quantitative ● Material developers ● MoE experts ● Experts form RSEBs ● Documents Content analysis interview 1.4 What is the status of material distribution for Grades 5–8? (This does not reflect READ TA’s performance, but it is valuable data.) Quantitative Qualitative ● Schools ● Principals ● Teachers ● Cluster supervisors Classroom observation Interview 1.5 What kind of support has READ TA made available for CTEs? Qualitative ● CTE Staff ● READ TA staff Interview 6 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV IR2: Language-Specific Teaching and Learning Methodologies and Strategies That Focus on Helping Students Learn to Read and Write Effectively Are Applied 2.1 How do teachers view the content of Grades 1–4 and Grades 5–8 student textbooks and teacher guides in terms of the use of language-specific teaching and learning methodologies and strategies? Quantitative ● Teachers Questionnaire 2.2 A. How did READ TA manage mentoring training? B. How did READ TA manage refresher trainings and training master trainers and teacher trainers for Grades 5–8? C. How did READ TA manage grants administration? Qualitative ● READ TA ● RSEBs ● MoE ● Master and teacher trainers Interview IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids 3.1 D. What has READ TA done on the deployment and use of ICT package for CTEs? E. What is the status of the use of assistive technology? F. What has READ TA done on the development of multimedia teaching aids for Grades 1–8? Qualitative ● READ TA ● MoE ● RSEBs ● CTEs Interview Document (if any) Observation at CTEs IR4: Technical Support to RSEBs and MoE for the READ Institutional Improvement 4.1 What progress has READ TA made on developing course modules for preservice teacher training? Qualitative ● CTEs Interview Document (if any) 4.2 What has READ TA done to address gender sensitivity and disability inclusion in the program? Qualitative ● MoE ● RSEBs ● READ TA ● Material developers Interview Document (if any) USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 7 2. METHODOLOGY 2.1. EVALUATION METHODOLOGY The research team employed a mixed-methods approach to conduct the performance evaluation of READ TA. The team relied on qualitative data to answer most of the performance evaluation questions, because the depth and richness of interviews and observation allowed the evaluation team to fully explore key stakeholders’ experiences in developing materials, conducting teacher trainings, and improving pedagogical practices. The evaluation team used quantitative data to gather information about broader project trends. Data sources are summarized in Table 2. TABLE 2: DATA SOURCES AND EVALUATION INSTRUMENTS DATA SOURCE INSTRUMENT EVALUATION QUESTIONS MoE and RSEBs In-depth interview 1.1, 1.2, 1.3, 2.2,3.1, 4.2, READ TA staff In-depth interview 1.1, 1.2, 1.5, 2.2, 3.1, 4.2 Principals and cluster supervisors In-depth interview 1.4 CTE staff In-depth interview 1.5, 3.1, 4.1 Master Trainers and Teacher Trainers In-depth interview 2.2 Material developers In-depth interview 1.2, 1.3, 4.2 Teachers In-depth interview and questionnaire 1.4, 2.1 Classroom and teaching aids and resource centers Observation checklist 1.4, 3.1 The semi-structured in-depth interview guide set the stage for a focused discussion, while still being flexible enough to capture a range of experiences and opinions. Enumerators used classroom observation tools in Grades 1–4 to observe patterns in the use of new student textbooks and teacher guides and to calculate the student-to-new-textbook ratio. Enumerators also used a questionnaire to survey teachers’ views on the new textbooks and teacher guides. Additionally, the research team reviewed READ TA documents to answer evaluation questions 1.2 and 1.3. Finally, the team looked at documents related project takeover and sustainability to examine READ TA’s exit strategy. 2.2. SAMPLING OF RESPONDENTS The sample design collected data across multiple levels of the national education system, beginning with zones from five regional states. The evaluation team used stratified random sampling to select zones from the Tigray, Amhara, Oromia, and Somali regional states. We generated random numbers from Microsoft Office Excel 2010 to select eight zones from Amhara, Oromia, Tigray, and Somali. All zones, except those zones from Ethiopia Somali Region which had security problems, were included in the sampling frame. Zones in the Southern Nations, Nationalities, and People’s Region (SNNPR) were predetermined because of the languages of intervention in the region. Altogether, 11 zones were selected from five regional states in this end-term performance evaluation. After selecting zones, the evaluation team generated random numbers using Microsoft Office Excel 2010 to select woredas from each of the selected zones. READ M&E included all woredas from the selected 8 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV zones in the sampling frame. The number of woredas selected varied from region to region. The number of woredas selected from each region corresponded with the region’s population size: four woredas each from Amhara and Oromia, three woredas each from Tigray and Somali, and two woredas from each of the three zones of SNNPR. READ M&E selected a total of 20 woredas for this end-term performance evaluation. For the third level of sampling, data collectors selected urban and rural schools from a list of full-cycle schools provided by woreda education offices. The team then used the lottery method to randomly select two schools from each woreda. All full-cycle (Grade 1-8) schools in the woreda were included in the sampling frame. To ensure equal representation of urban and rural schools, data collectors selected one school from the group of schools in town and one school from the group of schools located in rural areas. READ M&E selected schools in proportion to each region’s population size, which means that the team selected eight schools each from Amhara and Oromia, six schools each from Tigray and Somali, and four schools from each of the three zones of SNPPR. When there was an insufficient number of mother-tongue teachers for conducting interviews or administering questionnaires, data collectors included an additional one or two schools in the sample. For qualitative in-depth interviews, READ M&E selected individuals based on their position, their involvement in the project, and availability. To this end, the team selected individuals from the MoE, RSEBs, ZEDs (in SNNPR), READ TA, material developers, and CTE staff. By selecting interview participants from multiple stakeholder groups, READ M&E was able to capture the range of experience of those involved in READ TA activities. Data collectors interviewed participants until they reached saturation (i.e., when the interviews reveal no new information). Tables 3 and 4 show the number and type of respondents that READ M&E interviewed for this performance evaluation. TABLE 3: NATIONAL-LEVEL DATA COLLECTION (COVERING MOE INTERVIEWS AND READ TA STAFF INTERVIEWS) Number of Ministry of Education Interviews Number of READ TA Staff Interviews Addis Ababa 2 3 TABLE 4: REGION-LEVEL DATA COLLECTION (COVERING RSEB AND ZONE EXPERTS, MATERIAL DEVELOPMENT EXPERTS, READ TA STAFF, AND CTE STAFF) REGION NUMBER OF RSEB AND ZONE EXPERTS NUMBER OF MT* MATERIAL DEVELOPERS READ TA STAFF FROM REGION NUMBER OF CTE STAFF MATERIAL REVIEWERS Amhara 1 4 1 7 - Oromia - 1 2 5 2 Tigray 2 2 1 3 2 SNNPR Sidama 2 - 1 1 Wolayta 2 2 1 3 1 Hadiya 1 2 1 3 - Somali 4 2 1 3 2 Total 10 15 7 25 8 *MT stands for mother-tongue USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 9 READ M&E secured the names and cell phone numbers of experts involved in the English language textbook review and selected one expert for each grade level for the in-depth interview. Although the review took place at the READ TA headquarters, the reviewers came from other regions and zones. At the school level, data collectors selected teachers, school principals, and cluster supervisors to participate in the research. Data collectors selected different teachers for qualitative and quantitative data. The team interviewed 66 mother-tongue teachers who taught Grades 1-4, two from each school included in the study. These teachers had the new materials, received the 10-day training, and were teaching at least one class in Grades 1–4. Data collectors asked 186 mother-tongue language teachers (96 women and 90 men) to complete a self-administered questionnaire. The teachers who completed the questionnaire were not interviewed nor observed in their classrooms. The team interviewed principals from half of the schools in the sample and interviewed cluster supervisors from the remaining half of schools. In each sampled school, data collectors observed at least two randomly selected classrooms. In each woreda, data collectors observed Grades 1 and 3 in the first school and Grades 2 and 4 in the second school. To ensure equal representation of urban and rural schools, if the team randomly selected an urban school in one woreda, they would purposively select the other school from a rural area. When there was an insufficient number of mother-tongue teachers for conducting interviews or administering questionnaires, data collectors included an additional one or two schools in the sample. School-level data collection is summarized in Table 5. While the data are not nationally representative, they are an adequate source of information from which we can draw inferences. TABLE 5: SAMPLE SIZE FOR SCHOOL-LEVEL DATA COLLECTION REGION # OF ZONES # OF WOREDAS # OF SCHOOLS CLASSROOM OBSERVATION PRINCIPALS (IN HALF OF SCHOOLS) SUPERVISORS (IN HALF OF SCHOOLS) TEACHERS PER SCHOOL PER SCHOOL TOTAL PER SCHOOL TOTAL PER SCHOOL G1–4 TOTAL G1–4 TOTAL Amhara 2 4 10 ½ 16 1 8 1 6 1/2 11 Oromia 2 4 8 ½ 16 1 3 1 4 1/2 10 Tigray 2 3 5* ½ 10 1 3 1 3 1/2 12 SNNPR 1/2 Sidama 1 2 4 ½ 8 1 2 1 2 1/2 5 Wolayta 1 2 4 ½ 12 1 2 1 2 1/2 12 Hadiya 1 2 6 ½ 13 1 2 1 2 1/2 8 Somali 2 3 6 ½ 20 1 3 1 3 1 / 2 8 Total 11 20 43 95 23 22 66 *The data collection team was unable to access one school in Tigray The team collected information from 462 respondents for the performance evaluation. The team interviewed 10 experts from RSEBs and zones; two experts from the MoE; seven regional and three head-office READ TA staff; 15 material developers; eight material reviewers; 25 CTE staff; 23 principals 10 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV and 22 cluster supervisors; and 66 mother-tongue teachers. They also observed 95 mother-tongue language teachers in the classroom and collected self-administered questionnaires from 186 mother￾tongue teachers. 2.3. DATA COLLECTION PROCEDURES READ M&E recruited 18 experienced qualitative and quantitative data collectors with a minimum of a Master of Arts or Master of Education for this final performance evaluation. READ M&E conducted a 3- day intensive training about the content and purpose of data collection tools and procedures. The training also included information about how to approach the respondents, use the data collection tools, record the data, maintain respondents’ privacy, and transcribe audio-recorded interviews based on the data collection protocol. The training highlighted best practices from the past two data collection processes, problems encountered, actions taken, and implications for the current data collection process. The afternoon session of the first day and the entire second day of the training were devoted to introducing and discussing the data collection tools. On the third day, the trainees simulated data collection among themselves. Afterward, participants discussed confusing issues and potential solutions for problems they encountered. Two data collectors interviewed experts from RSEBs and ZEOs, READ TA regional staff, CTE staff, teachers, school principals, and cluster supervisors at each data collection site. One data collector facilitated the interview, while the other took notes and recorded the interview. In addition, the data collectors conducted classroom observations, site observations at CTEs, and collected self-administered questionnaires. The data collectors completed data collection within 2 weeks. Furthermore, READ M&E team members interviewed experts from the MoE and the READ TA headquarters. READ M&E staff also reviewed relevant documents from the READ TA headquarters. 2.4. DATA ANALYSIS AIR adheres to strict data analysis principles, regardless of the type of data being collected. All qualitative data were coded and analyzed. The READ M&E team created a preliminary coding outline and structure based on the evaluation questions, interview protocols, and field notes taken during data collection. This codebook was prepared using NVivo software and served as a tool to organize and subsequently analyze the information gathered in the interviews. The outline is a living document that may be modified as new themes and findings emerge during data analysis. The outline is accompanied by a list of definitions for the codes, so that coders may categorize data using the same standards. Before the actual coding, coders conducted an inter-rater reliability test. Inter-rater reliability ensures that coders agree on how to code the same responses within a given dataset. The team leader randomly selected two transcribed interviews for each type of respondent, and asked coders to code them based on the qualitative codebook. The team leader then ran a query to ascertain whether the coders had the same understanding. The overall agreement of the three raters exceeded 99% (teachers = 99.80; CTE = 99.49; materials developers = 99.24; principal and supervisor = 99.94; READ TA staff = 99.71), which indicates that the raters had a similar understanding of the nodes and could code independently and in the same manner. The team subsequently coded the data into NVivo following the structures given in the codebook. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 11 Using these coded data, the READ M&E team identified themes and categories and refined the concepts to inform the overall findings. During the data reduction process, the READ M&E team categorized prevalent responses, examined differences among groups, and identified key findings and themes related to the evaluation questions. Meanwhile, the research team analyzed quantitative data using Statistical Package for Social Scientists version 21. They entered the data via a prepared template and subsequently cleaned it to ensure accuracy. Researchers used descriptive statistical analyses—mostly percentages and proportions—to support and triangulate findings from the qualitative data. 2.5. ETHICS REVIEW AIR conducts rigorous ethical reviews through its Institutional Review Board (IRB) for all internal research activities and provides this service for a variety of subcontractors and collaborators. AIR’s IRB has conducted and expedited full board reviews of research involving human subjects for more than 16 years. AIR is registered with the Office of Human Research Protection as a research institution and conducts research under its own federal wide assurance. The AIR ethics board approved the data collection process, including procedures for participants’ informed consent and maintaining participant confidentiality throughout the data collection process. 3. FINDINGS This section synthesizes the qualitative and quantitative findings gathered through semi-structured interviews, questionnaires, and observational instruments. The findings are organized by READ TA’s intermediate results. IR1. READING AND WRITING MATERIALS APPROPRIATE FOR PRIMARY CLASSROOMS AND PRESERVICE AND IN-SERVICE TEACHER TRAINING DEVELOPED 3.1. MATERIAL DEVELOPMENT AND REVISION MATERIAL DEVELOPMENT IN MOTHER-TONGUE LANGUAGES READ TA began the material development process with a review of content, pedagogy, and assessment strategies of the previous mother-tongue language curricular materials. READ TA then launched a Mother Tongue Task Group to review the mother-tongue syllabus and the Minimum Learning Competencies (MLCs). Once task group revised the national mother-tongue syllabus, READ TA staff invited representatives from the MoE, RSEBs, USAID, and the READ TA consortium to participate in a national mother-tongue workshop in Adama from June 19–21, 2013. During this workshop, the participating stakeholders validated the newly revised national mother-tongue syllabus for Grades 1–8. Project staff then finished developing 112 separate textbooks and teacher guides, which covered seven mother-tongue languages in Grades 1–8 in two phases. During the first phase, the team developed the materials for Grades 1–4, while the materials for Grades 5–8 were developed during the second phase. The team followed a similar process to develop materials for the two cycles. However, team used what they learned during the development process for Grades 1–4 to improve the material development process for Grades 5–8. A READ TA respondent echoed the sentiment of other respondents when she shared her experience developing the materials: 12 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Before we started developing the materials for Grades 5–8, we reviewed the experiences related to material development for Grades 1–4. As a result, we were well prepared in terms of the kinds and number of experts needed, time management, template to be followed, resources, and material needed for developing the materials for Grades 5–8. Regional and headquarter READ TA staff cited examples of what they learned from materials development for Grades 1–4 to improve the process for Grades 5–8. For example, the team did not include designers and illustrators on the material development team for Grades 1–4, which resulted in considerable back and forth between the development team and the illustrators and designers. The illustrators and designers were included in the team for Grades 5–8, which improved the effectiveness and efficiency of the process. In addition, READ TA trained linguists and experts in materials development before they participated in the materials development process, which contributed to the quality of materials for Grades 5–8. More importantly, the team did not test materials for Grades 1–4 in the field, which would have allowed the task group to learn from teachers’ hands-on experiences with the new materials. To rectify this, the development team for Grades 5–8 included a well-planned field test in the development process. An expert from READ TA explained why field tests were useful: A group of teachers teaching Grades 5–8 were selected from each language and were given hands-on practice on using the new materials in the classrooms as [a] field test. The trainees submitted the experiences and lessons learned from the field test every Friday using a common reporting modality prepared for the purpose. The outcomes of the month[ly] lessons were very clear, and the field test was very successful. The submitted lessons were checked and regulated by process coordinators for each language. In this way, quality assurance was guaranteed by the coordinators. Respondents believed that the task group was more mindful of students’ age, grade level, and level of understanding when developing materials for Grades 5–8. This was not the case when developing materials for Grades 1–4. Furthermore, RSEBs were heavily involved in developing Grades 5–8 mother￾tongue materials. Most of the material developers believed that the experienced international consultants contributed considerably to the material development process and helped build local staff capacity. One material developer from SNNPR noted that the consultants helped ensure that the materials met international standards. According to another material developer from Amhara, the international consultants recommended incorporating an interactive learning approach in the materials: International consultants brought international experience into [the] material development process. The development of the new mother-tongue materials follows three methodological principles of “I do,” “we do,” and “you do.” This is a very interactive approach for students to learn. First teachers do, students practice, and then students and the teacher do the task together. All chapters were prepared based on this methodological approach and the students’ learning needs. A material developer from Amhara said that the READ TA staff did their best to coordinate material development activities, provide necessary materials and deliver materials development training. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 13 Most respondents agreed that they encountered several challenges while developing the mother-tongue materials for Grades 1–4. First, designers and illustrators were not members of the curriculum development team, which unnecessarily prolonged the development process. Second, the team did not use field test lessons, which could have provided the team with useful feedback. Third, the material development process took longer than expected. The task group learned from these challenges and addressed then during development of student textbooks and teacher guides for Grades 5–8. As a result, the material development team developed well-prepared mother-tongue materials for Grades 5– 8 in a shorter period of time. ENGLISH-LANGUAGE SYLLABUS AND TEXTBOOK REVISION READ TA conducted a desk review and situational analysis to inform the process for revising the existing English textbooks for Grades 1–8. The desk review used checklists to analyze the content of existing English language textbooks and to identify their weaknesses and strengths. The situational analysis gathered empirical data from key stakeholders (e.g., teachers, students, school principals, department heads) about the English language curricula and textbooks. According to READ TA experts, the desk review and situational analysis identified gaps in the scope and sequence of the existing textbooks; found inconsistencies in the approaches used to write the textbooks; and assessed the textbook contents’ difficulty level. The results revealed that the books taught English as a subject rather than as a medium of instruction, and therefore did not prepare students for the transition to the new materials. The textbooks were also developed by foreigners who had no understanding of the Ethiopian context. In general, the findings showed the need for revising the syllabi and textbooks for Grades 1–8. The MoE, RSEBs, and READ TA agreed with the desk review findings and agreed upon a plan to revise the English syllabi and textbooks at all levels (preprimary to Grade 12). The interviewees firmly stated that the READ TA headquarters led the revision of the syllabi for grades kindergarten through 12 and English textbooks for Grades 1–8. READ TA received input and guidance from experts selected from each region. English language professionals represented RSEBs from all participating regions. These professionals revised the English syllabus and the scope and sequence of English language materials. An expert from the READ TA headquarters shared an account of the revision process: In the revision of English textbooks for Grades 1–8, we used professionals from all regions. There were teachers from primary schools, secondary schools, and universities. Regional contexts were taken into account during the revision process. Now we are at the completion phase. The revision process followed similar steps considered for the development of mother￾tongue materials. There would be field test, quality assurance mechanisms, validation process, and approval of the final version of the textbooks. 14 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV A READ TA staff member from SNNPR described the revision process: We had made two types of evaluation on the English textbooks and teacher guides. The READ TA did desk study and field study. Using the desk study, the books were reviewed using checklists. Second, we had collected data from 75 schools in the country. Teachers, students, school principals, department heads, and other stakeholders participated in the field study and identified some problems from the existing textbooks and teacher guides. So the English situational analysis result revealed gaps such as inconsistency of the approach and the inappropriateness of the difficulty level to the grade level. The results were presented to MoE and concerned stakeholders. Then MoE decides on the revision of the curricular materials for all levels from preprimary school materials and the high school materials. And [a] syllabus was developed and followed by scope and sequence. The revision works of Grades 1–8 English materials were carried out by English professionals from all over the country, and the same professionals developed the [Grades] 1–4 and [Grades] 5–8 materials. Now the first draft of the Grades 1–8 books are completed, [and] international proofreaders are proofreading the English textbooks and teacher guides. The next job will be validation. I hope the work will be completed soon because the project has very [a] short time; so I think the project will accomplish it until June 2017. Another READ TA discussed the progress that had been made toward revising English language textbooks and teacher guides at the time of data collection (April to May 2017): … The first drafts of the English textbooks for Grades 1–8 have been completed; international consultants (native speakers of the language with the necessary expertise) have proofread it, and now the next job will be validation and preparation of the final version of the textbooks and teacher guides. I hope the work will be completed soon because the project has [a] very short time to terminate. Respondents generally praised the quality of the new textbooks, especially when comparing them to the previous materials. A reviewer from Hadiya zone described the difference between old and new textbooks: The existing textbooks and teacher guides were prepared by foreigners who had no idea about the context. But this time, these materials were revised locally and highly contextualized. The revised one is better in quality and appropriate for [the] local context. Another reviewer from the Tigray argued that success is measured by students’ academic performance and behavioral changes. For him, students’ illiteracy and poor academic performance were clear indicators that the previous materials were not effective. He concluded that it would be difficult to say that the previous syllabi and textbooks were successful. Reviewers also spoke positively about the degree of alignment between teacher guides and student textbooks with the MLCs. The most recent teacher guide was revised in conjunction with the student textbook to ensure consistency between them. A material reviewer from Hadiya zone stated, “While revising the materials, the level of alignment between the textbooks and teacher guides with the MLCs received great emphasis.” However, staff from READ TA and material reviewers reported that the review process took more time than expected. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 15 Despite complaints about the amount of time the team took to revise the syllabi and textbooks, participants generally had favorable opinions of the final product. First, the team conducted a desk review and situational analysis to ascertain the weaknesses and strengths of the previous syllabi and textbooks. This was an essential task that informed the revision process. Second, unlike the previous materials, the newly revised materials were prepared by local professionals who know the context. Third, the team tested and evaluated the materials through field tests at different levels of the education system. All these factors indicate that READ TA oversaw a structured revision process that went through multiple stages of review. However, despite favorable evaluations of the revised materials, the textbooks and teacher’s guides were not yet ready for distribution at the time of data collection. TRAINING MATERIAL DEVELOPERS A material developer from Wolayita believed that the international experts provided effective and comprehensive training on material development. According to this respondent: The training offered during the development process was exhaustive. It was for 30 days. Participants were from different regions. After training there was always activity to be completed. It was not a mere training. The trainers have the culture of shaping what participants did not capture properly. It was a perfect training. All but one respondent agreed that the international consultants facilitated a well-run training. The lone exception, a material developer from Tigray, said that not all foreign trainers were incompetent, but that he knew of local experts who know the native languages and who could handle the training on material development more effectively: The trainers were international consultants. We do have local trainers who could handle the training better. I believe that it would be better if the task was [sic] given to experts who know the native languages. Really, I am not saying all the foreign trainers were incompetent. But some of them were not better than the ability of local experts. So it would be more effective had such weaknesses be[en] corrected earlier. There is a consensus among material developers that the training offered was effective and productive and provided them with the opportunity to learn new information and skills related to material review and development. 3.2. COORDINATING MATERIAL DEVELOPMENT AND REVISION ACTIVITIES MOTHER-TONGUE MATERIAL DEVELOPMENT ACTIVITIES FOR GRADES 5–8 As previously mentioned, READ TA led mother-tongue material development for Grades 5–8 in selected regions. READ TA invited professionals from participating regions to attend training sessions in Hawassa. After the trainings, the mother-tongue language professionals returned to their respective regions to develop the needed materials. In SNNPR, three groups of developers developed materials for three mother-tongue languages: Sidaama Afoo, Hadiyyisa, and Wolayttatto. Zone Education Offices (ZEOs) took ownership of the mother-tongue language materials by helping to recruit professionals, providing support, and following up on progress made during the material development process. 16 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV READ TA, RSEBs, and the MoE coordinated the material development process in different regions. However, the degree of coordination among these partners may differ from one region to the other, which makes it important to examine similarities and differences across the regions. Material developers, experts from regions and zones, and regional READ TA staff believed that READ TA, RSEBs, and the MoE worked closely to implement material development activities. All the partners did their best to coordinate material development activities, provide necessary materials, and provide training. Two professionals who contributed to material development in Amhara believed that material development endeavor was well planned and organized. Similarly, a READ TA staff member from Amhara stated that “[o]ne of the best lessons we got is the joint planning and implementation between READ TA and the Amhara RSEB. We worked cooperatively as a team, starting from the task force.” Material developers also praised the collaboration between READ TA, MoE, Oromia RSEB, and material developers. The material developers believed that READ TA and Oromia RSEB’s support helped facilitate material development activities. Material developers expressed positive opinions on the degree of coordination among stakeholders during the mother-tongue material development for Grades 5–8 in SNNPR. According to one of the material developers: READ TA, MoE and RSEBs had [a] close relationship. There were READ TA experts, MoE and RSEB experts of each language who assisted us in the development of mother-tongue materials. The RSEB helped us in providing experts and rooms. Therefore, the collaboration between the three parties was very good. A material developer from SNNPR explained that each party plays a role in the material development process: “MoE prepared guidelines that enabled the uniformity of the curriculum materials across languages. Then, the region requested the zones to recruit and present experts for their respective languages.” The three stakeholders ensured that each party had clearly defined roles and responsibilities, which improved collaboration and understanding. One respondent pointed out that the three stakeholders regularly consulted one another and worked to solve problems as a team. However, another respondent believed that the zones accomplished more than the MoE and the RSEBs. While the respondent admitted that the three parties worked well together, the respondent from Hadiyyisa zone argued that “[ou]r communication was more or less with the zone, because the owner of the language is the zone. Even the validation process also took place in the presence of zonal representatives.” Respondents generally agreed that READ TA, the MoE, and RSEBs effectively coordinated the materials development by providing experts, following up on training progress, and monitoring the material development process. Furthermore, each ZEO was involved in the follow-up and validation process. There is a general a consensus among participants in the material development process that the MoE, Tigray RSEB, and READ TA had a strong working relationship. In the words of the READ TA staff: We never had any problem with the bureau. We were consulting and working together with the bureau head and vice head. Our main success was working collaboratively with the bureau. However, the three organizations still encountered challenges and difficulties. A READ TA staff member from Amhara explained that some of the material development team members, particularly those from colleges and the Amhara RSEB, had additional responsibilities in their respective organizations. These members had competing responsibilities and priorities, which resulted in delays. Similarly, a READ TA USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 17 staff member from Oromia reported that the team had difficulties implementing activities according to the original timeline because some colleagues had large workloads in their office. As a result, they could not complete their assignments according to the timeline. READ TA staff from SNNPR reported similar problems for some of their colleagues, particularly those from universities and colleges who had to teach courses. An expert from Tigray described a different set of challenges: (1) Sometimes, when READ TA and the Tigray RSEB made progress in their work, the MoE delayed the decision-making process, and (2) MoE interfered in matters concerning Grades 5–8, which is clearly the mandate of RSEB. In Somali, the RSEB experienced a high turnover of RSEB heads, which hindered the smooth function of the material development task force. The participants believed that READ TA, RSEBs, and the MoE had a strong working relationship, and coordinated their work effectively across all regions. Some problems that hindered the material development effort included (1) members whose responsibilities on the material development team competed with their regular job, (2) failure to complete individual and group assignments on time, and (3) failure to conduct meetings as scheduled. REVIEWING ENGLISH TEXTBOOKS FOR GRADES 1–8 Unlike the mother-tongue material development for Grades 5–8, READ TA coordinated the review of English language textbooks for Grades 1–8 at the central level. However, English language professionals were recruited to represent RSEBs from participating regions. MoE expressed full support of revising the English language textbook for Grades 1–8. The MoE wanted to ensure curricula and textbook uniformity across all grades, and requested that READ TA do the same for Grades 9–12. Despite the MoE’s demonstrated interest and support in the revision process, READ TA staff mentioned that MoE experts and higher officials did not show up to meetings on time. Their tardiness created unnecessary delays in the English language review process. A READ TA respondent commented that, “[E]ven the stage we reached now in the review process is because of our dedication.” This suggests that not all stakeholders are equally committed to revising the materials, or perhaps that they had difficulty coordinating amongst themselves. Several interviewees reiterated that the main problem was time. It took more than one year to actually start the revision. Even now, the textbooks have not yet been published and are not ready for use. These issues beg the question of whether stakeholders had agreed upon an implementation plan. If so, why did the review process take longer than originally thought? On the other hand, if the stakeholders did not agree upon an implementation plan, then this suggests that they had coordination challenges. The interview transcripts do not provide adequate evidence to draw a conclusion about stakeholders’ strategies to coordinate the review process. However, many interviewees are clearly unhappy with the time it took the team to complete the review process. Those who are looking forward to using the revised textbooks and implementing the new syllabi will have to wait longer. 18 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 3.3. ALIGNING THE NEW MOTHER-TONGUE MATERIALS AND SYLLABUS FOR GRADES 5–8 AND THE MLCS Material developers and MoE and RSEB experts were asked to evaluate the level of alignment between the new mother-tongue materials (student textbooks and teacher guides) for Grades 5–8 with the syllabi and the MLCs. The material developers, in particular, were asked to rate the level of alignment on a five-point scale that ranges from very weak to very strong. Among the 19 material developers, 13 rated the level of alignment as very strong, whereas five of them rated it as somewhat strong. The remaining respondent did not provide a clear response to the question. All but one of the material developers and reviewers indicated that the level of alignment was either very strong or somewhat strong. Thus, the material developers and reviewers provided mostly positive ratings. When the material developers were asked how they ensured strong alignment between the materials, syllabus, and the MLCs, one respondent from Wolayita zone described their process: First, we revised the MLCs. Then we started revising the textbook and teacher’s guide. While revising these materials, we checked whether there was alignment between the MLCs and the materials. Also, there were individuals who were responsible for checking this. Whenever they found a problem, they would tell us to revisit the materials. According to the respondent, the team designated individuals who were responsible for ensuring alignment during the mother-tongue material development. In addition, the team ensured alignment by developing the materials in a parallel process, rather than sequentially. A material developer from Tigray explained that they developed the student textbooks and teacher guides at the same time. When the team developed a lesson in one unit, they would also develop the teaching methodology for that lesson and incorporate it in the corresponding unit of the teacher guide. How did experts evaluate the level of alignment? Did they agree with the material developers? The interview results show that experts also evaluated the alignment between the materials and the MLCs favorably. Experts stated that material developers paid careful attention to the alignment between textbooks, teacher guides, and the MLCs during the development process. RSEB experts evaluated the draft materials against a set of criteria, including the level of alignment among textbooks, teacher guides, the syllabus, and the MLCs. In some regions, such as the SNNPR, RSEB and zone experts evaluated the materials three times before they approved of the final version. This demonstrates how much RSEBs valued alignment during the material development process for Grades 5–8. An expert from Somali confirmed that Somali RSEB would not have approved the final products if the materials, the syllabus, and the MLCs were not properly aligned. Another expert from the same region described the procedure they followed: First, we tried to identify the minimum learning competencies. So the syllabus was prepared based on the identified MLCs. Similarly, the textbook was prepared based on the syllabus. So I can confidently say there is appropriate alignment between the textbook and the MLCs. A third expert from Somali said: USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 19 The level of alignment between the textbooks and teacher guides for Grades 5–8 with the MLCs is high since [the] material development process is guided by the MLCs of the country. In addition, the materials developed took into account the level, age, and cognitive development of the students. An expert from Tigray described the link between the mother-tongue materials for Grades 5–8 and the MLCs: First, we tried to identify the minimum learning competences. So the syllabus was prepared based on the identified MLCs. And then the textbook was prepared based on the syllabus, taking into account the sequence and scope. So I can confidently say there is appropriate alignment between the textbook and the MLCs. The material developers and experts agreed that the newly developed materials were strongly aligned among the textbooks, teacher guides, the syllabus, and the MLCs. The development team took several measures to ensure strong alignment. First, the development team were intentional about ensuring alignment during the development process. Second, reviewers had the opportunity to evaluate the materials’ level of alignment. Third, the material developers had a chance to revise the materials based on evaluators’ feedback. The respondents describe a methodical procedure that ensured strong alignment among the mother-tongue textbooks, teacher guides, the syllabus, and the MLCs for Grades 5–8. 3.4. MATERIAL DISTRIBUTION As part of the performance evaluation, READ M&E examined the use of student textbooks and teacher guides in class. Although READ TA is not responsible for providing textbooks and teacher guides, this indicator is important because the effectiveness of material development relies on the dissemination and use of the new materials. This report presents the distribution of student textbooks and teacher guides for Grades 5–8 and Grades 1–4 in separate sections. 3.4.1. DISTRIBUTING MOTHER-TONGUE MATERIALS FOR GRADES 5–8 RSEBs collaborated with the MoE to successfully distribute student textbooks and teacher guides for Grades 5–8 among woredas. However, some woredas distributed the books to their schools while others did not. Most principals and cluster supervisors confirmed that they recently received copies of both student textbooks and teacher guides for Grades 5–8, whereas some teachers and principals reported that they have not yet received the books. In other cases, teachers were not yet trained in the new materials, and therefore have not started using the textbooks and teacher guides. Many principals reported that teachers in their schools were not using the new materials because they had not yet been trained on how they should use the guide in teaching the mother-tongue language. In Oromia, principals recollected the distributed books to prevent untrained teachers from using them, while other regions distributed the books and allowed untrained teachers to use the new the teacher guide. A principal in Hadiyyisa stated, 20 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Yes, we have given the new textbooks to students and started teaching Hadiyyisa in Grades 5–8 using the new books. Teachers in Grades 5–8 are teaching with the new books without taking any training about the new books. We decided to use the books because we received enough number of copies. I know teaching without the training is difficult, but this is the solution that we have. The data show that few schools have started teaching Grades 5–8 with the new materials. Out of 95 classrooms observed, data collectors only observed four classrooms using the new materials. Most teachers have not yet started using the materials because they had not yet been trained in how to use them. Principals and cluster supervisors explained that their schools received the books during the middle of the 2016–17 academic year. It would have been a significant challenge to use the new textbooks while preparing students for the Grade 8 regional examination, which takes place at the end of May or beginning of June. The schools that were observed using the new textbooks did so because they had a shortage of the old textbooks, not because they fulfilled the prerequisites in accordance with the plan. A cluster supervisor from Amhara said: Yes! The school has started teaching with the new mother-tongue textbooks and teacher guides of Grades 5–8. The school is using the new mother-tongue materials together with the old student textbooks and teacher guide following the exist[ing] shortage of old textbooks of Grades 5–8. In general, respondents expressed the need to train mother-tongue language teachers before they use the new mother-tongue materials in Grades 5–8. The new materials will most likely be implemented during the 2017–18 academic year (2010 E.C.). Respondents noted the trainings for Grades 1–4 teachers consumed a large portion of READ TA’s training budget, which created a budget shortage that delayed trainings for Grades 5-8 mother-tongue language teachers. 3.4.2. DISTRIBUTING MOTHER-TONGUE MATERIALS FOR GRADES 1–4 The research team used the teacher questionnaire to assess MoE and RSEB progress in distributing the mother-tongue materials for Grades 1–4. The findings for student textbooks and teacher guides are organized into separate sections. STUDENT TEXTBOOKS Almost all teachers surveyed who teach Grades 1–4 (95.2%) reported that their students have the new mother-tongue textbooks. Three teachers from the Somali Region and one teacher from the Oromia Region reported that their students do not have the new mother-tongue textbooks. Five teachers did not respond to this question. In a follow-up question, the survey asked teachers whether every student in their class has his or her own textbook. The majority of teachers responded affirmatively (71.5%). When their responses were differentiated by language, most of the teachers who responded negatively spoke Afaan Oromo. Figure 2 presents teachers’ responses by language. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 21 FIGURE 2. PERCENTAGE OF STUDENTS REPORTED TO HAVE THEIR OWN TEXTBOOK IN GRADES 1-4 As clearly shown in Figure 2, the majority of teachers in all languages, except Afaan Oromo, reported that every student in their class has his or her own textbook. Teachers were asked to describe the reason(s) why some of their students do not have their own textbook. Responses included: • Shortage of textbooks; • Improper management of textbook distribution logistics; • Delayed provision of textbooks; • Difference between the number of students and the textbooks received; and • Lack of RSEB funds to distribute the materials. The majority of teachers (61.3%) said that the student-to-textbook ratio is 1:1, which meets the Federal MoE’s goal of ensuring that every student gets a textbook. Table 6 presents the details. 96.7 75.8 30.8 79.2 70 85 85 3.3 21.2 69.2 16.7 30 15 15 0 3 0 4.2 0 0 0 0 20 40 60 80 100 120 Tigrigna Amharic Afaan Oromo Af Somali Sidaamu Afoo Wolayttatto Hadyissa Yes No No Response 22 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV TABLE 6: THE DISTRIBUTION OF STUDENT-TO-TEXTBOOK RATIO IN GRADES 1–4, BY LANGUAGE LANGUAGES RATIO TIGRINYA (%) AMHARA (%) AFAAN OROMO (%) AF￾SOMALI (%) SIDAAMA AFOO (%) WOLAYTTATTO (%) HADIYISA (%) TOTAL 1:1 90 69.7 46.2 70.8 15.0 80.0 50.0 61.3 1:2 3.3 6.1 10.3 12.5 40.0 5.0 35.0 14.0 1:3 3.3 15.2 35.9 4.2 25.0 5.0 5.0 15.1 1:4 0 3 2.6 0 5.0 0 0 1.6 1:5 3.3 3 2.6 0 10.0 5.0 5.0 3.8 1:6 0 0 2.6 0 0 5.0 5.0 1.6 NR 0 3 0 12.5 5.0 0 0 2.7 This table shows that the student-to-textbook ratio varies by language. Of the teachers who reported that their students have mother-tongue textbooks, 90% of respondents from Tigray reported that the student-to-textbook ratio is 1:1, whereas only 15% respondents from Sidama zone reported the same. This is a large difference between the two language groups. Moreover, a significant proportion (35.9%) of respondents from Oromia reported that the student-to-textbook ratio is 1:3. The majority of students observed have their own textbooks. Classroom observations show slightly different results. Out of 4,005 students observed in the class (2,172 boys and 1,833 girls), only 1,757 students (43.9%) had the new textbook, which translates into a textbook-to-student ratio of 1:2.28. That is, there were 100 student textbooks for every 228 students who were present in the class. This proportion does not necessarily reflect the number of students who have the new textbook but, rather, it reports the students who brought their textbook to class at the time of observation. 922 out of 2,172 male students brought the new mother-tongue textbooks to class, while 835 out of 1,833 female students did the same. The textbook-to-student ratios for male and female students were 1:2.36 and 1:2.19, respectively. TABLE 7: TEXTBOOK-TO-STUDENT RATIO, BY GRADE AND LANGUAGE LANGUAGE GROUP GRADE 1 GRADE 2 GRADE 3 GRADE 4 ALL GRADES Tigrinya 1:1.32 1:1.32 1:1.15 1:1.42 1:1.29 Amharic 1:1.83 1:1.42 1:1.14 1:1.22 1:1.40 Afaan Oromo 1:3.92 1:4.24 1:3.63 1:2.74 1:3.57 Af-Somali 1:2.99 1:2.62 1:3.48 1:2.27 1:2.71 Sidaama Afoo 1:4.21 1:6:47 1:6.14 -⃰ 1:5.76 Wolayittatto 1:1.79 1:1.74 1:1.38 1:1.70 1:1.63 Hadiyyisa 1:4.84 1:3.00 1:4.96 1:2.46 1:3.28 ⃰Data collectors did not observe Grade 4 class for this language group. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 23 The quantitative findings in Table 7 show that the textbook-to-student ratio for Grades 1–4 is high for the Sidaama Afoo, Afaan Oromo, and Hadiyyisa language groups. For Tigrinya and Amharic, the textbook-to-student ratio is nearly 1:1. When asked how often students bring the new mother-tongue textbooks to class, 69.9% of teachers reported that their students always bring the new textbook to every mother-tongue language class, whereas 22.6% of teachers reported that their students sometimes bring the textbooks to class. When differentiated by language, a very small proportion of respondents from Oromia (12.8%) indicated that students bring the new textbooks to every mother-tongue language class, compared to other regions. Figure 3 presents teachers’ responses differentiated by language. FIGURE 3. FREQUENCY OF BRINGING MOTHER-TONGUE TEXTBOOKS TO CLASS (GRADES 1–4) The data in Figure 3 clearly show that most students bring their new textbooks to every mother-tongue language class in all languages, except Afaan Oromo. For Afaan Oromo-speaking classrooms, 64.1% of teachers reported that their students sometimes bring their new textbooks to class, while 17.9% of teachers reported that their students rarely do so. When asked about textbook usage, most teachers (97.8%) reported that students use the new mother-tongue textbooks during class and at home when they have homework. Data collectors observed how often students used their textbooks during classroom observations. They found that students in 43 of the classrooms observed often used their textbooks; students in 26 classrooms used their textbooks moderately; and students in 19 classrooms used their textbooks infrequently. Teachers must bring student textbooks and teacher guides and use them to teach in the mother-tongue languages. The classroom observation revealed that 81.1% of teachers brought the new mother-tongue textbooks to the class. The observation also revealed that 76.8% of teachers used their own copy of the student textbooks, or borrowed textbooks from their students. 90 90.9 12.8 58.3 85 95 90 10 6.1 64.1 33.3 15 0 5 0 3 17.9 0 0 0 0 0 0 0 0 0 0 0 0 0 5.1 8.3 0 5 5 0 10 20 30 40 50 60 70 80 90 100 Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadyissa Always Sometimes Rarely Not at all Non-response 24 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV AVAILABILITY AND USE OF TEACHER GUIDES The READ M&E team asked teachers if they had access to the newly developed teacher guides in their schools. Of the teachers who completed the questionnaire, 161 (86.6%) teachers reported that they have the new mother-tongue language teacher guide. However, having the teacher guide is one thing; using it is another. Teachers must rely heavily on the teacher guides to effectively teach the mother￾tongue language. The questionnaire asked teachers how often they used the teacher guide while teaching. The majority of teachers (78.5%) reported that they use the teacher guide every time they teach the mother-tongue language, whereas 5.9% of teachers indicated that they never used the teacher guide while teaching. Figure 4 shows how frequently teachers use the teacher guide, by language. FIGURE 4. FREQUENCY OF TEACHER GUIDE USAGE DURING TEACHING (GRADES 1–4), BY LANGUAGE The majority of teachers in all languages use the teacher guide while teaching, although the proportion of teachers who always use teacher guides varied across languages. For example, all teachers teaching Wolayttatto consistently use teacher guides, whereas only 65% of teachers teaching Hadiyyisa use teacher guides each time they teach the language. The difference is statistically significant (χ2 = 57.076, p < 0.05). In general, the results show that teachers frequently use the teacher guides, because using the new mother-tongue textbooks without teacher guides is a difficult task. Classroom observation data revealed that 65 (68.4%) mother-tongue teachers brought the new mother￾tongue teacher guides to the class. Of teachers who brought the new mother-tongue teacher guide, 50 teachers (76.9%) used the guide during instruction. On the other hand, 77 (81.1%) out of all teachers observed brought mother-tongue student textbook to the class and 73 (94.9%) of them used student textbooks while teaching the mother-tongue language. Comparatively, teachers were observed using student textbooks rather than teacher guides. 38.9% of all teachers observed used both teacher guides and student textbooks while teaching the mother-tongue language. Figure 5 presents the details on teachers’ use of mother-tongue materials during instruction. 86.2 78.8 71.8 70.8 85 100 65 6.9 18.2 2.6 0 5 0 0 0 3 12.8 0 0 0 5 3.4 0 7.7 4.2 10 0 15 3.4 0 5.1 25 0 0 15 0 20 40 60 80 100 120 Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadyissa Always Sometimes Rarely Not at all Non-response USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 25 FIGURE 5. OBSERVATION DATA FOR TEACHERS’ USE OF MOTHER-TONGUE MATERIALS DURING INSTRUCTION (GRADES 1–4) Figure 5 shows that the majority of teachers were observed using student textbooks instead of teacher guides. This pattern was observed for teachers of all languages, except those who taught Af-Somali. All teachers of Afaan Oromo and Wolayttatto used student textbooks during instruction, whereas only 62.5% teachers of Afaan Oromo and 75.0% teachers of Wolayttatto used teacher guides during mother￾tongue language instruction. Most of the teachers who taught Tigrinya, Sidaama Afoo, and Hadiyyisa did not use teacher guides during mother-tongue language instruction. 3.5. READ TA SUPPORT PROVIDED TO CTES READ M&E explored the support that READ TA provided to CTEs for the purpose of improving the mother-tongue teacher training program. All CTE staff confirmed that they were aware of the support that READ TA provided to CTEs. The interviewees mentioned that READ TA provided different types of supports, one of which is training teachers who train mother-tongue language teachers. One CTE staff from SNNPR stated that “READ TA supported the development of modules for CTEs, delivered trainings for CTE staff who train primary school mother-tongue teachers on how to use the new modules, how to teach, and how to evaluate students.” A CTE staff explained how valuable it was to receive training and support from READ TA: READ TA supported us in many ways. We have acquired important experience in regard to preparation of high standard modules for colleges and for the nation as well. Now we have [a] better understanding of preparing a well-articulated module which is technology oriented with appropriate contents and new methods especially that enhances reading skill. According to this staff member, READ TA trained CTE students (prospective primary school teachers) in the revised primary school curriculum so that what they learn in CTE will be consistent with what 70.0% 87.5% 100.0% 30.0% 75.0% 100.0% 92.3% 10.0% 68.8% 62.5% 75.0% 12.5% 75.0% 23.1% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% 120.0% Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadiyssa Teacher’s Guide Student Textbook 26 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV they teach in schools after graduation. READ TA reviewed existing CTE courses and prepared modules for seven CTE courses so that the preservice training would incorporate the newly developed textbooks and teacher guides. Another CTE staff member appreciated the timing of READ TA’s support: Generally, I believe that either directly or indirectly, [the] READ TA project has supported colleges and would-be teachers. This project came in a crucial time when the reading and writing skills of students was significantly low. So the project accomplished a lot and paved the way to solve this problem focusing on producing quality materials and highly trained teachers in the preservice training. Even though respondents generally recognized the value of READ TA’s trainings and support, some CTE staff expressed concerns about the training schedule: “We need such trainings, and our teachers benefited a lot from it. Since in most cases these trainings were not properly scheduled, they created pressure on the teaching learning activities of the CTE.” In addition, several CTE staff believed that the trainings were too short, which didn’t allow participants to internalize and master the topics covered during the training. As a result, participants had difficulties applying what they learned in the classroom. Respondents highlighted the need for longer training sessions and post-training support. However, most CTE staff believed that the training was effective and achieved its intended purpose. According to CTE staff, the revised curriculum calls for a new, technology-oriented teaching approach instead of the “chalk and talk” approach. READ TA supported this new approach by providing material and technological support, including tablets, LCD projectors, cameras, wireless internet access, model videos, and modules among other things. Each mother-tongue language teacher received a tablet installed with modules, reference materials, and other resources. In addition, each CTE has one fully furnished language lab with an overhead projector, wireless internet access, and other equipment. Students who were training to be mother-tongue language teachers practiced teaching in the language lab to improve their pedagogical skills. An interviewee described how READ TA’s technology support had a positive impact on teachers’ motivation: Our college language laboratory is well organized with the support of READ TA.… The technological inputs that were provided by READ TA are creating [a] conducive teaching learning environment. The technological support by READ TA motivates educators to be more creative and hard working. READ TA supported CTEs in a variety of ways, such as providing (1) training, which built participants’ professional capacity; (2) material support, such as modules developed by teachers who received READ TA’s training; and (3) technological support, which facilitated the shift toward a technology-oriented pedagogical approach in mother-tongue language teaching. The majority of CTE staff expressed satisfaction with READ TA’s support. One CTE staff member expressed positive opinions on READ TA’s material support, but found that READ TA’s capacity building support was moderately satisfactory. This CTE staff member reported that he was only partly satisfied with READ TA’s capacity building USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 27 because he believed that mastering technology requires continuous technical support, which READ TA did not provide. IR2: LANGUAGE-SPECIFIC TEACHING AND LEARNING METHODOLOGIES AND STRATEGIES THAT FOCUS ON HELPING STUDENTS LEARN TO READ AND WRITE EFFECTIVELY ARE APPLIED 3.6. TEACHERS’ VIEWS ABOUT THE NEW MATERIALS READ TA developed 112 separate textbooks and teacher guides for Grades 1–8 and adapted 192 region-specific materials in seven mother-tongue languages (a total of 304 separate materials). During the midterm evaluation, READ M&E assessed teachers’ views on the new Grades 1–4 textbooks and teacher guides, which have been in use for quite some time. However, textbooks and teacher guides for Grades 5–8 have reached most schools but have not yet been used for language instruction. Grades 5–8 teachers have not yet been trained to teach with the new materials, thus limiting their usage in the classroom. The team explored Grades 1–4 teachers’ views on the new student textbooks and teacher guides. They used eight items to assess teachers’ level of satisfaction with the contents of the new textbooks and guides; nine items to assess the materials’ content and their link to the embedded learning methodologies and strategies; and three items on a five-point Likert scale to assess the materials’ physical qualities. A higher score on the scale shows satisfaction and positive views on the materials. To simplify data presentation, READ M&E combined the two positive scale values (very satisfied and satisfied) into a single scale value (satisfied). Similarly, READ M&E merged the two negative scales values (very dissatisfied and dissatisfied) into one scale value (dissatisfied). The middle scale value was used to indicate teachers’ mixed views or uncertainty about their satisfaction with the materials. The results show that the majority of teachers (72%) expressed positive views on the materials’ content, whereas 23.7% were uncertain about their satisfaction. The teachers who expressed uncertainty cited concerns over the continuity of lessons, grade-level appropriateness, level of difficulty, readability, relevance, and comprehensiveness. Only 4.3% of teachers reported unfavorable views about the new materials’ contents. 156 (83.9%) respondents stated that the continuity of lessons is within grade level, and 120 (64.5%) respondents indicated that they were satisfied with the continuity of lessons from one grade level to the next. When differentiating teachers’ responses by language they teach, the results show a statistically significant difference between the language groups (χ2 = 35.547, p < 0.05). Nearly 96% of respondents from Somali and 95% respondents from Wolayttatto expressed satisfaction with the content of the student textbooks and teacher guides. The details of the responses are presented in Figure 6. 28 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV FIGURE 6. TEACHERS’ LEVEL OF SATISFACTION WITH THE CONTENTS OF THE NEW MOTHER-TONGUE MATERIALS (GRADES 1–4) A high proportion of teachers (45%) who teach Afaan Oromo expressed mixed views or uncertainty about their satisfaction with the contents of the student textbooks and teacher guides, while 31% of teachers who teach Tigrinya feel the same. A relatively high proportion of teachers (15.2%) who teach Amharic were not satisfied with the contents of the new mother-tongue materials, compared to other language groups. The data also revealed that 81.2% of teachers showed positive viewpoints and expressed satisfaction with the link between the content and the learning methodologies embedded in the new materials. 15.1% of teachers expressed mixed views or uncertainty about their satisfaction with the same, while 3.8% expressed their dissatisfaction. Figure 7 summarizes the data, differentiated by language. 15.2 2.5 4.2 5.0 31.0 21.2 45.0 0.0 25.0 5.0 20.0 69.0 63.6 52.5 95.8 70.0 95.0 80.0 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadyissa Unsatisfied Unsure Satisfied USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 29 FIGURE 7. TEACHER’S OVERALL SATISFACTION ON THE LINK BETWEEN CONTENTS AND EMBEDDED METHODOLOGY, BY LANGUAGE As depicted in Figure 7, there is a slight, statistically insignificant difference in the proportions of teachers in all language groups who were satisfied with the link between the content and teaching methodologies incorporated in the new materials. All teachers teaching Wolayttatto and a large proportion of teachers teaching Tigrinya (89.7%) showed more satisfaction than those teaching Amharic. Teachers were asked to provide feedback on the physical quality of the new mother-tongue student textbooks and teacher guides. In particular, they were asked about the quality of the cover pages; paper and binding; and print, including text and illustrations. Most teachers (76.3%) were satisfied with the physical quality of the new mother-tongue materials, whereas 21% of teachers expressed moderate views. Fewer than 3% of teachers expressed dissatisfaction with the physical quality. Figure 8 summarizes teachers’ responses to the new materials’ physical quality, differentiated by language 0.0 12.1 0.0 8.3 5.0 0.0 0.0 10.3 21.2 22.5 16.7 10.0 0.0 15.0 89.7 66.7 77.5 75.0 85.0 100.0 85.0 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadyissa Unsatisfied Unsure Satisfied 30 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV FIGURE 8. TEACHERS’ SATISFACTION WITH THE PHYSICAL QUALITY OF THE NEW MATERIALS, BY LANGUAGE (GRADES 1–4) As shown in Figure 7, all teachers who teach Af-Somali and nearly all teachers who teach Sidaama Afoo and Wolayttatto were satisfied with the physical quality of the new materials, compared to 48.5% of teachers who teach Amharic and 55% of teachers who teach Afaan Oromo. The difference is statistically significant (χ2 = 46.993, p < 0.05). In general, quantitative and qualitative data show the majority of teachers in all language groups, except those who teach Amharic and Afaan Oromo, believe that the student textbooks are well prepared and meet the basic standards of quality. 3.7. SELECTING MASTER AND TEACHER TRAINERS READ TA is responsible for planning, coordinating, and financing activities related to teacher training. The training included participants selected from different levels of the education system, including primary and high school teachers, teacher trainers from CTEs, and experts from education bureaus. READ TA conducted the training in cooperation with professionals from higher education institutions 9.1 2.5 5 17.2 42.4 42.5 5 10 82.8 48.5 55 100 95 95 90 0 10 20 30 40 50 60 70 80 90 100 Tigrigna Amharic Afaan Oromo Af-Somali Sidaamu Afoo Wolayttatto Hadyissa Unsatisfied Unsure Satisfied USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 31 and with national and international consultants. READ TA staff from Somali Region explained the process: The training was based on [a] cascading model and conducted in three steps: training of master trainers, training of trainers, and training of the teachers. The criteria for selecting participants at various levels were developed by MoE in collaboration with READ TA. READ TA provided technical support to the MoE and RSEBs and identified the basic qualification required from master and teacher trainers. Based on this, MoE articulated the whole criteria and gave [these criteria] to regions, and the regions also sent these criteria to zonal education departments to select and send the qualified professionals. An expert from Amhara listed a set of criteria for selecting teacher trainers: a bachelor’s degree or higher; experience teaching a mother-tongue language in primary schools, currently teaching in secondary schools, knowledge of Grades 1–4 mother-tongue language curriculum, and involvement in similar, previous trainings organized by READ projects. The selection criteria are used to select participants for the first round of trainings, with minor modifications. READ TA identified a significant number of professionals who met the criteria, making it easy to select qualified individuals with experience and expertise. A READ TA staff from SNNPR explained that READ TA involved different professionals in the material development process, which was an important, yet unprecedented act. Knowledgeable people with expertise in different areas, including language and curriculum development, played an important role in developing the mother-tongue materials. READ TA, RSEBs, and the MoE selected the master trainers, and WEOs selected the teacher trainers. At the beginning of the project, the MoE and READ TA agreed on a set of criteria for selecting master and teacher trainers. Master trainers and teacher trainers must be mother-tongue language teachers with significant experience and a bachelor’s degree or higher. In addition, they must have fluent reading, writing, and speaking skills. The team favored individuals had experience teaching mother-tongue languages in primary school and knowledge of the Grades 1–4 mother-tongue language curriculum. Some of the master trainers contributed to the preparation of the educational materials; others had many years of experience in training teachers; and yet others were high school teachers with many years of experience in teaching. The team used the same criteria to select master trainers and teacher trainers for the second-round training. READ TA and Save the Children collaborated to administer and evaluate the regional trainings for Grades 5–8 teachers. USAID provided the training funds, while READ TA managed the grant funds. Save the Children was responsible for training the trainers and handling the logistics related to the second￾round of trainings. The MoE and RSEBs benefitted from the training programs, which were held in seven languages and in five regions of the country. The trainers used the same training manual for both rounds of training, as well as teacher guides and student textbooks. Staff from RSEBs, Save the Children, WEOs, and READ TA supervised the training programs. All of the stakeholders shared the responsibility of managing the teacher trainings: READ TA, MoE, RSEBs, ZEOs, WEOs, and Save the Children. READ TA took the lead in a collaborative effort to develop the training material, train master trainers and teacher trainers, and train the mother-tongue language teachers. 32 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 3.8. MANAGING REFRESHER TRAININGS RSEBs in collaboration with READ TA selected 646 (257 women and 389 men) master and teacher trainers from all languages to receive refresher trainings, and a few additional teacher trainers from the Addis Ababa City Administration and from Amhara. The selected master and teacher trainers were language teachers who were mainly from CTEs and senior secondary schools. The refresher training for master trainers took place in Addis Ababa while the training of teacher trainers took place in regions. The refresher training for master trainers was facilitated by READ TA language specialists in collaboration with consultants who developed the training materials. The refresher trainings for teacher trainers were facilitated by master trainers of each language. Interviews with READ TA staff revealed that each project partner had a role in conducting the refresher trainings. READ TA, for example, was responsible for handling technical content of the training lessons. Save the Children was responsible for managing the training logistics, and RSEBs and CTEs were responsible for selecting the trainers and following up on training-related issues. 3.9. GRANT MANAGEMENT USAID redirected a grant initially allocated to the MoE and awarded it to READ TA. USAID provided grant funds to READ TA in phases, which READ TA used to train teachers. Funding in subsequent phases were contingent on achieving preset conditions in the preceding phase. Staff from the READ TA headquarters believed that they managed the grant according to preset criteria. One interviewee felt that the grant management task provided the project team with additional opportunities by enabling the team to look at what was happening at the lower level: The task of grant management put pressure on the facilitators of the trainings to be very serious about what they do. They had to make sure that the criteria for each phase had been achieved for the work to be successful. We ourselves had seen these when we visited the training centers. 3.10. CAPACITY BUILDING The READ TA project built the capacity of everyone involved in the material development process. The material developers gained new experiences and skills by working with trainers and consultants. A material developer from Somali stated, “…the major and remarkable achievements made by READ TA were its capacity-building efforts, which enabled the mother-tongue language material developers and CTEs staff [to] develop very important skill and gain knowledge that can be considered as [a] resource for future use.” Another material developer agreed, saying that: The capacity building effort helped all, such as mother-tongue material developers, story writers, designers, illustrators, trainers, reviewers, and other consultants in every area. I strongly believe the capacity-building process was successful. A READ TA employee from the headquarters emphasized how successfully READ TA built local capacity, “…the work that is done now is mostly done by the local experts with the support of the foreign experts. I believe this is a huge achievement.” USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 33 READ TA also built local capacity by training education leaders to support mother-tongue education. As a project respondent explained, READ TA identified the MoE’s capacity development needs and organized different capacity building trainings for the ministry, regional, and education leaders. In addition, READ TA build ministry capacity by training ministry staff on monitoring and evaluation and leadership. 737 educational leaders participated in the project’s training programs. Stakeholders hope that these trainings will enable the regions to sustain READ TA activities beyond the life of the project. Respondents expressed satisfaction with the way that READ TA has managed its activities and accomplished its objectives. In particular, it has made significant progress in materials development, training master and teacher trainers, grants management, and capacity building. IR3: LANGUAGE TEACHING AND LEARNING SUPPORTED BY APPROPRIATE TECHNOLOGY AND TEACHER AIDS 3.11. DEPLOYING AND USING ICT PACKAGES FOR CTES According to interviewees from CTEs and READ TA, all mother-tongue language teacher trainers at CTEs received tablets loaded with five new and revised modules. In addition, READ TA established mother￾tongue language laboratory rooms and equipped them with a Synology server, the latest projector, LCDs, CDs, a document camera, training videos, and teaching and learning tools. One staff from the CTE in Hawassa stated: We have videos loaded on the tablets. For example, [the] phonological awareness module has videos on blending and segmenting. We have videos on texts like expository texts. I think they are more than 20. Textbooks and teacher guides are also loaded on the tablets. A respondent from the CTE in Adwa echoed the sentiment of many others interviewed: Mainly, we were given tablets to each of us which had all documents/resources including videos uploaded. It is equivalent to 21 GB that addressed the five principles: from sound recognition to concept understanding. The fact that the videos are presented using LCDs, the company has also equipped us with this equipment. The LCD is one of the best qualities and is easy to use it upon demand. It works for [a] long duration without any problem. We have also camera to take pictures in line with the concepts to be taught….Our college language laboratory is well organized with the support of READ TA. It also gave us projectors, documents, and other important teaching materials. Similarly, a staff member from the Debre Birhan CTE in Amhara said: The language laboratory is established in our college in one room, and READ TA had provided us with the server, tablets, video cameras with lenses, magnifiers, cables, jacks, and switches, LCD projectors, and other important materials like white garments. The same respondent emphasized the positive impact of READ TA’s technological support: 34 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV The technological support made available by READ TA motivated language educators to be more creative and diligent in their work. In general, technological inputs provided by READ TA have created [a] conducive mother-tongue language teaching environment in our college. A staff member from the CTE in Hosaina described how useful the tablets were: The materials are useful. With these uploaded materials one can teach a large number of students easily in a very interactive method using video, camera, and LCD projector. However, some interviewees were dissatisfied with some aspects of READ TA’s technological support. A respondent from Hosaina said: Many videos in English and Hadiyyisa are uploaded that show instructors teaching sound names, letter names, word segmenting and blending, different genres and different types of writing systems which are produced by Whiz Kids. However, some of the videos are not authentic, and they are far behind the standard, because the teachers recoded in the videos are not language professionals who cannot be models. In fact, we have identified the low-standard videos and communicated this to READ TA/RTI to discard these less standard videos produced by Whiz Kids. And there is also a need for additional videos that focus on poetic reading of poems. Some respondents complained that they couldn’t easily open the uploaded videos on their tablets. In addition, very few interviewees believed that READ TA’s efforts reached the classroom level. According to them, the recorded lesson videos did not reach schools, but instead remained in the CTE. All interviewees mentioned that READ TA provided numerous trainings to help teachers effectively use these technological materials. According to one staff member from the CTE in Assela, Oromia: READ TA has trained the ICT professionals of the college so that they can manage, maintain, and make the ICT package sustainable. The training should be continuous, and the IT staff and instructors should be continuously helped in the use of the technologies. Technologies are changing now and then. However, this has not happened in my college. An employee at the Nekemte CTE expressed concerns on the practicality of using the technological equipment provided by READ TA: We are not happy with regard to the use of technology by the college. Some of the materials were not being used for the purposes they were meant. Others were stolen. Still others were misplaced. Therefore, I have some reservations on the practical implementations of the materials for the purposes they were received. A READ TA respondent from Oromia explained that there is a knowledge and skill gap among CTE teachers. Younger teachers are more skilled in using the technology, while older teachers are reluctant to use it. Even after receiving refresher training, some CTE staff do not use the technology as expected. There are concerns about the authenticity of videos; the proper use of technology for their intended purpose at CTEs; and READ TA’s inability to provide direct technological support to primary schools. Nonetheless, nearly all experts and CTE staff agree that READ TA’s ICT support to CTEs has been USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 35 extremely valuable. They also agree that READ TA has played a significant role in implementing the revised mother-tongue language curriculum in seven languages. 3.12. ASSISTIVE TECHNOLOGY The research team asked READ TA staff and teachers whether children with special needs received assistive technology support. Respondents explained that assistive technology support is in the pilot stage. A READ TA employee described the progress they made toward providing assistive technology support: In order to address the needs of students with hard of hearing and visual impairment, a pilot project on the use of assistive technology is underway in 10 sampled schools. This is only on a trial stage that did not include other schools. That means READ TA is trying on 10 out of 8,000 schools, which implies that READ TA has not started implementing the use of assistive technology on [a] large scale. The focus was on those students who are slightly challenged to hearing and see[ing]. The pilot focused on preparing special teacher guides that would help teachers in accommodating these groups of children in the teaching of mother-tongue language. According to READ TA staff, the project provided assistive technology to children with disabilities in pilot schools: Assistive technology was another technology made available to those students who have problem[s] of hearing and seeing. A [s]martphone was provided to the teachers of Grade 2 as a pilot to help teachers provide appropriate support for students to read the letters of Afaan Oromo correctly. READ TA provided all teachers in pilot schools with two trainings on the use of assistive technology. READ TA staff followed up with the teachers to ensure that they provided assistive technology to their students with disabilities. READ TA developed inclusive multimedia lesson plans and worked with inclusive education experts to integrate explicit accommodations for children with vision or hearing impairment into the newly developed teacher guide. READ TA embedded audio-files for all phonemic awareness activities and stories in the lessons, which is in keeping with the evidence-based principles of Universal Design for Learning. However, a READ TA staff from Oromia explained that there were challenges around encouraging teachers to use assistive technology. Although READ TA provided a variety of trainings to teachers, including refresher trainings, they did not effectively encourage teachers to use the assistive technology materials at least during the data collection period. Although the PMP did not clearly specify a strategy for introducing assistive technology materials, the use of assistive technology software is included under IR 3, indicator 3.2 (number of supplemental teaching and reading promotional aids developed for improved teacher training or reading instruction in seven mother-tongue languages or English) had been part of READ TA's cooperative agreement since the beginning of READ TA. According to the PMP, READ TA was supposed to introduce assistive technology in seven mother-tongue languages. However, READ TA was still piloting the introduction of assistive technology toward end of the project. Please note that the plan to conduct assistive technology pilots in select schools RTI's proposal and. 36 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 3.13. DEVELOPING MULTIMEDIA TEACHING AIDS FOR GRADES 1–8 READ TA developed two sets of multimedia teaching aids during the life of the project. READ TA produced model teacher videos for Grades 1–8 during Years 3 and 4 and produced eight 10-minute “Tsehay Loves Learning” episodes in seven mother-tongue languages during the last year of the project. By introducing technology learning tools to CTEs and producing videos for preservice teacher training, READ TA is providing preservice teacher trainees with knowledge of multimedia learning possibilities. The goal is to encourage these future teachers to use multimedia teaching tools when teaching primary school pupils. READ TA worked with Whiz Kids Workshop to develop model teacher videos for seven mother￾tongue languages and produced the model videos. An expert from READ TA summarized the production activities: We were working with Whiz Kids [Workshop], a local organization involved in the production of multimedia materials for education purpose[s]. CTE mother-tongue language teachers, in collaboration with primary school teachers, recorded classroom teaching practices of teachers who were carefully selected. The lessons recorded were repeatedly edited, validated, and were recorded as model teacher videos. These video materials were uploaded on the tablets of mother-tongue teachers at colleges of teacher education. READ TA and Whiz Kids effectively collaborated to produce multimedia learning materials and load them onto mother-tongue teachers’ tablets in CTEs. Nonetheless, a respondent from Nekemte expressed doubts about using multimedia learning aids in school classrooms: I do not think that the READ TA’s effort reached the classroom level in terms of supporting mother-tongue language instruction with multimedia aids. I mean, the recorded teacher videos were not provided to schools. They remained in the CTEs. Despite the challenges with implementing assistive technology and the development of multimedia teaching aids for Grades 1–8, nearly all respondents acknowledged and highly appreciated READ TA’s contribution in the deployment and use of ICT packages for CTEs. Experts from RSEBs and staff from CTEs expressed their satisfaction by repeatedly saying, “Yes, we are aware of READ TA’s contribution and effort to make use of technologies to incorporate multimedia presentation in the training modalities.” IR4: TECHNICAL SUPPORT TO RSEBS AND MOE FOR READ INSTITUTIONAL IMPROVEMENT 3.14. DEVELOPING MODULES FOR PRESERVICE TEACHER TRAINING READ TA played a leading role in developing modules for the preservice teacher training. At the central level, mother-tongue teachers were trained by local and international professionals and consultants. READ TA, with the help of Florida State University, reviewed the existing teacher training curriculum and USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 37 developed modules for training preservice mother-tongue language teachers. A READ TA staff member from the headquarters described the module development process: In line with the materials prepared for the training of mother-tongue language teachers, READ TA had developed and adapted four modules for preservice training for mother-tongue language teachers of CTEs in Years 3 and 4 of the project period and developed and validated three more modules in the fifth year of the project period. The English version of the developed module was adapted into seven mother-tongue languages. All in all, READ TA developed seven modules during the project period. The process of module development took into account the ability of students, cultural practices of the communities, and also the views of teachers. After the modules [were] adapted into the seven languages, a sample copy of each module was sent to respective colleges for evaluation. Based on the feedback, first the English version was revised, and following the revision of the English version, the adapted modules were revised two times, and then the final version of the adapted modules was sent to each CTE for use. Another READ TA staff member from the headquarters stated, The validation, the trainings, the assessment work [were] also conducted under the support of READ TA.…All mother-tongue teachers who would teach the modules, particularly in the first phase, received induction training in Addis Ababa. CTE staff witnessed the positive impact READ TA’s work on the preservice training curriculum. A staff member from the CTE in Hawassa mentioned that the modules were prepared and presented differently. The materials were of high quality compared with those of the previous modules. RSEB experts echoed this sentiment. Additionally, a team of teachers trained in monitoring and evaluation provide post-implementation follow-up. These teachers evaluate the teaching-learning process through classroom observations with the purpose of improving the modules. A staff member from a CTE in Nekemte, Oromia, pointed out that some modules had as many as 200 pages and contained many repetitions. There were also frequent translation issues due to conceptual differences between the English version and the Afaan Oromo translation and adaptation. The interviewee stated, The modules are very good. However, some of the modules are very bulky, and there are repetitions as well. But, it is common to see such repetition in any module preparations. However, frequency of the repetition in the prepared modules seems very high, and the volumes are large. But…the modules are still very good and appropriate for the trainees. However, CTE respondents acknowledged and highly appreciated READ TA’s support in module development as well as training. A mother-tongue language instructor from the Hosaina CTE described how READ TA’s support benefitted the CTEs: I know READ TA supported our college in the development of modules for [the] Hadyissa preservice mother-tongue teacher training program. For any college, a well-developed high standard teaching-learning material is a requirement so as to produce efficient, competent, and skillful teachers for primary education.…The materials developed are of high quality and are closely integrated to the materials that are used in the schools. 38 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Another respondent from the CTE in Debre Birhan also praised READ TA’s support: There is much improvement now, because during curriculum revision, care had been taken in relation to alignment of the materials. Everything was done with the assumption that graduates from the college can discharge their activities with confidence. Before the intervention by the project, the training offered was not focused. So far, students used to graduate from CTEs, including mine, to teach without getting sufficient knowledge on what to teach and how to teach mother-tongue language in schools. Now that problem is solved. The approach we follow is completely student centered. There is [a] big difference anyway. So I can say we got a very good support from the project. READ TA reviewed the existing curriculum, developed preservice teacher training modules, and provided follow-up support. There is a general consensus that READ TA’s support in preservice mother-tongue language teacher training has greatly benefitted CTEs. 3.15. ADDRESSING GENDER SENSITIVITY AND DISABILITY INCLUSION Gender sensitivity and disability inclusion in the development of modules has been one of the core issues of concern. Gender and disability experts were included in the material development team and helped develop the mother-tongue textbooks and teacher guides for Grades 5–8 for each language. The gender and disability experts participated in the material development process starting from determining the scope and sequence to the end of material development to keep gender balance and address disability issues throughout the entire process. A material development expert from Sidaama Afoo said, Gender and disability issues were discussed before starting the actual preparation. From the discussion, we learn[ed] that we need to assign equal roles for both males and females in the materials. As a result, the material developers developed the materials in such a way that they addressed the issue of gender equity and disability issues. For example, if a male was assigned a leadership role in the text, we did the same for females. It was prepared in [a] balanced way. Across the themes and illustrations, gender equality was maintained. We did this with knowledge. Similarly, disability issues were well addressed while developing the materials. Attention was paid to the kind of language used and text choices to make sure that both sexes are represented and the issues of disability are addressed. A material developer from Oromia discussed the extent to which the materials addressed children with special needs: In terms of including issues as topics to be discussed in the materials developed that deal with students with special needs, I cannot say that the materials addressed these kinds of issues. There are no separate sections dealing with these issues. However, in terms of properly representing children with special needs, I do not see any problem. Another material developer from Amhara said, If, for example, four or five pictures were used to illustrate certain concepts, one or two pictures were dedicated to issues related to special need students. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 39 The teacher guides also contain instructions that help teachers to engage students with special needs during the teaching-learning process. Students who have disabilities are represented in the pictures and the text. Furthermore, the new materials address issues concerning disability, gender, and socioeconomic status. Material developers assured the research team that gender and disability were properly addressed during the material development process. A material developer from the Wolayttatto language group summarizes the team’s efforts to include gender and disability, which was confirmed by other respondents: Serious attention was given to gender balance, and females were sufficiently depicted in pictures and textual matters. There was 50% representation of females in the materials developed. We also considered issues related to disabilities, culture, religion etc., in our everyday preparation of the materials. Women are sufficiently depicted both in the texts [and in the] examples and illustrations. There is no gender bias. We tried to make the materials free from gender stereotypes, and we were careful while addressing issues related to disabilities. Education professionals made notable contributions by incorporating gender and disability during the materials development process. 4. PROJECT LIMITATIONS AND CHALLENGES The READ TA team faced a number of challenges and limitations during project implementation. The following are some imitations and challenges that the project encountered during implementation. 4.1. REGISTRATION READ TA encountered major challenges related to securing registration as a legal institution. Although READ TA became registered after a very frustrating and difficult process, staff encountered registration￾related issues throughout the life of the project. Every time the project submitted an annual report to the Charity and Society Agency, for example, READ TA had to re-explain the project details and defend its registration. 4.2. WORKING WITH THE MOE AND RSEBS READ TA also struggled with the MoE and RSEB’s lack of coordination and slow decision-making processes. READ TA staff explained that this was one factor that delayed implementation of planned activities, such as reviewing the English language textbooks and teacher guides for Grades 1–8 and training teacher mentors. Whenever there was a task at hand, it was hard for READ TA staff to figure out which part of the ministry was responsible for it. Most of READ TA’s planned activities were meant to be accomplished by MoE, RSEBs, and other partner organizations. The MoE and RSEBs identified and selected experts who participated in material development. The READ TA staff reported that in some cases, the MoE and RSEBs did not identify and select the most experienced experts who can accomplish the task, but rather, chose individuals with the purpose of providing them with work experience. One READ TA staff member explained: 40 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV There were times we requested RSEBs to send us experts we know who had the competence to do the work, but RSEBs used to send us experts who were new and less competent to do the work. These experts did the work below the required standard and were submitting the tasks late. This kind of situation forced READ TA to work beyond the schedule. Another READ TA staff member shared similar comments, Another challenge was related to compromising the desire to build capacity of regional experts, maintaining the standard and quality of the outputs, and meeting the time for delivering the outputs. This challenge was reflected in the recruitment of consultants for the work. While working on different modules, we wanted the same people that worked on the previous ones to work on the next modules, but we couldn’t agree on this with CTEs, which wanted different people to take part in the work. One READ TA employee described how the project experienced difficulties interacting with individuals with different interests and goals: There were times where we wondered whether some directorates and some people in the regions were working for their country. Many times, the approaches we followed to accomplish the task were tuned with the personality of the person with whom we were working. There were proactive directorates, but also difficult people. The project staff also experienced challenges working with some government staff who sought to personally benefit from project activities. At times, READ TA had difficulties conducting consultative meetings without some form of payment, such as a per diem. 4.3. DELAY OF PLANNED ACTIVITIES An expert from the Somali RSEB expressed concern that READ TA failed to achieve some of its planned activities: READ TA, for example, planned to train teachers teaching Grades 5–8 years ago, but it has not implemented such activity yet, and this has brought delay in the use of student textbooks and teacher guides prepared for Grades 5–8. Therefore, I can say the project has not accomplished the tasks as they were planned. 4.4. DIVISION OF ROLES AND RESPONSIBILITIES READ TA and partner organizations have divided roles and responsibilities among themselves. READ TA experienced considerable challenges in getting other parties to do their work on time. According to one READ TA staff: While READ TA’s responsibility was to prepare mother-tongue materials for seven languages, printing and distribution of the materials was the responsibility of the MoE. MoE was working with other partners to get this done. The ministry had its own way of doing things. However, the task of printing and distributing the materials took too much time, and this had an implication on the implementation of the project. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 41 4.5. DATA ACCURACY AND TIMING Project staff were unable to obtain accurate data that would help READ TA plan teacher trainings. Implementing partners provided READ TA with inaccurate figures, which READ TA used to develop inappropriate timelines. A READ TA staff mentioned, When partners were asked to provide data, it took them months to respond, and when they responded, it was with wrong figures, and this had affected planning of the training of teachers. 4.6. MATERIAL DEVELOPMENT AND REVISION READ TA staff explained that its activities are participatory. However, an expert from the Hadiya education department said that his zone was not invited to participate in revising the English language textbooks and teacher guides for Grades 1–8. The expert stated, Since we are the implementers of the curriculum, I believe we should have been allowed to participate in the revision work. READ TA also failed to train teachers for Grades 5–8, which presented challenges to implementing the revised curricula. An expert from the Somali Education Bureau explained, The project is on the verge of termination, but teachers teaching [the] mother-tongue languages for Grades 5–8 are not yet trained. Another expert from the same region affirmed that teacher training is a prerequisite for curriculum implementation: English language textbooks have been revised and made ready for printing at about the end of [the] READ TA project. I believe that such endeavor should be followed by the training of teachers teaching English for Grades 1–8. Offering the training in line with the revised materials and making teachers ready for teaching the subject would remain a huge challenge for its implementation. 4.7. TEACHER TRAINING The project also noted a shortage of books during teacher training. Some regions failed to provide an adequate number of student textbooks and teacher guides in time for teacher trainings. Regional experts believe that the teacher trainings adequate for teaching with the new materials. However, an expert from the Tigray RSEB explained that not all teachers have the same skill and commitment to use what they learned during training. READ TA did not develop a tool or mechanism that provides these teachers with continuous support, which is a significant limitation in its curriculum implementation strategy. 42 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 5. PROJECT SUSTAINABILITY The evaluation team examined READ TA’s project exit and sustainability strategy. READ TA staff were asked to describe any efforts the project has made to ensure sustainability beyond the life of the project. One staff member described how READ TA is consolidating the project’s process documentation: READ TA had finished the preparation of process documentation and process guidance documents for curriculum development and use. READ TA has been encouraging those partners involved in the work to do the same so that there would be guidelines for someone who wants to come and make changes and contributions in this regard in the future. Every template, tool kit, and even PowerPoints have been documented for reference for future use. Other READ TA staff also mentioned that READ TA hosted the national reading symposium, which contributes to the project’s exit strategy. Regional experts note that RSEBs have worked to ensure project sustainability. An expert from the Tigray RSEB stated, I do not believe that the regional education bureau has [a] written plan which indicates a direction to continue this project at project close-up. However, there are movements on the part of the regional office, although initiated by READ TA, in order to own and continue the project. It is trying to identify what must be done in this regard. The expert said that the RSEB is aware of the need for project sustainability, and is working with other partners to prepare supplementary reading material. This will help RSEB staff build internal capacity to do so themselves. An expert from the Hadiya ZED reported that the zone has no plan for sustaining the project, but mentioned that the education department is interested in extending its work, and has submitted a proposal to the zone administration. The zone has begun preparing to revise the mother￾tongue materials for Grades 9–12. Apart from preparing materials and documenting activities to serve as guides for future activities, it is difficult to claim that READ TA had a well-defined exit and sustainability strategy. 6. CONCLUSION READ M&E conducted a final performance evaluation of READ TA activities. The evaluation investigated 10 questions under four intermediate results and used data findings to demonstrate the project’s successes and limitations. A close analysis of data results shows that READ TA has successfully accomplished the project activities per the contractual agreement. The following conclusions can be drawn for each of the intermediate results. IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In-Service Teacher Training Developed READ TA introduced new material development practices in Ethiopia and successfully produced 304 Grades 1–8 student textbooks and teacher guides in accordance with rigorous procedures. READ TA used the lessons learned from developing student textbook and teacher guides for Grades 1–4 to USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 43 facilitate a smoother development process for materials for Grades 5–8. The team used procedures to ensure that the materials were aligned with one another, and with the MLCs. READ TA adapted the new mother-tongue instructional materials to 12 languages, including languages that were not originally included in the READ programs. In addition, READ TA successfully revised the English textbooks and teacher guides for Grades 1–8. Local English language professionals used rigorous procedures to revise the English textbooks and teacher guides. However, the materials are not yet ready for distribution. The collaborative work among READ TA, MoE, RSEBs, and other partner organizations were essential to the success of the development and review of the mother-tongue materials. In spite of the challenges that the partners encountered, they collaborated in an effective and efficient manner. Although READ TA was not responsible for distributing the mother-tongue materials, the project effectively disseminated the new textbooks and teacher guides. Data collectors observed a 1:1 textbook-to-student ratio in classes that teach in Tigrinya and Amharic. Mother-tongue materials for Grades 5–8 reached most schools but have not yet been distributed among teachers and students. Teachers have not been trained in the new Grades 5–8 materials, and as a result, many schools have not yet started using them. READ TA has successfully supported preservice mother-tongue language teacher training programs. Most CTE staff are satisfied with READ TA’s technological, technical, material, and capacity-building support. Interviews with CTE staff also revealed that mother-tongue teachers relied on technological support when teaching the modules uploaded in the tablets. IR2: Language-Specific Teaching and Learning Methodologies and Strategies That Focus on Helping Students Learn to Read and Write Effectively Are Applied A majority of teachers, except those who teach Afaan Oromo, believe that READ TA prepared high quality mother-tongue materials. The teachers expressed satisfaction with the materials’ content, teaching methodology, the absence of words or phrases from other languages, and physical qualities of Grades 1–4 student textbooks and teacher guides. READ M&E did not gather data on teachers’ perception of the physical quality of mother-tongue materials prepared for Grades 5–8. READ TA was responsible for managing teacher training grants. READ TA collaborated with Save the Children to provide refresher training to Master Trainers and Teacher Trainers, and the MoE and RSEBs facilitated a10-day training for teachers. The teachers appreciated the trainings. IR3: Language Teaching and Learning Supported by Appropriate Technology and Teaching Aids READ TA equipped CTEs’ mother-tongue language labs with ICT materials such as servers, the latest projectors, LCDs, CDs, document cameras, and videos designed to improve mother-tongue language teachers’ pedagogical skills. These material contributions provide invaluable support to CTEs. Furthermore, READ TA is piloting the use of assistive technology. Every mother-tongue language teacher in the CTE has received a tablet with modules and video materials uploaded. Mother-tongue 44 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV teachers used these tablets while training prospective mother-tongue language teachers in the pre￾service training program. IR4: Technical Support to RSEBs and MoE for the READ Institutional Improvement READ TA conducted different trainings on educational leadership, monitoring and evaluation, and selected topics as a capacity-building endeavor. 737 educational leaders benefited from the capacity￾building trainings throughout the life of the project. 7. LESSONS LEARNED READ TA worked closely with the MoE and RSEBs throughout the project, which facilitated a smooth working relationship and improved collaboration. However, the division of roles and responsibilities among the three stakeholders opened the door for delays and misunderstandings. For example, in some instances, the responsible party failed to complete their role in a given task, which left the task unattended. In other cases, concerned government officials challenged implementation of project activities and denied knowing about the project activities. In the future, key stakeholders should agree upon and clearly communicate each party’s role and responsibilities. These roles and responsibilities must be outlined in a signed agreement between all parties, so that staffing changes will not affect project implementation. READ TA, MoE, and the RSEBs must ensure continuity of project activities, regardless of staff turnover in government positions. Staff turnover and decentralized agencies affect the continuity of project activities. For example, when certain positions experience turnover, documents and knowledge may not be smoothly transitioned to new staff. New government hires are generally unfamiliar with the project, and may be unwilling to provide support when needed. This may affect coordination among team members and may result in project delays. To rectify this, partner organizations organized periodic meetings to follow up on project implementation and address any outstanding issues. Moreover, government institutions lack a focal person who can direct project implementers to corresponding different directorates, departments, or sections. This presents a major challenge that should have been incorporated into the project design. Future projects should consider these issues during project design and development. Projects must design their approach to reach the grassroots level. READ TA, as it was designed, was not involved in direct implementation of the READ program. Its task was to review the curriculum, develop mother-tongue materials, and build the capacity of MoE and RSEB staff. It did not visit schools to evaluate the mother-tongue materials’ strengths and weaknesses and to assess how teachers and students used this support. Developing high-quality materials alone is not a sufficient approach, the project design must incorporate interventions at the classroom level to encourage their use of the new materials. Therefore, future project implementers should design and develop reading projects that incorporate school-level interventions that assess the change in classroom behaviors, and identify best practices and lessons learned. READ TA initially developed mother-tongue materials for seven languages and successfully adapted the new mother-tongue materials for 12 languages. SNNPR also adapted the new materials to selected languages in the region. This approach helped READ TA reach diverse communities, beyond the communities originally targeted to receive interventions. However, READ TA should have originally USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | 45 designed its project activities to reach diverse language communities. Future projects should be more mindful of project design and approach, so that the project will address the needs of beneficiaries who are not originally included in the project. What do the findings mean to the continuing improvement of READ in Ethiopia? Qualitative and quantitative data from diverse groups of READ TA stakeholders indicate that, overall, READ TA successfully implemented project activities per the contractual agreement. That being said, the challenges identified raise questions about the extent to which READ TA anticipated early implementation pitfalls that could have affected the deep implementation of project activities. Lessons from the implementation sciences literature should help similar projects be more deliberate about anticipating such pitfalls at least at the project exploration and installation phases. Fixsen, Blasé, Metz, and Van Dyke (2015) 1divide the implementation continuum into four stages, exploration, installation, initial implementation, and full implementation. Exploration refers to the activities undertaken to consider the need for change, the availability of the necessary supports to satisfy the identified need(s), and the readiness of system leaders to own, initiate, and lead the process of improvement beyond initial program implementation. The installation stage refers to foundational provisions targeted at ensuring that READ TA was led/championed by project and government staff who have the power, interest, and the technical, relational, political, and administrative know-how necessary for learning from early implementation dips (Fullan, 2014)2 to maximize deep implementation and sustainability of READ TA. This performance evaluation documented reports of delays of planned activities, problems with division of labor, difficulty working with MoE and RSEB staff, delays in the distribution of revised materials, challenges in securing fully dedicated staff to material development, technological interventions criticized as unsustainable, and uncertainty about the impact of READ TA outputs on student reading and writing outcomes. These challenges indicate that while trainings were provided, and good practices were implemented, questions arise on whether READ TA outputs will endure beyond the project funding period. As READ activities continue to be implemented, this performance evaluation takes stock of the fact that READ TA has put in motion important national innovations targeted at improving reading achievement in Ethiopia. Having noted this, the challenge moving forward is to turn these innovations into self￾sustaining improvement processes that lead to the deep implementation of READ TA achievements by the organizational bottlenecks identified in this performance evaluation. Unlike initial implementation which is more fraught with risk because it involves disturbing the status quo, fundamental changes in existing structures, deep and full implementation of READ TA will be achieved when half or more of the intended users of READ TA outputs implement them with fidelity. 1 Fixsen, D., Blase, K., Metz, A., Van Dyke, M., 2015. Implementation Science. In: James D. Wright (editor-in-chief), International Encyclopedia of the Social & Behavioral Sciences, 2nd edition, Vol 11. Oxford: Elsevier. pp. 695–702. 2 Fullan, M. (2014). Leading in a culture of change personal action guide and workbook. John Wiley & Sons. 46 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV At the heart of initial and full implementation are engaged leaders and persistent support (Marzano et al., 2005)3. As USAID and the MOE seek to create a self-sustaining improvement process, this performance evaluation shows that future READ technical assistance will to be accompanied by multi-tiered systems of support for reading achievement. These would be co-led by the Ministry of Education (MoE), the Regional State Education Bureaus (RSEBs), City Administration Education Bureaus (CAEBs), Colleges of Teacher Education (CTEs), and Woreda Education Offices (WEOs). One of the red flags in this performance evaluation is the absence of focal entities (persons or units) that can direct project implementers to corresponding different directorates, departments, and sections. This was found to hamper the smooth implementation of project activities. In implementation sciences, these individuals represent the engine of deep implementation and sustainability of the project. Their absence or high turnover undermined the potential of the implemented activities to be continued after the project ends. Therefore, it is critical to identify the skilled implementation teams that were trained and coached by READ TA, and ensure that the improvement experience they accumulated is further leveraged to reach the tipping point of more than half intended users (administrators, supervisors, teachers) implementing READ TA practices with integrity and fidelity. In conclusion, from a performance evaluation perspective, the evaluation data collected about READ TA indicate satisfaction with the quality of technical assistance delivered to improve reading and writing. The way forward, therefore, is to ensure that the developed capacity resulting from READ TA activities is embodied in leaders who wield the necessary influence, interest, and the know-how (Eden & Ackermann, 1998)4 necessary for accelerating the improvement of early grade reading and writing outcomes for all children in Ethiopia. 3 Marzano, R., Waters, T., McNulty, B., 2005. School Leadership that Works: From Research to Results. Association for Supervision and Curriculum Development (ASCD), Alexandria, VA. 4 Eden C and Ackermann F (1998). Strategy Making: The Journey of Strategic Planning. Sage: London. USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | A–1 ANNEX A. CLASSROOM OBSERVATION CHECKLIST INSTRUCTION FOR DATA COLLECTOR: The following checklist is what you’ll use while observing four classes: two classes from Grades1-4 and another two classes from Grade 5–8. Please refer to the data collection manual for further guidance on the procedures to follow. Write the grade you are observing: Grade ___________ Observer’s Name _____________________________________________________________ Complete as much of the table as you are able before the class begins. Teacher Gender M ______ F _______ Region Zone Woreda Name of school Date of observation Time of observation Start: End: # of students in the classroom (measures attendance) Boys ______ Girls ____ # of students with mother tongue textbooks in the class Boys ______ Girls ____ Teacher received training Yes ______ No ______ No. Description Yes No Remark 1. 1.1 Teacher brought the new mother tongue textbook to the classroom 1.2 Teacher brought the new mother tongue teacher’s guide to the classroom 1.3 Teacher uses new mother tongue textbook during instruction 1.4 The teacher refers/uses the teacher guide during instruction 2. Does the teacher engage males and females equally (engaged with the teacher- e.g. answering questions)? 3. Does the teacher use the “I do, we do, you do” methodology? 4. Does the teacher use any technology during teaching learning process (e.g. Model videos, assistive teaching aids, etc.) 5. The degree of students’ use of the textbooks during instruction 1. High 2. Moderate 3. Low 6. There is interactive teaching and learning processes 1. Not interactive 2. Least interactive 3. Interactive 4. Highly interactive 7. Does the teacher use assessment strategies during instruction? (Does the teacher ask students to do classroom activities and mark at the end of the class other than what has been mentioned in the “I do, we do, you do” methodology? etc.) 1. Not at all 2. Only Once 3. Twice 4. Three or more times Additional Comments: ___________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | B–1 ANNEX B. INTERVIEW GUIDE FOR PRINCIPALS AND SUPERVISORS MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. B–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you! DEMOGRAPHIC DATA Code of participant Age Gender Position Principal Supervisor Years of service Level of education Region Zone Woreda Name of school Date of Interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. USEFULNESS AND QUALITY OF TEACHING MATERIALS 1. Would you please tell us about the presence of the student textbooks and teacher’s guide in your school for Grade 1–8? Probe: the adequacy of the number of student textbooks and teacher’s guide. 2. Have you started using the new mother tongue language student textbooks and teacher guides for teaching mother tongue language for Grade 5–8? 2.1. When did you start teaching mother tongue language with the new materials? 3. Do you perceive Grade 5–8 materials as useful tools for instruction? 3.1. Do all mother tongue language teachers of Grade 5–8 currently use the materials for instruction? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | B–3 3.2. What proportion of Grade 1–4 and Grade 5–8 students have the new textbooks? 3.3. What are the challenges of using or implementing the new student textbooks and teacher’s guide for Grade 5–8? 4. As principal/cluster supervisor of this school, have you got the chance to go through the new Grade 5–8 student textbooks and/or teacher guides or at least discussed about these materials with mother tongue language teachers? 4.1. If yes, would you please briefly explain the strength and weakness of the new student textbooks and teacher guides? 5. Are you satisfied with the overall nature of the new Grade 5–8 student textbooks and teacher guides developed by READ TA? LANGUAGE SPECIFIC TEACHING METHODOLOGY 6. Would you please describe if the new materials have helped Grade 5–8 teachers in changing their method of teaching? 7. In what ways, if any, have the newly developed Grade 5–8 student textbooks and teachers guide enhanced reading and writing skills of students in your school? 8. Have you seen any difference in the progress of Grade 5–8 students in regards to learning reading in mother tongue language quicker/slower than students in the previous years? USE OF TECHNOLOGY 9. Can you please speak about any new uses of technology in your school? 9.1. Where did they come from? 9.2. When did you receive them? 9.3. Are they helpful in teaching and learning mother tongue? INCLUSIVE EDUCATION 10. Has your school received any support related to addressing the needs of students with special needs? Data Collector: Probe for the kinds of support made available to the school and which organization has made this support accessible. OVERALL IMPRESSIONS ABOUT THE MATERIALS DEVELOPED BY READ TA 11. What is your overall impression about the materials developed (student textbooks, teacher’s guide, and the CDs) for Grade 5–8? 12. If there is another chance of developing these materials, what aspects of the materials do you think should be improved? Why? 13. What effort has been made to sustain the best practices of the newly developed materials, methodologies and technology? 14. Is there anything that you want to add? B–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV CLOSING Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work • Lessons learned • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS! USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | C–1 ANNEX C. INTERVIEW GUIDE FOR CTE STAFF MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting Final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. C–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you DEMOGRAPHIC DATA Code of participant Age Gender Educational background Occupation Years of service Name of the CTE Your role in the project Dean Department Head Teacher Educator Date of interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. NATURE OF SUPPORT PROVIDED 1. Are you aware of the support provided by READ TA to CTEs to develop and adapt materials for pre-service training for mother tongue language teachers? 1.1. If you are aware of the support, would you please describe the types of support READ TA made available related to the development of modules and the use of technologies? 1.2. How many modules has READ TA developed so far? Why? 1.2.1.How do you evaluate the modules in terms of: 1.2.1.1.Appropriateness to the level of teacher trainees? 1.2.1.2.Level of difficulty? 1.2.1.3.Cultural appropriateness? 1.2.1.4.Language use and readability? 1.2.1.5.Gender balance? 1.2.1.6.Addressing disability issues? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | C–3 1.2.1.7.Continuity of lessons from one topic to the other? 1.2.1.8.Logical link among topics and sub-topics? 1.3. Would you please specifically describe the technological support made available by READ TA? 1.4. How many video materials have been uploaded? Why? 2. Describe the level of support you received from READ TA to develop and adapt pre-service training materials for mother tongue language teachers? 2.1. What kind of support do you think has not been addressed very well? USE OF TECHNOLOGY 3. What are the deployed ICT packages to support pre-service mother tongue instruction in the CTE? 3.1. What capacity building efforts has been taken to support the CTE on the ICT package? 3.2. Are you using the installed ICT packages? 3.3. Did you get any technical support? 3.4. What efforts has been done for maintenance and sustainability of ICT packages? CHALLENGES AND RECOMMENDATIONS 4. What were the challenges encountered while working with the READ TA? 4.1. How did you cope with? 5. What suggestions do you have for an overall improvement of the support that READ TA made available to CTEs? PROJECT TAKE OVER AND SUSTAINABILITY ISSUES 6. Does your college have any agreement signed with READ TA regarding project exit strategy? 6.1. Does your college have any sustainability plan? If yes, can you please explain the content and objective of the plan? 6.2. How does your college sustain completed activities? 6.3. Are there activities not completed by READ TA? If yes, what is the plan of your college to complete these activities? INCLUSIVE EDUCATION 7. Has your college received any support related to addressing issues related to students with special needs? Data Collector: Probe for the kinds of support made available to the college and which project has made this support accessible. CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Thank you very much for participating in this discussion today. C–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work in the interview • Lessons learned about their interview • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS! USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | D–1 ANNEX D. INTERVIEW GUIDE FOR EXPERTS FROM MOE AND RSEBS MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent non-profit research organization called American Institutes for Research (AIR). We are conducting final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. D–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you DEMOGRAPHIC DATA Code of participant Age Gender Current position Work experience in years Region (If applicable) Organization where you are working MoE RSEB Date of Interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. STATUS OF CURRICULUM DEVELOPMENT FOR GRADE 5–8 AND ENGLISH LANGUAGE 1. Mother Tongue curriculum for Grade 5–8 1.1. Would you please describe the process of curriculum review and development for Grade 5–8? (Data Collector: Please probe for roles, major achievements, and major challenges). 1.2. Do you think that the revised mother tongue curricula for Grade 5–8 have adequately addressed the problems with the previous curricula? Can you give us an example? 1.3. What has been done for adaptive regions on adaptation of mother tongue curricula materials for both Grade 1–4 and Grade 5–8? 2. New English Language curriculum 2.1. Would you please describe what has been done on the new English language curricula materials? (Data Collector: Please probe for roles, major achievements, and major challenges). 2.2. Is there anything that has not been done/achieved as per the implementation plan? Why? (Data Collector: Please probe for any solution that has been put in place). 2.3. Do you think that the newly developed English language curricula will adequately addressed the problems with the previous curricula? Can you give us an example? How? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | D–3 MATERIAL DEVELOPMENT, ADAPTATION, AND REVIEW 3. What is the status of the development and regional adaptation of student textbooks and teacher guides for Grade 5–8? 3.1. Has regional adaptation of student textbooks been completed in all mother tongue languages as planned for Grade 1–4? 3.2. How was the coordination between READ TA and the MoE and RSEBs in terms of the development of these materials? (Data Collector: Please probe for what has been done in terms of adaptation for Grade 5–8). 3.3. Was there any agreed upon mechanism to guide the coordination/collaboration activities? (Data Collector: Please probe for details and problems encountered in due course of co. coordination/collaboration). 4. What is the status of the review process of English language textbooks for Grade 1–8? 4.1. What do you know about the process? 4.2. What were the problems identified before the beginning of reviewing the textbooks? How were these problems identified? 4.3. Do you think that the review process is appropriate to address the problems related to the textbooks? (Data Collector: Please probe for what has been done to address the problem of alignment of textbooks with the MLCs). QUALITY OF MATERIALS DEVELOPED 5. How would you describe the level of alignment between the textbooks and teacher guides for Grade 5–8 with the MLCs? Please describe the alignment for each grade separately, as you are able. 6. How was the appropriateness of vocabulary, readability, complexity of structure, etc. of the materials determined during the development of mother tongue student textbooks and teacher guides for Grade 5–8? 6.1. Do the newly developed mother tongue student textbooks and teacher guides for Grade 5–8 address the gaps identified in the previous learning and teaching materials? TECHNOLOGY AND TEACHING AIDS 7. Would you please describe the status of multimedia teaching aids that have been put in place by READ TA to facilitate preservice training of mother tongue language teachers? 7.1. Have CTEs started using these technologies? Do you think, the technology put in place is appropriate for the purpose? 7.2. What are the strengths and weaknesses of the multimedia teaching aids developed by READ TA? 7.3. What are the technology types, costs, and implementation and sustainability requirements being proposed for use by CTEs and RSEBs? 7.4. Are you satisfied with what has been achieved with regard to the use of appropriate technology and teaching aids? D–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV REFRESHER TRAINING OF MASTER TRAINERS AND TEACHER TRAINERS 8. How was the selection of Master trainers conducted for the refresher training? What about the selection of teacher trainers? 8.1. Who did the selection? 8.2. Are there new criteria employed for selecting master and teacher trainers for refresher training other than what had been used before? If yes, why? 8.3. Are there new experts selected as master or teacher trainers? Why? How was the selection carried out? Was the training separate? What was the duration? 8.4. How was the refresher training organized? 8.5. How do you evaluate the organization and success of the refresher training? Has it achieved its purpose? GENDER AND DISABILITY ISSUES 9. Is there anything that READ TA has done in terms of addressing gender and disability issues other than addressing gender sensitivity and disability issues in the materials developed? Have the issues of gender sensitivity and disability reflected in the new Grade 5–8 student textbooks and teacher guides? CHALLENGES, PROJECT TAKE OVER, AND SUSTAINABILITY ISSUES 10. What are the major challenges that MoE encountered while working with READ TA in relation to material development, adaptation, and revision of English textbook? : 11. What has been done so far from the part of MoE and/or RSEBs for project takeover and sustainability of activities? (Data Collector: Please probe for the presence of project exit plan and activities accomplished, if any, to sustain the project). CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work • Lessons learned • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS! USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | E–1 ANNEX E. INTERVIEW GUIDE FOR MATERIAL DEVELOPERS AND REVIEWERS MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent non-profit research organization called American Institutes for Research (AIR). We are conducting final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. E–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you DEMOGRAPHIC DATA Code of participant Age Gender Educational background Occupation Years of service Name of the Organization/Institution where you work Position within organization Date of interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. KNOWLEDGE OF MLCS AND ALIGNMENT 1. Were you involved in the development of student textbooks and teacher guides for Grade 1–4? 2. Before the start of material development for Grade 5–8 and review English Language textbooks for Grade 1–8, did you receive any training and/or orientation about MLCs? 2.1. If yes, for how long? 2.2. By whom, where, and when? 2.3. If yes, please tell me about the training and/or orientation. In which ways was it helpful? 2.4. Did you have a copy of the MLCs when developing the mother tongue textbooks and teacher guides and reviewing Grade 1–8 English textbooks? 2.5. Did you refer to the MLCs while developing and reviewing the textbooks? 3. How would you describe the level of alignment between the produced and reviewed textbooks and teacher guides with the MLCs? Please explain for each grade that you are familiar with. 3.1. How do you rank the level of alignment on a five points scale: 5 – Very Strong, 4 – Somewhat Strong, 3 – Neutral, 2- Somewhat Weak, and 1- Very Weak? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | E–3 PROCESS OF MATERIAL DEVELOPMENT FOR GRADE 5–8 (ONLY FOR DEVELOPERS) 4. How was the collaboration between READ TA, MoE, and RSEBs in developing student textbooks and teacher guides for Grade 5–8? (Data Collector: Please probe for details and if there is no collaboration, as for reasons). QUALITY OF THE MATERIAL DEVELOPED (ONLY FOR DEVELOPERS) 5. How was the appropriateness of vocabulary, readability, complexity of structure, etc. of the materials determined during the development of Grade 5–8 mother tongue student textbooks and teacher guides? 6. Has there been sufficient representation of women thorough-out mother tongue material development process? 6.1. Are women sufficiently depicted in the illustrations and content of the books? If it is insufficient, do you have any idea what the barriers were? 7. Are the developed materials consider student with special need in the illustrations and content? PROCESS OF REVIEW OF ENGLISH LANGUAGE TEXTBOOKS FOR GRADE 1–8 (ONLY FOR REVIEWERS) 8. What approach did you follow to review the English textbooks? 8.1. Have you been given standards to follow while reviewing the English textbooks? Who developed these standards? What was your role in the process of developing these standards or criteria? 8.2. What were the weaknesses of the previous English textbooks? 8.3. Do you think the revision will fix the weaknesses that the textbooks have? How do you evaluate the revised English textbooks in terms of helping children develop English language skill? OPPORTUNITY FOR CAPACITY BUILDING 9. What do you think were the major opportunities for capacity building during the process of material development and review of English textbooks (capacity of RSEBs, FMOE, and CTEs Zones)? 10. As material developer or reviewer, what is the most important part of this process in terms of your own learning? CHALLENGES ENCOUNTERED AND RECOMMENDATIONS FOR FURTHER IMPROVEMENT 11. What do you think were the challenges during the material development process? 12. If you are given an opportunity to revise the materials developed which part of the materials do you think would be critically revised? Why? CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? E–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work • Lessons learned • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS! USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | F–1 ANNEX F. INTERVIEW GUIDE FOR TEACHERS MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent non-profit research organization called American Institutes for Research (AIR). We are conducting final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. F–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you! DEMOGRAPHIC DATA Code of participant Age Gender Years of service in teaching Years of service teaching mother-tongue Level of education Region Zone Woreda Name of school Grade currently teaching Date of Interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. AVAILABILITY OF THE NEWLY DEVELOPED MATERIALS 1. Are the newly developed student textbook and teacher guide for Grade 5–8 available to you at the school you teach? 1.1. If not, do you know why the textbooks are not available to you? 1.2. If yes, when did the textbooks and teacher guides become available? 2. If yes, what is the student to new text book ratio in your school/class room? 2.1. Is this ratio reasonably adequate? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | F–3 USEFULNESS AND QUALITY OF TEACHING MATERIALS 3. Are you currently using these materials for instruction? 3.1. Do you perceive the materials as useful tools for instruction? 3.2. Are the students using the textbooks? 3.3. Do students bring textbooks to the class whenever they have mother tongue language class? What proportion of students bring the textbooks when there is a mother tongue language class? 3.4. What are the challenges of using or implementing the new student textbooks and teacher’s guide for Grade 5–8? 4. Are the content, breadth, and depth of the student textbooks and teacher guides appropriate to the grade level you teach? (Data Collector: Please probe for cultural appropriateness of contents and the relevance of illustrations used in the textbooks). 4.1. What, in your view, is missing or hasn’t been given due attention in the course of developing the textbooks and the teacher guides for Grade 5–8? 4.2. Are the materials adequately inclusive and gender sensitive? 4.3. Would you please briefly explain the strength and weakness of the new student textbooks and teacher guides? 4.4. Are you satisfied with the overall nature of the new student textbooks and teacher guides? LANGUAGE SPECIFIC TEACHING METHODOLOGY 5. Are the methodologies and strategies for language specific teaching depicted in the textbooks and teacher guides helpful in teaching students to learn to read and write effectively? 5.1. Have the newly developed mother tongue textbooks and teacher’s guide helped you in changing your teaching methodology and strategies? If yes, how? (Data Collector: Please probe for I do, We do, You do approach). 5.2. How do you view the content of teacher guides and students’ textbooks in terms of language use and methods of presentation? 5.3. In what ways, if any, have the newly developed student textbooks and teachers guide enhanced reading and writing skills of students? 5.4. Have you seen any difference in the progress of your students in regards to learning reading in mother tongue language quicker/slower than previous? OVERALL IMPRESSIONS ABOUT THE MATERIALS DEVELOPED BY READ TA 6. What is your overall impression about the materials developed (student textbooks and teacher’s guide) for Grade 5–8? 7. If there is another chance of developing these materials, what aspects of the materials do you think should be improved? Why? 8. Is there anything that you want to add? F–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV CLOSING Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work • Lessons learned • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS! USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | G–1 ANNEX G. QUESTIONNAIRE FOR TEACHERS The purpose of this questionnaire is to collect data for the final performance evaluation of READ TA project funded by USAID and implemented by RTI. The READ TA project is providing technical assistance to support the development and implementation of mother tongue curricula for grades 1-8 in seven local languages. You are invited to participate in the evaluation of this project. Information collected from participants will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this questionnaire will be shared with USAID and the MoE, no individual will be identified in any report. Participation in completing this questionnaire is voluntary. If you have questions about the questionnaire, please contact (Daniel Tefera, the principal investigator of this evaluation) at 0911663402. You can also email the principal investigator at: dtefera@air-read￾me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. We are very much great full for your participation on this evaluation! PART I: DEMOGRAPHIC DATA Instructions: In this part of the questionnaire, you are asked to provide personal information. Read each question and circle the number of your choice or provide the information in writing on the space provided. Please note that any of your personal information will not be communicated to the third party in any way that your responses will be traced. 1. Region 1. Tigray 2. Amhara 3. Oromia 4. SNNPR 5. Somali 2. Zone _____________________________________________________________________ 3. Woreda___________________________________________________________________ 4. Name of the School __________________________________________________________ 5. Age of the respondent ________________________________________________________ 6. Gender of the respondent______________________________________________________ 7. Grade level currently teaching __________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ 8. Years of service as teacher _____________________________________________________ G–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV 9. Years of service in teaching mother tongue language ___________________________________ 10. Level of education ___________________________________________________________ PART II: STUDENT TEXTBOOK AND TEACHER’S GUIDE Instructions: In this part of the questionnaire, you are asked to provide your responses to items related to the use of student textbooks and teacher’s guide. Read each question and circle the number of your choice, put “X” under your choice or provide the information in writing on the space provided. 1. What is the student-textbook ratio (E.g. 1:5, 1 text book for 5 students)? Grade 1__________ Grade 2___________ Grade 3___________ Grade 4___________ Grade 5__________ Grade 6___________ Grade 7___________ Grade 8___________ 2. What is the reason why every student does not have his/her own textbook, as you understand it? _________________________________________________________________________ 3. How often do students bring textbooks to the class? Grade 1 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 2 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 3 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 4 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 5 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 6 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 7 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all Grade 8 a. Always (whenever they have class) b. Sometimes c. Rarely d. Not at all 4. How often do you use teacher’s guide? a. Always b. Sometimes c. Rarely d. Not at all NO. QUESTION GRADE 1 GRADE 2 GRADE 3 GRADE 4 GRADE 5 GRADE 6 GRADE 7 GRADE 8 YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO 1. For the grade (s) you teach, do your students have the new mother tongue textbooks? 2. For these grades, does every student have his/her own textbook? 3. Are students using the new textbooks? 4. Are the newly developed teacher guides available to you at the school you teach 5. Are there any electronic teaching aids in your school for the grade you teach? 6. If yes, are they helpful in improving your method of teaching? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | G–3 5. If your school has the textbooks, when did the textbooks become available? Month ______________Year__________________ 6. If your school has the teacher’s guide, when did the guide become available? Month_______________ Year__________________ 7. Are there electronic teaching aids in your school? a. Yes b. No 7.1. If yes, from where does the school obtain these teaching aids? _____________________________________________________________________ _____________________________________________________________________ PART III: VIEWS ABOUT THE MATERIALS DEVELOPED FOR GRADE 1–8 Instructions: In this part of the questionnaire, there are statements about how the contents of student textbooks are organized, the use of language specific methodologies and the technical qualities of student textbooks and teacher’s guide. We expect you to rate these materials of a scale of satisfaction: 1 – Very dissatisfied • 2 – dissatisfied • 3 – unsure • 4 – satisfied • 5 – Very satisfied. For each statement given below, check the box to the right that best fits your ratings towards the aspect of the materials indicated in the statement. Put “X” under your choice. For the next series of questions, please write the grade you teach with the new mother tongue textbook and answer the questions ONLY for that grade textbook. About which grade are you responding? Grade NO. ITEMS 1 2 3 4 5 Textbook content: 1. Continuity of lessons from one topic to the other 2. If you teach multiple grades, continuity of lessons from one grade level to the next grade 3. Grade level appropriateness 4. Level of difficulty 5. Readability 6. Cultural appropriateness for your local context 7. Relevance in terms of 7.1 Representing both boys and girls 7.2 Inclusiveness for students with special need 7.3 Promoting life skills, such as healthy habits like washing hands after toilet, telling the truth, or good manner 8. Comprehensiveness How do you view the content of the teacher’s guide and students’ textbooks in terms of the use of language–specific teaching and learning methodologies and strategies? 9. The alignment between the student textbooks and teacher’s guide 10. The arrangement of topics from simple to complex 11. Exclusive use of mother tongue language in the textbook G–4 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV NO. ITEMS 1 2 3 4 5 12. Good use of illustrations 13. Providing opportunities for continuous performance assessment and assessing student progress 14. The teaching strategy in the teacher’s guide are designed to be interactive 15. The capacity to trigger interest in the learners 16. Pedagogical appropriateness of the sequence of objectives in the teacher guides 17. Strategies promoting active learning How do you view the physical qualities of student textbooks and teacher guides in terms of: 18. Quality of cover pages 19. Quality of the paper and bindings 20. Quality of print (including text and illustrations) USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | H–1 ANNEX H. INTERVIEW GUIDE FOR EXPERTS FROM READ TA MATERIALS INCLUDE Voice recorder CHECKLIST OF PROCEDURES • Obtain verbal consent from participant. • Debrief following each interview • Conduct interview • Please type up notes in tables! Interviewer’s name ______________________________________________________________ Interview’s Code _______________________________________________________________ Region_______________________________________________________________________ Language Group ________________________________________________________________ Cell phone of the interviewer ______________________________________________________ BACKGROUND AND INFORMED CONSENT My name is (name of the interviewer) and I’ll be talking with you today. Also, here with us today is (name of the note taker), who will be taking notes. We work for an independent non-profit research organization called American Institutes for Research (AIR). We are conducting final performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@air-read-me.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45 minutes. We have a lot of questions to ask, but we want to respect your time. So, I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. H–2 | READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT USAID.GOV Do you have any questions? Do you agree to participate today? • Option 1: If the participant does not grant permission, end here. • Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you DEMOGRAPHIC DATA Code of participant Age Gender Current position Work experience in years Region (If applicable) Date of Interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. ACHIEVEMENTS DURING THE PROJECT PERIOD 1. Would you please describe your achievements during the project period in detail in relation to: 1.1. Provision of refresher training for master and teacher trainers? (Data Collector: Please probe for the selection process, criteria for selection if any, the duration of the selection, and the way it was organized). 1.2. Material development in general for Grade 5–8 in particular? 1.3. Adaptation of mother tongue textbooks and teacher guides for adapting regions? 1.4. Technologies and multimedia materials for CTE? 1.5. Development of modules, video materials, and other technological products for CTEs? 1.6. Reviewing English language textbooks for Grade 1–8? (Data Collector: Please probe for the processes followed to reach at the level of textbook revision). 1.7. Capacity building? 2. Is there anything that has not been done/achieved as per the implementation plan? Why? 3. Are there activities you have carried out not indicated in the project document or implementation plan? Why? USAID.GOV READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE PROJECT | H–3 CHALLENGES ENCOUNTERED 4. What are the major challenges that READ TA encountered during project implementation in relation to: 4.1. Collaborating with MoE, RSEBs, CTEs, partner organizations, and other government agencies? 4.2. Material development and adaptation? 4.3. Use of technologies and multimedia productions? 4.4. Capacity building? GENDER AND DISABILITY ISSUES 5. Is there anything that READ TA has done in terms of addressing gender and disability issues other than addressing gender sensitivity and disability issues in the materials developed? Have the issues of gender sensitivity and disability reflected in the new Grade 5–8 student textbooks and teacher guides? PROJECT TAKE OVER AND SUSTAINABILITY ISSUES 6. What has been done for project takeover and sustainability of activities? 6.1. Is there any sustainability plan? If yes, can you please explain the content and objective of the plan? 6.2. What has been done to sustain completed activities? 6.3. What has been done on preparation for closeout of projects? LESSONS LEARNED AND IMPLICATIONS FOR FUTURE 7. What are the major positive experiences that READ TA has come across while implementing its project? 8. What has READ TA learned from its project implementation? 9. What are the implications of such lessons for future planning and implementation of similar projects? CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to: • What worked and did not work • Lessons learned • Issues to address in future interviews • Issues to follow up on with the research team or client • Key points to note in data analysis PLEASE TYPE UP NOTES AS SOON AS POSSIBLE AFTER INTERVIEWS!