[May 2016] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by READ M&E under the guidance of the American Institute for Research (AIR). Mid-Term Evaluation Report: Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA) MID-TERM EVALUATION REPORT: READING FOR ETHIOPIA’S ACHIEVEMENT DEVELOPED TECHNICAL ASSISTANCE (READ TA) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Table of Contents ........................................................................................................................................ i List of Figures.............................................................................................................................................. ii List of Tables .............................................................................................................................................. iii Acronyms .................................................................................................................................................. iii Executive Summary ................................................................................................................................... iv Evaluation Purpose & Evaluation Questions ........................................................................................... 1 Evaluation Purpose .................................................................................................................................... 1 Evaluation Questions ................................................................................................................................. 1 Project Background .................................................................................................................................... 2 READ Projects ...................................................................................................................................... 2 READ Technical Assistance (READ TA) ............................................................................................ 2 Methodology ................................................................................................................................................ 4 Design ................................................................................................................................................... 4 Data Sources ......................................................................................................................................... 4 Data Collection Instruments.................................................................................................................. 8 Piloting of Instruments.......................................................................................................................... 9 Data Collection Procedures ................................................................................................................... 9 Data Analysis ........................................................................................................................................ 9 Analysis of Qualitative Data ................................................................................................................. 9 Analysis of Quantitative Data ............................................................................................................. 10 Analysis of Data Obtained from Document Review........................................................................... 10 Ethics Review .......................................................................................................................................... 11 Findings...................................................................................................................................................... 12 Intermediate Result 1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre￾Service and In-Service Teacher Training Developed .............................................................................. 12 1.1 Status of Development for the New Mother Tongue Curricula .............................................. 12 1.2 Coordination between READ TA, MOE, and RSEBs ............................................................ 13 1.3 Alignment of the Mother Tongue Syllabus, Student’s Textbooks, and Teacher’s Guide with the MLCs................................................................................................................................. 14 1.4 Student to New Textbook Ratio .............................................................................................. 15 1.5 Student Textbook Use ............................................................................................................. 18 1.6 READ TA Support to Colleges of Teacher Education............................................................ 22 Intermediate Result 2: Application of Language-Specific Teaching, Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write ...................................................... 24 2.1 Respondents’ Views of New Materials ................................................................................... 24 2.2 Content of the New Mother Tongue Language Textbooks ..................................................... 25 ii 2.3 Physical Qualities of Teacher’s Guide and Students’ Textbooks............................................ 29 2.4 Master Trainer Selection ......................................................................................................... 33 2.5 Training of Master Trainers .................................................................................................... 33 Intermediate Result 3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aid ............................................................................................................................................. 33 3. 1 Technology Use & Multimedia Teaching Aids....................................................................... 33 Intermediate Result 4: Technical Support to RSEBs and MOE for READ Institutional Improvement .. 33 4.1 READ TA’s support of the READ Institutional Improvement program................................. 33 4.2 READ TA Influence on Curriculum Development Capacity .................................................. 34 4.3 Gender Sensitivity & Disability Inclusion in READ TA ........................................................ 35 Conclusions ................................................................................................................................................ 37 Recommendations ..................................................................................................................................... 38 5.1 Recommendations for USAID, MoE, and RSEB ....................................................................... 38 5.2 Recommendations for READ TA ............................................................................................... 39 Annexes ...................................................................................................................................................... 42 Annex I: Findings Conclusions and Recommendations .......................................................................... 42 Annex 2: Planned sample size by region and language ........................................................................... 46 Annex 3: Data Collection Instruments..................................................................................................... 48 List of Figures Figure 1: Teachers' report on how often students bring textbooks to class ................................................ 19 Figure 2: Percentage of Teachers who brought to class the new Mother Tongue Textbook ...................... 21 Figure 3: Frequency of use of teachers guide by language ......................................................................... 22 Figure 4: Teachers who brought the new Mother Tongue Teachers' Guide to class .................................. 22 Figure 5: Level of textbook use in the classroom by students .................................................................... 23 Figure 6: Overall satisfaction of teachers on the link between contents and embedded methodology by language ...................................................................................................................................................... 26 Figure 7: Overall satisfaction of teachers on the content of the new materials by language ……………..28 Figure 8: Overall satisfaction of teachers on the physical qualities of the materials .................................. 31 iii List of Tables Table 1: READ TA material development approach ................................................................................... 3 Table 2: Sample size distribution …………………………………………………………………………. 7 Table 3: Teacher's report of the availability of textbooks ………………………………………………...16 Table 4: Average textbook to student ratio by language and grade level ……………………………...…16 Table 5: Availability of Teacher's Guide …………………………………………………………...…….17 Table 6: Distribution teachers' responses to items on the link between the contents of the new textbooks/guides and learning methodologies and strategies employed ……………………………...… 26 Table 7: Mean and standard deviation of satisfaction scores on the link between the contents of the new textbooks/guides and learning methodologies and strategies employed …………………………………27 Table 8: Mean and standard deviation of satisfaction scores for content by language …………………. 28 Table 9: Response of teachers to contents of textbooks ……………………………………………… .29 Table 10: Mean and standard deviation of satisfaction scores for physical qualities of new materials ...... 32 Table 11: Mean and SD of satisfaction scores for physical qualities of new materials by gender………. 31 Table 12: Distribution of teachers' responses to items measuring quality .................................................. 32 ACRONYMS AIR American Institutes for Research CSA Charities Societies Agency CTEs Colleges of Teacher Education FMOE Federal Ministry of Education FO Field Office FSU Florida State University HO Head Office IRB Institutional Review Board IRR inter-rater reliability IRs Intermediate Results MLCs Minimum Learning Competencies MOE Ministry of Education MoU Memorandum of Understanding MT Mother Tongue MT TF Mother Tongue Task force MTCD Mother Tongue Curriculum Development NEAEA National Educational Assessments and Examination Agency NEAEA City Administration Education Bureaus OHRP Office of Human Research Protection READ Reading for Ethiopia’s Achievement Developed READ CO READ Community Outreach READ II READ Institutional Improvement READ M&E READ Monitoring and Evaluation READ TA READ Technical Assistance RSEBs Regional State Education Bureaus RTI Research Triangle Institute SIL LEAD Summer Institute of Linguistics – Language Education and Development SPSS Statistical Package for Social Scientists USAID United States Agency for International Development WEO Woreda Education Offices iv EXECUTIVE SUMMARY Under the umbrella of the Reading for Ethiopia’s Achievement Developed (READ) projects, launched by USAID in collaboration with the Ethiopian Ministry of Education (MoE), the READ Monitoring and Evaluation (READ M&E) project provides external monitoring and evaluation services for the READ suite of projects. Evaluations undertaken by READ M&E provide external accountability, determine level of performances and accomplishments, identify challenges on implementation, and provide stakeholders with opportunity for learning and decision making for the respective READ projects under evaluation. This performance evaluation examines the major midterm project outcomes of Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The READ TA project is a five-year (January 2012 - December 2017) countrywide intervention, which works to improve reading and writing in mother tongue language in collaboration with the Ministry of Education (MoE), the Regional State Education Bureaus (RSEB), City Administration Education Bureaus (CAEB), Colleges of Teacher Education (CTE), and Woreda Education Offices (WEO). The project supports the development and implementation of Mother Tongue (MT) Curricula for Grades 1-8 in seven targeted Mother Tongue languages (Amharic, Afan Oromo, Af-Somali, Hadiyissa, Sidama Afoo, Tigrinya and Wolaytatto) and English as a Second Language. READ TA’s design assumes that the revision of mother tongue curriculum and development of student books and teachers’ guides, and supporting CTEs with teaching modules and technological assistance will improve reading skills. READ TA’s in-service program focuses on professional development of teachers through the training of teacher trainers while its pre-service program addresses the gap in the training of mother tongue language teachers through the provision of technical and technological support to CTEs. USAID provides funding for the project and Research Triangle International (RTI) implements it in partnership with the following organizations: Save the Children, Florida State University (FSU), SIL LEAD, Inveneo, and Whiz Kids Workshop. READ TA’s main activities are concentrated around four intermediate results (IR): IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre￾Service and In-Service Teacher Training Developed IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are applied IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement Process The mid-term performance evaluation of the READ TA program employed a mixed methods approach with both qualitative and quantitative data. Data were collected from the following respondents: MoE, RSEB, and zone experts (N=40); CTE (N=18) READ TA staff (N=7); cluster supervisors (N=33); Principals (46); and Material developers (N=22). READ M&E interviewed v 123 teachers, administered questionnaires to 245 teachers and conducted 192 classroom observations. The sampling strategy was generally purposive, and therefore READ M&E does not claim that the samples are statistically representative of the populations as a whole, i.e., all teachers, experts, material developers, CTE staff, principals and supervisors in the seven language groups. However, the selection of zones except for the SNNP, where the zones were predetermined by the project, is random. After excluding the inaccessible zones from Tigray, Amhara, Oromia, and Somali Regional States, a table of random numbers was used to select two zones from each of these regional states. Accordingly, READ M&E selected eight zones from Tigray, Amhara, Oromia, and Somali Regional States and three zones from SNNP (eleven zones) to conduct this evaluative study. Seven data collection instruments and interview guides were prepared and used to collect data from respondents from seven language groups. The instruments were developed collaboratively with USAID representatives and READ TA. READ M&E hired seven language experts (one from each language) to translate the instruments initially prepared in English into the seven local languages. Interview guides for principals and cluster supervisors (a supervisor assigned for a group of 5 to 7 schools) and interview guide for experts from MoE, RSEBs, Zones and READ TA were translated into four local languages viz. Amharic, Afan Oromo, Tigrigna, and Af-Somali. Since the working language for SNNP is Amharic, it is possible to communicate easily with school directors, cluster supervisors and experts from this region in Amharic. Hence, the Amharic versions of the interview guides for principals and cluster supervisors and experts from MoE, RSEBs, Zones and READ TA were used. Furthermore, the interview guides for material developers and CTE staff were not translated into local languages for the reason that they can read and write in English. In addition, the classroom observation checklist was not translated into local languages since data collectors used it. READ M&E piloted the instruments in Amharic and Afan Oromo to ensure appropriateness of the tools. READ M&E recruited eighteen experienced qualitative and quantitative data collectors; two per language except for Afan Oromo and Amharic each had four. After a one-day intensive training, nine teams of data collectors were deployed to eleven study sites (East and South East Tigray, South Gondar, South Wollo, West Arsi, East Shoa, Fafan and Siti of Somali, Sidama, Wolayta and Hadiya) from December 20, 2015 to January 8, 2016. As a strategy of quality assurance, three READ M&E teams supervised the data collection process. Nine experienced consultants transcribed 282 interviews and then the READ M&E team coded the qualitative data. READ M&E entered 245 questionnaires and 192 observation checklists into SPSS for quantitative analysis. Finding and Conclusions The major findings of the evaluative study for each IR follow: IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In-Service Teacher Training Developed READ TA has completed textbook and teacher guide development of 112 separate materials (as of December 2015) with the addition of region specific adapted version of 192 materials (a total of 304 separate materials as of June 2016), covering seven mother tongue languages in grades 1-8. vi READ TA followed rigorous procedures for developing the new mother tongue instructional materials. There is consensus that the coordination and collaboration among stakeholders is exemplary. In an effort to ensure effective pre-service training in teaching primary Mother Tongue reading and writing, READ TA has supported CTEs in the development of the new CTE MT curriculum and modules as well as in training CTE MT instructors on the new MT Curriculum and modules. Each of the modules addresses a specific content area, and supports a specific course within the new CTE MT Curriculum. The technological support provided by READ TA for CTEs has transformed the pre-service training of mother tongue language teachers. The multimedia teaching aids and technologies include model videos for teachers, audio companions, and software in support of MT reading and writing instruction for grades 1-8 in all seven languages. The teaching aid and technologies are intended for use at the Colleges of Teacher Education. There are concerns regarding the alignment of the syllabus and MT instructional materials with the national MLCs and the approach followed to check for alignment. IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are applied Teachers are highly satisfied with the content, embedded methodology, and the physical qualities of the new mother tongue instructional materials. However, there is a consensus among teachers that the contents of a lesson for a day cannot be covered within the time allotted. Hence, many teachers tend to rush to finish the daily lesson. IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids READ TA provided technology and technological instruction to the CTEs to improve MT instruction. IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement Many of the respondents praised the work done by READ TA on curriculum review and material development capacity building of the MOE, RSEB, Zones and CTEs related to material development. Key recommendations for further action The following key recommendations are taken from the longer list in section 5.1&5.2: Recommendation #1: READ M&E recommends that the procedures followed for developing the new mother tongue instructional materials be taken as a model for developing other instructional materials at the national level. Hence, USAID and MoE should use such approach in similar future projects. Recommendation #3: There are concerns related to the alignment of the new mother tongue instructional materials with MLCs. Hence, a comprehensive and objective alignment and content analysis study should be conducted. vii Recommendation #5: READ M&E recommends that the contents of the lessons should match the time allotted. Hence, it would be more appropriate if measures are taken to review the materials and actual classroom practice so that the time allotted is suited for the content of the daily lesson. Recommendation #10: Supporting classroom instruction with multimedia teaching aids would improve the quality of instruction and retain the attention of the students. It is recommended that the production of such teaching aids be preceded by a thorough examination of the classroom settings. 1 EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE Evaluations undertaken by READ M&E are intended to provide external accountability, determine level of performances and accomplishments, identify challenges on implementation, and provide stakeholders with opportunity for learning and decision making for the respective READ projects under evaluation. This performance evaluation examines the major midterm project outcomes. This report utilizes evaluation instruments developed collaboratively with USAID representatives and READ TA. EVALUATION QUESTIONS The following questions under the four major intermediate results (IR) guided READ M&E’s performance evaluation of READ TA. The development of the questions was done in consultation with USAID and READ TA. IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In￾Service Teacher Training Developed 1.1 What is the status of the development of the new mother tongue curricula? 1.2 How was the coordination between READ TA and the MOE and RSEBs for material development? 1.3 What is the level of alignment of the mother tongue syllabus, student’s textbooks, and teacher’s guide with the MLCs? 1.4 What is the student to new textbook ratio? (Does not reflect on READ TA performance but valuable data to gather at this time) 1.5 Are the students using the textbooks? (Does not reflect on READ TA performance but valuable data to gather at this time) 1.6 What is the level of support provided by the READ TA to CTEs to train MT teachers and develop materials for pre-service training at CTEs? IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are applied 2.1 How do teachers view student textbooks and teachers’ guide prepared by READ TA? 2.2 How do teachers view the content of the teachers’ guide and students’ textbooks in terms of the use of language - specific teaching and learning methodologies and strategies? a) What / how was the procedure used to select master trainers? b) How were the Master Trainers (MT) trained? IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids 3.1 What has been done on use of technology and teaching aids? IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement 4.1 How was READ TA’s support of the READ II program? 2 4.2 How has READ TA affected the curriculum development capacity of the MOE, RSEBs, and CTEs? 4.3 How has READ TA included gender sensitivity and disability inclusion in their program? PROJECT BACKGROUND READ Projects The Reading for Ethiopia’s Achievement Developed Monitoring and Evaluation (READ M&E) is a five-year United States Agency for International Development (USAID)-funded project facilitated by the American Institutes for Research (AIR), tasked with monitoring and evaluating early grade reading in seven mother tongues as well as the M&E needs of the overall USAID READ intervention. The READ M&E project works in partnership with the Ethiopian Federal Ministry of Education (MoE), the National Educational Assessments and Examination Agency (NEAEA), Regional State Education Bureaus (RSEBs,) and City Administration Education Bureaus (CAEBs). Under the umbrella of USAID READ suite of interventions, the READ Technical Assistance (READ TA) program focuses on curriculum revision, development of textbooks, teacher’s guides, teacher training manuals, and supplementary reading materials, training of teacher trainers, and provision of technical assistance. All activities are designed to improve early grade reading and writing. Close collaboration with the Ministry of Education (MOE), Regional State Education Bureaus (RSEBs), Colleges of Teacher Education (CTEs), and other key development partners is inherent in the design. READ Institutional Improvement (READ II) consisted of direct grants to the MOE and RSEBs to conduct in-service teacher training and build the capacity of the education system to improve early grade reading and writing. READ II finished in 2015. READ Community Outreach (READ CO) develops the capacity of parents and communities to engage in promoting early grade reading and writing. READ CO supports schools and community libraries and establishes reading centers. The projects complement each other and are expected to improve reading and writing in Ethiopian primary schools. Evaluations undertaken by READ M&E provide external accountability for the respective READ projects evaluated. This performance evaluation examines the major midterm project outcomes of READ TA that covers all performances between the beginning of the project and end of 2015. READ Technical Assistance (READ TA) The READ TA project is a five-year (January 2012 - December 2017) countrywide intervention, which works to improve reading and writing in mother tongue language in collaboration with the Ministry of Education (MoE), the Regional State Education Bureaus (RSEB), City Administration Education Bureaus (CAEB), Colleges of Teacher Education (CTE), and Woreda Education Offices (WEO). The project supports the development and implementation of Mother Tongue (MT) Curricula for Grades 1-8 in seven targeted Mother Tongue languages (Amharic, Afan Oromo, Af-Somali, Hadiyissa, Sidamu Afoo, Tigrigna and Wolayttatto) and 3 English as a Second Language. READ TA’s design assumes that the revision of mother tongue curriculum and development of student books, teacher guides, and supplementary reading materials will lead to improved reading scores. READ TA provides in-service and pre-service continuous professional development of teachers, school principals, supervisors, and teacher educators through the training of teacher trainers. USAID provides funding for the project and Research Triangle International (RTI) implements it in partnership with the following organizations: Save the Children, Florida State University (FSU), SIL LEAD, Inveneo, and Whiz Kids Workshop. READ TA’s main activities are concentrated around four intermediate results (IR): IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In￾Service Teacher Training Developed The major activities to date for READ TA have been under this IR. The project has completed textbook and teacher guide development of 112 separate materials (as of December 2015) with the addition of region specific adapted version of 192 materials (a total of 304 separate materials as of June 2016), covering seven mother tongue languages in grades 1-8. Printing and distribution of the materials is not the responsibility of READ TA, it is the responsibility of the Government of Ethiopia. When the government was unsuccessful in distributing an adequate number of grade 1-4 textbooks and teacher guides, READ TA distributed additional materials. Material development focused on addressing key problems observed from an initial needs assessment and review of existing materials. Issues with the original textbooks and how those problems were address in the development of new materials is summarized in the table below: Table 1: READ TA Material Development Approach Problem Activity Previous text was not active (teacher centered). Current text uses active/student methodology “I do, we do, you do.” Textbooks are not sensitive to cultural differences within Ethiopia. Includes regional/cultural appropriate issues (such as traditionally harmful practices). Illustrations are interesting, culturally appropriate, and gender and disability inclusive. Lack of practical guidance to teachers Teachers’ guides have specific instructions for teachers such as time guidelines, simple to complex concepts organized in order, assessments for teachers to use (only in hard copy) A key approach to this activity involved working closely with the MOE, RSEBs, and local language and pedagogical experts in the development of materials as well. IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are applied 4 Activities around teaching methodology were integrated into the teacher guides in IR1, and the READ II support in IR4. IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids READ TA developed multimedia teaching aids and technologies for reading instruction in line with the revised curriculum. Designed for use by the Colleges of Teacher Education (CTE), the multimedia teaching aids and technologies include model videos for teachers, audio companions, and software in support of MT reading and writing instruction for grades 1-8 in all seven languages. IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement READ TA subcontracted Save the Children to provided technical support in planning and monitoring the training of the first cohort of teacher trainees. Additionally, READ TA worked with MoE, RSEBs and the CTEs on teacher pre-service training. This work included adaptation and validation of the national pre-service MT teacher education curriculum into seven MTs; Development, adaptation to seven MTs, and validation of three CTE course modules (MT 201, TMT 222, and TMT 224) and Development of training manuals and revision of CTE Module 1 (MT 201). EVALUATION METHODS METHODOLOGY Design This mid-term performance evaluation of the READ TA program employs a mixed methods approach with both qualitative and quantitative data. Qualitative data provides depth and richness through interviews and observation. This allows the evaluation team to explore fully the experiences of those involved in READ TA materials development process. It also allows teachers’ use of the new materials and teacher trainings to be heard. Quantitative data was collected to allow the evaluation team to gather a wider representative population to add to information from teachers about their perception of the physical quality and the contents of the new mother tongue textbooks and teacher’s guide. Data Sources MOE and RSEB officials and experts, zone experts, and READ TA staff are the data sources for research question #1.1, #1.2, #3.1, #4.1, #4.2 and #4.3.  Cluster supervisors, school principals, and teachers are the data sources for evaluation question #1.3 and #2.1.  Teachers are the data source for evaluation question #1.4, #1.5, and #2.2.  CTE staff are the data source for evaluation question #1.6, #3.1, and #4.2.  Materials developers are data source for evaluation question #1.3.  Documents mainly from READ TA are data sources for answering research question #1.1, #1.2, #3.1, #4.1, #4.2 and #4.3.  Classroom observations are data sources for evaluation question #1.4 and #1.5. 5 Sampling Procedure The sampling strategy used in this evaluation is generally purposive, and therefore cannot be considered statistically representative of the populations as a whole, i.e., all teachers, experts, material developers, CTE staff, principals and supervisors in the seven language groups. However, the selection of zones except for the SNNP, where the zones were predetermined by the project, is random. After excluding the inaccessible zones from Tigray, Amhara, Oromia, and Somali Regional States, a table of random numbers method was used to select two zones from each of these regional states. Accordingly, READ M&E selected eight zones from Tigray, Amhara, Oromia, and Somali Regional States and three zones from SNNP (eleven zones) for conducting this evaluative study. Please see table 2 for sample size and distribution. Selection of Woredas The data collector in the field carried out the random selection of woredas. Zone education departments provided the list of woredas. The data collectors, together with zone experts, selected the woredas randomly using the lottery method. READ M&E initially planned to select two woredas from each language group except for Afan Oromo and Amharic groups. We planned to take four woredas in Afan Oromo and Amharic. This method accounts for the population size of the regions. Afan Oromo and Amharic have a slightly higher number of woredas. Twenty-five woredas were selected. The details are presented in Table 5. One of the possible explanations for the inclusion of additional woredas is the introduction of new administrative structure in some of the woredas. Another reason may be the absence of teachers in some of the classrooms planned to be observed. A teacher teaching the specific grade needed might not be available for classroom observation in the sampled woreda. In such situations, data collectors, in consultation with experts from the Addis office, selected other nearby woredas to have access to appropriate grade levels. Selection of Schools The selection of schools also followed a similar pattern. It is assumed that all schools received the new mother tongue student textbooks and teacher’s guides, therefore all schools were eligible to be selected for the sample. Data collectors secured the list of schools from the selected woreda education office and together with experts from woreda; education office selected one urban and one rural school randomly using a lottery method. In a situation where classrooms were unacceptable due to the absence of teachers teaching that grade level, data collectors were advised to select another school in the same woreda. READ M&E considered the population size of the regions and assigned more schools in Amhara and Oromia region (8 each) and less schools in the other regions (6 each). We initially planned to select forty-six schools. However, for the reasons mentioned above, the number of schools actually selected exceeded the planned number of schools. Sixty-six schools constituted the final sample size. For the details, see the annex. Selection of Respondents Sampling of Officials and Experts: Sampling of officials from the MoE and RSEB, MTCD team, READ TA, CTEs, and principals and cluster supervisors was entirely purposive. Officials of the directorates at MoE who are working with READ TA, RSEB experts and officials who were involved in the program, READ TA language experts from regions, and READ TA head office staff (Regional Director, DCOP for Technical and Curriculum Expert) who took part in the work carried out by READ TA were purposively selected. Material developers from each 6 language were selected based on availability. Similarly, eighteen CTE staff who had been involved in one way or another were selected for the sample. The deviation from the original plan in the number of CTE staff was done because the questions in the interview guide required the responses of staffs involved in different activities related to the work of READ TA at CTEs. It was observed during the interview, for example, that a person involved in coordination might answer some of the questions best rather than a person involved in the development of modules. Sampling of Teachers: Teachers selected for the sample were mother tongue language teachers who were teaching mother tongue language with the new mother tongue textbooks and teacher’s guide. READ M&E, as agreed during the data collection training, selected three teachers for the interview from each sample school based on availability. Every teacher in the selected school had the chance to participate in the study through interviews, questionnaire, or being observed. Data collectors automatically selected those who were in the classroom at the time of visit or had class during the visit selected for the classroom observation. Hence, data collectors selected one to four teachers purposively for classroom observations. As long as these teachers taught using the new mother tongue textbooks and teacher’s guide, they were eligible for inclusion. In the absence of adequate numbers of teachers for classroom observations, teachers interviewed were also observed while teaching. Data collectors followed similar procedures for selecting teachers for completing questionnaire. Those who were interviewed and observed were not selected for the questionnaire. Hence, the team observed 192 teachers while teaching and 245 teachers completed the questionnaire. For the details on the sample sizes by region and schools, refer to the annex. 7 Table 2: Sample Size and Distribution Region Zone Woreda School Sample Size Language CTE of School Material Developers Experts from RSEBs & Zones Principals Observation Teachers Cluster Interview Questionnaire Supervisors Amhara 2 5 13 5 6 7 10 40 30 51 6 Amharic Oromia 2 4 10 3 4 5 8 32 24 44 6 Afan Oromo Tigray 2 3 8 2 2 6 4 24 6 26 3 Tigrigna SNNP 1 3 10 2 2 7 5 24 18 39 4 Sidamu Afoo 1 4 9 2 2 3 6 24 18 37 5 Wolayttatto 1 3 9 2 2 2 7 24 17 30 6 Hadiyissa Somali 2 3 7 2 4 10 6 24 10 18 3 Af-Somali Total 11 25 66 18 22 40 46 192 123 245 33 8 Data Collection Instruments A total of seven data collection instruments were prepared and used to collect data from respondents from seven language groups. READ M&E hired seven language experts (one from each language) to translate the instruments initially prepared in English into local languages. Accordingly, questionnaire for teachers and interview guide for teachers were translated into Amharic, Afan Oromo, Tigrigna, Af-Somali, Sidamu Afoo, Wolayttatto, and Hadiyissa. Interview guides for principals and cluster supervisors (a supervisor assigned for a group of 5 to 7 schools) and interview guide for experts from MoE, RSEBs, Zones and READ TA were translated into four local languages viz. Amharic, Afan Oromo, Tigrigna, and Af-Somali. Since the working language for SNNP is Amharic, it is possible to communicate easily with school directors, cluster supervisors and experts from this region in Amharic. Hence, the Amharic versions of the interview guides for principals and cluster supervisors and experts from MoE, RSEBs, Zones and READ TA were used. Furthermore, the interview guides for material developers and CTE staff were not translated into local languages for the reason that they can read and write in English. In addition, the classroom observation checklist was not translated into local languages since data collectors used it. Interview Guides: Five separate interview protocols were prepared for teachers, principals and cluster supervisors, experts from the MoE, RSEBs, Zones, and READ TA, material developers, and CTE staff. The interview guide was semi-structured in design to allow for a focused discussion that still has the flexibility to capture a range of experiences and opinions. Classroom Observation Checklist: Classroom observations showed what is actually happening in classrooms in the use of new student textbooks, teacher’s guide and observe student to new textbook ratio and use of the new method of teaching. Teacher Questionnaire: Another tool used in this evaluation was a teacher questionnaire. This tool was used to answer evaluation question #2.1. The questionnaire asked teacher’s perception of the new mother tongue textbooks and teacher’s guide. The questionnaire helped the READ M&E team gather information relevant to the evaluation question from a large group of respondents. The questionnaire was a self-report questionnaire and teachers filled it out themselves. Piloting of Instruments Before using the instruments for data collection, READ M&E conducted a pilot study in two languages (Amharic and Afan Oromo) to ensure the appropriateness of the tools for the intended purposes. Data was collected from samples selected from the two languages and the READ M&E team involved in the data collection made the analysis. Results of the pilot suggested some changes to the instruments. In the teacher interview guide two questions were found to elicit similar responses and considered to be redundant. Hence, the team retained only one of the two questions. In the same instrument, another two questions were found to have the same meaning and one of the two items was omitted. The pilot result showed that two questions with their sub-questions in the interview guide for 9 experts from MoE, RSEBs, Zones and READ TA were found to be inappropriate for experts from zone education offices and decided that experts from zone education offices should not be asked these questions. In addition, READ M&E rephrased alternatives for one of the questions to include an additional option. The pilot result did not suggest major changes in the interview guide for principals and supervisors except for adding one question. The new question asked about the roles of supervisors and principals in the implementation of the new curricula. Finally, the piloting of the questionnaire resulted in a change of format and the cancellation of two questions. One of the questions was cancelled as it was not a relevant question for all respondents and the second question was eliminated because it asked teachers to provide information on the syllabus. It was noted that teachers have no information about the syllabus used to prepare the new mother tongue language materials. Data Collection Procedures READ M&E recruited eighteen experienced qualitative and quantitative data collectors for each language group (two per language except for Afan Oromo and Amharic each had four). The minimum qualification of the data collectors was MA/MEd degree. READ M&E conducted a one￾day training. The training began with an overall orientation to READ programs and the materials produced by READ TA. A brief description of the tools developed was given followed by an intensive training on how to approach the respondents, use the data collection tools, record the data collected, keep the privacy of the respondents, etc. based on the data collection protocol and data collection training manual. During and after the training, a number of issues were raised and clarified. On some of the issues agreement was reached to create consistency in the data collection process. For example, it was agreed that interviews conducted with all groups of respondents would continue until the point of saturation is reached (when no new information comes from the interviews). After the instruction, the data collectors simulated the instrument amongst themselves in pairs. Post-practice, data collectors and READ M&E staff discussed problems encountered and suggested solutions. Nine teams of data collectors were deployed to eleven study sites (East and South East Tigray, South Gondar, South Wollo, West Arsi, East Shoa, Fafan and Siti of Somali, Sidama, Wolayta and Hadiya) and the data collection was held from December 20, 2015 to January 8, 2016. As a strategy of quality assurance, three READ M&E teams supervised the data collection process. Additionally, READ M&E team members conducted interviews in Addis with MoE and READ TA staff. READ M&E also undertook a document review in close collaboration with appropriate partners. Data Analysis Analysis of Qualitative Data Nine experienced consultants transcribed 282 interviews conducted in seven languages with teachers, school principals and cluster supervisors, CTE staff, material developers, RSEB and Zone experts, READ TA staff, and experts from MoE. READ M&E Addis team, with the help of the home office, evaluated for inter-rater reliability (IRR) before coding started using qualitative 10 software NVivo 10. Five experts (four staff from AIR Addis Office and one from AIR DC Office) randomly selected one transcription from each respondent type (six transcriptions) and coded by five experts for the inter-rater reliability. The IRR confirmed that the initial coding made by READ M&E field office and home office as highly consistent and reliable. Then coding of 130 transcriptions selected randomly for each respondent group from the seven languages was carried out using NVivo 10. After coding had been completed for the large amount of qualitative data, READ M&E randomly selected representative transcriptions from each respondent group for analysis. AIR adheres to strict data analysis principles, regardless of the type of data being collected. The READ M&E team created a preliminary coding outline and structure on the basis of the evaluation questions, interview protocols, and memos of ideas that emerged during data collection. This codebook served as the tool to organize and subsequently analyze the information gathered in the interviews. The outline was a living document that may be modified as new themes and findings emerge during data analysis. A list of definitions for the codes accompanied the outline, so that coders categorized data using the same standards. Using the coded data, the qualitative team used grounded theory to identify themes, categories, and theories that emerged from the data and that confirmed or refuted the researchers’ initial impressions. That is, rather than basing the analysis on a hypothesis, the team created concepts and categories based on the data, refining the concepts as they go along to eventually inform the overall findings. During this process of data reduction, the team characterized the prevalence of responses, examined differences among groups, and identified key findings and themes related to the research questions. Analysis of Quantitative Data Separate data entry templates were prepared using Statistical Package for Social Scientists (SPSS) version 21 for the questionnaire and observation checklist. A total of 245 questionnaires and 192 observation checklists were entered, cleaned, and analyzed. Data cleaning ensured the accuracy of data entered. Simple percentages, graphs, other descriptive statistics and simple parametric (t-test for independent samples and One-way ANOVA) were used to analyze the data. For parametric tests, READ M&E created three composite scores by scoring and adding the responses of the respondents for all items on a five point Likert Scale. One score was for satisfaction with the contents of the materials, one for satisfaction with the link between the contents of the new textbooks/guides and the learning methodologies and strategies employed, and one for satisfaction with the physical qualities of the new mother tongue materials. Higher score on the scale indicates satisfaction while lower score on the scale indicates dissatisfaction. The scores derived were analyzed for comparison purposes. Analysis of Data Obtained from Document Review The READ M&E team reviewed different documents including annual reports. Thorough examination of the documents revealed both quantitative and qualitative data and the data was presented both in narrative form and quantitatively. 11 Ethics Review AIR conducts rigorous ethical reviews though our Institutional Review Board (IRB) for all of our own internal research activities and provides this service for a variety of subcontractors and collaborators. AIR’s IRB has conducted expedited and full board reviews of research involving human subjects for more than 16 years. AIR is registered with Office of Human Research Protection (OHRP) as a research institution and conducts research under its own Federal Wide Assurance. The data collection took place after securing full approval from the AIR ethics board. Data collectors began data collection after securing and recording the verbal consent of those being interviewed. The data collectors assured the confidentiality of data throughout the evaluation process. 12 FINDINGS This section presents a synthesis of the qualitative and quantitative findings gathered through the semi-structured interviews, questionnaires, and observational instruments. READ TA’s main activities are concentrated around four Intermediate Results (IRs); as such, findings are organized by intermediate results. Intermediate Result 1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In-Service Teacher Training Developed 1.1 Status of Development for the New Mother Tongue Curricula READ TA completed textbook and teacher guide development of 112 separate materials, covering seven mother tongue languages in grades 1-8.1 READ TA began the process with a review of the previous language learning materials to identify gaps in the reading curriculum> The review included content, pedagogy, and assessment. READ TA then launched a Mother Tongue Task Group to review the Mother Tongue Syllabus and the Minimum Learning Competencies (MLCs). Upon complete revision of the National MT Syllabus, a National Mother Tongue Consultative Workshop was held in Adama from June 19- 21, 2013. Representatives from the MoE, RSEBs, USAID, READ TA and partners, validated the newly revised National MT Syllabus for grades 1-8. Consistent with READ TA’s documentation, respondents across language groups similarly describe the completion of the development of MT instructional materials. A regional READ TA expert in Amhara details the granular steps undertaken to develop the MT materials as follows: “1) syllabus was validated at the national level; 2) syllabus was reviewed and approved by a task force at the regional level; 3) READ TA trained material development teams, which included story writers, editors, illustrators, and quality assurance experts; 4) content for grades 1 - 4 textbooks were developed by the trained material development teams with technical assistance from READ TA.; 5) pilot testing of new materials; 6) stakeholders, including religious and political groups, cultural bureau reviewers, and others, provide input on final draft; 7) final materials submitted to READ TA head office”. Respondents discussed the careful attention paid to regional differences in the materials. READ TA adapted of the mother tongue student textbooks and teacher guides for grades 1 - 8 for twelve language communities (Amharic Textbook was adapted for eight language communities in different regions. Afan Oromo was adapted for three language communities and Af-Somali for one language community). The adaptation work focused on accommodating unique cultural and linguistic aspects of the twelve language communities in various parts of the country. Contextualization was accomplished with technical support from international reading consultants, Reading, Curriculum and Language Experts from READ TA, and Summer Institute of Linguistics – Language Education and Development (SIL LEAD). Their work made the texts 1 READ TA Year 3 Quarterly Report (October – December 2014) 13 more relevant to children in different environments. For example, the Amharic textbook was contextualized for children living in the Addis Ababa urban environment. An expert with READ TA Reading, Curriculum and Language Experts explains the adaptation work for the Wolaytatto language as follows: For example, for the Wolayta we made it to reflect Wolayta culture. To do this we consulted all materials prepared in Wolaytatto, we visited culture and tourism office and consulted individuals. Generally, we used original sources. Thus, in the SNNP we made the Wolaytatto [textbook] to reflect Wolayta culture; the Hadiyissa to reflect Hadiya culture; and Sidama Afo to reflect Sidama culture. In general, the review of documents and the analysis of qualitative data regarding the status of development of the MT instructional materials indicate that READ TA did remarkable work and has successfully completed the development of mother tongue instructional materials. 1.2 Coordination between READ TA, MOE, and RSEBs Despite initial challenges, respondents describe the overall coordination as productive with a shared sense of ownership. READ TA cites a number of issues in the October 2012 – June 2013 annual implementation plan of challenges to project coordination and implementation. Initially, the lengthy process to obtain a Certification of Registration from the Charities Societies Agency (ChSA) and difficulties in scheduling a meeting with the Ministry of Education (MoE), due to the MoE’s busy schedule, which warranted repetitive rescheduling of meetings, is reported to have resulted in delaying activities. Furthermore, obtaining prior approval from the RSEB for every activity determined READ TA’s pace in the implementation of its plan. In an effort to harmonize coordination, READ TA supported the RSEBs in the establishment of a Mother Tongue Task Force to review the templates to be used by Regional Mother Tongue Curriculum Development Team for developing lessons, student textbook and teacher guide. RSEB staff was available for the various reviews and meetings held by Regional Mother Tongue Curriculum Development Team 2 . In guiding the coordination of MT instructional material development, READ TA utilized a Memorandum of Understanding (MOU) signed between the MoE and RSEBs, Implementation Plan documents, and the guideline for the formation of task force. Despite these mechanisms for managing coordination, unanticipated issues were acknowledged to have caused challenges, primarily at the regional level. Respondents generally commented on the strong collaboration that worked according to the guidelines set out to govern the process. A Reading, Curriculum and Sidamu Afo Language Experts from READ TA regional field office states: From the Ministry of Education to lower level zonal education department, there were taskforces. There were memorandums of understanding between these organizations. The work has been done in solid collaboration with those organizations. 2 READ TA Annual Implementation Plan (October 2012 – June 2013) 14 In one region in particular, SNNPR, issues with regional level coordination was noted as material development was undertaken in three different languages, as detailed by an expert from the zone education office: There are three languages in this region. MOE was only contacting regions and there were gaps. The region was troubled and confused. We offered directions to the region and fixed the problem. In a meeting organized by MOE at Adama just before the final validation workshop for grade 1 to 4 books we discussed the problems. It was finally decided that the three zones in the region have to be considered as regions. Despite the acknowledged challenges, overall coordination is predominantly described by the respondents as generally being productive with a shared sense of ownership. One respondent, albeit, from the Amhara region READ TA field office, reflects on the overall coordination as follows: There was no activity or step that contradicted with the interest and intention of the MOE. But the day to day coordination was between the regional level READ TA field office and task force at regional level. Hence, there were no major challenges in the coordination scheme and implementation. The READ TA instead of handing over the finished product brought all the international best practices, organized, and mobilized local capacities. Accordingly, with all technical assistance and financial support from the READ TA the RSEB managed to get the current new materials. Respondents from all language groups share a similarly positive perception on the overall coordination. A respondent from Somali Region states: The coordination among the partners was very successful because of their consultation, transparency, and commitment. I personally assumed that all parties were working as one collaborating in all aspects of the project. Another respondent from SNNP Education Bureau states: The good part of this collaboration was the transmission of skill, knowledge and experience. There were international and local experts from the Ministry of Education, zones and regions. There was very good knowledge transmission process. Despite the initial challenges to the coordination activities, the majority of the respondents across language groups report that the coordination was smooth and productive. In many instances, respondents report coordination facilitated “knowledge/skill transmission”, “commitment”, “collaboration”, “consultation”, and “sense of ownership.” 1.3 Alignment of the Mother Tongue Syllabus, Student’s Textbooks, and Teacher’s Guide with the MLCs The evaluation explored the level of alignment of the mother tongue syllabus, student’s textbooks, and teacher’s guides have with the minimum learning competency (MLC). Despite the workshop trainings and validation checks, the respondents, regarding the alignment of the syllabus, students’ textbook and teachers’ guide with the national Minimum Learning Competency (MLC), provided conflicting accounts. One respondent involved in material development in the Afan Oromo language states: 15 I received training about MLCs that focused on how to develop teaching materials for mother tongue language that align with the MLCs before the start of material development. The training organized and provided by the reading specialists of READ TA was highly useful in clearly specifying the MLCs expected of children of each grade level and how to develop material that exactly reflects or considers the MLCs identified for mother tongue language and for each grade level as well. Similarly, a respondent involved in development of materials in the Wolaytatto language states: At the beginning, we got the MLCs document and we have started our work by having that document; for all grade levels from grade one to four. Based on this where do we expect the child to be in grade one first semester, in second semester; in grade two and the like; the contents and lessons are aligned to the expected competencies. Despite the workshop trainings and validation checks held, a few respondents state that the final developed MT materials lack alignment with the national MLCs. As one material developer explains: There was poor preparation and lack of alignment with the MLC and the syllabus. I have some reservation on grade 3 and 4 materials regarding their perfect alignment with the MLC and their overall quality standard despite the fact that they passed through quality assurance. On the other hand, I feel that grade 1-2 materials are well developed and prepared. On the other hand, one of the respondents articulated that judging about the alignment of the syllabus and materials with the MLCs during the development phase may not reveal conclusive results and an objective mechanism may be waiting for students’ achievement on existing measures such as national standardized exams. This respondent state, It is difficult to rate the degree of the alignment the syllabus and mother tongue materials have with MLCs. Although we claim that there is perfect alignment during the development phase, the true alignment should be examined in terms of students’ achievement. This in turn requires field visit and empirical evidence. In general, there are different views regarding the alignment of the newly developed MT instructional materials with the national MLCs. Some respondents describe the presence of alignment while others state that it is difficult to judge at the development phase. Students’ results on national standardized exams may be a more objective measure indicative of the level of alignment between the newly developed MT instructional materials and the national MLCs. 1.4 Student to New Textbook Ratio READ TA is not responsible for providing textbooks to schools or guides to teachers. The government of Ethiopia is responsible for printing and distributing textbooks. However, READ M&E queried respondents about the extent of the availability of textbooks and teacher’s guides in their school as it contextualizes the work of READ TA. Quantitative findings indicate insufficient distribution of MT language student textbooks across language groups. The majority of teachers who responded to the questionnaire reported the availability of the new mother tongue textbooks in their classrooms. However, most of these respondents indicated that the availability of textbooks is not one per student. Approximately 16 77% of grade four teachers confirmed this, reiterating that the distribution of the new mother tongue textbooks is not one - to - one. Refer to the following table for availability and ratio of student textbooks disaggregated by grade level. Table 3: Teachers' report of the availability of textbooks Items Responses Yes No Total N % N % N % Do your students have the new mother tongue textbooks? Grade One 72 97.3 2 2.7 74 100.0 Grade Two 66 94.3 4 5.7 70 100.0 Grade Three 57 89.1 7 10.9 64 100.0 Grade Four 57 87.7 8 12.3 65 100.0 Does every student have his/her own textbook? Grade One 22 29.7 52 70.3 74 100.0 Grade Two 16 23.2 53 76.8 69 100.0 Grade Three 12 19.4 50 80.6 62 100.0 Grade Four 14 21.5 51 78.5 65 100.0 Data obtained from the questionnaire indicates that on average the textbook to student ratio for the first three grade levels is 1:4. Similarly, the classroom observation revealed that out of 4,237 boys in the class, 1,070 boys had the new textbook which makes textbook-student ratio for boys 1:3.96 i.e., there are 100 books for 396 boys. Similarly, out of 4,200 girls in the classrooms observed, 1,171 of them had the new textbooks. The textbook-student ratio for girls is 1:3.59 i.e., there are 100 books for every 359 girls in the classrooms observed. The textbook-student ratio for both sexes is approximately 1:4. Please refer to the following table for a breakdown of average student to textbook ratio by language and grade level. Table 4: Average Textbook to Student ratio by language and grade level READ M&E asked the respondents if they know the reason why the ratio of students to textbooks is not 1:1. Out of 195 teachers, 160 (82.1%) indicate that the reason is shortage of printed textbook. About 8.7% of the teachers knew that their schools got very few textbooks when distributed or adequate number of textbooks did not arrive at school. Furthermore, 64.8% of the respondents reported the availability of teacher’s guide in their school while 32.2% of the respondents indicated the absence of teacher’s guide for use in their schools. Language Group Grade 1 Grade 2 Grade 3 Grade 4 All Grades Afan Oromo 1:7 1:4 1:3 1:5 1:4 Amharic 1:3 1:3 1:4 1:3 1:3 Tigrigna 1:2 1:1 1:2 1:1 1:1 Sidamu Afoo 1:10 1:12 1:12 1:8 1:10 Wolayttatto 1:4 1:5 1:3 1:3 1:4 Hadiyissa 1:7 1:9 1:7 1:13 1:5 Af-Somali 1:6 1:9 1:9 1:3 1:6 17 Table 5 shows the details of the responses. 18 Table 5: Availability of Teachers' Guide Grade Level Are the newly developed teacher’s guides available to you at the school where you teach? Yes No Non Respondent Total N % N % N % N % Grade 1 53 71.6 20 27.0 1 1.4 74 100.0 Grade 2 43 61.4 22 31.4 5 7.1 70 100.0 Grade 3 37 57.8 26 40.6 1 1.6 64 100.0 Grade 4 44 67.7 20 30.8 1 1.5 65 100.0 Total 177 64.8 88 32.2 8 2.9 273 100.0 As shown above, more teachers from Grade 1 (71.6%) indicated the presence of teacher’s guide in their schools, where more teachers from Grade 3 reported the lowest level of availability, with a full 40.6% reporting no availability. Qualitative and quantitative findings reveal significant shortage of both students’ textbooks and related teachers’ guides, with slight variation between grades. Although, READ TA is not responsible for printing and distribution of Grades1-4 students’ textbooks and teachers’ guides in the seven mother tongue languages, it is valuable data to gather and report as it has implications for the impact of the READ projects. The majority of the respondents, across the language groups, reported limited availability of the MT instructional materials in their schools up to the time of data collection (December 20, 2015 to January 8, 2016). The availability of the new MT teacher guides is also similarly reported as insufficient. One respondent, a primary school principal in SNNP region, states: The books are scarce. For example, the number of grade one students is 832, but the books available are around 120; grade two students are 519, whereas the number of books is about 75; grade three students are 375, the number of books is 80; grade four students are 514, the books available are 75. In contrast to the above, a teacher in Tigray accounts: The distribution is complete. The newly developed textbooks of the Tigrigna curricula for grades 1-4 are adequate. The ratio of textbooks to students is 1 to 1; the distribution is adequate. A few respondents account partial availability of the MT textbooks without accompanying MT teacher guides: There was only one teacher’s guide of grade 3 in our school and there is no teacher’s guide for the other grade levels at all. A teacher respondent in the Amhara region explains: The new textbook is available to us and to students however, we teachers are sharing one teacher’s guide for three sections. And there is a shortage of student textbooks and teacher’s guides. Respondents describe efforts taken to increase access to the materials. Strategies such as encouraging sharing of textbooks by students and schools taking initiative to make copies are reported. A principal in Amhara region states: 19 The distribution is very inadequate; one book is given out for three and sometimes four students. As a result, it is creating problem in the teaching process for us. Similarly, a respondent explains: The number of books is not enough for students. I wish the ratio of books to student would be 1:1. This may make the teaching and learning process easy for the teacher. Correspondingly, a principal in Oromia Region states: The teacher’s guides are totally absent in our school. Even, the students’ textbooks are not available in our school in adequate number or copies. For instance, there are only 10-20 textbooks in each class, varying across each grade level. This means, there is a mismatch or incompatibility between the student population and the number of the new textbooks distributed to our school. To curb this situation, the management of our school has tried to photocopy some of the textbooks. But due to budget deficit, the school could not be able to satisfy a large student population. 1.5 Student Textbook Use Almost all respondent teachers state that their students are using the newly developed MT textbooks where available. Students were organized to access the MT textbook in groups and/or in rotation. Students who have received MT textbooks reportedly bring those textbooks to class regularly. A respondent, teacher in Amhara region explains: It is mandatory to bring the textbook to the class. Because books are inadequate and were given to them in a group hence, all students who have the textbooks bring them to Amharic class. If a student misses the class, he/she should send it to the class. Since the number of books in the class is inadequate, I can say that 100% of students who have the textbooks bring the textbooks to the class regularly. Correspondingly, most of the teachers interviewed provided similar answers. For example, a teacher from one of the schools in Oromia states: There is severe scarcity of the new students’ textbooks. Approximately, there is one textbook for 10 students in each class and each grade level. As a rule, the students are organized to use the textbooks by rotating among themselves. A teacher from Hadiyissa language group says: All the students who have the book always bring it to the class because I will not allow them to site in the class if they forget to bring. In contrast, a teacher in Somali region states: No, the students are not using the textbooks, as the textbooks have not reached most of the students yet. In line with the above qualitative findings, the quantitative findings yielded similar results in terms of material availability. The quantitative instruments used for data collection explores how often students bring the new mother tongue textbooks to the class. The majority of respondents (69.5%) indicated that those students who have the textbooks bring them to the class ‘always’ when they have mother tongue language class. Almost 20% (19.7%) of the respondents report that students ‘Sometimes’ bring the new textbooks to the class. The details are presented in Figure 1. 20 Figure 1: Teachers' report on how often students bring textbooks to class READ M&E observed no difference in the responses of teachers teaching at different grade levels with regard to their responses to the question “how often students bring textbooks to the class?” Data gathered through qualitative classroom observations yielded similar result. On the average 5.57 boys and 6.1 girls (approximately 12 students per class) brought the new mother tongue textbooks to the class with standard deviations of 5.398 and 5.834, respectively. In other words, 25.3% of males and 27.9% of females did bring the new mother tongue textbooks to the class. Similarly, the classroom observation revealed that 84% of teachers brought the new mother tongue textbooks to the class whereas 16% teachers who did not bring the textbooks were observed either borrowing the textbooks from students, using the note they prepared based on teacher’s guides and student textbooks, or consulted student’s textbooks while students were using. The analysis of data by languages revealed the following results. 21 Figure 2: Percentage of Teachers who brought to class the new Mother Tongue Textbook The data shows that relatively more number of teachers from Sidamu Affoo and Af-Somali languages did not bring the new mother tongue textbooks to the classroom. On the other hand, all mother tongue language teachers in the classrooms observed from Tigrigna language group brought the textbooks to the class. Out of those teachers who brought the student textbook to the class, 93.8% teachers were using the student textbook while teaching. Only 10 (6.2%) teachers did not use the new textbooks while teaching. Related to this, teachers were asked whether they use teacher’s guide while teaching. Out of 177 teachers who described the presence of teacher’s guide in their school 127 (71.8%), respondents said that their teaching heavily relies on the use the new teacher’s guide. The details are presented in Figure 3. 22 Figure 3: Frequency of use of teachers guide by language Contrary to the results of the quantitative questionnaire, the classroom observation yielded that the majority of teachers (63%) did not bring the new mother tongue teacher’s guides to the classroom at the time of observation. That is, data collectors did not observe these teachers using or referring to teacher’s guide teaching. When analyzed by language groups the data revealed that relatively more teachers from Tigrigna, Af-Somali and Wolayttatto languages brought teachers guide to the class than teachers from the other language groups. It is important to note at this point that desirability bias may affect the responses to the survey tool and the classroom observation may provide better representation of the actual situations in the classroom. Figure 4: Teachers who brought the new Mother Tongue Teachers' Guide to class As the figure depicts, there is difference across language groups in the proportions of teachers who brought the new mother tongue language teacher’s guide to the class. Out of those 23 teachers who brought the new teacher’s guide to the class 70% of them used the guide while teaching. Student’s use of the new textbook during instruction was observed with respect to whether students were reading the textbook, copying from textbook or doing exercise from the textbook. The data gathered show that the use of the new mother tongue textbooks was high only in 20.8% of the classrooms observed. The details are shown in Figure 5. Figure 5: Level of textbook use in the classroom by students The above figure shows that textbook use during instruction by the students was moderate and low for equal proportions of the classrooms observed. In about sixteen (8%) classrooms observed, there were no student textbooks. Teachers were teaching mother tongue language using their own copies of student textbooks and teacher’s guides. Most of these classrooms (12 out of 16) were in Somali Regional State and the other four classrooms were in Oromia Regional State. READ M&E observed that where textbooks are available, students across all regions bring MT instructional textbooks to class on regular bases. Overall, teachers report the majority (69.3%) of students ‘always’ bring textbooks to class. This is a significantly favorable finding, although this indicator is not used to rate the performance of the READ TA project; this has implications on the success of this project. 1.6 READ TA Support to Colleges of Teacher Education The majority of respondents expressed general satisfaction with the level of support provided by READ TA to Colleges of Teacher Education (CTE). In an effort to ensure effective pre￾service training in teaching primary Mother Tongue reading and writing, READ TA has supported CTEs in the development of the new CTE MT curriculum and modules as well as in training CTE MT instructors on the new MT Curriculum and modules. Each of the modules addresses a specific content area, and supports a specific course within the new CTE MT Curriculum. Additionally, READ TA provided CTEs with ICT packages of a tablet for each lecturer of mother tongue instruction at the CTE, a server, projector, speakers, and document camera. Furthermore, trainings on the use and maintenance of the ICT packages were provided to the 24 ICT teams of both the RSEBs and CTEs. CTE instructors were trained on the use of the ICT packages. A respondent from a CTE in Oromia explains: The types of support that READ TA made available to our CTE are enormous. It helped us in providing training for teachers teaching the mother tongue language (Afan Oromo), developing the pre-service training materials (modules) for mother tongue language teachers, and providing the soft and hard copies of these materials both the English and Afan Oromo versions. In addition to preparing teaching materials both for the primary schools and the pre-service training in the CTE, READ TA has also provided technical support such as giving orientation to teachers teaching mother tongue language on how to make use of modules and the information loaded on tablet and videos. The majority of respondents expressed general satisfaction with level of support provided by READ TA to CTEs in their efforts to train MT teachers and develop materials for pre-service training at CTEs. Different respondents from Amhara, Hadiya and Tigray respectively state: The support is really good. We cannot say that it is something simple. The support is in terms of both material and knowledge. The support aimed at producing well equipped mother tongue language teachers who can properly address the problems related to poor reading and writing skill with mother tongue language. The support is really very strong. The type of modules prepared and the teaching methodology planned to be followed while training mother tongue language teachers is very strong. I would say the support provided by READ TA is very good. The effects of the support offered by READ TA are lasting and benefit the college, school children and the next generation too. Since the project was initiated after identifying the real problems of children and the materials were developed passing through rigorous steps, I believe that the support provided would properly address the reading deficiencies of our students. I would describe the level of support we received so far from READ TA as very good; because the support made is very adequate. Generally, the respondents perceived the technical and technological supports provided by READ TA for CTEs favorably. The respondents appreciated the technical assistance offered by READ TA in the development of the new CTE MT Curriculum and modules along with support in training of teachers. Similarly, positive reflections are received on the technological resources provided. Some even mentioned the extent that the type of technological support made available is the first of its kind to most of the CTEs and has helped individual instructors in changing their methods of teaching and becoming more effective. 25 Intermediate Result 2: Application of Language-Specific Teaching, Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write 2.1 Respondents’ Views of New Materials Teachers report varying degrees of satisfaction with student textbooks and teachers’ guide prepared by READ TA. A large portion of the teachers interviewed report “being satisfied with the new books” because of their “clarity and simplicity” especially in comparison to previous learning materials. One principal from Tigray states: Compared with previous materials, the new materials are satisfying, though I would not take them to be complete on their own. A principal from Hadiyya zone described the improvement further: The old textbooks had problems related to word choice; some of the words were vague for another dialect speaker; the new mother tongue textbook is much better than the old textbook. In addition to improved word choice, increased use of mother tongue was also cited by teachers: I am very satisfied with the materials. First the textbooks are prepared in students’ mother tongue language. I believe that learning with ones’ own mother tongue language is very important to enhance skills and understandings. Also mentioned by teachers, principals, and supervisors about the improved presentation of the new materials, numerous teachers reported that the new materials’ design were “inspirational” and helped to hold students’ attention. One principal states that: The book is elegant and interesting starting from the outer cover. Though the majority of teachers, principals, and supervisors positively describe the learning materials developed by READ TA, many report the need for further improvements. Teachers’ main complaints focus on the quantity of material included in the READ TA learning materials: …The materials are vast and thus they need to be reduced to make them appropriate to the level of students in these grades. Issues due to vast content are also mentioned by principals: Teachers commented that the materials are vast. There are 3 to 4 exercises in each lesson, and according to the weekly distribution scheme the topics in each semester are so vast that they cannot be completed in 15 weeks. Similarly, teachers also suggested changes in the sequencing of lessons. According to one teacher from Amhara mentions changes to the student textbook that still need to be made: What needs to be improved in this new textbook is the sequence of the contents should be re-arrange over again. Some of the contents are difficult for student. For example, arrangement and it is better if those compound words place at the end of the chapters rather than at the beginning of the chapter. The majority of respondents agreed that the newly developed READ TA materials were an improvement from their old instructional materials with varying degrees of overall satisfaction. Teachers often mentioned that READ TA materials were better than the previous books at 26 incorporating mother tongue into the lessons. However, teachers frequently mentioned that it was difficult to cover all of the materials in the given instructional time. 2.2 Content of the New Mother Tongue Language Textbooks The majority of teachers mentioned that the use of language-specific teaching and learning strategies were very “helpful.” Several teachers stated that READ TA’s division of materials into specific days and weeks made “implementation more effective and convenient.” Respondents also cited that the new methods were more “interactive” and helped to keep students’ attention. One respondent from the Somali region described how students’ attention improved with the use of the new teaching methods: The organization, the structure, the contents and activities of the newly developed mother tongue curriculum has attracted the students so that they eagerly wait for the Somali language period whereas the previous curriculum used to make them passive. In addition, one teacher from the SNNP region mentioned that the new methods helped students to read: Segmenting words into syllables, we have considered that it is too much; and why not other items taken in balanced way; yet, segmenting words into syllables allowed students to identify where sounds break and that enabled students to learn reading quickly. Principals, across language groups, reported a high level of satisfaction with the use of language￾specific teaching and learning methodologies. According to one principal from Oromia region, students’ performance has improved as a result of the new materials: I am highly confident that the new materials on the mother tongue language have helped teachers in our school to change their method of teaching the mother tongue language. This is because significant changes are being observed in the students’ performance compared to the previous times, where this could be attributed to the new method that teachers are using at a moment. Besides the qualitative data, teachers’ overall level of satisfaction of the link between the contents of the new textbooks/guides and the learning methodologies and strategies embedded was assessed on nine items on five point Likert scale. However, for the sake of simplicity in data presentation, READ M&E combined the two positive scale values (Very Satisfied and Satisfied) into a single scale (satisfied) and similarly the two negative scales values (Very Dissatisfied and Dissatisfied) were merged together into one scale (dissatisfied). The result shows that the majority of teachers (69%) expressed their satisfaction with the link between the contents of the new textbooks/guides and the learning methodologies and strategies employed. On the other hand, 25.7% of teachers were not able to tell whether they are satisfied or dissatisfied with the link between the contents of the textbooks and the methodology embedded in the new mother tongue materials. Only 4.5% of teachers expressed their dissatisfaction. The data was further analyzed by language and the results are summarized in Figure 7. 27 Figure 6: Overall satisfaction of teachers on the link between contents and embedded methodology by language As depicted in Figure 6, more teachers from Wolayttatto, Sidamu Afoo, Afan Oromo, and Tigrigna language groups expressed greater satisfaction than teachers from other language groups. Further analysis of the responses for individual items confirms this result. Table 6: Distribution of teachers' responses to item on the link between the contents of the new textbooks/guides and learning methodologies & strategies employed No. Items Response Categories Satisfied Unsure Dissatisfied Total N % N % N % N % 1 The alignment between the student textbooks and teacher’s guide 171 70.1 38 15.6 21 8.6 230 94.3 2 The arrangement of topics from simple to complex 202 82.8 10 4.1 30 12.3 242 99.2 3 Exclusive use of mother tongue language in the textbook 216 88.5 4 1.6 37 15.2 242 99.2 4 Good use of illustrations 194 79.5 9 3.7 34 13.9 237 97.1 5 Providing opportunities for continuous performance assessment and assessing student progress 200 82.0 13 5.3 29 11.9 242 99.2 6 The teaching strategy in the teacher’s guide are designed to be interactive 183 75.0 33 13.5 19 7.8 235 96.3 7 The capacity to trigger interest in the learners 217 88.9 6 2.5 18 7.4 241 98.8 8 Pedagogical appropriateness of the sequence of objectives in the teacher’s guides 181 74.2 33 13.5 24 9.8 238 97.5 9 Strategies promoting active learning 207 84.8 10 4.1 21 8.6 238 97.5 As depicted in the above table, relatively high proportion of teachers expressed their dissatisfaction related to the exclusive use of mother tongue language in the textbooks followed by the use of illustrations in the textbooks and teacher’s guide. In addition, the total composite 28 scores on the link between the contents and the learning methodologies and strategies were compared across language groups. The minimum and maximum possible scores on this scales were 9 and 45, respectively. Table 7. Mean and standard deviation (SD) of satisfaction scores on the link between the contents of the new textbooks/guides and learning methodologies & strategies employed Language N Mean SD Tigrigna 26 37.73 6.576 Amharic 51 31.82 9.010 Afan Oromo 44 38.93 6.120 Sidamu Afoo 39 38.54 6.656 Wolayttatto 36 39.53 5.074 Hadiyissa 29 34.69 5.733 Af-Somali 18 36.28 5.550 Total 243 36.63 7.267 As shown in Table 7, the mean satisfaction scores of the respondents from the different language groups appear to be similar and high. The closer the scores to the maximum possible score (45), the higher is the level of satisfaction. Only the mean score of the respondents from Amharic language appeared to be low. A One-way ANOVA3 revealed statistically significant difference between language groups in the level of satisfaction with respect to the link between the contents of the new textbooks/guides and the learning methodologies and strategies. However, the difference does not imply that respondents from the Amharic language group are not satisfied with the link between the contents of the new textbooks/guides and learning methodologies and strategies employed. Tukey pairwise mean comparison4 revealed statistically significant difference in the level of satisfaction of teachers from Amharic language with those from Tigrigna, Afan Oromo, Sidamu Afo, Wolaytatto languages. There is no statistically significant difference in the level of satisfaction between any other pair of languages concerning the link between the contents of the new textbooks/guides and the learning methodologies and strategies employed. Further analysis was made to see if there is difference between male and female teachers in their ratings of the contents of the new mother tongue textbooks and teacher’s guides in terms of the use of language specific methodologies and strategies. The analysis revealed no gender difference in terms of level of satisfaction with respect to the link between the contents of the new textbooks/guides and learning methodologies & strategies employed. 3 A one-way analysis of variance (ANOVA) is used to determine whether there are any significant differences between the means of three or more independent groups. 4 Significant ANOVA does not tell which groups differ significantly. Tukey pairwise mean comparison is one of the procedures used to find means that are significantly different from each other. It compares all possible pairs of means or groups on the basis of the variable of interest. 29 Similarly, the evaluation assessed teachers’ level of satisfaction with the contents of the new mother tongue instructional materials. A majority of teachers 197 (80.4%) were satisfied with the contents of the new mother tongue languages textbooks while five (2%) respondents were not satisfied with the contents. Interestingly, forty-one (16.7%) of teachers were unsure of their level of satisfaction regarding the contents of the new textbooks. They preferred to take the middle position on the scale. The analysis by language groups revealed the following results. Figure 7: Overall satisfaction of teachers on the content of the new materials by language Teachers were asked to rate the contents of the new mother tongue textbooks in terms of continuity of lessons from one topic to the other, grade level appropriateness, level of difficulty, readability, cultural appropriateness for the local context, relevance in terms of representing both boys and girls, and promoting life skills. Life skills includes, healthy habits like washing hands after toilet, telling the truth, or good manners. Teachers were asked to rate the comprehensiveness on a five-point scale. A higher score on the scale (>29) shows satisfaction while a lower score (< 18) shows dissatisfaction with the contents of the new textbooks. The minimum possible score is 8 while that of maximum possible score is 40. Table 8 provides descriptive measures on satisfaction scores of the respondents. Table 8: Mean and standard deviation (SD) of satisfaction scores for content by language No. Language N Mean SD 1 Tigrigna 26 33.12 4.983 2 Amharic 51 28.61 7.161 3 Afan Oromo 44 36.30 3.501 4 Sidamu Affo 39 34.03 5.417 5 Wolayttatto 36 34.11 5.888 6 Hadiyissa 29 33.31 2.989 7 Af-Somali 18 34.00 3.678 Total 243 33.13 5.780 30 The analysis of the data yielded statistically significant difference in level of satisfaction between the respondents of the Amharic language group and the rest of the language groups. It appeared from the data that level of satisfaction is relatively low for respondents from the Amharic language. However, this does not mean that the respondents from Amharic language group are dissatisfied with the contents of the new mother tongue textbooks. Rather compared to respondents from other language groups, the level of satisfaction of respondents from Amharic language group is low. There is no difference in the level of satisfaction of respondents from Tigrigna, Afan Oromo, Sidamu Afoo, Wolaytatto, Hadiyissa, and Af Somali languages. In other words, the level of satisfaction of teachers with regard to the contents of the new mother tongue language textbooks is the same for these languages. Similarly, further analysis of the data by gender failed to reveal statistically significant difference between male and female teachers with regard to their level of satisfaction with the content of the new mother tongue language textbooks. Further analysis of the responses for individual items confirms this result. Table 9 shows the responses of participants to the items used to measure their level of satisfaction about the contents the new textbooks. Table 9: Response of teachers to contents of textbooks No. Characteristics or Quality of the Materials Response Categories Satisfied Unsure Dissatisfied Total N % N % N % N % 1 Continuity of lessons from one topic to the other 205 84.0 9 3.7 26 10.7 240 98.4 2 Grade level appropriateness 196 80.3 12 4.9 35 14.3 243 99.6 3 Level of difficulty 175 71.7 15 6.1 44 18.0 234 95.9 4 Readability 225 92.2 3 1.2 14 5.7 242 99.2 5 Cultural appropriateness for the local context 216 88.5 9 3.7 14 5.7 239 98.0 6 Representing both boys and girls 224 91.8 6 2.5 11 4.5 241 98.8 7 Promoting life skills 217 88.9 8 3.3 13 5.3 238 97.5 8 Comprehensiveness 189 77.5 13 5.3 25 10.2 227 93.0 Relatively, more number of teachers (18% and 14%) expressed their dissatisfaction related to level of difficulty of the textbooks and grade level appropriateness, respectively. In all other cases, the proportion of respondents who expressed dissatisfaction does not exceed 10%. 2.3 Physical Qualities of Teacher’s Guide and Students’ Textbooks In addition to improvements to the content of READ TA materials, teachers also suggest that certain physical aspects of the learning materials may be changed. Teachers and school principals frequently mentioned that the textbook size created issues for their students. According to one Tigray principal, the younger students were not able to handle the textbook size, “The size of the textbooks is big for children at these grade levels.” Teachers also described the need to improve the font size, the printing quality, and quality of illustrations used in the materials. Teachers frequently recommended that the textbooks should be in color. 31 Respondents from READ TA note that READ TA followed the template and prescription given by the MoE in all of these matters. Quantitative data explored the physical quality of student textbooks and teacher’s guides. Teachers were asked to evaluate the physical qualities of the new books in terms of quality of cover pages, quality of the papers and bindings, and quality of print including text and illustrations using five-point scale. The minimum possible value is 3 and the maximum possible value is 15. Higher score on the scale (12 and above) shows satisfaction while lower score (< 6) shows dissatisfaction with the quality of teacher’s guides and student textbooks. The analysis yields that 72% of the respondents are satisfied with the quality of the new books while 8.2% teachers report that they are dissatisfied with the quality of the materials. On the other hand, the analysis revealed that 18.4% of teachers participated in this evaluation are unsure of judging about the quality of the materials produced and hence do not know whether they are satisfied or dissatisfied with the physical quality of the new mother tongue textbooks and teacher’s guides. Further analysis of the data by language groups revealed that a greater number of respondents from the Amharic language group expressed dissatisfaction with the physical qualities of the student textbooks than respondents from other language groups. The details of the results are summarized in the figure below. Figure 8: Overall satisfaction of teachers on the physical qualities of the materials As shown in Figure 8 above, nearly all respondents from Sidamu Afo language group expressed satisfaction with the physical qualities of the new mother tongue instructional materials. Table 10 presents the descriptive measures on physical qualities of the new materials by language. 32 Table 10: Mean and standard deviation of satisfaction scores for physical qualities of new materials Language N Mean SD Tigrigna 26 12.54 2.803 Amharic 51 8.61 3.889 Afan Oromo 44 13.07 1.934 Sidamu Afoo 39 14.31 1.151 Wolayttatto 36 13.44 1.796 Hadiyissa 29 12.41 1.991 Af-Somali 17 11.47 2.741 Total 242 12.14 3.182 Although the statistical analysis yielded gender differences in terms of evaluating the quality of the new mother tongue materials, close observation of the data (see Table 11) does not really show the difference in real life situation among males and females. Generally, male respondents tended to rate the quality of the new mother tongue materials favorably than females). However, this does not imply that female teachers were dissatisfied with the quality of the materials. Table 11. Mean and SD of satisfaction scores for physical qualities of new materials by gender Gender of the Respondent N Mean SD Male 116 12.62 2.613 Female 127 11.64 3.616 Similarly, the analysis across language groups revealed statistically significant difference among the respondents of different language groups with respect their evaluations of the physical qualities of the new mother tongue textbooks and teacher’s guides. Consistently, respondents from the Amharic language group tended to rate their level of satisfaction lower than respondents from other language groups. Besides, respondents from Hadiyissa and Af Somali language groups rated their level of satisfaction at statistically significant lower levels than respondents from Sidama Afoo language group. The responses of teachers to individual items measuring the physical quality of mother tongue textbooks and teacher’s guides are summarized in Table 12 below. Table 12: Distribution of teachers' responses to items measuring quality No. How do you view the physical qualities of student textbooks and teacher’s guides in terms of quality of: Response Categories Satisfied Unsure Dissatisfied Total N % N % N % N % 1 Cover pages 192 78.7 11 4.5 40 16.4 243 99.6 2 Papers and bindings 191 78.3 11 4.5 41 16.8 243 99.6 3 Print (including text and illustrations) 205 84.0 5 2.0 33 13.5 243 99.6 As shown in the above table, the majority of teachers who participated in this evaluation are satisfied with the physical quality of the materials produced by READ TA. In general, the 33 findings suggest that the majority of the respondents rated the physical qualities of the new mother tongue instructional materials favorably 2.4 Master Trainer Selection The majority of experts interviewed state that READ TA used predetermined criteria in the selection of master trainers. Respondents from READ TA describe the selection of master trainers as entirely made by RSEBs. READ TA worked with the MoE to establish the selection criteria. However, the distinction of roles was not apparent to all participants. One expert from Tigray explains the roles of the various stakeholders in the development of the selection as, “criteria for selecting master trainers were prepared by TDP/MOE in collaboration with READ TA.” According to one expert from Oromia, criteria included, “gender issues, candidates’ capacity, and experience.” Though the majority of experts state that, “the plan was to select an equal number of males and females” to hold master trainer positions this was not always possible due to “low number of female instructors’ or underqualified female applicants. Additionally, experts mention that it was necessary for potential applicants to “know the language and the culture carefully and be able to give training.” An expert from Tigray zone states: The master trainers were from schools, colleges and universities; they were degree graduates in Tigrigna language; they have good performance records; they were involved in developing the materials and are believed to have developed their capacities as a result, and are considered to be capable of helping teachers improve their capacities. One expert explains the master trainer selection process: To be frank even when we selected from the zones we were very strict in our communication and following the criteria firmly. When we completed our task and sent back the twenty-six experts we assigned ranks for each of them. We have fully recorded effectiveness of each person and who has what weakness; we have recorded this for future work. Our selection always is based on whether the person can deliver it or not; job centered. For instance, last time when we recruited teachers for video recording we brought two of them; one was effective and accomplished it but the other was not. We immediately sent back the one who failed to do the task; without any compromise. Numerous experts mention being confident in the selection criteria as the result of the caliber of the selected master trainers. One expert states: After the trainers [have been] selected and engaged in the work I have seen that they are competent and I have no doubt that they are selected based on the criteria developed. The majority of experts mention that the predetermined criteria for master trainer selection resulted in high caliber candidates. 2.5 Training of Master Trainers In collaboration with the MoE, RTI and Save the Children, READ TA selected twenty-two (6F; 16M) experts to develop and adapt the Mother Tongue In-Service Teacher Training Manual. The Development Workshop was held in Addis Ababa from April 23 – May 21, 2014. The 34 objective of the workshop was to develop a Teacher Training Manual for Master Trainers and Training of Trainers (ToT) to train and support teachers in implementing the newly developed Student Books and Teacher Guides. READ TA provided training, guidance, support and a gender checklist. The gender checklist ensures gender and inclusive education considerations throughout the development of the manual. Though experts made few comments about the quality of the training, one expert from Oromia shares their thoughts about the training: I participated in the master training when the training agenda was on special needs and gender issues. During those days, I found that the training was participatory. Expert trainers express concerns about how attending trainings would influence their teaching responsibilities: We also wanted the training of master trainers to take place in regions. TDP personnel only see that training is conducted without considering the time lost in training would affect teaching…We appealed that the training should not take place when classes are on session with explanations of the problems that this might cause. So the training time was changed. It was conducted for ten days in January when schools had break. The problem we were confronted was the next training time would fall in the teaching time. Taking 10 days of teaching in self-contained classes in a 30 period week would mean a lot. We are not losing one subject teacher but teacher of all subjects in a section. This would have a negative impact on the quality of education. Intermediate Result 3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aid 3. 1 Technology Use & Multimedia Teaching Aids READ TA developed multimedia teaching aids and technologies for reading instruction in line with the revised curriculum. The multimedia teaching aids and technologies include model videos for teachers, audio companions, and software in support of MT reading and writing instruction for grades 1-8 in all seven languages. The teaching aid and technologies are intended for use at the Colleges of Teacher Education. The experts involved in the development of the multimedia teaching aids and technology are aware of READ TA’s development of multimedia aids and technologies. Intermediate Result 4: Technical Support to RSEBs and MOE for READ Institutional Improvement 4.1 READ TA’s support of the READ Institutional Improvement program Respondents from READ TA indicate that READ TA’s mandate was to support READ II to plan the initial round of teacher training. READ II received a single grant from USAID to MOE to train 64,000 teachers. READT TA helped READ II plan the initial round of teacher training. Accordingly, many of the respondents are unaware of READ TA’s support of READ II. One of the few comments received is form a respondent in the Tigray regional READ TA office who states: Read TA’s focuses in supporting READ II are: (1) in the development of materials, (2) selection and assignment of human power, and (3) making the necessary arrangements for training- make preparations to start training… 35 Other respondents explain that READ II phased out and READ TA commenced from where READ II stopped. Respondents from the Amhara and Wolaytatto language groups explain: The activities of READ II were largely on training of teachers. But now READ TA has got grant for the activity that was previously done by READ II. READ II is a project. When it accomplishes its mission either it proceeds to other issues or it phases out. For example, READ TA is a project and when it accomplishes its mission it would hand over the things to regional educational office. Although, very few respondents commented on READ TA’s support of the READ II project, the consensus among those respondents who commented is that READ II preceded READ TA on a continuum of intervention focusing on MT instructional material development and training on the developed instruments. 4.2 READ TA Influence on Curriculum Development Capacity READ TA, under the leadership of Florida State University (FSU) and in collaboration with MOE, RSEB, Zones and CTEs, led the development of four modules adapted into the seven MT languages. Training workshops were held to prepare MT pre-service teacher educators to teach and implement each of the four modules at CTEs. The development and implementation of the course syllabi aims to build the capacity of MOE, RSEBs and CTEs to support initiatives to improve the quality of early grade reading in MT languages. The courses and instructional approaches are in alignment with the newly revised primary school curricula. Effective implementation, including design and delivery, of these courses are assumed adequate in preparing primary school teachers to teach reading and writing in mother tongue in Grades 1-8. Throughout this process, MoE, RSEB and CTEs professionals realized opportunities for curriculum development capacity building. A respondent from the READ TA office in Amhara region states: One of the capacity building and success is that the RSEB has got opportunity to scale up the new materials and pedagogy to all other mother tongues in the region that are not covered by this project. Furthermore, experts at woreda, zone, and RSEB have increased their capacity of curriculum development in general and that of material development in particular. Respondents explain the experiences garnered doing this project may be capitalized on in further adaptation of the instructional materials into other languages as well as in future curriculum development. A respondent from the READ TA Sidama and Wolayta regional office states: At the preparation time it helps the experts to develop their capacity. Other experts also develop their capacity when they go through validation process... From each preparation and experience sharing activities professionals of Bureau and zones have created curriculum experts. They produced human power. Now if the region wants to prepare curriculum, those experts can be used. I have been raising this; even if READ TA completes its mission and goes there are numerous experts that were trained in this process that will serve for similar works in the future. There are for about 30 experts in Wolaytatto; 30 in Hadiyissa; and 30 in Sidama Afo. There is a possibility to do the same in other language groups using the materials 36 produced. The other one is the documents that we have produced and submitted to the region are highly valuable to the region. There are teaching learning materials, guidelines developed to write these materials and other supporting documents that were provided during the process; all these are assets for the region. All these materials are well compiled and would show the way for any future activity. Similarly, respondents from CTEs in Hadiya and Tigray explain: Well I think the main support is a support for the development of knowledge and skills. They have provided us with training when we participated in the material development. As I said earlier, starting from development to layout of materials were done by people from our RSEB and from outside, from colleges also, they were introduced to new technologies in developing materials and training. I would say there was transfer of capacity in developing materials. In addition, it has been done in close collaboration. Universities have also cooperated with our RSEB- they supported us by assigning their teachers and professional to work with us- this has helped us develop our capacities. Many of the respondents praise the work by READ TA on curriculum development capacity of the MOE, RSEB, Zones and CTEs. Respondents explain the potential to capitalize on the trainings and knowledge/skill obtained throughout the curriculum development process in further adaptation of the instructional materials into other languages as well as in future curriculum development work. 4.3 Gender Sensitivity & Disability Inclusion in READ TA Respondents predominantly expressed that they believed gender and inclusion were given appropriate attention. Gender and Inclusion are two of the crosscutting issues significantly considered throughout the READ TA project. In recruitment of professionals to staff the READ TA project, efforts were made to recruit women and individuals with disabilities. Inclusive and Special Needs Agencies were provided with advertisements for all positions5 . Measures were taken to assure the representation of women in the instructional material development process. As the following respondents from the Amhara and Oromia language groups explain: Yes this issue was given attention. There were focal persons from the part of the RSEB who oversee the inclusion of these issue. Females were made to participate in the development process. Materials were evaluated and checked that they have properly addressed gender sensitivity and equity, and disability issues. Furthermore, the developed MT instructional materials are reflective of gender and disability inclusion. A respondent from the regional Oromia READ TA office articulates: Starting from the preparation of the template, focus was given to gender and disability issues. Care was taken in making everything gender sensitive. This was reflected in the 5 READ TA Year 3 Quarterly Report (October – December 2014) 37 topic, in writing the texts, in the use of vocabularies, illustrations and activities. … Similarly, focus was given for disabilities. Derogatory words were avoided. The role of children with disabilities in the society was emphasized. Besides, in the front matters of the teachers’ guide how to assist and provide for individual difference was emphasized. In line with the above, a zonal expert from the Oromia language group explains: Yes our activities and support include gender and disability issues. These are considered as cross cutting issues. There are experts on gender and special needs education at all levels of the project from center to the regional office. Throughout the project cycle, READ TA has given sufficient consideration to gender and inclusion as crosscutting issues; as iterated by the respondent who have predominantly expressed positive views on READ TA’s efforts. 38 CONCLUSIONS READ M&E conducted this mid-term performance evaluation to provide external accountability and examine the major themes of READ TA’s work to date. There are nine evaluation questions under four intermediary results (IR1 to IR4). The results of the evaluation show that READ TA has followed exemplary procedures for development of the new mother tongue instructional materials in collaboration with all stakeholders. Additionally, there are a number of specific findings worth mentioning under each intermediate result. 4.1 IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In-Service Teacher Training Developed READ TA followed rigorous procedures for developing the new mother tongue instructional materials and has successfully completed the development of student textbooks and teacher’s guides for grades 1 – 4 based on the National MT Syllabus and the MLCs. The development of student textbooks and teacher’s guides is followed by a series of adaptation works. READ TA has successfully adapted the new mother tongue instructional materials for 12 language communities. This has helped READ TA to reach language communities, which were not originally included in the READ programs. Despite the reported initial challenges to coordination, there is consensus that the coordination and collaboration among stakeholders is exemplary. All parties involved learned from the way READ TA coordinated and guided activities through proper delegation of tasks. The coordination undertaken by READ TA to produce the new MT instructional materials facilitated “knowledge/skill transmission”, “commitment”, “collaboration”, “consultation”, and “sense of ownership.” The technological support provided by READ TA for CTEs has transformed the pre-service training of mother tongue language teachers. The technical assistance is related to the development of the new CTE MT Curriculum and modules along with capacity building training for teachers and technological assistance through the provision of ICT packages. There are concerns regarding the alignment of the syllabus and MT instructional materials with the national MLCs and the approach followed to check for alignment. 4.2 IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are Applied The procedures followed to develop the new mother tongue instructional materials resulted in new high quality mother tongue instructional materials completely different from the previous materials. Generally, teachers were found to be highly satisfied with the content, embedded methodology, and the physical qualities of the new mother tongue instructional materials. However, there is a consensus among teachers that the contents of a lesson for a day cannot be covered within the time allotted. Hence, many teachers tend to rush to finish the daily lesson. Data suggests that the size of the textbooks is not appropriate for the youngest age group. 39 Although there are concerns regarding the number of females and persons with disabilities recruited as master trainers, the majority of experts mentioned that READ TA’s predetermined criteria for master trainers selection resulted in the recruitment of high caliber candidates. Throughout the project cycle, READ TA has given sufficient consideration to gender and inclusion as crosscutting issues. 4.3 IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids READ TA provided technology and technological instruction to the CTEs to improve MT instruction. 4.4 IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement Many of the respondents praised the work done by READ TA on curriculum review and material development capacity building of the MOE, RSEB, Zones and CTEs. RECOMMENDATIONS Based on the conclusions drawn from the findings, the following recommendations may be provided for concerned stakeholders. 5.1 Recommendations for USAID, MoE, and RSEB Recommendation #1: READ M&E recommends that the procedures followed for developing the new mother tongue instructional materials be taken as a model for developing other instructional materials at the national level. Hence, USAID and MoE should use such approach in similar future projects. Recommendation #2: The pedagogy embedded in the materials for organizing lessons and delivering the contents should be taken as ‘standard’ or ‘best practice’ while developing instructional materials for other courses at the national level. Recommendation #3: There are concerns related to the alignment of the new mother tongue instructional materials with MLCs. Hence, a comprehensive and objective alignment and content analysis study should be conducted. Recommendation #4: Support provided to CTEs has received favorable feedback from the beneficiaries. Regional governments, RSEBs and CTEs should provide more training on the modules and provide more tech resources. They should strive towards providing quality training following the approach introduced by READ TA. Recommendation #5: READ M&E recommends that the contents of the lessons should match the time allotted. Hence, it would be more appropriate if measures are taken to review the materials and actual classroom practice so that the time allotted is suited for the content of the daily lesson. Recommendation #6: While revising the textbooks let the publishing company be advised to print the textbooks with A4 size so that the textbooks for the younger learners fit the locally prepared bag students’ use for carrying the books particularly in the rural area. 40 5.2 Recommendations for READ TA Recommendation #7: Lessons on coordination should be capitalized on for future projects and READ TA should prepare lessons learned/best practice guiding document on coordination. Recommendation #8: READ TA should continue the use of their established criteria to ensure the selection of high caliber candidates for any kind of training it is conducting. However, READ TA should use all possible strategies to include female applicants and persons with disabilities for similar positions without compromising the level of expertise needed. Recommendation #9: READ TA should develop lessons learned/ best practice guiding document on the provision of support provided and capacity building activities. Recommendation #10: Supporting classroom instruction with multimedia teaching aids would improve the quality of instruction and retain the attention of the students. It is recommended that the production of such teaching aids be preceded by a thorough examination of the classroom settings. 41 42 ANNEXES Annex I: Findings Conclusions and Recommendations The following is a summary chart of the salient qualitative and quantitative findings and related conclusion and recommendations: Finding Conclusion Recommendation IR1: Reading and Writing Materials Appropriate for Primary Classrooms and Pre-Service and In-Service Teacher Training Developed  READ TA has documented review of the previous language learning materials for purpose of identifying gaps in the reading curriculum including content, pedagogy, and assessment;  Mother Tongue Task Group was established to review the Mother Tongue Syllabus and the Minimum Learning Competencies (MLCs) and develop the revised National MT Syllabus.  The National Mother Tongue Consultative Workshop was held in Adama from June 19- 21, 2013 to validate the newly revised National MT Syllabus for grades 1-8.  READ TA also engaged in adaptation of student textbooks and teacher guides for grades 1-4.  The adaptation work has been completed for 12 language communities  READ TA followed set procedures for reviewing the existing curriculum and developed the National MT Syllabus for grades 1-8 based on the MLCs.  READ TA has successfully completed the development of student textbooks and teacher’s guides for grades 1 – 8.  The adaptation work has been completed.  READ M&E recommends that the procedures followed for developing the new mother tongue instructional materials be a model for developing other instructional materials at the national level.  The methodology embedded in the materials for organizing lessons and delivering the contents should be taken as ‘standard’ or ‘best practice’ for developing instructional materials for other courses at the national level.  In guiding the coordination of MT instructional material development, READ TA prepared different documents and established task forces.  Despite the reported initial challenges to coordination, there is consensus that the coordination among stakeholders facilitated “knowledge/skill transmission”,  Lessons on coordination should be capitalized on for future projects and READ TA should prepare lessons learned/best practice guiding document on coordination. 43  Overall, there was good coordination among various stakeholders in terms of managing activities related to review of the curriculum and the development of the new mother tongue instructional materials.  Despite these mechanisms for managing coordination, unanticipated issues are acknowledged to have caused minor challenges, primarily at the regional level. “commitment”, “collaboration”, “consultation”, and “sense of ownership” which resulted in development of the new MT instructional materials  Trainings were organized prior to commencement of material development for material developers on how to use various documents like MLCs and syllabus for guiding their material development effort  Validation checks are reported to have taken place after the development of materials  However, conflicting responses were provided by the respondents regarding the alignment of the MT syllabus, students’ textbook and teachers’ guide with the national Minimum Learning Competency (MLC)  Overall, there is a tendency among respondents to view the alignment positively  There are concerns regarding the alignment of the syllabus and MT instructional materials with the national MLCs and the approach followed to check for alignment is not appropriate.  Comprehensive and relatively objective approach should be followed to conduct alignment study including content analysis  In efforts to ensure effective pre-service training in teaching primary reading and writing, READ TA has supported CTEs in the development of the new CTE MT Curriculum and modules  READ TA has also trained mother tongue language teachers in the CTEs on the modules developed and on the use of the new teaching method supported by ICT materials  Generally favorable responses are received from respondents regarding the support provided by READ TA for CTEs.  The support provided is classified into two categories, technical and technological. o Technical assistance is related to the development of the new  Support provided to CTEs is among the services rendered by READ TA that has received more favorable feedback from the beneficiaries. Training on the modules and the expansion and provision of technological resources for improving the training of mother tongue language teachers should be handled by the government and CTEs should strive towards providing quality training following the approach introduced by READ TA. 44  Additional trainings were organized on use and maintenance of the ICT packages to the ICT teams of both the RSEBs and CTEs and on the use and maintenance of ICT materials.  Additionally, READ TA provided CTEs with ICT packages which consist of a tablet for each mother tongue language lecturer, a server, projector, speakers, and documentation camera for each CTE.  However, there are concerns among some of the CTEs staff that there are modules that have not yet been developed and trainees graduate this year without learning with the new modules  The facilities that are made available are not adequate enough for the number of trainees CTEs have in their pre-service programs CTE MT Curriculum and modules along with capacity building training for teachers. o Technological assistance is related to the provision of ICT packages. IR2: Language-Specific Teaching and Learning Methodologies and Strategies that Focus on Helping Students Learn to Read and Write Effectively are Applied  The majority of respondents agreed that the newly developed READ TA materials were an improvement from their old instructional materials despite varying degrees of overall satisfaction.  Teachers often mentioned that the newly developed materials were better at incorporating mother tongue into the lessons.  However, teachers frequently mentioned that it was difficult to cover the vast content of the materials in the given instructional time.  Additionally, teachers frequently mentioned  Largely, teachers were found to be highly satisfied with the contents, methodology embedded, and the physical qualities of the new mother tongue instructional materials.  However, the contents of a lesson for a day cannot be covered with the time allotted. Hence, many teachers tend to rush at about the end of the lesson. READ M&E recommends that the contents of the lesson should match the time allotted. Hence, it would be more appropriate if measures are taken to balance the time allotted with the content of the daily lesson. While revising the textbooks, the publishing company may be advised to print the textbooks with A4 size so that the textbooks fit the locally prepared ‘bag’ student’s use for carrying the books particularly in the rural area. 45 that the textbook size created issues for some students. The size is big for children to keep it safe and carry it to the class every time.  The majority of teachers mentioned that the use of language-specific teaching and learning strategies were very “helpful.” Several teachers stated that the division of materials into lessons of specific days and weeks made “implementation more effective and convenient.”  Respondents also cited that the new methods were more “interactive” and helped to keep students’ attention.  It appeared from the data that the size of the textbooks is not appropriate for the age and developmental level of the learners.  According to respondents the criteria for selecting master trainers included, “gender issues; knowledge of the language and culture; candidates’ capacity, and experience to give training.”  Though “the plan was to select an equal number of males and females” to hold master trainer positions this was not always possible due to low number of qualified female applicants for the post. The majority of experts mentioned that READ TA’s predetermined criteria for master trainer selection resulted in high caliber candidates.  READ TA should continue the use of their established criteria in order to ensure high caliber candidates for master trainers. However, READ TA should use all possible strategies to include female applicants for similar positions without compromising the level of expertise needed.  Lessons learned/ best practice documentation is recommended IR3: Language Teaching and Learning Supported by Appropriate Technology and Teacher Aids READ TA undertook development of multimedia teaching aids and technologies for reading instruction in line with the revised MT curriculum. The multimedia teaching aids and technologies include model videos for teacher, audio There is no provision of multimedia teaching aids/technologies for primary school teachers for supporting their classroom instruction Most primary school teachers, Supporting classroom instruction with multimedia teaching aids would improve the quality of instruction and retain the attention of the students. It is recommended that the production of such teaching aids be preceded by a thorough examination of the classroom settings. 46 companions and software in support of MT reading and writing instruction for grades 1-8 in all seven languages. Although experts involved in the development of the multimedia teaching aids and technology are aware of READ TA’s development of multimedia aids and technologies, the intended users, primary school teachers, are largely unaware of the development of these aids. principals and supervisors are unaware of any multimedia and technological teaching aids for use in primary classroom settings. IR4: Technical Support to RSEBs and MOE for the READ Institutional Improvement Very few comments on READ TA’s support of the READ II project are received from respondents. Many of the respondents are unaware of READ TA’s support of READ II. There is a consensus among the respondents that READ II preceded READ TA on a continuum of intervention focusing on the training if MT language teachers. The development and implementation of the project resulted in building the capacity of MOE, RSEBs and CTEs in terms of producing local experts who can handle curriculum review process and material development. The courses and instructional approaches in CTEs are in alignment with the newly revised primary school curricula. Respondents explain the potential to capitalize on the trainings and knowledge/skill obtained throughout the curriculum and material development process and in further adaptation of the instructional materials into other languages as well as in future curriculum development work. Many of the respondents praised the work that has been done by READ TA on curriculum review and material development capacity building of the MOE, RSEB, Zones and CTEs. Lessons learned/ best practice documentation is recommended. Respondents predominantly expressed that they Throughout the project cycle, Lessons learned/ best practice documentation is 47 believed gender and inclusion were given appropriate attention. READ TA has given sufficient consideration to gender and inclusion as crosscutting issues. recommended. Annex 2: Sample Size by region and language Region Zone Woreda School Sample Size Language of CTE School Material Developers Experts from RSEB, Zones, Woredas Principals Classroom Observations Teachers Cluster Supervisors Amhara 2 4 8 1 2 6+ 8 (3Int & 5Q) 8 32 (8Int & 24Q) 4 Amharic Oromia 2 4 8 1 2 6+ 8 (3Int & 5Q) 8 32 (8Int & 24Q) 4 Afan Oromo Tigray 2 2 6 1 2 6 6 (2Int & 4Q) 6 24 (6Int &18Q) 2 Tigrigna SNPP 1 2 6 1 2 6 6 (2Int & 4Q) 6 24 (6Int &18Q) 2 Sidamu Afoo 1 2 6 1 2 6 6 (2Int & 4Q) 6 24 (6Int &18Q) 2 Wolyttatto 1 2 6 1 2 6 6 (2Int & 4Q) 6 24 (6Int &18Q) 2 Hadiyissa Somali 2 2 6 1 2 6 6 (2Int & 4Q) 6 24 (6Int &18Q) 2 Af-Somali Total 11 18 46 7 14 42 48 46 184 18 48 49 Annex 3: Data Collection Instruments Interview Guide for Principals and Supervisors Procedures for informed consent Verbal consent will be obtained from participants. Materials include Interviewer clock Voice recorder Checklist of procedures Obtain verbal consent from participant. Conduct interview. Debrief following each interview. Please type up notes in tables! Interviewer’s name Interview’s Code Region Language Group Cell phone of the interviewer Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting mid-term performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@gmail.com. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202- 403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. So I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today? 50  Option 1: If the participant does not grant permission, end here.  Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you! Introductory Questions Please tell us your name and your favorite pastime. Demographic Data Code of participant Age Gender Position Principal Supervisor Years of service Level of education Region Zone Woreda Name of school Date of Interview Time of interview Usefulness and Quality of Teaching Materials 1. Would you please tell us about the presence of the student textbooks and teacher’s guide in your school? Probe: the adequacy of the number of student textbooks and teacher’s guide. 2. Have you started using the new mother tongue language student textbooks and teacher’s guides for teaching mother tongue language in your school? 2.1 When did you start teaching mother tongue language with the new materials? 3. Do you perceive the materials as useful tools for instruction? 3.1 Do all mother tongue language teachers in your school currently use the materials for instruction? 3.2 What proportion of students have the new textbooks? 3.3 What are the challenges of using or implementing the new student textbooks and teacher’s guide? 4. As principal/cluster supervisor of this school, have you got the chance to go through the new student textbooks and/or teacher’s guides or at least discussed about these materials with mother tongue language teachers? 4.1 If yes, would you please briefly explain the strength and weakness of the new student textbooks and teacher guides? 5. Are you satisfied with the overall nature of the new student textbooks and teacher guides developed by READ TA? 51 Language Specific Teaching Methodology 6. Would you please describe if the new materials have helped teachers in your school in changing their method of teaching? 7. In what ways, if any, have the newly developed student textbooks and teachers guide enhanced reading and writing skills of students in your school? 8. Have you seen any difference in the progress of students in your school in regards to learning reading in mother tongue language quicker/slower than students in the previous years? 9. Can you please speak about any new uses of technology in your school? [Probes: Where did they come from? When did you receive them? Are they helpful?] Overall Impressions about the Materials Developed by READ TA 10. What is your overall impression about the materials developed (student textbooks, teacher’s guide, and the CDs)? 11. If there is another chance of developing these materials, what aspects of the materials do you think should be improved? Why? 12. Is there anything that you want to add? Closing Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to:  What worked and did not work  Lessons learned  Issues to address in future interviews  Issues to follow up on with the research team or client  Key points to note in data analysis Please type up notes as soon as possible after interviews! 52 Interview Guide for Experts from MoE, RSEBs and READ TA Procedures for informed consent Verbal consent will be obtained from participants. Materials include Interviewer clock Voice recorder Checklist of procedures Obtain verbal consent from participant. Conduct interview. Debrief following each interview. Please type up notes in tables! Interviewer’s name Interview’s Code Region Language Group Cell phone of the interviewer Background and Informed Consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting mid-term performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@gmail.com. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202- 403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. So I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today?  Option 1: If the participant does not grant permission, end here.  Option 2: If the participant grants permission, please proceed to the next section and begin the interview. 53 Thank you Demographic Data Code of participant Age Gender Current position Work experience in years Region (If applicable) Organization where you are working MoE RSEB READ TA Date of Interview Time of interview Introductory Questions Please tell us your name and your favorite pastime. Status of Curriculum Development 1. What was your role with curriculum development? 2. Would you please describe what has been completed on the new mother tongue materials? 2.1.What are the major achievements? 2.2.What have been the major challenges? 3. What is it that has not been done/achieved as per the Implementation Plan? 3.1.Do you have any thoughts on how this occurred? 3.2.What did you put in place, as a solution, to ensure that activities move as per the Implementation Plan? 4. Do you think that the newly developed mother tongue curricula have adequately addressed the problems with the previous curricula? Can you give us an example? 5. What is the most important thing you have learned from the process? Level of Coordination 6. How was the coordination between READ TA and the MoE and RSEBs for material development and adaptation? 6.1.What is the status of the development and regional adaptation of student textbooks and teacher guides? 6.2.Has regional adaptation of student textbooks been completed in all mother tongue languages as planned? 6.3.What was the role and involvement of MoE and RSEBs in the development and adaptation of mother tongue textbooks, and other related modules? 6.4.Was there any agreed upon mechanism to guide the coordination/collaboration activities? 6.4.1.If yes, would you briefly explain about that? If no, what was the coordination like between and among the three partners? 54 6.4.2.Did you encounter any problem(s) in due course (of the coordination/collaboration)? If yes, what were the challenges and how did you resolve them? 6.4.3.What are the lessons that can be drawn from this coordination/collaboration? Quality of Materials Developed 7. How would you describe the level of alignment between the textbooks and teacher guides with the MLCs? Please describe the alignment for each grade separately, as you are able. 7.1.How do you rank the level of alignment on a three points scale: 3 - Very Strong, 2 - Moderately Strong, 3 – Weak? 7.2.Can you give us an example? 8. How was the appropriateness of vocabulary, readability, complexity of structure, etc. of the materials determined during the development of mother tongue student textbooks and teacher guides? 8.1.Do the newly developed mother tongue student textbooks and teacher guides address the gaps identified in the previous learning and teaching materials? Technology and Teacher Aids 9. What type of appropriate technology and teacher aids has READ TA made available to enhance mother tongue language teaching? 9.1.What multimedia teaching aids are currently being developed, piloted, and or deployed to support pre-service and in-service teacher development as well as use in classrooms to support MT instruction? 9.2.What are the strengths and weaknesses of the multimedia teaching aids developed by READ TA (if any)? 9.3.What are the technology types, costs, and implementation and sustainability requirements being proposed for use by CTEs, RSEBs, and teachers? 9.4.Have all the seven mother tongue language groups received and used the appropriate technology and teaching aids made available by READ TA? If no, please describe the reason. 9.5.Are you satisfied with what has been achieved with regard to the use of appropriate technology and teaching aids? Training of Master Trainers 10. How was the selection of Master trainers conducted? [Probe: Who and how was it decided who to select and invite?] 10.1. Were there any criteria used in the selection of Master Trainers? 10.1.1. Who developed these criteria? 10.1.2. Was there agreement among MoE, RSEBs, and READ TA on the criteria developed? 10.1.3. What were the criteria? 10.2. Do you think that the selection of Master Trainers was based on the criteria developed? 10.3. How would you describe the input of each partner in this undertaking? 10.4. What is the gender mix like in the selection of Master trainers? 10.5. What are the strengths and weaknesses of the selection process? 10.6. How was the training of Master Trainers? [Probe: What went wrong in the training of Master Trainers?] Support Made Available 11. [Only READ TA Staff] What types of support has READ TA made available to CTEs? Please describe in detail the type of support provided to CTEs for the development of pre-service and in￾service training materials for Mother Tongue Language teaching/instruction? 55 11.1. How would you describe the relevance of the support provided by READ TA to CTEs? 11.2. How often do CTE’s relevant staff and READ TA’s staff hold meetings to discuss on matters related to the support? 11.3. What challenges emerged in your work with CTEs? How were they resolved? 11.4. How satisfied were you with your relationships with CTEs and the work they did? 11.5. What suggestions do you have for an overall improvement of the support that READ TA makes available to CTEs? 12. In more specific terms what were READ TA’s focuses in supporting READ II programs? Please give examples. 12.1. Are the MoE and RSEBs satisfied with the support provided? 13. What capacity building objectives for the MoE and RSEBs have been realized as a result READ TA’s initiative? 13.1. What capacity building activities have been carried out by READ TA? 13.2. What activities were planned and what have been achieved so far? 13.3. What are the specific capacities that have been developed? 13.4. Have the capacities of MoE and the RSEBs been built to produce mother tongue language reading curriculum? 13.5. What further capacity building is planned? 13.6. Has the capacity building effort included ICT applications to support reading and writing? If yes, how? What is planned? Gender and Disability Issues 14. Has READ TA included gender sensitivity and disability issues in its program? Where it is that issue of gender sensitivity and disability are reflected? CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to:  What worked and did not work  Lessons learned  Issues to address in future interviews  Issues to follow up on with the research team or client  Key points to note in data analysis Please type up notes as soon as possible after interview 56 Interview Guide for Teachers Procedures for informed consent Verbal consent will be obtained from participants. Materials include Interviewer clock Voice recorder Checklist of procedures Obtain verbal consent from participant. Conduct interview. Debrief following each interview. Please type up notes in tables! Interviewer’s name Interview’s Code Region Language Group Cell phone of the interviewer BACKGROUND AND INFORMED CONSENT Background and informed consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting mid-term performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@gmail.com. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202- 403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 60 minutes. We have a lot of questions to ask, but we want to respect your time. So I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today?  Option 1: If the participant does not grant permission, end here. 57  Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you! INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. DEMOGRAPHIC DATA Code of participant Age Gender Years of service in teaching Years of service teaching mother tongue language Level of education Region Zone Woreda Name of school Date of Interview Time of interview AVAILABILITY OF THE NEWLY DEVELOPED MATERIALS 1. Are the newly developed student textbook and teacher guide available to you at the school you teach? 12.1 If not, do you know why the textbooks are not available to you? 12.2 If yes, when did the textbooks and teacher guides become available? 12.3 If yes, what is the student to new text book ratio in your school/class room? 12.4 Is this ratio reasonably adequate? USEFULNESS AND QUALITY OF TEACHING MATERIALS 13. Do you perceive the materials as useful tools for instruction? 13.1 Are you currently using the materials for instruction? What do you find most important about this text book and teacher guides? 13.2 Are the students using the textbooks? 13.3 Do students bring textbooks to the class whenever they have mother tongue language class? What proportion of students brings the textbooks when there is a mother tongue language class? 13.4 What are the challenges of using or implementing the new student textbooks and teacher’s guide? 14. Are the content, breadth, and depth of the student textbooks and teacher guides appropriate to the grade level you teach? 14.1 Are the mother tongue learning materials culturally appropriate? 14.2 Are the illustrations in the mother tongue learning materials relevant to the text? 14.3 What do teachers find most useful about these textbooks? 58 14.4 What do teachers find least useful about these textbooks? 14.5 What, in your view, is missing or hasn’t been given due attention in the course of developing the textbooks and the teacher guides? 15. Would you please briefly explain the strength and weakness of the new student textbooks and teacher guides? 16. Are you satisfied with the overall nature of the new student textbooks and teacher guides? LANGUAGE SPECIFIC TEACHING METHODOLOGY 17. Are the methodologies and strategies for language specific teaching depicted in the textbooks and teacher guides helpful in teaching students to learn to read and write effectively? 17.1 In what ways, if any, has the newly developed materials changed your teaching methodology? Please give example. 17.2 In what ways, if any, have the newly developed student textbooks and teachers guide enhanced reading and writing skills of students? 17.3 How do you view the content of the teacher’s guide and students’ textbooks in terms of language use and methods of presentation? 17.4 In what ways, if any, have the newly developed mother tongue textbooks and teacher’s guide helped you in changing your teaching methodology and strategies? If yes, would you please elaborate? [Probe for I do, We do, You do approach] 17.5 In what ways, if any, have the new materials engaged your students? 17.6 Have you seen any difference in the progress of your students in regards to learning reading in mother tongue language quicker/slower than previous? 17.7 What is your overall perception of the newly developed student textbooks and teacher’s guide prepared by READ TA? 18. Can you please speak about any new uses of technology in your class? [Probes: Where did they come from? When did you receive them? Are they helpful?] OVERALL IMPRESSIONS ABOUT THE MATERIALS DEVELOPED BY READ TA 19. What is your overall impression about the materials developed (student textbooks, teacher’s guide, and the CDs)? 19.1 If there is another chance of developing these materials, what aspects of the materials do you think should be improved? Why? 19.2 Is there anything that you want to add? CLOSING Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to:  What worked and did not work  Lessons learned  Issues to address in future interviews  Issues to follow up on with the research team or client  Key points to note in data analysis 59 Interview Guide for Material Developers Procedures for informed consent Verbal consent will be obtained from participants. Materials include Interviewer clock Voice recorder Checklist of procedures Obtain verbal consent from participant. Conduct interview. Debrief following each interview. Please type up notes in tables! Interviewer’s name Interview’s Code Language Group Cell phone of the interviewer BACKGROUND AND INFORMED CONSENT Background and informed consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting mid-term performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TAin collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@gmail.com. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202- 403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45minutes. We have a lot of questions to ask, but we want to respect your time. So I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today?  Option 1: If the participant does not grant permission, end here.  Option 2: If the participant grants permission, please proceed to the next section and begin the interview. 60 Thank you DEMOGRAPHIC DATA Code of participant Age Gender Educational background Occupation Years of service Name of the Organization/Institution where you work Position within organization Date of interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. KNOWLEDGE OF MLCS AND ALIGNMENT 1. My next questions will be about the knowledge you have about the MLCs. Before the start of material development, did you receive any training and/or orientation about MLCs? 1.1. If yes, for how long? 1.2. By whom, where, and when? 1.3. If yes, please tell me about the training and/or orientation. In which ways was it helpful? Did you have a copy of the MLCs when developing the mother tongue textbooks and teacher guides? Did you refer to the MLCs? 2. How would you describe the level of alignment between the produced textbooks and teacher’s guides with the MLCs? Please explain for each grade that you are familiar with. 2.1. How do you rank the level of alignment on a five points scale: 5 – Very Strong, 4 – Somewhat Strong, 3 – Neutral, 2- Somewhat Weak, and 1- Very Weak? PROCESS OF MATERIAL DEVELOPMENT 3. How was the collaboration between READ TA, MoE, and RSEBs in developing the student textbooks and teacher guides? [Probe: Was there collaboration? Why, why not?] QUALITY OF THE MATERIAL DEVELOPED 4. How was the appropriateness of vocabulary, readability, complexity of structure, etc. of the materials determined during the development of mother tongue student textbooks and teacher guides? 5. Has there been sufficient representation of women thorough-out mother tongue material development process? Were women involved in the development? Are women depicted in the illustrations and content of the books? If it is insufficient, do you have any idea of what the barriers were? 61 OPPORTUNITY FOR CAPACITY BUILDING 6. What do you think were the major opportunities for capacity building during the process of material development (capacity of RSEBs, FMOE, and CTEs Zones)? 7. As material developer, what is the most important part of this process in terms of your own learning? CHALLENGES ENCOUNTERED AND RECOMMENDATIONS FOR FURTHER IMPROVEMENT 8. What do you think were the challenges during the material development process? 9. If you are given an opportunity to revise the materials developed which part of the materials do you think would be critically revised? Why? CLOSING Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to:  What worked and did not work  Lessons learned  Issues to address in future interviews  Issues to follow up on with the research team or client  Key points to note in data analysis Please type up notes as soon as possible after interviews! 62 Interview Guide for CTE Staff Procedures for informed consent Verbal consent will be obtained from participants. Materials include Interviewer clock Voice recorder Checklist of procedures Obtain verbal consent from participant. Conduct interview. Debrief following each interview. Please type up notes in tables! Interviewer’s name Interview’s Code Language Group Cell phone of the interviewer BACKGROUND AND INFORMED CONSENT Background and informed consent My name is (name of the interviewer) and I’ll be talking with you today. Also here with us today is (name of the note taker), who will be taking notes. We work for an independent nonprofit research organization called American Institutes for Research (AIR). We are conducting mid-term performance evaluation of the project undertaken by Reading for Ethiopia’s Achievement Developed Technical Assistance (READ TA). The purpose of today’s interview is to learn about the various activities carried out by READ TA in collaboration with the MoE and RSEBs. We will be recording this conversation to help capture what you say accurately. All information collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this discussion will be shared with USAID and the MoE, no individual will be identified in any report. Participation in this discussion is voluntary and you do not have to answer any questions you do not want to answer. If you have questions about the discussion, please contact Daniel Tefera - the principal investigator at 0911 663402. You can also email Daniel Tefera at: dtefera@gmail.com. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202- 403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. The interview will last approximately 45minutes. We have a lot of questions to ask, but we want to respect your time. So I may ask you to move to another topic at some point in the discussion. Please let us know if you have any time constraints. Do you have any questions? Do you agree to participate today?  Option 1: If the participant does not grant permission, end here. 63  Option 2: If the participant grants permission, please proceed to the next section and begin the interview. Thank you DEMOGRAPHIC DATA Code of participant Age Gender Educational background Occupation Years of service Name of the CTE Your role in the project (Coordinator/ Participant) Date of interview Time of interview INTRODUCTORY QUESTIONS Please tell us your name and your favorite pastime. NATURE OF SUPPORT PROVIDED 10. Are you aware of the support provided by READ TA to CTEs to develop materials for pre-service training for mother tongue language teachers at CTEs? 10.1. If you are aware of the support, what types of support did READ TA make available to CTEs? Please note the types of support that helped in developing the pre-service training materials for mother tongue language teachers (like, technical, budgetary/financial, material support etc.) 10.2. How often do READ TA and CTE relevant staff hold meetings to discuss on matters related to the support? 11. How would you describe the level of support you received from READ TA to develop pre-service training materials for mother tongue language teachers? [Strong, Weak, etc.] 11.1. What are the major contributions of READ TA support to the development of pre-service training materials for mother tongue language teachers? 11.2. What is your impression of the support provided by READ TA? On a scale of 1-4, how would your rank the support? (4 - Very Good, 3 - Good, 2 - Fair, 1 – poor) Please explain your reason for the choice. 11.3. What kind of support do you think has not been addressed very well? QUALITY OF THE MATERIAL DEVELOPED 12. Are you aware of the training modules developed by READ TA for pre-service training for mother tongue language teachers? If yes, how do you evaluate the modules in terms of: 12.1. Appropriateness to the level of teacher trainees? 64 12.2. Level of difficulty? 12.3. Cultural appropriateness? 12.4. Language use and readability? 12.5. Gender balance? 12.6. Addressing disability issues? 12.7. Continuity of lessons from one topic to the other? 12.8. Logical link among topics and sub-topics? CHALLENGES AND RECOMMENDATIONS 13. What were the challenges encountered while working with the READ TA? How were they resolved? 14. What suggestions do you have for an overall improvement of the support that READ TA makes available to CTEs? CLOSING Is there anything I did not ask about that you would like to share with me? Do you have any additional thoughts about what we have discussed today? Thank you very much for participating in this discussion today. Staff facilitating the interview will debrief following the interview to discuss issues including but not limited to:  What worked and did not work in the interview  Lessons learned about their interview  Issues to address in future interviews  Issues to follow up on with the research team or client  Key points to note in data analysis Please type up notes as soon as possible after interviews! 65 Questionnaire for Teachers The purpose of this questionnaire is to collect data for the mid-term performance evaluation of READ TA project funded by USAID and implemented by RTI. The READ TA project is providing technical assistance to support the development and implementation of mother tongue curricula for grades 1-8 in seven local languages in Ethiopia. All information that is collected will be treated confidentially. Your name will be kept private and separate from the evaluation. While results from this questionnaire will be shared with USAID and the MoE, no individual will be identified in any report. Participation in completing this questionnaire is voluntary. If you have questions about the questionnaire, please contact Daniel Tefera (the principal investigator of this evaluation) at 0911663402. You can also email the principal investigatorat:dtefera@read-me-air.org. If you have concerns or questions about your rights as a participant, contact the American Institutes for Research Institutional Review Board (which is responsible for the protection of project participants) at IRB@air.org, or +1 202-403-5542, or by postal mail: AIR c/o IRB, 1000 Thomas Jefferson Street, NW, Washington, DC 20007, USA. We are very much great full for your participation on this evaluation! Part I: Demographic Data Instructions: In this part of the questionnaire, you are asked to provide personal information. Read each question and circle the number of your choice or provide the information in writing on the space provided. Please note that any of your personal information will not be communicated to the third party in any way that your responses will be traced. 1. Region 1.Tigray 2. Amhara 3.Oromia 4.SNNP 5. Somali 2. Zone _________________________________________________________________ 3. Woreda ___________________________________________________________________ 4. Name of the School __________________________________________________________ 5. Age of the respondent_______________________________________________________ 6. Gender of the respondent_____________________________________________________ 7. Years of service as teacher ___________________________________________ 8. Years of service in teaching mother tongue language _______________________ 9. Level of education ___________________________________________________ Part II: Student Textbook and Teacher’s Guide Instructions: In this part of the questionnaire, you are asked to provide your responses to items related to the use of student textbooks and teacher’s guide. Read each question and circle the number of your choice or provide the information in writing on the space provided. 1 For the grades you teach, do your students have the new mother tongue textbooks? Grade 1: Yes No Grade 2: Yes No 66 Grade 3: Yes No Grade 4: Yes No 2 For these grades, does every student have his/her own textbook? Grade 1: Yes No Grade 2: Yes No Grade 3: Yes No Grade 4: Yes No 3 If no, what is the student: textbook ratio? Grade 1: ratio ______ student: _______textbook Grade 2: ratio ______ student: _______textbook Grade 3: ratio ______ student: _______textbook Grade 4: ratio ______ student: _______textbook 4 If no, what is the reason why every student does not have his/her own textbook, as you understand it? __________________________________________________________ 5 Are students using the new textbooks? Grade 1: Yes No Grade 2: Yes No Grade 3: Yes No Grade 4: Yes No 6 How often do students bring textbooks to the class? Grade 1: 1. Always (whenever they have class) 2. Sometimes 3. Rarely 4. Not at all Grade 2: 1. Always (whenever they have class) 2. Sometimes 3. Rarely 4. Not at all Grade 3: 1. Always (whenever they have class) 2. Sometimes 3. Rarely 4. Not at all Grade 4: 1. Always (whenever they have class) 2. Sometimes 3. Rarely 4. Not at all 7 If your school has the textbooks, when did the textbooks become available? Month_______________ Year_______________ 8 Are the newly developed teacher guides available to you at the school you teach? Grade 1: Yes No Grade 2: Yes No Grade 3: Yes No Grade 4: Yes No 9 How often do you use teacher’s guide? 1. Always 2. Sometimes 3. Rarely 4. Not at all 10 If your school has the teacher’s guide, when did the guide become available? Month_______________ Year__________________ 11 Do you have the syllabus for the new student textbooks and teacher’s guide? Grade 1: Yes No Grade 2: Yes No Grade 3: Yes No Grade 4: Yes No 12 Are there electronic teaching aids in your school? 1. Yes 2. No 3. Don’t know 67 13 If yes, from where does the school obtain these aids? _________________________ ____________________________________________________________________ 14 If yes, are they helpful in improving your method of teaching? 1. Yes 2. No Part III: Views about the Materials Developed Instructions: In this part of the questionnaire, there are statements about how the contents of student textbooks are organized, the use of language specific methodologies and the technical qualities of student textbooks and teacher’s guide. We expect you to rate these materials of a scale of satisfaction: 1 – Very dissatisfied • 2 – dissatisfied • 3 – unsure • 4 – satisfied • 5 – Very satisfied. For each statement given below, check the box to the right that best fits your ratings towards the aspect of the materials indicated in the statement. Put “X” under your choice. 15 For the next series of questions, please select one grade you teach with the new mother tongue textbook and answer the questions ONLY for that grade textbook. About which grade are you responding? Grade 1__ 2___ 3___ 4____ (select one) No. Items Very dissatisfi ed Dissatisfi ed Unsure Satisfied Very Satisfied 1 2 3 4 5 Textbook content : 16 Continuity of lessons from one topic to the other 17 Grade level appropriateness 18 Level of difficulty 19 Readability 20 Cultural appropriateness for your local context Relevance in terms of 21 representing both boys and girls 22 promoting life skills, such as healthy habits like washing hands after toilet, telling the truth, or good manner 23 Comprehensiveness How do you view the content of the teacher’s guide and students’ textbooks in terms of the use of language – specific teaching and learning methodologies and strategies? 24 The alignment between the student textbooks and teacher’s guide 25 The arrangement of topics from simple to complex 26 Exclusive use of mother tongue language in the textbook 27 Good use of illustrations 28 Providing opportunities for continuous performance assessment and assessing student progress 29 The teaching strategy in the teacher’s guide are designed to be interactive 30 The capacity to trigger interest in the learners 31 Pedagogical appropriateness of the sequence of objectives in the teacher’s guides 68 32 Strategies promoting active learning How do you view the physical qualities of student textbooks and teacher’s guides in terms of: 33 Quality of cover pages 34 Quality of the paper and bindings 35 Quality of print (including text and illustrations) Classroom Observation Checklist Instruction for Data Collector: The following checklist is what you’ll use while you observe four classes: one from each grade (Grade 1 to Grade 4). Please refer to the data collection manual for further guidance on the procedures to follow. Write the grade you are observing: Grade _____________________ Observer’s Name __________________________________________________________ Complete as much of the table as you are able before the class begins. Teacher Gender M ______ F _______ Region Zone Woreda Name of school Date of observation Time of observation Start: End: # of students in the classroom (measures attendance) Boys ______ Girls ____ # of student mother tongue textbooks in the class Boys ______ Girls ____ Teacher received training Yes ______ No ______ No. Description Yes No Remark 1. 1.1 Teacher brought the new mother tongue textbook to the classroom 1.2 Teacher brought the new mother tongue teacher’s guide to the classroom 1.3 Teacher uses new mother tongue textbook during instruction 2. The teacher refers/uses the teacher guide during instruction 3. Does the teacher engage males and females equally (engaged with the teacher- e.g. answering questions)? 4. Does the teacher use the “I do, we do, you do” methodology? 5. The degree of students’ use of the textbooks during instruction 1. High 2. Moderate 3. Low 6. There is interactive teaching and learning processes 1. Not interactive 2. Least interactive 3. Interactive 4. Highly interactive 7. Does the teacher use assessment strategies during instruction? (Does the teacher ask to do classroom activities and mark at the end of the class? etc.) 1. Not at all 2. Only Once 3. Twice 4. Three or more times 69