July 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Nancy Horn, Eric Allemano, Feleke Desta, Erango Ersado, Habtamu Mammo, and James Wile under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania TLMP in Ethiopia Intentionally left blank EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA TLMP IN ETHIOPIA Prepared by: Nancy Horn, Team Leader Eric Allemano, Team Member Feleke Desta, Team Member Erango Ersado, Team Member Habtamu Mammo, Team Member James Wile, Team Member July 18, 2013 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP – TLMP in Ethiopia i TABLE OF CONTENTS LIST OF ACRONYMS AND TERMS ............................................................................. iii EXECUTIVE SUMMARY................................................................................................. iv Evaluation Methodology ............................................................................................................................. v Findings ........................................................................................................................................................... v TLMP Output and Dissemination ........................................................................................................ v Project Management and Partnerships ............................................................................................... v Program Implementation ......................................................................................................................vi Lessons Learned.........................................................................................................................................vii Recommendations ....................................................................................................................................viii Materials Improvement .......................................................................................................................viii Systems Improvement .........................................................................................................................viii Towards a More Comprehensive Approach to Language Development..................................ix Linking TLMP to USAID Priorities in Early Grade Reading ..........................................................ix 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES.................................. 1 1.1 The Background to TLMP ................................................................................................................... 1 1.2 Evaluation Objectives........................................................................................................................... 1 2. TLMP ETHIOPIA PROJECT BACKGROUND ......................................................... 2 2.1 AAMU Background Research............................................................................................................. 2 2.2 MOE Context and the Educational System..................................................................................... 2 2.2.1 Teacher Training and the Languages of Instruction............................................................... 4 2.2.2 Systemic Textbook Issues............................................................................................................ 4 2.2.3 Issues Associated with Approaches to Teaching Literacy.................................................... 4 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS.................................. 5 3.1 Background Research Conducted on AAMU................................................................................. 5 3.2 Research Activities in Addis Ababa, Oromiya and Amhara Environs....................................... 5 3.3 Limitations of the Study ....................................................................................................................... 7 4. FINDINGS AND CONCLUSIONS ............................................................................ 8 4.1 AAMU Achievements and Challenges.............................................................................................. 8 4.2 TLMP Output......................................................................................................................................... 8 4.3 Project Management and Partnerships............................................................................................. 8 4.4 Project Implementation ................................................................................................................... 10 4.4.1 Materials Development............................................................................................................ 10 4.4.2 Assessment of TLMs by Evaluation Team............................................................................. 10 4.4.3 Comments on the TLMs Made by RSEB and Woreda Education Officials, Principals and Teachers.......................................................................................................................................... 11 4.4.4 Comments on Training Made by RSEBs and Woreda Education Officials, Principals, Teachers and Master Trainers........................................................................................................... 13 4.5 Outputs and Outcomes................................................................................................................... 14 4.5.1 Teacher Observations.............................................................................................................. 14 4.5.2 Learner Reading Assessments and Outcomes..................................................................... 15 4.6 Stakeholder Interest and Use of Materials/Sustainability ......................................................... 16 4.6.1 World Bank ................................................................................................................................ 16 4.6.2 TTCs ............................................................................................................................................ 16 4.6.3 Other Sustainability Issues....................................................................................................... 16 5. LESSONS LEARNED................................................................................................. 16 Evaluation of TLMP – TLMP in Ethiopia ii 6. RECOMMENDATIONS.............................................................................................17 6.1 Materials Development ..................................................................................................................... 17 6.2 Systems Improvement...................................................................................................................... 17 6.3 Towards a More Comprehensive Approach to Language Development ............................ 18 6.4 Linking TLMP to USAID Priorities in Early Grade Reading...................................................... 18 ANNEX A. SCOPE OF WORK ....................................................................................20 ANNEX B. QUESTIONS POSED OF AAMU TLMP DIRECTOR............................22 ANNEX C. OTHER DATA COLLECTION INSTRUMENTS...................................24 ANNEX D. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES..33 ANNEX E. MATERIALS DEVELOPMENT PROCESS - GRADES 3 AND 4.........377 ANNEX F. TLM ASSESSMENTS .................................................................................39 ANNEX G. CONFLICT OF INTEREST STATEMENTS.........................................422 Evaluation of TLMP – TLMP in Ethiopia iii LIST OF ACRONYMS AND TERMS AAMU Alabama Agricultural & Mechanical University AEI President’s African Education Initiative AIR American Institutes for Research CA Cooperative Agreement CD Curriculum Director COR Contracting Officer Representative CPD Continuous Professional Development CSU Chicago State University ECSU Elizabeth City State University EFA Education for All EFL English as a Foreign Language EGRA Early Grade Reading Assessment ELIP English Language Improvement Program ESL English as a Second Language IBTCI International Business & Technical Consultants, Inc. IFESH International Foundation for Education and Self Help INSET In-service Teacher Training KG Kindergarten MLC Minimum Learning Competencies MOE Ministry of Education MSI Minority Serving Institution MT Master Trainers PAC Project Advisory Committee RSEB Regional State Education Bureau SCSU South Carolina State University TELL Teach English for Lifelong Learning TL Team Leader TLM Textbooks and Learning Materials TLMP Textbooks and Learning Materials Program TOT Training of Teachers TTC Teacher Training Colleges USAID United States Agency for International Development USAID/E United States Agency for International Development/Ethiopia USAID/W United States Agency for International Development/Washington UTSA University of Texas, San Antonio VSO (historically) UK Voluntary Service Overseas Evaluation of TLMP – TLMP in Ethiopia iv EXECUTIVE SUMMARY OBJECTIVES In 2005 Alabama Agricultural and Mechanical University (AAMU) was awarded a Cooperative Agreement (CA) to implement the Textbook and Learning Materials Program (TLMP) in Ethiopia to generate English language textbooks for grades 6-8. Subsequently, in 2009, the CA was extended to generate English language textbooks for grades 1-4 and supplemental reading materials (folktales) for the same grades. The in-country evaluation of this program took place in March 2013, and had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based Minority Serving Institutions (MSIs), in-country institutions (Ministries of Education (MOEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. AAMU’s vision for TLMP was to build capacity in Ethiopia not only to increase the number of textbooks that children use, but also to enhance the teaching and learning process by integrating the most effective instructional methods into books and teacher’s guides. Collaboration between AAMU and the MOE,, and selected educationalists brought about a number of improvements in English language instruction in grades 1-4. TLMP activities in Ethiopia were to include: • Conducting a new needs assessment;1 • Writing TLMs with Ethiopian educators in accordance with national curricula that are culturally, socially, and academically appropriate and that challenge gender stereotypes; • Field testing all TLMs in Ethiopian rural and urban schools to determine understandability, effectiveness, and cultural appropriateness; • Revising and editing field-tested TLMs based upon the results of surveys and other field data collection; • Collaboratively reviewing camera-ready copy of all TLMs with all relevant partners; • Assisting the MOE in developing a distribution and delivery strategy; and • Conducting an impact study on the use of TLMs (not done). TLMP in Ethiopia actually had two parts: 1) production of English language textbooks and teacher’s guides for grades 2, 3, and 4 (and the finalization of grade 1 materials); and 2) the production of supplementary reading materials through a participatory writing process (similar to the materials development process used by University of Texas, San Antonio (UTSA) in South Africa and Malawi). For this latter activity, the Curriculum Directors (CDs) from all the regions were invited to participate with university and Teacher Training Colleges (TTC) faculty (this was the very first time that all of the 1 A second needs assessment was not conducted because the AEI evaluation (of Phase 1 of the project in all African countries) served as a situation analysis and made several recommendations as to the way forward. Evaluation of TLMP – TLMP in Ethiopia v CDs came together to work on a common set of materials). They created anthologies for grades 1-2 and 3-4, teacher’s guides, and two “big books” with stories for grade 1 and grades 1-2. The big books are meant for teachers to read aloud in the classroom. The educational policy and structural environment in which TLMP operated included a system in which a Federal MOE created the structure of education as well as curriculum, syllabi, and the development of learning materials. However, the nine Regional State Education Bureaus (RSEBs) and two City Administrations play a role, especially in early grade education in each mother tongue. TTC relates closely to both the Federal and regional/city MOEs as the language of instruction is supposed to be English, but teachers also require language education in the mother tongue to be effective in their classrooms. The system is struggling to cope with all the language acquisition needs as well as the content knowledge needs of teachers who teach in self-contained classrooms in grades 1-4. Part of this problem will be addressed in the development of teachers specialized in language education, math and science, and ethics. Evaluation Methodology A six-person team was assembled (three Americans and three Ethiopians, each with expertise in TLMP￾related areas) to conduct a literature review, individual interviews with senior Federal and regional MOE officials, interviews with school principals and teachers, observations of teachers using the TLMs and assessments of children’s ability to read and understand the TLMs. The team was divided into two, with one sub-team concentrating on research in Oromia and one in the Addis Ababa environs. The whole team conducted research in Amhara. Findings TLMP Output and Dissemination • Grade 1 – 4.3 million materials prepared (TLMP paid for 1.5 million student books and 25,000 teacher’s guides; the rest was paid for by USAID/Ethiopia (USAID/E), • Grade 2 – 1,700,284 student books and 50,000 teacher’s guides (TLMP paid), • Grades 3 and 4 – 45,000 teacher’s guides at each grade level (TLMP paid) (USAID/E printed 5.5 million student books for grades 2, 3, and 4), and • Folktale books – 700,000 anthologies and big books and 50,000 teacher’s guides (TLMP paid). The dissemination of the TLMs was highly irregular and inconsistent. The grade 1 materials were distributed in 2009, but grade 2 materials were distributed only in September 2012 and grades 3 and 4 materials had just been distributed before this evaluation began in March 2013. Project Management and Partnerships During the 2009-2012 period of implementation (Phase 2), AAMU hired a local coordinator. The bulk of the development activities had been completed by 2011; the remainder of the work required timely decision-making and oversight. At AAMU, the Director assembled a management team composed of a program coordinator, an office manager and herself. She also hired an illustrator who had been engaged in Ethiopia, but who then moved to Alabama to attend school. However, no individual was designated to develop a formal Monitoring and Evaluation (M&E) system and to track the production of the materials as well as the progress of teachers and students. A major partner to TLMP was USAID/E itself. Initially, USAID/E’s role was to liaise with the MOE and to help establish and maintain the relationship between AAMU and the MOE. Since USAID/Washington (USAID/W) had not included training on the use of the TLMs in the award, USAID/E provided funding Evaluation of TLMP – TLMP in Ethiopia vi to the American Institutes for Research (AIR) under the “Teach English for Lifelong Learning” (TELL) project to provide training to 50,000 English teachers. The actual training that TELL provided was met with challenges: • The budget was limited (although USAID/E offered what it could); • No pre-workshop assessment was undertaken to determine the English language capability of teachers (high school or primary school); • One-off training sessions were held for only 3.5 days; • There was no classroom follow up; • No provision was made for refresher training; and • The content of the training did not fully address the new pedagogy being introduced. Program Implementation Materials Improvement - Teachers and other educators who had attended 220 hours of classes offered in the English Language Improvement Program (ELIP) constituted the pool of individuals qualified to participate in the development of the TLMs. After attending three workshops, writers were able to write textbook lessons (or skeletons thereof) and/or folk tales that would constitute supplementary reading materials. Materials were then reviewed by the Ethiopian Coordinator, sent to AAMU for editing and illustrations, and then sent to the MOE for review. The MOE does not have a specific textbook review board nor was it aware of the leveling that was needed to make the TLMs accessible for teachers and students. It is not clear what criteria were used for these final assessments and edits. Assessment of TLMs - Overall, the TLMs produced under this initiative were far superior to the materials they replaced in terms of structure, print quality, and pedagogy. The TLMs reflect the MOE’s Minimum Learning Competencies (MLCs) for each grade level and were generally well-illustrated with color drawings. In general, all materials produced were in alignment with the MOE syllabi. In terms of the student textbooks, not all content moved evenly from simple to complex, and – taking the materials as a graded set – a “linguistic cliff” had to be climbed between the grades 1 and 2 materials. Content correctness and relevance to Ethiopian culture in certain instances was erroneous, and only minimal attention was paid to cross-cutting themes. The illustrations were numerous and helpful, although sometimes choice of color made picture detail difficult to distinguish. Topics were relevant for each grade, although pedagogical content seemed unsystematic. Opportunities for continuous assessment are included, but there is no specific instruction on how this will be done. The language used is far above what children (and teachers) are capable of understanding. The colorful presentation and large number of illustrations are very attractive and children want to look at the books. While the physical quality of the books is generally very durable, the size is cumbersome for children and should be divided by terms. In terms of the teacher’s guides, too many lessons are included under each unit and each lesson does not have its own objectives. Often there is a mismatch between what is in the guide and what is in the textbook. Sequencing is often a problem. Scripted lessons are presented for each topic, but the language used is beyond the capability of most teachers at the grades 1-4 level. Many new teaching ideas are presented, but these form a new, non-didactic, participatory approach to teaching to which teachers have not been introduced. There is significant cross referencing to other materials to be used, but teachers do not have much experience in using a range of materials to teach one subject in a classroom. The size of the guides is cumbersome, the text is very dense, there are no illustrations, and some explanations are missing; there is no glossary or other helpful drills that can be constructed in teaching different language acquisition skills. Evaluation of TLMP – TLMP in Ethiopia vii In the development of the materials, there was significant ambiguity in what was needed. The indecisiveness of the MOE as to which approach to take emerged in the materials and in the confusion that teachers and trainers felt in teaching the materials. Overall, the teacher’s guides were composed of scripted lessons. However, using this technique as an approach to overcoming teacher shortcomings is not viable. Teacher Training - In general, when new teaching materials are introduced, training on how to use them is critical. Although USAID/E graciously found the means to fund a project that trained teachers during both phases of TLMP, the model used was inadequate to meet the needs of the teachers. One￾off training left teachers confused about how to proceed largely because of their own English language inadequacies in being able to understand the new teaching methodology inherent in the teacher’s guides. Teacher Observations - There is a sufficient number of deficiencies and “not observed” comments to warrant a more comprehensive assessment of teacher ability to use the TLMs effectively and to conduct more rigorous training in the future. Learner Reading Assessment - The performance of a small sample of 10 cannot be used to make generalizations, but some observations and trends are noteworthy as they mirror the Early Grade Reading Assessment (EGRA) findings in certain important areas: • Children seem to make progress as they move up the grades; • Children perform better at tasks that are more behaviorist than cognitive (memorization, over￾thinking) and may reflect a preference of teaching style and pedagogy; • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension; • Children at both grades have minimal expressive language facility (e.g., none of the children spoke in complete English sentences); and • Given that test items were drawn from the student’s grade level TLMP textbook, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. Lessons Learned • In designing and implementing a TLM development program, teacher training on how to use the new materials must be included at both the pre- and in-service levels so that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use. • Scripted lessons cannot replace a systematic professional development program, particularly when the learning outcomes are cognitive, not rote learning. • When piloting newly-developed TLMs, teachers should be given them to use in the classroom for at least a month (better for three months), and then research should be undertaken by an independent party/publisher to determine challenges teachers faced and/or overcame in using the materials so that they can be revised before final printing. • Before launching a TLM production project, thorough research needs to be undertaken into the educational system: how teachers are trained (and for how long), the frequency and substance of In￾Service Teacher Training (INSET) programs, the level of English language competency of teachers (for English language textbooks), and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. • Before launching a language-based TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project: the learning outcomes and results achieved Evaluation of TLMP – TLMP in Ethiopia viii in using those books, and the level of language ability children have under past practices, so as to avoid making assumptions about what children are able to hear, speak, read and write. • Launching an innovative TLM production project works best when the various directorates of the MOE are working together in an overall learning improvement program that is not competing with other programs to create results. • Leveling any language materials is absolutely essential to creating positive learning outcomes among children; appropriate review policies, procedures and professional person power must be available to carry out these tasks. • For a U.S.-based university to work in an African country successfully, the staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts hampering long-distance communications) as well as with the donor. Recommendations Materials Improvement • If U.S.-based institutions are to participate in textbook development in Ethiopia, they should: • Hire only experienced individuals in textbook development for designated grades; • Be properly introduced to the country, the culture, and the practice of teaching in Ethiopia; • Have experience teaching the grades and subjects for which they will write textbooks; and • Be able to share their expertise with Ethiopians who have not developed textbooks before. • When conducting textbook field tests: • A broad array of schools must be included so that teachers working in different circumstances and with limited English capability can be included. • International project leaders must understand the cultural reluctance Ethiopians feel in criticizing another person’s work. • Teachers must be given between a month and three months after training to become familiar with the materials and the methodology, and to practice teaching before pilot/ assessment observations and interviews are held. Systems Improvement • Any project that creates textbooks MUST: • Include a teacher training program over the course of the project and beyond; • Include a strong M&E system that includes indicators on production, delivery and use in the classroom; and • Use a continuous professional development approach. • A sustainable strategy for training teachers and administrators in the use of the materials is critical. This strategy must include pre-service training at TTCs, continuous professional development through the use of the cluster centers, and the training of “master trainers” who can provide initial capacity building and then be available to deliver refresher courses. If the books are to be distributed to all children in all primary schools (and that is the goal), then all teachers must be capacitated to use them over the long term. • The cascading approach can be cost effective, but significant oversight needs to be exercised by designated focal persons or faculty members of TTCs to ensure that the training delivered is the same at all levels. • To ensure the use of TLMs, training organizers should plan to include all stakeholders in the trainings, such as supervisors/inspectors, and curriculum specialists at the woreda and sub-city level so that they can be supportive of teachers. Evaluation of TLMP – TLMP in Ethiopia ix Towards a More Comprehensive Approach to Language Development • Language Policy: The MOE needs to clarify its own language acquisition policy and determine goals that are pedagogically sound; building further capacity in curriculum and syllabi development within the MOE is cornerstone to this process. • Language Teaching: A more comprehensive approach to teacher training in language needs to be taken. Questions such as the following need to be addressed: • What entrance requirements should be satisfied for future teachers to be accepted into a TTC? • What will be the curriculum, especially for English, Amharic and mother tongues? • How will English language capability of all TTC entrants be improved so that English classes can be conducted in English, and teachers will be able to provide English language experience to their students? • What is the best approach to teaching English and other languages – ESL, EFL, and grammar translation? • How can methodologies for teaching language be integrated across the board so that teachers can utilize similar methodologies in teaching all languages? • How can the cluster center school structure be utilized to present on-going professional development in-service training so that teachers and administrators have an opportunity to learn continuously? • Teacher Licensing - The Licensing Directorate in the MOE is working on standards that will allow primary schools to become licensed. Part of this effort includes language tests, the results of which should be used in redesigning the TTC and English language curricula in all pre-service, in-service, and other professional development activities. Linking TLMP to USAID Priorities in Early Grade Reading High quality, well-illustrated TLMs were produced by AAMU in Ethiopia for grades 1-4. The above recommendations notwithstanding, the investment made by USAID/W and USAID/E in these materials should be an integral part of the new USAID priorities in early grade reading. The current contractor implementing this project should be encouraged to do the following: • Determine how the English language textbooks might be edited to reflect a greater step-wise reading/English language development process, beginning with phonics (in English and the various mother tongues targeted for Ethiopia), taking students through a sequential learning process that will build upon skills developed each day. • Consider how the methodology developed by the UTSA is generating TLMs through providing writing workshops to teachers and other educationalists, who then develop culturally relevant materials in the various mother tongues. The project would then take the materials through a range of leveling and pilot testing processes to ensure that children would be able to learn, in a step-wise manner, how to read in their home languages. • Consider how the supplementary reading folktales might be translated into mother tongues, edited and leveled, to be used in English and early grade reading, teaching, and learning in the targeted languages. • Work with the Curriculum Directorate of the MOE to outline a realistic curriculum and syllabus for each grade level so that it can incorporate strategies for mother tongue and English language reading skills development. • Although early grade reading is the focus of this new initiative, if possible, the new contractor might also review the TLMs produced for grades 6-8 to bring them more into line with earlier skill development in English and to work to incorporate these in the curriculum as textbooks rather than supplemental materials. Evaluation of TLMP – TLMP in Ethiopia x • Generate a strong teacher training program that integrates TTCs, woreda Education Offices, RSEBs, and others in the teacher training planning and implementation process. Evaluation of TLMP – TLMP in Ethiopia 1 1. Evaluation Purpose, Question and Objectives 1.1 The Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched by the United States Agency for International Development (USAID/W) in 2005 in conjunction with the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort to improve the management capacity of education sector personnel in a number of countries in Sub-Saharan Africa. TLMP was extended for another three years in 2008/09 to 2012. Each of five Minority-Serving Institutions (MSIs) was responsible for managing and implementing the TLMP in a specific country and with achieving specific outputs and results in accordance with its respective Cooperative Agreement (CA); Alabama Agricultural and Mechanical University (AAMU) was first awarded a CA to work in Ethiopia in 2005- 2008/09 on Textbook and Learning Materials (TLMs) for grades 6 through 8, and was awarded a second CA to generate TLMs for grades 1 through 4 in Ethiopia 2009-2012. Each MSI was responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s MOE and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 1) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA; 2) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program; and 3) Ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). Originally a program to develop TLMs for primary schools only, MSIs conducted initial needs assessments and determined that in three countries MOE priorities were for middle and secondary school TLMs. In Ethiopia, the MOE determined that English language textbooks should be developed for grades 1-4. To implement the project, AAMU (and their fellow MSIs) received about the same level of funding - approximately US$8 million. 1.2 Evaluation Objectives This performance evaluation covered work led by AAMU in Ethiopia during the period 2009-2012 and was intended to satisfy the following objectives (see ANNEX A – SCOPE OF WORK): • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets, measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. Evaluation of TLMP – TLMP in Ethiopia 2 2. TLMP Ethiopia Project Background 2.1 AAMU Background Research AAMU’s vision for TLMP was to build capacity in Ethiopia not only to increase the number of English language textbooks that children use, but also to enhance the teaching and learning process by integrating the most effective instructional methods into books and teacher’s guides. Collaboration between AAMU and the MOE and selected educators brought about a number of improvements in English language instruction in grades 1-4. The decision was made by the MOE for TLMP to produce TLMs in English in Ethiopia as it is the official international language and is taught beginning in grade 1. The goals of TLMP during 2009-2012 included: • Conducting a new needs assessment;1 • Writing TLMs with Ethiopian educators in accordance with national curricula that are culturally, socially, academically appropriate and that challenge gender stereotypes; • Field testing all TLMs in Ethiopian rural and urban schools to determine understandability, effectiveness, and cultural appropriateness; • Revising and editing field tested TLMs based upon the results of surveys and other field data collection; • Collaboratively reviewing camera-ready copy of all TLMs with all relevant partners; • Procuring bids for printing; • Assisting the MOE in developing a distribution and delivery strategy; and • Conducting an impact study on the use of TLMs (not done). TLMP 2009-12 in Ethiopia actually had two parts: 1) production of English language textbooks and teacher’s guides for grades 2, 3, and 4 (and the finalization of grade 1 materials); and 2) the production of supplementary reading materials through a participatory writing process (similar to the materials development process used by UTSA in South Africa and Malawi). For this latter activity, the Curriculum Directors (CDs) from all the regions were invited to participate with university and Teacher Training College (TTC) faculty; this was the very first time that all of the CDs came together to work on a common set of materials. They created anthologies for grades 1-2 and 3-4, teacher’s guides, and two “big books” with stories for grade 1 and grades 1-2. The “big books” are meant for teachers to read aloud in the classroom. Severe difficulties in implementation began in the last two years of the project. For example, TLMP staff had not been paid since August 2012, the printing of the supplementary reading materials was stopped due to lack of payment, and the small administrative office that was opened in Addis Ababa was closed. While interviews were held with all other MSIs prior to field research, our Contracting Officer Representative (COR) advised us not to conduct campus interviews with AAMU staff. However, a telephone interview was held with the Project Director at her new place of employment. 2.2 MOE Context and the Educational System Ethiopia is a Federal Republic composed of nine Regional States and two City Administrations. The Federal MOE in Addis Ababa determines the overall curriculum, facilitates the development of textbooks, produces syllabi for each subject in each grade, determines the type and duration of pre￾service teacher training, promotes In-service Teacher Training (INSET), determines the criteria for licensing (in progress), and other similar activities. Each Regional State and City Administration has its 1 This second needs assessment was not conducted because the AEI evaluation (of Phase 1) served as a situation analysis and made several recommendations as to the way forward. Evaluation of TLMP – TLMP in Ethiopia 3 own Regional State Education Bureau (RSEB) or equivalent that has oversight over such things as examinations and learning assessments at grades 4 and 8 (administered in the predominant mother tongue of the region), certain curriculum regulations, determining at what grade English is taught as a subject and at what grade it becomes the language of instruction. Some lines of authority are not entirely clear and this has given rise to a number of curricular ambiguities as well as challenges in the types of materials to be used in teaching. English is taught as a subject beginning in grade 1 throughout the country. In two regions/city administrations, English is used as the language of instruction beginning in grade 5; in seven regions, at grade 7; and in two regions at grade 9. Textbooks are not available in all languages and TTC are hard￾pressed to train teachers in all of these languages. (Teachers, in the past, could be assigned to schools in any region; their ability to teach the local language was not necessarily a criterion. However, according to officials at the MOE, this is slowly changing in an effort to create language-teaching specialists.) Both the national school leaving examinations in the 10th and 12th grade are in English. The school-age population in 2008 was 19,573,771. Of these, 15,340,786 (78%) were in primary schools: 79.5% in rural schools and 20.5% in urban schools. Boys comprised 53% and girls 46.5% of the student population.2 Since Education for All (EFA) was adopted, public education at the primary level became free, although it is not compulsory, and the number of students entering school has “exploded,” especially for girls. Many rural, school-age students are not enrolled due to: distance of homes from schools; opportunity cost of child labor; poor health of the children and/or parents; and poverty. These same causes have created a high drop-out rate, in particular between grades 1 and 2, with a very heavy drop-out rate of females at grade 4 (the end of first cycle primary that generally coincides with the onset of menses), as reported in the AEI evaluation. Large class size, a severe lack of textbooks, and the level of training that teachers receive all contribute to very difficult teaching conditions. The textbook – student ratio ranges in urban areas between1:1 to 1:6, but can be as high as 1:100 in the rural areas. Complicating this structure are the differences in the ability of teachers to teach in a self-contained classroom, which is the norm for grades 1-4, and the fact that many schools operate on a shift system. This structure requires that teachers teach all subjects to children according to the curriculum and timetable. With language instruction taking no less than 10 class periods a week (mother tongue and English; and more if Amharic as an additional language is included), and with the instructional day no longer than three hours for grades 1 and 2, and no longer than four hours for grades 3 and 4, a teacher must be well-trained to cope with the number of children in a classroom as well as deal with the range of subjects that must be taught in the time allotted. The Federal MOE is moving toward teacher specialization in three areas – language, math and sciences, and ethics. When fully operational, children will be taught languages (mother tongue, Amharic and English) by language specialists, math and sciences by specialists in these areas, and ethics in the mother tongue by specialists in this field. Other rurally￾based problems include: • Teachers are often absent from class, especially during harvest time; and • Instructional radio broadcasts are now conducted in the mother tongue in each region, so they are not repeated as frequently as when they were all in Amharic or English; equipment is in disrepair. 2 During our research, we noted that in all six primary schools in which data was collected, the enrollment of girls in all grades was higher than that for boys, sometimes by as much as 200%; this is most likely the result of adoption of the Education for All (EFA) policy. Evaluation of TLMP – TLMP in Ethiopia 4 2.2.1 Teacher Training and the Languages of Instruction The teacher training system has only recently been stabilized, according to MOE informants. Approximately ten years ago, TTC entrance requirements included a pass in the 12th grade leaving exam; TTC attendance covered only one year. Thereafter, because an insufficient number of teachers were being trained to meet the demand, admission requirements were lowered to include those who passed the 10th grade leaving exam (having earned only a 2.0 grade); students were then trained for one year at the TTC. Subsequent MOE research, however, indicated that TTC students did not have enough subject matter knowledge to teach in self-contained classrooms; hence, the period for training at a TTC was extended to three years, or to the diploma level. The Federal MOE is providing the diploma￾qualifying learning opportunity to teachers already in service through instruction given on weekends and break times. It is not clear what the content of the training is. Complicating the teacher training process is the language of instruction at each TTC. Although English is supposed to be the language of instruction in learning institutions from grade 9, even the TTCs find it difficult to teach in English. Moreover, the regional states and cities where the TTCs are located have different languages of instruction in their respective primary schools. For example, the TTC in Hawassa in the Southern Nations, Nationalities, and Peoples Regional State (SNNPR) focuses on Sidama as a mother tongue. Since materials in this language are quite sparse, faculty and students face daunting challenges as a number are themselves not fluent in Sidama. The language and literacy program at most TTCs begins with mother tongue skill building, and then goes on to English and Amharic. Reading in all of these languages can be taught using a phonics approach, but it is not clear that this is what is done. Evidence at schools demonstrated a phonics approach in teaching children how to read in Afan-Oromo, but the teaching methods are not utilized across the language curricula, i.e., the same phonics-based decoding skills are not used to teach reading in each language. 2.2.2 Systemic Textbook Issues There are several discrepancies between textbook development and examinations, with the latter department not being forewarned of a new textbook in a particular subject matter. For example, we were informed that in one instance the Examinations Directorate was given an advance copy of a new textbook so as to include the subject matter on the examination. At examination time, the students were in an uproar because the materials covered on the examination had not been addressed in class because the textbooks had not been delivered. In interviews conducted with the Curriculum Directorate, we learned that the former Prime Minister had issued a directive that textbooks were to be distributed on a 1:1 ratio of children to books. When the TLMP could not deliver all the materials in a year’s time, an international tender was sent out not only for English books for grades 5-8, but also for other subjects. Despite the rush to get the materials developed, printed, and distributed, it still took two years to complete the process. We were also informed that the textbooks produced were not as acceptable as those developed by TLMP. 2.2.3 Issues Associated with Approaches to Teaching Literacy The MOE has also included a number of reading and language programs in its curricula, some of which are based on phonics in the earliest grades and others that take more of a whole language approach. With the launch of the new early grade reading project (READ), yet another approach will be developed not only for mother tongue language and literacy development, but also for English. It is not clear whether the grades 1-4 TLMP materials will go the same way as the TLMP materials for grades 6-8, which are now in the category of “supplemental” materials. Evaluation of TLMP – TLMP in Ethiopia 5 The MOE is searching for the best way it can address language and literacy development in several languages. Every five years, the MOE undergoes a curriculum review. At the time TLMP was launched (AEI in 2005 and TLMP in 2009), reform had just taken place. Prior to 2009, the MOE decided that the whole language approach was to be used in teaching English, although the syllabi that emerged could easily have adopted an English-as-a-Second Language (ESL) approach. The syllabi had been produced by an external consultant, meaning that a full understanding of what Ethiopian teachers and children need to acquire language and reading skills in several languages using different orthographies may not have been explored fully. The Curriculum Directorate has determined that the whole language approach is not producing the desired results, and the next curriculum revision – the team was told -- will chart a more comprehensive course in language and reading education. For this to take place, however, the MOE must have a deeper understanding of the purpose for the current language programs, design and implement an overall language policy, and create the curriculum and syllabi that best meets the new policy. In conclusion, the TLMP activity in Ethiopia was not necessarily about teaching reading. It was primarily an English language program into which USAID/E introduced reading pedagogy ahead of its new primary grades reading initiative. The ambiguity as to whether this was an English language, reading, or ESL/EFL program is demonstrated in the syllabi as well as in the TLMs produced. 3. Evaluation Questions, Methods & Limitations The evaluation questions are listed in Section 1.2 above. The evaluation of TLMP in Ethiopia made use of a “mixed-methods” methodology that included: review of project documents; analysis of the TLMs produced; key informant interviews with AAMU’s former director, Ethiopian educators involved with the project, Ethiopian education officials at different levels of the hierarchy, and school personnel; assessments of the reading capability of students at different grade levels; discussions with Mission staff; and discussions with personnel from various development partners. 3.1 Background Research Conducted on AAMU It was not feasible for evaluation staff to make a site visit to AAMU or to review most project documents. However, the Team Leader (TL) was able to interview AAMU’s TLMP Director at her new university (see ANNEX B – QUESTIONS POSED OF AAMU TLMP DIRECTOR). 3.2 Research Activities in Addis Ababa, Oromiya and Amhara Environs The data collected from each category of stakeholders in each location are presented in ANNEX C – OTHER DATA COLLECTION INSTRUMENTS. The order with which the research was conducted is found in ANNEX D – SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES. Over the two-week period (March 3-19, 2013) in which the team was in Ethiopia, data were collected at the following venues/from the following stakeholders: • Visits to two TTCs, Kotebe in Addis Ababa and Debre Berhan (Amhara); • Observations and interviews at seven primary schools: three in Oromiya: Mulugeta Gedle, Burayu, and Dukem 1, three in Addis Ababa: TTC, Wondrad Demonstration, and Sefre Selam, and one in Amhara: the Debre Berhan TTC; • Interviews with nine school principals/deputy principals; • Observations of 10 teachers; • Focus group interviews with 22 teachers; • Forty reading assessments with 20 first graders and 20 second graders (20 boys and 20 girls) at five schools in Addis Ababa and Oromia; 1st grade age range: 7-14; 2nd grade age range: 8-15; Evaluation of TLMP – TLMP in Ethiopia 6 • Interviews with six English curriculum specialists; • Interviews with three Master Trainers; • Meetings with four textbook developers and a telephone interview with one coordinator; • Meetings at two RSEBs and seven woredas/sub-city officers; • At the MOE, interviews with the State Minister, Director of Curriculum, and Heads of English Curriculum, Teacher Development, Examinations, Licensing, Stores/Stockkeeper; • A telephone interview with the Graphic Designer; • Interviews with two International Foundation for Education and Self Help (IFESH) Volunteers in early grade reading in Addis Ababa and at the Debre Berhan TTC in Amhara; • Interview with the World Bank Procurement Contractor; and • Interviews in Addis Ababa with the TLMP Project Director and Project Coordinator. In identifying schools, we wanted to maintain a focus on at least three regions where different mother tongues have conversational dominance, so we selected schools in Oromia (3), Amhara (2), and Addis Ababa (6). (Due to an outbreak of meningitis, it was not possible to travel to SNNPR as originally planned.) Because we wanted to understand how the TTCs were incorporating the TLMs into their courses, in Addis Ababa we went to the Kotebe TTC; we had the same plan for the Debre Berhan TTC, but circumstances noted below under “Limitations” prevented us from collecting significant data at this TTC. To accommodate the limited English language ability of school principals and teachers, the team was divided in two: one team of three (two Ethiopians with facility in Afan-Oromo and the TL) collected data in Oromia; and one team of three (two Americans and one Ethiopian) collected data at the Kotebe TTC and two schools where English was spoken by all those to be interviewed. This division was made so as not to overburden the Ethiopian members of the team. The Oromia team traveled to Mulugeta Gedle, Burayu, and Dukem #1 primary schools, each of which was located in different directions approximately 40 kilometers from Addis Ababa. No principal or teachers could be interviewed in English, nor could the woreda or RSEB heads. The Addis Ababa team interviewed the director, English language instructors, curriculum specialists, and materials developers at Kotebe TTC. Also interviewed was the Peace Corps Volunteer whose job it was to train TTC students on how to use the TLMs. The team then went on to the TTC demonstration school to observe teachers and assess students. Another school where the team collected data was Sefre Selam, a school noted for its strong community support. The Addis Ababa team also went to three other schools - at which curriculum heads, materials development educators, or master trainers were located. However, as these schools were not primary schools, no others were interviewed. The Ethiopia TLMP evaluation presented a somewhat unique opportunity: assessing the TLMs produced for each grade level. The team developed a rubric that considered specific quality indicators of content, pedagogy, and publishing quality, and then worked in pairs (one Ethiopian and one US expert) to evaluate the materials. One pair evaluated grade 1 student textbooks and teacher’s guides, another evaluated grade 2 TLMs, and the third evaluated both grades 3 and 4. Discussion of the evaluations reflected the various professional backgrounds of each of the team members and yielded insight into the strengths and shortcomings of the TLMs, including literacy, language, culture, context, level of capability, teaching methodology, etc. Protocol for Conducting Reading Assessments. Since our research design included assessing children’s ability to read a portion of the grade-specific TLMs, we had to be at a school when classes Evaluation of TLMP – TLMP in Ethiopia 7 started at 8:00am as English is generally the first subject taught. We, therefore, conducted the teacher observations first and then took two boys and two girls from Grade 1 and Grade 2 classes to conduct the reading assessments. As discussed under Limitations below, Grade 2 and Grade 4 materials were not yet in use. 3.3 Limitations of the Study While as planned we did visit locales where different first languages are spoken and which are written using alternatively the Amharic syllabary or the Western alphabet. Overall, there was insufficient time to conduct an evaluation that was geographically inclusive. We also did not have time to collect data on schools that had not received the books to determine what difference having the books made to the children’s language facility and ability to read. Our research design had originally included trips to the regional capitals of SNNPR (Hawassa) and in Amhara (Bahr Dar). When we interviewed the education team at USAID/E on the first day of field research, we were told not to go south to Hawassa as several cases of meningitis had been reported. We also learned that the key people we were to interview in Bahr Dar were in Addis for a meeting, so we interviewed them in Addis. When the whole team went to the Debre Berhan TTC (identified by the MOE as a “center of excellence”), we learned that the former director had recently resigned and the new director had been in his position only two months. Also, contrary to what we had been told by the Federal MOE, the TLMs had not been distributed at the TTC, so no action was being taken in the classroom or elsewhere in training future teachers on how to use the TLMs. (Kotebe TTC in Addis Ababa did have the TLMs and was making use of them.) In Debre Berhan, we had also scheduled appointments at the cluster school in which the TTC provided INSET on a periodic basis. However, when we arrived, we learned that the school was closed as all teachers were participating in a math and science workshop. When we went to the Woreda Education Office, we learned that the two senior officials with whom we had made appointments had been called elsewhere for meetings, and other staff were not available. Hence, apart from the meetings with RSEB staff who happened to be in Addis, we were not able to collect any data in Amhara. When we conducted our interviews at the MOE, we learned that due to administrative issues, the books for grades 3 and 4 had just been delivered to the RSEBs. Some had been picked up from the RSEB warehouse by school leaders but were not yet being used in the classroom as the year was half over (each textbook is to be used for a year). Since most grade 3 and 4 children had not been exposed to the materials, we could not assess their reading ability based on the use of the TLMs. A final limitation was our inability to contact Mondography Printers to determine whether TLMP had any impact on the company’s development. A road was being built outside of the business premises that prevented access. Electricity had been cut off, as had telephone communication. We obtained a cell phone number for the proprietor but repeated attempts to get in touch failed. Hence, we have no direct data on the Ethiopia-based printer, except from the TLMP Director, who informed us that the supplementary readers were now being produced as Mondography had received partial payment from AAMU. Evaluation of TLMP – TLMP in Ethiopia 8 4. Findings and Conclusions This section presents the findings that resulted from the use of the data collection instruments and procedures mentioned in the Methodology section above. We begin with the data collected on AAMU, present data on the print output of the project, and then go on to present findings on different aspects of the project. 4.1 AAMU Achievements and Challenges While AAMU was on a trajectory to make institutional changes supportive of international development activities in education, the evaluators were not in a position to collect data at AAMU to confirm these activities. Consequently, we have no information as to institutional changes that may have been made at AAMU as a result of TLMP participation. 4.2 TLMP Output By the end of 2008, TLMP had produced 3.2 million culturally- and research-based TLMs, and 70,000 teacher’s guides for grades 6, 7, and 8. Final numbers for 2009-12 outputs include the following: • Grade 1 – 4.3 million (TLMP paid for 1.5 million student books and 25,000 teacher’s guides; the rest was paid for by USAID/E); • Grade 2 – 1,700,284 student books and 50,000 teacher’s guides (TLMP paid); • Grades 3 and 4 – 45,000 teacher’s guides at each grade level (TLMP paid) (USAID/E printed 5.5 million student books for grades 2, 3, and 4); and • Folktale books – 700,000 anthologies and “big books” and 50,000 teacher’s guides (TLMP paid). The dissemination of the TLMs was highly irregular and inconsistent. The grade 1 materials were distributed in 2009, but grade 2 materials were distributed only in September 2012, and grades 3 and 4 materials had just been distributed before this evaluation began in March 2013. Distribution was carried out largely by the MOE and RSEBs, with documentation maintained by the central storekeeper as to when the materials were delivered. Since RSEBs do not have the means to transport the materials the rest of the way to the schools or even to zones (the level of government immediately below the region), both of these arranged to pick up the number of books allocated to a given school from the RSEB warehouse. In keeping with the former Prime Minister’s directive to create a 1:1 ratio of textbooks to students, USAID provided significant financial support to print the number of textbooks needed to fulfill this goal. Because of the dollar limitation USAID/E has in issuing local tenders, they had to issue an international tender. The printing award went to a South African company, who then sub-contracted to a printing company in China. Consequently, with the exception of materials produced in Ethiopia by either Mega or Mondography Printers, all other textbooks were produced in China. 4.3 Project Management and Partnerships A major partner to TLMP was USAID/E itself. Initially, USAID/E’s role was to liaise with the MOE and to help establish and maintain the relationship between AAMU and the MOE. However, during the AEI period, it became very clear to USAID/E that training on how to use the grade 6-8 textbooks was a necessity as the books introduced new methods of teaching. If teachers and students were to benefit (teacher training per se was not covered in the CA), USAID/E determined that it should facilitate a teacher training process. It made a special agreement with a service provider to organize the training: the American Institutes for Research (AIR) was contracted under the Teach English for Lifelong Learning Evaluation of TLMP – TLMP in Ethiopia 9 (TELL) project to provide training. Under this contract, approximately 20,000 English teachers were trained on the grade 6-8 materials. USAID/E continued this project during the 2009-12 period, and TELL trained 50,000 more English teachers (many of whom taught in self-contained classes) in the use of the grades 1-4 textbooks. The TLMP Director wrote the training manual (in Alabama), and TELL implemented it. The TLMP trained 32 Master Trainers (MT), who then cascaded the training to 300 high school teachers. On the basis of individual action plans, the high school teachers then trained primary school teachers, all with the support of TELL. While this endeavor made a “dent” in training needs, there are over 300,000 teachers who need the training. The actual training that TELL provided was met with challenges: • The budget was limited (although USAID/E offered what it could), • No pre-workshop assessment was undertaken to determine the English language capability of teachers (high school or primary school), • One-off training sessions were held for only 3.5 days, • There was no classroom follow up, • No provision was made for refresher training, and • The content of the training did not fully address the new pedagogy being introduced. Underlying the training challenges was the English language capability of teachers. The MOE recognizes this shortcoming and so has arranged with the British Council to again undertake another English Language Improvement Program (ELIP) to train 40,000 more teachers in English. This program is offered by the Teacher Development Directorate. As noted above, USAID/E played a major financial role in having the books printed. When USAID/W reduced its funding some time in 2010/2011, TLMP was hard-pressed to print the number of books needed to go beyond the pilot schools. USAID/E paid for the printing of all grade 3 and 4 textbooks, and a percentage of the grade 2 books. Essentially, the project would not have worked without these two areas of mission involvement. Owing to the increased activity of TLMP during 2009-12, AAMU hired a local coordinator in Ethiopia who had very rich experience in curriculum and planning at the MOE. Based in AAMU, the Director continued to make several trips to Ethiopia per year, but the bulk of the development activities had been completed by 2011; the remainder of the work required timely decision-making and oversight. At AAMU, the Director assembled a management team composed of a program coordinator, an office manager and herself. She also hired an illustrator who had been engaged in Ethiopia, but who then moved to Alabama to attend school. However, no individual was designated to develop a formal M&E system and to track the production of the materials as well as the progress of teachers and students. Working with the Ethiopian coordinator in Addis Ababa was a former Voluntary Services Overseas (VSO) volunteer who had been working with the MOE Curriculum Directorate during AEI. She began her employment with TLMP in 2010 and was responsible for the development, review and revision of grades 3 and 4 materials. Review included field testing at sites within easy traveling distance from Addis Ababa in the Oromiya and Amhara Regions. A partnership was developed between TLMP and Ethiopia Reads (ER), an organization that builds and stocks libraries throughout Ethiopia. Libraries were established and stocked at schools in each of the 11 RSEBs and administrative cities. Schools had to agree to set aside space to house the library and ER Evaluation of TLMP – TLMP in Ethiopia 10 provided US$10,000 worth of books for each one. It was envisioned by TLMP that all of the TLMP materials would be placed in the library, but in the two libraries observed (Mulugeta Gedle and Tsehai Chora – our pilot school in Addis Ababa), they were not in evidence. 4.4 Project Implementation 4.4.1 Materials Development AEI in Ethiopia ended with instructional materials in the pipeline. These included print-ready materials for grade 1 (developed solely by the TLMP Director), and preliminary work on the grade 2 student book. It is unclear what process was used to develop the grade 2 textbooks as they differ significantly from those produced for grade 1 and do not necessarily follow the grade 1 materials. In fact, the team determined that there was a “language cliff” that had to be climbed between grades 1 and 2 that assumed children could read and could address some very complicated spellings, grammatical structures, and learning concepts. For an explanation of how the grade 3 and 4 materials were developed, please (see ANNEX E - MATERIALS DEVELOPMENT PROCESS GRADES 3 AND 4). Teachers and other educators who had attended the 220 hours of ELIP training constituted the pool of individuals deemed qualified to participate in the development of the TLMs. After attending three workshops, writers were able to write textbook lessons (or skeletons thereof) and/or folk tales that would constitute supplementary reading materials. Materials were then reviewed by the Ethiopian Coordinator, sent to AAMU for editing and illustrations, and then sent to the MOE for review. The MOE does not have a specific textbook review board nor was it aware of the leveling that was needed to make the TLMs accessible for teachers and students. It is not clear what criteria were used for these final assessments and edits. 4.4.2 Assessment of TLMs by Evaluation Team Student Textbooks - The criteria on which the assessment of pedagogical quality/content was conducted in student textbooks included: alignment with syllabus, organization of content, correctness of content/conformity with Ethiopian culture, integration with other subjects and cross-cutting issues, quality of illustrations, relevance and appropriateness for learners and teachers, opportunities for assessing student progress, language and communication, and enjoyment. In terms of the physical design of the student textbooks, the following criteria were used: cover and paper stock, bindings, size and dimension, quality of print, typeface, and structures. In general, all materials produced were in alignment with the MOE syllabi. In terms of the student textbooks, not all content moved evenly from simple to complex, and – taking the materials as a graded set – a “linguistic cliff” had to be climbed between the grades 1 and 2 materials. Content correctness and relevance to Ethiopian culture in certain instances was questionable, and only minimal attention was paid to cross-cutting themes. The illustrations were numerous and helpful, although sometimes choice of color made picture detail difficult to distinguish. Topics were relevant for each grade, although pedagogical content seemed unsystematic. Opportunities for continuous assessment are included, but there is no specific instruction on how this should be done. The language used is far above what children (and teachers) are capable of understanding. The colorful presentation and large number of illustrations are very attractive and children want to look at the books. While the physical quality of the books is generally very durable, the size is cumbersome for children and should be divided by terms (rather than having one book for the whole year). Evaluation of TLMP – TLMP in Ethiopia 11 Teacher’s Guides – The criteria on which the assessment on pedagogical quality/content was conducted in teacher’s guides included: objectives, organization and presentation, teaching and learning strategies, teaching ideas and referencing. In terms of the physical design of the teacher’s guides, the same criteria as the assessment for the student textbooks were used. Too many lessons are included under each unit and each lesson does not have its own objectives. Often there is a mismatch between what is in the guide and what is in the textbook. Sequencing is often a problem. Scripted lessons are presented for each topic, but the language used is beyond the capability of most teachers at the grades 1-4 level. Many new teaching ideas are presented, but these form a new, non-didactic, participatory approach to teaching to which teachers have not been introduced. There is significant cross-referencing to other materials to be used, but teachers do not have much experience in using a range of materials to teach one subject in a classroom. The size of the guides is cumbersome, the text is very dense, there are no illustrations, and some explanations are missing; there is no glossary or other helpful drills that can be constructed in teaching different language acquisition skills. In summary, the materials require improvement for them to be truly useful to both teachers and students. (For a full assessment of the TLMs by the evaluation team, see ANNEX F – TLM ASSESSMENTS.) 4.4.3 Comments on the TLMs Made by RSEB and Woreda Education Officials, Principals and Teachers It was initially planned that there would be three sections reporting on the findings of each of the officials and teachers identified. However, the overlap in what each of the groups of respondents reported was so significant that findings have been consolidated. The reader will note that many of the points raised by stakeholders about the TLMs are similar to the assessment findings made by the team. Pilot Testing: In 2008, 2009, and 2010, grade 1, grade 2, and grades 3 and 4 TLMs, respectively, underwent extensive pilot studies. The materials were evaluated based on the following criteria: Criteria on Which TLMs Were Evaluated During Pilot Studies No. Evaluation criteria Remark 1 Content of stories is appropriate to the target age group. 2 Stories are interesting to the reader in the target group. 3 Stories will help to improve reading habits in the target group. 4 Level of language difficulty is suitable for the target group. 5 Book does not contain grammatical errors. 6 Illustrations are interesting and appropriate to the text. 7 Illustrations are culturally appropriate. 8 Layout and design are attractive and suitable for the target group. 9 Stories encourage integration of positive values. 10 Stories take into account cultural diversity. 11 Treatment of gender in the text and illustrations of the textbook is appropriate/balanced. 12 Questions are relevant and suitable for the target learners. 13 Questions are relevant and suitable for the target learners 14 The Instructional Guide will be helpful for teachers. 15 The Instructional Guide promotes communicative, active learning. Evaluation of TLMP – TLMP in Ethiopia 12 The results of this assessment, conducted at pilot schools where teachers had received more input, were mostly positive, and very few critical comments were made. Missing from this and other instruments were questions on the challenges and difficulties teachers faced in using the materials. Without a period (between one and three months) during which teachers are equipped to use the materials and then actually use them in their classes, responses to this data collection instrument were largely hypothetical. Current Evaluation: The range of school-based stakeholders the team interviewed reported the following (some comments are made by principals, while others are made by teachers): Positive Responses • The colorful pictures help students understand the content better; • The books motivate students to read more; • The physical part of the books seems well-organized, attractive and durable; • The textbooks fit the MOE curriculum well; • Gender equality is well presented; • The diversity of names in the texts reflect the different regions of the country; • Students respond well to the diverse learning activities; • Almost all students in grades 1 and 2 have textbooks; • Books are learner-centered as they focus on the interests of the children; and • The books are innovative as the teacher role shifts from being directive to facilitating. Challenges and Negative Responses • Teacher English Language Deficiency - Teachers themselves don’t have a command of English, and there are few opportunities to practice English outside of class. ELIP needs to be extended over the long term so teachers can improve their English. Teachers have to use dictionaries to look up vocabulary words in the TLMs. • Number of Books Insufficient until Recently - To overcome the shortfall of the ratio of textbooks to students, one school director who became impatient waiting for the books said: “We photocopied some of the books, but much was lost when we could not reproduce colored pictures.” • Continuous Assessment – A principal said: “Teachers don’t care to assess students because they feel it is a waste of time since students are promoted to the next class anyway.” • Content of Student Books - It is very difficult to relate the content of the textbooks to the context of children’s lives. More drills are needed on different structures so that children can incorporate them into the way they speak. Students cannot understand the questions in the books. Grade 3 stories are too long and teachers have a lot of vocabulary to teach; grade 4 students are sent to the library to look up words. Teachers are unable to complete the grade 2 textbook in a year’s time as there are too many lessons; many of the songs and games are skipped because teachers cannot understand the English used. Teaching children in grade 1 how to write the alphabet takes a considerable amount of time. Many pictures appear in the books without an explanatory legend beneath them. The grade 2 textbook has many very packed pages and it is really beyond the capacity of the children. There are no “explicit” grammar exercises on tenses, parts of speech, conjunctions, sentence structure or other aspects of grammar. Sentences for grade 4 materials are longer than those for grade 3 and they are too difficult. Reading passages lack review questions. • Physical Structure of Books -TLMP books are fragile; they wear out in two years. Evaluation of TLMP – TLMP in Ethiopia 13 • Teacher’s Guides (TG) – Teachers noted: answers to exercises in the TGs do not always match the student book exercises and need illustrations; it is almost impossible to teach the materials without the TG because each lesson is scripted and does not leave much room for teacher creativity. Conclusion Overall, the TLMs produced under this initiative were far superior to the materials they replaced in terms of structure, print quality, and pedagogy. The TLMs reflect the MOE’s Minimum Learning Competencies (MLCs) for each grade level and those were generally well-illustrated with color drawings. While the textbooks and teacher’s guides were very welcome by all, the inability of teachers to use them appropriately in self-contained classrooms was a major setback to project achievements, largely because of teacher’s own English language deficiencies. All those interviewed addressed the levels of English language used as being too high, the lessons being too long, the materials in the student books not matching the materials in the teacher’s guides, and the teacher’s guides being long and “bulky.” In the development of the materials, there was significant ambiguity in what was needed (even though the AEI evaluation recommended a more focused ESL approach), and the indecisiveness of the MOE as to which approach to take emerged in the materials and in the confusion that teachers and trainers felt in teaching the materials. Overall, the teacher’s guides were composed of scripted lessons. However, using this technique as an approach to overcoming teacher shortcomings is not viable. 4.4.4 Comments on Training Made by RSEBs and Woreda Education Officials, Principals, Teachers and Master Trainers Although training on the use of the materials was not a focus of TLMP, USAID/E supported the TELL project (described above) to help teachers learn how to use them. RSEBs, Woreda Officials, Principals, Teachers, Master Trainers, and TOTs made the following comments about training: Positive Comments • We have learned some new teaching techniques from the TG such as how to use flash cards. • The developers have learned how to collaborate with a university in the U.S. to create materials for our children. Challenges and Negative Comments • Teachers need more training in pronunciation/spoken English and in writing. • The underlying pedagogy inherent in the materials was not a focus of the training; so many teachers missed the point and just follow along with the lesson script without realizing that they are also learning how to teach in a different way. • The training offered to teachers through the cascade model was “watered down” at each level, with primary school teachers receiving the least training; in essence, the cascade model as implemented is ineffective. • When we taught the teachers how to use the materials, a big piece missing was how to assess the children and what they learned/skills they acquired. • Training needed to be conceptualized as a long-term process that provided more than a one-off workshop; there was no real follow-up. • The TTCs have not taken up the TLMs and so there is no other training available for teachers to learn how to use the materials. Conclusion Evaluation of TLMP – TLMP in Ethiopia 14 In general, when new teaching materials are introduced, training on how to use them is critical. Although USAID/E graciously found the means to fund a project that trained teachers in the use of the TLMs, the model used was inadequate to meet the needs of the teachers. One-off training left teachers confused about how to proceed, largely because of their own English language inadequacies in being able to understand the new teaching methodology inherent in the teacher’s guides. 4.5 Outputs and Outcomes Grade 1 TLMs were developed and distributed to Ethiopian primary schools beginning with the 2010-11 school year. Since then grade 2 materials were distributed in time to be used for the 2012-13 school year. Grades 3 and 4 TLMs had just been delivered to RSEBs at the time of evaluation. There was a major gap between the time teachers were trained in the use of the materials and when they actually received the textbooks for classroom use (approximately two years); it is highly likely that teachers forgot what they had learned and were in need of a refresher course. The teacher-generated Ethiopian folk tale books were developed in the last year of the project and made camera ready, but printing was halted due to lack of payment by AAMU. Development of the materials called upon teachers and other educators to record folk tales that were then edited by the Coordinator’s team and the AAMU team. Production of the 700,000 anthologies, “big books,” and 50,000 teacher’s guides is now on track. 4.5.1 Teacher Observations A total of 10 teachers were observed, 8 female and 2 male, in grades 1-3 (teachers in the Wondrad Demonstration School associated with the Kotebe TTC in Addis Ababa had received the grade 3 textbooks). Although most of the items identified were “observed,” significantly not observed were the following: • The teacher gives and corrects homework using the TLMs (10); and • The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson (8). Deficiencies were also noted in the following areas (“not observed”): • The teacher has prepared an authentic lesson that uses the TLMs (2). • The teacher explains the goal and purpose of the class lesson (5). • The teacher identifies, pronounces and defines any difficult vocabulary (3). • The teacher begins the class activity with questions that review previous activities using the TLMs (2). • The teacher uses learning aids/materials produced by TLMP (6). In terms of the artifact inventory, “not observed” were the following: • Lesson objectives are written on the board in English (8); • Learning aids/materials are posted in the classroom (8); • Word walls display words in English (8); and • There is evidence that teachers mark exercise books (9). Conclusion There is a sufficient number of deficiencies and “not observed” comments to warrant a more comprehensive assessment of teacher ability to use the TLMs effectively and to conduct more rigorous training in the future. Evaluation of TLMP – TLMP in Ethiopia 15 4.5.2 Learner Reading Assessments and Outcomes The Assessment Instruments and Process The purpose of the TLMs is to help Ethiopian children learn to speak, listen, read and write in English. An assessment instrument was developed by the evaluation team to measure student performance in English – speaking, listening, reading and writing (included in Annex C). The assessment was administered by both Ethiopians and Americans on the team, with the American acting largely as the recorder. • Speaking was assessed by asking the child to respond to a series of questions about himself/herself in English (e.g., What is your name? Are you a boy or a girl? How old are you? What grade are you in?). • Letter identification was conducted by asking children to name 10 letters in English (both upper and lower case). • Word recognition was conducted by children reading up to 10 words that had been drawn from the grade appropriate textbook. • Listening comprehension included a passage that one member of the team read aloud from a grade textbook, and then “wh” questions were posed to determine if the student understood the passage. • Oral reading fluency (word accuracy and phrasing) was assessed using a passage drawn from a grade appropriate textbook. • Writing competence was measured by asking the child to write three letters, write his/her name and/or one word of her/his choice. Results The children assessed had success in certain language skills (letter naming) and great difficulty in other areas (expressive oral communication). Characteristic First Graders Second Graders Comment Letter Identification 75% identified 70% 90% identified at least 70% Listening & Speaking 100% 100% One-word responses Word Recognition 20% recognized 70% 35% identified at least 70% Comprehension Not able 25% answered 60% of questions Reading Not able 50% read w/high accuracy 15% able to read One-word responses Writing 40% able to write 3 letters 100% able to write names 35% able to write a word in English 80% able to write 3 letters 100% able to write names 35% able to write a word in English Evaluation of TLMP – TLMP in Ethiopia 16 Conclusion The performance of this small sample size can be compared with students who were given the EGRA in 2010 which contained similar tasks but in mother tongue languages. The sample size in this assessment cannot be used to make generalizations, but some observations and trends are noteworthy as they mirror the EGRA findings in certain important areas: • Children seem to make progress as they move up the grades; • Children perform better at tasks that are more behaviorist than cognitive (memorization over thinking) and may reflect a preference of teaching style and pedagogy; • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension; • Children at both grades have minimal expressive language facility (e.g., none of the children spoke in complete English sentences); and • Given that test items were drawn from the student’s grade level TLMP textbook, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. 4.6 Stakeholder Interest and Use of Materials/Sustainability 4.6.1 World Bank The World Bank has taken an interest in the English for Ethiopia TLMs and has agreed to fund the reprinting of several volumes to provide them to more schools. 4.6.2 TTCs Since we only had a sample of two TTCs in our research, with one using the TLMs in their pre-service and in-service training program, and the other not having received any of the TLMs yet, no pattern was established. MOE officials had said that all TTCs had received the books, but this was erroneous. Until they are delivered to all schools and the language curriculum can accommodate them, it is difficult to determine how TTCs will use them. 4.6.3 Other Sustainability Issues The Teacher Development Directorate is concerned at this time with improving English language competency in 40,000 primary school teachers. Working with the British Council, the Directorate will roll out an English Language Improvement Program (ELIP) beginning in September 2013. Special modules and materials are being prepared for the launching of this program. It is not connected to any other teacher training program. In the 4th Education Sector Development Program 2010-2015, published by the MOE in 2010, priority areas to be addressed include improved teacher training, curriculum and textbook assessment, school improvement, and increased access to early childhood care and education, especially in rural areas and for girls. All of these foci are relevant to the TLMP, but for our purposes here, we wish to focus on the increased attention to kindergarten (KG). While the Plan stops short of incorporating KG into the MOE curriculum, schools themselves are launching KG programs to start the school socialization process, and to introduce children to reading in the mother tongue. How KG teachers will be trained to launch the socialization and reading/ language development process is not clear. 5. Lessons Learned • In designing and implementing a textbook and learning materials development program, teacher training on how to use the new materials must be included at both the pre- and in-service levels so Evaluation of TLMP – TLMP in Ethiopia 17 that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use. • Scripted lessons cannot replace a systematic professional development program, particularly when the learning outcomes are cognitive, not rote learning. • When piloting newly-developed textbooks and learning materials, teachers should be given them to use in the classroom for at least a month (better for three months) and then research should be undertaken by an independent party/publisher to determine challenges teachers faced and/or overcame in using the materials so that they can be revised before final printing. • Before launching a TLM production project, thorough research needs to be undertaken into the educational system, how teachers are trained (and for how long), the frequency and substance of in￾service teacher training programs, the level of English language competency of teachers (for English language textbooks), and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. • Before launching a language-based TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project, the learning outcomes and results achieved in using those books, and the level of language ability children have under past practices so as to avoid making assumptions about what children at particular grade levels are able to understand, speak, read and write. • Launching an innovative TLM production project works best when the various directorates of the MOE are working together in an overall learning improvement program that is not competing with other programs to create results. • Leveling any language materials is absolutely essential to creating positive learning outcomes among children; appropriate review policies, procedures and professional person power must be available to carry out these tasks. • For a US-based university to work in an African country successfully, staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts hampering long-distance communications) as well as with the donor. 6. Recommendations 6.1 Materials Development • If US-based institutions are to participate in textbook development in Ethiopia, they should: • Hire only experienced individuals in textbook development for designated grades; • Be properly introduced to the country, the culture and the practice of teaching in Ethiopia including how the education system is structured, how it functions, etc., with some insight on its history; • Have experience teaching the grades and subjects for which they will write textbooks; and • Be able to share their expertise with Ethiopians who have not developed textbooks before. • When conducting textbook field tests: • A broad array of schools must be included so that teachers working in different circumstances and with limited English capability can be included. • International project leaders must understand the cultural reluctance Ethiopians feel in criticizing another person’s work. • Teachers should be given between a month and three months after training to become familiar with the materials and the methodology, and to practice teaching before pilot/ assessment observations and interviews are held. 6.2 Systems Improvement • Any project that creates textbooks MUST: Evaluation of TLMP – TLMP in Ethiopia 18 • Include a teacher training program over the course of the project and beyond; • Include a strong M&E system that includes indicators on production, delivery and use in the classroom; • Use a continuous professional development approach; and • Ensure that the textbooks consider integration, continuity, and sequence [horizontally across other subjects in the same grade, vertically up the grades within the same cycle ie grade 1-4, and 5-8]. • A sustainable strategy for training teachers and administrators in the use of the materials is critical. This strategy should include pre-service training at TTCs, Continuous Professional Development (CPD)through the use of the cluster centers, and the training of “master trainers” who can provide initial capacity building and then be available to deliver refresher courses. If the books are to be distributed to all children in all primary schools (and that is the goal), then all teachers must be capacitated to use them over the long term. • The cascading approach can be cost-effective, but significant oversight needs to be exercised by designated focal persons or faculty members of TTCs to ensure that the training delivered is the same at all levels. • To ensure the use of TLMs, training organizers should plan to include all stakeholders in the trainings, such as supervisors/inspectors, curriculum specialist at the woreda and sub-city level, so that they can be supportive of teachers. 6.3 Towards a More Comprehensive Approach to Language Development • Language Policy: The MOE needs to clarify its own language acquisition policy and determine goals that are pedagogically sound. Building further capacity in curriculum and syllabi development within the MOE is cornerstone to this process. • Language Teaching: A more comprehensive approach to teacher training in language needs to be taken. Questions such as the following need to be addressed: • What entrance requirements should be satisfied for future teachers to be accepted into a TTC? • What will be the curriculum, especially for English, Amharic and mother tongues? • How will English language capability of all TTC entrants be improved so that English classes can be conducted in English and teachers will be able to provide English language experience to their students? • What is the best approach to teaching English and other languages – ESL, EFL, and grammar translation? • How can methodologies for teaching language be integrated across the board so that teachers can utilize similar methodologies in teaching all languages? • How can the cluster center school structure be utilized to present on-going professional development in-service training so that teachers and administrators have an opportunity to learn continuously? • Teacher Licensing: The Licensing Directorate in the MOE is working on standards that will allow primary schools to become licensed. Part of this effort includes language tests, the results of which should be used in redesigning the TTC and English language curricula in all pre-service, in-service, and other professional development activities. 6.4 Linking TLMP to USAID Priorities in Early Grade Reading High quality, well-illustrated TLMs were produced by AAMU in Ethiopia for grades 1-4. The above recommendations notwithstanding, the investment made by USAID/W and USAID/E in these materials should be an integral part of the new USAID priorities in early grade reading. The current contractor implementing this project should be encouraged to do the following: Evaluation of TLMP – TLMP in Ethiopia 19 • Determine how the English language textbooks might be edited to reflect a greater step-wise reading/English language development process, beginning with phonics (in English and the various mother tongues targeted for Ethiopia), taking students through a sequential learning process that will build upon skills developed each day. • Consider the feasibility of replicating the methodology developed by UTSA in generating TLMs through providing writing workshops to teachers and other educationalists, who then develop culturally relevant materials in the various mother tongues. The project would then take the materials through a range of leveling and pilot testing processes to ensure that children would be able to learn, in a step-wise manner, how to read in their home languages. • Consider how the supplementary reading folk tales might be translated into mother tongues, edited and leveled, to be used in English and early grade reading teaching and learning in the targeted languages. • Work with the Curriculum Directorate of the MOE to outline a realistic curriculum and syllabus for each grade level so that it can incorporate strategies for mother tongue and English language reading skills development. • Although early grade reading is the focus of this new initiative, if possible, the new contractor might also review the TLMs produced for grades 6-8 to bring them more into line with earlier skill development in English and to work to incorporate these in the curriculum as textbooks rather than supplemental materials. • Generate a strong teacher training program that integrates TTCs, woreda education offices, RSEBs and others in the teacher training planning and implementation process. Evaluation of TLMP – TLMP in Ethiopia 20 ANNEX A. SCOPE OF WORK DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID‟s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners‟ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MoE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History • TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over 3 million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings. • TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over 6 million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English. • TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers guides and training materials and trained nearly four thousand teachers on methodological classroom usage. • TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 1.8 million materials in both French and English for grades 2-10 in science, mathematics, and language arts. Over 160 teachers were trained on utilizing the materials as part of their curriculum. • TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over 6,000 teachers. The work was completed in 2009. Evaluation of TLMP – TLMP in Ethiopia 21 • TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over 1,200 teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection. Task 2 – Data Review. Task 3 – Coordination and Management. Task 4 – Site Visit. Task 5 – Data Analysis. IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives: • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based MSIs, in-country institutions, Ministries of Education (MoEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Strategy_fe b2011.pdf 6 V. OPERATING CONSTRAINTS / LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance Evaluation of TLMP – TLMP in Ethiopia 22 can be found here: http://www.usaid.gov/policy/ads/200/203.pdf The new USAID Evaluation Policy can be found here: ttp://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. ANNEX B. QUESTIONS POSED OF AAMU TLMP DIRECTOR TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did you university provide? What challenges emerged in your work with the MOE? How were they resolved? • What process did you use to review and revise the materials? How were the materials, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? Evaluation of TLMP – TLMP in Ethiopia 23 • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? Project Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the project improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school (Ghana – kindergarten)? About distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? Evaluation of TLMP – TLMP in Ethiopia 24 ANNEX C. OTHER DATA COLLECTION INSTRUMENTS In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the expertise developed in textbook production by the MOE being leveraged to obtain other, similar grants? What would prevent this from happening? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in Ethiopia? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • How satisfied were you with your relationships with Alabama A&M and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOE Administrators (triangulate with different administrators at each level) • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOE provide? How frequently? • How do the woreda inspectors assess teachers? Were they trained in the use of TLMs? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • How was the decision made to include supplementary/complementary readers in the TLMP? Who made the decision that these materials should be based in folk tales? How did the process of developing these materials differ from that used in developing the TLMs for grades 1-4 English classes (textbooks)? Was there an advantage of one approach over the other? Please explain. • What will the MOE do to continue the production of TLMs now that the project has ended? Have any other donors been found to continue this activity? What new textbook policies have been developed as a result of TLMP? Evaluation of TLMP – TLMP in Ethiopia 25 • Is the MOE ready to use its own funding in the creation of TLMs in the future? In the redevelopment and printing of the grades 6, 7, and 8 materials? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? How do you know? • How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with AAMU? How could they be improved? 3) Material Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How did the production process and personnel differ between the development of the textbooks and the development of the supplementary reading materials (folk tales)? Did one process have an advantage over the other? If so which one, and how were results different? Would you recommend one process over the other for future book production? • How often did the two US- and Ethiopia-based teams meet? What were the results of these meetings? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? What cross-cutting themes did you include? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? • How satisfied are you with the collaborative production process between yourselves and AAMU? What worked well/did not work well? What would you change to improve the process? • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? (describe both the textbooks and the supplementary reading materials) What did the MOE do (specify unit)? What did the project Evaluation of TLMP – TLMP in Ethiopia 26 do? What challenges emerged in your work with the MOE? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOE play (specify unit)? What roles did your office play? What guidelines did the MOE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Who was responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who provided the training? For how long? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Gender distribution of those who attended? Did TTC faculty attend? From which TTCs? Did university faculty attend? How many? From which universities? • What other in-service teacher training is provided by the MOE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes • What was the TLMP management structure in Ethiopia? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ethiopia Project Advisory Committee (PAC)? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of project work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with AAMU? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery Evaluation of TLMP – TLMP in Ethiopia 27 system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? • How satisfied were you with the relations established with AAMU? How could they be improved? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? • How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparity in enrollment/attendance? If so, what have the results been? • What is the linguistic background of the learners at this school? What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? At what grade does English become the language of instruction? Do you believe your students are adequately prepared in English to learn entirely in English? What needs to be done to prepare students better? Evaluation of TLMP – TLMP in Ethiopia 28 • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long? • How many of each textbook and workbook did you request for the school? How many of each text/work books did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on (cascade) what they have learned to their colleagues? Do they receive any incentive to do this? • How has the cluster center training and resource system enhanced the ability of teachers to be more learner-centered? How has the TLMP enhanced the adoption of learner-centered teaching practices? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom and Head Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree) • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching English? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLMs? Please explain. • How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP textbooks do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive the TLMs for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? Evaluation of TLMP – TLMP in Ethiopia 29 • When did you receive training on the use of TLMs? How long did it last? Did someone from the TLMP project or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this? • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1- 4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the supplementary readers? 3) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? • Why do you think you were chosen to be a trainer? Evaluation of TLMP – TLMP in Ethiopia 30 • On a 1-4 scale, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? • How familiar were you with the TLMs before you delivered the training? What materials were you provided to be a trainer? What materials did you provide to your trainees? How confident were you after the TOT that you could teach others in how to use the TLMs? What else did you need? • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs/ETHIOPIA School _________________________ Village/Town/City ________________________ Teacher Sex _____ M; _____F Grade Level: ____________________________ No. of Students: ______ M; ______ F; ______Total Languages spoken in this community: _______________________________________ Home Language/Mother Tongue of learners in the class: ________________________ Home Language/Mother Tongue of teacher: __________________________________ No. of books: _______________ No. & type of learning materials __________________ Indicator Observed Not Observed Other Teaching Using TLMs Instruction 1. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP 7. The teacher can read and explain TLM content to the students 8. Students can read and understand the subject matter in the TLMs. Evaluation of TLMP – TLMP in Ethiopia 31 Indicator Observed Not Observed Other 9. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment) 10. The teacher gives and corrects homework using the TLMs. 11. The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson 12. Students and teachers use mother tongue/English (French) when asking and responding about TLMs (circle which language) 13. Teacher demonstrates personal mastery of English TLMs/Artifact Inventory 14. Lesson objectives are written on the board in English 15. Learning aids/materials are posted in the classroom (TLMP produced and others). 16. TLMs are locked up in the cupboard. 17. Word walls display key words in English 18. Sentences appear on the chalkboard or on a chart 20. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books) 21. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment Comment_____________________________________________________________________ _____________________________________________________________________________ 10) Learners Reading Competency - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read a few sentences aloud, and then rank the performance in the following manner: 1) fluent; 2) little difficulty; 3) very haltingly; 4) can read only a few words; 5) cannot read at all. If a learner tries to read a word, identify what strategy he/she is using to read: 1) sound it out – phonics; 2) sight reading; 3) other strategy (TBD). Reading Comprehension - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read the passage silently and after they have finished, have learners explain what they have just read. Evaluation of TLMP – TLMP in Ethiopia 32 TLMP READING ASSESSMENTS/ETHIOPIA School Name ______________________________ Region ____________________ Woreda ___________________ Book 1 _____; Book 2 _____; Book 3 _____; Book 4 _____ Letter Recognition – x/10 Word Recognition – x/10 Reading Fluency Passage Comprehension Writing Grade Level Sex M/F Age Home Language KG Upper Lower Sight Words 1) Fluent; 3) Haltingly; 5) Can’t Read Can put passage into own words: 1) without difficulty; 3) with difficulty; 5) cannot put passage into own words PROMPTORS: Who? What happened? Why? How? What do you think…? Name; Any Known Word (specify) Evaluation of TLMP – TLMP in Ethiopia 33 ANNEX D. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES Date Time Organization Person Interviewed Title 3/4 4:00 Ethiopia Reads Dana Roskey Director, TESFA & ER 3/5 3:00 IBTCI Team Meeting 3/6 8:00 USAID Alison Wainer Befekadu Gebretsadik Education Chief COTR; mission activity manager for this activity 3/6 2:00 AAU/Addis Ababa Meeting at Dreamliner (schools choice) Tizazu Asare Local TLMP Coordinator (formerly MOE Head of Curriculum and Head of Planning) 3/6 3:15 MOE Girma Alemayehu Director, Curric. & Instr. 3/6 3:15 MOE (New Bldg., 121) Ejeta Negeri Head of English 3/7 8:00 MOE Ibrahim Fuad (edited TLMs) State Minister of Education (GES) 3/7 10:00 Sheraton Dr. Mary Spor and Tizazu Director, TLMP and Manager, TLMP 3/7 3:00 Dreamliner Desalegn Garsamo, Former COP AIR/Tell Project MOE, Teacher Training 3/7 4:00 IBTCI Team Planning Meeting 3/8 7:30 Tsehay Chora PS Principal, Teachers, Learners Addis Ababa 3/8 11:30 Amhara Curriculum Spec. Eyasu Aemiro From Bahr Dar MOE Ejeta Negeri FG w/Aemiro ADDIS ABABA (Team 1) AA Schools Evaluation of TLMP – TLMP in Ethiopia 34 3/11 7:30 Kotebe TTC Dean Teferi Belew Almaz Debru MT Jennifer Miller PCV English Language Faculty Addis Ababa Wondrad Cluster Center School (Kotebe) Director, English Teachers 1-4, Students Addis Ababa 3/12 7:30 SefreSalam (L) Director, English Teachers 1-4, Students Addis Ababa 3/12 1:30 AA Kolfe Kereano Sub-City Curriculum Head Addis Ababa AA RSEB Addis Ababa OTHER ADDIS ABABA 3/11 1:30 MOE Ejeta follow-up 3/11 3:00 MOE Examinations Department KefelengTsigie Deputy Head 3/11 4:00 MOE Stockkeeper Metike Head, AA 3/12 2:00 MOE Licensing Directorate Sahlu Bayissa Abaweloo 3/12 3:00 World Bank/MOE Zelalem Tadessa Procurement Contractor 3/12 3:30 MOE Curriculum Head Girma follow-up 3/13 9:00 Menelik Primary Tesfanesh Mulugeta Worku Belay English Curriculum Spec Bole High School Askalu Kifle 3rd Grade writer; master trainer 3/13 1:30 MOE Teacher Training Shetu Deputy, Teacher Development, English 3/16 7:00 VSO Volunteer Judith Althous Coordinator, Materials Evaluation of TLMP – TLMP in Ethiopia 35 Development 3/17 1:30 Graphic Designer Mulualem Fanta (Ms Wainer’s husband) Telephone Interview 3/18 9:30 MOE – RTI Reach Project Helen Boxwill Curriculum, RTI Read Project Not Held Printer Bennyam Girma Mondography Printers (TG 1&2, books, folktales) 3/14 12:30 Kotebe TTC Askalu Kifle Seife Hassan Almaz Debru Fisseha Matuma Grade 3 Textbook Writers Grade 4 Textbook Writers OROMIYA (Team 2) OROMIYA SCHOOLS 3/11 7:00 Mulugeta Gedle (L) Mulugeta Megersa Tadelech Gutema Almaz Yigletu Mulu Meseret Beyene Dechasa Demissie Senbeth Principal Vice Principal English Curriculum Spec Master Trainers 3/11 1:30 Woreda Tsehay Debele Fitassa Guluma Woreda Educ.Bureau Head Woreda Curriculum Sp. Evaluation of TLMP – TLMP in Ethiopia 36 3/12 7:30 Burayu PS Degitu Yadessa Muleta Lami Principal Vice Principal 11:00 Woreda Dirba Tafesse Birhanu Assefa Woreda Statistician Curriculum Expert 2:00 Oromiya RSEB Fite Abera REB Vice Head 3/13 7:30 Dukem #1 Chanyalew Belay Keneni Bajiga Principal Vice Principal 1:00 Woreda Mesfin Ababe Shashitu Deresa Curriculum Team Leader Curriculum Specialist 3/15 6:30 DEBRE BERHAN TTC Deputy Dean Neqnike Gebeyehu Yrsmaw, MT Tell Project English Curric. Spec and English Faculty Suzanne IFESH Volunteer Debre Berhan Cluster Center School, Atse Zeray Yacob PS Aklilu Wolde Amanuel Deputy Director 3/19 1:00 USAID DEBRIEF Alison Wainer, Befekadu, Assefa, Demissie, Warkaye Education Team Evaluation of TLMP – TLMP in Ethiopia 37 ANNEX E. MATERIALS DEVELOPMENT PROCESS - GRADES 3 AND 4 The process for generating grade 3 and 4 textbooks included the identification of teachers and teacher trainers (at TTCs) that had participated in the UK-established English Language Improvement Program (ELIP) and performed in English very well. Those chosen were exposed to the minimum learning competencies (MLC) for each grade and were then asked to write a sample lesson covering a specific topic. Those whose samples were chosen were invited to a one-half day of orientation jointly led by the TLMP Director and Coordinator, the MOE, and USAID/E. The selected writers were then participated in a three-day workshop during which they were trained in the pedagogy and format of the TLMs. The writers were then assigned different units (and/or lessons within the units) to produce as well as a model lesson. Writers were divided into two grade level teams. Each individual was assigned a specific unit. The units had already been defined by a topic flow chart for grades 1-4. Some units in each textbook were assigned to Ethiopian English as a Foreign Language (EFL) experts. Other units were assigned to primary education experts at AAMU (who did not necessarily have expertise in African education or in ESL). The American writers were expected to contribute reading pedagogy while the Ethiopian writers were expected to contribute a contextualize perspective and a sense of classroom reality in Ethiopia’s diverse education settings. American writers based at the AAMU, presumably faculty and graduate students from the school of Education, were chosen to develop and then edit materials produced in Ethiopia. It is unclear what criteria were used in the selection process of the American writers/editors and on what basis editorial decisions were made. The American and Ethiopian members of the writing team met in Addis for an initial workshop to begin drafting and critiquing draft units and lesson plans. Ethiopian writers raised concerns over the level of difficulty of the linguistic demands of the syllabus and over certain teaching methods as being too complicated for primary school teachers in Ethiopia. These concerns were apparently not addressed, nor were the format and content substantively modified. The English for Ethiopia series relies on extensive use of illustrations. Writers specified illustrations to guide student understanding and to introduce new vocabulary. These specifications were forwarded to graphic illustrators in the US and were adapted in some cases by illustrators/graphic designers in Ethiopia. Although units were developed on an individual independent basis, the Ethiopian writers convened on their own to review each other’s work and to provide suggestions. From each grade level a writer was selected to serve as a team coordinator whose main task was to keep the other writers on schedule. The Ethiopian writers would also visit the TLMP office to discuss their writing with the Coordinator during manuscript development as instructions were not always clear. A second workshop was held in Adama/Nazret. None of the AAMU writers werepresent at this event. The purpose of this workshop was to refine draft units, share experiences, and solve Evaluation of TLMP – TLMP in Ethiopia 38 problems. This was the final support to writers before submitting final drafts for compilation and review. A final workshop was held to review the completed draft textbooks and was attended only by the Ethiopian writers. Other participants included curriculum specialists from the MOE. The review focused mainly on the political and cultural issues reflected in the textbooks. Although the participants were given a guide to review the materials, this guide did not direct reviewers to provide feedback on pedagogy, readability, and content. The MOE does not appear to have a rigorous textbook evaluation process. The MOE does not have a standing review board and a systemic process for evaluating textbooks according to language, pedagogy, and learning outcomes. Materials developers worked for between 6 and 8 months, and at no time during that period were they informed of any field testing results. The developers were not aware of any specific changes that emerged from such testing (see the discussion on field testing below). The Ethiopian writers interviewed were unanimous in their opinions that participation in the TLMP was a significant professional and personal experience. One writer commented: This was my first opportunity to develop instructional materials. I learned how to develop content from the syllabus. I learned how to simplify things so they would be suitable for young children. I learned about vocabulary and how to select appropriate words for a grade level. Another pointed out how the activity affected his teaching of pre-service primary school teachers: I teach a course on materials evaluation. It was good to have this experience of developing a textbook then critiquing the structure, the content, the connection to the syllabus. I now assign my students the task of reviewing a textbook and I use some of the guidelines we were given to help my students evaluate the textbooks. Another writer commented on the personal side of the experience: It was important that we were working as a real team. We shared the work; we gave each other feedback and helped each other. We don’t usually get that chance to work together. While these anecdotes point to the value that teachers obtained from participating in the writing process, none of these teachers is directly employed at the MOE in the Curriculum Directorate. Hence, although the MOE asserted that “we have had our capacities built in textbook development,” in fact they have not: individual teachers and trainers have. Evaluation of TLMP – TLMP in Ethiopia 39 ANNEX F. TLM ASSESSMENTS 1. Student Books A. Pedagogical Content of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Alignment with Syllabus All textbooks are aligned with the syllabus, and meet the MLCs for each grade level Organization of Content Textbooks for grades 1, 3, and 4 appear to move from simple to complex, but the grade 2 text does not, with vocabulary and sentence structure too long and difficult. Correctness of Content/ Conformity with Ethiopian Culture “Lions live in the forest and prey on small animals” is erroneous, indicative of other passages. Content generally reflected urban/westernized culture and did not depict all religious and ethnic groups. In an attempt to be gender sensitive, some tasks failed to recognize that they are a part of religious culture. Integration with other Subjects & Cross-cutting Issues Only minimal attention was paid to linking content to other cross-cutting issues that would likely occur in other subject areas such as math and environmental science. Quality of Illustrations The textbooks make a good use of illustrations to guide vocabulary; the picture quality of the grade 2 text was often blurry because contrasting colors were not used. Relevance & Appropriateness for Learners and Teachers The topics were relevant and appropriate for children of typical age grades. However, the pedagogical content was not appropriate as a basic language development program (presentation seemed unstructured and unsystematic). Opportunities for Assessing Student Progress The texts provide ample opportunity for continuous performance assessment, but the format lacks opportunities for unit review and assessment. Language and Communication “While many stories and activities are innovative and engaging, they are too difficult for the English abilities of most learners and even many teachers at the third grade level.” There seems to be a mismatch between expectations of teachers’ capacity to read and model the level of English required at each grade level and unrealistic expectations about pupils’ cumulative language development. This mismatch may result in pedagogies that require the teacher to translate English content and directions into native languages, an overemphasis on the mechanical components of language (letter identification) and a learning strategy that emphasizes memorization of vocabulary and sentence frames. Evaluation of TLMP – TLMP in Ethiopia 40 Enjoyment Index Textbooks are colorful and full of color illustrations; however, the grade 1 textbooks looks like a penmanship workbook, and the content of the other grades neglected opportunities to introduce jokes, riddles, songs, etc., that might stimulate learner interests. B. Physical Design of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Cover and Paper Stock Durable and appropriate stock, but may be more durable if the textbooks were produced in two volumes rather than one to preserve the longevity of the books. Durability would be enhanced if children were provided book covers. Durability is projected as 2 years, but may be as much as 4. Bindings Durable and appropriate; should have a shelf life of 3-4 years, provided they are handled appropriately. Size and Dimension The grade 1 textbook is more of a workbook that provides children the opportunity to write letters and sound them out when a phonics approach is used. Other textbooks should be divided in two so that children and teachers can use each part each term. Quality of Print Good quality and legible; some illustrations lack sharp contrast. Typeface The sans serif font is good for grade 1 as it is clean and simple. However this typeface does not match frequently used fonts in texts (a, g, t). Structures Supportive materials to enhance children’s learning might have included a pupil picture dictionary, high frequency language phrases, and some grammar models/sentence frames. 2. Teacher Guides A. Pedagogical Quality of Teacher Guides Assessment Characteristic Team Assessment Objectives MLCs and unit objectives are presented at the opening of the unit, but are absent for each lesson and do not identify learning outcomes. The number of lessons to be covered per week is beyond the overall number of days a child is to be in the classroom over the school year. Objectives are not order, in sequence, and do not necessarily build on what has already been learned. Organization & The Guides contains lessons that are not in the student text, Evaluation of TLMP – TLMP in Ethiopia 41 Presentation resulting in an odd labeling system that is clumsy and confusing (e.g., units in the student’s text may open with lesson three). Teaching & Learning Strategies Teaching activities are designed to be interactive rather than didactic. The format uses a scripted text for teachers; however, teacher English competency is not sufficient for them to read and understand the scripts. Moreover, the scripted format will not accommodate irregularities in the pupils’ texts nor will it help teachers address children’s questions as they arise. Teaching Ideas The guides introduce or model a number of high interest techniques to promote active learning and language skill development. Referencing The guides provide extra materials to reinforce or enrich lessons, such as crossword puzzles or suggestions for making flashcards. It also lists the answers to the activities in the student book. However, the guides do not orient the teacher to resources in other books, publications or the Internet. The teacher will need a dictionary to teach the meaning of the vocabulary. B. Physical & Design Quality of Teacher Guides Assessment Characteristic Team Assessment Cover & Paper The guides are not attractive, and have no picture on the front cover. The paper quality is appropriate, durable and appears water-resistant. Binding Adequate Size & Dimension Foolscap paper was used for the guides; the number of pages is daunting, with one volume 274 pages. Page length is long due to the scripted nature of the lessons. Quality of Print Good, with a good mix of boldface and textboxes; however, there is simply too much print on each page. More white space is needed so that teachers can “bracket” some portions or write ideas/reminders in the text. Structural Elements There is a vocabulary list, but no definitions. Language patterns (sentence frames) do not provide the name of the pattern, e.g., irregular plurals, verb tenses, adjectives, etc., only an example is given. The front part of the guides do not provide useful background information for the teacher on issues such as language development, lesson planning, and assessment. This type of content assumes a language proficiency that is beyond the competency levels of most teachers. Evaluation of TLMP – TLMP in Ethiopia 42 ANNEX G. CONFLICT OF INTEREST STATEMENTS Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Nancy Horn Title Team Leader Organization IBTCI Evaluation Position [X]Team Leader Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date February 26, 2013 Evaluation of TLMP – TLMP in Ethiopia 43 Name Eric Allemano Title Organization IBTCI Evaluation Position Team Leader [X] Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date February 26, 2013 Evaluation of TLMP – TLMP in Ethiopia 44 Name Feleke Desta Title Consultant Organization IBTCI Evaluation Position Team Leader [X] Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 19, 2013 Evaluation of TLMP – TLMP in Ethiopia 45 Name Erango Ersado Title Organization IBTCI Evaluation Position Team Leader [X] Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 19, 2013 Evaluation of TLMP – TLMP in Ethiopia 46 Name Habtamu Mammo Title Organization IBTCI Evaluation Position Team Leader [X] Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 25. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 26. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 27. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 28. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 29. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 30. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 19, 2013 Evaluation of TLMP – TLMP in Ethiopia 47 Name James M. Wile Title Organization IBTCI Evaluation Position Team Leader [X] Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 31. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 32. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 33. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 34. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 35. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 36. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date January 08, 2013