December 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. The principal authors are Edward Jay Allan and Nancy E. Horn under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Volume II – Country Reports Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania 1 EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA FINAL EVALUATION REPORT VOLUME II – COUNTRY REPORTS Principal Authors Edward Jay Allan, Project Director Nancy Horn, Team Leader Other International Team Members Eric Allemano Carol Benson Thomas Tilson James Wile National Team Members Erango Ersado (Ethiopia) Feleke Desta (Ethiopia) Habtamu Mammo Ethiopia) Kofi Marfo (Ghana) Kofi Mereku (Ghana) Denis Khasu (Malawi) Hannock Mateche (Malawi) Babacar Diouf (Senegal) Alhousseynou Sy (Senegal) Rakgadi Phatlane (South Africa) Richard Chediel (Tanzania) Cathleen Sekwao (Tanzania) December 3, 2013 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP – Evaluation Report Volume II i TABLE OF CONTENTS ACRONYMS iii INTRODUCTION vi EVALUATION REPORT – ETHIOPIA Executive Summary 1 1. Evaluation Purpose, Question and Objectives 7 2. TLMP Ethiopia Project Background 8 3. Evaluation Questions, Methods & Limitations 11 4. Findings and Conclusions 13 5. Lessons Learned 22 6. Recommendations 23 ANNEX A. Questions Posed of AAMU TLMP Director 26 ANNEX B. Other Data Collection Instruments 28 ANNEX C. Schedule of Stakeholder Interviews and Activities 37 ANNEX D. Materials Development Process - Grades 3 and 4 41 ANNEX E. TLM Assessments 43 EVALUATION REPORT – GHANA Map of Southern Ghana 46 Executive Summary 47 1. Evaluation Purpose, Question and Objectives 52 2. TLMP Ghana Program Background 53 3. Evaluation Methods & Limitations 55 4. Findings and Conclusions 57 5. Lessons Learned 75 6. Recommendations 75 ANNEX A. Questions Posed in Various Interviews Conducted at CSU 78 ANNEX B. Data Collection Instruments, Ghana Stakeholders. 82 ANNEX C. School of Interviews and Meetings 87 ANNEX D. Reading Assessment Form and Evaluation of Learner Reading Competency 92 ANNEX E. Printing and Distribution – A Case Study 94 ANNEX F. Review of TLMP Pupil’s Workbooks and Teacher’s Guides 96 ANNEX G. The Original 14 TLMP Districts 116 ANNEX H. Workbooks and Instructional Materials Prepared by TLMP 2009-2012 117 EVALUATION REPORT – MALAWI Executive Summary 118 1. Evaluation Purpose, Question and Objectives 122 2. TLMP Malawi Project Background 123 3. Evaluation Questions, Methods & Limitations 125 4. Findings and Conclusions 126 5. Lessons Learned 137 6. Recommendations 138 ANNEX A-1. Questions Posed In Various Interviews Conducted At UTSA 141 ANNEX A-2. Data Collection Instruments, Malawi Stakeholders 144 ANNEX B. Schedule of Stakeholder Interviews and Activities 154 Evaluation of TLMP – Evaluation Report Volume II ii ANNEX C. Reading Assessment Methodology 157 EVALUATION REPORT - SENEGAL Executive Summary 159 1. Evaluation Purpose, Question and Objectives 163 2. TLMP Senegal Project Background 164 3. Evaluation Methods & Limitations 166 4. Findings and Conclusions 168 5. Lessons Learned 176 6. Recommendations 177 ANNEX A. The Senegalese School System 179 ANNEX B. Data Collection Instruments 180 ANNEX C. Master Data Collection Instruments 181 ANNEX D. Schedule of Stakeholder Interviews and Activities 190 ANNEX E. Summary of Findings 202 ANNEX F. Evaluation of Selected TLMs 204 ANNEX G. List of Clarifications on the TLMs for Possible MOE Distribution 206 EVALUATION REPORT – SOUTH AFRICA Executive Summary 207 1. Evaluation Purpose, Question and Objectives 210 2. TLMP South Africa Project Background 211 3. Evaluation Questions, Methods & Limitations 214 4. Findings and Conclusions 215 5. Lessons Learned 220 6. Recommendations 221 ANNEX A: Map of Limpopo Province 224 ANNEX B: Schedule of Stakeholder Interviews and Activities 225 EVALUATION REPORT – TANZANIA Executive Summary 227 1. Evaluation Purpose, Question and Objectives 232 2. TLMP Tanzania Project Background 232 3. Evaluation Questions, Methods & Limitations 236 4. Findings and Conclusions 238 5. Lessons Learned 248 6. Recommendations 249 ANNEX A. Questions Posed of SCSU Project Team 251 ANNEX B. Schedule of Data Collection Activities 253 ANNEX C. Data Collection Instruments 256 ANNEX D. Mathematics Example of Adaptation Process 265 ANNEX E. Textbook Reviews 266 Evaluation of TLMP – Evaluation Report Volume II iii ACRONYMS AAMU Alabama Agricultural & Mechanical University ADEA Association for the Development of Education in Africa ADP Area Development Program AEI President’s African Education Initiative AERA American Education Research Association AIR American Institutes for Research APS Annual Program Statement CA Cooperative Agreement CAPS Curriculum and Assessment Policy Statement CD Curriculum Director COR Contracting Officer Representative CPD Continuous Professional Development CRDD Curriculum Research and Development Division CRECCOM Creative Center for Community Mobilization CSU Chicago State University DBE (South Africa) national Department of Basic Education DEO District Education Office/Officer DOE Department of Education (provincial) ECSU Elizabeth City State University EFA Education for All EFL English as a Foreign Language EGRA Early Grade Reading Assessment ELIP English Language Improvement Program EMAC Educational Materials Approval Committee ESL English as a Second Language GER Gross Enrollment Rate GES Ghana Education Service HBCU Historically Black Colleges and Universities HUB Historically Underutilized Businesses IA Inspection de l’Académie (Regional Education Office, Senegal) IBTCI International Business & Technical Consultants, Inc. IBTCI International Business & Technical Consultants, Inc. IDEN Inspection Départementale de l’Education Nationale (Departmental Education Office, Senegal) IFESH International Foundation for Education and Self Help INEADE Institut National d’Etude et d’Action pour le Développement de l’Education (National Institute for Applied Research in the Development of Education) INSET In-service Teacher Training Ithuba Ithuba [“Opportunity”] Writing Project KG Kindergarten KZN KwaZulu-Natal L1 First language, mother tongue L2 Second language LEA Language Experience Approach LIEP Language-in-Education Policy Evaluation of TLMP – Evaluation Report Volume II iv LOLT Language of Learning and Teaching LT Lead Teacher M&E Monitoring and Evaluation Manuel School textbook (aligned with/representing the official curriculum) Manuel d’appoint Reference book (supporting but not aligned with the curriculum) MEN Ministère de l’ Education Nationale (Ministry of Education, Senegal) MIE Malawi Institute of Education MLC Minimum Learning Competencies MOE Ministry of Education MOEST (Malawi) Ministry of Education, Science and Technology MOEVT Ministry of Education and Vocational Training Molteno Molteno Institute for Language and Literacy MOU Memorandum of Understanding MRALG Ministry of Regional Administration and Local Government MSI Minority Serving Institution MT Master Trainers MTPDS Malawi Teacher Professional Development Support (Program) MTPDS Malawi Teacher Professional Development Support program NALAP National Acceleration Literacy Program NCS National Curriculum Statement NECTA National Examination Council of Tanzania NER Net Enrollment Rate NESP (Malawi) National Education Sector Plan 2008-2017 NGO Non-governmental organization NIH National Institutes of Health NPC (Malawi) National Primary Curriculum NQATF National Quality Assurance Task Force OUP Oxford University Press P1-P3 Primary 1 - Primary 3 PAC Program Advisory Committee PCAR (Malawi) Primary Curriculum and Assessment Reform PE Program Evaluation PEA (Malawi) Primary Education Advisors PMP Performance Monitoring Plan PTA Program Technical Assistance PTA Parent-Teacher Association PTL Program Team Leader READ READ Educational Trust REO Regional Education Office/Officer RFP Request for Proposal RM Read Malawi RSEB Regional State Education Bureau RTI Research Triangle Institute S1-S3 (Malawi) Standard 1-Standard 3 SCSU South Carolina State University SIPS Société Industrielle de Papeterie au Sénégal Evaluation of TLMP – Evaluation Report Volume II v SOSA (Malawi) The name given to Book One, Standard 1, Chichewa SSA Sub-Saharan Africa SVT Sciences de la Vie et de la Terre (Life and Earth Sciences, Senegal) TDC Teacher Development Center TELL Teach English for Lifelong Learning TIE Tanzania Institute of Education TL Team Leader TLM Textbooks and Learning Materials TLMP Textbook and Learning Materials Program TOT Training of Teachers TPR Total Physical Response TT Teacher Training TTC Teacher Training College UEW University of Education, Winneba UL University of Limpopo (now the University of the North) UM Uhuru Media UNICEF United Nations Children’s Fund UNICEF United Nations International Children’s Fund UP University of Pretoria USAID United States Agency for International Development USAID/E United States Agency for International Development/Ethiopia USAID/G United States Agency for International Development/Ghana USAID/M United States Agency for International Development/Malawi USAID/S United States Agency for International Development/Senegal USAID/SA United States Agency for International Development/South Africa USAID/T United States Agency for International Development/Tanzania USAID/W United States Agency for International Development/Washington UTA University of Texas, Austin UTSA University of Texas, San Antonio VSO (historically) UK Voluntary Service Overseas WASH Water, Sanitation, and Hygiene Evaluation of TLMP – Evaluation Report Volume II vi INTRODUCTION Volume II is the companion volume to the main Project Report, Volume I, and contains the individual reports on the implementation of the expansion phase of TLMP in each of the six countries involved. The country reports all follow the same basic structure but with adjustments to reflect the varied circumstances of each implementation and also the “learning curve” of the members of the team as commonalities and important variations in the different implementations became clearer over the course of the evaluation. To reduce the risk of inter-rater variability, each evaluator (with the exception of Dr. Allan, who worked only on the evaluation of the South Africa ITHUBA project) served on at least two country teams and with different international team members. The final schedule for country visits was: 1. Jan 26-Feb 9 Senegal Eric Allemano, Carol Benson 2. Feb 17-March1 Tanzania Jim Wile, Carol Benson 3. Feb 17-March 2 Malawi Nancy Horn (Team Leader), Tom Tilson 4. March 3-19 Ethiopia Nancy Horn, Eric Allemano, Jim Wile 5. May 5-16 South Africa Ed Allan (Project Director), Carol Benson 6. June 9-20 Ghana Tom Tilson, Eric Allemano The international members of the TLMP evaluation project are very appreciative of the solid contributions, personal as well as professional, made by our host country evaluation colleagues. We are also all appreciative of the time, energy, and support provided throughout by staff of each of the Missions involved; staff of the MSIs both at their home institutions and in the countries where TLMP worked and the host country organizations and individuals who collaborated in producing so many materials; and the many, many educators in each country at all levels of the education systems, from classroom level to national MOE, whom our team members met. And please know that we are most grateful for having had the opportunity to meet with so many of the children and youth who benefitted from all of the TLMP materials developed. Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania EVALUATION REPORT - ETHIOPIA This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Nancy Horn, Eric Allemano, Feleke Desta, Erango Ersado, Habtamu Mammo, and James Wile under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Intentionally left blank TLMP Evaluation Report Volume II - Ethiopia 1 EVALUATION REPORT - ETHIOPIA EXECUTIVE SUMMARY Objectives In 2005 Alabama Agricultural and Mechanical University (AAMU) was awarded a Cooperative Agreement (CA) to implement the Textbook and Learning Materials Program (TLMP) in Ethiopia to generate English language textbooks for grades 6-8. Subsequently, in 2009, the CA was extended to generate English language textbooks for grades 1-4 and supplemental reading materials (folktales) for the same grades. The in-country evaluation of this program took place in March 2013, and had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based Minority Serving Institutions (MSIs), in-country institutions (Ministries of Education (MOEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. AAMU’s vision for TLMP was to build capacity in Ethiopia not only to increase the number of textbooks that children use, but also to enhance the teaching and learning process by integrating the most effective instructional methods into books and teacher’s guides. Collaboration between AAMU and the MOE,, and selected educationalists brought about a number of improvements in English language instruction in grades 1-4. TLMP activities in Ethiopia were to include: • Conducting a new needs assessment;1 • Writing TLMs with Ethiopian educators in accordance with national curricula that are culturally, socially, and academically appropriate and that challenge gender stereotypes; • Field testing all TLMs in Ethiopian rural and urban schools to determine understandability, effectiveness, and cultural appropriateness; • Revising and editing field-tested TLMs based upon the results of surveys and other field data collection; • Collaboratively reviewing camera-ready copy of all TLMs with all relevant partners; • Assisting the MOE in developing a distribution and delivery strategy; and • Conducting an impact study on the use of TLMs (not done). TLMP in Ethiopia actually had two parts: 1) production of English language textbooks and teacher’s guides for grades 2, 3, and 4 (and the finalization of grade 1 materials); and 2) the production of supplementary reading materials through a participatory writing process (similar to the materials development process used by University of Texas, San Antonio (UTSA) in South Africa and Malawi). For this latter activity, the Curriculum Directors (CDs) from all the regions were invited to participate with university and Teacher Training Colleges (TTC) faculty (this was the very first time that all of the 1 A second needs assessment was not conducted because the AEI evaluation (of Phase 1 of the project in all African countries) served as a situation analysis and made several recommendations as to the way forward. TLMP Evaluation Report Volume II - Ethiopia 2 CDs came together to work on a common set of materials). They created anthologies for grades 1-2 and 3-4, teacher’s guides, and two “big books” with stories for grade 1 and grades 1-2. The big books are meant for teachers to read aloud in the classroom. The educational policy and structural environment in which TLMP operated included a system in which a Federal MOE created the structure of education as well as curriculum, syllabi, and the development of learning materials. However, the nine Regional State Education Bureaus (RSEBs) and two City Administrations play a role, especially in early grade education in each mother tongue. TTC relates closely to both the Federal and regional/city MOEs as the language of instruction is supposed to be English, but teachers also require language education in the mother tongue to be effective in their classrooms. The system is struggling to cope with all the language acquisition needs as well as the content knowledge needs of teachers who teach in self-contained classrooms in grades 1-4. Part of this problem will be addressed in the development of teachers specialized in language education, math and science, and ethics. Evaluation Methodology A six-person team was assembled (three Americans and three Ethiopians, each with expertise in TLMP￾related areas) to conduct a literature review, individual interviews with senior Federal and regional MOE officials, interviews with school principals and teachers, observations of teachers using the TLMs and assessments of children’s ability to read and understand the TLMs. The team was divided into two, with one sub-team concentrating on research in Oromia and one in the Addis Ababa environs. The whole team conducted research in Amhara. Findings TLMP Output and Dissemination • Grade 1 – 4.3 million materials prepared (TLMP paid for 1.5 million student books and 25,000 teacher’s guides; the rest was paid for by USAID/Ethiopia (USAID/E), • Grade 2 – 1,700,284 student books and 50,000 teacher’s guides (TLMP paid), • Grades 3 and 4 – 45,000 teacher’s guides at each grade level (TLMP paid) (USAID/E printed 5.5 million student books for grades 2, 3, and 4), and • Folktale books – 700,000 anthologies and big books and 50,000 teacher’s guides (TLMP paid). The dissemination of the TLMs was highly irregular and inconsistent. The grade 1 materials were distributed in 2009, but grade 2 materials were distributed only in September 2012 and grades 3 and 4 materials had just been distributed before this evaluation began in March 2013. Project Management and Partnerships During the 2009-2012 period of implementation (Phase 2), AAMU hired a local coordinator. The bulk of the development activities had been completed by 2011; the remainder of the work required timely decision-making and oversight. At AAMU, the Director assembled a management team composed of a program coordinator, an office manager and herself. She also hired an illustrator who had been engaged in Ethiopia, but who then moved to Alabama to attend school. However, no individual was designated to develop a formal Monitoring and Evaluation (M&E) system and to track the production of the materials as well as the progress of teachers and students. A major partner to TLMP was USAID/E itself. Initially, USAID/E’s role was to liaise with the MOE and to help establish and maintain the relationship between AAMU and the MOE. Since USAID/Washington (USAID/W) had not included training on the use of the TLMs in the award, USAID/E provided funding TLMP Evaluation Report Volume II - Ethiopia 3 to the American Institutes for Research (AIR) under the “Teach English for Lifelong Learning” (TELL) project to provide training to 50,000 English teachers. The actual training that TELL provided was met with challenges: • The budget was limited (although USAID/E offered what it could); • No pre-workshop assessment was undertaken to determine the English language capability of teachers (high school or primary school); • One-off training sessions were held for only 3.5 days; • There was no classroom follow up; • No provision was made for refresher training; and • The content of the training did not fully address the new pedagogy being introduced. Program Implementation Materials Improvement - Teachers and other educators who had attended 220 hours of classes offered in the English Language Improvement Program (ELIP) constituted the pool of individuals qualified to participate in the development of the TLMs. After attending three workshops, writers were able to write textbook lessons (or skeletons thereof) and/or folk tales that would constitute supplementary reading materials. Materials were then reviewed by the Ethiopian Coordinator, sent to AAMU for editing and illustrations, and then sent to the MOE for review. The MOE does not have a specific textbook review board nor was it aware of the leveling that was needed to make the TLMs accessible for teachers and students. It is not clear what criteria were used for these final assessments and edits. Assessment of TLMs - Overall, the TLMs produced under this initiative were far superior to the materials they replaced in terms of structure, print quality, and pedagogy. The TLMs reflect the MOE’s Minimum Learning Competencies (MLCs) for each grade level and were generally well-illustrated with color drawings. In general, all materials produced were in alignment with the MOE syllabi. In terms of the student textbooks, not all content moved evenly from simple to complex, and – taking the materials as a graded set – a “linguistic cliff” had to be climbed between the grades 1 and 2 materials. Content correctness and relevance to Ethiopian culture in certain instances was erroneous, and only minimal attention was paid to cross-cutting themes. The illustrations were numerous and helpful, although sometimes choice of color made picture detail difficult to distinguish. Topics were relevant for each grade, although pedagogical content seemed unsystematic. Opportunities for continuous assessment are included, but there is no specific instruction on how this will be done. The language used is far above what children (and teachers) are capable of understanding. The colorful presentation and large number of illustrations are very attractive and children want to look at the books. While the physical quality of the books is generally very durable, the size is cumbersome for children and should be divided by terms. In terms of the teacher’s guides, too many lessons are included under each unit and each lesson does not have its own objectives. Often there is a mismatch between what is in the guide and what is in the textbook. Sequencing is often a problem. Scripted lessons are presented for each topic, but the language used is beyond the capability of most teachers at the grades 1-4 level. Many new teaching ideas are presented, but these form a new, non-didactic, participatory approach to teaching to which teachers have not been introduced. There is significant cross referencing to other materials to be used, but teachers do not have much experience in using a range of materials to teach one subject in a classroom. The size of the guides is cumbersome, the text is very dense, there are no illustrations, and some explanations are missing; there is no glossary or other helpful drills that can be constructed in teaching different language acquisition skills. TLMP Evaluation Report Volume II - Ethiopia 4 In the development of the materials, there was significant ambiguity in what was needed. The indecisiveness of the MOE as to which approach to take emerged in the materials and in the confusion that teachers and trainers felt in teaching the materials. Overall, the teacher’s guides were composed of scripted lessons. However, using this technique as an approach to overcoming teacher shortcomings is not viable. Teacher Training - In general, when new teaching materials are introduced, training on how to use them is critical. Although USAID/E graciously found the means to fund a project that trained teachers during both phases of TLMP, the model used was inadequate to meet the needs of the teachers. One￾off training left teachers confused about how to proceed largely because of their own English language inadequacies in being able to understand the new teaching methodology inherent in the teacher’s guides. Teacher Observations - There is a sufficient number of deficiencies and “not observed” comments to warrant a more comprehensive assessment of teacher ability to use the TLMs effectively and to conduct more rigorous training in the future. Learner Reading Assessment - The performance of a small sample of 10 cannot be used to make generalizations, but some observations and trends are noteworthy as they mirror the Early Grade Reading Assessment (EGRA) findings in certain important areas: • Children seem to make progress as they move up the grades; • Children perform better at tasks that are more behaviorist than cognitive (memorization, over￾thinking) and may reflect a preference of teaching style and pedagogy; • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension; • Children at both grades have minimal expressive language facility (e.g., none of the children spoke in complete English sentences); and • Given that test items were drawn from the student’s grade level TLMP textbook, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. Lessons Learned • In designing and implementing a TLM development program, teacher training on how to use the new materials must be included at both the pre- and in-service levels so that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use. • Scripted lessons cannot replace a systematic professional development program, particularly when the learning outcomes are cognitive, not rote learning. • When piloting newly-developed TLMs, teachers should be given them to use in the classroom for at least a month (better for three months), and then research should be undertaken by an independent party/publisher to determine challenges teachers faced and/or overcame in using the materials so that they can be revised before final printing. • Before launching a TLM production project, thorough research needs to be undertaken into the educational system: how teachers are trained (and for how long), the frequency and substance of In￾Service Teacher Training (INSET) programs, the level of English language competency of teachers (for English language textbooks), and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. • Before launching a language-based TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project: the learning outcomes and results achieved TLMP Evaluation Report Volume II - Ethiopia 5 in using those books, and the level of language ability children have under past practices, so as to avoid making assumptions about what children are able to hear, speak, read and write. • Launching an innovative TLM production project works best when the various directorates of the MOE are working together in an overall learning improvement program that is not competing with other programs to create results. • Leveling any language materials is absolutely essential to creating positive learning outcomes among children; appropriate review policies, procedures and professional person power must be available to carry out these tasks. • For a U.S.-based university to work in an African country successfully, the staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts hampering long-distance communications) as well as with the donor. Recommendations Materials Improvement • If U.S.-based institutions are to participate in textbook development in Ethiopia, they should: • Hire only experienced individuals in textbook development for designated grades; • Be properly introduced to the country, the culture, and the practice of teaching in Ethiopia; • Have experience teaching the grades and subjects for which they will write textbooks; and • Be able to share their expertise with Ethiopians who have not developed textbooks before. • When conducting textbook field tests: • A broad array of schools must be included so that teachers working in different circumstances and with limited English capability can be included. • International project leaders must understand the cultural reluctance Ethiopians feel in criticizing another person’s work. • Teachers must be given between a month and three months after training to become familiar with the materials and the methodology, and to practice teaching before pilot/ assessment observations and interviews are held. Systems Improvement • Any project that creates textbooks MUST: • Include a teacher training program over the course of the project and beyond; • Include a strong M&E system that includes indicators on production, delivery and use in the classroom; and • Use a continuous professional development approach. • A sustainable strategy for training teachers and administrators in the use of the materials is critical. This strategy must include pre-service training at TTCs, continuous professional development through the use of the cluster centers, and the training of “master trainers” who can provide initial capacity building and then be available to deliver refresher courses. If the books are to be distributed to all children in all primary schools (and that is the goal), then all teachers must be capacitated to use them over the long term. • The cascading approach can be cost effective, but significant oversight needs to be exercised by designated focal persons or faculty members of TTCs to ensure that the training delivered is the same at all levels. • To ensure the use of TLMs, training organizers should plan to include all stakeholders in the trainings, such as supervisors/inspectors, and curriculum specialists at the woreda and sub-city level so that they can be supportive of teachers. TLMP Evaluation Report Volume II - Ethiopia 6 Towards a More Comprehensive Approach to Language Development • Language Policy: The MOE needs to clarify its own language acquisition policy and determine goals that are pedagogically sound; building further capacity in curriculum and syllabi development within the MOE is cornerstone to this process. • Language Teaching: A more comprehensive approach to teacher training in language needs to be taken. Questions such as the following need to be addressed: • What entrance requirements should be satisfied for future teachers to be accepted into a TTC? • What will be the curriculum, especially for English, Amharic and mother tongues? • How will English language capability of all TTC entrants be improved so that English classes can be conducted in English, and teachers will be able to provide English language experience to their students? • What is the best approach to teaching English and other languages – ESL, EFL, and grammar translation? • How can methodologies for teaching language be integrated across the board so that teachers can utilize similar methodologies in teaching all languages? • How can the cluster center school structure be utilized to present on-going professional development in-service training so that teachers and administrators have an opportunity to learn continuously? • Teacher Licensing - The Licensing Directorate in the MOE is working on standards that will allow primary schools to become licensed. Part of this effort includes language tests, the results of which should be used in redesigning the TTC and English language curricula in all pre-service, in-service, and other professional development activities. Linking TLMP to USAID Priorities in Early Grade Reading High quality, well-illustrated TLMs were produced by AAMU in Ethiopia for grades 1-4. The above recommendations notwithstanding, the investment made by USAID/W and USAID/E in these materials should be an integral part of the new USAID priorities in early grade reading. The current contractor implementing this project should be encouraged to do the following: • Determine how the English language textbooks might be edited to reflect a greater step-wise reading/English language development process, beginning with phonics (in English and the various mother tongues targeted for Ethiopia), taking students through a sequential learning process that will build upon skills developed each day. • Consider how the methodology developed by the UTSA is generating TLMs through providing writing workshops to teachers and other educationalists, who then develop culturally relevant materials in the various mother tongues. The project would then take the materials through a range of leveling and pilot testing processes to ensure that children would be able to learn, in a step-wise manner, how to read in their home languages. • Consider how the supplementary reading folktales might be translated into mother tongues, edited and leveled, to be used in English and early grade reading, teaching, and learning in the targeted languages. • Work with the Curriculum Directorate of the MOE to outline a realistic curriculum and syllabus for each grade level so that it can incorporate strategies for mother tongue and English language reading skills development. • Although early grade reading is the focus of this new initiative, if possible, the new contractor might also review the TLMs produced for grades 6-8 to bring them more into line with earlier skill development in English and to work to incorporate these in the curriculum as textbooks rather than supplemental materials. TLMP Evaluation Report Volume II - Ethiopia 7 • Generate a strong teacher training program that integrates TTCs, woreda Education Offices, RSEBs, and others in the teacher training planning and implementation process. 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES 1.1 The Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched by the United States Agency for International Development (USAID/W) in 2005 in conjunction with the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort to improve the management capacity of education sector personnel in a number of countries in Sub-Saharan Africa. TLMP was extended for another three years in 2008/09 to 2012. Each of five Minority-Serving Institutions (MSIs) was responsible for managing and implementing the TLMP in a specific country and with achieving specific outputs and results in accordance with its respective Cooperative Agreement (CA); Alabama Agricultural and Mechanical University (AAMU) was first awarded a CA to work in Ethiopia in 2005- 2008/09 on Textbook and Learning Materials (TLMs) for grades 6 through 8, and was awarded a second CA to generate TLMs for grades 1 through 4 in Ethiopia 2009-2012. Each MSI was responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s MOE and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 1) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA; 2) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program; and 3) Ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). Originally a program to develop TLMs for primary schools only, MSIs conducted initial needs assessments and determined that in three countries MOE priorities were for middle and secondary school TLMs. In Ethiopia, the MOE determined that English language textbooks should be developed for grades 1-4. To implement the project, AAMU (and their fellow MSIs) received about the same level of funding - approximately US$8 million. 1.2 Evaluation Objectives This performance evaluation covered work led by AAMU in Ethiopia during the period 2009-2012 and was intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets, measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and TLMP Evaluation Report Volume II - Ethiopia 8 • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. 2. TLMP ETHIOPIA PROJECT BACKGROUND 2.1 AAMU Background Research AAMU’s vision for TLMP was to build capacity in Ethiopia not only to increase the number of English language textbooks that children use, but also to enhance the teaching and learning process by integrating the most effective instructional methods into books and teacher’s guides. Collaboration between AAMU and the MOE and selected educators brought about a number of improvements in English language instruction in grades 1-4. The decision was made by the MOE for TLMP to produce TLMs in English in Ethiopia as it is the official international language and is taught beginning in grade 1. The goals of TLMP during 2009-2012 included: • Conducting a new needs assessment;2 • Writing TLMs with Ethiopian educators in accordance with national curricula that are culturally, socially, academically appropriate and that challenge gender stereotypes; • Field testing all TLMs in Ethiopian rural and urban schools to determine understandability, effectiveness, and cultural appropriateness; • Revising and editing field tested TLMs based upon the results of surveys and other field data collection; • Collaboratively reviewing camera-ready copy of all TLMs with all relevant partners; • Procuring bids for printing; • Assisting the MOE in developing a distribution and delivery strategy; and • Conducting an impact study on the use of TLMs (not done). TLMP 2009-12 in Ethiopia actually had two parts: 1) production of English language textbooks and teacher’s guides for grades 2, 3, and 4 (and the finalization of grade 1 materials); and 2) the production of supplementary reading materials through a participatory writing process (similar to the materials development process used by UTSA in South Africa and Malawi). For this latter activity, the Curriculum Directors (CDs) from all the regions were invited to participate with university and Teacher Training College (TTC) faculty; this was the very first time that all of the CDs came together to work on a common set of materials. They created anthologies for grades 1-2 and 3-4, teacher’s guides, and two “big books” with stories for grade 1 and grades 1-2. The “big books” are meant for teachers to read aloud in the classroom. Severe difficulties in implementation began in the last two years of the project. For example, TLMP staff had not been paid since August 2012, the printing of the supplementary reading materials was stopped due to lack of payment, and the small administrative office that was opened in Addis Ababa was closed. While interviews were held with all other MSIs prior to field research, our Contracting Officer Representative (COR) advised us not to conduct campus interviews with AAMU staff. However, a telephone interview was held with the Project Director at her new place of employment. 2.2 MOE Context and the Educational System Ethiopia is a Federal Republic composed of nine Regional States and two City Administrations. The Federal MOE in Addis Ababa determines the overall curriculum, facilitates the development of 2 This second needs assessment was not conducted because the AEI evaluation (of Phase 1) served as a situation analysis and made several recommendations as to the way forward. TLMP Evaluation Report Volume II - Ethiopia 9 textbooks, produces syllabi for each subject in each grade, determines the type and duration of pre￾service teacher training, promotes In-service Teacher Training (INSET), determines the criteria for licensing (in progress), and other similar activities. Each Regional State and City Administration has its own Regional State Education Bureau (RSEB) or equivalent that has oversight over such things as examinations and learning assessments at grades 4 and 8 (administered in the predominant mother tongue of the region), certain curriculum regulations, determining at what grade English is taught as a subject and at what grade it becomes the language of instruction. Some lines of authority are not entirely clear and this has given rise to a number of curricular ambiguities as well as challenges in the types of materials to be used in teaching. English is taught as a subject beginning in grade 1 throughout the country. In two regions/city administrations, English is used as the language of instruction beginning in grade 5; in seven regions, at grade 7; and in two regions at grade 9. Textbooks are not available in all languages and TTC are hard￾pressed to train teachers in all of these languages. (Teachers, in the past, could be assigned to schools in any region; their ability to teach the local language was not necessarily a criterion. However, according to officials at the MOE, this is slowly changing in an effort to create language-teaching specialists.) Both the national school leaving examinations in the 10th and 12th grade are in English. The school-age population in 2008 was 19,573,771. Of these, 15,340,786 (78%) were in primary schools: 79.5% in rural schools and 20.5% in urban schools. Boys comprised 53% and girls 46.5% of the student population.3 Since Education for All (EFA) was adopted, public education at the primary level became free, although it is not compulsory, and the number of students entering school has “exploded,” especially for girls. Many rural, school-age students are not enrolled due to: distance of homes from schools; opportunity cost of child labor; poor health of the children and/or parents; and poverty. These same causes have created a high drop-out rate, in particular between grades 1 and 2, with a very heavy drop-out rate of females at grade 4 (the end of first cycle primary that generally coincides with the onset of menses), as reported in the AEI evaluation. Large class size, a severe lack of textbooks, and the level of training that teachers receive all contribute to very difficult teaching conditions. The textbook – student ratio ranges in urban areas between1:1 to 1:6, but can be as high as 1:100 in the rural areas. Complicating this structure are the differences in the ability of teachers to teach in a self-contained classroom, which is the norm for grades 1-4, and the fact that many schools operate on a shift system. This structure requires that teachers teach all subjects to children according to the curriculum and timetable. With language instruction taking no less than 10 class periods a week (mother tongue and English; and more if Amharic as an additional language is included), and with the instructional day no longer than three hours for grades 1 and 2, and no longer than four hours for grades 3 and 4, a teacher must be well-trained to cope with the number of children in a classroom as well as deal with the range of subjects that must be taught in the time allotted. The Federal MOE is moving toward teacher specialization in three areas – language, math and sciences, and ethics. When fully operational, children will be taught languages (mother tongue, Amharic and English) by language specialists, math and sciences by specialists in these areas, and ethics in the mother tongue by specialists in this field. Other rurally￾based problems include: • Teachers are often absent from class, especially during harvest time; and • Instructional radio broadcasts are now conducted in the mother tongue in each region, so they are not repeated as frequently as when they were all in Amharic or English; equipment is in disrepair. 3 During our research, we noted that in all six primary schools in which data was collected, the enrollment of girls in all grades was higher than that for boys, sometimes by as much as 200%; this is most likely the result of adoption of the Education for All (EFA) policy. TLMP Evaluation Report Volume II - Ethiopia 10 2.2.1 Teacher Training and the Languages of Instruction The teacher training system has only recently been stabilized, according to MOE informants. Approximately ten years ago, TTC entrance requirements included a pass in the 12th grade leaving exam; TTC attendance covered only one year. Thereafter, because an insufficient number of teachers were being trained to meet the demand, admission requirements were lowered to include those who passed the 10th grade leaving exam (having earned only a 2.0 grade); students were then trained for one year at the TTC. Subsequent MOE research, however, indicated that TTC students did not have enough subject matter knowledge to teach in self-contained classrooms; hence, the period for training at a TTC was extended to three years, or to the diploma level. The Federal MOE is providing the diploma￾qualifying learning opportunity to teachers already in service through instruction given on weekends and break times. It is not clear what the content of the training is. Complicating the teacher training process is the language of instruction at each TTC. Although English is supposed to be the language of instruction in learning institutions from grade 9, even the TTCs find it difficult to teach in English. Moreover, the regional states and cities where the TTCs are located have different languages of instruction in their respective primary schools. For example, the TTC in Hawassa in the Southern Nations, Nationalities, and Peoples Regional State (SNNPR) focuses on Sidama as a mother tongue. Since materials in this language are quite sparse, faculty and students face daunting challenges as a number are themselves not fluent in Sidama. The language and literacy program at most TTCs begins with mother tongue skill building, and then goes on to English and Amharic. Reading in all of these languages can be taught using a phonics approach, but it is not clear that this is what is done. Evidence at schools demonstrated a phonics approach in teaching children how to read in Afan-Oromo, but the teaching methods are not utilized across the language curricula, i.e., the same phonics-based decoding skills are not used to teach reading in each language. 2.2.2 Systemic Textbook Issues There are several discrepancies between textbook development and examinations, with the latter department not being forewarned of a new textbook in a particular subject matter. For example, we were informed that in one instance the Examinations Directorate was given an advance copy of a new textbook so as to include the subject matter on the examination. At examination time, the students were in an uproar because the materials covered on the examination had not been addressed in class because the textbooks had not been delivered. In interviews conducted with the Curriculum Directorate, we learned that the former Prime Minister had issued a directive that textbooks were to be distributed on a 1:1 ratio of children to books. When the TLMP could not deliver all the materials in a year’s time, an international tender was sent out not only for English books for grades 5-8, but also for other subjects. Despite the rush to get the materials developed, printed, and distributed, it still took two years to complete the process. We were also informed that the textbooks produced were not as acceptable as those developed by TLMP. 2.2.3 Issues Associated with Approaches to Teaching Literacy The MOE has also included a number of reading and language programs in its curricula, some of which are based on phonics in the earliest grades and others that take more of a whole language approach. With the launch of the new early grade reading project (READ), yet another approach will be developed not only for mother tongue language and literacy development, but also for English. It is not clear whether the grades 1-4 TLMP materials will go the same way as the TLMP materials for grades 6-8, which are now in the category of “supplemental” materials. TLMP Evaluation Report Volume II - Ethiopia 11 The MOE is searching for the best way it can address language and literacy development in several languages. Every five years, the MOE undergoes a curriculum review. At the time TLMP was launched (AEI in 2005 and TLMP in 2009), reform had just taken place. Prior to 2009, the MOE decided that the whole language approach was to be used in teaching English, although the syllabi that emerged could easily have adopted an English-as-a-Second Language (ESL) approach. The syllabi had been produced by an external consultant, meaning that a full understanding of what Ethiopian teachers and children need to acquire language and reading skills in several languages using different orthographies may not have been explored fully. The Curriculum Directorate has determined that the whole language approach is not producing the desired results, and the next curriculum revision – the team was told -- will chart a more comprehensive course in language and reading education. For this to take place, however, the MOE must have a deeper understanding of the purpose for the current language programs, design and implement an overall language policy, and create the curriculum and syllabi that best meets the new policy. In conclusion, the TLMP activity in Ethiopia was not necessarily about teaching reading. It was primarily an English language program into which USAID/E introduced reading pedagogy ahead of its new primary grades reading initiative. The ambiguity as to whether this was an English language, reading, or ESL/EFL program is demonstrated in the syllabi as well as in the TLMs produced. 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS The evaluation questions are listed in Section 1.2 above. The evaluation of TLMP in Ethiopia made use of a “mixed-methods” methodology that included: review of project documents; analysis of the TLMs produced; key informant interviews with AAMU’s former director, Ethiopian educators involved with the project, Ethiopian education officials at different levels of the hierarchy, and school personnel; assessments of the reading capability of students at different grade levels; discussions with Mission staff; and discussions with personnel from various development partners. 3.1 Background Research Conducted on AAMU It was not feasible for evaluation staff to make a site visit to AAMU or to review most project documents. However, the Team Leader (TL) was able to interview AAMU’s TLMP Director at her new university (see ANNEX A – Questions Posed of AAMU TLMP Director). 3.2 Research Activities in Addis Ababa, Oromiya and Amhara Environs The data collected from each category of stakeholders in each location are presented in ANNEX B – Other Data Collection Instruments. The order with which the research was conducted is found in ANNEX C – Schedule of Stakeholder Interviews and Activities. Over the two-week period (March 3-19, 2013) in which the team was in Ethiopia, data were collected at the following venues/from the following stakeholders: • Visits to two TTCs, Kotebe in Addis Ababa and Debre Berhan (Amhara); • Observations and interviews at seven primary schools: three in Oromiya: Mulugeta Gedle, Burayu, and Dukem 1, three in Addis Ababa: TTC, Wondrad Demonstration, and Sefre Selam, and one in Amhara: the Debre Berhan TTC; • Interviews with nine school principals/deputy principals; • Observations of 10 teachers; • Focus group interviews with 22 teachers; • Forty reading assessments with 20 first graders and 20 second graders (20 boys and 20 girls) at five schools in Addis Ababa and Oromia; 1st grade age range: 7-14; 2nd grade age range: 8-15; • Interviews with six English curriculum specialists; TLMP Evaluation Report Volume II - Ethiopia 12 • Interviews with three Master Trainers; • Meetings with four textbook developers and a telephone interview with one coordinator; • Meetings at two RSEBs and seven woredas/sub-city officers; • At the MOE, interviews with the State Minister, Director of Curriculum, and Heads of English Curriculum, Teacher Development, Examinations, Licensing, Stores/Stockkeeper; • A telephone interview with the Graphic Designer; • Interviews with two International Foundation for Education and Self Help (IFESH) Volunteers in early grade reading in Addis Ababa and at the Debre Berhan TTC in Amhara; • Interview with the World Bank Procurement Contractor; and • Interviews in Addis Ababa with the TLMP Project Director and Project Coordinator. In identifying schools, we wanted to maintain a focus on at least three regions where different mother tongues have conversational dominance, so we selected schools in Oromia (3), Amhara (2), and Addis Ababa (6). (Due to an outbreak of meningitis, it was not possible to travel to SNNPR as originally planned.) Because we wanted to understand how the TTCs were incorporating the TLMs into their courses, in Addis Ababa we went to the Kotebe TTC; we had the same plan for the Debre Berhan TTC, but circumstances noted below under “Limitations” prevented us from collecting significant data at this TTC. To accommodate the limited English language ability of school principals and teachers, the team was divided in two: one team of three (two Ethiopians with facility in Afan-Oromo and the TL) collected data in Oromia; and one team of three (two Americans and one Ethiopian) collected data at the Kotebe TTC and two schools where English was spoken by all those to be interviewed. This division was made so as not to overburden the Ethiopian members of the team. The Oromia team traveled to Mulugeta Gedle, Burayu, and Dukem #1 primary schools, each of which was located in different directions approximately 40 kilometers from Addis Ababa. No principal or teachers could be interviewed in English, nor could the woreda or RSEB heads. The Addis Ababa team interviewed the director, English language instructors, curriculum specialists, and materials developers at Kotebe TTC. Also interviewed was the Peace Corps Volunteer whose job it was to train TTC students on how to use the TLMs. The team then went on to the TTC demonstration school to observe teachers and assess students. Another school where the team collected data was Sefre Selam, a school noted for its strong community support. The Addis Ababa team also went to three other schools - at which curriculum heads, materials development educators, or master trainers were located. However, as these schools were not primary schools, no others were interviewed. The Ethiopia TLMP evaluation presented a somewhat unique opportunity: assessing the TLMs produced for each grade level. The team developed a rubric that considered specific quality indicators of content, pedagogy, and publishing quality, and then worked in pairs (one Ethiopian and one US expert) to evaluate the materials. One pair evaluated grade 1 student textbooks and teacher’s guides, another evaluated grade 2 TLMs, and the third evaluated both grades 3 and 4. Discussion of the evaluations reflected the various professional backgrounds of each of the team members and yielded insight into the strengths and shortcomings of the TLMs, including literacy, language, culture, context, level of capability, teaching methodology, etc. Protocol for Conducting Reading Assessments. Since our research design included assessing children’s ability to read a portion of the grade-specific TLMs, we had to be at a school when classes started at 8:00am as English is generally the first subject taught. We, therefore, conducted the teacher TLMP Evaluation Report Volume II - Ethiopia 13 observations first and then took two boys and two girls from Grade 1 and Grade 2 classes to conduct the reading assessments. As discussed under Limitations below, Grade 2 and Grade 4 materials were not yet in use. 3.3 Limitations of the Study While as planned we did visit locales where different first languages are spoken and which are written using alternatively the Amharic syllabary or the Western alphabet. Overall, there was insufficient time to conduct an evaluation that was geographically inclusive. We also did not have time to collect data on schools that had not received the books to determine what difference having the books made to the children’s language facility and ability to read. Our research design had originally included trips to the regional capitals of SNNPR (Hawassa) and in Amhara (Bahr Dar). When we interviewed the education team at USAID/E on the first day of field research, we were told not to go south to Hawassa as several cases of meningitis had been reported. We also learned that the key people we were to interview in Bahr Dar were in Addis for a meeting, so we interviewed them in Addis. When the whole team went to the Debre Berhan TTC (identified by the MOE as a “center of excellence”), we learned that the former director had recently resigned and the new director had been in his position only two months. Also, contrary to what we had been told by the Federal MOE, the TLMs had not been distributed at the TTC, so no action was being taken in the classroom or elsewhere in training future teachers on how to use the TLMs. (Kotebe TTC in Addis Ababa did have the TLMs and was making use of them.) In Debre Berhan, we had also scheduled appointments at the cluster school in which the TTC provided INSET on a periodic basis. However, when we arrived, we learned that the school was closed as all teachers were participating in a math and science workshop. When we went to the Woreda Education Office, we learned that the two senior officials with whom we had made appointments had been called elsewhere for meetings, and other staff were not available. Hence, apart from the meetings with RSEB staff who happened to be in Addis, we were not able to collect any data in Amhara. When we conducted our interviews at the MOE, we learned that due to administrative issues, the books for grades 3 and 4 had just been delivered to the RSEBs. Some had been picked up from the RSEB warehouse by school leaders but were not yet being used in the classroom as the year was half over (each textbook is to be used for a year). Since most grade 3 and 4 children had not been exposed to the materials, we could not assess their reading ability based on the use of the TLMs. A final limitation was our inability to contact Mondography Printers to determine whether TLMP had any impact on the company’s development. A road was being built outside of the business premises that prevented access. Electricity had been cut off, as had telephone communication. We obtained a cell phone number for the proprietor but repeated attempts to get in touch failed. Hence, we have no direct data on the Ethiopia-based printer, except from the TLMP Director, who informed us that the supplementary readers were now being produced as Mondography had received partial payment from AAMU. 4. FINDINGS AND CONCLUSIONS This section presents the findings that resulted from the use of the data collection instruments and procedures mentioned in the Methodology section above. We begin with the data collected on AAMU, present data on the print output of the project, and then go on to present findings on different aspects of the project. TLMP Evaluation Report Volume II - Ethiopia 14 4.1 AAMU Achievements and Challenges While AAMU was on a trajectory to make institutional changes supportive of international development activities in education, the evaluators were not in a position to collect data at AAMU to confirm these activities. Consequently, we have no information as to institutional changes that may have been made at AAMU as a result of TLMP participation. 4.2 TLMP Output By the end of 2008, TLMP had produced 3.2 million culturally- and research-based TLMs, and 70,000 teacher’s guides for grades 6, 7, and 8. Final numbers for 2009-12 outputs include the following: • Grade 1 – 4.3 million (TLMP paid for 1.5 million student books and 25,000 teacher’s guides; the rest was paid for by USAID/E); • Grade 2 – 1,700,284 student books and 50,000 teacher’s guides (TLMP paid); • Grades 3 and 4 – 45,000 teacher’s guides at each grade level (TLMP paid) (USAID/E printed 5.5 million student books for grades 2, 3, and 4); and • Folktale books – 700,000 anthologies and “big books” and 50,000 teacher’s guides (TLMP paid). The dissemination of the TLMs was highly irregular and inconsistent. The grade 1 materials were distributed in 2009, but grade 2 materials were distributed only in September 2012, and grades 3 and 4 materials had just been distributed before this evaluation began in March 2013. Distribution was carried out largely by the MOE and RSEBs, with documentation maintained by the central storekeeper as to when the materials were delivered. Since RSEBs do not have the means to transport the materials the rest of the way to the schools or even to zones (the level of government immediately below the region), both of these arranged to pick up the number of books allocated to a given school from the RSEB warehouse. In keeping with the former Prime Minister’s directive to create a 1:1 ratio of textbooks to students, USAID provided significant financial support to print the number of textbooks needed to fulfill this goal. Because of the dollar limitation USAID/E has in issuing local tenders, they had to issue an international tender. The printing award went to a South African company, who then sub-contracted to a printing company in China. Consequently, with the exception of materials produced in Ethiopia by either Mega or Mondography Printers, all other textbooks were produced in China. 4.3 Project Management and Partnerships A major partner to TLMP was USAID/E itself. Initially, USAID/E’s role was to liaise with the MOE and to help establish and maintain the relationship between AAMU and the MOE. However, during the AEI period, it became very clear to USAID/E that training on how to use the grade 6-8 textbooks was a necessity as the books introduced new methods of teaching. If teachers and students were to benefit (teacher training per se was not covered in the CA), USAID/E determined that it should facilitate a teacher training process. It made a special agreement with a service provider to organize the training: the American Institutes for Research (AIR) was contracted under the Teach English for Lifelong Learning (TELL) project to provide training. Under this contract, approximately 20,000 English teachers were trained on the grade 6-8 materials. USAID/E continued this project during the 2009-12 period, and TELL trained 50,000 more English teachers (many of whom taught in self-contained classes) in the use of the grades 1-4 textbooks. The TLMP Director wrote the training manual (in Alabama), and TELL implemented it. The TLMP trained 32 TLMP Evaluation Report Volume II - Ethiopia 15 Master Trainers (MT), who then cascaded the training to 300 high school teachers. On the basis of individual action plans, the high school teachers then trained primary school teachers, all with the support of TELL. While this endeavor made a “dent” in training needs, there are over 300,000 teachers who need the training. The actual training that TELL provided was met with challenges: • The budget was limited (although USAID/E offered what it could), • No pre-workshop assessment was undertaken to determine the English language capability of teachers (high school or primary school), • One-off training sessions were held for only 3.5 days, • There was no classroom follow up, • No provision was made for refresher training, and • The content of the training did not fully address the new pedagogy being introduced. Underlying the training challenges was the English language capability of teachers. The MOE recognizes this shortcoming and so has arranged with the British Council to again undertake another English Language Improvement Program (ELIP) to train 40,000 more teachers in English. This program is offered by the Teacher Development Directorate. As noted above, USAID/E played a major financial role in having the books printed. When USAID/W reduced its funding some time in 2010/2011, TLMP was hard-pressed to print the number of books needed to go beyond the pilot schools. USAID/E paid for the printing of all grade 3 and 4 textbooks, and a percentage of the grade 2 books. Essentially, the project would not have worked without these two areas of mission involvement. Owing to the increased activity of TLMP during 2009-12, AAMU hired a local coordinator in Ethiopia who had very rich experience in curriculum and planning at the MOE. Based in AAMU, the Director continued to make several trips to Ethiopia per year, but the bulk of the development activities had been completed by 2011; the remainder of the work required timely decision-making and oversight. At AAMU, the Director assembled a management team composed of a program coordinator, an office manager and herself. She also hired an illustrator who had been engaged in Ethiopia, but who then moved to Alabama to attend school. However, no individual was designated to develop a formal M&E system and to track the production of the materials as well as the progress of teachers and students. Working with the Ethiopian coordinator in Addis Ababa was a former Voluntary Services Overseas (VSO) volunteer who had been working with the MOE Curriculum Directorate during AEI. She began her employment with TLMP in 2010 and was responsible for the development, review and revision of grades 3 and 4 materials. Review included field testing at sites within easy traveling distance from Addis Ababa in the Oromiya and Amhara Regions. A partnership was developed between TLMP and Ethiopia Reads (ER), an organization that builds and stocks libraries throughout Ethiopia. Libraries were established and stocked at schools in each of the 11 RSEBs and administrative cities. Schools had to agree to set aside space to house the library and ER provided US$10,000 worth of books for each one. It was envisioned by TLMP that all of the TLMP materials would be placed in the library, but in the two libraries observed (Mulugeta Gedle and Tsehai Chora – our pilot school in Addis Ababa), they were not in evidence. TLMP Evaluation Report Volume II - Ethiopia 16 4.4 Project Implementation 4.4.1 Materials Development AEI in Ethiopia ended with instructional materials in the pipeline. These included print-ready materials for grade 1 (developed solely by the TLMP Director), and preliminary work on the grade 2 student book. It is unclear what process was used to develop the grade 2 textbooks as they differ significantly from those produced for grade 1 and do not necessarily follow the grade 1 materials. In fact, the team determined that there was a “language cliff” that had to be climbed between grades 1 and 2 that assumed children could read and could address some very complicated spellings, grammatical structures, and learning concepts. For an explanation of how the grade 3 and 4 materials were developed, please (see ANNEX D - Materials Development Process Grades 3 and 4). Teachers and other educators who had attended the 220 hours of ELIP training constituted the pool of individuals deemed qualified to participate in the development of the TLMs. After attending three workshops, writers were able to write textbook lessons (or skeletons thereof) and/or folk tales that would constitute supplementary reading materials. Materials were then reviewed by the Ethiopian Coordinator, sent to AAMU for editing and illustrations, and then sent to the MOE for review. The MOE does not have a specific textbook review board nor was it aware of the leveling that was needed to make the TLMs accessible for teachers and students. It is not clear what criteria were used for these final assessments and edits. 4.4.2 Assessment of TLMs by Evaluation Team Student Textbooks - The criteria on which the assessment of pedagogical quality/content was conducted in student textbooks included: alignment with syllabus, organization of content, correctness of content/conformity with Ethiopian culture, integration with other subjects and cross-cutting issues, quality of illustrations, relevance and appropriateness for learners and teachers, opportunities for assessing student progress, language and communication, and enjoyment. In terms of the physical design of the student textbooks, the following criteria were used: cover and paper stock, bindings, size and dimension, quality of print, typeface, and structures. In general, all materials produced were in alignment with the MOE syllabi. In terms of the student textbooks, not all content moved evenly from simple to complex, and – taking the materials as a graded set – a “linguistic cliff” had to be climbed between the grades 1 and 2 materials. Content correctness and relevance to Ethiopian culture in certain instances was questionable, and only minimal attention was paid to cross-cutting themes. The illustrations were numerous and helpful, although sometimes choice of color made picture detail difficult to distinguish. Topics were relevant for each grade, although pedagogical content seemed unsystematic. Opportunities for continuous assessment are included, but there is no specific instruction on how this should be done. The language used is far above what children (and teachers) are capable of understanding. The colorful presentation and large number of illustrations are very attractive and children want to look at the books. While the physical quality of the books is generally very durable, the size is cumbersome for children and should be divided by terms (rather than having one book for the whole year). Teacher’s Guides – The criteria on which the assessment on pedagogical quality/content was conducted in teacher’s guides included: objectives, organization and presentation, teaching and learning strategies, teaching ideas and referencing. In terms of the physical design of the teacher’s guides, the same criteria as the assessment for the student textbooks were used. TLMP Evaluation Report Volume II - Ethiopia 17 Too many lessons are included under each unit and each lesson does not have its own objectives. Often there is a mismatch between what is in the guide and what is in the textbook. Sequencing is often a problem. Scripted lessons are presented for each topic, but the language used is beyond the capability of most teachers at the grades 1-4 level. Many new teaching ideas are presented, but these form a new, non-didactic, participatory approach to teaching to which teachers have not been introduced. There is significant cross-referencing to other materials to be used, but teachers do not have much experience in using a range of materials to teach one subject in a classroom. The size of the guides is cumbersome, the text is very dense, there are no illustrations, and some explanations are missing; there is no glossary or other helpful drills that can be constructed in teaching different language acquisition skills. In summary, the materials require improvement for them to be truly useful to both teachers and students. (For a full assessment of the TLMs by the evaluation team, see ANNEX E – TLM ASSESSMENTS.) 4.4.3 Comments on the TLMs Made by RSEB and Woreda Education Officials, Principals and Teachers It was initially planned that there would be three sections reporting on the findings of each of the officials and teachers identified. However, the overlap in what each of the groups of respondents reported was so significant that findings have been consolidated. The reader will note that many of the points raised by stakeholders about the TLMs are similar to the assessment findings made by the team. Pilot Testing: In 2008, 2009, and 2010, grade 1, grade 2, and grades 3 and 4 TLMs, respectively, underwent extensive pilot studies. The materials were evaluated based on the following criteria: Criteria on Which TLMs Were Evaluated During Pilot Studies No. Evaluation criteria Remark 1 Content of stories is appropriate to the target age group. 2 Stories are interesting to the reader in the target group. 3 Stories will help to improve reading habits in the target group. 4 Level of language difficulty is suitable for the target group. 5 Book does not contain grammatical errors. 6 Illustrations are interesting and appropriate to the text. 7 Illustrations are culturally appropriate. 8 Layout and design are attractive and suitable for the target group. 9 Stories encourage integration of positive values. 10 Stories take into account cultural diversity. 11 Treatment of gender in the text and illustrations of the textbook is appropriate/balanced. 12 Questions are relevant and suitable for the target learners. 13 Questions are relevant and suitable for the target learners 14 The Instructional Guide will be helpful for teachers. 15 The Instructional Guide promotes communicative, active learning. The results of this assessment, conducted at pilot schools where teachers had received more input, were mostly positive, and very few critical comments were made. Missing from this and other instruments were questions on the challenges and difficulties teachers faced in using the materials. Without a period (between one and three months) during which teachers are equipped to use the TLMP Evaluation Report Volume II - Ethiopia 18 materials and then actually use them in their classes, responses to this data collection instrument were largely hypothetical. Current Evaluation: The range of school-based stakeholders the team interviewed reported the following (some comments are made by principals, while others are made by teachers): Positive Responses • The colorful pictures help students understand the content better; • The books motivate students to read more; • The physical part of the books seems well-organized, attractive and durable; • The textbooks fit the MOE curriculum well; • Gender equality is well presented; • The diversity of names in the texts reflect the different regions of the country; • Students respond well to the diverse learning activities; • Almost all students in grades 1 and 2 have textbooks; • Books are learner-centered as they focus on the interests of the children; and • The books are innovative as the teacher role shifts from being directive to facilitating. Challenges and Negative Responses • Teacher English Language Deficiency - Teachers themselves don’t have a command of English, and there are few opportunities to practice English outside of class. ELIP needs to be extended over the long term so teachers can improve their English. Teachers have to use dictionaries to look up vocabulary words in the TLMs. • Number of Books Insufficient until Recently - To overcome the shortfall of the ratio of textbooks to students, one school director who became impatient waiting for the books said: “We photocopied some of the books, but much was lost when we could not reproduce colored pictures.” • Continuous Assessment – A principal said: “Teachers don’t care to assess students because they feel it is a waste of time since students are promoted to the next class anyway.” • Content of Student Books - It is very difficult to relate the content of the textbooks to the context of children’s lives. More drills are needed on different structures so that children can incorporate them into the way they speak. Students cannot understand the questions in the books. Grade 3 stories are too long and teachers have a lot of vocabulary to teach; grade 4 students are sent to the library to look up words. Teachers are unable to complete the grade 2 textbook in a year’s time as there are too many lessons; many of the songs and games are skipped because teachers cannot understand the English used. Teaching children in grade 1 how to write the alphabet takes a considerable amount of time. Many pictures appear in the books without an explanatory legend beneath them. The grade 2 textbook has many very packed pages and it is really beyond the capacity of the children. There are no “explicit” grammar exercises on tenses, parts of speech, conjunctions, sentence structure or other aspects of grammar. Sentences for grade 4 materials are longer than those for grade 3 and they are too difficult. Reading passages lack review questions. • Physical Structure of Books -TLMP books are fragile; they wear out in two years. • Teacher’s Guides (TG) – Teachers noted: answers to exercises in the TGs do not always match the student book exercises and need illustrations; it is almost impossible to teach the materials without the TG because each lesson is scripted and does not leave much room for teacher creativity. TLMP Evaluation Report Volume II - Ethiopia 19 Conclusion Overall, the TLMs produced under this initiative were far superior to the materials they replaced in terms of structure, print quality, and pedagogy. The TLMs reflect the MOE’s Minimum Learning Competencies (MLCs) for each grade level and those were generally well-illustrated with color drawings. While the textbooks and teacher’s guides were very welcome by all, the inability of teachers to use them appropriately in self-contained classrooms was a major setback to project achievements, largely because of teacher’s own English language deficiencies. All those interviewed addressed the levels of English language used as being too high, the lessons being too long, the materials in the student books not matching the materials in the teacher’s guides, and the teacher’s guides being long and “bulky.” In the development of the materials, there was significant ambiguity in what was needed (even though the AEI evaluation recommended a more focused ESL approach), and the indecisiveness of the MOE as to which approach to take emerged in the materials and in the confusion that teachers and trainers felt in teaching the materials. Overall, the teacher’s guides were composed of scripted lessons. However, using this technique as an approach to overcoming teacher shortcomings is not viable. 4.4.4 Comments on Training Made by RSEBs and Woreda Education Officials, Principals, Teachers and Master Trainers Although training on the use of the materials was not a focus of TLMP, USAID/E supported the TELL project (described above) to help teachers learn how to use them. RSEBs, Woreda Officials, Principals, Teachers, Master Trainers, and TOTs made the following comments about training: Positive Comments • We have learned some new teaching techniques from the TG such as how to use flash cards. • The developers have learned how to collaborate with a university in the U.S. to create materials for our children. Challenges and Negative Comments • Teachers need more training in pronunciation/spoken English and in writing. • The underlying pedagogy inherent in the materials was not a focus of the training; so many teachers missed the point and just follow along with the lesson script without realizing that they are also learning how to teach in a different way. • The training offered to teachers through the cascade model was “watered down” at each level, with primary school teachers receiving the least training; in essence, the cascade model as implemented is ineffective. • When we taught the teachers how to use the materials, a big piece missing was how to assess the children and what they learned/skills they acquired. • Training needed to be conceptualized as a long-term process that provided more than a one-off workshop; there was no real follow-up. • The TTCs have not taken up the TLMs and so there is no other training available for teachers to learn how to use the materials. Conclusion In general, when new teaching materials are introduced, training on how to use them is critical. Although USAID/E graciously found the means to fund a project that trained teachers in the use of the TLMs, the model used was inadequate to meet the needs of the teachers. One-off training left teachers confused about how to proceed, largely because of their own English language inadequacies in being able to understand the new teaching methodology inherent in the teacher’s guides. TLMP Evaluation Report Volume II - Ethiopia 20 4.5 Outputs and Outcomes Grade 1 TLMs were developed and distributed to Ethiopian primary schools beginning with the 2010-11 school year. Since then grade 2 materials were distributed in time to be used for the 2012-13 school year. Grades 3 and 4 TLMs had just been delivered to RSEBs at the time of evaluation. There was a major gap between the time teachers were trained in the use of the materials and when they actually received the textbooks for classroom use (approximately two years); it is highly likely that teachers forgot what they had learned and were in need of a refresher course. The teacher-generated Ethiopian folk tale books were developed in the last year of the project and made camera ready, but printing was halted due to lack of payment by AAMU. Development of the materials called upon teachers and other educators to record folk tales that were then edited by the Coordinator’s team and the AAMU team. Production of the 700,000 anthologies, “big books,” and 50,000 teacher’s guides is now on track. 4.5.1 Teacher Observations A total of 10 teachers were observed, 8 female and 2 male, in grades 1-3 (teachers in the Wondrad Demonstration School associated with the Kotebe TTC in Addis Ababa had received the grade 3 textbooks). Although most of the items identified were “observed,” significantly not observed were the following: • The teacher gives and corrects homework using the TLMs (10); and • The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson (8). Deficiencies were also noted in the following areas (“not observed”): • The teacher has prepared an authentic lesson that uses the TLMs (2). • The teacher explains the goal and purpose of the class lesson (5). • The teacher identifies, pronounces and defines any difficult vocabulary (3). • The teacher begins the class activity with questions that review previous activities using the TLMs (2). • The teacher uses learning aids/materials produced by TLMP (6). In terms of the artifact inventory, “not observed” were the following: • Lesson objectives are written on the board in English (8); • Learning aids/materials are posted in the classroom (8); • Word walls display words in English (8); and • There is evidence that teachers mark exercise books (9). Conclusion There is a sufficient number of deficiencies and “not observed” comments to warrant a more comprehensive assessment of teacher ability to use the TLMs effectively and to conduct more rigorous training in the future. 4.5.2 Learner Reading Assessments and Outcomes The Assessment Instruments and Process The purpose of the TLMs is to help Ethiopian children learn to speak, listen, read and write in English. An assessment instrument was developed by the evaluation team to measure student performance in English – speaking, listening, reading and writing (included in Annex B). The assessment was TLMP Evaluation Report Volume II - Ethiopia 21 administered by both Ethiopians and Americans on the team, with the American acting largely as the recorder. • Speaking was assessed by asking the child to respond to a series of questions about himself/herself in English (e.g., What is your name? Are you a boy or a girl? How old are you? What grade are you in?). • Letter identification was conducted by asking children to name 10 letters in English (both upper and lower case). • Word recognition was conducted by children reading up to 10 words that had been drawn from the grade appropriate textbook. • Listening comprehension included a passage that one member of the team read aloud from a grade textbook, and then “wh” questions were posed to determine if the student understood the passage. • Oral reading fluency (word accuracy and phrasing) was assessed using a passage drawn from a grade appropriate textbook. • Writing competence was measured by asking the child to write three letters, write his/her name and/or one word of her/his choice. Results The children assessed had success in certain language skills (letter naming) and great difficulty in other areas (expressive oral communication). Characteristic First Graders Second Graders Comment Letter Identification 75% identified 70% 90% identified at least 70% Listening & Speaking 100% 100% One-word responses Word Recognition 20% recognized 70% 35% identified at least 70% Comprehension Not able 25% answered 60% of questions Reading Not able 50% read w/high accuracy 15% able to read One-word responses Writing 40% able to write 3 letters 100% able to write names 35% able to write a word in English 80% able to write 3 letters 100% able to write names 35% able to write a word in English Conclusion The performance of this small sample size can be compared with students who were given the EGRA in 2010 which contained similar tasks but in mother tongue languages. The sample size in this assessment cannot be used to make generalizations, but some observations and trends are noteworthy as they mirror the EGRA findings in certain important areas: • Children seem to make progress as they move up the grades; • Children perform better at tasks that are more behaviorist than cognitive (memorization over thinking) and may reflect a preference of teaching style and pedagogy; TLMP Evaluation Report Volume II - Ethiopia 22 • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension; • Children at both grades have minimal expressive language facility (e.g., none of the children spoke in complete English sentences); and • Given that test items were drawn from the student’s grade level TLMP textbook, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. 4.6 Stakeholder Interest and Use of Materials/Sustainability 4.6.1 World Bank The World Bank has taken an interest in the English for Ethiopia TLMs and has agreed to fund the reprinting of several volumes to provide them to more schools. 4.6.2 TTCs Since we only had a sample of two TTCs in our research, with one using the TLMs in their pre-service and in-service training program, and the other not having received any of the TLMs yet, no pattern was established. MOE officials had said that all TTCs had received the books, but this was erroneous. Until they are delivered to all schools and the language curriculum can accommodate them, it is difficult to determine how TTCs will use them. 4.6.3 Other Sustainability Issues The Teacher Development Directorate is concerned at this time with improving English language competency in 40,000 primary school teachers. Working with the British Council, the Directorate will roll out an English Language Improvement Program (ELIP) beginning in September 2013. Special modules and materials are being prepared for the launching of this program. It is not connected to any other teacher training program. In the 4th Education Sector Development Program 2010-2015, published by the MOE in 2010, priority areas to be addressed include improved teacher training, curriculum and textbook assessment, school improvement, and increased access to early childhood care and education, especially in rural areas and for girls. All of these foci are relevant to the TLMP, but for our purposes here, we wish to focus on the increased attention to kindergarten (KG). While the Plan stops short of incorporating KG into the MOE curriculum, schools themselves are launching KG programs to start the school socialization process, and to introduce children to reading in the mother tongue. How KG teachers will be trained to launch the socialization and reading/ language development process is not clear. 5. LESSONS LEARNED • In designing and implementing a textbook and learning materials development program, teacher training on how to use the new materials must be included at both the pre- and in-service levels so that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use. • Scripted lessons cannot replace a systematic professional development program, particularly when the learning outcomes are cognitive, not rote learning. • When piloting newly-developed textbooks and learning materials, teachers should be given them to use in the classroom for at least a month (better for three months) and then research should be undertaken by an independent party/publisher to determine challenges teachers faced and/or overcame in using the materials so that they can be revised before final printing. TLMP Evaluation Report Volume II - Ethiopia 23 • Before launching a TLM production project, thorough research needs to be undertaken into the educational system, how teachers are trained (and for how long), the frequency and substance of in￾service teacher training programs, the level of English language competency of teachers (for English language textbooks), and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. • Before launching a language-based TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project, the learning outcomes and results achieved in using those books, and the level of language ability children have under past practices so as to avoid making assumptions about what children at particular grade levels are able to understand, speak, read and write. • Launching an innovative TLM production project works best when the various directorates of the MOE are working together in an overall learning improvement program that is not competing with other programs to create results. • Leveling any language materials is absolutely essential to creating positive learning outcomes among children; appropriate review policies, procedures and professional person power must be available to carry out these tasks. • For a US-based university to work in an African country successfully, staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts hampering long-distance communications) as well as with the donor. 6. RECOMMENDATIONS 6.1 Materials Development • If US-based institutions are to participate in textbook development in Ethiopia, they should: • Hire only experienced individuals in textbook development for designated grades; • Be properly introduced to the country, the culture and the practice of teaching in Ethiopia including how the education system is structured, how it functions, etc., with some insight on its history; • Have experience teaching the grades and subjects for which they will write textbooks; and • Be able to share their expertise with Ethiopians who have not developed textbooks before. • When conducting textbook field tests: • A broad array of schools must be included so that teachers working in different circumstances and with limited English capability can be included. • International project leaders must understand the cultural reluctance Ethiopians feel in criticizing another person’s work. • Teachers should be given between a month and three months after training to become familiar with the materials and the methodology, and to practice teaching before pilot/ assessment observations and interviews are held. 6.2 Systems Improvement • Any project that creates textbooks MUST: • Include a teacher training program over the course of the project and beyond; • Include a strong M&E system that includes indicators on production, delivery and use in the classroom; • Use a continuous professional development approach; and • Ensure that the textbooks consider integration, continuity, and sequence [horizontally across other subjects in the same grade, vertically up the grades within the same cycle ie grade 1-4, and 5-8]. TLMP Evaluation Report Volume II - Ethiopia 24 • A sustainable strategy for training teachers and administrators in the use of the materials is critical. This strategy should include pre-service training at TTCs, Continuous Professional Development (CPD) through the use of the cluster centers, and the training of “master trainers” who can provide initial capacity building and then be available to deliver refresher courses. If the books are to be distributed to all children in all primary schools (and that is the goal), then all teachers must be capacitated to use them over the long term. • The cascading approach can be cost-effective, but significant oversight needs to be exercised by designated focal persons or faculty members of TTCs to ensure that the training delivered is the same at all levels. • To ensure the use of TLMs, training organizers should plan to include all stakeholders in the trainings, such as supervisors/inspectors, curriculum specialist at the woreda and sub-city level, so that they can be supportive of teachers. 6.3 Towards a More Comprehensive Approach to Language Development • Language Policy: The MOE needs to clarify its own language acquisition policy and determine goals that are pedagogically sound. Building further capacity in curriculum and syllabi development within the MOE is cornerstone to this process. • Language Teaching: A more comprehensive approach to teacher training in language needs to be taken. Questions such as the following need to be addressed: • What entrance requirements should be satisfied for future teachers to be accepted into a TTC? • What will be the curriculum, especially for English, Amharic and mother tongues? • How will English language capability of all TTC entrants be improved so that English classes can be conducted in English and teachers will be able to provide English language experience to their students? • What is the best approach to teaching English and other languages – ESL, EFL, and grammar translation? • How can methodologies for teaching language be integrated across the board so that teachers can utilize similar methodologies in teaching all languages? • How can the cluster center school structure be utilized to present on-going professional development in-service training so that teachers and administrators have an opportunity to learn continuously? • Teacher Licensing: The Licensing Directorate in the MOE is working on standards that will allow primary schools to become licensed. Part of this effort includes language tests, the results of which should be used in redesigning the TTC and English language curricula in all pre-service, in-service, and other professional development activities. 6.4 Linking TLMP to USAID Priorities in Early Grade Reading High quality, well-illustrated TLMs were produced by AAMU in Ethiopia for grades 1-4. The above recommendations notwithstanding, the investment made by USAID/W and USAID/E in these materials should be an integral part of the new USAID priorities in early grade reading. The current contractor implementing this project should be encouraged to do the following: • Determine how the English language textbooks might be edited to reflect a greater step-wise reading/English language development process, beginning with phonics (in English and the various mother tongues targeted for Ethiopia), taking students through a sequential learning process that will build upon skills developed each day. • Consider the feasibility of replicating the methodology developed by UTSA in generating TLMs through providing writing workshops to teachers and other educationalists, who then develop culturally relevant materials in the various mother tongues. The project would then take the TLMP Evaluation Report Volume II - Ethiopia 25 materials through a range of leveling and pilot testing processes to ensure that children would be able to learn, in a step-wise manner, how to read in their home languages. • Consider how the supplementary reading folk tales might be translated into mother tongues, edited and leveled, to be used in English and early grade reading teaching and learning in the targeted languages. • Work with the Curriculum Directorate of the MOE to outline a realistic curriculum and syllabus for each grade level so that it can incorporate strategies for mother tongue and English language reading skills development. • Although early grade reading is the focus of this new initiative, if possible, the new contractor might also review the TLMs produced for grades 6-8 to bring them more into line with earlier skill development in English and to work to incorporate these in the curriculum as textbooks rather than supplemental materials. • Generate a strong teacher training program that integrates TTCs, woreda education offices, RSEBs and others in the teacher training planning and implementation process. TLMP Evaluation Report Volume II - Ethiopia 26 ANNEX A. QUESTIONS POSED OF AAMU TLMP DIRECTOR TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did you university provide? What challenges emerged in your work with the MOE? How were they resolved? • What process did you use to review and revise the materials? How were the materials, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? TLMP Evaluation Report Volume II - Ethiopia 27 Project Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the project improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school (Ghana – kindergarten)? About distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? TLMP Evaluation Report Volume II - Ethiopia 28 ANNEX B. OTHER DATA COLLECTION INSTRUMENTS In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the expertise developed in textbook production by the MOE being leveraged to obtain other, similar grants? What would prevent this from happening? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in Ethiopia? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • How satisfied were you with your relationships with Alabama A&M and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOE Administrators (triangulate with different administrators at each level) • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOE provide? How frequently? • How do the woreda inspectors assess teachers? Were they trained in the use of TLMs? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • How was the decision made to include supplementary/complementary readers in the TLMP? Who made the decision that these materials should be based in folk tales? How did the process of developing these materials differ from that used in developing the TLMs for grades 1-4 English classes (textbooks)? Was there an advantage of one approach over the other? Please explain. • What will the MOE do to continue the production of TLMs now that the project has ended? Have any other donors been found to continue this activity? What new textbook policies have been developed as a result of TLMP? TLMP Evaluation Report Volume II - Ethiopia 29 • Is the MOE ready to use its own funding in the creation of TLMs in the future? In the redevelopment and printing of the grades 6, 7, and 8 materials? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? How do you know? • How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with AAMU? How could they be improved? 3) Material Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How did the production process and personnel differ between the development of the textbooks and the development of the supplementary reading materials (folk tales)? Did one process have an advantage over the other? If so which one, and how were results different? Would you recommend one process over the other for future book production? • How often did the two US- and Ethiopia-based teams meet? What were the results of these meetings? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? What cross-cutting themes did you include? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? • How satisfied are you with the collaborative production process between yourselves and AAMU? What worked well/did not work well? What would you change to improve the process? • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? (describe both the textbooks and the supplementary reading materials) What did the MOE do (specify unit)? What did the project TLMP Evaluation Report Volume II - Ethiopia 30 do? What challenges emerged in your work with the MOE? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOE play (specify unit)? What roles did your office play? What guidelines did the MOE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Who was responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who provided the training? For how long? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Gender distribution of those who attended? Did TTC faculty attend? From which TTCs? Did university faculty attend? How many? From which universities? • What other in-service teacher training is provided by the MOE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes • What was the TLMP management structure in Ethiopia? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ethiopia Project Advisory Committee (PAC)? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of project work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with AAMU? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery TLMP Evaluation Report Volume II - Ethiopia 31 system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? • How satisfied were you with the relations established with AAMU? How could they be improved? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? • How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparity in enrollment/attendance? If so, what have the results been? • What is the linguistic background of the learners at this school? What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? At what grade does English become the language of instruction? Do you believe your students are adequately prepared in English to learn entirely in English? What needs to be done to prepare students better? TLMP Evaluation Report Volume II - Ethiopia 32 • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long? • How many of each textbook and workbook did you request for the school? How many of each text/work books did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on (cascade) what they have learned to their colleagues? Do they receive any incentive to do this? • How has the cluster center training and resource system enhanced the ability of teachers to be more learner-centered? How has the TLMP enhanced the adoption of learner-centered teaching practices? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom and Head Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree) • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching English? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLMs? Please explain. • How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP textbooks do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive the TLMs for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? TLMP Evaluation Report Volume II - Ethiopia 33 • When did you receive training on the use of TLMs? How long did it last? Did someone from the TLMP project or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this? • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1- 4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the supplementary readers? 3) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? • Why do you think you were chosen to be a trainer? TLMP Evaluation Report Volume II - Ethiopia 34 • On a 1-4 scale, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? • How familiar were you with the TLMs before you delivered the training? What materials were you provided to be a trainer? What materials did you provide to your trainees? How confident were you after the TOT that you could teach others in how to use the TLMs? What else did you need? • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs/ETHIOPIA School _________________________ Village/Town/City ________________________ Teacher Sex _____ M; _____F Grade Level: ____________________________ No. of Students: ______ M; ______ F; ______Total Languages spoken in this community: _______________________________________ Home Language/Mother Tongue of learners in the class: ________________________ Home Language/Mother Tongue of teacher: __________________________________ No. of books: _______________ No. & type of learning materials __________________ Indicator Observed Not Observed Other Teaching Using TLMs Instruction 1. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP 7. The teacher can read and explain TLM content to the students 8. Students can read and understand the subject matter in the TLMs. TLMP Evaluation Report Volume II - Ethiopia 35 Indicator Observed Not Observed Other 9. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment) 10. The teacher gives and corrects homework using the TLMs. 11. The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson 12. Students and teachers use mother tongue/English (French) when asking and responding about TLMs (circle which language) 13. Teacher demonstrates personal mastery of English TLMs/Artifact Inventory 14. Lesson objectives are written on the board in English 15. Learning aids/materials are posted in the classroom (TLMP produced and others). 16. TLMs are locked up in the cupboard. 17. Word walls display key words in English 18. Sentences appear on the chalkboard or on a chart 20. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books) 21. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment Comment____________________________________________________________________________ ______________________________________________________________________ 10) Learners Reading Competency - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read a few sentences aloud, and then rank the performance in the following manner: 1) fluent; 2) little difficulty; 3) very haltingly; 4) can read only a few words; 5) cannot read at all. If a learner tries to read a word, identify what strategy he/she is using to read: 1) sound it out – phonics; 2) sight reading; 3) other strategy (TBD). Reading Comprehension - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read the passage silently and after they have finished, have learners explain what they have just read. TLMP Evaluation Report Volume II - Ethiopia 36 TLMP READING ASSESSMENTS/ETHIOPIA School Name ______________________________ Region ____________________ Woreda ___________________ Book 1 _____; Book 2 _____; Book 3 _____; Book 4 _____ Letter Recognition – x/10 Word Recognition – x/10 Reading Fluency Passage Comprehension Writing Grade Level Sex M/F Age Home Language KG Upper Lower Sight Words 1) Fluent; 3) Haltingly; 5) Can’t Read Can put passage into own words: 1) without difficulty; 3) with difficulty; 5) cannot put passage into own words PROMPTORS: Who? What happened? Why? How? What do you think…? Name; Any Known Word (specify) TLMP Evaluation Report Volume II - Ethiopia 37 ANNEX C. Schedule of Stakeholder Interviews and Activities Date Time Organization Person Interviewed Title 3/4 4:00 Ethiopia Reads Dana Roskey Director, TESFA & ER 3/5 3:00 IBTCI Team Meeting 3/6 8:00 USAID Alison Wainer Befekadu Gebretsadik Education Chief COTR; mission activity manager for this activity 3/6 2:00 AAU/Addis Ababa Meeting at Dreamliner (schools choice) Tizazu Asare Local TLMP Coordinator (formerly MOE Head of Curriculum and Head of Planning) 3/6 3:15 MOE Girma Alemayehu Director, Curric. & Instr. 3/6 3:15 MOE (New Bldg., 121) Ejeta Negeri Head of English 3/7 8:00 MOE Ibrahim Fuad (edited TLMs) State Minister of Education (GES) 3/7 10:00 Sheraton Dr. Mary Spor and Tizazu Director, TLMP and Manager, TLMP 3/7 3:00 Dreamliner Desalegn Garsamo, Former COP AIR/Tell Project MOE, Teacher Training 3/7 4:00 IBTCI Team Planning Meeting 3/8 7:30 Tsehay Chora PS Principal, Teachers, Learners Addis Ababa 3/8 11:30 Amhara Curriculum Spec. Eyasu Aemiro From Bahr Dar MOE Ejeta Negeri FG w/Aemiro TLMP Evaluation Report Volume II - Ethiopia 38 ADDIS ABABA (Team 1) AA Schools 3/11 7:30 Kotebe TTC Dean Teferi Belew Almaz Debru MT Jennifer Miller PCV English Language Faculty Addis Ababa Wondrad Cluster Center School (Kotebe) Director, English Teachers 1-4, Students Addis Ababa 3/12 7:30 SefreSalam (L) Director, English Teachers 1-4, Students Addis Ababa 3/12 1:30 AA Kolfe Kereano Sub￾City Curriculum Head Addis Ababa AA RSEB Addis Ababa OTHER ADDIS ABABA 3/11 1:30 MOE Ejeta follow-up 3/11 3:00 MOE Examinations Department KefelengTsigie Deputy Head 3/11 4:00 MOE Stockkeeper Metike Head, AA 3/12 2:00 MOE Licensing Directorate Sahlu Bayissa Abaweloo 3/12 3:00 World Bank/MOE Zelalem Tadessa Procurement Contractor 3/12 3:30 MOE Curriculum Head Girma follow-up 3/13 9:00 Menelik Primary Tesfanesh Mulugeta Worku Belay English Curriculum Spec TLMP Evaluation Report Volume II - Ethiopia 39 Bole High School Askalu Kifle 3rd Grade writer; master trainer 3/13 1:30 MOE Teacher Training Shetu Deputy, Teacher Development, English 3/16 7:00 VSO Volunteer Judith Althous Coordinator, Materials Development 3/17 1:30 Graphic Designer Mulualem Fanta (Ms Wainer’s husband) Telephone Interview 3/18 9:30 MOE – RTI Reach Project Helen Boxwill Curriculum, RTI Read Project Not Held Printer Bennyam Girma Mondography Printers (TG 1&2, books, folktales) 3/14 12:30 Kotebe TTC Askalu Kifle Seife Hassan Almaz Debru Fisseha Matuma Grade 3 Textbook Writers Grade 4 Textbook Writers OROMIYA (Team 2) OROMIYA SCHOOLS 3/11 7:00 Mulugeta Gedle (L) Mulugeta Megersa Tadelech Gutema Almaz Yigletu Mulu Meseret Beyene Dechasa Principal Vice Principal English Curriculum Spec Master Trainers TLMP Evaluation Report Volume II - Ethiopia 40 Demissie Senbeth 3/11 1:30 Woreda Tsehay Debele Fitassa Guluma Woreda Educ.Bureau Head Woreda Curriculum Sp. 3/12 7:30 Burayu PS Degitu Yadessa Muleta Lami Principal Vice Principal 11:00 Woreda Dirba Tafesse Birhanu Assefa Woreda Statistician Curriculum Expert 2:00 Oromiya RSEB Fite Abera REB Vice Head 3/13 7:30 Dukem #1 Chanyalew Belay Keneni Bajiga Principal Vice Principal 1:00 Woreda Mesfin Ababe Shashitu Deresa Curriculum Team Leader Curriculum Specialist 3/15 6:30 DEBRE BERHAN TTC Deputy Dean Neqnike Gebeyehu Yrsmaw, MT Tell Project English Curric. Spec and English Faculty Suzanne IFESH Volunteer Debre Berhan Cluster Center School, Atse Zeray Yacob PS Aklilu Wolde Amanuel Deputy Director 3/19 1:00 USAID DEBRIEF Alison Wainer, Befekadu, Assefa, Demissie, Warkaye Education Team TLMP Evaluation Report Volume II - Ethiopia 41 ANNEX D. Materials Development Process - Grades 3 and 4 The process for generating grade 3 and 4 textbooks included the identification of teachers and teacher trainers (at TTCs) that had participated in the UK-established English Language Improvement Program (ELIP) and performed in English very well. Those chosen were exposed to the minimum learning competencies (MLC) for each grade and were then asked to write a sample lesson covering a specific topic. Those whose samples were chosen were invited to a one-half day of orientation jointly led by the TLMP Director and Coordinator, the MOE, and USAID/E. The selected writers were then participated in a three-day workshop during which they were trained in the pedagogy and format of the TLMs. The writers were then assigned different units (and/or lessons within the units) to produce as well as a model lesson. Writers were divided into two grade level teams. Each individual was assigned a specific unit. The units had already been defined by a topic flow chart for grades 1-4. Some units in each textbook were assigned to Ethiopian English as a Foreign Language (EFL) experts. Other units were assigned to primary education experts at AAMU (who did not necessarily have expertise in African education or in ESL). The American writers were expected to contribute reading pedagogy while the Ethiopian writers were expected to contribute a contextualize perspective and a sense of classroom reality in Ethiopia’s diverse education settings. American writers based at the AAMU, presumably faculty and graduate students from the school of Education, were chosen to develop and then edit materials produced in Ethiopia. It is unclear what criteria were used in the selection process of the American writers/editors and on what basis editorial decisions were made. The American and Ethiopian members of the writing team met in Addis for an initial workshop to begin drafting and critiquing draft units and lesson plans. Ethiopian writers raised concerns over the level of difficulty of the linguistic demands of the syllabus and over certain teaching methods as being too complicated for primary school teachers in Ethiopia. These concerns were apparently not addressed, nor were the format and content substantively modified. The English for Ethiopia series relies on extensive use of illustrations. Writers specified illustrations to guide student understanding and to introduce new vocabulary. These specifications were forwarded to graphic illustrators in the US and were adapted in some cases by illustrators/graphic designers in Ethiopia. Although units were developed on an individual independent basis, the Ethiopian writers convened on their own to review each other’s work and to provide suggestions. From each grade level a writer was selected to serve as a team coordinator whose main task was to keep the other writers on schedule. The Ethiopian writers would also visit the TLMP office to discuss their writing with the Coordinator during manuscript development as instructions were not always clear. A second workshop was held in Adama/ Nazret. None of the AAMU writers were present at this event. The purpose of this workshop was to refine draft units, share experiences, and solve problems. This was the final support to writers before submitting final drafts for compilation and review. A final workshop was held to review the completed draft textbooks and was attended only by the Ethiopian writers. Other participants included curriculum specialists from the MOE. The review focused mainly on the political and cultural issues reflected in the textbooks. Although the participants were given a guide to review the materials, this guide did not direct reviewers to provide feedback on pedagogy, readability, and content. TLMP Evaluation Report Volume II - Ethiopia 42 The MOE does not appear to have a rigorous textbook evaluation process. The MOE does not have a standing review board and a systemic process for evaluating textbooks according to language, pedagogy, and learning outcomes. Materials developers worked for between 6 and 8 months, and at no time during that period were they informed of any field testing results. The developers were not aware of any specific changes that emerged from such testing (see the discussion on field testing below). The Ethiopian writers interviewed were unanimous in their opinions that participation in the TLMP was a significant professional and personal experience. One writer commented: This was my first opportunity to develop instructional materials. I learned how to develop content from the syllabus. I learned how to simplify things so they would be suitable for young children. I learned about vocabulary and how to select appropriate words for a grade level. Another pointed out how the activity affected his teaching of pre-service primary school teachers: I teach a course on materials evaluation. It was good to have this experience of developing a textbook then critiquing the structure, the content, the connection to the syllabus. I now assign my students the task of reviewing a textbook and I use some of the guidelines we were given to help my students evaluate the textbooks. Another writer commented on the personal side of the experience: It was important that we were working as a real team. We shared the work; we gave each other feedback and helped each other. We don’t usually get that chance to work together. While these anecdotes point to the value that teachers obtained from participating in the writing process, none of these teachers is directly employed at the MOE in the Curriculum Directorate. Hence, although the MOE asserted that “we have had our capacities built in textbook development,” in fact they have not: individual teachers and trainers have. TLMP Evaluation Report Volume II - Ethiopia 43 ANNEX E. TLM Assessments 1. Student Books A. Pedagogical Content of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Alignment with Syllabus All textbooks are aligned with the syllabus, and meet the MLCs for each grade level Organization of Content Textbooks for grades 1, 3, and 4 appear to move from simple to complex, but the grade 2 text does not, with vocabulary and sentence structure too long and difficult. Correctness of Content/ Conformity with Ethiopian Culture “Lions live in the forest and prey on small animals” is erroneous, indicative of other passages. Content generally reflected urban/westernized culture and did not depict all religious and ethnic groups. In an attempt to be gender sensitive, some tasks failed to recognize that they are a part of religious culture. Integration with other Subjects & Cross-cutting Issues Only minimal attention was paid to linking content to other cross-cutting issues that would likely occur in other subject areas such as math and environmental science. Quality of Illustrations The textbooks make a good use of illustrations to guide vocabulary; the picture quality of the grade 2 text was often blurry because contrasting colors were not used. Relevance & Appropriateness for Learners and Teachers The topics were relevant and appropriate for children of typical age grades. However, the pedagogical content was not appropriate as a basic language development program (presentation seemed unstructured and unsystematic). Opportunities for Assessing Student Progress The texts provide ample opportunity for continuous performance assessment, but the format lacks opportunities for unit review and assessment. Language and Communication “While many stories and activities are innovative and engaging, they are too difficult for the English abilities of most learners and even many teachers at the third grade level.” There seems to be a mismatch between expectations of teachers’ capacity to read and model the level of English required at each grade level and unrealistic expectations about pupils’ cumulative language development. This mismatch may result in pedagogies that require the teacher to translate English content and directions into native languages, an overemphasis on the mechanical components of language (letter identification) and a learning strategy that emphasizes memorization of vocabulary and sentence frames. Enjoyment Index Textbooks are colorful and full of color illustrations; however, the grade 1 textbooks looks like a penmanship workbook, and the content of the other grades neglected opportunities to introduce jokes, riddles, songs, etc., that might stimulate learner interests. TLMP Evaluation Report Volume II - Ethiopia 44 B. Physical Design of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Cover and Paper Stock Durable and appropriate stock, but may be more durable if the textbooks were produced in two volumes rather than one to preserve the longevity of the books. Durability would be enhanced if children were provided book covers. Durability is projected as 2 years, but may be as much as 4. Bindings Durable and appropriate; should have a shelf life of 3-4 years, provided they are handled appropriately. Size and Dimension The grade 1 textbook is more of a workbook that provides children the opportunity to write letters and sound them out when a phonics approach is used. Other textbooks should be divided in two so that children and teachers can use each part each term. Quality of Print Good quality and legible; some illustrations lack sharp contrast. Typeface The sans serif font is good for grade 1 as it is clean and simple. However this typeface does not match frequently used fonts in texts (a, g, t). Structures Supportive materials to enhance children’s learning might have included a pupil picture dictionary, high frequency language phrases, and some grammar models/sentence frames. 2. Teacher Guides A. Pedagogical Quality of Teacher Guides Assessment Characteristic Team Assessment Objectives MLCs and unit objectives are presented at the opening of the unit, but are absent for each lesson and do not identify learning outcomes. The number of lessons to be covered per week is beyond the overall number of days a child is to be in the classroom over the school year. Objectives are not order, in sequence, and do not necessarily build on what has already been learned. Organization & Presentation The Guides contains lessons that are not in the student text, resulting in an odd labeling system that is clumsy and confusing (e.g., units in the student’s text may open with lesson three). Teaching & Learning Strategies Teaching activities are designed to be interactive rather than didactic. The format uses a scripted text for teachers; however, teacher English competency is not sufficient for them to read and understand the scripts. Moreover, the scripted format will not accommodate irregularities in the pupils’ texts nor will it help teachers address children’s questions as they arise. Teaching Ideas The guides introduce or model a number of high interest techniques to promote active learning and language skill development. Referencing The guides provide extra materials to reinforce or enrich lessons, such as crossword puzzles or suggestions for making flashcards. It also lists the answers to the activities in the student book. However, the guides do not TLMP Evaluation Report Volume II - Ethiopia 45 orient the teacher to resources in other books, publications or the Internet. The teacher will need a dictionary to teach the meaning of the vocabulary. B. Physical & Design Quality of Teacher Guides Assessment Characteristic Team Assessment Cover & Paper The guides are not attractive, and have no picture on the front cover. The paper quality is appropriate, durable and appears water-resistant. Binding Adequate Size & Dimension Foolscap paper was used for the guides; the number of pages is daunting, with one volume 274 pages. Page length is long due to the scripted nature of the lessons. Quality of Print Good, with a good mix of boldface and textboxes; however, there is simply too much print on each page. More white space is needed so that teachers can “bracket” some portions or write ideas/reminders in the text. Structural Elements There is a vocabulary list, but no definitions. Language patterns (sentence frames) do not provide the name of the pattern, e.g., irregular plurals, verb tenses, adjectives, etc., only an example is given. The front part of the guides do not provide useful background information for the teacher on issues such as language development, lesson planning, and assessment. This type of content assumes a language proficiency that is beyond the competency levels of most teachers. This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Thomas Tilson (Team Leader), Eric Allemano, Kofi Mereku, and Kofi Marfo under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania TLMP in Ghana Cover Photo Credit: Chicago State University Evaluation of TLMP – TLMP in Ghana 46 MAP OF SOUTHERN GHANA With locations visited marked Performance Evaluation of TLMP – Ghana 47 EXECUTIVE SUMMARY As part of the President’s African Education Initiative (AEI), Chicago State University (CSU) was awarded a Cooperative Agreement (CA) in 2005 to meet the needs of Ghana’s Ministry of Education (MOE) in creating curriculum, generating textbooks and teacher’s guides, and teacher training to support its recent decision to include kindergarten (KG) in its curriculum. CSU focused its attention on producing KG materials in English in literacy, numeracy, and environmental science. In addition, CSU was tasked with developing literacy, mathematics, and science materials for primary grades 1-3 (P1-P3). In 2008 CSU received another CA to allow it to continue the work it had started for KG and P1-P3, but owing to budget cuts, the materials developed for P1-P3 were not printed or distributed. The final level of funding of US $8 million in 2012 was to enable CSU to expand implementation of TLMP in Ghana; the total funding was comparable to the funds received by the other MSIs. CSU has continued work on materials development with support from USAID/Ghana. Of the six countries in which TLMP operated, Ghana and Ethiopia were the only ones with a focus on primary level throughout the total life of the activity. Further, in Ghana there were major changes in policy and practice with respect to languages of instruction and to incorporation of kindergarten (KG) into basic education during the overall life of TLMP plus the addition of the major NALAP (National Accelerated Literacy Program) associated with the restructuring, so this country report on the expansion phase of TLMP provides more information on the broader contexts surrounding TLMP than do the other country reports. The program evaluation of the expansion phase of TLMP had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. Beginning in October 2012, the Chicago-based project team leader, Dr. Nancy Horn, had a number of in-person and telephone/e-mail interactions with CSU TLMP staff. The in-country component of the evaluation was originally scheduled for late November 2012, but at the request of the Mission (e.g., to accommodate national elections), it took place in June 2013. A Snapshot of TLMP in Ghana • TLMP was highly successful in producing teacher’s guides and workbooks for KG1 and KG2 in literacy, numeracy and environmental science and distributing 6.1 million of these materials throughout Ghana. In addition, it produced 220,000 wall posters that were distributed as well. The success of printing and distributing the materials was due to a capable local printer, Buck Press. TLMP also produced and field tested P1-P3 teacher’s guides and workbooks in English, mathematics and natural science, but for lack of funds these were not printed. • TLMP developed and implemented an in-service teacher training program for 2,000 teachers, but did not become engaged with pre-service teacher education. • Major factors affecting implementation were the decision of the MOE to change the language of instruction for lower primary from English to local languages and its decision to have expendable workbooks, which could not be reused and which costs kept from being reprinted, as a core part of the TLMP materials. Performance Evaluation of TLMP – Ghana 48 Implementation of TLMP TLMP in Ghana began with a needs assessment in 2005, which was about the time the MOE decided to incorporate KG into the education structure, a move that took place in 2007. As a result, the MOE requested that TLMP focus on developing materials for KG, as Ghana didn’t have any materials for KG. Also, there was recognition that the teaching and learning of English remained an ongoing challenge. As a result of perspectives at that time, the MOE requested that TLMP develop materials for KG1 to P3 in English literacy, numeracy and science and that the materials should be in English. TLMP in Phase I (2005-2008) reprinted existing materials and, then in Phase II (2009-2012), developed new materials for KG1 to P3 in literacy, numeracy and science, and then printed and distributed 5.6 million copies of the KG1 and KG2 materials only – first to an initial 14 deprived districts, but then to all 170 districts throughout the country. The materials were available to all public schools and registered private schools; the latter were particularly appreciative because they give special attention to teaching English. The materials were highly valued by teachers. They were especially appreciative that the teacher’s guides were closely aligned with the official KG curriculum. In addition, teachers felt that the workbooks were especially valuable in helping children learn basic pre-reading skills. But the decision to use disposable workbooks was ill-advised, as in the majority of schools they were consumed in one year and, thus, the teachers were unable to continue using TLMP as designed. Only in about one-third of the classes visited by the evaluation team did the teachers make special efforts to preserve the books for re￾use, e.g., by having the children write answers to questions in the workbooks in a separate notebook. However, the major challenge facing the relevance and sustainability of TLMP was an important shift in focus of the Ministry. Shortly after the start of TLMP, the Ministry, with USAID support, decided to emphasize teaching in mother tongue (L1) in the early grades as the best way of engaging children in school, and for the transition to English by P4. The result of this initiative was NALAP (National Accelerated Literacy Program), a program for KG1 to P4 in which mother tongue dominated instruction in the early grades and there was only a limited role for English; there were only 10 minutes of ESL out of 90 minutes total for language arts in KG1, gradually moving to a 50-50 split between local language and English (L2) by P3 and P4. With this shift in policy, there was no longer a need for an intensive English program such as TLMP. As a result, one of the last major activities of TLMP was to merge the best parts of its program into the NALAP teacher’s guides in order to strengthen the limited, although important, English component. This merger or fusion process is nearly complete and TLMP, which is now primarily a mission-driven project, will now print the revised NALAP guides and support a major teacher training program for over 100,000 teachers starting in late 2013. Currently, USAID/Ghana has taken TLMP under its own wing, with a scheduled end of early 2014 (The Mission-based activities fall outside the Scope of Work for this evaluation, and are discussed to provide broader context). The Ministry would like to draw on the best of TLMP to strengthen the limited English component of the new bilingual early childhood education program. This has resulted in the current merger or fusion process, which is taking the best and most relevant activities and content of TLMP and strengthening the English component of the NALAP teacher’s guides. This will be an important contribution, but the Ministry is no longer interested in the reprinting or separate identification of TLMP. The new merged materials are to be referred to as the Child Literacy Program. TLMP will also print the new merged teacher’s guides and support the in-service training of over 100,000 teachers on the use of these new guides. CSU has worked most closely with the MOE’s Curriculum Research and Development Division (CRDD) to create the materials and implement the program. This approach helped to create a commitment and ownership of TLMP by GES, but it also limited technical assistance that might have been useful in areas such as early childhood education and M&E. In addition, the project engaged some faculty of the Performance Evaluation of TLMP – Ghana 49 University of Cape Coast and the University of Education, Winneba and some of the colleges of education in creating the instructional materials and implementing the in-service teacher training program, as well as some members of the PAC and PTA. However, TLMP failed to engage with these universities and colleges in terms of trying to influence pre-service teacher education. In addition, TLMP failed to engage the Teacher Education Division (TED) in teacher training activities. Evaluation Methodology The evaluation used a mixed-methods methodology, primarily qualitative, comprised of key informant interviews, review of materials, and site visits with classroom observations and discussions with educators, both in the settings of their schools and offices and in the context of a writing workshop at the Ghanaian National Association of Teachers (GNAT). To these activities was added assessment of the literacy skills of some students in the Volta Region. The major limitation, apart from time, was that relatively few schools are actually using the TLMP materials as planned. The evaluation team visited 16 schools, 11 public and five private, in four regions, where four different major local languages are spoken. The evaluation team had intended to observe a number of TLMP classes, but in the end could not do so because so few schools – only five of the 16 schools visited, all but one of them in Volta Region, the last region where school visits took place – were currently using TLMP. The team also met with officials in seven district offices, as well as individuals at the University of Cape Coast and University of Education, Winneba and officials in three colleges of education. Key Findings: Impact of TLMP on CSU An important objective of the TLMP projects for all the MSIs was to develop interest and skills in supporting education projects in developing countries. After an uneven start, this objective was clearly met with CSU, which has learned how to manage a large international project funded by USAID. Ten CSU College of Education faculty members and doctoral candidates worked with the MOE and GES is designing, writing, and producing the instructional materials and in assessment activities. In addition, 20 CSU students served as interns in Ghana on the project, and there were two Ph.D. dissertations based on work in Ghana. CSU also established the Center for International Education on its campus, stemming from the work in Ghana. Key Findings: TLMP’s Legacy in Ghana TLMP succeeded in producing and distributing very large numbers of materials quickly, but otherwise the legacy of TLMP, per se, in Ghana is limited, largely in the form of posters and wall charts. As noted above, TLMP in the Ghanaian locales visited realistically is no longer in use as intended, due to combinations of change in Ministry approaches towards literacy, being subsumed by more recent projects, and the lack of TLMP workbooks. All but one of the private schools received the TLMP materials, but only 30% of the schools still had a supply of workbooks and were using them. The teachers in the other schools continued to use the teacher’s guides and, in some cases, the workbooks as reference materials. Many of the schools had started to use the NALAP materials. None of the teachers interviewed had received any TLMP training. At two (Cape Coast and Effutu) of the seven district offices, the education officials had not heard of TLMP; this highlighted a problem with communications between TLMP and the field. For example, the team was given to understand that TLMP did not orient the District Directors to the project when the materials were distributed nationally, thereby losing an opportunity for getting District Director support for implementing TLMP. Regarding the universities, TLMP did engage several lecturers and even had a MoU with the University of Cape Coast and the University of Education, Winneba, but this linkage did not result in incorporating Performance Evaluation of TLMP – Ghana 50 any TLMP activities or materials in the universities’ academic programs. The lack of information and TLMP materials was also evident at the three colleges of education. The M&E activities of TLMP were weak. There was no M&E officer in the TLMP office and, thus, very limited ability to track implementation activities or to assess the impact of TLMP in the classroom and on student learning. CSU and the Ministry conducted just one learning assessment with KG2 tests in all three areas (literacy, numeracy and environmental science) in a limited number of experimental and control schools. The results showed a modest improvement in learning in the TLMP classes in literacy, but basically no difference in numeracy and science. The evaluation team administered a basic literacy test to a very small sample of KG children in three schools in Jasikan and Ho, in the Volta Region. (A wider sample wasn’t possible because the majority of schools were no longer using the TLMP program.) The children assessed had success in certain language skills (word recognition and naming pictures of familiar objects) and great difficulty in other areas (describing a moderately complex picture in the workbook). In doing word-picture assessment the researcher found that in some cases students seem to have memorized the word-picture sets in the order they appeared on the page. If the researcher chose the word-picture sets at random, some students had difficulty giving the right answer. Key Recommendations The recommendations are divided into improving materials and strengthening the education system. Materials Improvement • One fundamental recommendation is that TLMP’s work could have been more effective if TLMP could have been able to shift to first language instruction earlier workbooks. The workbooks and teacher’s guides could continue to serve a limited market including private schools that emphasize English language instruction and cosmopolitan public schools where it is not possible to teach in a common mother tongue. To help meet this need, the workbooks should be made available in the commercial market so that schools or parents could purchase them. Although it was not advisable to create disposable workbooks because of the unaffordable recurrent costs, theoretically, the KG1workbook might now be used over two years (KG1 and KG2) because of the more limited English program implemented in the context of NALAP focus on L1. However, there are no plans to reprint the workbooks. Because workbooks are highly valued by the Ministry and teachers, other forms of workbooks should be explored such as plasticized pages that children could write on, but then be erased so the books could be used the next year. As an alternative to workbooks (note that the NALAP has no workbooks), it would be desirable to create “activity books.” These would be non-disposable books that would have many of the advantages of workbooks, but where children would write in their exercise books rather than in the books themselves. • The approach to reading would be much improved by having a greater emphasis on phonics. In fact, the teacher’s guides and, especially, the TLMP workbooks are better suited to teaching in mother tongue rather than English because they assume the children have a greater level of language ability than is the case in English. In this sense, the TLMP materials, if translated into L1, might be better suited to NALAP than even the TLMP program. At least based on the very limited sample of students assessed, there is little actual learning to read taking place. Systems Improvement • Although TLMP in Ghana was more successful in this regard than various other TLMP implementations, a major design flaw was the failure to pay adequate attention to the need to train Performance Evaluation of TLMP – Ghana 51 teachers in how to use textbooks and learning materials. • Although TLMP has done limited in-service teacher training and plans a massive teacher training program beginning near the end of 2013, the program gave no attention to the important pre￾service programs. The new NALAP program is working with pre-service institutions, and efforts should continue and be strengthened to ensure that the newly trained teachers are competent in bilingual education and teaching basic reading skills in L1 and L2. • The Early Childhood Education (ECE) Coordinators should be trained so that they better understand the new Child Literacy program and teaching L1, and to then be able to develop or support district-level or school-based training. • The book distribution process went well, thanks in large part to Buck Press, but the system can be strengthened to ensure that books are distributed everywhere where needed and in the correct numbers. Performance Evaluation of TLMP – Ghana 52 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES 1.1 The Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched by the United States Agency for International Development in 2005 as part of the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort to develop and distribute learning materials to improve sub￾Saharan African host country partners’ management capacity in the education sector. For the expansion phase of TLMP, which began in 2008/9, each of the five Minority-Serving Institutions (MSI) was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP were to: 4) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 5) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 6) Ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP’s original focus was to develop TLMs for primary schools. In Ghana the Ministry of Education (MoE) determined that because it had very recently included kindergarten (KG) in its purview and there were no official texts or teacher’s guides, Chicago State University (CSU) should focus its attention on producing KG materials in literacy, numeracy, and environmental science. CSU was also tasked with the responsibility of developing literacy materials for primary grades 1-3 (P1-P3). TLMP was launched in Ghana with these responsibilities in 2005; in 2008 CSU’s CA was extended to allow the CSU project to expand on the work it had started for KG and P1-P3. However, owing to cuts in the TLMP budget, the materials developed for P1-P3 were not printed or distributed. The final level of funding for CSU to expand implementation of TLMP in Ghana was approximately US$8 million, comparable to the final levels of funding received by the other MSI. 1.2 Evaluation Objectives This performance evaluation covered work led by CSU during the expansion phase, 2008-2012, and was intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. Performance Evaluation of TLMP – Ghana 53 2. TLMP GHANA PROGRAM BACKGROUND Of the six cooperative agreements for the extension of TLMP, TLMP in Ghana and in Ethiopia were the only ones to reflect a continuing focus on primary level literacy throughout both phases of the TLMP activity and, unlike TLMP in Ethiopia, it was affected by the very major change in education policy vis-à￾vis the primary language of instruction. Further, an additional set of major education interventions, the National Acceleration Literacy Program (NALAP), began in the course of the expansion phase. Therefore, to provide a broader context, this Ghana TLMP report provides more information as to predecessor and concurrent activities, which were not the subjects of this evaluation, than do the other country reports. 2.1 CSU Responsibilities For the period 2005-2008, TLMP was a component of the centrally-funded President’s African Education Initiative (AEI). CSU launched its project in 2006 with a rapid needs assessment, which was followed by a more comprehensive assessment. The CSU leadership team identified subject areas, grade levels, potential partners, teacher training institutes, and issues related to the educational system. After holding a number of workshops with the Curriculum Research and Development Division (CRDD) of the Ghana Education Service (GES), the CSU leadership team identified development of materials to meet the new needs of the MOE in their efforts to incorporate KG as the highest priority. CSU identified a Ghanaian representative and opened a small office from which to manage the activity. Cape Coast University, the Education University, Winneba (UEW), and CRDD partners worked in concert with CSU to identify a writing team and to implement the program. Unfortunately, in the AEI phase of the work, no early childhood specialist or bilingual experts were made a part of the team. Although university partners in Ghana included these specialists in their teams, CSU’s failure to provide support in this area made for difficulties and misunderstandings in the partnership. Moreover, all materials were developed in English initially and then were to be translated into Twi, one of 11 languages of instruction, but funding cuts prevented the translation from occurring. As many KG teachers do not have a good command of English, many had a difficult time understanding the teacher’s guides. Consequently, as part of its own education portfolio, USAID/Ghana has been supporting the translation of TLMP teacher’s guides into Twi. The difficulties and misunderstandings between the partners were exacerbated when the new materials were edited. CSU did not believe materials had been properly edited by Ghanaian partners and so preceded to edit them again. Eventually, issues were resolved and CRDD had the final say. It should be noted that some of the illustrations in the children’s workbooks were inappropriate, especially regarding gender stereotypes, but were retained, e.g., males depicted as doctors, policemen, footballers, and soldiers, while women are depicted as teachers and nurses – a stereotypical set of gender-based professions; depiction of community leaders as almost all men, etc. CSU had books that were exemplary in structure, content, and illustrations identified for use as models and CSU reprinted or edited and printed over 2.5 million colored teaching and learning materials, which included numeracy, literacy and environmental studies workbooks and teachers’ guides, as well as over 300,000 wall charts for use in the Kindergarten through Primary 3 classrooms. USAID/Ghana paid for the first print run, produced by All Good Books. Subsequently, Buck Press was identified as a printer who could take on the work, which ultimately included printing, storage, and transport for the distribution of books nationally. Prior to the MOE’s assuming responsibility for KG, the private (for￾profit, faith-based, other NGO, etc.) KG all had their separately developed curricula and materials. Because the MOE wanted for there to be standardized KG curricula and materials, TLMP distributed its materials to both public and private KGs. Starting with the AEI phase of TLMP, there were a number of miscues with respect to printing and distribution. CSU purchased printing equipment for GES, believing that GES was responsible for printing Performance Evaluation of TLMP – Ghana 54 educational materials. After the equipment arrived in Ghana, CSU learned that GES does not do its printing in-house. Storage and distribution was to have been in the hands of GES; however, Buck Press ended up taking care of this as an in-kind contribution to TLMP, and school districts picked up the TLMs themselves. In fact, there was never any formalized method for distribution, contributing to a lack of accountability. CSU envisioned providing training to school lead teachers on the use of the materials. GES was to have selected these teachers in each district (books were eventually delivered to all districts), but then did not complete the selection process. Hence, the training consisted of two week-long workshops for the teachers of the 14 districts that had been selected initially. CSU staff informed the team that it was not satisfied with the extent of the training it provided, but further training was restricted due to budget considerations. Among the many goals included in the CA that CSU sought to achieve in the 2008-12 period were to: • Sign a new MoU with the MOE, specifying roles of each partner • Conduct a new needs assessment • Continue to write and field test materials • Evaluate and monitor the effectiveness of achieving specified results – with focus on deprived Ghanaian primary schools that have not reached the gender parity index. • Continue to establish partnerships with the Curriculum Research & Development (CRDD) division of the Ghana Education Service (GES), University of Cape Coast, University of Education at Winneba, Centre for Education Development Evaluation and Management (CEDEM) and others guided by a Program Management Committee (PAC) • Branding deliverables • Disseminate results to other MSIs • Print instructional materials based on deliverables • Expand into 16 new districts, in addition to the existing 14 districts in ten regions • Train 5,000 teachers in use of materials and in interactive methods • Introduce teachers to cooperative learning • Support sustainability in the GES • Continue to form new partnerships CSU experienced difficulty in trying to meet these goals, as discussed in 4, Findings and Conclusions, below. 2.2 MOE Context and the Educational System The Ministry of Education (MOE) has overall responsibility for education policy in Ghana; within it the Ghana Education Service (GES), which was established in 1974, is responsible for the implementation of approved national policies and programs related to pre-tertiary education. GES has several divisions relevant to TLMP including basic education, teacher training, and CRDD. The MOE has been undergoing decentralization since the 1980s as one step to address several critical issues, including poor quality of teaching and learning, attrition and low achievement especially by girls; poor/no infrastructure; lack of well-trained and motivated teachers, especially in the rural and deprived urban areas; and poor quality teaching and learning materials. In conjunction with this, the MOE recognized the importance of early childhood education on the ability of children to succeed in school in subsequent years, and decided to incorporate a two-year KG program in 2004, rather than leaving it to social service agencies and private individuals, which historically had been the case. In 2007 the Ministry formalized an 11-year primary school system: 2 years KG, 6 years primary and 3 years junior high school. This is followed by a 4-year high school program. Performance Evaluation of TLMP – Ghana 55 Thus, when the TLMP needs assessment took place in 2005, the MOE was beginning to incorporate kindergartens into its structure, yet there were few qualified KG teachers, virtually no learning materials for kindergartens, and a continuing interest in trying to strengthen English language in schools. As a result, the MOE requested that TLMP focus on developing English language materials for kindergartens and the early primary grades. TLMP was committed to developing programs for KG1, KG2 and P1-P3 in language, numeracy, and science. The aim was to get good quality English language materials into the classroom as soon as possible. The inclusion of numeracy and science was also a response to MOE priorities. In 2007, the MOE added to lower primary (P1-P3) environmental studies and math that emphasized problem solving. The MOE sought to make teaching more practical and child￾centered using more interactive teaching methods that encourage enquiry and problem solving skills through the use of project work using local materials. The MOE also took note of research that suggests that teaching in the mother tongue in the first years of schooling has many benefits, including a smoother transition to a second language (English, in the case of Ghana). As a result, just as TLMP was starting, the context quickly changed with the development of the National Accelerated Literacy Program (NALAP). Although NALAP includes some English language instruction (L2), the emphasis is on mother tongue instruction (L1). In fact, out of a 90-minute class period for language and culture, only 10 minutes is allocated to English in K1, 20 minutes in K2, and slowly increasing to nearly 45 minutes (50%) of the time by P3 with a transition to English as the medium of instruction in P4. By 2010 there was considerable progress in the expansion of KG reaching 97% Gross Enrollment Rate (GER) and 59% Net Enrollment Rate (NER). Yet 70% of the KG teachers lacked formal qualifications (information from appraisal of Ghana’s Education Sector Plan 2010-2020, p. 19). Student progress in English proficiency by P3 still lagged with only 40% mean score on a national test, minimum skills at 57% and proficiency at only 20% (NEA, National Education Assessment). These results suggested the need for an even stronger English program in the early grades, to which TLMP was responding. Yet, the Ministry was committing itself to the importance of L1. In 2012 it published the document “Scaling up National Quality KG Education in Ghana.” It is worth noting that Ghana’s history of the use of L1 as the language of early grade instruction long predates Independence although with a number of detours along the way.4 Thus, the current policy is not new, even though the renewed emphasis on L1 is new. While the MOE is committed to L1 in the early grades, and is even considering delaying the transition to English as the medium of instruction by one year, English remains important and is the language of the leaving exam at the end of P6. English is perceived by the public as critical to being successful in Ghana and, thus, much of the public is not supportive of the new emphasis on L1. The current language policy is controversial and came up in virtually all of the interviews conducted by the evaluation team. Finally, one unanticipated variable within the MOE was that the Minister changed three times in two years, so the program had to go through all the approvals several times. Also, each Minister had a different outlook and perspective on TLMP, which required ongoing adjustments by CSU. 3. EVALUATION METHODS & LIMITATIONS The evaluation of Chicago State University’s implementation of TLMP in Ghana was primarily qualitative and made use of (a) Review of project documents, of the literature, and of materials prepared under the CA; (b) Interviews with CSU administrators and project staff in the U.S.; (c) Key informant interviews with USAID staff, implementers, educators at various levels, and other stakeholders in Ghana; (d) Classroom observations; and (e) Oral assessments of selected students. The overall Team Leader for 4 See, e.g., Charles Owu-Ewie, “The Language Policy of Education in Ghana: A Critical Look at the English-Only Language Policy of Education,” Selected Proceedings of the 35th Annual Conference on African Linguistics, 2006 Performance Evaluation of TLMP – Ghana 56 TLMP, Dr. Nancy Horn, conducted interviews at CSU in late 2012 and developed the research design. The field evaluation team consisted of two Americans (Dr. Thomas Tilson, Team Leader, and Dr. Eric Allemano) and two Ghanaian experts in education (Dr. Kofi Damian Mereku and Mr. Peter Kofi Marfo), who conducted their field work in Ghana between June 9 and June 20, 2013. The team conducted a number of interviews in Accra with a range of stakeholders: USAID/Ghana, MOE and GES officials, various stakeholders including Buck Press and UNICEF, and then visited district offices and schools in four regions (Greater Accra, Eastern, Central and Volta), where data were collected from directors, teachers and children. The research teams also interviewed representatives at the University of Cape Coast, the Education University at Winneba, Aburi College of Education, Jasikan College of Education, and the OLA (Our Lady of the Apostle) College of Education. (See APPENDIX C – SCHEDULE OF RESEARCH ACTIVITIES). 3.1 Research Conducted at CSU The overall evaluation research design called for research to be conducted at the MSIs to determine 1) the background to the program and how it was implemented, and 2) how participation in a USAID￾funded program built the capacity of MSIs to lead future projects. The interviews conducted at CSU were based on several sets of questions posed of leaders, program managers and participants (see APPENDIX B1 – QUESTIONS POSED IN VARIOUS INTERVIEWS AT CSU). 3.2 Research Activities in Ghana With the background information provided by CSU, the Program Team Leader (PTL) developed a background paper/research design that included different categories of questions to be posed to different stakeholders (see APPENDIX B2 – DATA COLLECTION INSTRUMENTS, GHANA STAKEHOLDERS). From the research design, the evaluation team developed a shortened version of key evaluation questions: 1. What has been the impact of TLMP in terms of: a. Materials produced by grade level and subject b. Materials distributed by region and district c. Use of the materials by teachers and children d. Student learning e. Local book printing and distribution 2. What have been the strengths and weaknesses of program management? 3. What are the major lessons learned? 4. Regarding sustainability, to what extent has the Ministry taken ownership of this program and prepared for its continuation? 5. What are the major recommendations for improving the program and to help ensure its continuation? 3.3 Limitations of the Study While, as intended, the evaluation did encompass schools in areas where different languages predominate, the major limitation was insufficient time to conduct an evaluation that included schools outside a radius of one day’s travel each way from Accra. This meant, particularly, that schools in northern Ghana, where opportunities for education have historically been limited, were necessarily left out. In addition, the time limited the number of officials and partners that could be interviewed. Another limitation to the study was the fact that the Ghana trip was postponed multiple times to accommodate USAID Mission scheduling requests, making it necessary to replace local consultants, whose schedules could not accommodate changed dates. The country visit to Ghana was to have been the first to be conducted (in November 2012) but proved to be the last of the six country visits (June Performance Evaluation of TLMP – Ghana 57 2013). By the time the evaluation was actually undertaken, CSU was very involved in its transition and merger of TLMP materials in literacy, numeracy and science from KG1 to P3 with the NALAP teacher’s guides, a USAID/Ghana program launched in 2010 and adopted by the MOE as the standard curriculum for the early grades. So much attention was diverted to NALAP by CSU and the MOE/GES that by the time of the evaluation, many TLMP activities had almost become “invisible,” making it extremely difficult to collect data specifically on TLMP at any level. Another important limitation related to efforts to collect some achievement data from KG2 students and to observe TLMP classes. The majority of classes visited were no longer using the full TLMP program as the workbooks had been filled out by the children during the first year (as instructed) and, thus, were no longer available for use. Also, many teachers were now using the teacher’s guide from NALAP, as per MOE policy, and using TLMP only as resource materials. There were some exceptions to this, especially in the Volta Region, but the number of children tested was very limited and classes were observed in only a few schools were TLMP was still in use e.g., Living Star (private in Ga East, Denguanao and St. Mary’s (private) in Koforidua. In addition, a couple of classes visited in the Volta Region had been strongly influenced by the new Israeli-sponsored program MASHAV that emphasizes low-cost education materials, activity-based learning, and the elimination of separate subjects on the timetable in favor of a thematic or topical approach to organizing learning. GES recognized that many KG teachers had been confused about being presented both TLMP and NALAP programs. TLMP emphasized English and stuck closely to the official KG curriculum, whereas NALAP emphasized L1 and did not adhere as closely to the official curriculum. But, in the end, NALAP became the official KG curriculum and materials were distributed to all schools and KG1 – P3 teachers received initial training on NALAP. The evaluation team visited some districts (Ga East, Effutu, and Cape Coast) and schools (Philip Quaque Girls’ School, Cape Coast and Effutu Municipality KG and Primary School, Winneba/Effutu) where the District Directors and head teachers didn’t even recognize the name TLMP, even though the TLMP materials had, in fact, been delivered to the districts and schools. In some cases, the individuals arrived after the materials had been distributed, but in other cases TLMP no longer had name recognition. Nevertheless, given the combination of NALAP officially replacing TLMP and the fact that in many schools the TLMP workbooks had been used up, it was not surprising to find that TLMP was no longer being used in the majority of classes visited by the evaluation team. The exception was in Volta Region. 4. FINDINGS AND CONCLUSIONS We begin this section of the report with the data collected at CSU. We then move on to the output of the program so that the reader is aware of the number of items produced and to whom they were distributed. Findings and conclusions on the implementation of the program are then presented, as are the results of teacher comments and student reading assessments. 4.1 CSU Achievements and Challenges 4.1.1 Achievements CSU counts the following among its achievements associated with TLMP: • 10 CSU College of Education faculty members and doctoral candidates were engaged with the MOE and the GES in design, writing, production and assessment activities. • 20 CSU students were supported as TLMP interns in Ghana during the grant period. Students worked as researchers, observers and assistants in primary schools and returned to share their experiences with the CSU community. Two Ph.D. dissertations were produced by CSU students with data from TLMP. • Initially, extra salary was provided to CSU faculty for participation, but now participation entails Performance Evaluation of TLMP – Ghana 58 obtaining released time; this has produced better quality and reduced any conflict over extra salaries in departments. • The CSU College of Education created the Center for International Education (at the University). • Faculty development in the international field has taken place. • Papers were delivered at national, professional meetings including six papers presented at CIES • International experience is part of the tenure process. • The diverse experience of faculty prepared them better to address Chicago international education issues to help keep kids in school. They want to bring the lessons learned in Ghana back home – CSU developed international experience to teach students locally. • CSU developed skills and confidence in working with USAID. • CSU strengthened its administrative systems, so that it could wait for reimbursement. This helped to increase CSU’s credit rating. The board of trustees wanted formal reports, so greater accountability was established. • TLMP established national and international partnerships with private sector and commercial entities and raised funds to support the TLMP goals and objectives in the US. For example, United Parcel Services (UPS) donated $6,000 toward shipping. Hinsdale Medical Supply donated 3,000 emergency kits, and UBS Promotional donated 5,000 mathematical sets and school supplies. Buck Press, the Ghanaian printer, donated distribution and warehousing services valued at over $150,000. 4.1.2 Challenges Overall Although CSU had some important successes, it also faced many challenges, some of which were a result of early design decisions. Others evolved over the course of implementation. • There was a major design issue with TLMP as a whole from the outset in that the focus was almost entirely on the development and production of materials, with very little attention being paid to the training of teachers in how to use the materials that were produced. TLMP in Ghana was somewhat atypical in that there were plans to train 5,000 teachers, although ultimately only about 2,000 teachers were trained under the project. In Ghana as in other countries, the teachers themselves generally had relatively little experience in using textbooks or learning materials in their own learning, and training in the use of textbooks is by no means a systematic part of in-service training and is not included in pre-service training. • CSU’s financial management was an issue from the beginning as senior TLMP leadership at CSU was not as effective as was expected during the AEI phase, largely because of lack of internal capacity and skills. • Cutbacks in USAID funding prevented the translation of the materials into Twi, which – to make the materials useful – necessitated a USAID/G response to merge TLMP into NALAP to translate at least the teacher’s guides into Twi. • In addition, lack of funding precluded printing of the P1-P3 materials and the reprinting of the KG workbooks. • CSU had difficulty being taken seriously by GES as the Ghanaian partners considered the results of the initial needs assessments as inadequate. • CSU was beset with lack of cooperation by GES in identifying teachers for and participating in the TOT. Challenges presented by GES and MOE created a number of difficulties in project implementation that led the CSU leadership team to believe that a full commitment by to TLMP was never made and that it will not be sustainable. • The findings of the assessment report in 2010 (an edited version appearing in 2012) were not addressed in developing the framework of operations, leading to a number of ambiguities and misunderstandings. • While USAID/G provided only limited support during the AEI phase, during the expansion phase it Performance Evaluation of TLMP – Ghana 59 became totally involved and, in fact, took over financing and management of TLMP in 2012. • GES and CSU jointly decided that children should be given disposable workbooks with no firm plans to reprint the workbooks. Thus, in about two-thirds of the classes visited by the evaluation team, the children had written in the workbooks (as directed by the project), and, thus, the workbooks were not available in subsequent years. As a result, TLMP stalled in many classes. • KG teachers had limited English language ability and teachers had difficulty in understanding the directions provided in the teacher’s guides. • The evolving change in the direction of language policy in the MOE created confusion between TLMP and the USAID/G program, NALAP (see discussion below). As discussed above, although the MOE was interested in strengthening English when TLMP began, soon thereafter the MOE shifted its emphasis to the use of mother tongue or L1. In early 2010 NALAP (a USAID/Ghana-supported project) took the place of TLMP as the major curriculum in the early grades, and this has had a major impact on TLMP. NALAP quickly became the focus of the Ministry and is now the official language and culture program for KG1 – P3. NALAP has three major components: 1) the development of teaching and learning materials in eleven Ghanaian languages; 2) the publishing and distribution of over 5 million textbooks, teacher’s guide and other materials such as “big books”; and 3) the training of 80,000 teachers. These components have been completed and NALAP has been implemented nationwide. It focuses on L1, although it also includes a limited amount of English instruction. Thus, the initial emphasis of TLMP on English turned out to be a challenge under NALAP. 4.2 TLMP Output TLMP was initially launched in 14 pilot districts that were considered by the MOE as among the most deprived in terms of resource allocation for effective teaching and learning and were also facing serious challenges with regard to gender equity. Criteria used in choosing these districts includes the lack of: access road, accommodation, transport, electricity, communication, water, health facility, school infrastructure, and teachers. Schools were chosen in the following regions/districts. PILOT DISTRICTS CHOSEN Region District District Capital Ashanti 1. Amansie West Manso Nkwanta Brong Ahafo 2. Pru Yeji Central 3. Assin North Assin Fosu Eastern 4. Birim North New Abirem Greater Accra 5. Accra Metro Accra 6. Ga East Abokobi Northern Region 7. Sawla-Tuna-Kalba Sawla 8. Saboba 9. Chereponi Saboba Upper East 10. Garu Tempane Garu Upper West 11. Nadowli Nadowli Volta 12. Krachi West Kete-Krachi 13. Ho Ho Western 14. Bia Bia The major deliverables of TLMP were the development, printing and distribution of instructional materials – teacher’s guide, workbooks, and wall charts or posters. • 8,170,172 teacher’s guides and workbooks were printed and distributed by Buck Press under Phase 2 for kindergarten classes throughout the country. This number includes 4,149,353 million books that were printed for national distribution in 2010/2011; this number exceeded the CA by more than 100 per cent. The approximate cost of each TLM was $1.28. • 220,000 laminated, colored wall-charts were developed and printed by Chicago Laminating. Performance Evaluation of TLMP – Ghana 60 Inc. based in Arlington Heights, Illinois at a cost of $136,710. They were shipped to Ghana and distributed for use as supplemental teaching materials. Some information obtained from interviews and our document review identified a number of anomalies in distribution that could not be clarified. • Communication between TLMP leadership and District Education Directorates was not effective enough since 20 - 60% of the Directors and Store Keepers did not receive the information that materials were coming (Monitoring and Distribution Report 2012. GES. P. vii). This assessment was verified by the visits of the evaluation team to district offices that were part of the national distribution where the team was told that neither the District Director nor the Early Childhood Coordinator were informed about TLMP when the materials were sent to the District Logistics Office. • The quantities of materials delivered at the various regions and districts matched the enrolment of KG1 and KG2 pupils in the majority of the Districts. The mismatches increase as one moves from the District level to the school and classroom levels. 4.3 Program Management and Partnerships 4.3.1 Program Management at CSU and in Ghana Lack of Adequate or Appropriate Staffing. CSU maintained a small administrative staff in Accra and did not have its own full-time technical staff. It relied almost totally on MOE personnel, especially in CRDD, to develop the materials, provide teacher training and to conduct some M&E. The small staff had an important positive aspect in that the direction of the project and the activities were firmly entrenched within GES, which, in turn, helped to ensure ownership by the Ministry. Nevertheless, the evaluation team questions whether it was a good decision for TLMP not to have any of its own long-term technical staff – no one in early childhood education, English and bilingual education, or monitoring and evaluation, either in Ghana or in the U.S. An early childhood specialist was added only after a review and change in program leadership in 2010. Although CSU provided some technical assistance in Ghana and training for MOE staff in the United States, TLMP seems to have been handicapped by not having professional technical staff of its own based in Ghana. This severely limited in-country technical assistance and training and, as described below, greatly limited important M&E activities. Insufficient Communications. The program did not have a strong advocacy component and communication was limited. For example, when the materials were distributed nationwide, there was communication with only some District Directors or the ECE Coordinators. As a result, although the materials did, in fact, get distributed to schools through the District offices, most senior officials were not in a position to offer support or guidance to the teachers. “… the two District Directors from each of the four of the Regions, namely Ashanti, Northern, Upper West and Western said they received information before the arrival of the materials at the District Education Office. As many as six districts from six Regions reported that they did not receive any information prior to the arrival of the materials” (Monitoring and Distribution report, p. 7). “The process of communication between TLMP Officials and District Education Directorates was not effective enough since 20 - 60% of the Directors and Store Keepers did not receive the information” (Monitoring and Distribution report, p. 26). Partly as a result of the lack of communication at the district level, follow-up teacher training did not take place either at the district or school levels. The evaluation team visits to district offices also confirm the lack of communication. As part of the national distribution of materials, the District Directors and EC Coordinators that the team met did not receive any information from TLMP about the materials being distributed. Thus, they were not in position to support the teachers or to help implement TLMP. Inadequate M&E. While a monitoring and evaluation structure had been developed for the initial Performance Evaluation of TLMP – Ghana 61 phase of TLMP in Ghana, for the expansion phase, there was no Performance Monitoring Plan (PMP), no annual needs assessment, and no regular data collection from school visits including interactions with the head teachers, teachers, parents, and children. In addition, there was only one small study that sought to measure the impact on learning of TLMP. Each of the quarterly reports provided by CSU to USAID/W documented certain activities, but the in-depth monitoring of the production, distribution and use of the materials was not undertaken consistently. However, the following assessments were carried out once CSU’s TLMP senior leadership changed in 2010. The information generated by these assessments was valuable, but still represented a limited M&E effort: • TLMP Program Assessment Report (March 2011) • TLMP Impact on Academic Achievement (March 2012) • TLMP Field Test Results (Basic 1 thru Basic 3s) (June 2012) • NALAP Formative Evaluation Report, Ghana (April 2011) 4.3.2 Partnerships with MOE CSU’s primary interaction with the MOE was with the CRDD within GES. From the beginning, CRDD took a leadership role in TLMP, most importantly, determining in the beginning that the program should support the newly incorporated KG classes, and that the materials should be in English and cover literacy, numeracy and science for KG1 – P3. In addition, CRDD took the lead role in developing the new curriculum and materials, with CSU providing important training and technical assistance. From the perspective of CSU, however, it appeared that CRDD initially only wanted the materials and did not express an interest in support for teacher training, training in the US, or in evaluation. CSU stated that working with the GES was problematic. GES would not provide a list of all those who needed to be paid with the topping up funds. Also GES did not want a “project” run by outsiders. To overcome any ambiguity, the project director established contracts with all people and put money in the GES account designated for teacher training food and lodging and required bills to support the use of funds. These administrative issues aside, the senior GES officials including the Director of CDRR, the EC Coordinator, the Deputy Directors General, and the Director of Basic Education all spoke positively about TLMP and its role in helping to meet the important needs for KG. The Director of CRDD noted the following: • Children can't speak adequate English in P4. • There are different schools of thought regarding English and L1, yet the high-stakes exam at the end of junior high school is in English and there is much parental pressure to emphasize English language instruction. • CRDD doesn't have language specialists and is understaffed, with only 18 of some 45 positions filled. • There is only one semester on local languages in the colleges of education, which is not enough to develop necessary L1 skills for the teachers. • There is need to change attitude towards L1 as much of the public remains supportive of emphasizing English language instruction. • The TLMP program created and distributed a large number of books, which helped to close the gap in pupil to book ratio. • Capacity was strengthened at CRDD, including in computer graphics. • CSU has provided some technical expertise, especially language specialists. They provided five days of training at CRDD, sent people to the CIES conference, and provided one month training in the United States for some CRDD staff. Although CRDD was the key partner for TLMP and made major contributions in terms of materials development, the project would have liked for CRDD to have had greater ownership, not just assigning Performance Evaluation of TLMP – Ghana 62 persons to work on the project, but to have been more pro-active and to have provided additional financial commitment, not just in-kind contributions. The GES Teacher Education Division (TED) is responsible for quality assurance in education. However, it appears that TED has been completely sidelined with respect to TLMP. For example, its Director informed the team that he was never consulted about TLMP, was not aware on the initial TOT and has no report of any trainings by TLMP, and had never seen any of the materials prior to the interview. The evaluation team met with district officials in the following seven districts – Accra Municipality, Ga East, Koforidua (New Juben Municipality), Cape Coast, Effutu Municipality, Jasikan, and Ho. In every location, the team also visited schools. The following is a brief summary of the highlights relating to TLMP of the district and school visits. This summary excludes visits to universities and colleges of education, which are described in the next section. Koforidua (New Juaben Municipality) – On June 12, the team split with some members going to Koforidua where they visited the Denguano M/A Basic School and St. Mary’s Day Care and Preparatory School (private). Accra – On June 13, the team visited the Accra Metro Education Director Mr. Alfred Kofi Osie and Mrs. Grace Agyepong Dankyria, the ECE Coordinator, and then visited the following schools: • Osu Presbyterian KG School (actually three schools). This was a well-supported public school. The TLMP materials had arrived and were used for two years, but then the workbooks were used up and, thus, they are not using TLMP any more, although the teachers still use the teacher’s guide as a resource material. Of particular interest was the new Israeli MASHAV program, which has a strong emphasis on locally available materials, active student involvement, and school days organized by topics rather than subject matter. • Osu Presbyterian KG School Annex. In contrast to the above school, this one was small and impoverished with no parental support. TLMP was welcomed, but then discontinued when the workbooks were used up. The teacher’s guides were still being used as a reference. Ga East – On June 14, the team visited Ga East Municipal Education Office and met with the Director Ms. Florence Damali. She was not familiar with TLMP. • The team met with staff at Pantang Presbyterian Basic School. They used the TLMP materials for one year only until the workbooks were used up; the teachers still refer to the teacher’s guides. • The team then visited the private Living Star school. The school received TLMP materials for two years. The workbooks were used up, but the parents would buy them if available on the commercial market. The team visited four KG classes. Cape Coast – One sub-team visited Cape Coast on June 17. Neither the Deputy Director for Supervision (previously a Circuit Supervisor) nor the ECE Coordinator had any knowledge about TLMP. There had been no contact with the District office when the materials were sent to the Stores. • Philip Quaque Girls’ School. The school used TLMP for one year and now uses NALAP. Sometime TLMP materials are used supplementary activities or for homework. • Abakam CRAN Presbyterian School. The head teacher was not there, but the KG2 teacher had not heard of TLMP. • Golden Treasure International School (private). This school did have some TLMP workbooks for KG2, although most were at least partially filled out. Only with effort did the KG1 teacher find the teacher’s guide (no workbooks). Effutu – The Central Region team visited Winneba, Effutu Municipality on June 18 and met with the Director Ms. Hilde Eghan and Ms. Thelma Essuman, ECE Coordinator. The Director has heard of TLMP, but only based on her previous work at GES in Accra. She has not heard of TLMP in her current position (only three months). The ECE Coordinator had heard of TLMP about two years ago at a meeting, but was not informed when the TLMP materials were delivered to the District Stores. The ECE Performance Evaluation of TLMP – Ghana 63 Coordinator said that two teachers had come up with ways to extend the life of the workbooks, including one teacher who had the children write with chalk. The teachers valued the workbooks. No training was provided to the teachers, although the ECE Coordinator thought that use of the materials was straightforward. Teachers are now using NALAP, but use TLMP as a resource. • Effutu Municipality KG and Primary School where the head teacher was not familiar with TLMP. However, the TLMP had arrived and there were many workbooks still available, some fresh and some partially used. The teachers were mending the books and also trying to erase what the student had marked. • ACM KG and Primary School. Only a couple of posters and teacher’s guides in sight. The workbooks were used for one year only. • Ebenezer Memorial Education Centre (private). Again, they used the TLMP books for one year and, currently, the teacher’s guides are just a resource material. Jasikan. The Volta Region sub-team visited Jasikan on June 17, where the Deputy Director Mr. Dompreh and J. K. Daniel verified that they had received TLMP materials. • Jasikan College of Education Demonstration School. The team administered a basic literacy test to eight students. Ho. On June 18, the Volta Region team visited the Ho Municipal Education Directorate and met with Mrs. Dora Agorsor, Director, and Mrs. Beauty Agbeve, ECE Coordinator. This was a TLMP pilot district and they were supplied twice with materials. • Mawuli E.P. Primary School. The second set of books arrived in 2012 and the children were writing in their exercise books in order to preserve the workbooks. • Kpenoe E. P. Primary School. The teachers use the books, although instruction is largely in L1. The team tested 8 students. Accra. On June 19, a team member visited the North Legon Little Campus (private), but found that they have had no experience with TLMP. 4.3.3 Partnerships and Linkages with Colleges of Education and Universities Regarding partnership with the universities and colleges of education, the efforts were mixed. Staff from some of the universities and colleges were involved in helping to create the new materials and to support teacher training initiatives, but when the evaluation team visited the University of Cape Coast, the University of Education, Winneba (UEW), and four colleges of education, they found only limited familiarity with TLMP. Almost none of the staff met by the evaluation team knew about TLMP. In addition, none of the individuals met recalled receiving any of the TLMP materials. There is no evidence that any of the pre-service teacher training programs at the universities or colleges included any TLMP materials or methodology. TLMP signed MoU with University of Cape Coast and University of Education, Winneba to collaborate in the production KG materials. At Cape Coast, three senior lecturers were selected for the implementation committee. Various workshops were held during the first year, but the University withdrew because CRDD took over and sidelined the University. The University did not receive TLMP materials. At the University of Education, Winneba, the team met with a senior lecturer in the Department of Early Childhood Education. Although they teach a course on the preparation of TLMs using local materials, they do not have the TLMP or NALAP materials and no library of their own. The lecturer says that TLMP never involved UEW or those lecturers who were training teachers. He has never seen the TLMP materials. The Principal at OLA College of Education was aware of TLMP only because she attended a workshop on the merger process in her capacity as Secretary for the Association of Principals of Colleges of Performance Evaluation of TLMP – Ghana 64 Education. Otherwise she has not been aware of TLMP and has never received any of their materials. This is unfortunate because they teach the preparation and usage of TLMs in schools. At Aburi College of Education, the senior officials said that they had not heard about TLMP and have never used any of the TLMP materials. The Jasikan College of Education began its Early Childhood Education program in 2008 following a selection process by TED. The Coordinator of ECE said that he had never heard of TLMP and that the college does not use the TLMP materials. One of the reasons for visiting this College is that they are active participants in the new Israeli MASHAV program. From the visits to the universities and colleges, it appears that TLMP did not reach out to the pre￾service teacher training institutions and did not send them TLMP materials. In the initial stages, according to TLMP documents and the report of one senior UCC lecturer, TLMP did invite some individuals to help create the new materials, to assist with the teacher training activities, and for membership on the PAC. Yet, these initial important steps seem not to have carried over into the training programs at these institutions. The training efforts of TLMP focused only on in-service teacher training. 4.3.3 PAC and PTA The Program Advisory Committee (PAC) was established in 2006 with senior level personnel from CSU and the Ministry – Ministry, GES, the Health Book Development Council, the Faculty of Education of the University of Cape Coast and the University of Education, Winneba, among others. The following February PAC held a workshop that established three teams – Design, Writing, and Assessment. Shortly after this meeting, the Assessment Team selected already existing materials for printing. PAC conducted a materials assessment and then identified content area experts to develop the literacy and environmental science materials at both KG1 and 2 levels. They reviewed the national standards to conform the books to them, and identified the topics to be covered in the content. PAC used to meet 2-3 times a year and played an important role as described above. However, PAC had not met for many months before the arrival of the evaluation team. In practice, PAC has evolved into PTA (Program Technical Advisors) that are attuned to activities associated with Modification 8 to the CA. It brings in relevant parties and adds individuals as may be needed. The PTA committee is made up of four CSU experts and four Ghanaians. Typically the PTA invites principals of the colleges of education, teachers, lecturers and other stakeholders to meetings. The last planning meeting took place in June 2013, after the evaluation team left Ghana. 4.4 Program Implementation 4.4.1 Materials Development After the initial printing of existing and revised teachers’ guides and student workbooks, the next materials were developed at CRDD using some of its own staff, plus many individuals from the outside, especially from the Universities and Colleges of Education. CSU provided initial input into some of the design factors, conducted a workshop at CRDD, and brought some CRDD staff to Chicago for training. Nevertheless, this was principally a CRDD activity. CRDD then sent the draft materials to CSU for review and feedback. This was a valuable step. CRDD gave very useful feedback, which was then reviewed by CRDD. Most of the feedback was incorporated into the final documents, but some suggestions, especially in the illustrations, were felt to be culturally inappropriate. Once CRDD revised the materials, it field tested them in a range of schools including different regions and both urban and rural locations. Final revisions were then made before the printing and distribution. Performance Evaluation of TLMP – Ghana 65 A significant design flaw was the decision to use disposable workbooks at the KG1 and KG2 levels. The use of workbooks was strongly valued by the MOE, but neither the MOE nor TLMP had projected funding to continually reprint and distribute them. As a result, in the majority of classes that the evaluation team visited, the use of TLMP materials largely collapsed because the workbooks had been consumed in the first year. Just five of the sixteen public and private schools visited were still using the workbooks. In the Central Region, all but one class visited by the team had used up the workbooks and, as a result, had largely stopped using TLMP. However, in the Volta Region, the team found most classes had figured out a way to continue to use the workbooks: teachers copied the exercises on the chalkboard and then children copied them in their regular exercise books. “The majority of the pupils confirmed that they were not allowed to take the materials home ... Head Teachers and Teachers gave realistic reasons for not allowing the children to take the books home. Parental commitment and involvement are required in order to allow children to take home the books” (Monitoring and Distribution Report, p. 29). However, many schools reported that, at the end of the year, the children were given the workbooks that they had filled out and took them home for good. 4.4.2 Materials Assessment by Evaluation Team The in-country evaluation team had the following comments on the TLMP materials: • The TLMs are colorful, attractive and well laid out. • There are a few pictures that are gender-stereotyped or inappropriate in Ghana (One exception was a picture of a girl and a boy. She holds a doll, he a car). • The TLM teacher’s guides and student books seem to be designed to teach reading to students who already have some understanding of English. • The workbooks and teacher’s guides do not have glossaries with translations of vocabulary in a Ghanaian language. • There is no guide to pronunciation or exercises in sound recognition. This would have been useful since sometimes L and R sounds are reversed by Ghanaian speakers, “TH” sounds are not pronounced clearly and the stress pattern of English sentences during oral work seems to follow that of Ghanaian languages. • There is no presentation of grammar. Tenses are limited to the simple present and present progressive when describing pictures in the student workbook. It would be difficult for students to “talk about” some of the more complex pictures in their workbooks with much spontaneity unless there are some exercises with verbs, prepositions and conjunctions. • Overall, the keen language-learning ability of young children is not put to adequate use. • In general, the materials seem not to take into adequate account the fact that the children are not English-speaking. • There are lists of vocabulary words, but no definitions or translations into any Ghanaian languages. • There are two-word verbs such as “sit down,” “get up,” and “lie down,” but no guidance about how to teach these verbs other than via illustrations. The difference in meaning between, e.g., “sit down” and “sit,” “lie down” and “lie” needs to be explained. • The simple present, past and past progressive tenses appear in the student books and need explanation. • More relevant materials to northern areas of Ghana. • The materials need a pacing chart. • There should be more 3-4 letter words to match pictures. • Pictures need labels and better identification. • Some drawings are not in natural colors. • Some drawings are too difficult. Previously, the evaluation’s Team Leader, Dr. Horn, conducted a detailed review of several workbooks Performance Evaluation of TLMP – Ghana 66 and teacher’s guides, reflecting (a) pedagogical quality and (b) physical and design quality. Annex F begins with a delineation of the major evaluation criteria for the materials and continues with detailed analyses for KG1 in literacy and numeracy, and KG1 and KG2 environmental science. In general, TLMP can be commended on the quality following the workbooks and teacher’s guides. In Literacy for KG1, some negative comments include – children looking sad, inappropriate illustrations, emphasis on sight reading rather than phonics, lack of assessment activities, and marginal durability. In numeracy, overall very good, but it may have been better to start with numbers rather than classification. The KG 1 Environmental Science has many strong characteristics, but may draw on vocabulary and writing exercises that are beyond what has been taught in the literacy lessons. The only materials that were printed and used in classrooms were for KG1 and KG2. However, TLMP did complete the development and field testing of P1-P3 teacher’s guides and student workbooks in literacy, mathematics, and natural science, a total to 18 books. However, there are no funds in TLMP to print these materials. In addition, the national curriculum has changed, so there is need to review these materials. It seems likely that the mathematics and science materials would be useful in supporting the existing curricula in these subjects. And, as part of the merger process, these materials were reviewed and some content and activities have now been incorporated into the NALAP teacher’s guides. 4.4.5 Teacher Training Teacher training was included in this CA from the outset. CSU held two training-of-trainers (TOT) workshops for educators in 2008, one in Sunyani in Brong-Ahafo and the other in Koforidua in the Eastern Region. Trainers were equipped to facilitate workshops with 300 KG teachers in the 14 districts on the use of the textbooks and learning materials. Based on this experience, CSU determined that during the 2009-12 expansion period, other TOTs would be held, this time targeting 2,000 additional teachers. The objectives of this latter training were to: • Enhance teachers’ expertise in using the TLMs • Equip them with a deeper understanding of the best practices for use of the materials • Prepare participants to use the TOT model to disseminate best practices to teachers The first tier of the TOT model was implemented in May 2011 to 24 trainers and was evaluated to make decisions about the effectiveness of the TOT model in providing professional development workshops. Five training modules were created focusing on the following areas: how children learn, learning styles, teaching methods, teaching strategies, and songs, poems. Districts that participated in this training included: Accra Metro; Tema Metro; Ga East and West Municipal; Kokuku Krowor Municipal; Akwapen North; Birim North; Kwahu North; New Juaben, and Suhum Kraboa Coaltar from the Greater Accra and Eastern regions. Training was seen as a two-step process: the TOT provided to district level facilitators, and the delivery of workshops by these facilitators in their respective districts. It is unfortunate that only the first part of this training was evaluated and there were no efforts to follow up with the facilitators in the field to determine how any subsequent workshops were implemented with teachers and how frequently this was done. However, CSU compiled and published a book on the topic, CSU Impact on Literacy in Ghana, West Africa (ISBN 978-0-9837851-3-2), which includes the needs assessment, mid-term assessment, teacher training handbook and modules, two doctoral dissertations, guidelines for writing, editing and formatting TLMs, guidelines for researchers, and the sustainability report. Portions of this book were provided to the team, but an outcomes/learning assessment was not included. This was to have been completed by CRDD and other Ghanaian partners. Under the initial terms of the CA, CSU envisioned training 5,000 teachers. Several issues prevented the reaching of this target: 1) the lack of cooperation by the GES in identifying teachers, especially in light of the adoption of the MOE of the NALAP curriculum; 2) USAID/W’s budget cuts that prevented payment Performance Evaluation of TLMP – Ghana 67 of accommodation, fees, and payment of facilitators; and 3) failure of GES and printers to agree in a timely manner on the distribution of books. The final number of teachers to be trained was 2,000. Facilitators were to have trained lead teachers at schools in each district, who then were supposed to train their KG colleagues. GES started but then did not complete the selection process throughout the country. Hence, the training covered only those schools in the 14 districts that had been part of the pilot (although TLM distribution, ultimately, was national). Providing an honorarium for lead teachers was not recommended. Moreover, it was determined that KG teachers, especially in the deprived areas, lacked the competence to use the TLMs due to the fact that most of them were untrained, the materials were new in the GES curriculum, and many could not communicate well in English. This made using the teacher’s guides extremely challenging for teachers. 200 LTs were to have been chosen; many GES staff attended TOT along with the 150 teachers. Trainees were district trainers who were to train the teachers in their districts, but only one did this – others did not as there was no incentive for cascading. When the TOT was presented, it was supposed to be first in a series, but then the budget was cut and the TOT had to stop. Unfortunately, a report on the training was never written or the documentation is missing. The GES did one TOT but never shared their materials. In spite of the TLMP training of 2,000 and the subsequent TOT, the evaluation team did not come across a single teacher who said that he or she had received training on TLMP. 4.4.6 The Current and Anticipated Future Status of CSU and TLMP In some ways TLMP seems to have stalled during the past year. All the KG materials had been developed, printed and distributed and many teachers trained. And the P1-P3 materials were developed, but there are no funds for printing. Also, even in the schools, although some teachers had used “work-arounds” such as having learners write in regular exercise books, the majority of teachers met by the evaluation team had stopped using TLMP as the student workbooks have been used up. In addition, NALAP materials had been distributed and instructions given to teachers that NALAP was now the official program. Nevertheless, TLMP was developing and implementing an important exit strategy consisting of the following: 1. The merger of key concepts and activities from TLMP into the NALAP teacher’s guides. The goal of the merger is to capture and combine the best practices of TLMP into the NALAP program in terms of methodology, content, and activities in order to strengthen the English component. In addition, some of TLMP practices would also strengthen the dominant L1 program within NALAP 2. To reach consensus concerning the proper balance and usage of L1 to facilitate a smooth and successful transition to English instruction by P4. 3. To print, and distribute the revised NALAP teacher’s guides. 4. To train all KG1-P3 teachers in the new teacher’s guides 5. To use effectively technology to reduce costs and to ensure a high quality teacher training program. The evaluation team attended one of the workshop activities where the merger or fusion process was taking place, a process that began in October 2012. Although there still wasn’t total clarity regarding the possible future existence of TLMP materials, the subject matter teams had make much progress and expected to complete the merger of the materials by the end of July 2013. Shortly after the evaluation team departed Ghana, TLMP held an important workshop in June including the CSU project director that brought together the key players in Ghana to review the merge process including some of the new materials, and to finalize the strategy to train over 100,000 KG-P3 teachers at the end of 2013 and the beginning of 2014. Performance Evaluation of TLMP – Ghana 68 A part of the strategy is to create and distribute DVDs with model lessons, presentations on methods on teaching basic reading skills, etc. that can be used to support the training process and even reviewed by individual teachers at a later date. The DVDs should help to ensure that the quality of the training is maintained throughout the cascade process. TLMP plans to train a 54-person National Resource Team (5 days). These people, in turn will train 1,100 master teachers (3 days) at the district level, who will then train approximately 100,000 KG1-P3 teachers. 4.5 Outputs and Outcomes 4.5.1 School/Teacher/Classroom Observations The evaluation team visited seven district offices and sixteen schools (eleven public and five private) in four regions. The predominant local languages in these regions are the Fante and Twi dialects of Akan, Ga, and Ewe. School Location PUBLIC Osu Presby. Accra (Greater Accra) Osu annex Accra Abakam Cape Coast (Central Region) Philip Quaque Cape Coast ACM Effutu (Central Region) Effutu Effutu Presby. Basic School Ga East (Greater Accra) Kpenoe Ho (Volta Region) Mawuli Ho Jasikan Demo Jasikan (Volta Region) Denguano Koforidua (Eastern Region PRIVATE North Legon Accra Golden Treasures Cape Coast Ebenezer Effutu Living Star Ga East St. Mary’s Koforidua (*) Schools marked in italics were still using the TLMP workbooks. (**) Schools in bold were in the original 14 districts, which were identify as some of the deprived districts. The other schools were added as part of the national distribution of materials. The major findings from visits to the sixteen schools are as follows: • All but one of the private schools and all of the public schools had received the TLMP materials. The Ministry approved the distribution of the materials to properly registered private schools, but there is no information on the number of private schools that received the materials. • All of the private schools emphasized the teaching of English; one school had a sign in the KG classroom stating “English is the medium of instruction.” Given the emphasis on English, it Performance Evaluation of TLMP – Ghana 69 would appear that the TLMP materials would continue to be very useful. • Of the sixteen schools visited, only five schools or 30% still had a supply of workbooks and were using them. This included only one of the private schools as well. • All of the schools spoke highly of the materials, especially the workbooks, which they said were important learning tools for the children. • The majority of the schools (10) said that they had used up the workbooks, with most schools having the children take home the books at the end of the school year. Except for two schools that had received a surplus of books, these schools had no more books for the following year and, thus, effectively stopped using TLMP. • The five schools that still had books did so as the result of teachers who realized that more workbooks would not be forthcoming, and they figured out ways to preserve the books. Some teachers arranged to erase most of what the children had written. Other teachers had the children write in their exercise books. A few teachers even tried to photocopy some pages, but this proved too expensive. In short, the decision to use disposable workbooks was ill-advised, and even when the materials were sent to all districts, there was no advice from TLMP about strategies to preserve the books. • Some teachers said that if the workbooks were available on the open market, they would encourage parents to buy them. • None of the teachers met by the evaluation team had received any training on TLMP. • The posters were in evidence in most schools and are likely to be the longest lasting of the TLMP materials, although there was little evidence that they were actually being used. The TLMP materials could continue to be of special value in both private schools and public schools in cosmopolitan areas where there are so many local languages that the school needs to use English as the medium of instruction. 4.5.2 Learner Reading Assessments and Outcomes There are two sources of data on learning achievement. The first, described below, is an English test administered by the evaluation team to 24 students in three schools in the Volta Region. The second is a test administered by TLMP. The Assessment Instruments and Process The purpose of the materials prepared by TLMP is to help Ghanaian children in kindergarten to prepare for listening, speaking, reading and writing in English. A learning assessment instrument was developed by team member Dr. Eric Allemano to measure student performance in English – speaking, listening, reading and writing (included in Appendix C) and was administered to children in three schools in and around Jasikan and Ho in the Volta Region. The sample was comprised of a total of 24 students – two boys and two girls in both KG1 and KG2 in each school. The sample is small because so few schools visited by the team – five of the sixteen public and private schools – were still using the TLMP program. The remaining schools never used the materials or had stopped using TLMP once the workbooks had been filled out by the children and, thus, were no longer available for subsequent years. Because of the limited sample, it is not possible to generalize from these results to the impact of TLMP on student learning. Nevertheless, the results give some indication of the ability in English of at least these KG1 and KG2 students. Certain factors made the assessment challenging. First of all, the children were very young (aged 4 in KG1 and 5 in KG2) and had little or no contact with English outside of the classroom. Another factor was the reduced number of hours devoted to English since the introduction of instruction in national languages under NALAP. Although the research was done close to the end of the 2012-2013 school year, the students in the Volta Region had covered only half of the lessons in the TLMP materials for their grade level. As a result, the children had had only half of the instruction that they were supposed Performance Evaluation of TLMP – Ghana 70 to have received and their vocabularies were very limited. A further factor made it difficult to assess letter recognition in English. In Ghana, children generally learn the sound of a letter rather than its name. For instance, they learn that the letter A is ah rather than “Ay”, the letter B is bh rather than “Bee” and the letter C is kh instead of “See”. This made it challenging for the American researcher to test letter recognition among the students as he was not certain of how “C” and “K” were differentiated or whether vowels were long or short in the Ghanaian system. Literacy and speaking skills were assessed by five methods. 1. Listening and speaking were tested by asking the child to answer a series of questions about himself/herself in English (e.g., What is your name? Are you a boy or a girl? How old are you? What grade are you in?). 2. Letter identification was conducted by asking children to name 10 letters in English (both upper and lower case). 3. Oral (speaking) fluency (word accuracy and phrasing) was assessed by asking KG2 students to describe a picture they were familiar with in the TLMP workbook (for example, a boy sitting at a table, reading a book near a bookcase, a woman in a shop, choosing fruit). The exercise was too difficult for KG1 students so they were asked to name items that the researcher pointed to on the TLMP wall charts (which did not have word labels). 4. The KG1 students were also asked to identify pictures with words (ex. fish, in the workbook) and to read the word under the picture. Word recognition for KG2 students was conducted by having the children read 10 words drawn from the KG2 workbook. 5. Writing competence was measured by asking the child to write three letters, write his/her name and/or one word of her/his choice. Results The children assessed had success in certain language skills (word recognition and naming pictures of familiar objects) and great difficulty in other areas (describing a moderately complex picture in the workbook). In doing word-picture assessment the researcher found that in some cases students seem to have memorized the word-picture sets in the order they appeared on the page. If the researcher chose the word-picture sets at random, some students had difficulty giving the right answer. Characteristic KG 1 KG 2 Comment 1. Listening & Responding 60% answered at least 1 question 100% answered at least 1 question 1 correct answer: usually the child’s name. 2. Letter Identification 70% identified 80% 90% identified at least 80% 3. Oral Fluency 0% able to describe a picture 25% able to describe a familiar picture in the workbook. The researcher had to ask probing questions. 4. Word Recognition 20% recognized 70% of pictures in the workbook 35% recognized 80% of the pictures chosen on a wall chart 35% identified at least 70% from workbook 65% recognized at least 90% of the pictures chosen on a wall chart 5. Writing 65% able to write 3 letters 65% able to write their names 15% able to write a word in English 90% able to write 3 letters 100% able to write their names 65% able to write a word in English Performance Evaluation of TLMP – Ghana 71 Conclusion It is evident that students do improve their skills in English by the end of KG2 and that the TLMs seem to be correctly used. There is evidence that the TLMs help the students using them to outperform students in schools that do not use them, as shown by test results collected by CSU in 2012. However, it is difficult to compare the results of the testing done in the three schools visited in the Volta Region with those of testing done a year earlier by CSU. Literacy Content Domains TLMP Sch. Control Sch. Mean S.D Mean S.D Conversation 1.83 0.346 1.79 0.367 Visual Discrimination 2.70 0.519 2.56 0.622 Phonological Awareness 0.28 0.395 0.39 0.413 Auditory/Visual Comprehension 1.49 0.613 1.35 0.603 Awareness of Positions 1.01 0.840 0.74 0.772 Source: Assessment of the Impact of TLMP on Pupil Achievement in Some Selected Districts The results of the testing done by CSU indicate that with the exception of Phonological Awareness Content Domain, TLMP school students outperformed their control-school counterparts on content Domains in language and literacy. • Children do seem to make progress in English literacy skills as they move from KG1 to KG2 • Children perform better at tasks that are more behaviorist than cognitive (memorization over thinking) and may reflect a preference of teaching style and pedagogy • There were no obvious correlations between letter naming and word reading accuracy, or between letter naming and comprehension • Children in both KG1 and 2 have minimal expressive language facility (e.g., none of the children spoke in complete English sentences). • Given that most of the test items were drawn from the student’s grade level TLMP workbook or the wall charts, it is apparent that both the linguistic level and the instructional objectives of the TLMs are above most children’s language and reasoning abilities. The TLM reading method is quite reminiscent of “Dick and Jane”, the main characters in popular basal readers that were used for decades to teach American children to read. The books relied on the whole word or sight word reading method, and repetition, with some limited attention to phonetic analysis. This approach has been essentially replaced in English-speaking countries by various adaptations of phonics. However, the TLMP materials that were developed for Ghana do not include phonics or any exploration of sentence structure and grammar.. One difficulty of using the sight word reading approach with KG1 and 2 students in Ghana is that for the most part, they do not know English. While teachers may translate words and sentences, the TLMs appear to assume that the learner already knows how to speak English reasonably well but needs to learn how to read it. The absence of phonics/pronunciation exercises and of practice with sentence structure and basic conjugations is a further limitation to the TLMs in Ghana. An ESL/EFL approach would have been more appropriate for the vast majority of learners. 4.5.3 Other Evidence of Learning TLMP did conduct one study that measured learning achievement. It consisted of a limited number of classes and children.5 A total of about 474 students were tested in the three domains – literacy, 5 TLMP Evaluation Final Report, 2013 Performance Evaluation of TLMP – Ghana 72 numeracy and environmental science. Half of the children were in TLMP schools and the other half in control schools. The results showed only a small difference in favor of TLMP classes in literacy, and almost no difference in numeracy and science, as shown in the table below. Literacy Numerac y Scienc e TLMP 7.30 8.84 7.96 Control 6.65 8.78 7.96 It should also be noted that the TLMP Assessment Report, Final (2011) included a finding that, of the teachers who returned the questionnaire “91% had observed positive changes in the children’s achievement since the TLMP workbooks and materials have been used.” P. 33. But as clearly stated in the Report, this finding was based only on teacher perceptions and not any test data. 4.6 Educator Comments on TLMs Staff at all but one of the sixteen schools visited by the evaluation team were familiar with TLMP, but only five schools were continuing to use the workbooks on a regular basis. This notwithstanding, staff at most of the other schools indicated that the teacher’s guides and workbooks were good reference materials. All the teachers who had used the materials highly praised the teacher’s guides and workbooks. They especially appreciated that the teacher’s guides closely followed the curriculum and, thus, continue to be a good reference source. In addition, all the teachers spoke highly of the workbooks, saying that they were an important learning tool for the children. In schools where the workbooks were no longer available, and where parents were buying other workbooks in the commercial market, the teachers said that they would highly recommend to the parents that they buy the TLMP books, if they were available. 4.7 Stakeholder Interest In and Use of Materials/ Sustainability CSU worked with the Mission to try to ensure sustainability of TLMP, and PAC also sought ways to ensure sustainability. But the evolution and dominant role of NALAP means that the legacy of TLMP will likely be a stronger English component within the NALAP teacher’s guides resulting from the merger or fusion process. The GES senior personnel met by the evaluation team were very supportive of TLMP and valued the contribution that the materials and training made in support of the newly incorporated KG classes into the educational system. They would still like to see some of the TLMP content used to support the English language component of NALAP, which has been the major focus of the recent merger or fusion process. They would not like, however, to see a continuing separate identity for TLMP. Nevertheless, the TLMP materials are perceived as good, especially for teaching L1, and should be translated into local languages to help children learn to read. NALAP, with its focus on L1, but also with an English component, has become the new official language program of the Ministry and will subsequently be called just Child Literacy. The TLMP teacher’s guides that are currently in the schools can continue to be used as reference materials, and the workbooks that still remain in some schools can continue to be used until they are completely worn out. From discussion with the two GES Deputy Director Generals, the MOE is not interested in having these workbooks reprinted and, in any case, TLMP does not have the funds to do so. It should be noted, however, that NALAP does not include workbooks for the children, although the MOE recognizes the value of having them. Based on the TLMP experience, it would not be a good Performance Evaluation of TLMP – Ghana 73 decision to develop and print workbooks to be filled out by children because the Ministry cannot afford to replace them every year. CSU and its partners have struggled with alternatives to disposable workbooks, which might be applied to other reading activities. Some ideas are as follows: • Use “Activity Books,” which may be similar in many respects to workbooks, but children would not write in them. The written exercises would be done in the children's exercise books. One variation of this Activity Book would be an occasional page that the teachers could photocopy for the children. Also, the teacher might copy some exercises on the blackboard. • Alternatively, should there be funds for some repeated printings, perhaps with plasticized pages of the workbook so that children could write on them, perhaps with crayons, which could later be erased. Costs could be reduced by using black-and-white illustrations for the workbook instead of color (Buck Press estimated a 20% savings.) The Director of CRDD noted that in any event, materials need to be replaced every three years. The Scope of Work for this evaluation did not call for review of current approaches for teaching literacy in Ghana. However, the team noted that the TLMP materials assumes a greater ability in English among the KG students than the team observed. The methodology, however, could be applicable for enhancing learning in L1. 4.7 Conclusions The overall conclusions of the field study are the following: Positive: • Support of KG. TLMP served a useful purpose, as the KGs had just been incorporated into the Ministry and there were not instructional materials. TLMP developed good teacher’s guides that were closely aligned with the curriculum, pus attractive workbooks and posters. The focus was on English language instruction, which met a felt need of the Ministry at that time. It helped to give important support to early childhood education and to developing the foundations for reading skills in English. • Materials. TLMP produced good teacher’s guides, attractive workbooks, and posters for KG1 and KG2 that were highly valued by the teachers. Virtually all of the persons whom the evaluation team met appreciated that the teacher’s guides were aligned with the curriculum and that the workbooks were effective in helping to children to develop preliminary skills for reading. • Printing and distribution. TLMP is to be highly commended for printing and distributing such a large number of good instructional materials – over 6 million. This is a major accomplishment. • Shift to focus on L1. Although the changing circumstances emphasizing L1 were outside of TLMP’s control, TLMP was still able to make a contribution by strengthening the limited English component of the new NALAP teacher’s guides through the “merger” process. • Local leadership. From the beginning, CSU sought to place responsibility and the leadership of TLMP in the hands of GES. CRDD was, in fact, the lead organization for developing the new materials, guiding the teacher training program, and for all other aspects of the program. In addition, both the PAC and PTA included a range of senior Ghanaian professionals in education. • Capacity development. CSU had several activities designed to strengthen local capacity, especially in CRDD. They held a one-week workshop in Accra, hosted CRDD staff for training at CSU, sponsored educators to attend a CIES conference and, perhaps, most importantly, provided valuable ongoing feedback on the new instructional materials being developed. Negative: • Workbooks The decision to create disposable workbooks was unfortunate as most of the schools, as instructed by TLMP, had the children faithfully fill out the workbooks during the first year and then, at the end of year, sent the workbooks home with the children. Thus, there were no workbooks for the subsequent year and, in a majority of the schools, the teachers were unable to Performance Evaluation of TLMP – Ghana 74 continue with TLMP, only using the teacher’s guide and, perhaps a remaining workbook as reference materials. It was unrealistic to think that either the donors or the Ministry would be able to reprint the guides in subsequent years. And even though this fact should have been obvious when it was decided to distribute the materials nationwide, there was no guidance for the teachers on how they might preserve the life of the workbooks. • Language policy. The impact of TLMP will be limited because of a change in Ministry policy in support of L1 and the NALAP program. Although this change is not the fault of TLMP, perhaps TLMP could have responded earlier to the changing environment. While appreciative of the role that TLMP played and the materials developed, the Ministry no longer supports TLMP as a separate entity or the printing and distribution of the materials. NALAP is the new official program, and TLMP’s last useful initiative is to merge the useful concepts and activities into the NALAP teacher’s guides in order to strengthen the small English component. In addition, TLMP will provide additional support by printing the new teacher’s guides and supporting a large-scale teacher training program at the end of 2013 and beginning of 2014. • P1-P3 materials. Both teacher’s guides and workbooks were developed and piloted, but they were never printed because of a shortage of funds. Now, with the new NALAP program, there are no plans to print these materials, which may waste potentially useful resources, especially in support of the existing mathematics and natural science curricula. • Pedagogy. For all the positive aspects of the TLMP materials, the approach to teaching basic reading skills seem a bit off in in two regards. First, both the teacher’s guides and the workbooks assume an excessively high level of English skills among the children. For example, the pace of vocabulary development is far too fast and the language required to discuss the drawings in the workbook or posters is beyond the ability of the children beginning in KG1. This approach would be much better suited for instruction in L1. Perhaps the program would be useful if translated and adapted to teach L1. The second concern is a clear strategy for teaching basic reading skills. The materials over￾emphasize sight recognition or a “whole-word” approach, and do not give enough emphasis to phonics. • Communication. TLMP fell short in its communication. For example, when the materials were distributed nationally, there did not seem to be any communication with the District Directors or the ECE Coordinators about the materials and how these key officials could help to support the program. The evaluation team met many individuals who had not heard of TLMP itself. A few individuals, such as head teachers, had, in fact, been aware in the past when materials were delivered, but had simply forgotten about TLMP for lack of any further communication. • Universities and Colleges of Education. Although TLMP involved some individuals in creating the materials and carrying out teacher training from the universities and colleges (including MoUs with UCC and UEW), there was no follow-up to help ensure that their pre-service programs incorporated the TLMP program and materials. • In-service teacher training. TLMP limited its teacher training activities to in-service, yet even this was too limited. TLMP trained about 2,000 teachers and also had a planned TOT program whereby district officials would carry out local teachers, but this did not happen. In fact, every one of the teachers interviewed by the evaluation team said that they had not received training in TLMP. • M&E. The inadequacy of M&E activities was a major shortcoming. TLMP had no ongoing ability to follow-up on their training initiative, the distribution of materials, any issues related to program implementation, and measurement of the impact of the program on learning. CSU did manage to carry out about four studies, but this could not replace the need for regular monitoring and evaluation activities. In fact, CSU did not even have a PMP, which seems unusual for a USAID cooperative agreement. • Lack of adequate or appropriate staffing. Part of the deficiencies stem from lack of professional expertise on the TLMP team. The program staff was largely administrative and there were no full￾time professionals in ECE, bilingual education, or M&E. Performance Evaluation of TLMP – Ghana 75 5. LESSONS LEARNED • Need for experienced implementing institutions. A major objective of the USAID/W initiative was to strengthen the capacity of MSIs by giving them exposure to international development and working with USAID. Yet, at least in the initial stages, CSU had difficulties in administration and finance that stemmed, in part, from lack of experience in this area. Even on the programmatic side, a more experienced institution, or with appropriate project staff, might not have decided to have disposable workbooks, or to use an approach to reading that was better suited to L1 than L2, or such a limited M&E capacity. • Full-time technical staff in-country. A project of this magnitude needs at least minimal professional staff, either local or international, in areas such as ECE, bilingual education, reading, and M&E. • M&E support. There is always a need for good M&E support, beginning with a PMP that identifies major indictors of success and guides data collection and analyses, plus a strong monitoring program to assess implementation progress. In addition, especially with a program such as TLMP whose objective is to improve reading skills, there is need for a strong summative evaluation program.. • Sustainability. Consideration of sustainability is important from the very beginning of a project. In the case of TLMP, there was a major error in not recognizing that future funding to reprint workbooks was unlikely. In addition, as soon as NALAP was identified as the new language program focusing on L1, TLMP might have been able to initiate collaborative efforts much sooner for the benefit of both programs. • Need for good communication. A program such as TLMP needs a good communication strategy, especially one that reaches out to key partners. For example, when the materials were distributed nationally, there was need to information District Directors and ECE Coordinators on TLMP – the objectives, materials and the potential role of the districts to support the program. Periodic communication including the solicitation of feedback would be a great help • Include key partners. TLMP focused on in-service teacher training, but neglected the importance pre-service training. It is important to include adequately all relevant institutions, in this case, the universities, colleges of education, and TED. 6. RECOMMENDATIONS We have divided the recommendations into two categories. The first is on improving the materials. The workbooks in their current form are not likely to have any further use in Ghana given the new NALAP program. An exception to this could be in private schools where English language is emphasized or public schools in cosmopolitan areas where English is used because there are too many mother tongues represented in the classes. We also recommend that the TLMP workbooks be made available in the commercial market. In any of these situations, the following recommendations to improve the materials may be useful. The second category is on recommendations to improve the relevant education systems. 6.1 Materials Improvement • Create new activity books for NALAP based on the TLMP workbooks. The activity books could cover both the mother tongue and English components. The major difference between an activity book and a workbook is that the children would not write in the activity books, which avoids the major problem of annually having to replace workbooks at a cost that is unaffordable to the Ministry. • As mentioned elsewhere in this report, the methodology of the TLMP materials is a better match for teaching introductory reading skills in L1 rather than L2 because the exercises, vocabulary, and grammatical structure in the KG1 materials assume a level of English that is beyond the level of young Ghanaians. The materials and methodology, however, could be applied to the L1 NALAP materials. Performance Evaluation of TLMP – Ghana 76 • The instructional approach to reading should rely much more on phonics rather than word sight recognition. • Before abandoning completely the idea of a workbook, examine some options to preserve the workbooks, while still giving children the opportunity to write in them. One such option might be to use plasticized pages on which the children might write with crayons or chalk, which could be removed at a later time. • There is need for a glossary in English and L1 languages. 6.2 Systems Improvement There are a number of ways in the educational system might be strengthened to support TLMP: • Once it was known that NALAP was to become the official language curriculum for KG and the early primary grades, perhaps USAID, CSU and GES could have begun a process much earlier than the current “merger” initiative to determine the best options for TLMP to support the change. • The MOE could have had greater accountability – greater commitment than just assigning a person to work on the project, be active and provide more financial commitment up front. They should make a financial commitment, not just in kind. • It would have been helpful if the MOU with the MOE had specified sustainable policies relating to ECE and language teaching, the use of L1 and L2, the printing of materials and teacher training. • There was need for a more clear structure for payment of MOE personnel who helped to develop materials or conducted teacher training. • There is need to ensure that all key partners are fully and properly involved such as the universities, colleges of education, and the GES’ Teacher Education Division. • KG teacher training at colleges and universities need to be synchronized and made relevant to the curriculum and materials in schools. • There is need for a full training program at the pre-service as well as in-service level. • The ECE Coordinators need special training and support so that they, in turn, could guide the training and support of the KG teachers in their districts. • The book distribution system, although basically sound with the support of Buck Press, needs fine tuning because some locations in the country were not well served. • There is need for a good M&E system for both formative and summative evaluation tasks. 6.3 Linking TLMP to USAID Priorities in Early Grade Reading Early grade reading is a high priority of USAID as stated in the USAID Education Strategy, 2011 Goal one: “Improved reading skills for 100 million children in primary grades by 2015.” Improved literacy has a modest impact on national economic growth and helps to ensure the retention and success in subsequent grades. The strategy also recognizes the importance of early childhood education: “The strategy recognizes the importance of educating children in their native languages where possible in the earliest grades” (USAID Education Strategy, 2011 Goal one, P. 9). TLMP fits well with these USAID priorities, especially early childhood education and the development of basic reading skills. In hindsight, the choice of focusing on English language instruction turned out not to be in line with both changing priorities in Ghana and within USAID – shift towards more emphasis on developing basic skills in the mother tongue. USAID/Ghana, in fact, supported two projects with a mother tongue emphasis, first EQUAL and then NALAP. By the end of TLMP, its legacy will be an improved English language component within NALAP. TLMP also missed out on two other relevant areas regarding reading. While it developed good teacher’s guides, workbooks and posters, it seemed not to have a clear strategy for teaching basic reading skills. Some other USAID programs such as MTPDS in Malawi developed excellent phonics- Performance Evaluation of TLMP – Ghana 77 based guides for teachers. Also, TLMP lacked a rigorous approach to measuring the impact on learning of its materials and teacher training initiatives. In part, this will now be corrected by the development and implementation of an EGRA program being implemented with Mission funds with the support of RTI. Performance Evaluation of TLMP – Ghana 78 ANNEX A. QUESTIONS POSED IN VARIOUS INTERVIEWS CONDUCTED AT CSU GENERIC QUESTIONS POSED OF ALL MSIs TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did you university provide? What challenges emerged in your work with the MOE? How were they resolved? • What process did you use to review and revise the materials? How were the materias, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? Performance Evaluation of TLMP – Ghana 79 • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? Program Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the program improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school (Ghana – kindergarten)? About distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? SPECIFIC QUESTIONS POSED OF TLMP LEADERSHIP TEAM AT CSU 1. National Curriculum • What is the national curriculum for K1? • Get copy of K2 materials (only have K1) (K2 includes: auditory discrimination, auditory comprehension, conversation, storytelling, pre-reading activities such as: sorting, writing, left to right eye movement, visual discrimination/memory skills, visual comprehension, understanding positions, reading pictures, phonological awareness, introduction to the letters of the alphabet, associating lower and upper case letters, matching objects with letters, letter sounds, formation of two-three letter words/sounds and rewriting activities) • What changes are being made in the national curriculum for English, math and environmental studies (NALAP addendum, p. 9) • What is the current language policy in education in Ghana? • What happened to the P1-3 materials? • What impact has the project had on the national curriculum? On other policies related to KG? Performance Evaluation of TLMP – Ghana 80 2. Research Reports • Get March 2012 study of three schools on achievement • Have you videotaped any of the teachers delivering lessons in each of the three areas using the TLMs? • Get field test reports for P1-3 materials 3. Book Printing/Distribution • What happened to the printing equipment that was purchased originally and given to the GES? • How was the distribution of books monitored? Who was ultimately responsible for distribution? • From your studies, how many workbooks were delivered to each district/school? Does every KG child have a workbook? How do teachers monitor the work done by the children? 4. Merger with NALAP • Get original proposal; transfer of responsibility from USAID/W to USAID/Ghana – who will be your COTR? • How different is the TOT model to be implemented in the merger vs. in TLMP? What lessons were learned? • Pages missing from 8/22/12 document • How will the bilingual/L1-L2 transition affect the TLMP materials in all three areas? What did you learn in how these materials need to be adapted to a bilingual environment rather than just English? • What will the review/editing/field testing process look like? How will it be different from the TLMP process? What lessons have been learned? • What lessons were learned in the earlier phases of TLMP that will inform how you do things under the merger? • What team will you need to assemble for the merged activities? 5. Teacher Training • In choosing the master teachers, how did they cope with English? • How was this monitored after the TOT? • How many teachers did each TOT trainee train? • Was the whole country ever covered? What is the plan for including all districts? • How did TLMP TT plan fit in MOE/GES overall teacher training activities, both pre-service and in-service? • Are TLMP materials now included in pre-service TT? • What were the evaluation results of the cascaded model? • How did teachers measure improvements in their students after teachers were trained and students used the workbooks and materials? 6. Book Production • In the earliest phase of the TLMP project, certain already-existing titles were reproduced. How many of these books/titles were reproduced vs. actual number of materials produced, printed and distributed? • How were the books field tested? • The original intention of the project was to create materials in the mother tongue. What challenges did you face when the budget was cut and the materials were not translated? Performance Evaluation of TLMP – Ghana 81 7. Team • How did you become involved in TLMP? What were your roles and responsibilities? How have you seen the project change over time? What were its strengths? Its weaknesses? What were the lessons learned? • Who are the CSU team members now? What are their skills/abilities/subject matter specializations? Who can comment further on TLMP up to now? • Who are the Ghanaian team members now? What are their skills/abilities/subject matter specializations? Who can comment further on the TLMP up to now? • Names and contact information in Ghana of all partners. Performance Evaluation of TLMP – Ghana 82 ANNEX B. DATA COLLECTION INSTRUMENTS, GHANA STAKEHOLDERS In Accra • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the experience of TLMP leveraging or leading to Chicago State Univ. working on other educational development activities in Ghana or elsewhere in the future? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in this country? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • Would you say that the model for TLM creation developed by Chicago State Univ. (working with teachers to write the materials) can be utilized in other areas? Or was this a process you feel can be used for only supplementary or complementary materials? How would it need to be improved to be useful in other contexts? • How satisfied were you with your relationships with Chicago State Univ. and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar program in the future? 2) MOE Administrators • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the program or were program responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOEST/GES instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOEST/GES provide? How frequently? • How do PEAs assess teachers? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • What will the MOEST/GES do to continue the production of TLMs now that the program has ended? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLMs in the future? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? • How was TLMP monitored by the MOEST/GES? What indicators did you use? How often did Performance Evaluation of TLMP – Ghana 83 you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOEST/GES benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with Chicago State Univ.? What could be improved? 4) TLMP Program Administrators Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? What did the MOEST/GES do (specify unit)? What did the program do? What challenges emerged in your work with the MOEST/GES? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOEST/GES play (specify unit)? What roles did your office play? What guidelines did the MOEST/GES provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? • What other teacher training is provided by the MOEST/GES? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Program Management and Outcomes • What was the TLMP management structure in Ghana? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ghana PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit program implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? Performance Evaluation of TLMP – Ghana 84 • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Headteachers? District/Provincial administrators? PEAs? The MOEST/GES itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of your work? • What accomplishments are you most proud of? If a TLMP-type program were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with Chicago State Univ.? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What is your opinion of the management training that was provided to you by Chicago State Univ.? What key lessons did you learn that you applied to the management of your operations? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? In Field 6) District Directors • What is your current position? How long have you been in this position? • Have you been involved in TLMP and, if so, what was your particular involvement? • How many schools with KG classes, total KG classes, and KG teachers do you have? • What percent of your KG teachers are trained? • When did you receive the TLMP materials? Do you know how many materials arrived by grade level and subject? What percent of your KG classes receive these materials? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? What recommendations do you have to improve the delivery service? • How many of your KG teachers received the initial TOT training on TLMP? • How many of these teachers subsequently taught other teachers? How many? • Has the district given any additional support or training to the KG teachers on using TLMP? • How many of our KG classes initially participated in TLMP? • How many of your KG classes are currently using the TLMP materials on a regular basis. If the number of teachers actively using TLMP has decreased, why was this so? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank TLMP in terms of helping KG children learn English including basic reading skills? What do you recommend to improve the program? In Schools: 7) Head Teachers • What is your current position? How long have you been a principal at this school? • What are the grades in your school? How many boys and girls? Performance Evaluation of TLMP – Ghana 85 • How many KG classes and teachers do you have? What is the enrollment in each KG class? • Are you familiar with TLMP and, if so, is it being used in your school on a regular basis? • How many set of TLMP materials did your school receive? Were these sufficient? • What has happened to the workbooks given that children are supposed to write in them and they have not been replaced? • Have the teachers tried to extend the life of these workbooks and, if so, what have they done? • Are some teachers no longer using TLMP because there are no workbooks? • Is TLMP being used more, less, or the same as compared to three years ago? If there has been a change in the use of TLMP, why is this so? • How many of your KG teachers were trained in TLMP? Were they trained in a TOT or did other teachers provide the training? Are they still at the school, or have some departed or shifted to another grade level? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom Teachers • How long have you been a teacher? How long at this school teaching KG? • How would you classify your ability to interact in and teach in English? (excellent, very good, good, somewhat deficient, poor) • How many students do you have in your classes? Specify class and number of students. • Have you been trained in TLMP? If so, who did the training and how long was the training? How would you rate the teaching? • If you attended the TLMP TOT training, how many other teachers did you teach afterward? What worked well? What difficulties did you encounter in doing this? • If you did not attend any training related to the materials, how did you learn how to use them? • Do you have the TLMP KG Teacher’s Guide? • How many TLMP workbooks do you have? How many children are in your class? • If you don’t have enough workbooks for all the children, what do you do? • Are these workbooks already filled in by children in previous years? If so, how do the children use these workbooks? • Have you stopped teaching TLMP because you don’t have workbooks? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs materials? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes have you observed and recorded in children’s achievement in English since the TLMP workbooks/materials were introduced? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? Performance Evaluation of TLMP – Ghana 86 • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? Performance Evaluation of TLMP – Ghana 87 ANNEX C. SCHOOL OF INTERVIEWS AND MEETINGS DATE TIME ORGANIZATION PERSON TITLE ACCRA 10 June 8:00 TLMP Mrs. Ewura-Abena Ahwoi In-Country Coordinator 10:00 CRDD/GES Mr. Jacob Kor Director, CRDD 12:00 GES Mrs. M. Okae National ECD Coordinator 3:00 PM GES Mr. C. Dowuona￾Hammond Liaison Officer June 11 9:30 USAID M. Perez, A. Jehando, J. Petro Education Office Chief, Education Management Specialist, Education Advisor 12:00 TLMP Mrs. F. Boakye-Yiadom Head, Merger Implementation Team June 12 Aburi College of Education Mrs. Beatrice Boateng￾Sampong, Vice Principal Aburi College of Education Mr. Baffoe Bonmie, Teaching Practice Coordinator New Juaben Municipality Mrs. Docia Abbah Director Mrs. Victoria Obeng Deputy Director for Basic Schools Mr. Peter A Odei Deputy Director for Supervision and Monitoring Densuano M/A Basic School Mr. Bosoompem George Bismark Head teacher St. Mary’s Day Care and Preparatory School, Koforidua Ms. Mirium Ofosu Appiah Proprietress RTI Mr. Paa Nii Bossman M&E Officer?? Performance Evaluation of TLMP – Ghana 88 DRCC/GES Mr. Isaac Asiegbor Director, Assessment Services Unit UNICEF Mr. Madeez Adamu Issah Education Specialist June 13 8:30 Accra Metro Education Directorate Ms. Grace Dankyira ECE Coordinator 9:30 Osu Presbyterian KG, Ga East Lydia Offei Kwapong Head teacher 11:30 Osu Presbyterian Annex KG Gladys Ayiku Head teacher 3:00 PM CRDD (former) Mrs. Sarah Agyeman-Duah Retired former Director 4:00 PM GES Mr. Charles Aheto Tsegah Deputy General Director June 14 9:00 Ga East Municipal Education Directorate Ms. Florence Damahli Director 9:30 Pantang Presbyterian Basic School Mr. Samuel Nkotia￾Dardom Head teacher 11:00 Living Star School (private) Mrs. Mary Ameyaw Proprietress Mr. Joseph Obeng-Yeboah Assistant Head Teacher 2:30 PM Buck Press Mr. Kofi Buckman Managing Director Mr. Tony Boamah Director of General Operations Mr. James Ottumani Marketing Director June 17 9:30 Jasikan College of Education, Volta Mr. Ansi Samuel Vice Principal Performance Evaluation of TLMP – Ghana 89 Mr. Torkonyo Moses Martey ECE Coordinator Mr. Agbenor Akpeteasi Head of Languages Mr. Nyavor Kunche Nyavor Sports and Health Tutor 11:30 Jasikan College of Education Demo School Mr. Feda Robert Kwasi Head teacher 11:00 Jasikan District Education Office Mr. Dompreh J.K. Daniel Deputy Director (HRMD) Ms. Kikiri Victoria ECE Coordinator Mr. Yao Augustine Wuabu Storekeeper 9:30 Metro Cape Coast Education Office Mr. Felix Ansah Director following Supervision 10:00 Philip Quaque Girls’ School, Cape Coast Ms. Olivia Appian Head teacher 11:00 Abakam CRAN school, Cape Coast Teachers 11:30 Golden Treasure International School (private) Ms. Rita Dodoo ECE Coordinator 12:00 OLA College of Education Rev. Elizabeth Amoako Principal 1:00 PM Metro Cape Coast Education Office Ms. Vivian Etroo Director 2:30 PM University of Cape Coast Dr. Kafui Etsey (Mrs) Senior Lecturer Dr. Clement Agezo Head, Department of Basic Education June 18 8:30 Ho District Education Office Mrs. Dora Agorsor Director Mrs. Beauty Agbeve Coordinator ECE Performance Evaluation of TLMP – Ghana 90 10:00 Mawuli E.P. Primary School, Ho Mrs. Ocloo Mawuse Head teacher 12:00 Kpenoe E.P. Primary School, Ho Mr. Appiah Samuel Head teacher 9:00 Effutu Municipality Education Directorate (Winneba) Mrs. Hilda Eghan Director Mrs. Thelma Essuman ECE Coordinator 10:30 Effutu Municipal KG and Primary School Ms. Ruth Duncan-William Head teacher 11:00 ACM KG and Primary School Teacher 12:30 Ebenezer Memorial Education Centre (private) Mrs. Grace Anakyewa Ehintoh Proprietress Mrs. Agnes Appeadu Head teacher 2:30 PM University of Education, Winneba Mr. W. K. Agbeke Senior Lecturer, Department of ECE June 19 9:30 GES Mr. Stephan Adu Deputy Director General, Basic Education 11:00 TLMP Mrs. Ewura-Abena Ahwoi In-Country Coordinator 1:00 PM North Legon Little Campus (private) Ms. Nicholina Bannerman Director, Administration Mr. Douglas Djarbeng Managing Director Mr. Simon Peter Attah￾Cats Director, Academic June 20 10:30 Chicago State University Ms. Jamille Watkins-Barnes Assistant Director, TLMP 4:00 PM USAID Ms. Cheryl Anderson Ms Marisol Perez Ms Adama Jehanfo, J. Petro Mission Director, Education Office Chief, Education Management Specialist, Education Advisor Performance Evaluation of TLMP – Ghana 91 ANNEX D. READING ASSESSMENT FORM AND EVALUATION OF LEARNER READING COMPETENCY Following is the protocol that was to be used for schools where assessments of students were to be made. 1. Class selection - K2 only 1.1. Interview all teachers together 1.2. For class observation, ensure teacher has been trained and materials available 1.3. Then select section A, or A and B. Try to get 1 male and 1 female teacher 2. Child selection 2.1. 2 boys and 2 girls 2.2. Select at random. Do NOT let teachers select the students. Perhaps count children to 15 and then 30 for adjacent boy and girl 3. Test administration 3.1. By our team only. (Probably without the teacher) 3.2. Set and adhere to time limit 3.3. Fill out data sheet for each child 3.4. Describe to student what is to be done 3.4.1. Learn about English skills 3.4.2. Not interested in any individual child, but want to find out in general abilities in English 4. Conduct a modified EGRA test, assessing: 4.1. Letter recognition 4.2. Word recognition 4.3. Ability to read a simple sentence In practice, it proved necessary to adjust this protocol. After the teachers were interviewed, the consultant explained the learner literacy skills test to be administered to KG1 and KG2 students. The learner reading competency test was revised to accommodate conditions found in the three schools visited in the Volta Region, as described in the text. After an interview with the teachers and an explanation of the literacy skills test, the researcher selected four KG1 and four KG2 students to be tested. In each grade-level group there were four boys and four girls. Because someone had to explain to the children in a Ghanaian language what the testing procedure was going to be, and to explain steps as the test progressed (especially in KG1), a teacher had to be present during the testing. The testing was done in the corner of the classroom so that the wall charts would be visible. The skills tested were the following: 1. Listening and responding The researcher asked each child (KG1 and KG2) to answer these questions: “What is your name?”; “How old are you?”; “What is this?” (pointing to the table where the testing took place). 2. Letter identification Each child was asked to identify these letters: M, b, r, P, A, g, F, T, Z s. 3. Oral fluency in describing a A) A picture was chosen from the workbook and the child was asked to describe it (ex. The boy is sitting on a chair). Performance Evaluation of TLMP – Ghana 93 picture B) Pictures were pointed to on the wall charts and the student was asked to name them. 4. Word recognition Pictures and the associated words were chosen by the researcher for the student to read from the workbook. Some were chosen in the sequence in which they appeared on the page, others at random. 5. Writing The researcher asked the student (with the help of a teacher who explained the task) to do the following: 1. Write three letters (either capital or lower case, as the child preferred). 2. Write their first name. 3. Write a word of personal choice in English. Performance Evaluation of TLMP – Ghana 94 ANNEX E. PRINTING AND DISTRIBUTION – A CASE STUDY Per IBTCI’s January 2013 consultations with the Africa Bureau’s Education Team, review of the physical production of TLMP materials was to be only of secondary concern for the evaluation and, therefore, IBTCI has developed and presented only the most relevant information on this aspect of TLMP for the other CAs and MSIs. However, since each of the MSIs was required to produce a minimum of 600,000 copies of materials and procurement of this amount of work presented challenges novel both to the MSIs and often to the printers and since CSU was, to the best of our knowledge, the only MSI actually to procure significant equipment, we feel that the example of production of TLMP materials in Ghana would provide a useful case study of potential value both to Missions and to implementers who may find themselves with similar needs. According to Buck Press Ltd, they responded to invitation to Bid (Bid No. 08211S), published Friday March 28 and Saturday March 29, 2008 in the Daily Graphic and Ghanaian Times. It competitively won the Tender as per Notification of Award of Contract dated April 24, 2008 and signed by the CSU Director of Purchases. Buck Press was identified as the primary printing contractor as they were willing to produce, store and transport the books nationally for distribution to the schools (storage was BP’s in-kind contribution). Transporting them was to have been covered by GES, but due to lack of funds/misunderstandings, school districts picked up the TLMs themselves. According to Mr. Buckman, the following issues emerged during the contract period: • Variations (increases) in pages of some of the titles. • Delays in submitting artwork due to corrections in the scripts. As such, Buck Press, Ltd had to work under pressure to deliver within limited time frame. • Storage of finished titles for very long time. • Breaking the materials by Districts involved extra packaging. • Buck Press advanced the cost for distributing TLMs to the Districts before being reimbursed. The evaluation team in its visit to Buck Press noted that it has grown to become a full-service printing (not publishing) company that provides good quality printing services. The TLMP contract made a significant difference to Buck Press, which was able to purchase new equipment and upgrade its storage capacities. The project did not provide technical or managerial capacity building services for Buck Press. During the visit, the team observed the production of a mathematics textbook for 7th grade. The glossy paper, the print, the binding and the illustrations were of high quality. Buck Press is currently producing eight other textbooks for a different client. In addition, the company has also undertaken large employment schemes to facilitate the provision of secure identification card services in Ghana, Liberia and beyond. It also ensures that its waste paper is recycled. For TLMP, it was able to print 1,458,006 items in three weeks. The value of the contract was $930,525.50 USD. The project paid the Buck Press invoice two months after delivering TLMs on environmental studies, numeracy and literacy to the regional education offices’ store houses. The number of TLMs dispatched to each regional education office was determined by the number of students and teachers listed in the MOE’s statistical yearbook for KG1 and KG2. In 2010 and 2011 Buck Press printed 3,881,353 teachers’ guides and student workbooks at a cost of $4,915,000, including distribution costs. Mr. Buckman explained that printing in black and white would have been about 20 per cent cheaper than full color printing (the option chosen by the project). Buck Press explained that the Chicago State/USAID contract had been challenging. The finished products were to have been delivered to the MOE warehouse in Tema and the Ministry was to have ensured forwarding to its district offices. However, when the production of the first run of TLMs was completed in September 2008, the Tema warehouse was full. Buck Press then delivered the books to the Central Region and Accra at no extra charge. This was a gesture of “corporate social responsibility” valued at $150,000. Buck Press distributed the TLMs to District Stores (depots) serving 170 districts in the Performance Evaluation of TLMP – Ghana 95 quantities determined by the Ministry and the CSU project, and the district store managers signed bills of lading. Buck Press confirmed that it had not delivered any TLMs to teacher training colleges or universities with teacher training departments. It should be noted that early in TLMP CSU made the mistake of buying its own print equipment. There are two offset printers in the TLMP office, neither of which have worked for some time. However, with new additional funding, TLMP plans to repair the printers and to use them to print the newly revised NALAP teacher’s guides for national distribution. The background of the printers is not totally clear. It appears that they were purchased with the expectation, initially, that GES would produce all the TLMs. However, the equipment was never used by GES as the contract was then developed with Buck Press. And the former Director of CRDD says that the printers “did not satisfy the specifications of CRDD.” The printing equipment was initially placed in CRDD, but then moved to TLMP with the idea that it could be used to print materials for piloting of materials. A comment from one senior officer was that there has been substantial maintenance costs and that it might have been more cost effective to have the printing done commercially. For the more limited runs, perhaps it would have been more cost-effective to have purchased a copy-printer or just a high capacity photocopying machine. Performance Evaluation of TLMP – Ghana 96 ANNEX F. REVIEW OF TLMP PUPIL’S WORKBOOKS AND TEACHER’S GUIDES SUMMARY OF MATERIALS EVALUATION CRITERIA A. Pedagogical Quality 1. Conformity to the syllabus 2. Organization and presentation of content 3. Correctness of information 4. Conformity to socio-cultural values in Ghana 5. Integration of skills and cross cutting skills 6. Adaptability and applicability to local and global settings 7. Relevance to the learner’s age and interests 8. Use of quality illustration 9. Appropriateness for learner and teacher 10. Content appropriate for learners of diverse abilities 11. Presence of enjoyment index 12. Assessment of student learning 13. Language and communication aspects B. Physical and Design Quality 1. Type and quality of paper 2. Type and quality of binding 3. Size and dimension 4. Quality of print 5. Type face and size of sentence length 6. Structure Each of the characteristics above is further divided into success stories-categories. PUPIL’S WORKBOOK – KG1 LITERACY A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing Each lesson includes a selection of the following: talk about the picture, sorting, coloring/drawing/ matching, tracing letters, visual discrimination, same/different, pattern making, what is missing,. Emphasis is on vocabulary building, although there are no strategies to use the vocabulary except once during Performance Evaluation of TLMP – Ghana 97 each lesson when a child must describe a picture. In general, the children are depicted as looking sad – largely the result of drawing the mouths of people illustrated. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors Good 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country The back of a nude male is presented on p. 34 as he is washing. Boys do not jump rope as depicted on pp. 32 and 43. 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues The 24 units (plus one for revision) cover the “normal” topics – myself, my family, body, food, colors, home, shops, school, friends, animals, clothes, safety, and health. The topics could lend themselves to further exploration, but at the KG level, this could be inappropriate. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality Content is local and is designed to have children acquire the vocabulary of different locales and people’s activities. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests All relevant. However, almost all lessons require coloring or drawing and a student must have access to crayons or colored pencils to complete each lesson. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations All appropriate 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Pictures and activities can be expanded in any way the teacher decides. 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners Learners can proceed in accordance to the teacher’s or their own pace. Performance Evaluation of TLMP – Ghana 98 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests There is a considerable variety of activities, although no songs/games are considered. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed None is present in the workbook, although there are three revision units in which the same type of activities as the other units are included/repeated. 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Emphasis is on vocabulary and learning how to sight read (rather than sound out letters/phonics) with names of items appearing under a picture of the item. Reading is only at the picture/word level and does not address any skill building in sound/ symbol correlation. Talking about each picture at the outset of each lesson requires a certain amount of vocabulary; it is not clear if this is in English or home language. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Size of book is A4 with a sturdy cover, although it may not be water resistant. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum Binding is just two staples and does not appear sturdy. 3 Size and dimension • Size and dimension in relation to learners’ age A4 – may be too big for KG. 4 Quality of print • Legibility of ink (text and illustrations) Very good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Size letters and illustrations are good. No sentences are used except to give directions (which would probably be given orally by the teacher). 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Unit balance; all almost the same. No end matter. Workbook is 91 pages and it is not clear how the lessons progress; if the workbook is to be used over a year, it Performance Evaluation of TLMP – Ghana 99 may not withstand rough treatment. TEACHER’S GUIDE – KG1 LITERACY A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Emphasis on listening and speaking. Outcome objectives are clearly set forth at the outset of each lesson. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book TG includes the materials presented in the children’s workbook, identifying and explaining different activities to be undertaken to teach the lesson. It is not clear what level of English teachers have achieved; understanding the TG requires a significant amount of English as well as an understanding of early childhood education, learner￾centered methods, and how children learn using different strategies, 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) The TG takes the teacher through a series of age-specific activities for children to introduce them to English and give them language experience. Discussion of the pictures requires children to use English language constructions that have not really been introduced – the teacher says something, and the children parrot: e.g., I am walking (and illustrates while walking). Use of the total physical response method is good, but more practice is needed in using this construction and others. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Many different types of activity are provided, along with how to implement them to achieve learning objectives. A final part in each unit includes suggestions for parents that teachers can make in parent conferences. Writing is introduced through making scribbles and coloring so that children can become comfortable in holding writing implements. Teaching songs is a good way to help children learn vocabulary and structure. All of the songs and poems are in the Appendix. Most are from Performance Evaluation of TLMP – Ghana 100 the US. However, it takes time to teach and to learn these and it is not clear how much time each lesson should take. No time allocations are made to teach each unit and each element in each unit. A significant number of materials to manipulate are required to teach the units and it is not clear where a teach would get these implements, e.g., bottles, rattles, drums, etc. Assessment is largely “be mindful of benchmarks” without actually noting who has mastered what. Teachers are to gather books recommended in the appendix for “read aloud” activities, with two excerpts from the “Ananse” stories given; it is not clear how to systematically use these materials to enhance language development. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Lots of ideas are present, but no time frame is given in teaching them. 6 Referencing • Extent to which the guide helps identify other materials and resources Many in the appendix, but no idea how teachers can access these. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book The student workbook is a practice book in which students practice a limited set of skills, with the teacher providing a great deal more in the classroom. There are no questions in the students’ workbooks. B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Same cover and paper as children’s workbooks; not sure if cover is water resistant. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum Binding is more substantial gum, but in humid climates may not be durable. 3 Size and dimension A4 Performance Evaluation of TLMP – Ghana 101 • Size and dimension of the book 4 Quality of print • Legibility Very good 5 Typeface and size • Typeface and size are easy to read God use of headings and separation; dense, but separations make it easy to read. 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) The TG is 243 pages long, includes the student workbook (divided so as to show where expanded lessons should be taught), but does not give instructions on letter recognition even where children are asked to “read” them based on their associated pictures. PUPIL’S WORKBOOK – KG1 NUMERACY A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing The workbook begins with classification/grouping rather than with actual numbers (which begins only on p. 19 – Unit 2). Groupings are by color, size, shape, matching and ordering, all of which may not be appropriate if the numeracy skills are to be supported by literacy skills. The illustrations of people are much better in the numeracy book than the literacy book; the people are smiling. Beginning with Unit 2, children practice tracing and writing the numbers after they count items in pictures. Pictures are of items common in the environment. Unit 3 introduces shapes and solids, although some of this was in Unit 1. 3 Correctness of information All good Performance Evaluation of TLMP – Ghana 102 • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Yes, pictures are of things common in Ghanaian culture; the other items n/a 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues N/A 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality N/A 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Learners are asked to count items 1-10, to trace and write numbers, and to distinguish between shapes. Illustrations all support these activities. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations Good. Milo can was at first indistinguishable. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Good progression 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners Workbook pages can be done as assigned or as the children pace themselves. 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests Nothing presented in the workbook. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition Nothing presented in the workbook. Performance Evaluation of TLMP – Ghana 103 • Variety of assessment techniques employed 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Starting with classification was inappropriate; should have started outright with numbers. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Cover paper is of a card stock that is not water resistant. Weight of paper of book similar to all books produced and may have a limited life. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum For 38 pages of A4 paper, only two staples bind the pages, so book may have limited life. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be cumbersome for KG 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Only sentences are instructions, which are probably given orally by teacher. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Number of pages of each unit uneven. It is not clear what the classification section is all about as it is not linked to the other two units on counting and shapes and patterns. There is no end matter. TEACHER’S GUIDE - KG1 NUMERACY A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives (realistic & measurable) Outcome objectives are clearly stated at the beginning of each unit. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book Book includes children’s workbook split up into activities that are augmented by the teacher. Each unit lists materials required, but some may not be available to teachers. 3 Level of cognition The first unit on grouping and Performance Evaluation of TLMP – Ghana 104 • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) sorting is misplaced and children may not have the appropriate vocabulary for the items identified. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Practical activities are varied, but could be expanded on, i.e., number songs; does not have the same variety of activities as literacy book. Each lesson has suggested activities for parents. Unit 3 on shapes and patterns is introductory to geometry and could be addressed later, paying attention instead to numbers beyond 10, 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Not very creative; follows workbook fairly closely without many added activities. 6 Referencing • Extent to which the guide helps identify other materials and resources None 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book Not many in this subject at this level. No questions. B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Card stock used for cover; not water resistant. Paper same as other books and TGs 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum For a 68-page book on A4 paper, just two staples hold the materials together, suggesting limited durability. 3 Size and dimension • Size and dimension of the book A4 – OK for teachers 4 Quality of print • Legibility Good 5 Typeface and size • Typeface and size are easy to read Good 6 Structure Limited explanation of how to teach numeracy to 4-year olds (the Performance Evaluation of TLMP – Ghana 105 • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) target audience). Children who start KG can be much older. No end matter. PUPIL’S WORKBOOK – KG1 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing The structure of this book does not conform to that of the others. Materials are presented in sections and then subdivided into units, whereas the literacy and numeracy books are divided in units and then subdivided into lessons. There are three sections: building a healthy individual, the physical environment and the social environment. The first unit introduces the self, the body and care of the body. Body parts are not introduced in the literacy book until unit 5 (page 13), but begins the section in ES. There will most likely be a vocabulary shortfall in this book. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Girl dressed in traditional clothes named Fatima, a Muslim name, which may require different dress. 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues Pictures and exercise promote eating fruits and vegetables and how to cover your mouth when sneezing. Also covered is how everyone in the family helps to keep the house and the surroundings clean. No other Performance Evaluation of TLMP – Ghana 106 items covered. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality N/A for the most part, but point above can be applied anywhere. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Vocabulary may be a bit above children’s level – e.g., “quench” on p. 12, “bamboo” on p. 27, “dolphin” on p. 32, etc., etc. When naming animals, it cannot be assumed that just because a child lives in Africa that they will be familiar with the animals of the savannah and know their habitats. The summary of all sections call for a lot of writing which the children have not learned to do in literacy. 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations Illustration on p. 10 shows a “healthy” baby and another that looks emaciated to illustrate why fruits and vegetables are eaten. This is inappropriate. Illustration on p. 14 asks child to circle best way of washing hands; it is not clear what is being taught. The illustration on p. 16 of children brushing teeth is not clear as the “stick” that the boy is using is no sufficiently detailed to differentiate it from a toothbrush. The illustration of the back side of a nude boy taking a shower is inappropriate. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Topics covered are not interrelated and do not cover items sequentially; rather, sections have their own vocabulary. 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners As a stand alone, the book could be used according to the teacher’s or the student’s pace. However, children will have difficulties in vocabulary and in writing as these have not yet been introduced in literacy. 11 Presence of enjoyment index None Performance Evaluation of TLMP – Ghana 107 • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, practice and review activities • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed Summaries appear at the end of each section, but they require writing which the children may not yet be able to do. No assessments in workbook. 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Language level may be a problem as there is so much new vocabulary. No other materials are present. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight It appears there is a water￾resistant cover; paper quality is the same as all other books. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum There are 52 A4 pages held together with two staples meaning they are not durable. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be too big for children 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Good – but sentences introduced in summaries and this is the first time they appear, except for instructions. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Uneven; nothing at end. Performance Evaluation of TLMP – Ghana 108 TEACHER’S GUIDE – KG1 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Outcome objectives are set at the beginning of each lesson – check to see if conforms to syllabus. 2 Organization and presentation • Sequencing of topics in relation to pupils’ book Two manners of presentation include discovery teaching (child-centered) and direct teaching (review, present, guide, provide feedback, and encourage independent practice). 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) Exercises appear to be age appropriate, but sight words require additional understanding and teaching. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Key sight words are identified for each lesson, but phonics is not part of the lesson. Materials needed are specified, but these may not be available to the teacher (e.g., large sheets of brown paper, fruits). Writing is required for summary exercises, and children have not yet been taught to do this. Songs are provided in the appendix, but no time is set as to how long it should take for each part of the lesson or to teach the song. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction A broad range of hands-on activities are presented for each lesson. 6 Referencing • Extent to which the guide helps identify other materials and resources The TG identifies five titles for further use and ends with notes the teacher can make on activities that work well and/or need adjustment. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book The TG has much more in it than the children’s workbook. The workbook give the child the chance to do certain exercises and become used to working with a book, but it is not a text per se. The TG provides many extension activities to learn a lesson using different approaches. Performance Evaluation of TLMP – Ghana 109 B. Physical Design and Quality No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Cover paper appears to be water resistant; pictures on contents are on the cover; paper used is standard for all books. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum The 91-page TG is held together with only two staples, thus reducing the durability of the book. 3 Size and dimension • Size and dimension of the book A4 paper seems appropriate for teachers 4 Quality of print • Legibility Good 5 Typeface and size • Typeface and size are easy to read Good 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) The same framework is used for each lesson; introductory materials spell out the approach to be used; end matter includes an appendix with all songs, poems and games to be used. Some are parodies from US-based songs and games. PUPIL’S WORKBOOK – KG2 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Conformity to the syllabus • Extent to which the text covers the entire syllabus • Depth and breadth of concepts • Arrangement of the material in respect to the pedagogical needs of the learner • Conformity to the general and specific goals of the subject area The same three sections that were in KG1 are in KG2 – individual, physical and social environment. It is not clear if these are what the Ghana syllabus calls for. 2 Organization and presentation of content • Sequencing of content in relation to sound pedagogical approach (simple to complex, known to unknown) • Use of a variety of presentation techniques • Use of illustrations, italics, bolds, subheads, paragraphing Healthy Individual - The first unit starts out with my physical needs – food, clothing, shelter and water with children asked to differentiate different uses of the basic needs. Pictures are appropriate. Writing through tracing letters is introduced. A rhyme in English is given, although it is not clear if instructions in the math book should be followed, i.e., teach in any language as it is the related concepts that are important. The summary requires a significant level of reading Performance Evaluation of TLMP – Ghana 110 ability in English. Unit two talks about spiritual needs and attributes of God. While Ghanaian culture promotes these, it is inappropriate for a religious unit to be included in materials paid for by the USG. Unit 3 focuses on taking care of boy waste with a picture of a boy defecating into a pit latrine and urinating into a toilet. The instructions are to cross out the “bad” things as it includes a boy putting a handkerchief around his mouth/nose while sneezing, and a girl sneezing into the air. The questions to be answered require writing and it is not clear if children have this skill. Physical Environment – Unit 1 addresses sources of light, but does not depict a fire and the dangers of having one – which is more common than an LED light bulb. Writing and reading are included although children may not have these skills. Unit 2, weather and seasons, has very good illustrations and has the children drawing different weather patterns. Unit 3 on keeping the environment clean does not depict implements used in Ghana. Unit 4 on safety finally depicts fire, and an electric iron, which is not necessarily used in the rural areas so children may not know what this is. Social Environment – Under unit 1 my school, three boys are depicted jumping rope; this is highly unlikely in Ghana as this is seen as a girls’ activity. While the flag is in the picture of the school, the instruction is to color it (not stating which color goes where) and nothing more is done to the flag. This is an opportunity to bring in civics, but the activities in the workbook do not direct the children in this way. Because this lesson has a lot of “letter tracing,” it is clear that children still do not know how to write, yet the summary lessons at the end of each unit require children to write. Unit 2 focuses on my community; the depiction of a health clinic would have opened up many areas for teaching, but only police station, PO, chief’s palace, a mosque and a church are depicted. The market is depicted in the next lesson, and only women are shown as vendors. Women selling foodstuffs is very Performance Evaluation of TLMP – Ghana 111 common, but men also sell. This is an excellent time to raise the question why women sell certain things and men others. It is only at the end of this unit that a hospital/clinic in the community as an important place, but there is no depiction. Under the unit festivals and other celebrations, only men are depicted in killing animals, drumming and dancing, and at a durbar. It may be that the queen mother is depicted behind the chief, but the picture is not clear are depicted also, and this is not a regular custom. Independence day is included in the next unit, with the children learning the national anthem and the national pledge, but no discussion or activities as to why this is celebrated. 3 Correctness of information • Correctness of facts and concepts • Accuracy of figures • Free from spelling and grammatical errors Well done 4 Conformity to socio-cultural values of Ghana • Content abides with the moral values of Ghana • Content shows tolerance to diverse beliefs • Content reflects love and respect for the country Eating nutritious food, drinking clean water, etc., are all in accordance with moral values. Using the proper receptacle to pass personal waste is also important as many people do not use sanitary facilities. By reinforcing appropriate behaviors, children will begin to act appropriately. However, the downside of poor behavior also needs to be addressed. Wearing the right clothes in different types of weather is presented, but the types of clothing depicted may not be a part of a household priority (rain coat, rain hat). The illustrations of why we go to school (pp. 68-9) are sexist – males are depicted as doctors, policemen, footballers, and soldiers, while women are depicted as teachers and nurses – a stereotypical set of gender-based professions. Birthdays are depicted as they are in the west (cake with candles, pp. 82-3) and do not take local customs into account. Birthday cards are also included and this is not a part of custom for children, especially in families that are resource poor. Sexism continues with the depiction of community leaders, all of which are men (except the mother in the picture of the parents) (pp. 89-90). This lesson clearly demonstrates that the writers knew little about the leadership roles women play, or Performance Evaluation of TLMP – Ghana 112 wanted to ignore them. 5 Integration of skills and cross cutting skills • Content helps learners build positive life skills • Content informs learners on HIV/AIDS, gender, OVC, children’s rights, safety, environmental issues There are a lot of misrepresentations in this section. However, the issue of safety is fairly well covered on an introductory level. 6 Adaptability and applicability to local and global settings • Content informs learners on global technological, economic and political issues • Content informs learners of current issues • Extent to which knowledge in the text can be applied by learners in their locality Only the final point is well covered. 7 Relevance to the learner’s age and interests • Activities in the text relate to the learner’s age, needs, and interests • Illustrations and pictures relate to the level of the learner’s development and interests Good 8 Use of quality illustrations • Meaningfulness and relevance • Clarity • Location and balance in relation to text • Effective use of color • Variety of illustrations • Presence of captions • Correctness of illustrations In general, these are good, but some are not as clear as others. 9 Appropriateness for learner and teacher • Extent to which learner/teacher are likely to enjoy the material • Extent to which content guides the learner to the intended knowledge Good coverage of issues in each section – movement from the individual to the family and to the community is good, although some other topics might have been included about social relationships and getting along (i.e., no bullying). 10 Content appropriate for learners of diverse abilities • Content considers both slow and fast learners OK 11 Presence of enjoyment index • Content integrates elements of enjoyment (humor, jokes, cartoons songs, puzzles) • Use of different techniques to stimulate learners’ interests Yes, songs and games are integrated. 12 Assessment of student understanding • Use of feedback, summaries, diagnostics, N/A Performance Evaluation of TLMP – Ghana 113 practice and review activities • Adequacy of assessment activities to test different levels of cognition • Variety of assessment techniques employed 13 Language and communication aspects • Extent to which the level of difficulty of language reflects the learners’ level • Extent to which concepts are exact • Correctness of spelling and grammar • Extent to which the material stimulates learners’ language usage (catchwords, proverbs, slogans) Not clear when writing is undertaken full scale; seems to be a mix-up between letter tracing and the need to actually write things. B. Physical and Design Quality No. Criteria Remarks 1 Type and quality of paper • Water resistant cover • Opacity of paper & weight Cover seems to be water resistant; paper that used for all texts. 2 Type and quality of binding • Type of binding in relation to pages • Strength of binding wire, thread or gum 102-page workbook is bound with glue, which may not be very durable in humid climates. 3 Size and dimension • Size and dimension in relation to learners’ age A4 may be too big for students. To reduce wear and tear, separate books might be developed for each section. 4 Quality of print • Legibility of ink (text and illustrations) Good 5 Type face and size and sentence length • Typeface and size in relation to learners’ level • Lengths of sentences in relation to learners’ level Not clear what learner’s level should be in KG2, although sentences appear to be short and direct. It is also not clear how and when sentence structure is introduced in written form. 6 Structure • Balance of chapters • Endmatter (glossaries, indices, appendices) Sections seem to be equally balanced. No end material in Children’s workbook. TEACHER’S GUIDE – KG2 ENVIRONMENTAL STUDIES A. Pedagogical Quality No. Criteria Remarks 1 Objectives • Objectives stated in the guide match the syllabus • Clarity of objectives ( realistic & measurable) Outcome objectives are established on the basis of activities, but not overall behavior. Not clear if what is covered is actually guided by the syllabus. Performance Evaluation of TLMP – Ghana 114 2 Organization and presentation • Sequencing of topics in relation to pupils’ book The move from individual, to the physical and then to the social environment seems OK, although it is not clear how these are integrated. 3 Level of cognition • Activities measure the level of understanding (knowledge, comprehension, application, analysis, synthesis, and evaluation) It appears that children are provided a sufficient number of activities to learn in different ways so that all of these are addressed in one activity or another. However, there is no real means of measuring these addressed in the lesson plan. 4 Teaching and learning strategies • Teaching and learning strategies enhance the teaching-learning process • Strategies help the teacher attain the stated objectives • Strategies give extra information compared to the pupils’ book Word walls are introduced for all sight words presented at the outset of each lesson and then reinforced/used throughout the lesson. Teachers are directed to producing charts, drawings or pictures of different scenes, etc., to expand the application of the lesson; teachers may not have the resources to produce these items. Children are directed in each lesson to write something, although it is not clear when they learned to write. A number of field trips in the community are suggested; the logistics of guiding many children (how many are in each KG class?) requires significant planning so as to have enough staff to oversee the children. 5 Teaching ideas • Guide provides new ideas for enhancing classroom instruction Many given and teacher can choose what to do or can create her/his own 6 Referencing • Extent to which the guide helps identify other materials and resources Other materials are noted in the appendices. 7 Extra questions/examples/Answers • Extent to which the guide gives extra questions and examples apart from those in the pupils’ book • Extent to which the guide provides correct answers to questions in pupils’ book Some are given, but value questions are somewhat sketchy as sexism is apparent in the illustrations. The only questions answered are the summary questions at the end of each section. B. Physical Design and Quality Performance Evaluation of TLMP – Ghana 115 No. Criteria Remarks 1 Type and quality of cover/paper • Attractiveness of cover • Water resistant cover • Opacity of paper • Weight of paper Appears to be water-resistant; paper same as all books. 2 Type and quality of binding • Type of binding in relation to number of pages • Strength of wire, thread or gum 139-page TG is bound with glue, which may not be durable in humid climates. 3 Size and dimension • Size and dimension of the book A4 seems appropriate for teachers 4 Quality of print • Legibility God 5 Typeface and size • Typeface and size are easy to read A shift has occurred with this TG in the size of the typeface in the introduction: it is much smaller and does not look like the children’s workbook, although all the topics are there. 6 Structure • Forematter (rationale, theoretical framework) • End matter (appendices, glossaries, indices) There is very little introduction that is relevant specifically to the book; it is the general introduction used in other books. There are four appendices: rhymes and action songs; stories; key words; parental support; and references. The book is structured in the same way as the others, with objectives, suggested teaching and learning procedures, key sight words, teaching and learning resources, several activities, and summaries. The systematic approach across all books is helpful in guiding teachers through the teaching and learning process. Performance Evaluation of TLMP – Ghana 116 ANNEX G. THE ORIGINAL 14 TLMP DISTRICTS CRITERIA FOR SELECTING THE DISTRICTS 1. Lack of Access Road 2. Lack of Accommodation 3. Lack of Transport 4. Lack of Electricity 5. Lack of Communication 6. Lack of Water 7. Lack of Health Facility 8. Lack of School Infrastructure 9. Lack of Teachers Districts REGION DISTRICT DISTRICT CAPITAL Ashanti 1. Amansie West Assin Manso Brong Ahafo 2. Pru Yeji Central 3. Assin North Assin Fosu Eastern 4. Birim North New Abirem Greater Accra 5. Accra Metro Accra 6. Ga East Abokobi Northern Region 7. Sawla-Tuna-Kalba Sawla 8. Saboba 9. Chereponi Saboba Chereponi Upper East 10. Garu Tempane Garu Upper West 11. Nadowli Nadowli Volta 12. Krachi West Kete-Krachi 13. Ho Ho Western 14. Bia Bia ANNEX H. WORKBOOKS AND INSTRUCTIONAL MATERIALS PREPARED BY TLMP 2009-2012 YEAR TITLES Environ. Env. Num. Num. Literacy Lit. Environ . Enviro n Num. Num. Literacy Litera cy TOTAL S Wk Bk 1 TG 1 Wk Bk 1 TG 1 Wk Bk 1 TG 1 Wk Bk 2 TG 2 Wk Bk 2 TG 2 Wk Bk 2 TG 2 2009 270,000 8,501 270,000 8,501 270,000 8,501 202,000 5,501 202,000 5,501 202,000 5,501 1,458,00 6 2010/1 1 1st print 471,489 12,918 471,489 12,918 471,489 12,918 359,443 11,712 359,443 11,71 2 362,150 11,712 2,569,39 3 2nd print 289,511 7,732 289,511 7,729 289,511 7,732 222,890 7,188 222,890 7,188 220,890 7,188 1,579,96 0 TOTA LS 1,031,00 0 29,151 1,031,00 0 29,14 8 1,031,00 0 29,151 784,333 24,401 784,333 24,40 1 785,040 24,401 5,607,35 9 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Nancy E. Horn (Team Leader), Tom Tilson, Denis Khasu, and Hannock Mateche under Task Order AFR￾12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania EVALUATION REPORT - MALAWI Intentionally left blank TLMP Evaluation Report Volume II – Malawi 118 EVALUATION REPORT - MALAWI EXECUTIVE SUMMARY The University of Texas San Antonio (UTSA) was awarded a Cooperative Agreement (CA) to implement a textbooks and learning materials program (TLMP) in Malawi in 2009 to generate supplemental reading materials for Standards (grades) 1-3. The evaluation of this program took place in February/March 2013, and had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based Minority Serving Institutions [MSIs]), in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. The program UTSA implemented included: • Writing workshops with teachers and others, as they were the original authors of the stories; • Follow-up editing that focused not only on the mechanics and content, but also on how the materials conformed to the curricula developed under the NPC; • Development of teachers’ guides for all volumes; • Illustrator workshops led by the South African illustrator who had done the graphic illustrations; • Capacity building workshops led by the Institute of Economic Development at UTSA (focus was on small business development) for Kriss Offset (the printer) to improve their management practices; • Community mobilization by the Creative Center for Community Mobilization (CRECCOM) to build awareness to the value of reading; • Workshops with lead teachers and primary education advisers (PEAs)on how to use the materials (cascaded to others in their respective clusters, then schools); • Continuous collaboration with the Malawi Institute of Education (MIE), English Curriculum Division (the lead unit working with UTSA) in research, outreach, workshop participation and leadership, etc.; and • Establish e-learning labs at the Lilongwe and Blantyre Teacher Training Colleges (TTC) (through a partnership with INTEL). The educational context in which the Read Malawi (RM) TLMP was implemented included a National Education Sector Plan 2008-2017 that sought to 1) equip students with basic knowledge and skills to enable them to function as competent and productive citizens in a free society; 2) reduce absenteeism, increase enrollment and reduce the dropout rate; 3) improve learning outcomes; and 4) retain girls to complete the primary cycle. UTSA envisioned that the generation of supplementary reading materials would advance this Plan and create a reading culture among children in the early grades. Competing with Read Malawi, however, was a number of other Ministry of Education, Science and Technology (MOEST) or donor-sponsored reading programs and materials that took a more phonics-based approach. RM materials took a whole word/whole language approach and assumed that children could already read. TLMP Evaluation Report Volume II – Malawi 119 This evaluation identified many stakeholders from whom to collect data. Protocols were developed for each group, and teacher observation forms as well as reading assessment instruments were implemented with teachers and students, respectively. Several limitations, however, prevented a comprehensive evaluation: 1) time allocated for research; 2) floods on roads that necessitated the choice of different schools; 3) a civil service strike that closed all the schools; and 4) the institutional failure of the Malawi Institute of Education (MIE) (the local implementing partner) to provide a full set of the TLMs for team review. Despite these challenges, the team was able to collect significant data on the different components of the program. 1. Impact of TLMP on UTSA – Several institutional changes were made as a result of project involvement: from local to international procurement; making sub-awards; streamlining travel policies to ensure a quick turn-around; creating a global reputation among faculty leading to increased enrollment in education abroad programs; an increased credit rating; and improving the university’s standing as a research university. 2. TLMP Output – A total of 5,260,548 materials were printed and distributed. On average, each of the 1,272 receiving institutions was given 3,700 books. 3. Management and Partnerships – UTSA developed several partnerships to implement RM, all of which contributed to the success of documents for the project. At the head of the list was the partnership with the MIE, the main implementing partner. A para-statal that is the curriculum development arm of the MOEST, the MIE – under the leadership of the English curriculum unit – spearheaded the process with UTSA in providing the writing workshops, and overseeing the development process. In developing the materials, RM had a major problem in leveling them so that they would be accessible to children in Standards (grades) S1-S3. Writers took the materials to demonstration schools and observed children using them; however, no attempt was made to involve teachers in using them and then providing feedback. The training provided to teachers was insufficient, and the methodology – cascading – led to different participants being trained for different durations with teachers themselves receiving a “watered down” version of the training. A partnership with a community mobilizing organization sought to increase appreciation for reading and conducted a number of innovative activities toward that end. Another partnership was created with UltiNets to install approximately 100 computers at the Lilongwe and Blantyre Teacher Training Colleges (TTC) for teachers to write the stories and for children in the adjacent demonstration school to learn about computers. Beyond this input, TTCs were not properly capacitated by UTSA to train teachers on the use of the TLMs either for pre-service or for in-service continuous professional development. 4. Project Implementation – Malawian teachers, curriculum specialists, artists, desktop designers, editors and printers were all identified by UTSA, the University of Texas, Austin (UTA), and MIE to develop and produce the TLMs. Through a series of writing workshops at two TTCs, authors of books (in-service and pre-service teachers, MOEST officials, TTC Lecturers and MIE curriculum specialists who were fluent in English) learned to write high quality complimentary reading materials that were situated in local contexts and took into account certain cross-cutting themes: gender; HIV/AIDS; and local TLMP Evaluation Report Volume II – Malawi 120 concerns. These materials were authored in both Chichewa and English. Authors used a combination of their lived experiences and topics of interest to learners as a basis for the books. Teacher’s Guides were developed in a similar manner. Printing of the materials was at first undertaken by a South African printer, but when shipments were shorted, pages printed upside down, etc., the organization hired to repackage the TLMs was hired to do the printing. They not only accomplished this task, but was able to donate approximately $50,000 worth of TLMs to the project. Already a well-resourced printer, Kriss Offset benefited from the TLMP order, but did not expand its business in any appreciable way. Although the materials were welcomed by stakeholders and helped some children to learn how to read, many challenges emerged in how to use them: they were too difficult for both the teachers and the students; there was too much material and too many lessons to cover in the time allotted to teaching the books; a special time was set aside on the timetable for the use of the books (Tuesdays and Thursdays) that had deleterious effects on the ability of teachers to address other subjects and other language programs; teachers mistakenly used the books as texts and so having just 20 copies of a title for use by 120 students was clearly insufficient; 20 copies of all 180+ titles were provided to selected schools, leaving approximately 4,500 schools without any TLMs and; the training provided to teachers was inadequate to change their teaching habits and learn how to incorporate supplementary materials into the curriculum. 5. Outputs and Outcomes –Teacher observations demonstrated – in schools that were not demonstration schools or had not received special training – that teachers were not able to cope with all the requirements presented in the teacher’s guides. Students could not cope either with the long class period during which time they were meant to listen to a story being read by the teacher. Children’s reading ability was non-existent in S1 but increased through S2 to S3. Not surprisingly, in the informal assessments conducted, most of the strides in reading were measurable in Chichewa rather than in English. 6. Stakeholder Interest – Many organizations became interested in purchasing the TLMs for their own purposes. Among such organizations is UNICEF, Save the Children, Concern Worldwide and World Vision. Each has either placed an order or is considering placing and order with the MIE to purchase considerable numbers of several titles to be used in their own programs. TTCs remain “outside the loop” in terms of receiving and giving training on the use of the TLMs; hence, without pre- and in-service training, it is not clear whether the use of the TLMs is sustainable. 7. Lessons Learned – Foremost is the noted absence of a sustainable teacher training program that would “guarantee” the use of the TLMs in the future. Assumptions made about children’s ability to read and thus enjoy the TLMs were erroneous and defined the entire approach to the development of the TLMs (e.g., use of a whole word/language approach instead of an integrated approach using phonics). Field testing of the books was inadequate; teachers should have been provided the materials for up to three months to try them out and then be requested to explain how they were used and the challenges they faced. Disagreements in the MOEST about the practicability of different approaches to reading led to confusion in the classroom as so many reading programs were being piloted at the same time as RM. TLMP Evaluation Report Volume II – Malawi 121 8. Recommendations Materials Improvement • The books, especially S1 and S2, need to be simplified, placing greater emphasis on the alphabet and syllables. The book should be reviewed and revised according to the realities within each grade. • Encourage teachers to split up the long lessons into several shorter classes in which the value of each book can be fully mined before proceeding to the next book. The notion of “flexibility” is critical. • The read aloud books should be shortened and different activities integrated so that children do not get bored and lose interest (one teacher used Total Physical Response (TPR) in reading the stories, i.e., when an action was identified she had the students get up and perform the action, thus providing one more activity to help comprehend the story). • Teacher’s Guides should not be sheets of paper, but should be bound into a booklet so that they can easily be handled and cannot easily be lost. • The allocation of time for each activity noted in the Teacher’s Guides needs to be reconsidered given the number of students in the classroom, the amount of time it takes to perform the activity, and the amount of time it takes to assess what has been learned. Encourage teachers to be creative and allow them to adapt instruction to their specific classroom needs. Systems Improvement • Introduce and teach basic reading skills including phonological awareness and phonics in S1 through the use of the new Ministry Primary Curriculum and Assessment Reform (PCAR) materials, Malawi Teacher Professional Development Support Program/Early Grade Reading Assessment (MTPDS/EGRA) materials, Timawerenga and Tikwere programs, build on these in S2, and begin using the readers in S3 once children know how to read. Then continue to use the TLMs as designed, as supplementary readers to the PCAR curriculum through S8. • Encourage teachers to learn from each other in using the TLMs by meeting with teachers at neighboring schools (along with Primary Education Advisors [PEAs]) by sharing innovative practices. • Train teachers on how to integrate supplemental materials into their lesson plans. • To increase the number of participating schools, redistribute the materials, perhaps sharing the180 titles among three neighboring schools, rather than all materials being in one school. The schools could then rotate or share the materials on a regular basis. • Provide more guidance on how to manage the books so that teachers can access titles that they need without difficulty. • Train all teachers up through S8 for several reasons: teachers often change grade levels and, thus, need to be prepared for RM should they be placed in S1-S3; and the books can be used in all grades beginning in S3 and ending in S8. • In communities, a greater effort can be made to establish mother and/or father groups to promote a reading culture. Parents could come to school in the afternoons to teach their children to read, and at home parents, older siblings, neighbors or relatives should assist children in reading. Towards a More Comprehensive Approach to Reading Development • The MOEST should be encouraged to follow-up with the development of the new Reading Strategy, which, among other things, would determine the proper role of the current multiple reading programs (or their components) including Read Malawi, Tikwere, MTPDS, and others. Critical to this strategy would be the teaching of phonics in S1 and S2, the use of the PCAR core TLMP Evaluation Report Volume II – Malawi 122 text for grammar in S1 and S2, and the introduction of the TLMs in S3 and their continued use through S8. • A less cumbersome system for reproducing the RM TLMs to satisfy the needs of NGOs must be developed. Additionally, the readers should be made available in bookshops on a commercial basis since books will be lost or damaged and there is no way to replace them. Also, many individuals stated that parents would be interested in procuring these books for their children. • Strengthen the e-Lab benefits for children by identifying instructional software (there’s much free software available on the Internet) and installing the software on the computers and providing guidance on the use. Linking TLMP to USAID Priorities in Early Grade Reading • The methodology used in creating the TLMs generated many culturally relevant, colorful, interesting, and contextualized stories. The strategy should be considered for producing future materials to enhance early grade reading in different mother tongue languages. Additions should include lessons on phonics/morphology, leveling, extensive piloting by teachers, and an agreement to produce the materials commercially so that other stakeholders can easily purchase the materials for their own use. • In rolling out the new early grade reading project, USAID should work with project leaders to determine how best to use the TLMs already produced in Chichewa and English to make them suitable as textbooks and/or to add certain sections or build upon other reading materials to strengthen the TLMs and make them more accessible to children and teachers. • The investment made in the production and distribution of the TLMs should not be wasted but should be built upon, taking the lessons learned from this report and adding other innovations to make the TLMs useful in building a culture of reading. In particular, teacher training should be provided on how to blend the phonics/whole language approach to assist children in learning how to read and to enjoy it. 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES 1.1 The Background to TLMP The Textbooks and Learning Materials Program (TLMP) was launched by the Africa Bureau of the United States Agency for International Development (USAID) in 2005 in conjunction with the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. TLMP was extended for another three years in 2008/09 to 2012. Based upon the provisions noted in their respective Cooperative Agreements (CA), each of the five Minority-Serving Institutions (MSIs) was responsible for managing and implementing the TLMP in a specific country and with achieving specific outputs and results. After having been awarded a CA to work in South Africa in 2005-2008/09, the University of Texas, San Antonio (UTSA), was awarded a second CA to implement TLMP in Malawi 2009-2012. Each MSI was responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 7) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA; TLMP Evaluation Report Volume II – Malawi 123 8) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program; and 9) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e., gender, health, etc.). Originally a program to develop TLMs for primary schools only, MSIs conducted initial needs assessments and determined that in three countries MOE priorities were for middle and secondary school TLMs. In Malawi, the Ministry of Education, Science and Technology (MOEST) determined that supplemental reading TLMs in Chichewa and English should be developed for standards (grades) 1-3. To implement the project, UTSA (and their fellow MSIs) received about the same level of funding - approximately US$8 million. 1.2 Evaluation Objectives This performance evaluation covered work led by UTSA in Malawi during the period 2009-2012 and was intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e., U.S.- based MSIs, in-country institutions, ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. 2. READ MALAWI PROJECT BACKGROUND 2.1 UTSA Responsibilities The TLMP CA implemented by UTSA was called Read Malawi (RM) because it sought to establish a reading culture among the youngest learners in primary schools in both Chichewa (the national language) and English. RM produced books in several categories in accordance with different methods: read aloud, shared reading, and independent reading. TLMs produced for use in the first year were written in Chichewa and those produced for children in the second year in English (translations of many of the stories read in Chichewa in the first year). Building on the teacher-based story writing methodology developed and implemented in South Africa, UTSA included the following program activities in Malawi: • Writing workshops with teachers and others, as they were the original authors of the stories; • Follow-up editing that focused not only on the mechanics and content, but also on how the materials conformed to the curricula developed under the NPC; • Development of teachers’ guides for all volumes; • Illustrator workshops led by the South African illustrator who had done the graphic illustrations for TLMP (“Ithuba”) in South Africa; • Capacity building workshops led by the Institute of Economic Development at UTSA (focus was on small business development) for Kriss Offset (the printer) to improve their management practices; TLMP Evaluation Report Volume II – Malawi 124 • Community mobilization by the Creative Center for Community Mobilization (CRECCOM) to build awareness to the value of reading; • Workshops with lead teachers and primary education advisers (PEAs) on how to use the materials (cascaded to others in their respective clusters, then schools); • Continuous collaboration with the Malawi Institute of Education (MIE), English Curriculum Division (the lead unit working with UTSA) in research, outreach, workshop participation and leadership, etc.; and • Establish e-learning labs at the Lilongwe and Blantyre Teacher Training Colleges (TTCs) through a partnership with Intel Corporation (Intel). TLMs were also produced in braille through a relationship with Montfort College (for the sight and hearing impaired). Montfort translated selected readers into braille and then distributed them to schools that blind students attended, but no training was provided to teachers on how to use them. To enhance teacher training, UTSA created a partnership with the University of California, Berkeley, to establish e-learning centers at the TTCs in Lilongwe and Blantyre. Through another contract with Intel, computers for teachers and students were installed at each TTC. 2.2 MOEST Context and the Educational System The MOEST has aggressively sought to raise its educational standards as a way of improving literacy levels. The Malawi Growth and Development Strategy, the Ministry of Education Guiding Policy Document, and the National Education Sector Plan 2008-2017 (NESP) called for primary schools to 1) equip students with basic knowledge and skills to enable them to function as competent and productive citizens in a free society; 2) reduce absenteeism, increase enrollment and reduce the dropout rate; 3) improve learning outcomes; and 4) retain girls to complete the primary cycle. The NESP launched a sector-wide improvement strategy, including revision of the National Primary Curriculum (NPC). The new curriculum, among other things, focused on reducing the high rates of illiteracy in the early school years. With additional donor support, MIE designed, printed and distributed textbooks in each of the core subject areas for Standards 1 to 8, including Chichewa and English language textbooks. A national effort was undertaken to orient all primary teachers in the more than 5,000 schools to this new outcomes-based education. However, the government recognized the need for additional materials to develop a culture of reading in the primary schools. Different approaches were taken in the development of language textbooks, each of which emphasized good practices developed on the basis of language acquisition and reading theories in other countries and supported by different donors: • SOSA – The name given to Book One, Standard 1, Chichewa under the previous curriculum that focused on phonological approaches to reading development. • PCAR – Primary Curriculum and Assessment Reform – a reform movement in Chichewa reading that has recently moved from the whole word to a more phonics-based approach. • MTPDS/EGRA – Malawi Teacher Professional Development Support (Program)/Early Grade Reading Assessment – a project supporting in-service teacher training with a special emphasis on teaching reading in Chichewa in S1 in seven of the 34 education districts. The approach stresses five core skills including phonological awareness and, for S1 classes, MTPDSP developed scripted lessons and associated readers. The project measured learning gains through EGRA. • Tikwere – A radio program designed to support teachers in the teaching of various concepts, especially mathematics and the teaching literacy, using a phonics-based approach. TLMP Evaluation Report Volume II – Malawi 125 TLMP was the first project the MIE actually worked on in such depth, although, according to senior MOEST officials, it was considered just another “pilot” activity. The MIE staff members were seconded to manage and participate in the project. Teachers who participated in the writing workshops were paid per diem and travel allowances, but did not have salaries topped off by the project. No fees were paid for writing the different stories. A local Program Coordinator was hired to manage the different processes. The MIE worked closely with the Primary Education Advisors (PEAs) who support all education operations of between 10 and 15 schools in their respective zones. PEAs, after initial training provided by MIE and UTSA, cascaded the training to teachers locally, but with each level of cascading, training became “watered down.” In the words of one senior MOEST official: “We need to get rid of at least one level in the cascading process as when the training actually reaches the teachers it is not as effective as the training provided to the master trainers.” Moreover, delivery of the books was not closely monitored initially so there was significant slippage observed in the way books were distributed (e.g., in a cluster of 12 schools, perhaps only nine received them). Once a UTSA manager devised a more foolproof delivery system, the slippage stopped. Once the books were delivered to schools, UTSA lobbied the MOEST for dedicated time in the timetable for teachers to use them. Time was made for their use on Tuesdays and Thursdays, often by extending the class day. (In S1 and S2, classes are only held from 7:30-10:30.) This change edged out other reading programs already in schools, including the use of the English core text (which was recently updated to include more decoding and comprehension skills) and acted as a disincentive for teachers to integrate the TLMs with other materials. This dedicated time was allowed as part of the pilot orientation because teachers were not used to integrating “supplemental” materials into their lesson plans. Devoid of such materials, teachers largely followed the core text. Hence, training was needed on how to incorporate and use other materials to expand children’s ability to read. In other words, according to one MOEST official, the RM materials were seen as “add-ons” that made more work for teachers. Without training on the use of supplemental materials, teachers either followed the teachers’ guides to the letter or did not use them at all. 3. EVALUATION QUESTIONS, METHODS AND LIMITATIONS 3.1 Research Conducted at UTSA The overall evaluation research design called for research to be conducted at the MSIs to determine 1) the background to the project and how it was implemented, and 2) how participation in a USAID-funded program built the capacity of MSIs to lead future projects. The interviews conducted at UTSA were based on several sets of questions posed of leaders, project managers and participants (see ANNEX A1 – Questions Posed in Various Interviews at UTSA). 3.2 Research Activities in Malawi (Lilongwe, Zomba and Blantyre Environs) With the background information provided by UTSA, the Team Leader (TL) developed a background paper/research design that included different categories of questions to be posed to different stakeholders (see ANNEX A2 – Data Collection Instruments, Malawi Stakeholders). The range of stakeholders include MOEST and MIE officials, writers, printers, school directors, teachers, children, community mobilizers, and IT people who worked with the TTCs in establishing and maintaining the computer centers at the two TTCs in Lilongwe and Blantyre. In Malawi, the team consisted of two American and two Malawian experts in education. The team conducted a number of interviews in Lilongwe with USAID/Malawi (USAID/M), MOEST officials, and TLMP Evaluation Report Volume II – Malawi 126 various stakeholders, and then moved on to Zomba and Blantyre to collect data at MIE, schools and from other stakeholders (see ANNEX B – Schedule of Stakeholder Interviews and Activities). In Lilongwe, the team piloted the data collection tools at the Lilongwe TTC and its respective demonstration school. Adjustments were made to the instruments prior to the team’s departure for Zomba, the central locus of project implementation (where the MIE is located). The RM Project Coordinator was hired to facilitate appointments to be made with officials and schools. Unfortunately, the schools chosen in Zomba were “high performing schools,” those that had received additional attention and were well resourced, and were those that had also benefitted from other projects (a concern which USAID/M cautioned us against). Hence, the results achieved for Zomba are not representative of what was achieved elsewhere. In Blantyre the team chose its own schools, following our own parameters and those of USAID/M, and was able to obtain information believed to be more generalizable to the broader school population. The Education Officer responsible for the project at USAID/M accompanied the team during the Blantyre data collection and was able to observe results for himself and corroborated other research conducted in Mission-based programs. At each school we interviewed the director and teachers who were using the TLMs and observed how teachers used the TLMs in the classroom. We also administered an informal reading assessment using both Chichewa and English TLMs that called for children to name certain letters, read certain sight words, and listen to a passage and answer questions in English about the passage. Assessment materials were taken directly from the grade-specific materials of the children tested, and with which the children ought to have been very familiar. The materials were from standards (grades) 1, 2 and 3 TLMs. When the PEAs were located at a school, we also interviewed them; otherwise we went to their offices. 3.3 Limitations of the Study Overall, there was insufficient time to conduct a full evaluation that was geographically inclusive and that included schools chosen on the basis of a random sample. We also did not have time to collect data on schools that had not received the books to determine what difference having the books made to the children’s ability to read. When the two American evaluators arrived, Malawi was experiencing heavy rains that interfered with our movements to schools. The road between Zomba and Blantyre (under construction) was flooded necessitating our making other plans (two days before moving to Blantyre, the road was re-opened). Two days after launching the research in Lilongwe, the Civil Service went on strike. The meetings we had scheduled with MOEST officials had to be rearranged as they were told not to come to the office. Two officials came to our hotel to be interviewed, while two others refused to do so. Another was addressing Parliament on the salary needs of teachers and so did not meet with us until we returned to Lilongwe from Blantyre. With teachers on strike, data collection at several schools was inhibited and we could not conduct the research as extensively as planned. The full set of TLMs produced was not available at any time for team review. At MIE we were told that the books had all been distributed and none were available – a “fact” we found very strange since MIE was the implementing partner, held the copyright, and was responsible for going out on bid to reproduce any of the materials. 4. FINDINGS AND CONCLUSIONS We begin this section of the report with the data collected at UTSA in Texas so that the reader can understand how an MSI dealt with the challenges of being awarded a USAID CA to conduct work in an TLMP Evaluation Report Volume II – Malawi 127 African country for the very first time. We then move on to the print output of the project so that the reader is aware of the number of items produced and to whom they were distributed. Findings and conclusions on the implementation of the project are then presented as are the results of teacher observations and student reading assessments. 4.1 UTSA Achievements and Challenges Although globalization is a primary focus for the 40-year-old university, some of its systems and procedures were not in sync to take on international development projects. Significant changes took place in Business Affairs and in Grants & Contracts: templates were redesigned for routing so as to reduce approval time frames, forms were generated to increase efficiency, travel disbursements were managed differently so as to allow for international/developing country expenditures and included per diem (no credit cards), etc. 4.1.1 Administrative Adjustments: Through implementation of the TLMP, the following changes were made: • Procurement (under VP) – An international bidding system was put in place to let out contracts for supplies in Malawi that emanated in Malawi, South Africa, or elsewhere. The university had to overcome the challenge of a legislative mandate to take bids from the historically underutilized businesses (HUB) roster in Texas. The ceiling for procurement was also lifted and now can go up to $1 million before needing board approval. • Consultants vs. Sub-awards – Initially, individuals from other institutions were to have been awarded consulting contracts by UTSA; with further inquiry, however, the Office of Research determined that sub-awards should be offered to the University of Texas, Austin (UTA) – the institution at which several TLMP consultants were based. This different approach triggered cost-sharing by several institutions so that UTSA did not have to provide total costs. • Changes in Per Diem and Travel Advance Practices – Payment of per diem, in accordance with USAID parameters, was instituted for UTSA faculty as well as MIE and other partner staff (when they came to the US). Faculty and staff also had to obtain travel advances as not many hotels/restaurants in Malawi would take credit cards. A system in compliance with IRS regulations for paying international visitors was also developed. • Travel Policies – Because travel had to be arranged quickly at times, the approval process was streamlined and new operational guidelines developed. 4.1.2 Beneficial Impacts on UTSA and its Community - Participation in TLMP, both through Ithuba in South Africa and through Read Malawi, had the following beneficial impacts on UTSA: • Community Impacts – Faculty who have participated in TLMP now share what they have learned in their classes in which students are largely teachers themselves. Faculty reputations have been enhanced by their international participation, and now more students are signing up for education abroad. • Increased Credit Score – With the inflow of grant funding, the credit rating of UTSA advanced to AAA, thus allowing them to borrow money to build more dormitories and classrooms. Ten years ago, international revenue was $200 million; it is now $450 million, 50% of UTSA income. • Institutional Internationalization – UTSA has gone on to sign a Memorandum of Understanding (MOU) with China for students to study at UTSA, and to develop projects in Peru, Honduras, Mozambique, and Pakistan. The Institute for Economic Development is TLMP Evaluation Report Volume II – Malawi 128 expanding its outreach beyond Mexico and South America and is looking forward to taking advantage of other small business development funding opportunities in other countries. Many departments, centers, and institutes are now exploring global initiatives. The Center for Vaccines has developed a vaccine against chlamydia and would like to explore how it could be used in developing countries. Collaborating schools have developed a cell phone solar charger and would like to explore how this could be used in developing countries. • International Reputation – UTSA was designated number 53 out of 100 top research universities under 50 years old. 4.2 TLMP Output Initially, in 2009, the project had a $13 million budget, but this was cut to $8,859,624 (USAID funding of $8,150,528 plus institutional cost-share) because of USAID overall budget cuts. With this level of funding, RM generated 183 titles (120 in Chichewa and 60 in English, two alphabet books and an overview guide) that were developed by teachers and other educators, illustrated, printed and distributed to 1,272 educational centers (945 primary schools, (three per zone) out of 5,300 primary schools, 6 TTCs, and 350 Teacher Development Centers (TDCs) at which PEAs were located). In addition, 90 Teacher’s Guides and training manuals were produced, all in accordance with NPC guidelines. The following is a summary of what was produced and distributed: • Read Aloud Books: Chichewa, 1,200; English, 600 • Guided Reading Books: Chichewa, 1,200; English 600 • Alphabet Books: Chichewa 120; English 80 • Overview Guide: English 6 • Teacher’s Guides: Chichewa 180; English 90 • Big Books: Chichewa 20; English 10 • Alphabet Posters: Chichewa 4; English 4 • How to Care for Books Poster: English 4 • Water Sources Poster: English 4 A total of 5,260,548 materials were printed and distributed. Overall, each receiving institution was given 3,700 books. The average cost of production/unit in each category was: • Teachers’ Guides – Average cost of $0.18/unit (printer 1 $0.16/unit; printer 2 $0.21/unit) • Learners’ Books – Average cost $0.33/unit (printer 1 $0.31/unit; printer 2 $0.35/unit) Unit costs do not include delivery as this occurred across units and cannot be divided between the two. Costs also do not include transportation, custom fees and school delivery of books initially produced in South Africa and sent to Malawi. 4.3 Project Management and Partnerships As RM was a centrally-funded project, USAID/M assigned one person to maintain liaison with RM and to exercise some oversight over its activities. The relationship between UTSA and USAID/M was cordial and it appears that many joint decisions were made on how to proceed. Early in the project, however, the head of the Mission’s Education section was transferred to Ghana, and she was not replaced for approximately a year, leaving responsibility for RM in the hands of an assigned officer. TLMP Evaluation Report Volume II – Malawi 129 When RM was launched, officials at the MOEST were skeptical about the ability of teachers and other educators to create stories as had been done in South Africa. Owing to the major reform movements taking place in Malawian education at the time, however, the MOEST decided to accept the program outlined by UTSA. That Malawians became authors of the supplemental readers became a source of pride not only for the MOEST but for the authors themselves. The MOEST delegated the task of partnering with UTSA in the development and implementation of RM to the MIE. The English curriculum unit was designated as the lead department, even though readers would be developed in both Chichewa and English. While several textbooks were being written (with other donor support) that focused on phonics, the focus of the supplementary readers was on whole word or whole language. This approach assumed that children beginning in Standard 1 were already able to read. Despite the expert input from several institutions, the ability of the project – according to an MOEST senior official – to accurately level the materials (that is, to assure that the level of the materials was appropriate for the learners’ age and knowledge base) was “missed.” Typical Bantu language morphology consists of syllables comprised of a vowel by itself at the beginning of a word or of a consonant followed by a vowel. However, this morphology was not characteristically followed in producing the books because the pedagogical approach did not reflect the use of phonics. In fact, frequently very long words and names were used that children could not decode regardless of how long or how hard they tried to figure out what was written. In schools where other donor materials based on phonics were being used, alongside RM readers, children were better able to decode. Comprehension, however, remained a problem. For sustainability purposes, we inquired into how TTCs dealt with the RM materials. At the Lilongwe TTC, English language faculty members stated that a maximum of only one hour of instruction (out of a year of classroom instruction and a year of a practicum) was dedicated to the RM materials under the Language and Literacy Syllabus. At the Blantyre TTC, RM books are dealt with under the three hours allocated for the year under the curriculum topic “Dealing with Supplementary Readers.” Although teacher training on the use of the materials was not included in TLMP CAs, UTSA determined that such training was necessary. Selected educators were trained, including PEAs, who were then charged to cascade the model to teachers in the schools for which they were responsible. The model was watered down, and teachers trained by PEAs and others did not benefit in the same way as those trained originally by RM staff. PEAs did provide RM support in their respective zones. In one zone, however, the PEA had issued a directive to stop using the RM materials as “the project is over.” This decision appears to have been made unilaterally and affected only one zone. The hundreds of books that had been delivered to the three schools in that zone remained in unopened boxes in locked storage rooms. While a Program Advisory Committee (PAC) was established at the outset of the project, it was not called into session to make any major decisions or to provide any specific input. A partnership created to assist RM in creating a reading culture was with CRECCOM, an organization that sensitized the community on how to care for books and the pedagogy of the RM instructional model. Community based activities within the CRECCOM model included: theatre for development; awareness/sensitization meetings; mother groups (reading groups led by mothers); volunteer assistants; role modeling; and school incentive packages, to name a few. The shortcoming of CRECCOM’s TLMP Evaluation Report Volume II – Malawi 130 participation was that it was in only one community per district and the organization’s ability to reach out further was limited by funding. A second partnership that was created was with UltiNets under the sub-project name of Intel Teach to establish e-Learning Labs at Lilongwe and Blantyre TTCs. Under this initiative, UTSA entered into an agreement with UC Berkeley and INTEL to establish computer labs (with 96 computers each, 48 for use by faculty and 48 for use by children at the demonstration school). The computers were used by teachers writing the RM stories. Beyond this initial use, it was unclear how RM used this resource. Additionally, there was no instructional software (except for one math program – Easy Learning) and no instructional syllabus available to teach teachers and students how to use the computers. A third partnership was created with Montfort College for the visually and hearing impaired. Under this agreement, Montfort transliterated 60 of the stories into braille, and then distributed them to 30 special education centers. However, no orientation or training was provided to teachers or administrators, leaving them unclear on how to use the readers. A final partnership was created for the printing of the TLMs with Kriss Offset, an impressive family￾owned printing company located in Blantyre. Printing of the first run of materials (developed prior to the launching of RM and not in accordance with RM processes) was undertaken in South Africa with a printer that was used during the implementation of UTSA’s TLMP in that country. When the order was received by Kriss Offset to repack for internal distribution, it was found to be short, pages were printed upside down, quality control measures were not taken, and importation of the materials was costly. Thereafter, UTSA entered into an agreement with Kris Offset to print the materials in Malawi. They did so successfully and economically. However, UTSA made some promises to Kris Offset about the number of books to be produced that never materialized owing to USAID budget cuts. Kriss Offset invested in building a warehouse to store books before they could be shipped to schools by the MOEST. The order was subsequently cut from 4,500 schools to 1,272 learning institutions. Kriss Offset also believed that other orders would be forthcoming, but the cumbersome bidding process required by MIE prevented Kriss Offset from winning any subsequent bids. Hence, Kriss Offset, while benefiting from the order they did receive, did not expand their operations to address this particular order. In conclusion, the management structure and partnerships created enabled UTSA to deliver most of what it promised. The major gap in all areas, however, was the training needed on the use of all materials and the integration of the partnerships with other administrative activities. 4.4 Project Implementation Because the MIE had not been a part of the early decision-making concerning the approach and operations of RM, UTSA acted on a number of assumptions about education in Malawi that were not valid: UTSA assumed that children acquired reading skills in the first grade and on this basis a “whole language” approach was used in developing the TLMs; there was no Malawian expert in early childhood education and early literacy on the project; the faculties of the TTCs were not consulted in terms of the type of training teachers were provided; and very little, if any, time was spent in the classroom to determine the challenges teachers faced nor were teachers asked how they cope. Once the materials were developed, writers themselves were asked to “field test” the TLMs on the basis of a check sheet rather than through actual use by teachers over time. Hence, teacher feedback was minimal. TLMP Evaluation Report Volume II – Malawi 131 4.4.1 Materials Development Guided Reading and Read Aloud Books Malawian teachers, curriculum specialists, artists, desktop designers, editors and printers were all identified by UTSA, UTA and MIE to develop and produce the TLMs. Through a series of writing workshops at two TTCs, authors of books (who were in-service and pre-service teachers, MOEST officials, TTC Lecturers and MIE curriculum specialists who were fluent in English) learned to write high quality complimentary reading materials that were situated in local contexts and took into account certain cross-cutting themes: gender; HIV/AIDS; and local concerns. These materials were authored in both Chichewa and English. Authors used a combination of their lived experiences and topics of interest to learners as a basis for the books. The educators writing the stories participated in three workshops. In the first workshop, they learned to document and describe experiential events of interest to children. The books were authored in Chichewa and translated into English for further development; authors revised their Chichewa versions recursively. During the second workshop, authors continued to revise their books; they were encouraged to keep their audience in mind through crafting mini-lessons. Included in these mini-lessons were the topics of content, word choice and imaging decisions. In the third workshop, the stories were field-tested with learners in classrooms. The field testing was problematic as 1) there were no specific guidelines and authors merely observed what the children could manage, and 2) much of the field testing was conducted at TTC demonstration schools where children and teachers receive more attention. Leveling then became a more definitive problem. Such items as word length and difficulty, sentence structure and length, repeated phrases, picture support, etc., were partially addressed, especially in the guided reading books; but the read aloud books included many challenges for both teachers and students. The illustrations for the materials were developed and designed by local artists either working for the Nation Newspaper or employed by the MIE. Typically, each book has between seven and ten images. All images were painted by hand (it was deemed by UTSA that the color quality from computer￾generated images was too poor). Illustrators reported that they had to do their work in a very short time frame (generally no more than three weeks) when the UTSA Project Director was in-country, and felt that this was not their best work. They were paid on the basis of their productivity, although some illustrations were rejected at the last minute, meaning that they were not fully paid for their output. Teacher’s Guides The Teacher’s Guides, developed in both Chichewa and English, were linked directly to the national curriculum and provided teachers with ways of developing literacy and subject area skills. Those written for English instruction supported second language acquisition practices and the teaching of reading strategies, including word identification/knowledge, fluency strategies and comprehension strategies. The Teacher’s Guides used a daily, patterned step-by-step approach that included (on the guided reading day) a preview of difficult words, guided writing, shared and guided reading, word wall time, and life skills inquiry/home connections, each with a stated amount of time to accomplish each task (for a total of up to105 minutes, depending on the grade level). On the “read aloud” day, time was allotted to reporting/ gathering time, read aloud, response activity, independent reading time, individual assessment and support, and promoting a reading culture (for a total of 105 minutes). Because each lesson is so constructed, once teachers grow accustomed to the patterns they can implement them in ways that are helpful to their learners. However, teachers reported that the time allotted to each activity was insufficient. At well over an hour the full lessons were too long, and the children lost interest because they also had to share 20 books among as many as 120 children in a classroom. Moreover, the transition time between activities TLMP Evaluation Report Volume II – Malawi 132 took time away from the substance of the activity, leaving the lesson rushed and/or incomplete. The time allotted was inadequate for the number of words that had to be previewed and the types of activities that had to be accomplished, but too long for the comfort and attention level of the learners. The Overview Guide includes six key components or “big ideas”: 1) The teacher uses a variety of tools to scaffold understanding between oral and written language; 2) In Guided Reading, the teacher gradually releases responsibility to learners; 3) The teacher uses the literate environment to scaffold connections between home/school and life skills subject area; 4) The teacher uses the Read Aloud books as an instructional practice to purposefully scaffold comprehension of text; 5) The teacher provides learners with opportunities to respond to the Read Aloud and Guided Reading books in creative ways; and 6) The teacher promotes independent reading and a culture of reading. While these big ideas were useful in guiding the development of the TLMs and Teacher’s Guides for each lesson, teachers were presented with many challenges in implementing them. 4.4.2 Comments on the TLMs Made by Teachers, Directors and PEAs - During the two-week field visit, the evaluation team visited 13 schools in six districts, interviewed 40 head teachers/management staff, 13 groups of teachers totaling over 50 teachers, and eight PEAs (who provided additional information on three other schools). The team also met with about 100 learners as part of the reading assessment. We had anticipated providing separate findings for each of the stakeholders noted; however, the responses from each of these groups were so similar that we consolidated them, noting differences when reported. Responses to our interview questions on the materials and their use were both positive and negative. Positive Responses Children. The largest number of positive responses (12) indicated that the materials helped some children to read, learn new words, and construct sentences. The materials motivated learners (8) because the books were attractive with colored pictures (5), and had interesting stories and meaningful (culturally-based) pictures (5). A smaller number of responses included an improvement in the reading culture, using pictures to help learners predict stories, and the opportunity for children to take books home where they could practice reading with their parents or others. Other responses included children learning about their past from stories (when books were taken home and discussed, parents and grandparents recalled hearing similar stories and discussed them with the children), the ability to tell stories, developing good listening skills, and developing skills to think critically. Teachers. Teachers reported that the training was good, that they gained knowledge on how to teach reading, and boosted their interest in stories and reading. The lesson plans, charts and books helped teachers to be more effective. Parents. Children and parents in certain areas borrowed books from the local school to take home, which helped to develop a reading culture, gave children an opportunity to practice reading at home, and increased communication between parents and teachers. TLMP Evaluation Report Volume II – Malawi 133 Negative Responses Books Too Difficult. Stakeholders reported that the books were too difficult, especially for Standard 1 (S1, equivalent to first grade) and Standard 2 (S2, equivalent to second grade). Books for S1 had long sentences and difficult vocabulary rather than letters, syllables, and simple words and phrases. Teachers pointed out that the children were not able to read without an introduction to the alphabet and syllable construction, and that the whole word approach was not effective for beginners. They reported that children were interested in the pictures, but not in reading, and often just tried to memorize what the teacher was reading from the books. The Read Aloud books tended to be too long and difficult and the children lost interest. Class Periods Too Long/Too much Material. Class periods ranged from approximately 90 minutes in S1 to up to 120 minutes in S2 and S3. One teacher noted that the extended day actually increased absenteeism. There was just too much material and children became bored during the long lessons. This was also stressful for teachers. Several teachers noted that it was “impossible to teach one book in one day,” which was the expectation set by RM. Under this system, the teachers and children rushed through a book, typically, with the children learning very little and, in most cases, unable to read any of the text. As one teacher said, it seems like the program is “teaching titles and not reading.” The instructions for teachers resulted in one book being used for one day, and then never seen again in that class. Thus, there is a situation in which the lessons are too long, yet there is more material than can be taught even in this long period. Timetable Issues. RM lessons were scheduled on the timetable for Tuesdays and Thursdays (without MOEST approval, according to a senior official) and created conflicts with other subjects. The programs “collide, overload, and confuse students.” Also, many teachers were struggling with how to deal with TLMs, especially when the lessons take up so much time and the teachers also have to attend to other reading and English language programs, in addition to the official PCAR curriculum - the radio Tikwere programs - and, in some districts, the MTPDS/EGRA lessons. Also, one teacher noted that these three programs were not examinable; only subjects that are officially on the timetable are included in the exams. Too Few Copies of Each Title. An overarching misconception of the TLMs on the part of stakeholders was that they were textbooks and not supplementary materials. Teachers were not used to integrating supplementary materials in their lesson plans (nor did they have training on this in their year-long TTC program). Hence, teachers believed that the TLMs were their textbooks. With that perception, teachers reported that 20 copies of a given title were too few for classes with up to 120 students. Too Many Books. Paradoxically, while each school may have too few copies of each title, only 20, at the same time they had too many titles to use. The teachers were not sufficiently knowledgeable about how to integrate this range of materials into their daily lesson plans. While teachers struggled with the limited number of books for a given title, in the storeroom of some schools, there were boxes of books that had never been opened. Also, while most schools have encouraged children to take books home, some of the schools have limited this practice because of books becoming lost. And, since the books are not available commercially or from the MOEST/MIE, there is no way to replace them. Training on the Use of the Books. Several teachers noted that the training was too short, and that the few teachers who attended the training did not always train their fellow teachers at the school level. Also the PEAs who were responsible for the training, were overloaded with too many responsibilities. There was limited school-based training, and some teachers felt that RM was imposed on them. Many teachers commented that the allowance money provided during the training was inadequate. TLMP Evaluation Report Volume II – Malawi 134 4.2.3 Conclusions: The number and type of books created by RM is commendable, especially in light of the methodology used to create them. The stories were written in Malawi and, thus, are relevant and of interest to children and teachers. The books are full of attractive pictures, with one illustration for each page of text in the guided readers. They are virtually the only additional print materials available to teachers and children. This is a major accomplishment. The children love to hold the books and enjoy the pictures, which serve as a jumping-off point to understanding the stories. The stories encourage children’s prediction and creative thinking skills. And the teachers report the program is helping to develop reading skills and a reading culture among both the children and parents. However, the full benefits of the program are constrained by several factors. First, the language of the books, especially in S1 and S2, is far too advanced for the children. In S1, for example, rather than starting with letters, syllables, and simple words, the books begin with full paragraphs, sentences and big words. There are claims that RM helps children learn to read, but the TLMs are not books for beginners. Once children have developed basic reading skills, these books can make a major contribution to develop further reading skills and create a reading culture. The TLMs produced by RM are not treated as “supplementary” but rather as a separate subject because they appear on the timetable every Tuesday and Thursday. The time allocated is very long for the early grades and children lose interest. Moreover, these long periods cut into and eliminate other subjects that are normally part of the curriculum. Another problem with the implementation of RM is that teachers are instructed to complete one book during a single class period, a virtual impossibility if the children are to get any significant reading skills out of the lesson. And the long lessons have forced many schools to extend the school day, which is not yet official policy. And while there are a large number of books, there are only 20 copies of each title, making RM difficult to use in even modest sized classrooms. There are too many titles for any one school, as evidenced by boxes of books remaining unopened, yet too few copies of each title to be used effectively in the classes. In summary, although RM was highly successful in developing a large number of attractive books, the books were not leveled properly, especially for S1 and S2, and the implementation framework and expectations have created problems in the schools and classrooms, which have limited the effectiveness of the program and created problems in relation to other subjects on the timetable that are to be taught. 4.5 Outputs and Outcomes 4.5.1 Teacher Observations: One element of the evaluation was observing teachers using the TLMs in their classrooms. Because of the civil service strike, we were unable to do this at each school and, in some cases, a lesson was “staged” because we came on a day other than Tuesday or Thursday. With these caveats, we nevertheless present a summary of the classroom observations. Of the teachers observed, six were teaching at the Domasi Government School (part of the MIE “lab” approach); four at the Domasi Demonstration school, a school that also got additional attention; four were at Mponda “High Implementing School” in Zomba town; three in rural Blantyre at PIM Primary School, and two at Mafe School. The average number of learners in the classes observed was 72 and the mother tongue was largely Chichewa, with a few in the class speaking Yao. Four of the teachers were males and fifteen were females. The observation form identified items to be checked that were “observed” or “not observed.” The highest number of “observed” checks was achieved by teachers at Domasi Demonstration School, and TLMP Evaluation Report Volume II – Malawi 135 the lowest was PIM Primary School. In only one instance (for standard/grade 3) was writing used to reinforce what was learned, and was continuous assessment of homework assignments in evidence (at Domasi Demo). In only six observations did the teacher explain the goals and purpose of the class, identify difficult words before undertaking the reading lesson, and explain the lesson in English (in other instances it was explained in Chichewa). In 50 percent of the lessons observed, children could not understand the English being used. In seven cases, the teacher did not write the lesson objectives on the board; and in at least five cases teachers did not demonstrate personal mastery of English. While we cannot draw strong conclusions based on this limited sample, we do take these findings as indicative of the ability of teachers to provide instruction in English, follow the teacher’s guides appropriately, and to fit the entire lesson into the time allotted. 4.5.2 Learner Reading Assessments and Outcomes: The range of reading/language development programs in each school made it difficult for us to attribute reading ability to the use of the TLMs. Nevertheless, we endeavored to ascertain children’s ability to decode and read the TLMs that they had recently focused on in their classes. The purpose of this assessment was not to carry out a rigorous impact evaluation, but to try to determine what the outcomes were of the lessons taught in the classroom. The results of these assessments need to be interpreted with caution for several reasons: • Schools were not selected at random. Because of the limited time for field activities, the evaluation team selected schools largely for their accessibility during periods of flood and teacher strikes. • Students were not always selected at random within a class. As a result of the strike, students who happened to be at the school in the appropriate grades were selected for the assessment. In other cases, where school was in session, teachers chose the students to participate, and often this resulted in the “best” students being chosen. • To minimize possible influence on reading scores as a result of other reading programs in a school, the team tried to choose districts and zones that did not have other programs. The one exception was Zomba Rural that was also using MTPDS/EGRA. Although the test scores would be influenced by MTPDS as well as RM, it was helpful to talk to teachers about the relative advantages of each program. • There are no comparison scores. The evaluation team did not have baseline scores or the results of comparable schools that did not participate in the RM program For a presentation of schools selected, a description of the instruments, and how the reading assessments were undertaken, see ANNEX C – Reading Assessment Methodology. Basic Reading The results show a steady improvement in reading scores at each level. For example, in Chichewa, virtually no one could read in S1, by S2 about one-third of the children were able to read, and by S3 the children who could read increased to 45 percent. Regarding letter recognition, the S1 children could identify some letters – 42percent could identify M and 26 percent could identify S. By S3, the letter recognition scores increased to 85 percent and 90 percent respectively. As expected, word recognition also increased substantially from 21 percent in S1 to 67 percent in S 3. TLMP Evaluation Report Volume II – Malawi 136 While these results show a significant improvement in reading scores from S1 to S3, there is no indication of RM’s influence in this improvement because there is no baseline or control group. Nevertheless, it is clear from these results, that many children are learning basic reading skills in the early primary grades. Yet, even by the second term in S3, only 45 percent could read fluently and 21 percent could not read at all. A summary of the findings by Standard/grade level is as follows: Standard Read Chichewa Fluently Can’t Read/ Identify Letters in Chichewa Read English Fluently Can’t Read/ Identify Letters in English 1 4% 96% 2 31% 59% 6% 88% 3 45% 21% 32% 47% Comprehension After reading the passages, the evaluation team members determined whether the child had enough reading ability to take the comprehension test. Those results are as follows: Language and Grade Level Number of Children Percentage Able to Comprehend Passage Chichewa 2 2 out of 2 100% Chichewa 3 11 out of 26 82% English 3 4 out of 22 25% The few children who took the comprehension test did well in Chichewa, but not well in English. Gender When the test results were disaggregated by sex, in general, the boys scored higher than girls. When all boys and all girls across both languages and the three grade levels are calculated, 35 percent of the boys and 26 percent of the girls can read. Conclusions The results of the reading assessments show a steady improvement in reading skills across the three grades. Whereas there was virtually no ability to read in S1, by S3 45 percent of the children could read fluently in Chichewa and 32 percent in English. Few of the children had enough reading skills to take the comprehension test, but those who did take the test in Chichewa scored well. Finally, with the exception of S2 in Chichewa, boys significantly outperformed the girls on the reading tests. Nevertheless, these reading scores need to be treated with caution because of the lack of a rigorous procedure for selecting schools and children, and the lack of any comparison data. It is impossible to tell to what extent RM may have contributed to these gains in reading scores. Particularly given the poor leveling process for the books in S1 and S2 and the complete lack of any introduction of basic skills in recognizing letters, syllables, and words, it seems unlikely that RM would have contributed significantly to the development of initial reading skills. However, once those skills had been obtained through the use of other teaching methodologies, the TLMs could have contributed to developing further reading skills. TLMP Evaluation Report Volume II – Malawi 137 4.6 Stakeholder Interest in and Use of Materials/Sustainability 4.6.1 NGO Reading Programs: The development and publication of so many supplementary readers came to the attention of UNICEF and other NGOs implementing reading programs in Malawi, i.e., Save the Children, Concern Worldwide, and World Vision. UNICEF - Will purchase 30 sets of the RM books for libraries in district schools by the end of the project in 2016. By that time UNICEF will have stocked 300 libraries with RM materials. Save the Children - Has placed an order for 10 titles in English – 35,000 books – and plan to purchase the same in Chichewa to support their Literacy Boost program in Zomba that includes Reading Camps with links to 40 primary schools in four zones. It is envisioned that 350 camps will be established with each school having 5-10 camps to support reading activities Concern Worldwide – Has placed an order for approximately 133,000 readers, teacher’s guides and TLMs in Chichewa and English for use in schools included in their field of operation. World Vision – Is considering purchasing an undetermined number of books for distribution in its Area Development Programs (ADP) throughout the country. A stumbling block to the fulfillment of these orders is the cumbersome bidding process that MIE, the owner of the copyright, must go through. Several of the NGOs consulted cited the long lead time needed for bids to go out, a winner to be chosen, and production to take place. While the MIE is looking into ways to cut down this time-consuming process, it has not achieved success, thus frustrating the NGOs who wish to use the materials. 4.6.2 TTCs: There are no current plans to include training on RM TLMs at TTCs. It is possible that when the Director of Inspectorate and Advisory Services (DIAS) completes his strategy paper on reading, literacy, and language instruction, a more expansive teacher training element will be included. There was no appropriate pre- and in-service teacher training on the use of supplementary reading materials. It is likely that the books not yet opened and stored in secure closets at receiving schools will not see the light of day. Teachers find them too much of a challenge to use in the way prescribed, and in self-contained classrooms take time away from studying other subjects. With regard to using the TLMs at TTCs, when it became clear that the TTCs needed assistance in developing Continuous Professional Development (CPD) modules, UTSA promised these would be provided. They were never developed. Hence, the TTCs are not prepared to provide training on the use of the TLMs to either pre-service or in-service teachers. It remains to be seen what the new strategy will be and how both core textbooks and supplementary readers will be integrated. 5. LESSONS LEARNED • In designing and implementing a textbooks and learning materials development program, teacher training on how to use the new materials must be included at both the pre- and in-service levels so that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use, and for the textbooks and learning materials to be used over the long term. • When piloting newly-developed textbooks and learning materials, teachers should be given the textbooks to use in the classroom for at least a month (better, for three months) and then research TLMP Evaluation Report Volume II – Malawi 138 should be undertaken to determine challenges teachers faced and/or overcame in using them so that they can be revised before final printing. • Before launching a TLM production project, thorough research needs to be undertaken into the educational system on how teachers are trained (and for how long), the frequency and substance of in-service teacher training programs, and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. • Before launching a TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project, the learning outcomes and results achieved in using those books, and the level of reading ability children have under past practices so as to avoid making assumptions about what children are able to read. • Launching an innovative TLM production project works best when the MOEST, MIE, and PEAs are working together in an overall learning improvement program that is not competing with other programs to create results. • “Leveling” any reading materials is absolutely essential to creating positive learning outcomes among children. • Having a management entity at MIE improved management, administrative and financial practices. In so doing, MIE not only had technical expertise in the subject matter, but also had management expertise, which led them to applying for grants on their own. • RM works well with learners that have already developed skills in identifying vowels and consonants, and are able to form basic syllables and simple words. • Delivery of TLMs worked best once a foolproof tracking method was developed. • For a U.S.-based university to work in an African country successfully, staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts preventing long-distance communications). 6. RECOMMENDATIONS While RM has experienced a number of successes, recommendations made below point to areas that need improvement: 6.1 Materials Improvement • The books, especially S1 and S2, need to be simplified, placing greater emphasis on the alphabet and syllables. The books should be reviewed and revised according to the realities within each grade. • Teachers should be encouraged to split the long lessons into several, shorter classes in which the value of each book can be fully mined before proceeding to the next book. The notion of “flexibility” is critical. • The read aloud books should be shortened and different activities integrated so that children do not get bored and lose interest (one teacher used Total Physical Response (TPR) in reading the stories, i.e., when an action was identified she had the students get up and perform the action, thus providing one more activity to help comprehend the story). • Teacher’s Guides should not be loose sheets of paper, but should be bound into a booklet so that they can easily be handled and cannot easily be lost. • The allocation of time for each activity noted in the Teacher’s Guides needs to be reconsidered, given the number of students in the classroom, the amount of time it takes to perform the activity, and the amount of time it takes to assess what has been learned. Encourage teachers to be creative and allow them to adapt instruction to their specific classroom needs. TLMP Evaluation Report Volume II – Malawi 139 6.2 Systems Improvement • Introduce and teach basic reading skills including phonological awareness and phonics in S1 through the use of the new Ministry PCAR materials, MTPDS/EGRA materials, and Tikwere programs, build on these in S2, and begin using the readers in S3 once children know how to read. Then continue to use the TLMs as designed, as supplementary readers to the PCAR curriculum through S8. • Encourage teachers to learn from each other in using the TLMs by meeting with teachers at neighboring schools (along with PEAs) by sharing innovative practices. • Train teachers on how to integrate supplemental materials into their lesson plans. • To increase the number of participating schools, redistribute the materials, perhaps sharing the 180 titles among three neighboring schools, rather than having all materials remain in one school. The schools could then rotate or share the materials on a regular basis. • Provide more guidance on how to manage the books so that teachers can access titles that they need without difficulty. • Train all teachers up through S8 for several reasons: teachers often change grade levels and, thus, need to be prepared for RM should they be placed in S1-S3; and the books can be used in all grades beginning in S3 and ending in S8. • In communities, a greater effort can be made to establish mother and/or father groups to promote a reading culture. Parents could come to school in the afternoons to teach their children to read, and at home parents, older siblings, neighbors or relatives should assist children in reading. 6.3 Towards a More Comprehensive Approach to Reading Development • The MOEST should be encouraged to follow-up with the development of the new Reading Strategy, which, among other things, would determine the proper role of the current multiple reading programs (or their components) including Read Malawi, Tikwere, MTPDS, and others. Critical to this strategy would be the teaching of phonics in S1 and S2, the use of the PCAR core text for grammar in S1 and S2, and the introduction of the TLMs in S3 and their continued use through S8. • A less cumbersome system for reproducing the RM TLMs to satisfy the needs of NGOs should be developed. Additionally, the readers should be made available in bookshops on a commercial basis since books will be lost or damaged and there is no way to replace them. Also, many individuals stated that parents would be interested in procuring these books for their children. • Strengthen the e-Lab benefits for children by identifying instructional software (there’s much free software available on the Internet) and installing the software on the computers and providing guidance on the use. 6.4 Linking TLMP to USAID Priorities in Early Grade Reading • Educators constitute a valuable resource in the development of culturally-relevant reading materials. In focusing on early grade reading in mother-tongue languages, it would be useful to utilize the method that UTSA developed in creating TLMs. Developing the stories in local languages by those who speak those languages and who were socialized into their use will create reading opportunities of interest to children, their families and teachers. However, integrated into this methodology should be lessons in the phonics/morphology of local languages and English (if books will also be produced for early reading in English). Leveling must also be paid considerable attention, as does the rigorous testing of materials in local languages and English to determine their suitability, level of enjoyment, etc. Once produced, materials should be made available commercially so that other stakeholders might be able to purchase the materials for their own use in a timely manner. TLMP Evaluation Report Volume II – Malawi 140 • USAID programs in early grade reading should consider how to use the TLMs produced, e.g., implementing many of the recommendations made in the above sub-sections. To make these TLMs suitable as textbooks, introductory workbooks or texts need to be developed in phonics/morphology and linked to the vocabulary, syntax, and style of the stories printed. Alternatively, materials developed for other reading programs can be used to introduce the TLMs in Chichewa and English. This strategy might require a “renumbering” of the TLMs to reflect the skill level developed in decoding and comprehension. • The reading resource developed through TLMP in Malawi utilized a fairly rigorous approach that produced TLMs that both children and teachers enjoy for their color, layout, content, cultural relevance, gender sensitivity, and contextual issues raised. That so many titles were produced in both languages is an attestation of UTSA’s and MIE’s desire to fulfill the project plan and provide students with opportunities to read they did not have before. This resource should not be wasted; on the contrary, a concerted effort should be made to link the early grade reading program focus of USAID with TLMP. That future reading programs will have an on-site management entity should smooth out any operational challenges as well as create a more integrated effort. TLMP Evaluation Report Volume II - Malawi 141 ANNEX A-1. Questions Posed In Various Interviews Conducted At UTSA 1. What have you learned from implementing the project in South Africa that you are using in Malawi – process as well as support people, etc.? 2. Of whom does the team consist at the University of Texas/Austin under Jim Hoffman’s direction; was this expertise not available at UTSA? 3. What role did the Institute of International Development at UTSA play in the two projects? 4. How was the collaborative process of creating materials developed? Was this led by Jim Hoffman? Who was involved in development? What was the process of creation, review, revision, etc.? How were the books field tested? Where? Who? What documentation/reports? Who developed the Teacher’s Guides? What process was undertaken for these? 5. How was the decision made about where to create the e-learning labs? Who made this decision? 6. What was problematic about obtaining copyrights for materials? 7. What is the Alliance for the International Study of Texts and Literacy? How are they involved? Get report of meeting April 6-28, 2010. 8. Of what does your exit plan consist? Get a copy of exit plan. 9. What was the purpose of the CB workshop for local printers? Did any of those trained receive any/part of the printing contract? Are they capable of taking on these activities in the future to bring it to Malawi instead of Uniprint in South Africa? 10. How were the rules of procurement between UTSA and Malawi changed to adapt to Malawi requirements? 11. Who are the members of the PAC? What roles did they play? How often did they meet? What was their SOW? 12. How did you handle payment or topping off MIE/MOEST/PAC members? How much? What for? 13. How was training and then use of materials monitored? What instruments were developed? Who was monitored? 14. What was the agenda for the annual international conferences? Who was invited? What were the outcomes? 15. How different are the approaches in teaching literacy in Chichewa from English? How is the system working to introduce the Chichewa materials in grade 1 and then the English version in grade 2, etc.? 16. What problems were encountered in the delivery of books from printer in SA to kitting in Malawi to delivery to districts to delivery to schools to use by children? Discuss the value chain. 17. Who determined pilot schools? How? How many? Where? Who/how were demonstration schools determined? And six TTCs? 18. Capacity Building Report – get a copy. 19. What changes in Teachers Guide to require a second edition? Any lessons learned here? 20. Report on National Roll Out 21. What were all the CB activities regarding how to do research? Who attended? Where? What have been the results? How will research continue to be conducted as part of the exit strategy/sustainability? 22. What changes in training teachers between read aloud and guided reading? What was the problem? What lessons learned? 23. What is the USAID funded MTPDS project? How does TLMP work with them? 24. What are “Efficacy” schools (30)? Where are they? (all three regions) M&Edone there – any reports? 25. Continuous professional development manual – get a copy. 26. What parameters for the production of the video at Mponda school in Zomba? How is this to be used? Do schools have the equipment to play? Who will facilitate? What monitoring afterwards? 27. Zomba Pilot Study. 28. Monitoring Tool for Village Heads and SMCs. TLMP Evaluation Report Volume II – Malawi 142 29. How have the materials been institutionalized through incorporation in their use at pre-service TTCs? 30. What difference did it make to your work when Malawi revised its national primary curriculum? Did you have to revise to make them conform? 31. What skill shortfalls did you encounter in your Malawi partners? How did you address these (besides hiring outside consultants)? What capacities were built for the future? 32. How were the 14 core trainers selected? What criteria were used? Get Read Malawi Training Manual for Complementary Books. How were 9 training videos used? Did training differ for all the various groups? National monitoring team, Divisional SEMAs, core team, PEAs, TTC lecturers, head teachers, teachers. 33. What indicators did you use for monitoring all training and implementation? 34. Given the range of training that was provided, and the number and type of challenges that still remained, what would you do differently in terms of timing and training? 35. What kind of outcomes have been achieved through community mobilization activities? By the community members? By the SCMs and PTAs? Were the needs of illiterate parents addressed at all? 36. Where were reading centers established? What is their purpose? Who manages them? What resources are there? How are they different from community/school libraries? 37. From what you have learned about teachers in the Malawi system, do they shift jobs often? What are the reasons for this? GENERIC QUESTIONS POSED OF ALL MSIs TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did your university provide? What challenges emerged in your work with the MOE? How were they resolved? TLMP Evaluation Report Volume II – Malawi 143 • What process did you use to review and revise the materials? How were the materials, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? Project Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the Mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the project improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school about distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? TLMP Evaluation Report Volume II – Malawi 144 ANNEX A-2. Data Collection Instruments, Malawi Stakeholders In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the Mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the experience of TLMP leveraging or leading to UTSA working on other educational development activities in Malawi or elsewhere in the future? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbooks and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in this country? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • Would you say that the model for TLM creation developed by UTSA (working with teachers to write the materials) can be utilized in other areas? Or was this a process you feel can be used for only supplementary or complementary materials? How would it need to be improved to be useful in other contexts? • How satisfied were you with your relationships with UTSA and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOEST/MIE Administrators • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOEST/MIE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOEST/MIE provide? How frequently? • How do PEAs assess teachers? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • What will the MOEST/MIE do to continue the production of TLMs now that the project has ended? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLMs in the future? What would prevent this from happening? TLMP Evaluation Report Volume II – Malawi 145 • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? • How was TLMP monitored by the MOEST/MIE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOEST/MIE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with UTSA? What could be improved? 3) Materials Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? How many males? Females? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How often did the two teams meet? What were the results of these meetings? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? How did you include gender concerns? How did you include HIV/AIDs-related concerns? How did you include cultural/contextual relevance? What other cross-cutting themes did you include? • How was the decision made (and who made it) that materials would be produced only in Chichewa and English (rather than multiple mother-tongue languages)? How was the decision made (and who made it) that certain materials should only be produced in Chichewa while others were produced in both Chichewa and English? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? • How satisfied are you with the collaborative production process between yourselves and UTSA? What worked well/did not work well? What would you change to improve the process? • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Zomba) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? What did the MOEST/MIE do (specify unit)? What did the project do? What challenges emerged in your work with the MOEST/MIE? How were they resolved? How did you liaise with all stakeholders? TLMP Evaluation Report Volume II – Malawi 146 • In implementing the project, what role did the MOEST/MIE play (specify unit)? What roles did your office play? What guidelines did the MOEST/MIE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? • What other teacher training is provided by the MOEST/MIE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes • What was the TLMP management structure in Malawi? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Malawi PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? PEAs? The MOEST/MIE itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of your work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with the UTSA? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What is your opinion of the management training that was provided to you by UTSA? What key lessons did you learn that you applied to the management of your operations? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? TLMP Evaluation Report Volume II – Malawi 147 • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many were males? Females? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? • How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparities in enrollment/attendance? If so, what have the results been? • What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? Are the TLMs in the appropriate language(s) for this school? If not, how should they be changed? • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long? TLMP Evaluation Report Volume II – Malawi 148 • How many of each textbook did you request for the school? How did you calculate this number to include entering KG1 students? How many of each textbook did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOEST/MIE provide? How often? Are those who attend expected to pass on what they have learned to their colleagues? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher (certificate, diploma, degree)? • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching in English? In Chichewa? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLMs? Please explain. • How many students do you have in your classes? Males? Females? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach reading? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP supplementary readers and TLMs do you have to teach? What TLMs do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive training on the use of TLMs? How long did it last? Did someone from TLMP or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • When did you receive the TLMP materials for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • From your point of view, how satisfied are you with the way gender issues are treated in the books? What can be done to improve this? From your point of view, how satisfied are you TLMP Evaluation Report Volume II – Malawi 149 with the way HIV/AIDS issues are treated in the books? What can be done to improve this? From your point of view, how satisfied are you with the way cultural/ contextual issues are included in the books? What can be done to improve this? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs materials? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this? • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1- 4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? Have you noticed any difference between males and females? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? • Why do you think you were chosen to be a trainer? • On a scale of 1-4, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? TLMP Evaluation Report Volume II – Malawi 150 • How familiar were you with the TLMP materials before you delivered the training? What materials were you provided to be a trainer? What materials did you provide the trainees? How confident were you after the TOT that you could teacher others in how to use the TLMs? What else did you need? • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any added compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs School: ______________________________ Village/Town/City: ____________________ Country: _________________________ Type of School: ___Primary ___Jr. Sec ___Sec Teacher Sex: _____ M _____F Grade Level: __________________________ No. of Students: ___ M ___ F ___Total No. of repeaters/overage learners: ___M ___ F ___Total Languages spoken in this community: ___________________________________________ Home Language/Mother Tongue of Learners in the Class: ____________________________ Home Language/Mother Tongue of Teacher: _____________________________________ No. of books: _________ No. & type of learning materials: _________________________ Indicator Observ ed Not Observed Teaching Using TLMs Instruction 2. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP. 7. The teacher can read and explain TLM content to the students. 8. The teacher uses TLMs throughout the class period. 9. Students use TLMs throughout the class period. 10. Students can read and understand the subject matter in the TLMs. 11. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment). TLMP Evaluation Report Volume II – Malawi 151 Indicator Observ ed Not Observed 12. The teacher makes sure learners make connections of TLMP content to daily lives. 13. The teacher gives and corrects homework using the TLMs. 14. The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson. 15. Students and teachers use mother tongue/English when asking and responding about TLMs (circle which language). TLMs/Artifact Inventory 16. Lesson objectives are written on the board in either local language or English. 17. Learning aids/materials are posted in the classroom (TLMP produced and others). 18. TLMs are locked up in the cupboard. 19. Word walls display key words in local languages and English. 20. To demonstrate language experience, sentences appear on the chalkboard or on a chart in the classroom. 21. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books). 22. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment. Comment_______________________________________________________________ _______________________________________________________________________ TLMP Evaluation Report Volume II – Malawi 152 9) Learners Reading Competency - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read a few sentences aloud, and then rank the performance in the following manner: 1) fluent; 2) little difficulty; 3) very haltingly; 4) can read only a few words; 5) cannot read at all. If a learner tries to read a word, identify what strategy he/she is using to read: 1) sound it out – phonics; 2) sight reading; 3) other strategy (TBD). Reading Comprehension - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read the passage silently and after they have finished, have learners explain what they have just read. In the case of KG and grade 1 where pre-reading text and workbooks have been developed, identify letters, ask what sound the letter makes, and then have the learner point out words that include this letter. Then have the child write the letter (or a word). Rank learners in the following manner: 1) instant recognition; 2) uses other cues to identify the letter; 3) does not associate the graphic presentation of a letter with the sound it makes; 4) cannot recognize a letter in a word; 5) cannot write a letter or a word. Use the following instrument for all grades: EVALUATION OF LEARNER READING COMPETENCY Note: Try to evaluate 12 learners per grade: 3 girls and 3 boys that the teacher has indicated are advanced learners, and 3 girls and 3 boys that the teacher has indicated are weak. School: __________________________ Village/Town/City: ________________________ Country: _________________________ Type of School: ___Primary ___Jr. Sec ___Sec Grade Level: ______________________ Class Subject Matter _____________________ Student Age: _____________ Sex: ____M ____F Repeater: ___ Yes ___No Home Language/Mother Tongue of Learner: ___________________________________ Language of Testing: _____________________________________________________ Title/Type of TLM used in Class: _____________________________________________________________________ Reading Aloud: Instructions: Identify 2 short passages (3 to 5 sentences) from one of the TLMs for the appropriate grade level and make a clean copy of each passage. [Please save passages to accompany evaluation results. They should preferably be cut from within stories or texts so that memorization can be ruled out. One passage should be easier than the other, so one can be taken from the beginning of the book and one from the middle, depending on how far into the school year they are.] Have the student read the more advanced passage aloud and note the results below. If s/he is successful, stop after this. If the advanced passage is too difficult, provide the easier passage and conduct the exercise again, noting the results. Reading Fluency Passage 1: ____________________ (Rate and comment) 1) Fluent _______________________________________________________________ TLMP Evaluation Report Volume II – Malawi 153 2) Little difficulty (specify words) _____________________________________________ 3) Very haltingly (specify words) ______________________________________________ 4) Can read only a few words (specify words) ____________________________________ 5) Cannot read at all ______________________________________________________ Passage 2: ____________________ 1) Fluent _______________________________________________________________ 2) Little difficulty (specify words) ______________________________________________ 3) Very haltingly (specify words) ______________________________________________ 4) Can read only a few words (specify words) ____________________________________ 5) Cannot read at all ______________________________________________________ Reading Comprehension/Silent Reading [Note that you will need the teacher or another local person to translate what each student says in lower grades.] Instructions: Identify two short passages (3-5 sentences) from one of the TLMs for the appropriate grade level and make a clean copy of each passage. [Please save passages to accompany evaluation results. These can come from the beginning of stories or texts so that the context is clear. As above, one passage should be easier than the other.] Have the student read the more advanced passage silently and then re-tell what the passage is about using his/her own words. Note the results and comment below. If s/he is successful, stop after this. If the advanced passage is too difficult, provide the easier passage and conduct the exercise again, noting the results. Ability of learner to paraphrase what s/he has read silently: 1) Can put into own words all that has been read without difficulty ____________________ 2) Can put into own words only selected points __________________________________ 3) Has difficulty putting passage into own words __________________________________ 4) Cannot put passage into own words _________________________________________ 10) Community Mobilizers/CRECCOM (Zomba) • What is the core business of your organization? How did you become involved in TLMP? • What did you do in communities to help people understand the importance of reading? Which strategies were the most effective? Do you believe that community mobilization was an appropriate activity for TLMP? Why? • What results did you achieve? What challenges did you face? How were these overcome? • What is your opinion of the supplementary reading TLMs produced by TLMP? What was the community’s/parents’ opinion? • What actions did community members take to demonstrate their value of reading had changed? (e.g., is there a demand for adult literacy classes; do parents read with/to their children? Etc.) • How satisfied were you with your relationship with UTSA? How could this relationship be improved? • If you could restart your involvement with TLMP, what would you do differently? TLMP Evaluation Report Volume II – Malawi 154 ANNEX B. Schedule of Stakeholder Interviews and Activities DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE LILONGWE 17/2 2:30 TLMP team at Wendal’s 18/2 8:00 Lilongwe Demonstration School Carolyn Majiga Head teacher 18/2 2:00 Lilongwe TTC Elick Kwanda. Goodson Jona Acting Director eLab 18/2 3:30 MOEST Meeting at Wendal’s Patrick Themu Residential Course Manager DTED 19/2 7:30 USAID Ramsey Sosola John Collins Chikondi Maleta Program Management Specialist Education Officer Program Management Specialist 19/2 10:30 MOEST Mrs. Chikondano Mussa Director of Secondary Education (formerly Deputy Director, Basic Education) 19/2 2:00 Ultinets Stuart Winga Head of Operations 19/2 3:30 Save the Children Lexon Ndalama Acting Senior Manager for Education 20/2 7:00 DEPART FOR ZOMBA ZOMBA 20/2 11:30 UTSA Project Coordinators Ms. Sellina Kanyerere Mr. Henri Chilora 20/2 3:00 CRECCOM Madalo Samati Director of Programs DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE 20/2 4:00 Illustrator -MIE Heath Kathawere 21/2 7:00 Schools Team 1 (Zomba) Mponda Primary school Dorothy Kalta Head teacher TLMP Evaluation Report Volume II – Malawi 155 Police Primary School M.M. Chagoma Head teacher 21/2 7:00 Schools Team 2 Domasi Demonstration School Domasi Government School James Kalomgonda Nehru Banda Head teacher Head teacher 21/2 2:30 Malawi Institute of Education (Zomba) Dr. William Susuwele￾Banda Executive Director 22/2 6:30 School Visits (Zomba, St. Anthony’s Zone) St. Anthony Girl's Primary School St. Anthony Boy’s Primary School Sr. Catherine Bulla Louis Matekenya Gregory Kamwendo Head Teacher DHT HT 22/2 10:00 St. Anthony’s Zone TDC Ms. Mphatso Makhumula PEA 22/2 1:00 Zomba Rural Education Office Mac Owen Ligomeha Hamilton Hon Saiti Alifo Ussi Paul Chindamda PEAs 2:30 Writers at MIE Foster Gama 23/2 8:30 TRAVEL TO BLANTYRE DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE BLANTYRE 10:30 Kriss Offset Sabri Gani Ashraf Patel MacDonld Lambole General Manager Finance Controller Sales and Marketing Manager 25/2 7:00 Schools Team 1 (Chiradzulu District) Malavi Primary School St. Theresa Primary School Maston Angiston Sitima Patron Alexdandef Makwinja Head teacher Head teacher 25/2 11:30 Montford College for the Blind Hastings Magombo Deputy Principal and Head Visual Impairment Department 25/2 2:30 MOEST Alex Chipungu Logistics Officer, Supplies TLMP Evaluation Report Volume II – Malawi 156 25/2 2:30 Blantyre TTC Mr. Magelegele Principal, Blantyre TTC 26/2 7:00 Schools Team 1 (Chiradzulu District) PIM Primary School Mafe Primary School C.T.K. Mbewe Raison Chidziwe Head teacher Head teacher 26/2 7:00 Schools Team 2 (Blantyre Urban District) South Lunzu Primary School Namilangu Primary School 26/2 2:30 Illustrators - Nation Newspaper James Kazemba Ralph Mwara Illustrators 2:30 PEAs at Ryalls Hotel Monica Kainja Luke Mamboya PEA – Ndirande Zone, BT Urban PEA – Bangwe Zone￾BT Urban 27/2 9:00 DEPART FOR LILONGWE LILONGWE 28/2 8:30 Team Meeting at Crossroads 28/2 Write Findings & PPT for USAID Debriefing Arrange meetings with UNICEF (Panjee 0888-384194 28/2 2:00 MoEST Mr. R. Agabu Director, DIAS 1/3 8:00 Debriefing at USAID 1/3 Finalization of work with team 2/3 10:00 Tom and Nancy Leave TLMP Evaluation Report Volume II - Malawi 157 ANNEX C. Reading Assessment Methodology Introduction The data collection instrument appearing in ANNEX B2 became very cumbersome; consequently the streamlined methodology appearing below was the one used. Schools and Children Selected District Zone School Blantyre Urban South Lunzu Namilango South Lunzu Chiradzulu Malavi Malavi St. Theresa PIM Mafe PIM Zomba Rural Mchengawede Domasi Demonstration Domasi Government St. Anthony St. Anthony Boys St. Anthony Girls Zomba Urban Mponda Mponda Within each school, two boys and two girls were selected from S1, S2 and S3 giving a total of 66 children. The children in S1 were tested in Chichewa and those in S2 and S3 were tested in both Chichewa and English using the Guided Readers that had recently been the subject of classroom lessons. Theoretically, the children should have been able to read the selected texts. Instruments and Testing One important shortcoming of the RM books, especially in S1, is that they did not include an introduction to letters, syllables, or words. Yet, as will be seen by the test results, almost no S1 children could actually read; thus, the evaluation team included in the testing process both letter and word recognition. Only then would the team ask the children to read a passage from the appropriate level TLMs. The team followed the same routine for all three grade levels. The test was administered to children individually, often with a teacher from the school present (especially when the team member was not from Malawi). After a brief introduction to each child, the test began by asking the child to point to the letter “M” anywhere on the page shown. Typically, the page inside the cover of the book was shown to the child where there were several examples of M as in Malawi. The second letter asked was S. In the English tests for S2 and S3, other letters were used such as d, g, k and u. The team member recorded whether the child could identify each letter and, if so, the number of attempts before actually identifying the correct letter. A correct answer was coded as a 1, and an incorrect answer as a 0. The second component of the test included the identification of words. The team member would say a word, and the child was then asked to point to that word on the page. In general, each child was asked TLMP Evaluation Report Volume II – Malawi 158 to identify two words. The team member tried to identify simple words, but even this was difficult in some of the books. Typical words in Chichewa included akuti (they say), agogo (grand pa/ma), anthus, and ndi (and). In English, words included fruit, village, fun, good, today, and brother. Depending on the number of words identified correctly, the child would receive a score from 1 (can identify all words without a problem) to a 5 (unable to read any words). Then, the child was asked to read one or two short passages selected at random within the reader. A reference to each passage was recorded by the team member. Each child would receive a score from 1- 5 according to the following scale: 1= fluent, 2 = little difficulty, 3 = very haltingly, 4 = can read only a few words, 5 = cannot read at all. Finally, if a child were successful in reading the passage, s/he was asked to read another passage silently and then describe in his or her own words what the passage said. The response for each child was coded as follows: 1 = can put into own words all that has been read without difficulty, 2 = can put into own words only selected parts, 3 = has difficulty putting passage into own words, or 4 = cannot put passage into own words. All of these exercises were completed in both Chichewa and English at all three grade levels, with the exception in S1 where only Chichewa was tested. During the analyses, the five scale responses were reduced to three. For example, the responses to reading a passage were coded as follows: the "fluent" and "little difficulty" were combined into a single score indicating ability to read. The middle category "very haltingly" was left as is. And the bottom two categories "can read only a few words" or "cannot read it all" were combined as “cannot read” measure. This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Eric Allemano (Team Leader) and Carol Benson, Babacar Diouf, and Alhousseynou Sy under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania Evaluation Report - Senegal Intentionally left blank TLMP Evaluation Report Volume II – Senegal 159 EVALUATION REPORT - SENEGAL EXECUTIVE SUMMARY Overview TLMP was implemented in Senegal between 2009 and 2012 under a Cooperative Agreement (CA) awarded by USAID’s Africa Bureau to Elizabeth City State University (ECSU) in North Carolina. TLMP was envisioned as a program to generate high quality textbooks and learning materials (TLM) developed in partnerships between Minority Serving Institutions (MSI) of higher education in the U.S. along with ministries of education (MOE) in several African countries. The overall aim of TLMP was to provide high-quality textbooks to African students. According to the TLMP White Paper produced by USAID, well over 25 million children have gained access to the textbooks produced, which consist of more than 500 titles in 13 languages. According to the CAs awarded to each MSI, the materials produced were to be fully aligned with national curricula, to focus on primary education, to be culturally relevant, and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. An initial inquiry by ECSU determined that in Senegal the focus of the TLMs should be on middle school and lower secondary, and on providing French-language support materials (manuels d’appoint) in mathematics and the sciences rather than on curriculum-based textbooks (manuels). According to the CA, the contents of the TLMs should reflect the official curriculum of Senegal as well as recommendations of the National Education Reform Committee and international trends in curricula. In addition, the TLMs were to be culturally sensitive, research-based, and supportive of gender equity and grade-age appropriateness. This evaluation of TLMP in Senegal is intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved in Senegal in relation to intended program targets as well as standardized and variable indicators (including quantitative and qualitative impacts on local materials production capacity, teaching and learning; • Determine if in-country institutions (with support from ECSU) were able to deliver services effectively in terms of coordinating material design, alignment, production and distribution; • Analyze usage of allocated USAID funding and overall cost effectiveness; • Highlight specific program accomplishments; • Explore the level of satisfaction on the part of the Senegalese Ministry of Education and educational stakeholders including teachers, parents and students; • Document challenges and lessons learned; • Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy (http://www.usaid.gov/) The evaluation was launched with interviews conducted in December 2012 by the IBTCI Project Director on campus at ECSU. Between January 27 and February 9, 2013, a four-person team comprised of two U.S. consultants and two Senegalese education specialists, all with expertise in TLMP-related areas, conducted in-country field work, which included a literature review, interviews with senior MOE officials, school principals and teachers, school visits and observations of teachers using TLMs, and discussions with students. The team worked as a whole for one week and was divided into two for the second week, with one sub-team concentrating on research in the North (Saint Louis, Louga and Thies) and the other in the South (Fatick, Kaolack and Kafrine). TLMP Evaluation Report Volume II – Senegal 160 TLMP in Senegal TLMP in Senegal was funded in two phases. Phase 1 covered October 1, 2005 to September 30, 2008, with the aim of producing at least one million textbooks and learning materials (TLMs). The original CA was funded in the amount of $3,000,000 (October 1, 2005) and was later amended to include an additional $2,000,000 (October 1, 2007), bringing the total amount for the three-year period to $5,000,000, and producing 1,634,000 books. With the exception of two teacher’s guides, all the TLMs were student materials. Phase 2, the expansion phase, which is the subject of this evaluation, ran from September 1, 2009 to December 31, 2012 with a final budget of $8,066,000 and produced 2,650,000 books (three million was the target in the CA). During Phase 2, ECSU facilitated the development of 20 titles for various grade levels and subjects while supporting the printing of workbooks for three subjects at the primary level. No teacher’s guides were produced during Phase 2. There was reportedly heavy involvement from ECSU during Phase 1, but during Phase 2 ECSU’s involvement was primarily in reviewing the materials that Senegalese educators had prepared. Based on the “lessons learned” from Phase 1, the operating principle for Phase 2 was to institutionalize the production of materials, both content and physical production, in Senegal to the greatest extent possible. Research Methods While this evaluation attempted to gain evidence of learning outcomes, i.e. impacts of the TLMs on student learning, the amount of time and need to cover rural and urban departments of many regions meant that the data is impressionistic rather than comprehensive in nature. However, the team visited schools in seven of 14 regions of the country, and was able to triangulate findings thus producing stronger conclusions. The team spoke with some parents, observed classes when possible, and looked for impacts on learning when books were in students’ hands, but most of the data speak to the processes of TLM development and distribution. The data presented constitute the opinions and experiences of Senegalese educators, teachers and learners who had contact with the TLMP processes or the outputs, the materials themselves. In sum, the fieldwork focused on assessing the processes of materials development, storage and distribution, and use of the TLMs by teachers and students. Successes and Challenges From the perspective of the ECSU community, participation in TLMP made a very significant contribution to strengthening the university’s capacity to address international activities, involving both faculty and students. Specific instances for which TLMP served as a catalyst include the establishment of a Global Education Center on campus, the strengthening of international student exchange programs with Senegal and China, and participation in the USAID-supported American-African Universities Collaborative of the African Presidential Center. The TLMs were produced in collaboration with the MOE and a team from the curriculum development unit. The project fit well with USAID’s strategic objectives, and the Mission played an important role by creating an advisory committee on which all stakeholders, including ECSU representatives, reviewed and approved all aspects of the program, and by actively participating in the development and review of the materials. The process exposed the Senegalese subject specialists and other participants to new ways to go from concept to draft product within a limited time frame, to working under pressure and to following strict production requirements. The TLMs were printed by two different printing companies in Dakar, whose capacity was improved. The printers were responsible for delivery of the TLMs to the regional education offices (Inspections de l’Académie), from which the packaged materials were distributed to the schools. TLMP Evaluation Report Volume II – Senegal 161 The achievements of the MOE in partnering with ECSU to implement TLMP include the improved ability of its staff and partner institutions to develop TLMs. Teams of Senegalese subject-matter specialists and review committees were created to develop and validate the draft TLMs before printing. However, little information is available about field testing of the materials. (We understand that some individual teachers used some chapters for some lessons, but the “validation” process seemed to involve discussions among specialists rather than piloting with students.) Finally, the MOE now has a supply of support TLMs of a reasonable quality in mathematics and science for middle and secondary schools where books were not previously available. Several outcomes of TLMP have the potential to boost student learning. One of these is the provision of resource materials (manuels d’appoint) in mathematics and science, most of which are generally useful for teaching the national curriculum, especially by skilled teachers. The TLMs provide a variety of texts and exercises which could potentially save teachers and students learning time, provided they no longer feel they need to copy everything onto and from the blackboard. The evaluation team found that the science and mathematics materials were produced in numbers that would significantly improve the pupil-tobook ratio in the lower and upper secondary schools where they were used in the classroom. In addition, TLMP helped the MOE by reproducing supplementary materials for the elementary level in 2009 and again in 2012, demonstrating flexibility in responding to the MOE’s need for TLMs. However, there were many challenges for the TLMP in Senegal. These have their basis in ECSU’s lack of experience when it came to TLM development, working with the Senegalese counterparts in French, dealing with the administrative constraints of USAID, consistently following the aims of the project, and following up on book development, distribution and measuring learning outcomes. First, the process of TLM development was very rushed, being based on U.S. fiscal years and having to observe (and miss) printing deadlines. TLMs went through a committee “validation” process but were not piloted with teachers or students. There were numerous inappropriate and non-African illustrations in the texts and highly limited gender awareness or inclusion of mandated cross-cutting themes. During Phase 2 (20092012) there were no teacher’s guides to familiarize teachers with how to use support materials to teach their subjects. Instead, according to available ECSU reports, a program of in-service training using a “cascade model” was set up to guide teachers in how to use the TLMs. There were, apparently, some trainers who were trained; however, the research team was unable to locate any independent documentation of this process or its results. There were obstacles in the distribution process and limited follow-up to correct them, since the process was not monitored. When TLMs did reach the schools, there was limited understanding of how books, which are free, were to be distributed, and many remained in storage or in libraries rather than in teachers’ and students’ hands. There were very positive exceptions to these findings in individual districts and schools, where well-trained and motivated teachers were able to put the TLMs to good use with students, and where students were able to use the TLMs either in the library or (having signed them out) at home. Finally, since copyrights are in the names of individual former faculty members of ECSU, there is the potential for complications and delays as to the MOE’s right to make revisions, reprint materials, etc. It should be noted, also, that various materials incorporated content, primarily illustrations, from other sources without permissions. Lessons Learned There are a number of project-specific lessons to be learned. One major set relates to M&E deficiencies. The TLMP project would have benefitted considerably had a logical framework analysis been done at the beginning to develop appropriate, achievable goals, objectives and specific indicators that could be assessed during implementation. Baseline data on pupil-to-textbook ratios should have been collected but were not. There should have been both monitoring and evaluation at each step to ensure that the original goals of the project were being adhered to, or that there was good reason to diverge from TLMP Evaluation Report Volume II – Senegal 162 them. There should have been follow-up activities (such as checking to see that the TLMs had actually reached the teachers and students) which would result in clear, budgeted actions. Another set of lessons relate to the materials themselves. The TLMs should have been properly piloted and criteria applied to check them for gender awareness, inclusion of cross-cutting themes, language level, grade/age level, and how usable they were for teachers. A third set relates to production. The process of procuring the printing of materials as separate procurements, rather than as a logical grouping, led to inefficiencies, resulting in higher costs, and this was exacerbated by production schedules missed by TLMP. However, a more critical set of “lessons learned” relates to a fundamental design flaw. The mere distribution of TLMs does not ensure that they will be used at all, let alone that they will be used the way they were intended, especially in the effective absence of teacher training. Teachers who have not themselves had access to TLMs do not automatically know how to teach with them. Early among the school visits the team started finding books stuck in storage rooms full of dust or in libraries still in boxes or lying pristine on shelves, and it is fair to assume that a major reason for this is lack of a level of comfort. The most capable teachers were clearly able to make use of the TLMs; the less competent were not. Teacher’s guides, along with TLM-related training, are necessary elements of the teaching￾learning process, not optional luxuries. Key Recommendations 1. Rigorous planning and both systemic and systematic M&E, with follow-up, is critical to assuring that projects run smoothly. 2. Improving the ability of Senegalese teachers and students to use TLMs. • Senegal’s MOE should provide guidance to inspectors and schools as to the status of TLMs and how they should be used. • The MOE should be supported in orienting school inspectors so that they can follow up in all school districts to ensure that students have use of the TLMs. In Annex G we provide a set of clarifications that the MOE could use to communicate with its field staff. • USAID should support the development of a set of charts showing how the TLMs build the subject- and grade-specific competencies of the national curriculum. • Teachers need to get the skills they need in using TLMs to promote learning. The TLMs, along with the set of charts linking them to the curriculum, should be used in this teacher professional development. • In-service training could have given a temporary boost to help teachers use the new materials. However, this is still challenging in countries like Senegal which, due to rapidly expanding school systems, must hire new, often inadequately trained teachers every year. There is also attrition as teachers either retire or leave the profession. 3. Development of the materials • For any project involving the development of educational materials, teachers with typical capacity should be involved, along with specialists in the development of draft TLMs, contributing to ownership and lending realism to the task. • Materials development specialists should have been engaged so that important steps in TLM development would be followed. • TLMs should be submitted to analysis using criteria for gender awareness, languages and levels of language, content and sequence, cross-cutting themes, and other criteria important to the target users. • Piloting with classroom teachers and students should be planned and budgeted. Teacher orientation/training should be planned and budgeted. Teacher’s guides should be developed for future TLMs that are designed for the actual audience. The research team found that Senegalese teachers who were not well trained had difficulty in using the TLMs. TLMP Evaluation Report Volume II – Senegal 163 4. Other • USAID and/or the MOE should encourage the use of procurement mechanisms that are more efficient and more cost efficient than ordering the production of each TLM as an individual procurement. One better approach, for example, would be to group TLMs with similar size print runs together. • As is commonly the case in sub-Saharan Africa, there were snags in getting materials from district offices to schools. Rather than hoping that principals will figure out their own ways to get TLMs, the MOE should strive to negotiate with other agencies which have vehicles, such as the police, to assist with distribution. • Insofar as copyrights that were assigned to ECSU faculty members, the right of the MOE to revise and/or reprint TLMs should be clarified. • TLM developers should obtain the rights to use intellectual property, such as illustrations, that belongs to others. (Preferable would be for Senegalese developers to develop their own.) 1. EVALUATION PURPOSE, QUESTIONS AND OBJECTIVES The Background to TLMP The overall aim of the TLMP was to provide high-quality textbooks to African students. According to the TLMP White Paper produced by USAID, well over 25 million children have gained access to the textbooks produced, which consist of more than 500 titles in 13 languages. The Cooperative Agreements (CA) awarded to each MSI state that the materials produced are to be fully aligned with national curricula, to focus on primary education, to be culturally relevant and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. The extension of TLMP between 2009 and 2012 built on the President’s African Education Initiative (AEI) launched by USAID’s Africa Bureau in 2005. It was designed to contribute directly to USAID’s effort in the development and distribution of learning materials. A related goal was to improve the management capacity of sub-Saharan African (SSA) host country partners in the education sector. CAs were awarded to Elizabeth City State University (ECSU) and four other Minority-Serving Institutions (MSI), each of which was responsible for managing and implementing the TLMP in a specific country – for ECSU, to continue work in Senegal – and with achieving specific outputs of TLM production. (The University of Texas, San Antonio was awarded separate CAs to finish work in South Africa and to replicate it in Malawi.) These materials were to be developed and/or adapted in partnership with the host partner country’s Ministry of Education (MOE) and other relevant local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S. MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). Originally a program to develop TLMs for primary schools, ECSU’s initial needs assessment determined that MOE priorities in Senegal were for French language middle and secondary school TLMs in history, geography, mathematics and science. This notwithstanding, in the AEI phase a total of 1.63 million TLMP Evaluation Report Volume II – Senegal 164 primary school textbooks was produced for the MOE before 2009. While most were reprints of existing books, the project produced a French grammar book and two teacher’s guides for environmental science, the latter using translated materials. The project also helped the MOE by reprinting several cahiers d’intégration, which were workbooks produced by a different project that were designed for use in a single school year and had to be reprinted every year. Evaluation Objectives This Performance Evaluation is intended to satisfy the following objectives with respect to the 20092012 period of the implementation of the Textbooks and Learning Materials Program (TLMP): • Validate stated program goals and impacts; • Assess the results achieved in Senegal in relation to intended program targets as well as standardized and variable indicators (including quantitative and qualitative impacts on local materials production capacity, teaching and learning; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Analyze usage of allocated USAID funding and overall cost effectiveness; • Highlight specific program accomplishments; • Explore the level of satisfaction on the part of the Senegalese Ministry of Education and educational stakeholders including teachers, parents and students; • Document challenges and lessons learned; • Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy (www.usaid.gov). The overall purpose of this evaluation provides USAID/Africa Bureau and USAID/Senegal with answers to these questions: • What documentation can be provided on the impact, if any, of TLMP on the way that students learn and teachers teach in Senegal? • What conditions have facilitated or constrained the impact of TLMP in Senegal? • Is there evidence that successes and lessons learned during the implementation of TLMP in Senegal have been institutionalized or incorporated into national education sector activities? If so, by which entities? This Performance Evaluation also assesses the impact of TLMP on the U.S. Minority Serving Institutions as well as on host country partners. 2. PROGRAM BACKGROUND ECSU Responsibilities ECSU’s vision for TLMP was not only to increase the number of textbooks to which students have access, but also to enhance the teaching and learning of mathematics and the sciences, particularly at the middle and lower secondary school levels, by integrating the most effective instructional methods into books and teacher’s guides. However, the decision was made to develop support materials (manuels d’appoint) rather than curriculum-based textbooks (manuels), because of the length of time and the bureaucratic obstacles impeding textbook approval in Senegal. Only two teacher’s guides were produced, and they were designed to accompany primary school textbooks on science and the environment. TLMP Evaluation Report Volume II – Senegal 165 TLMP in Senegal was funded in two parts. The first covered October 1, 2005 to September 30, 2008, with the aim of producing at least one million textbooks and learning materials (TLM). The original CA was funded in the amount of $3,000,000 (October 1, 2005) and was later amended to include an additional $2,000,000 (October 1, 2007), bringing the total amount for the three-year period to $5,000,000, and producing 1,634,000 million books. The second part ran from September 1, 2009 to December 31, 2012, with a final budget of $8,066,000, and produced approximately 2,650,000 books. During this part, ECSU facilitated the development of 20 titles for various grade levels and subjects, while supporting the re-printing of workbooks No teacher’s manuals were produced in Phase 2. There was heavy involvement from ECSU during the first part, but during the extension ECSU’s involvement was primarily in reviewing the materials that Senegalese educators had prepared. Based on the “lessons learned” from part 1, the operating principle for the second was to institutionalize the production of materials, both content and physical production, in Senegal to the greatest extent possible. The focus of the TLMs developed by TLMP in Senegal was on middle and lower secondary school, and on providing support materials (manuels d’appui). The reason cited for focusing on post-primary education was that the curriculum of the primary grades was undergoing revision at the time of the project. While ECSU helped develop a few TLMs for the elementary level (three titles), the bulk of the elementary materials had already been developed by the MOE and were simply reprinted by the TLMP project. ECSU affirmed in the CA that the contents of the TLMs should reflect the official curriculum of Senegal as well as recommendations of the National Education Reform Committee and international trends in curricula. In addition, the TLMs were supposed to be culturally sensitive, research-based, and supportive of gender equity and grade-age appropriateness. TLMP required significant USAID/Senegal involvement to bridge the cultural and linguistic gaps between ECSU and Senegalese government education officials. To this end, a Program Advisory Committee (PAC) was set up in Senegal.6 The role of this committee was to guide TLMP and ensure that potentially helpful entities and individuals were aware of and participated (as needed) in the project. The specific functions of the PAC were the following: 1. To provide quality control for program management; 2. To ensure that the best-qualified national specialists were contracted in the areas of curriculum, writing, illustrating and graphic production; 3. To ensure cost-effective choices of strategy, administration and logistics in the TLMP; 4. To assist in establishing vetting and revision procedures for the TLM drafts before printing; 5. To set standards of durability and shelf life of the TLMs in light of local conditions and costs. MOE Context and the Educational System Senegal has an educational system largely based on the French colonial model. Although there is experimentation with the use of mother tongues in formal education, French remains the major language of instruction at all levels of the educational system. Since 1996, educational governance has been delegated to the 14 administrative regions of the country, which have considerable legal responsibilities and financial autonomy. Communes and communautés rurales have jurisdiction over the management of basic education services, literacy training, vocational training and promotion of national languages. Central services, in their reduced role, concentrate on 6 PAC members were representatives of the MOE including the Minister or his/her delegate, directors of Elementary and Middle/Secondary Education Departments, the IGEN director, president of the national PTA, and other management and communication officers. TLMP Evaluation Report Volume II – Senegal 166 policy design, monitoring and evaluation, production of teaching and learning materials and coordination of the various levels of the education system. However, few local governments adequately fulfill their legal obligations because the budgets allotted are insufficient in relation to their needs. It is very often parent-teacher associations (PTAs) that pay school water, electricity and telephone bills. • The Inspections d’Académie (IAs), the regional inspectorates responsible for implementing national education policy, are in charge of supervising the implementing national policy on curriculum, teacher training and other functions. However, they do not have the tools and human resources needed to fulfill their mission, nor are they equipped to evaluate future needs for learning inputs. • The Inspections départementales (IDENs) play a strategic role in policy implementation. They provide technical support and pedagogical facilitation for the schools in their department. However, considerable efforts are needed for the IDEN to properly fulfill these functions. Local government involvement in the development of education has significant gaps. According to a separate analysis of the education system done by a team involving our two national consultants, local government resources are not allocated on the basis of objective criteria and it is difficult to obtain precise figures on the total amount of resources stipulated in the budget or effectively made available to the education system. The analysis found that among the many weak points in the decentralized management of the education system is a lack of transparency, administrative red tape, inefficient resource management, and problems of access and poor quality of services. This was reflected in the confusion over the distribution of the TLMs that the printers delivered to the IAs. The IDEN were sometimes unable to find the resources to take the cartons of books to the schools. Similarly, according to some school principals interviewed, the MOE had not disseminated clear guidelines about the status and intention of the TLMs. Still, progress has been made in the financing of schools by local government: their contribution is larger than the subsidies they receive from the state in the form of endowment funds and assistance.7 Development of new textbooks or choice of foreign textbooks is still a centralized process in Senegal. The Ministry of Education relies on the Institut National d’Etude et d’Action pour le Développement de l’Education (National Institute for Applied Research in the Development of Education) or INEADE, a special unit to develop textbooks that conform to the approved curriculum. This unit also approves commercially-available textbooks that are marketed in Senegal, primarily by French publishers. Some of the experts in this unit, most of whom work as education professors at the national university, have been writing and editing texts since 1984.8 3. EVALUATION METHODS AND LIMITATIONS Methods The team used a “mixed-methods” methodology that emphasized qualitative approaches. It consisted of review of project documents (including a number of the TLMs); interviews with senior officials of ECSU and the MOE, Senegalese and U.S. TLMP implementing staff and administrators, materials developers, MOE officials at district and regional level, particularly those responsible for training and pedagogical 7 Senegal. Effective delivery of education services: A review by AfriMAP and the Open Society Initiative for West Africa by Hady Guèye, Lamine Kane, Babacar Diop and Amadou Abdoul Sy (Nov. 2010). 8 An Outcomes and Impacts Evaluation of the President’s Africa Education Initiative. Country Study Report: Senegal. Aguirre Division of JBS International (2009). TLMP Evaluation Report Volume II – Senegal 167 support in each region visited, and the printers; classroom observations with discussions with principals, teachers, and students; and assessments of student performance. Research Conducted at ECSU The evaluation project director conducted a site visit to ECSU between December 4 and December 6, 2012. This visit was organized by Dr. Abdou Sene, who succeeded Dr. Johnny Houston as TLMP project director in December 2009, which included group and individual meetings with University Chancellor Willie Gilchrist and top university academic officials, Dr. Sene, Dr. Houston, and other TLMP staff, plus other university administrators. In addition to review of project documents and some materials and discussion of the mechanics of project implementation, a key focus of the sessions was on the impact of TLMP on the campus community. These are discussed in section 4.1 below. Research Activities in Senegal The in-country evaluation activities were conducted by Eric Allemano (Team Leader), Carol Benson, Alhousseynou Sy, and Babacar Diouf between January 27 and February 9, 2013. The field visits included interviews at USAID/Senegal, MOE officials at various levels, school principals and teachers, and classroom observations and quasi-focus group discussions with students and some collective interviews with teachers and trainers. The team also had the opportunity to talk with some parents. Sites were selected to reflect both capital city and distant settings with different demographics. In addition to visits to schools in the greater Dakar area, two-person sub-teams, comprised of one U.S. specialist and one Senegalese specialist, visited sites to the north (Saint-Louis, Louga and Thies) and to the southeast (Kaffrine, Kaolack and Fattick). Please see the map of Senegal above for site locations. School sites were selected to reflect capital city, peri-urban and remote rural settings with different demographics. (See ANNEX D – Schedule of Stakeholder Interviews and Activities) As a result of the focus on creating middle and lower secondary school TLMs as support materials (manuels d’appui) in mathematics and sciences rather than on curriculum-based textbooks (manuels), the questions posed at the MOE and in schools had to be adapted. While there were materials developed for elementary (three titles), the bulk of the elementary materials was developed by the MOE and simply printed by TLMP. This meant that the team needed to focus on middle school visits, while trying to check on the other two levels. The team’s initial interviews with stakeholders disclosed that the project had experienced challenges in the delivery and acceptance of materials, resulting in the desirability of exploring this challenge more deeply than initially envisioned. In order to obtain perspectives of the reach of TLMP to schools at some distance from the capital, a day of fieldwork was added to the original plan, enabling each team to work three full days outside Dakar and to see schools in seven of Senegal’s 14 regions. Nonetheless time for observation of classes in session was still limited, a factor that was complicated by a teachers’ strike at one point, so as an alternative the team opted to talk to available teachers and students and to ask students to show us the books they were using. In sum, the fieldwork focused on the processes of materials development, distribution, teacher training and use of the TLMs. The team did talk to some parents, observed classes when possible and looked for impacts on learning when books were in students’ hands, but most of the data say more about the processes of TLMP than any impacts, because of a number of challenges. For a description of the stakeholders targeted in the field and the themes on which they were questioned, see ANNEX B – Data Collection Instruments. TLMP Evaluation Report Volume II – Senegal 168 Limitations of the Study One inherent limitation was the time available, given the great desirability of visiting schools located away from Dakar. While the task order anticipated that a country visit would last for ten calendar days, the team spent 11 working days in Senegal, a duration which IBTCI deemed was needful in order to accommodate travel to different parts of the country. Complicating this research was a teachers’ strike, limiting our access to schools, teachers and students in the north-east, and classroom observation time was constrained. Because of time and data limitations, the use of formal quantitative methodologies was not feasible. For example, because of multiple confounding factors it would not have been possible to credibly come to any rigorous conclusions as to the effect of TLMP materials on learning, given the large number of titles and grades covered. The team does, however, report on the opinions and experiences of Senegalese educators. The TLMs developed in 2012 had not all arrived at the schools by the time of the in-country activities. This caused us to redirect our methodology somewhat to focus on questions that would be central to the project and processes as implemented in Senegal. 4. FINDINGS AND CONCLUSIONS ECSU Achievements and Challenges According to the Chancellor of ECSU and others at ECSU, participation in TLMP made a very significant contribution to strengthening ECSU’s capacity to address international activities involving both faculty and students. Specific campus instances for which TLMP served as a catalyst include: • The establishment of a Global Education Center equipped for teleconferencing, which serves as a campus focal point for a number of international activities. • Increasing internationalization of the curriculum and improving student and faculty awareness of international issues. • The establishment and strengthening of international student exchange programs with Senegal and China. • Funding of a scholarship for a student from Dakar. • The establishment of National Institutes of Health (NIH)-funded research linkages with Senegal, such as a study on hypertension and diabetes. • Participation in the USAID-supported American-African Universities Collaborative of the African Presidential Center, hosted by Boston University. The Collaborative provides the opportunity for ECSU to engage with dialogue with presidents of other HBCUs and of African universities and also to participate in colloquia on major issues with former heads of state of African nations. ECSU was also able to develop and implement acceptable financial procedures that made it possible for the printers to address the “Catch 22” elements associated with their need to procure large quantities of paper, etc. in order to print the materials on the basis of which they would ultimately be paid. Finance office staff also pointed to the ability of ECSU to serve as a source of information for area businesses seeking to work with potential international suppliers or customers. Specific Achievements In evaluating the work of ECSU in the context of the work of all TLMP implementers, it is important to keep in mind that ECSU was the only MSI not to be conducting essentially all of its work in English. TLMP Evaluation Report Volume II – Senegal 169 Based on the perceived needs of Senegal’s MOE, TLMP’s work focused from the beginning on postprimary education; however, significant attention was given to the TLMP and Goal 1 emphasis on primary education. The project produced or reprinted 750,000 literacy development and/or reading books for grades 1 to 6 between 2009 and 2012, although most of these materials were not developed by TLMP itself. It is not known, however, how many of these books were actually delivered to schools and in use by teachers and students. Section 4.2 discusses the overall output of TLMs. Jointly learning from experience, the development process changed considerably since the inception of the project: Table 1: Transfer of Ownership of the TLM Writing Process Time period ECSU Senegalese Explanation/examples 2003 100-90% 0-10% Book from USA machine translated and sent to Senegal for interpretation and adaptation by national specialists. 2004-2006 80% 20% Senegalese team(s) went to USA to work with ECSU counterparts 2007-2008 50% 50% Some STTA trips 2009-2011 20% 80% Some STTA trips 2012 0-10% 90-100% Some STTA trips ECSU strived to be responsive to the needs of its Senegalese partners holistically, in ways beyond professional development. This is exemplified in part through support provided by TLMP to print and distribute elementary school learning materials developed by the Ministry, which was short of money. Another TLMP success was the reinforcement of capacity at two local printing companies, SIPS (Société Industrielle de Papeterie au Sénégal) and Polychrome, both based in Dakar. Further, both printing companies reported hiring additional local staff at least on a temporary basis during print runs. Printers rather than publishing companies were chosen to avoid overhead costs and the outsourcing of printing and via competitive bidding to eliminate the potential for favoritism towards publishers, who were known to have relationships with MOE/INEADE staff. Distribution of TLMs to the regions was done by the printing companies, reducing the risk that books would “escape” to the market, and enabling proactive IAs and IDENs to facilitate distribution to schools. Specific Challenges As mentioned above, TLMP staff at ECSU made some serious errors at the beginning of AEI, e.g. by starting with U.S. materials and using simple internet translation; however, in time, they realized that Senegalese authors should do the development and writing of the materials, and the preparation of manuscripts was delegated to Senegalese teams. Such a costly “lesson learned” could have been avoided if ECSU had been given advice by specialists in curriculum/materials development. Once the main work was moved to Senegal, which was a key aspect of the expansion phase, technical advice was still needed, particularly on appropriate methodology and activities to include in the materials. The advice was given by local subject-matter specialists. It is not clear that these materials actually did represent effective participatory pedagogy, as TLMP aimed to do, since few teachers’ manuals were developed and teachers who had been trained on the TLMs were not observed. It appears that most of the technical support given by ECSU took place during materials development and finalization workshops, but these did not appear to take place systematically, and the extent to which U.S. and Senegalese subject-matter specialists collaborated in the meantime, especially given language issues, or the extent to which U.S. curriculum specialists were at all involved, was by no means clear. TLMP Evaluation Report Volume II – Senegal 170 According to a group of author/adapters with whom we met, the entire process of TLM development was rushed. They wished they had been given more training. They also felt their opinions were not always listened to; for example, one said the content of a mathematics TLM was too difficult and was not adapted to the Senegalese context; another said he did not see the connection with the curriculum; and all of them said they were against combining two grade levels in one book, but their opinions were overruled in the name of cost-saving. (The team heard a number of complaints in the field about the materials that combined grades, which would indicate that the project staff should have listened more closely to the Senegalese authors.) Regarding the development and validation of the TLMs, some do not seem to take into account the abilities of average Senegalese teachers (many of whom are poorly trained) or the level of students outside of the better urban schools. According to interviews conducted by the IBTCI research team, the TLMs were pilot-tested for 15 days in schools in Thies and Mbour. It is unknown how many schools participated or whether a balance of urban and rural schools was followed. The education curriculum advisor in these two cities worked closely with designated master teachers to collect data from the pilot test to pass back to the Senegalese editors. The data were collected by questionnaire and observations. (Apparently, no testing of learning results was done, as the details of the validation methodology were not available to the research team.) The sketchy information available on the validation process indicates that the drafts were finalized by a committee which did include one or two teachers from the Dakar area, but not from rural areas or other parts of Senegal. In any event, this does not constitute adequate piloting, which would have picked up on some of the issues we encountered with inappropriate, foreign photos and illustrations. One of the most blatant examples, which was referred to as problematic at least twice when we were in the field, was a set of drawings of naked non-African female and male children and adults in a description of maturation. The ECSU team should have advised the Senegalese writers about these issues. Furthermore, some of the principals and teachers we interviewed found the TLMs “too difficult”. They found the level of the mathematics books, particularly Mathématiques: Raisonnement quantitative (2008), is challenging for 7th and 8th grade students. For example, it contains a chapter entitled Nombres rationnels Leçon 5: Les Binomiaux, le triangle de Pascal et la théorie binomiale (Rational Numbers Lesson 5: Binomials, Pascal’s triangle and binomial theory). While the book, which had its origins in the AEI phase, is wellwritten and laid out, it would take a creative and well-trained teacher to use this book effectively in an average Senegalese classroom, which raises questions about how the draft of the book was pilot tested. From the point of view of the original aims of TLMP as expressed in the CA, it is disappointing to see so much unrealized potential of ECSU to increase access to high-quality teaching and learning materials in Senegal. The materials produced were to be fully aligned with national curricula, to focus on primary education, to be culturally relevant and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. Regarding the level, as already noted, the decision was made to support middle school/lower secondary mathematics and sciences due to expressed needs and the fact that the primary curriculum was undergoing revision at the time. The printing of a few primary TLMs was positive in terms of project flexibility and support to national structures, but the quality of those materials was not high and there is no evidence that TLMP specialists had any input; they were merely printed by the project. The TLMP middle school/lower secondary support materials had as much alignment as possible to existing curricula, but these curricula were revised and at least some teachers felt that project TLMs were not completely consistent with what they were required to teach. Unfortunately there were no accompanying teachers’ guides developed for the math or science TLMs. While there was no requirement to run orientations or trainings related to the TLMs, ECSU did train some trainers and had TLMP Evaluation Report Volume II – Senegal 171 a plan for a “cascade” model. (According to the ECSU newsletter, 350 teachers from the regions where books were to be distributed, along with some local/regional MOE trainers, were trained as Master Trainers.) Overlooking the fact that “cascades” are not known to be realistic nor sustainable, we were not able to verify how many actual trainings were done. We asked everyone in the field about participation in training related to the TLMs, but were not able to meet anyone who had been trained. Thus it appears that even if trainers were trained, there was no follow-through with the plan. Regarding cultural relevance, we have already mentioned the images of non-Africans; in addition, there were American/European stereotypes evident in the TLMs. There is no discernible integration of gender awareness; in fact there are few people represented in the texts, and when there are they are mostly men—girls and women are almost never depicted, nor are there many young people shown with whom learners can identify. Finally, we found no integration of other cross-cutting themes like HIV/AIDS, other than in one book (which the evaluation team did not see in the field) which seems to have been written to include HIV. There were a number of inefficiencies in print production. Bids were invited for each print run/title, which was not the most efficient way to work with printers, since costs for paper and ink would be lower if large print runs could be guaranteed. Dependence on U.S. fiscal year budgeting caused many aspects of the process to be unnecessarily rushed. For example, at least one title was printed by both companies to enable the TLMs to be produced in a short time frame, but this caused quality differences and other inefficiencies. Further, failure to submit manuscripts by the deadline caused cost overruns. Polychrome said they even invited TLM author-editors to their offices to facilitate the editing of final manuscripts. The printers said that longer-term planning (over two to three years) and guarantees of larger print runs, e.g. for entire series, would have allowed them to invest in permanent staff, better equipment and storage of extra paper and materials. Although contrary to the wishes of Senegalese educators, as expressed to the team, in a number of cases the text for two years of a particular subject was printed as a single (but bulky) volume, largely for cost reasons. The evaluation team did not have the data to do a cost-benefit analysis of this project, but from our meetings with the two printing companies and our general knowledge, we have reason to believe that printing costs were higher than necessary, for a variety of reasons. For example, a separate bidding process was initiated for each title, which meant that neither printer could be confident that it would have a sufficient amount of work to hire and train personnel. Delays in finalizing the TLM manuscripts meant that the printers had to work overtime to produce the books they had agreed to print, increasing costs. The printers said that if they had had contracts for a number of titles to be produced over a certain time, and if they had received the manuscripts as promised, the printing process would have been much more cost-effective and efficient. One result of receiving rush orders and requests for a different format (e.g. the primary TLMs, which used smaller paper and cover sizes) was that special types of paper had to be ordered by air, thereby increasing unit costs. TLMP Output From the end of Phase 1 through the end of Phase 2, there were over 2.5 million TLMs published by the project, according to the project newsletter. Table 2 shows the distribution between levels of education: Table 2: Summary of TLM Production 2009 – 2012 YEAR Elementary School Middle School Lower Secondary Total 2009 500,000 500,000 TLMP Evaluation Report Volume II – Senegal 172 2010 600,000 250,000 850,000 2011 200,000 800,000 1,000,000 2012 300,000 300,000 Total 1,000,000 1,400,000 250,000 2,650,000 Most of the titles developed by the project were for middle school and lower secondary mathematics, geography/history and the sciences, as shown in Table 3: Table 3: Summary of TLM Titles Produced 2010 – 2011 English title Grade Year Copies Life & Earth Sciences (SVT) 9-10 2010 250,000 Physical Science 9-10 2010 150,000 Life & Earth Sciences (SVT) 7 2010 200,000 Mathematics 11 2010 125,000 Physical Science 11 2010 125,000 History & Geography 10 2011 200,000 Life & Earth Sciences 8 2011 250,000 Life Skills (gen) 2011 350,000 The project also produced one elementary social studies title (Discovering Our World) in 2009 and three titles for elementary French language arts, as shown in Table 4: Table 4: Summary of Elementary French Literacy TLMs Produced 2011 – 2012 English title Year Copies Learning French 2011 100,000 Language & Communication 2011 100,000 Between Sounds (French) 2012 150,000 In addition, TLMP helped the MOE by reproducing supplementary materials for the elementary level in 2009 and again in 2012, according to Table 5: Table 5: Summary of TLM Supplementary Titles for Elementary Grades in 2009 and 2012 No. of titles Year Copies per title Total 5 titles 2009 50,000 250,000 3 titles 2012 50,000 150,000 On the positive side, it may be seen that the science and mathematics materials were produced in numbers that would significantly improve the book-to-pupil ratio in the middle schools and lower secondary. Regarding the elementary materials, the team felt it was significant that TLMP could be flexible enough to respond to the MOE’s need for resources in printing elementary school materials. TLMP Evaluation Report Volume II – Senegal 173 The titles printed for the MOE were elementary language arts and teachers’ guides, which we found were well distributed and in use, as there was no question about ownership. Certain anomalies arise when considering the TLMs. First, the numbers of TLMs, when cross￾referenced, differ when totaled. The information comes from different newsletters and reports, and unfortunately could not be aligned or verified across sources. Second, the content and images in both the MOE materials (reproduced by the project) and the TLMP-developed materials are often inappropriate. Project Management A central role of ECSU TLMP staff was to serve as intermediaries between U.S. and Senegalese culture. With the retirement of the original TLMP Director, who continued to serve as a consultant in mathematics, ECSU recruited and hired a Senegalese education specialist resident in the area at the beginning of the expansion phase. An additional Senegalese specialist was later hired as Program Manager. These and other members of the ECSU team were able to effectively serve as bridges and skills transfer agents between the U.S. and Senegal, ECSU and the Ministry of Education, working in English and in French, Wolof, and other Senegalese languages. Project Implementation Materials Development Conceptualization and Development: The TLMs were produced in collaboration with the MOE and a team from the curriculum development unit. The project fit well with USAID’s strategic objectives, and the Mission played an important role by creating the Project Advisory Committee (PAC) on which all stakeholders, including ECSU representatives, reviewed and approved all aspects of the program. Although there was an attempt to use Senegalese examples, photographs and references, many of the final materials lacked cultural sensitivity, and many items appear to have been downloaded in haste from the Internet without citing a source or permission given. On the other hand, one of the later books, the 9th grade history and geography book lists the source of borrowed photographs and other illustrations. However, this book is an exception to the general practice of the project. Nevertheless, despite these flaws in execution, the process exposed the Senegalese subject specialists and other participants to new ways to go from concept to draft product within a limited time frame, to working under pressure and to following strict requirements. The usual method of working was from a distance, with an occasional meeting at a hotel to work over the weekend. Authors said they learned a great deal during that time, though it was challenging and they could have used more training. The TLMP Director said that they gained capacity to use laptops, which were bought for the task by the project. In the 2010 fiscal year the technical teams in Senegal asked to develop each book jointly with the technical team in the United States. They felt that this would improve their ability to design and develop textbooks independently. One result of the collaboration was a seven-day workshop in December 2009, held in Senegal and attended by four people from the ECSU-Senegal Program in Elizabeth City (the TLM Program Director; the former Program Director/Mathematics specialist, and specialists in Life and Earth Sciences (Sciences de la Vie et de la Terre, SVT), and Physical Sciences), 12 Senegalese educators for each of the three subjects, and eight members of the PAC. The resultant manuscripts were to be completed in a workshop held in March 2010. This presented the authors with a very tight timeframe to produce the materials, requiring a lot of rushed and last-minute work. Some books have a list of more than 25 Senegalese authors; we do not know how the writing and editing tasks were coordinated. At the week-long March 2010 Validation Workshop, which engaged over 100 individual teachers and educators from around the country to review the materials, the review unfortunately missed culturally inappropriate illustrations such as scarecrows and Thanksgiving turkeys. Had they piloted the materials TLMP Evaluation Report Volume II – Senegal 174 with teachers, they might also have discovered that teachers did not appreciate the merging of two school years in one book. This concern had been expressed by some authors earlier in the process, but was apparently overruled for reasons of cost-effectiveness. ANNEX F – Evaluation of Selected TLMs presents an analysis of eight TLMs prepared by the project. Printing: As increasing local printers/publishers capacity was one of the goals of TLMP, the TLMs were printed by two different printing companies, SIPS and Polychrome, both based in Dakar. Publishers were avoided due to high costs and the potential for favoritism, as publishers were known to have relationships with insiders at the MOE. Bids were invited for each print run, i.e. each title, an inefficient system. The printers were responsible for delivering the TLMs to the IAs at the regional levels; to do so, they called ahead to be sure that someone at the IAs would be prepared to sign for delivery. (This system seemed to have functioned to the extent that someone was required to receive the materials; however, there was criticism of this practice at the IA level since the usual procedure would have been to receive word of the delivery from the Ministry.) There was no systematic follow-up by ECSU or by the MOE. Distribution was not monitored, nor was the use of TLMs verified in the schools. There was no monitoring and evaluation component in the TLMP CA, and no indicators upon which to evaluate project implementation. The team did see some students and some teachers using TLMs effectively in classrooms in some research locations (e.g. Kaolack, Kaffrine and Fatick), but in other locations the TLMs were not in use for a variety of reasons. Training: According to an ECSU newsletter, 350 teachers from the regions where books were to be distributed, along with some local/regional MOE trainers, were trained as Master Trainers. A plan was made to “cascade” the training to other teachers at the school level; however, the team found no evidence that further training was ever undertaken, there was no budget item for this, no apparent follow-up by the project, and none of the teachers we met in the field had been trained to use the TLMs. Achievements and Challenges for the Senegalese MOE The MOE indicated that its staff and partner institutions improved their skills in TLM development, especially in content, choice of images and validation. Equally important is the creation of teams of Senegalese subject specialists and review committees to develop and validate TLMs. The MOE also now has a supply of support TLMs of a reasonable quality in mathematics and science for middle and secondary schools where books were not previously available. The Senegalese author-editors who participated in developing the TLMs said their own capacity was raised through the process. Authors gained capacity in materials development as well as computer skills, according to the TLMP director. Several outcomes of TLMP have the potential to boost student learning. One of these is the provision of resource materials (manuels d’appoint) in mathematics and science, most of which are generally useful for teaching the national curriculum, especially by teachers who have some training. Also important is the opportunity for students in some schools we visited to take books home for periods ranging from a week to the entire school year. Another asset is the numerous images (particularly the colored graphs, maps, illustrations and photos), which some teachers felt improve the quality and efficiency of teaching. In addition, the TLMs provide a variety of texts and exercises, which could potentially save teachers and students learning time – providing they no longer feel they need to copy everything onto and from the blackboard. There was one possible “halo” effect of TLMP seen at the Regional Inspectorate of Thies, where additional resources were mobilized to train users of the TLMs. Also of interest are TLMP Evaluation Report Volume II – Senegal 175 (unsubstantiated) reports of improved student learning in classes with trained teachers that use the TLMs. As for the challenges for the MOE, the project was operated largely through ad hoc advisory committees (separate from the PAC) and specially-hired consultants. There does not appear to have been a strategy to integrate the methodologies of the TLMs into the formal curriculum or textbook development bodies of the MOE; this integration could have increased collaboration and capacity for both U.S. and Senegalese institutions and improved the sustainability of the TLMP interventions. It is unfortunate that the MOE did not take steps to ensure that the TLMs were fully aligned with the curriculum and able to serve as required textbooks rather than support materials. It was apparently not only the primary curriculum that was a moving target; the middle school and secondary curricula were also in flux during TLMP. Some principals did not distribute the TLMs because of doubts that they conformed to the official curriculum. There was no evidence that MOE-supported pre-service or inservice teacher training integrated the TLMs into their methodology instruction. Delivery and distribution of the TLMs was highly problematic. Some IA and IDEN heads reportedly rejected the materials or objected to the unusual delivery procedures (e.g. representatives of the printing companies calling and requiring signatures without official word from the MOE). A number of interviewees commented independently of each other that the TLMs had “fallen on our heads,” meaning that they were not forewarned of the arrival of the materials nor of the fact that they would be responsible for facilitating delivery to the schools. There was reportedly no clear message from TLMP management or the MOE about how the TLMs were to be distributed “free” to teachers and students, which meant that the response ranged from keeping the TLMs sealed in their cartons, taking some out and selling them in the market (although they were marked Not For Sale), storing cartons or books unused in storage spaces, putting them in libraries for use by teachers and students in school, borrowing to use in the classroom, or borrowing to take home for shorter or longer periods. During school visits we saw many books stored in places where they would not be available to students. All of these different behaviors depended on the local understanding of the potential of the TLMs and the capacity of the receivers to effectively exploit the materials for student learning. Finally, there was limited follow-up of distribution to the IAs and no budget or plan was in place to ensure that TLMs were in fact distributed to the school level and to the student level. A Vignette on How Teachers Actually Used the TLMs It would be informative to describe how teachers in two 10th grade classes used the TLMs; we believe this to be broadly representative of the classes we observed. The first was a physics and chemistry lesson in a rural school; the other was a life sciences lesson in an urban school. • Neither teacher had been trained on the TLMs he was using. • The pedagogy was teacher-centered, particularly in the urban school, where a neophyte teacher was observed. • The teachers tended to fall back on traditional methods of writing texts and diagrams on the blackboard for students to copy, rather than engaging in a discussion of the material in the TLMs, even though there was at least one copy for every two students. • The more experienced teacher (in the rural school) asked students to come up to the board and write answers to certain questions. The other teacher wrote the answers himself after students had responded orally. • There was some good use of questioning techniques, such as asking “why” questions that required the students to refer to their TLMs. TLMP Evaluation Report Volume II – Senegal 176 • The newer teacher, in the urban school, gave a short dictation to give the students practice in listening and writing texts with vocabulary such as proteids vs. proteins. 5. LESSONS LEARNED Lessons for Project Planning and Implementation To be noted both by USAID and by implementers, TLMP would have benefitted considerably from following essential planning and implementation elements which are normally part of a development project. First, at the planning stage, there should have been a logical framework analysis to develop appropriate aims, goals and specific indicators that could be assessed during implementation. Then, as TLMP was rolled out in Senegal, there should have been both monitoring and evaluation at each step, and a mid-term review not limited to financial reports. This would have ensured that the original goals of the project were being adhered to, or that there was good reason to diverge from them. Likewise, it would have helped ECSU and the MOE as well as other stakeholders to learn from their experiences and adapt so that follow-up activities (such as checking to see that the TLMs had not only reached the schools but were indeed being used by teachers and learners) would result in clear, budgeted actions. Adequate planning and regular monitoring would have uncovered the flaws of the project, including but not limited to: failure to develop TLMs that could serve as designated textbooks rather than support materials; failure to thoroughly pilot-test the materials; and failure to consider the needs of most teachers, either through development of teachers’ guides or follow-up of training or both. These flaws resulted in what we see as a failure of TLMP in Senegal to reach its full potential. Another lesson learned is that the project budgeting, financing, and procurement processes were not efficient from a planning point of view, particularly in the development and piloting of materials, which when done responsibly takes well over one year. It was also inefficient for the planning of print runs, whose cost-efficiency (and adherence to the initial bid) depends on advance acquisition of paper, toner and other supplies and the meeting of deadlines from the book development side. Lessons for Implementers The most important lesson was that it was not a good idea to begin with materials for American students. This was exacerbated by the fact that the early TLMs, produced under AEI, were machine￾translated from English to French. This may have seemed efficient but introduced too many inaccuracies, resulting in hard work for the Senegalese specialists and misunderstandings between American and Senegalese collaborators. Further challenges inherent in this U.S.-to-Senegal direction were that images were often culture-bound (not relevant to Senegalese learners), that books were not fully aligned with the national curriculum, and that the levels were too high. There were also issues associated with quality control of the TLMs. Later materials were created by Senegalese specialists, who received limited feedback from individuals at ECSU, and there was no piloting; meanwhile, the validation processes used failed to capture and correct these flaws. The rushed process from conception to distribution did not facilitate thoughtful application of project goals (such as raising capacity among stakeholders) nor did it cultivate effective practices (such as developing criteria for textbook evaluation such as gender balance, culturally appropriate illustration, etc.). There are also intellectual property issues. Lessons for USAID There was a fundamental flaw in the development hypothesis – an assumption that production and distribution of textbooks by itself will make much of a difference. The mere distribution of TLMs does not ensure that they will be used at all, let alone that they will be used the way they were intended. Teachers, who have not had access to TLMs, whether as students TLMP Evaluation Report Volume II – Senegal 177 themselves or through training, do not automatically know how to teach with them. During the course of the field work, TLMs were found in storage rooms full of dust or in libraries still in boxes or lying pristine on shelves. The most capable teachers were clearly able to make use of the TLMs; the others were not. Teachers’ guides along with TLM-related trainings are necessary elements of the process, not optional luxuries. 6. RECOMMENDATIONS Materials Improvement The implementation of the expansion phase of TLMP in Senegal suffered from the lack of adequately rigorous and systematic planning processes as well as from lack of adequate monitoring and evaluation as the project progressed. • Ensuring high-quality design and lay-out. Materials development specialists should be engaged so that important steps in TLM development are all followed—including, e.g., piloting with students, checking for gender balance, stereotyping, etc. in content and illustrations. In addition to subject matter experts, graphic design assistance should be called upon for expertise in choosing or developing illustrations, photographs, maps and charts. • Realistic timelines. Frequently, there was a very tight timeframe within which materials were to be planned, developed, and finalized. This resulted in work that was rushed and not adequately reviewed plus costly delays in production. • Piloting and Usability. It was not clear how many TLMs actually were piloted in classrooms, or whether schools operating in different types of environments were used for the piloting. Teachers of representative ability should be involved as part of the materials development and piloting process in order to identify areas that could present challenges to comparable teachers. • Teacher’s guides. It is important to develop a teacher’s guide for each TLM, since not all teachers will be familiar with the TLMs through pre- or in-service training or with the pedagogical principles being promoted. In Senegal, as in other developing countries, many teachers have inadequate preservice training and some may not even be certified to local standards. Such teachers tend to fall back on the methods by which they were taught as students; moreover, in many countries the teachers may themselves have gone through much of their own schooling with little contact with textbooks. • Workbooks. Some materials were developed with the expectation that they would be accompanied by workbooks in which learners would write. This inherently makes the workbooks usable only once, with a need to replace them every year. Materials should be developed instead such that learner responses can be put into regular copybooks, which are relatively readily replaceable, including through the use of plain paper if need be. Systems Improvement • Ensure closer involvement of MOE bodies concerned with learning material development and approval. It appears that there was not a very close collaboration between INEAD and other MOE entities concerned with curriculum and textbooks. In fact, the choice of producing supporting manuels d’appoint was a way of circumventing the admittedly complex procedures of formally approving textbooks as part of the official, examinable curriculum. While this saved time, it appears that the MOE was not proactive in supporting the distribution process or in clarifying to its personnel what the purpose of the TLMs was for teachers and students. This would be avoided with better collaboration and greater willingness to follow the appropriate procedures. • Support the MOE in orienting school inspectors so that they can follow up in all school districts to ensure that students have use of the TLMs. Whatever information was TLMP Evaluation Report Volume II – Senegal 178 provided at the time of distribution to the school districts was inadequate to ensure that school directors and teachers understood the aim of getting materials into students’ hands. A clear directive from the MOE, along with a budget for orientations, will help get TLMs out of cartons, off shelves and into teachers’ and learners’ hands, to be used as resources in and outside of classrooms. As a reference, we have listed some points that could be clarified in ANNEX G: List of Clarifications on the TLMs for Possible MOE Distribution. • Support development of a chart that links the TLMs to the national curriculum. To provide added value to the TLMs as support materials, a chart showing teachers how to link the TLMs (as well as other available resource materials) to the competencies in the national curriculum would show that they are part of an officially sanctioned program of the education system. Such an activity could also raise capacity among subject specialists at the MOE to help teachers implement the latest curriculum reform. Doing this for each subject at each level of schooling would help educators to understand progression/sequencing of skills and would address the challenge some teachers expressed with having one TLM for more than one grade level. The chart for each subject and grade level could use a template with columns for: competencies in curriculum, learning indicators, pages in TLM, lesson name, and assessment. • Link TLMs to teacher professional development. Since the TLMs for middle and secondary school levels were all support materials rather than standard textbooks, since some of their content was considered challenging, and since not all teachers knew how to use books as resources in the first place, a link to teacher in-service and pre-service courses could bring the TLMs into better circulation and raise teacher capacity. Even if focused simply on using resource materials, training is justified for all teachers and could be made part of USAID/Senegal’s education strategy. • Support school-based management of resources. USAID might consider developing support programs at Senegalese middle and secondary schools where subject-based departments (such as mathematics) collaborate on the utilization of TLMs and other resources. USAID support to secondary school directors and department heads might help the MOE to create criteria and models for future school-based decision making. TLMP Evaluation Report Volume II – Senegal 179 ANNEX A. The Senegalese School System École elementaire Elementary school (6 years, 3 stages of 2 years each) Collège Middle school (4 years, equivalent to grades 7-10) Sécondaire, lycée Secondary school (2-3 years) Directeur d’école School director = Person responsible for an elementary school Principal Person responsible for a middle school Proviseur Person responsible for a secondary school Elementary school Middle school Secondary school Stage Name Grade Name Grade Name Grade First CI CP 1 2 6ème 5ème 7 8 2ème 1ère 11 12 Second CE1 CE2 3 4 4ème 3ème 9 10 Términal 13 Third CM1 CM2 5 6 MAP OF SENEGAL Field visit sites are marked in yellow. TLMP Evaluation Report Volume II – Senegal 180 ANNEX B. Data Collection Instruments The following is a description of the stakeholders the in-country team targeted in the field and the themes on which we questioned them: Regional education heads 1. Regional education inspectors (Inspecteurs d’Académie) • Their evaluation of the TLMP school books with regard to their alignment with the national curriculum • Management of the books (reception, storage and distribution) Departmental education heads 2. Departmental inspectors (Inspecteurs départementaux) - rural - urban • Their evaluation of the TLMP school books with regard to their alignment with the national curriculum • Management of the books (reception, storage and distribution) • Impact of the books on educational quality and student learning Subject specialists/Trainers 3. Specialized inspectors (Inspecteurs des spécialités) in language, math and science 4. Heads of regional training centers (Responsables des Pôles régionaux de formation) 5. Travelling pedagogical advisors (Conseillers pédagogiques itinérants) • Their evaluation of the TLMP school books with regard to their alignment with the national curriculum • Management of the books (reception, storage and distribution) • Impact of the books on educational quality and student learning • Participation in any training in the use of the books with regard to the reported cascade model • Their evaluation of the capacity/level of teachers using the TLMP books School heads 6. Middle school principals (Principaux de collèges) 7. Elementary school principals (Directeurs d’écoles) • Their evaluation of the TLMP school books with regard to their alignment with the national curriculum • Management of the books (reception, storage, care and lending policies) • The impact of the books on educational quality and student learning • Their evaluation of any training given to the teachers Teachers 8. Elementary school teachers 9. Middle school teachers 10. Secondary school teachers • Their evaluation of the TLMP school books with regard to their alignment with the national curriculum • Distribution, care and lending of the books • The impact of the books on educational quality and student learning • Their evaluation of any training given to the teachers Parents 11. PTA representatives 12. Individual parents The impact of the books on educational quality and student learning (if they have seen the books) Students 13. Elementary students  Their evaluation of the attractiveness of the books, the images and the 14. Middle school students readability of the texts 15. Secondary students The impact of the books on their learning and school results Others (added in the field) 16. IA storage heads (comptables magasiniers) 17. IDEN storage heads 18. School librarians Management of the books (reception, storage, distribution, care and lending policies) TLMP Evaluation Report Volume II – Senegal 181 ANNEX C. Master Data Collection Instruments This annex contains the revised master set of instruments used as a model for the TLMP evaluation. These questions were adjusted based on the circumstances of TLMP implementation for each country. In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the expertise developed in textbook production by the MOE being leveraged to obtain other, similar grants? What would prevent this from happening? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in Ethiopia? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • How satisfied were you with your relationships with Alabama A&M and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOE Administrators (triangulate with different administrators at each level) • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOE provide? How frequently? • How do the woreda inspectors assess teachers? Were they trained in the use of TLMs? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • How was the decision made to include supplementary/complementary readers in the TLMP? Who made the decision that these materials should be based in folk tales? How did the process of developing these materials differ from that used in developing the TLMs for grades 1- TLMP Evaluation Report Volume II – Senegal 182 4 English classes (textbooks)? Was there an advantage of one approach over the other? Please explain. • What will the MOE do to continue the production of TLMs now that the project has ended? Have any other donors been found to continue this activity? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLMs in the future? In the redevelopment and printing of the grades 6, 7, and 8 materials? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? How do you know? • How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with AAMU? How could they be improved? 3) Material Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How did the production process and personnel differ between the development of the textbooks and the development of the supplementary reading materials (folk tales)? Did one process have an advantage over the other? If so which one, and how were results different? Would you recommend one process over the other for future book production? • How often did the two US- and Ethiopia-based teams meet? What were the results of these meetings? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? What cross-cutting themes did you include? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? • How satisfied are you with the collaborative production process between yourselves and AAMU? What worked well/did not work well? What would you change to improve the process? TLMP Evaluation Report Volume II – Senegal 183 • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? (describe both the textbooks and the supplementary reading materials) What did the MOE do (specify unit)? What did the project do? What challenges emerged in your work with the MOE? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOE play (specify unit)? What roles did your office play? What guidelines did the MOE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Who was responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who provided the training? For how long? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Gender distribution of those who attended? Did TTC faculty attend? From which TTCs? Did university faculty attend? How many? From which universities? • What other in-service teacher training is provided by the MOE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes • What was the TLMP management structure in Ethiopia? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ethiopia PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials? TLMP Evaluation Report Volume II – Senegal 184 • What types of policy changes, if any, have you observed as a result of project work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with AAMU? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? • How satisfied were you with the relations established with AAMU? How could they be improved? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? • How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced? TLMP Evaluation Report Volume II – Senegal 185 • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparity in enrollment/attendance? If so, what have the results been? • What is the linguistic background of the learners at this school? What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? At what grade does English become the language of instruction? Do you believe your students are adequately prepared in English to learn entirely in English? What needs to be done to prepare students better? • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long? • How many of each textbook and workbook did you request for the school? How many of each text/work books did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on (cascade) what they have learned to their colleagues? Do they receive any incentive to do this? • How has the cluster center training and resource system enhanced the ability of teachers to be more learner-centered? How has the TLMP enhanced the adoption of learner￾centered teaching practices? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom and Head Teachers TLMP Evaluation Report Volume II – Senegal 186 • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree) • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching English? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLMs? Please explain. • How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP textbooks do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive the TLMs for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? • When did you receive training on the use of TLMs? How long did it last? Did someone from the TLMP project or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this? TLMP Evaluation Report Volume II – Senegal 187 • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 14, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the supplementary readers? 3) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? • Why do you think you were chosen to be a trainer? • On a 1-4 scale, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? • How familiar were you with the TLMs before you delivered the training? What materials were you provided to be a trainer? What materials did you provide to your trainees? How confident were you after the TOT that you could teach others in how to use the TLMs? What else did you need? • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs/SENEGAL Used in the Regions of Kaolack, Kaffrine and Fatik only School: _________________________ Village/Town/City: ________________________ Teacher Sex: _____ M _____F Grade Level: ______________________________ No. of Students: ______ M ______ F ______Total Languages spoken in this community: __________________________________________ Home Language/Mother Tongue of learners in the class: ___________________________ TLMP Evaluation Report Volume II – Senegal 188 Home Language/Mother Tongue of teacher: _____________________________________ No. of books: _______________ No. & type of learning materials: __________________ Indicator Observed Not Observed Other Teaching Using TLMs Instruction 1. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP 7. The teacher can read and explain TLM content to the students 8. Students can read and understand the subject matter in the TLMs. 9. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment). 10. The teacher gives and corrects homework using the TLMs. 11. The teacher shows evidence of having used the Teacher’s Guide in Indicator Observed Not Observed Other presenting the lesson 12. Students and teachers use mother tongue/English (French) when asking and responding about TLMs (circle which language) 13. Teacher demonstrates personal mastery of English TLMs/Artifact Inventory 14. Lesson objectives are written on the board in English TLMP Evaluation Report Volume II – Senegal 189 15. Learning aids/materials are posted in the classroom (TLMP produced and others). 16. TLMs are locked up in the cupboard. 17. Word walls display key words in English 18. Sentences appear on the chalkboard or on a chart 20. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books) 21. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment Comment____________________________________________________________________________ ____________________________________________________________________________________ TLMP Evaluation Report Volume II – Senegal 190 ANNEX D. Schedule of Stakeholder Interviews and Activities People Interviewed at Elizabeth City State University Name Title Date(s) e-mail Mr. Abdou Maty Sene TLMP Project Director 4-6 Dec. Dr. Willie Gilchrist Chancellor 5-6 Dec. Dr. Anthony Brown Vice-Chancellor 5 Dec Dr. Ali A. Khan Provost 5 Dec Dr. Johnny L. Houston Former TLMP Director; Consultant 5-6 Dec Dr. Chérif Seck Director of International Programs & TLMP Coordinator 5-6 Dec. Ms Eundene Faulks Asst. Director, Sponsored Programs 5 Dec. Mr. Joshua Lassiter Finance Dept. 5 Dec. People Interviewed in Senegal Title Name Date(s) Place Project leadership ECSU TLMP Project Manager Mr Abdou Maty SENE 28-30 Jan, other days Dakar USAID representatives Chief Education Officer Mr Pape SOW 28 Jan, Dakar Education Officer Ms Michelle CHEN 28 Jan, Dakar MEN (national level) MEN Secretary General Mr.Baba Ousseynou LY 28 Jan Dakar Former Secretary General Mr Mafakha TOURE 28 Jan Dakar Dir Elementary Ed Mr Abdou DIAO 28 Jan Dakar Dir Middle School Ed Mr Ibrahima NDOUR 28 Jan Dakar Coord DEMSG Mr Papa SENE 28 Jan Dakar Director of Inspection (IGEN) Mr Ndiogou FAYE 31 Jan Dakar Printing company representatives SIPS Technical Director Mr Maher GHANDOUR 29 Jan Dakar Polykrome Director General Mr Walid ATTIEH (and staff) 29 Jan Dakar Authors/validators Science expert Mr Cheikh Tidiane DIOP 29 Jan Dakar Math expert Mr Samba DABO 29 Jan Dakar Middle school math teacher Mr Oumar SY 29 Jan Dakar Parent/teacher association (national) President of FENAPES Mr Bakari BADIANE 31 Jan Dakar Secretary general of federation Mr Dam SEK 31 Jan Dakar Secretary of communications Mr Mohamed Diem SESAY 31 Jan Dakar Regional (IA) directors and staff (including inventory/storage chiefs) IA Dakar Whole team Academy inspector 1 Feb Pikine Assistant inspector (IA Pikine) Mme DIABY 1 Feb Pikine Mr Musa Diaba HASSAN 1 Feb Pikine IA Saint-Louis CB & AS TLMP Evaluation Report Volume II – Senegal 191 Academy inspector Mr Ndar FALL 4 Feb Saint-Louis Storage Chief Mr Abdou Karim KA 4 Feb 4 Feb Saint-Louis 4 Feb IA Kaolack EA & BD Academy inspector 4 Feb 4 Feb 4 Feb IA Louga CB & AS Academy inspector Mr Lamine SARR 5 Feb 5 Feb 5 Feb Louga IA Kaffrine EA & BD Academy inspector 5 Feb 5 Feb 5 Feb IA Thiès CB & AS Assistant academy inspector Mr Ibrahima BAR 6 Feb Thiès Comptable matières de l’IA Mr Mamadou KANE Thiès 6 Feb President of the APE (Association of students’ Mr Mama KANTE 6 Feb Thiès parents) of Thiès IA Fattick EA & BD Academy inspector 6 Feb 6 Feb 6 Feb IS, CPI, etc. (subject specialists/trainers) Saint-Louis region (they cover Louga as well) Science Inspector (CPI) Mr Déthie Chiendella FALL 4 Feb Saint-Louis Science Inspector (Regional training center) Mr Mactar FALL 4 Feb Saint-Louis Coordinator of the regional training pole (for middle & secondary ed) Mr Amadou NDIAYE 4 Feb Saint-Louis Thiès region Trainer (former CPI) in sciences (SVT) Mr Papa Birane THIANDOUM 6 Feb Thiès Departmental (IDEN) directors and staff (including inventory/storage chiefs) IDEN Saint-Louis Departmental inspector Mr Samba Laobé DIOP 4 Feb Saint-Louis TLMP Evaluation Report Volume II – Senegal 192 4 Feb Saint-Louis IDEN Departmental inspector 4 Feb 4 Feb 4 Feb IDEN Louga Departmental inspector Mr Amady KEBE 5 Feb Louga Comptable matières Mr Ismaïla TALL 5 Feb 5 Feb Louga IDEN 5 Feb 5 Feb 5 Feb IDEN Thiès Departmental inspector for Thiès city Mr Talla FAYE 6 Feb Comptable matières Mr Khaty DIAGNE 6 Feb 6 Feb IDEN 6 Feb 6 Feb 6 Feb Schools, teachers, students and parents In Saint-Louis IDEN Mpal Elementary School Director: Mr Sidate DIOP Teachers Mr Insab BADJI, Ms DIABO, Ms Bintu (bilingual) 4 Feb Mpal Ngom Middle School Principal: Mr Ngange GUÈYE Grade 9 class Grade 10 class 4 Feb 4 Feb Ngom In __ IDEN Inspector 4 Feb In Louga IDEN Director of Keur Serigne Bara Elementary School Ms Khady Ndiaye GNIGUE 5 Feb Louga CM1 teacher at KSB Elementary School Mr Mamadou DIAW 5 Feb Louga Students at KSB Elementary School CM1 students 5 Feb Louga Principal of Elhadji Djily Mbaye Middle School Mr Abdoulaye DIALLO 5 Feb Louga Teachers responsible for EDM school library Mr Babacar NDOYE Mr Alias Ismaïla SYLLA Louga TLMP Evaluation Report Volume II – Senegal 193 Teachers of EDM math & science (SVT) Mr Mamadou THIAM Mr Abdoul CISSE 5 Feb Louga Father of middle school student in 4ème Mr Saliou SEYE 5 Feb Louga In Kaffrine IDEN 5 Feb 5 Feb 5 Feb In Thiès IDEN Director of Saïb Ndoye Elementary School Mr Cheikh MBENGUE 6 Feb Thiès Gr 6 teacher at SN Elementary School Mr Modou NDIAYE 6 Feb Thiès Gr 5 teacher at SN Elementary School Ms THIAM 6 Feb Thiès Gr 5 and Gr 6 students at SN Elementary school One gr5 class and one gr6 class 6 Feb Thiès Principal Mamadou Diaw Middle School (formerly Château d’Eau) Mr Amary MBAYE 6 Feb Thiès Math & science teacher at MD Middle School Mr Paul DIOP 6 Feb Thiès Librarians at MD Middle School Ms Pallé DIOP (resp) Ms Didé Mbodji COULIBALY 6 Feb Thiès IDEN 6 Feb 6 Feb 6 Feb TLMP Evaluation Report Volume II – Senegal 194 Schedule of Activities in Senegal, 27 Jan – 9 Feb 2013 Date/Time Schedule Meeting with Accompanied by 26 Jan, Sat (evening) Eric: arrival in Dakar Hotel Ibis 27 Jan, Sun (morning) Carol: arrival in Dakar Hotel Ibis 16:30-18:00 First team meeting at Hotel Ibis Dakar: selfpresentations, introduction to evaluation, methods and schedule. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 28 Jan, Mon Activities (whole team: Eric, Carol, Babacar and Alhousseynou) 9:00-11:30 Meeting at USAID with the Chief Education Officer: Background to TLMP and this evaluation, logistics Pape SOW Abdou SENE Former Chief of Party TLMP 12:00-12:20 Meeting at MOE with Secretary General: We presented evaluation, he gave impressions. Secrétaire Général : Mr. M. Baba Ousseynou LY “ 12:35-13:35 Meeting at MOE with Director of Elementary Education: Impressions, materials published under TLMP Directeur de l’Education Elémentaire : Mr. Abdou DIAO “ 14:10-15:10 Meeting at MOE with Director of Middle School Education (Dir de l’Enseignement moyen sécondaire général/DEMSG/collège): Impressions, recommendations. Dir DEMSG Ibrahima NDOUR ndourahim@yahoo.fr & Coord de DEMSG Papa SENE “ (Lunch) 16 :30-17 :25 Meeting at home of retired Secretary General of MOE: His impressions of national textbook policy, recommendations. Mr. Mafakha TOURÉ “ 29 Jan, Tues Activities (whole team) 9:00-9:30 Team meeting at Hotel Ibis. Abdou SENE 10:25-11:20 Meeting at SIPS (Société Industrielle de Papetérie au Sénégal/printing company) with Technical Director: Impressions of working with TLMP, recommendations. Dir Technique Maher GHANDOUR “ 11:50-12:55 Meeting at Polykrome (printing company) with Director General and staff: Their experiences with TLMP and recommendations on printing, distribution. Dir Général Walid ATTIEH “ (Lunch with team and authors at Hotel Novotel) TLMP Evaluation Report Volume II – Senegal 195 14:25-17:30 Meeting at Hotel Novotel with authors/validators/teacher: Process, collaboration, impressions, recommendations. Cheikh Tidiane DIOP, science expert Samba DABO, math expert Oumar SY, middle school math teacher in Dakar 17:30-19:00 Team meeting to synthesize findings, challenges, recommendations and questions Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 30 Jan, Wed 8:00-8:45 Breakfast meeting at hotel (EA&CB) 8:45-13:00 Work at hotel on report, translations of instruments, bank issues, planning of trips (EA&CB) while preparing school visit permissions and travel (AS&BD). (Lunch) 14:00-19:00 (Continued, team meetings to plan fieldwork) 31 Jan, Thurs 8:00-9:30 Team work at hotel. 10:15-11:15 Meeting at Inspectorate headquarters (Inspection Générale de l’Education Nationale) in Dakar. Mr. Ndiogou FAYE, InspectorGeneral of the MoE 12:00-14:30 Lunch meeting – team – and planning 15:00-16:15 Meeting at Fédération Nationale des Parents d’Elèves et des Enseignants du Sénégal (National Federation of PTAs). Bakari BADIANE, President Mr. Dam SECK, Secretary General Mohamed Diem SESAY Secretary for Communication, Training and Research 1 Feb, Fri 8:00 Team: Depart hotel for outer Dakar (Dept. of Pikine) 9:00-10:00 Meeting at IGEN (Inspection Générale de l’Education Nationale) in Guédiawaye Mme Ami Sene DIABY Deputy Inspector Musa Diaba HASSAN, Assistant TLMP Evaluation Report Volume II – Senegal 196 10:15-12:20 Meeting with Guédiawaye middle school principal and staff. Visit to library. Observation of SVT demonstration lesson by Mr DIAW. Cheikh Tidiane SY, Principal Cheikna FALL (Math teacher) Abdou SENE (Math math teacher) Demba DIAW (SVT teacher & librarian) Lunch Team lunch and planning of field visits. 14:00-17:00 Planning, documentation of meetings. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 2 Feb, Sat 8:00-16:00 Eric & Carol: Breakfast and planning, bank and copying, reporting and methodology of upcoming visits. Calls to Babacar and Alhusseynou, Abdou. 3 Feb, Sun Morning (Check out and arrangement of cars) 14:00-15:30 Team lunch near crossroads: admin and planning. Drivers Alioun & Mamadou Departure of two teams (one North, one SE) “ Evening AS&CB: Arrival in Saint-Louis, check in Hotel Sindone, calls by Alhousseynou to Regional Inspector and other officials to plan Mon. visits. Mamadou Evening EA&BD: Check into the Hotel de Paris in Kaolack. Calls by Babacar to Regional Inspector and other officials to plan Mon. visits. Alioun 4 Feb, Mon Eric ALLEMANO and Babacar DIOUF in Kaffrine TLMP Evaluation Report Volume II – Senegal 197 8:00 – 9:00 10:00 – 12:30 Interview at the Inspection d’Académie of Kaffrine. Discussion of issues related to insufficient quantities of TLMs delivered to the IA. Comments from principals suggest that: • the manuels d’appoint are not linked to the curriculum; • the TLMs for the primary grades are “too difficult” and should have been validated with children before being printed. The Academy was not involved in training on the TLMs. Visit to the Lycée de Boulel, a rural secondary school. TLMs available for science, history& geography, physics & chemistry for some grades but not others. Number of TLMs insufficient for some classes. Problem with combining two grades in one TLM: parts of the curricula for each grade left out. 7&8th grade math exercises “too easy” EA observes a physics & chemistry class. Mr. Samba BAKHOUM, IEVS Mr. Abdoulaye FAYE, BEMSG Tel. 221 775110894 Mme Rokhaya SARR, Pedagogical counselor for English Mr. Boubacar TIMERA accountant & supplies manager Mme Neifatou N’DIAYE, Science teacher; Mr. Malal DIOP, History & Geography teacher Mme Bernadette MENDY, History& Geography teacher Mr. Abdou CISSOKO, Physics & Chemistry teacher. Lunch 14:00 – 15:30 Collège d’Enseignement Moyen (Middle School) in Kaffirine city. EA observes a science lesson. Mr. Samba THIAM, Math, Physics & Chemistry teacher Mr. Abdoulaye KA, Science teacher 16:00 – 17:00 17:30 – 18:00 Inspection Départementale de l’Education Nationale (IDEN) Debriefing at the Inspection d’Académie Mr. Ousmane BA, Inspector Mr. Fodé SARR, Accountant & supplies manager 5 Feb, Tues EA and BD in Kaolack TLMP Evaluation Report Volume II – Senegal 198 8:00 – 9:30 10:30 – 12:30 Lycée de Sibassor (rural secondary school) Focus group discussion with teachers about the TLMs. Primary school in Ngothie (rural). Focus group discussion with teachers and other stakeholders about the TLMs. • Mr. Samba NOBUR, Principal and 11 teachers. • N’DIAYE, President of the School Management Committee • Mme Néné DIOUF, committee member • Mme Fatou FAYE, President of the Women’s Group • Alioun, Representative of the Sports Committee • Mr. Mame Kor THIARE, Representative of the Village Chief. Lunch 3:00 – 4:00 Meeting with the head of the Inspection Départementale de l’Education Nationale (rural areas) Issues of TLM use in schools. 6 Feb, Wed EA and BD in Fatick 8:00 – 9:00 9:30 – 10:00 11:00 – 12:00 Briefing with the Departmental Inspector N’diongolor Middle School (rural) • Focus group discussion with teachers. • Visit to library Collectif des Chefs d’Etablissements de Fatick (Fatick Principals Association) • Focus group discussion • Training needs Mr. Saïdou BA, Inspector Mr. Ardo N’DIAYE, Principal Mr. Donga Carrera, Science teacher Mr. Papa Mamour DIOP, Physics teacher Mme Ndeye Magatte GUEYE, Science teacher Mme Aïda Ndoye N’DIAYE Science teacher Mr. Cheikh DIONE, Deputy Inspector Fatick IA Mr. Saïdou BA, Departmental Inspector, Fatick Mr. Assane FAYE, Accountant & supplies manager, Fattick IA Mr. Mame Kor FAYE, Librarian Mr. Saly SARR, IT technician Lunch 14:00 – 16:00 Attended, with IA staff, a JICA workshop in Fatik city on optimizing the use of textbooks and other learning aids. Invited to attend by the Deputy Inspector of the Fattick IA. 16:00 EA & BD drive back to the Hôtel de Paris in Kaolack and pack, check out and return to Dakar. TLMP Evaluation Report Volume II – Senegal 199 4 Feb, Mon Alhousseynou SY and Carol BENSON in Saint-Louis 8:00-8:15 Team meeting to finish planning the day 8:30-9:30 Meeting with l’inspecteur départemental (IDEN) de Saint-Louis (departmental director) to discuss project and locate schools and trainers; he accompanied us to the schools. Departmental Inspector (IDEN) Mr Samba Laobé DIOP 10:00-10:30 Arrival at (rural) Mpal Primary School; discussion with school principal, tour of library, checked presence of materials. School principal Insab BADJI Teachers Mme DIABO, Mme Bintu (bilingual) 10:40-11:15 Arrival at (rural) Rawane Ngom Middle School; found materials in his office. Asked 2 classes (4ème and 3ème) about SVT book. School principal Ngange GUÈYE Inspector Mr. Samba L. DIOP 11:45-12:35 Went to Académie, met Inspector General of IA in hallway (planned meeting later) and stayed to talk with Mr NDIAYE, who organized discussions with Specialist Inspectors. Inspector Gen/ Coord. of Middle and Secondary Ed. Amadou NDIAYE 12:35-12:45 Meeting with Mathematics Specialist Inspector. Math Inspector (IA) Alioune DIOP 12:55-13:05 Followed the Storage Chief (Comptable Matière) to new IA storage locale (10 minutes away) to see books in storage. Storage Chief Mr. TALL 13:10-13:25 Went to Centre de Formation (teacher training college) to meet another Specialist Inspector, this one for SVT (Science). Science Inspector (IA) Mr. Bactar FALL 13:35-14:05 Back to Académie, waited, then were able to meet with Inspector General. Inspector General (IA) Mr Ndar FALL 14:10-14:50 Lunch 15:00-15:50 (Waited at teacher training college and talked to people participating in Malaria workshop) 15:50-16:15 Meeting with SVT (Science) Inspector Science Inspector (IA) Mr Déthie Chiendella FALL 16:30 (Back to hotel to write notes and organize meetings for Tuesday in Louga). Meeting to synthesize findings of day 5 Feb, Tues AS and CB in Louga 7:00 (Check out of Hotel Sindone) Depart Saint-Louis for Louga. 8:10-8:45 Meeting with Inspector General of Louga IA, interviewed and he helped arrange visit to nearby middle school Mr. Lamine SARR, Inspector Genl. of the Louga IA 8:55-9:05 Meeting with principal of Middle School, interview Principal Middle School Mr. Abdullai DIALLO Mr. Amady KEBE from IA 9:05-9:35 Past and present librarians showed us books and explained how loan system works Mr Babacar N’DIAYE Mr Alias Ismaïl SYLLA TLMP Evaluation Report Volume II – Senegal 200 9:40-10:00 Interview in teachers’ room with two maths/science teachers Mr Mamadou THIAM Mr Abdulwahab SESAY 10:20-10:55 Meeting with Inspector General of IDEN Louga (whom we met in Saint-Louis yesterday). Inspector Genl. Of the IDEN Mr Hamedy KEBE 11:00-11:20 Meeting with school principal at Khady Mdiaye School director Gning primary school; visit to class to see that they are using project TLMs (no lesson but they showed us their books—see photos). Ms Kourseline BARA Teacher Mr. Moussa DIAW 11:20-11:30 Went to the Centre de Formation looking for the CPIs (Conseilleurs pédagogiques itinérants) but they were not available. 11:30-11:50 Walked to the practice preschool beside the Centre de Formation to meet a parent (father of a middle school student) who is school principal. Preschool Principal Mr. SY 11:50-12:15 Back to the IA to witness people from the IDEN in Kebemer picking up TLM materials and others (see photos). Mr Alioune THIAM 12:15 Left for Thiès, lunch at Kebemer on the way. Evening Discussed day, Alhousseynou made calls to facilitate the next day’s visits. 6 Feb, Wed AS and CB in Thiès 8:00-8:30 Meeting with Inspector Gen at IDEN. Inspector General Thiès city Mr Talla FAYE IDEN 8:30-8:35 Viewed storage area and discussed distribution. Storage supervisor Mr Khatry DIAGNE 8:35-8:50 Discussed outside Mr. Sahid NDOY, chef Math Mr. Mamadou DJAW, collège Math/SVT 9:00-9:30 Meeting with Inspector Genl. Adjoint de l’IA and (during this meeting) quick interview of President of Parent-Teacher Association. Insp. GenL. Adjoint de l’IA Mr. Ibrahima BARR President of parent assoc. Mr. Moussa KANTE 9:30-9:40 Viewed storage area (locked, see photos taken from outside). 9:45-10:00 Meeting with principal of college. Principal of Mamadou Diaw middle school Mr Amarie MBAYE 10:00-10:20 Visited library and spoke with librarian and assistant. We recommended distributing the books that were just sitting on shelves and in boxes. School librarian Ms Pall Diop N’DIAYE Assistant Ms Dide Mbodji COULIBALY 10:30-10:45 Went to Training Centre, met with trainer. Mr Papa Birame THIANDOUNE (Return to Dakar, EA&CB check into Hotel Ibis) 7 Feb, Thurs TLMP Evaluation Report Volume II – Senegal 201 Morning EA and CB prepare presentation for USAID 15:00-17:15 Debrief at USAID Mr Pape SOW Ms Michelle CHEN 8 Feb, Fri Morning Team meeting, sharing of findings in the North and the South East field trips. Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF Afternoon Meeting with Abdou Maty Sene, Former Chief of Party TLMP Eric ALLEMANO Carol BENSON Alhousseynou SY Babacar DIOUF 9 Feb, Sat Morning Work on the preliminary draft of the IBTCI mission report Eric ALLEMANO Carol BENSON Evening Dinner for the team at Babacar Diouf’s home 10 Feb, Sun Evening Eric Allemano departs for Paris TLMP Evaluation Report Volume II – Senegal 202 ANNEX E. Summary of Findings Topic Findings Comments 1. Responsibility for writing the TLMs 2004-2006: 80% of the work done at ECSU with American authors 2007-2008: 50% of the work done at ECSU 2009-2011: 80% of the work done in Senegal 2012: 100% of the work done in Senegal by local authors. According to ECSU’s TLMP Program Manager. The lists of authors in the TLMs produced over the life of the project support this information. They are all Senegalese for the last books produced. 2. Variety of titles 28 titles produced for Grades 1 to 11. Subjects include French, math, science & environmental studies, history & geography, physics & chemistry, health and guidance. Three of the 28 titles (elementary school readers) are reprints of existing, non-TLMP books as were three workbooks. Only two teacher’s guides were produced, for environmental studies in Grades 1-2 and Grade 3. The post-primary books are “supplementary” books. The lack of training and teacher’s guides may explain why many teachers interviewed found the TLMs “too difficult” (esp. 7th-8th grade math, which some teachers in the Southeast refuse to use) or not “aligned” with the curriculum. The teachers were apparently not familiar with the concept of supplementary curriculum materials. 3. Validation of the TLMs Little is documented. An expert committee was supposed to evaluate the drafts and they were piloted for 15 days in two regions. Student learning was apparently not tested. The TLMs were developed in haste to meet deadlines. It is claimed that the TLMs are all aligned with the official syllabi, although this was disputed by some of the teachers met during the field work. 4. Copyright issues The copyright holders of the early books are the Project Director at ECSU and USAID Senegal. Later books add the MOE. TLMs that are translated and edited versions of American textbooks do not mention the original authors or copyright holders. Only a few TLMs document the sources of illustrations, graphs and charts, many of which were taken from the Internet or other sources. Many photographs and illustrations do not reflect Senegalese or African people, objects and environments. Males are represented far more frequently than females. The TLMs were supposed to be sensitive to culture, male-female equality, youth leadership, HIV/AIDS, etc. TLMP Evaluation Report Volume II – Senegal 203 Mention of HIV/AIDS is restricted to the TLMs on health. 5. Cultural sensitivity and special themes Overall, the printing process went well despite the fact that the printers had to do “rush” orders and deliver the books quickly to the IAs. The MOE seems to have been inadequately involved in the distribution process in terms of determining the number of titles to distribute to different IAs. In addition, the IAs and IDENs were not clearly informed about the procedures to follow to get the books to the schools and the students. Storage has been a problem although some schools keep the TLMs in their libraries or in specially-built rooms. Others are kept in the principal’s office or even home. In a few schools, the TLMs are exposed to damage from dust, wind, rain and termites. According to the printers, they had to bid separately for the printing of each TLM title, which was a cumbersome process. Some schools did not receive enough TLMs to reach a ratio of one booke per student. The team saw evidence of theft of the TLMs in at least one IA storage room. Some schools charge parents fees for letting students take books home. 6. Printing, distribution and storage of the TLMs Despite claims by the TLMP Project Manager and statements in newsletters that teachers had been trained, no evidence of this was found during the field visits. Training was apparently not budgeted. 7. Teacher training Whether or not the MOE informed the principals of the official status of the TLMs, many teachers expressed doubts about the issue. The absence of teacher’s guides for all but two subjects made it challenging for poorly￾trained teachers to use the TLMs. Principals find that the TLMs help teachers do their lesson planning better than before. Some say that they find that the TLMs have boosted student learning. Before the arrival of the TLMs, teachers in at least one region (Fatick) relied on the Internet for finding materials for their lessons. When there are not enough TLMs for individuals, students often use available copies for group work. The TLMs were more favorably viewed by: Educators in the South-East than in the North. Academy, Departmental and subject-matter inspectors than the teachers. Better-trained teachers than poorly-trained teachers. Parents had little to say but were generally pleased that the schools and the students had more books than before. A positive aspect of group use of a TLM is that it can promote collaborative learning. 8. Perceptions and uses of the TLMs TLMP Evaluation Report Volume II – Senegal 204 ANNEX F. Evaluation of Selected TLMS Math (2008) Suppl. 4ème￾3ème (9th￾10th) J Houston & USAID Ed-in-Chief J Houston Asst Ed A Sene 0 0 0 0 0 This manual d’appoint is “rich with the American approach” of situations/problems and is meant to complement study (viii). Authors: 2 ECSU (Houston with Lawrence), 2 Senegalese “author/adapters.” Intro by Laura Bush. Contains only French words and formulas. SVT (2010) Suppl. 5ème (8th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed C Seck 1 Men (group 9) 3 Boys (12,21, cartoon 25) 1 Women (white 163) 1 Girl (cartoon 54) Dakar (Hwy constructn 3, 10,39) Urban foods (45, 47) Deals with “Environm ental problems” Non-Africans (12,13,25,50,54,163) Non-African animals (77,81,94) “Support” book corresponding to “all of the points in the official curriculum of May 2008” (iii). Contains exercises to promote “mastery” or “methods.” Authors: 3 Senegalese (men). Few people represented, majority are boys. Some Senegalese photos illustrations, mostly urban. Phys Chem (2010) Suppl. 2ème Sec (11th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 16+ Men (13,15,27, 28,54,105,107,108, 116, 137,145, 154,157, 176, 201, 213) 0 Boys 1 Women (cartoon family 15) 0 Girls Dakar (crowds 98,99) 0 Non-Africans (13,15,27,28,54,105, 107,108, 116,137,145, 154,157,176,201, 213, white hands 261-262) French cheeses (312) Title (year) Type Level Copyright holder(s) Editors Representation of men/boys Representation of women/girls Representation of local culture Integration of transversals (In)appropriate content SVT (2010) Suppl. 4ème-3ème (9th￾10th) AM Sene, USAID & MINED Ed-in-Chief AM Sene Asst Ed C Seck 5 Men (pp 5, 58, 60, 79, 158) 3 Boys (15, 16, 29) 2 Women (photo 5, pregnant Euro 71) 2 Girls (cartoon 51, holding baby 78) Very little (kwashiorkor and goiter 5, cartoon leg 59, runners 58) No HIV or environmental issues Non-Africans (5,6,16,35,48,51,53, 59,63,65, 66,7173,77,80,97,99,159) This manual d’appoint is not intended to replace the textbook but “respects the objectives and competencies” of the curriculum (vii). Authors: 3 Senegalese (1 woman), 1 ECSU (J Houston). More males than females represented; of 4 females total in book, 1 woman pregnant and 1 girl holding baby (traditional roles). Overabundance of non-African cartoons and photos, e.g. drawings of naked white child/youth/adult of each sex (65) deemed highly inappropriate by informants. Highly unnecessary use of Western/non-African elements like white hands, eyes and ears, European family photos for genetics and Perrier water for experiment. TLMP Evaluation Report Volume II – Senegal 205 This “study instrument” conforms to the curriculum of 2008-2009. Each chapter has content, summaries and exercises (explained pp xiii-ix). Authors: 5 Senegalese (men). Almost exclusively men represented, almost all non-African. No transversals, little/no Senegalese culture. Phys Chem (2nd ed 2010) Suppl. 4ème￾3ème (9th￾10th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 9 Men 1 Boys 1 Women (grp 13) 1 Girls (cartoon 36) 0 0 Non-Africans (13,36,84,91,157,158, 163- 165,169, cartoon men 92,96,97) This book takes a “practical and simple American approach” combined with a “rigorous and precise Senegalese approach” (viii). Authors: 3 Senegalese (men), 1 ECSU (Houston). Intro by Laura Bush. Mostly non-African men in illustrations. SVT (2010- 2011) Suppl. 6ème (7th) AM Sene, USAID & MINED Ed-in-Chief AM Sene, Asst Ed J Houston 3 Men (43,44,46) 0 Boys 0 Senegal (3,9,44,65, 66,74) ? Non-Africans (134, 149) This “support” book “respects the objectives and competencies” of the curriculum of Oct 2008 (vii). Authors: 5 Senegalese (2 women, 3 men). No females represented. Many photos of Senegal. Hist & Geog (2011) Workbook Prim gr 3 USAID & MINED Ed-in-Chief AM Sene, Asst Ed C Seck (All men) 0 0 0 Old, low-quality photos, cartoons (9,29,49, 58) This “livret” (workbook) is not meant to be written in, but it provides exercises for students. Old, often distorted illustrations/photos/caricatures. Collaboration with USAID basic ed (EDB) project. No authors listed, only “editing team” of 6 Senegalese (men). Envir Sci Tech (2006) Suppl. 3ème (10th) J Houston & USAID Ed-in-Chief J Houston, Asst Ed M CoulsonClar k Men (non-Af 28,32, 35,41,53,77, 80,81,83- 85,88,90, 94,100101,112…) Boys (59) Women (non-Af 41,47,53,77,112, Muslim woman 116) Senegalese farmers (77) 0 Non-Senegalese (28, 41, white hands 42, 47, urban 53,59,100- 101,112,117) This manual d’appoint depends on U.S. photos. (Exceptions are one Muslim woman and some Senegalese male and female farmers.) Contributing authors appear to be 5 Americans and 2 Senegalese (including A Sene). TLMP Evaluation Report Volume II – Senegal 206 ANNEX G. List of Clarifications on the TLMs for Possible MOE Distribution The following is a list of clarifications for possible inclusion in an MOE letter to IA directors (copied to department directors) regarding TLMP materials. If possible, it could be attached to the proposed curriculum linking chart or brochure. 1. All regions have received TLMP materials (list them by level). These books financed by USAID are to be distributed free of charge to the schools. They should be considered school property but loaned to students and teachers for the appropriate level. It is up to the school to determine if this distribution is to be short-term (example: day of the lesson, 1-2 weeks from the library, or for the entire school year) but by the end of the school year the books should be returned to the school to be used the next year. School directors are instructed to do everything in their power to ensure return and re-use of books, however, it is understood that there will be occasional small losses due to wear and tear, misplacement, etc. 2. If you have not received these materials, please check with your IA or IDEN. If your school does not have the resources to pick up the materials at your IA or IDEN, we direct you to contact the Regional Governors and/or your city mayor to request transportation help (as for more remote schools Governors can use vehicles from the gendarmeries). 3. These materials were destined for enrichment of our national curricula even if they do not cover all points. Please review them within your subject department to see what is included and how content and activities could best be used. 4. They were also meant to be distributed to all teachers and pupils. They should not be kept in storage unless teachers and students have access to this storage and can sign out the books for their own use. 5. If you currently have cartons of TLMP materials in storage, please distribute them as soon as possible. We understand that the quantities may not allow for 1 student : 1 book ratio, but we recommend that school leaders (directors, principals) ensure that students have contact with these materials through one or more of these methods: • Students borrowing from school libraries or storage rooms, or from their teachers. • Students using them for in-class study or independent study. • Shared after-school activities (where 4-5 students living in the same neighborhood share one book, with 1 student responsible). • Teachers borrowing from libraries or storage rooms to use the TLMs in classrooms. • Teachers borrowing from libraries or storage rooms and signing them out to students. • Teaching students how to cover and care for books and return them at the end of each week/month/term/school year. Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania Evaluation Report – South Africa This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Edward Jay Allan (Project Director), Carol Benson, and Rakgadi Phatlane under Task Order AFR￾12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Intentionally Left Blank TLMP Evaluation Report Volume II – South Africa 207 EVALUATION REPORT – SOUTH AFRICA EXECUTIVE SUMMARY This evaluation differs from those conducted in the five other countries in which TLMP was implemented in that funding for Ithuba (the name given to the project in South Africa) was provided to complete work begun under the President’s African Education Initiative (AEI) and not to initiate continued activities; USAID-assisted activities ended by 2009. Unlike the other five projects funded during 2009-2012, which were being wrapped up in the course of the evaluation, the evaluation of Ithuba aimed to determine any lasting impacts that the project might have had on the South African educational system, administrators, teachers and students. The objectives of the study were as follows: Objectives • Determine if the planned outcome of facilitating the development of a “reading culture” in South Africa occurred. • Assess the results achieved and determine whether the South African TLMP had any spin-off effects on policies or practices of the Department of Basic Education (DBE), regional education offices, schools and implementing partners. • Seek evidence of how the practices and effects of TLMP may have continued since the end of TLMP in 2008. • Identity any long-term effects of teacher/educator participation in the development of supplemental readers. • Document challenges and lessons learned. • Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy (www.usaid.gov) Program Background The focus of the Ithuba Writing Project (Ithuba) was the development and production of storybooks in South Africa’s nine official African languages, plus Afrikaans and English, by training teachers to become authors and some to become illustrators. The lead South African partner was READ Educational Trust (READ), joined by the Molteno Institute of Language and Literacy (Molteno). The writing workshops were designed and implemented by TLMP leadership at UTSA and its partner, the University of Texas Austin (UTA). The workshops were first delivered to trainers (using a training-of-trainers (TOT) model) and then rolled out with teachers from target areas. Assisting UTSA in delivering further workshops were five trainers from Molteno and 10 trainers from READ, all of whom had been trained in materials development processes. The goal was to create a model for materials development so that teachers would have the capacity to use storybooks to promote literacy learning along with the capacity to create future iterations of stories in appropriate languages to suit their needs. Teachers’ guides in the form of 4-page leaflets linked the stories to the national curriculum, in hopes that teachers would make use of the stories to teach literacy across the curriculum. The storybooks were printed at Uniprint in Durban and distributed to target schools through the provincial Departments of Education (DOEs). Two writing centers were created, but the team could find no evidence of their continued existence at the time of the evaluation. Methodology And Limitations A “mixed methods” approach was used, comprised of: • Review of program documents and a sampling of TLMs produced TLMP Evaluation Report Volume II – South Africa 208 • Key informant interviews with UTSA staff, in-country partners, DOE personnel at various levels, staff of other stakeholders, and school principals and teachers • Site visits, including classroom observations and, as feasible, getting perspectives from learners During the in-country phase (May 5-16, 2013, inclusive), a team of two Americans and one South African collected data from a range of stakeholders in Limpopo and Gauteng provinces and READ’s representative from Mpumalanga, including: officials of DBE and the provincial DOEs; representatives of READ, Molteno and Uniprint; representatives of the University of Limpopo (UL) and the University of Pretoria (UP); and school principals, teachers of grades 4, 5, and 6 (the “Intersen” years), and students in three schools – a township school in Pretoria and two rural schools in Limpopo. Data collection was challenging due to the team’s limited time in country, a teachers’ strike and go-slows called by the unions, the fact that the project had ended in 2008/2009 and the reorganization of education offices since that time. We saw individual storybooks and teachers’ guides in the appropriate languages in the schools. Achievements of Ithuba There have been a number of achievements of Ithuba/TLMP in South Africa. The major achievements that could be verified are the following: • Storybook development: Ithuba contributed significantly to the number of high-quality supplementary reading storybooks available in schools in the nine official African languages. • Use of TLMs: At least some storybooks are being used in some classes in Gauteng, Limpopo, and Mpumalanga. We found Ithuba materials displayed on Ithuba stands, in rolling cupboards, on library shelves and in classrooms. Foundation phase (grade 1-3) teachers reported using the storybooks, along with teachers of grades 4-6, for whom they were intended. • Contribution to improved reading ability: Virtually all of the students we observed in grades 4 through 6 who were asked to read from the storybooks, whether chorally or individually, were able to do so with accuracy and fluency in their home languages. While there is no way to tie this directly to Ithuba, their presence and use would suggest that they have contributed significantly to children’s reading skills. A representative of Molteno reported that she saw students reading technically difficult graphemes (like consonant trigraphs in Sepedi) in the Ithuba books with comprehension and fluency, and the team saw that students were able to re-tell stories orally or in writing, as called for in the Ithuba teachers’ guides. • Contribution to teacher competence in developing literacy skills: Enlisting and training teachers to become authors was innovative and seems to have left lasting impressions. The Ithuba author workshops reportedly raised the capacity of teachers and trainers to be authors while motivating them to write actively, and it is hoped that they will also motivate their students to write. • Extension of Ithuba methods for storybook development: In one case, a library specialist at the Limpopo DOE convinced colleagues to help her develop storybooks in Sepedi (with a plan to extend the program to the other official African languages) for the Foundation phase. They gained the support of Irish Aid to develop mock-ups and field-test them in the schools. In another case, members of the English Department at UL were inspired by Ithuba so they sought funding from Belgium to create the “Multiple Literacies Project,” which produces high quality storybooks in home languages using the Ithuba model. • Ithuba storybooks are still appropriate under the new curriculum: The new Curriculum and Assessment Policy Statement (CAPS) emphasizes early literacy in the mother tongue and literacy across the curriculum, and the team was informed by South African educators that the Ithuba materials are congruent with the new curriculum standards. TLMP Evaluation Report Volume II – South Africa 209 Challenges to Ithuba There were also some challenges to TLMP in South Africa. • Sustainability issues within the DBE and USAID/SA: In principle, the only obstacle to the basic sustainability of the Ithuba materials is funding for the reproduction of existing materials plus limited funding for revisions/updating and, if desired (and recommended), funding for adaptation of existing materials into other languages and for training. There is an ongoing need – both real and perceived – for supplementary reading materials in local languages. The Ithuba materials appear to be high quality and are consistent with current curriculum guidelines. The materials have great acceptance by the teachers and students and provincial DOE personnel whom the team met, and other agencies have been willing to support expansion, to at least a limited extent. However, we understand from Mission staff that USAID intends to work on its new education initiative from scratch and national DBE personnel advise that they are not in a position to take the Ithuba materials up at this time. • Pedagogical issues: Most of the classroom observation demonstrations we saw were heavy on the “didactic” side and less on the “enjoyment” side. Teachers generally did have the idea of letting students read aloud independently as well as together, although there were occasional vestiges of traditional methods like reading first and asking students to repeat, and students at the Onane Primary School were encouraged to develop ad hoc skits based on the reading. • Consistent use of the TLMs: Adding to the challenge of methodology, it was difficult to determine whether or not teachers used the books regularly, because most of the sessions we saw were “staged” for us. We were told at each school that teachers used the books, but except at the Onane Primary School there was no instance where we saw use of the materials in the course of a class conducted in its regular time period. • Some logistical difficulties: Because the readers were supplemental and not meant to be used as texts, only 20 copies of each reader was shipped to a school, enough for sharing if a class had the 40 students called for by DBE policy but not enough if classes were larger. The 4-page teacher’s guides, which were developed as separate sheets (in strong cardboard) for each storybook, were not found at all schools. There were some reports of mismatches between the languages of the stories and the home languages of the learners. It was difficult for teacher-authors to write consistently at grade 4-6 levels. Lessons Learned Many constructive lessons were learned by the implementers, including the following: • Creating good relationships with education officials: According to UTSA implementers, many of the successes of the project emanated from the good relationships established between UTSA and the DBE. • Teachers as writers and editors: From UTSA reports, it was noted that once the South African team members developed skills in estimating reading levels and judging appropriateness of stories, they could have been relied on to do all of the editing, rather than sending materials to UTSA. • More evaluation and follow-up: Local implementers commented that there was inadequate M&E or follow-up. • Make TLMs available to parents: In the interest of creating a culture of reading, the storybooks should have been available to families, possibly through loaning schemes, promoting intergenerational activities around print. Key Recommendations 1. USAID/SA should engage in dialogue for continued use of Ithuba materials: At the very least, the mission should initiate a dialogue with DBE to determine interest. The major pedagogical TLMP Evaluation Report Volume II – South Africa 210 reasons are discussed above – the massive need for supplementary reading materials (into the production of which the American taxpayer has already invested millions of dollars), their compliance with current curriculum guidelines, their high acceptability. In addition, the materials are branded as being supported by the American people and will be seen in most South African households with a child in elementary school and can be used also to promote adult literacy and to develop a culture of reading. 2. Consider using the materials development model of TLMP (Ithuba) to develop new storybooks for early grade literacy: Involving teachers and other educators in the story writing process was designed to create long-term impacts in teachers’ approaches to reading and writing. Literacy (in home languages) has been identified as a priority by DBE in South Africa as well as by USAID. 3. Selection of future implementers: For a project involving TLM development, implementers should have: knowledge of the context; understanding of how TLMs satisfy curriculum requirements; experience integrating literacy promotion strategies into existing teacher training mechanisms; understanding of the sociolinguistic context and the language capabilities of teachers; knowledge of the printing and dissemination process utilized by public and private agencies; and ability to consistently monitor and evaluate (along with partners) project activities. 4. Consider the use of Annual Program Statements (APS) to complement ongoing education activities. It may not be workable for an education activity to incorporate the development of learning materials as part of its work; e.g., its primary focus might be on teacher training or educational administration. However, this does not obviate both the need and desirability for learning materials. An APS would give local entities, possibly in collaboration with U.S. partners, the opportunity to build its own capacity and the capacity of local educators to develop, produce, distribute, and use materials on, e.g., a provincial level with the possibility that they could develop approaches novel in the country to any of these aspects, maintaining appropriate interactions with other development partners; it could also provide opportunities for facilitating leveraging. 1. EVALUATION PURPOSE AND OBJECTIVES 1.1 Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched in 2005 by the Africa Bureau of the United States Agency for International Development (USAID/W) as a part of the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. TLMP was extended for another three years in 2009 to 2012. The funding mechanism was the use of Cooperative Agreements (CA) awarded to each of five Minority-Serving Institutions (MSI). The University of Texas San Antonio (UTSA) was awarded CAs for South Africa (2005-2008) and Malawi (2009-2012). Each MSI was responsible for generating (i.e., identifying, selecting, developing, adapting, printing, and assisting with distributing) a minimum of 600,000 copies of quality, cost-effective education materials produced in partnership with the national ministry of education and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 10) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA, 11) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 12) Ensure alignment with national curriculum and to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Evaluation Report Volume II – South Africa 211 Through an initial needs assessment and discussions with the national Department of Basic Education (DBE) in South Africa, it was determined that UTSA should develop supplemental reading materials for grades 4-6 in nine African languages, plus Afrikaans and English to be in alignment with the then current curriculum. To implement the project UTSA received approximately US$8 million. 1.2 Evaluation Objectives Since the South African TLMP (known as Ithuba) ended in 2008, this evaluation differs somewhat from those done on the other five TLMPs, the Africa Bureau was looking for an impact assessment. As such, the objectives of this evaluation include: • Determine if the planned outcome of facilitating the development of a “reading culture” in South Africa occurred. • Assess the results achieved and determine whether Ithuba had any spin-off effects on policies or practices of the DBE, regional education offices, schools and implementing partners. • Seek evidence of how the practices and effects of TLMP may have continued since the end of TLMP in 2008. • Identity any long-term effects of teacher/educator participation in the development of supplemental readers. • Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy (www.usaid.gov) The evaluation will also discuss the impact that implementing TLMP in South Africa has had on UTSA, the MSI partner. 2. PROGRAM BACKGROUND 2.1 TLMP Background in South Africa TLMP in South Africa took the form of the Ithuba Writing Project (Ithuba). The focus of its activities was the development and production of storybooks in the nine official African languages, plus Afrikaans and English, by training teachers to become authors and some to become illustrators. Teachers’ guides in the form of 4-page leaflets linked the stories to the national curriculum in hopes that teachers would make use of the stories to teach literacy across the curriculum. The storybooks were printed by Uniprint in Durban and distributed to target schools through the provincial DOEs. With the donation of the project laptops to two writing centers, one in Gauteng and one in Limpopo, it was planned that writing workshops would spin off to other teachers as well as learners. The idea was that if teachers were engaged in writing in their home languages they would engage learners in writing as well. UTSA and its partners developed a series of intensive writing workshops in Gauteng (largely urban) and Limpopo (largely rural) during which a total of 140 teachers from all nine language groups were taken through the steps of writing original children’s books in their own languages. Teachers were encouraged to find the intersection between their experiences/expertise and topics that children in grades 4, 5, and 6 (the “Intersen” years) would find interesting. Authoring teachers were asked to write their stories in the language in which they felt most comfortable. These teacher-authors were also provided a workshop on rendering a version of their story in South African English—not a translation, but a relevant adaptation of the story, accounting for differences between languages, cultures and experiences. They drafted their stories on laptop computers, giving them access to technology and increasing their skills sets to include word processing and formatting. The project implementers also designed an illustrators’ workshop to develop the skills of local artists, whose original works were then featured in the books. TLMP Evaluation Report Volume II – South Africa 212 The writing workshops were designed and implemented by TLMP leadership at UTSA and its partner university in the U.S., the University of Texas Austin (UTA). The workshops were first delivered to trainers (using a training-of-trainers (TOT) model) and then rolled out with teachers from target areas. Assisting UTSA in delivering further workshops were five trainers from Molteno Institute of Language and Literacy (Molteno) and 10 trainers from READ Educational Trust (READ), all of whom had been trained in materials development processes. The goal was to create a model for materials development so that teachers would have the capacity to use storybooks to promote literacy learning along with the capacity to create future iterations of stories in appropriate languages to suit their needs. The separate, spiral-bound training manuals, which were provided in each trainer kit covered the following topics: • Drafting Our Stories • Revising and Illustrating Stories • Revising and Field-Testing our Stories • Developing, Revising and Editing Stories • Implementing Ithuba Materials The latter manual covered read-aloud, shared reading/guided reading, independent reading, and discussion topics for teachers as well as matrices linking the stories to “standards” (competencies) in the National Curriculum Statement (NCS) curriculum and referring to the individual teachers’ guides for each storybook. As an intermediary step in the process, members of the development and editing team (which consisted of South African and U.S. literacy and content experts) worked on the English versions of the books to ensure their instructional quality. The team also worked to link the storybooks to the national curriculum. The newly-trained local illustrators drew pictures for mother tongue and English versions of the books, and the books were field tested with children in South African public schools by the authoring teachers. Acceptability criteria included: word choice (readability), comprehension levels, and story quality criteria (including relevant illustrations). Finally, a team of South African language experts edited the stories for appropriateness of word choice and conventionality. Eventually the entire effort resulted in the production of 140 storybooks in English and 140 storybooks among the nine official African languages. The books also had to include certain cross-cutting issues, in accordance with the CA: cultural relevance, gender roles, and HIV/AIDS. In keeping with this directive, the percentage of stories covering these and related topics are as follows: HIV/AIDS (26%), abuse and bullying (15%), crime, racism and classism (97%), and cultural traditions (25%). Based on the implementers’ own analysis, common themes the stories included were caring (31%), responsibility (23%), achieving goals (21%), earning respect (7.8%), traditional values (3 stories) and breaking gender barriers – all values that society and schools would want to instill in children. Each narrative was between 20 and 32 pages long, contained between 8 and 10 full-color illustrations and was approximately 4.5” X 5.5” in size. The stories were instructionally integrated with the NCS through teacher’s guides that served as an impetus for mathematics, natural science, or social studies lessons that followed the reading of the books. The contract to print the books was awarded to Uniprint. After print runs of all titles, kits were assembled that included storybooks and accompanying teachers’ guides in the language(s) chosen as the Language of Learning and Teaching (LOLT) at each target school. They also included an Ithuba Overview guide to assist teachers in integrating the teaching and learning materials (TLM) across the curriculum. TLMP Evaluation Report Volume II – South Africa 213 As integral TLMP partners, Molteno and READ facilitated the teacher workshops on the use of the storybooks. These two non-governmental organizations (NGOs) reported that they conducted workshops of 8 and 16 hours that trained a total of 5,903 teachers and principals on the use of the storybooks. By the end of the project, approximately 8,000 teachers had participated in these trainings, which included an orientation to using the Ithuba teacher’s guides accompanying each storybook. After TLMs were delivered to the provincial DOE (and subsequently made available in schools), Ithuba organizers reported achieving the following outcomes: • Educators who worked in learning areas outside of language development learned to integrate the stories into their specialized curriculum. • Teachers were introduced to Shared Book Reading and Read Aloud methods. • Educators interviewed reported using the supplementary readers during daily lessons and that their students enjoyed them. • Teachers were so impacted by the books that one teacher in Limpopo reportedly quit smoking after reading one of the Sepedi stories, Le reng le kgoga, which discusses the dangers of smoking. • Authoring teachers received 10 hours of Continuing Education credit from the University of Pretoria that can be used toward a degree program. When TLMP ended, READ offered to conduct a final print run of Ithuba materials and to offer teacher training based on those storybooks. READ also proposed to work closely with provincial education staff to add value to these endeavors, and to bear all administrative costs of these efforts, including the costs of additional training. During this same activity, READ took responsibility for the re-kitting of the 227 kits printed by Uniprint that were stripped out and repacked as some had not been appropriately packed for the regions to which they were to be delivered. Ithuba also helped to establish two writing centers, one in Limpopo and the other in Gauteng, with the idea of creating some sustainability in the process of training teachers to be storybook authors and the laptops used by Ithuba were donated to these centers. However, by the time that the evaluation team conducted its work in May 2013, the writing centers had become defunct. As a result of Ithuba, staffs at UTSA and the partner institutions have generated a number of publications and presentations at international meetings. The MSI implementers have expanded on lessons learned under Ithuba in implementing a subsequent TLMP project in Malawi. 2.2 DBE Background and Priorities The 1997 adoption by the South African government of its Language-in-Education Policy (LIEP) mandated that children be taught in their home languages throughout primary school. This was a challenge for South Africa as historically under Apartheid, several “Bantustans”/Homelands had been developed in each region, each constituting the separate homeland for a particular ethnic group that spoke a specific language. When the government changed, a decision had to be made in each province as to which African language would be the “official” language. For example, in Limpopo there had been approximately 14 homelands, with each speaking its own language and with each creating a homeland￾based educational system. The challenge for education became 1) deciding which language would be the “official” language of the province, and 2) how to dismantle the Apartheid-based homeland education departments and form one provincial department of education (DOE). DOE officials and different language communities were hard-pressed to make this decision, although they saw the reasoning behind it. The decision was ultimately made for each province, resulting in the nine “official” languages; however, because of the overwhelming number of languages spoken in several provinces, students must be fluent and be able to read in at least three languages: mother tongue, the language of learning and teaching (LOLT), and English or Afrikaans. This created huge problems in terms of providing learning TLMP Evaluation Report Volume II – South Africa 214 materials to schools to satisfy the DBE requirement to teach in the “official” mother tongue of the province. At the time of TLMP implementation, there was no specific curriculum development unit within the DBE. When UTSA presented its approach, DBE officials were skeptical as the prevailing assumption was that teachers would not be able to develop the skills necessary to produce the materials. With careful explanations and evidence from similar activities in the U.S., UTSA convinced the DBE of the efficacy of the approach. Thus Ithuba was the first to involve teachers and administrators in developing learning materials. Because the approach was the first of its kind in South Africa, new ways of collaborating with the DBE had to be developed. This challenge had to be met in the context of USAID/SA closing out its educational programs. Despite this programmatic closure, USAID/SA assigned two members of its staff in Pretoria to provide limited assistance. With the closure of the USAID/SA education program, according to UTSA, when TLMP ended in 2008 it could not be renewed; instead, UTSA went on to conduct similar activities in Malawi for grades 1-4. 3. EVALUATION METHODOLOGY AND LIMITATIONS 3.1 Overall Methodology In collecting data for this impact assessment, a “mixed methods” approach was used, comprised of: • Review of program documents and a sampling of TLMs produced • Key informant interviews with UTSA staff, in-country partners, DOE personnel at various levels, staff of other stakeholders, and school principals and teachers • Site visits, including classroom observations and, as feasible, getting perspectives from learners as well as from educators Site visits were used, inter alia, to determine whether books actually had been distributed to schools and, if so, if they were still being used. Materials were reviewed to determine gender balance and appropriateness; apparent bias with respect to urban-rural issues; apparent relevance and appropriateness of content, including illustrations; language level; alignment with curricula; etc. 3.2 Research Conducted at UTSA Dr. Nancy Horn (team leader) conducted advance on-site and follow-up field work at UTSA to determine 1) the background to the project and how it was implemented, and 2) how MSI participation in a USAID-funded program built its capacity to engage in future development projects. 3.2 Research Activities in South Africa The field work in South Africa was conducted between May 5 and May 16, 2013, inclusive, by Dr. Edward Allan (project director), Dr. Carol Benson (a specialist in language and education, with particular familiarity with South Africa), and Dr. Rakgadi Phatlane, an educational researcher from the University of Pretoria. Dr. Thabile Mbatha also provided valuable assistance, although ultimately we were not able to visit KwaZulu-Natal. With the background information provided by UTSA, the team leader developed a background paper/research design that included questions to be posed to different stakeholders. However, finding individuals to answer these questions was highly problematic owing to the passage of four years’ time since the project ended. During this time, officials were reassigned or transferred; partner staff members resigned or passed away; leadership changed in all areas; the reorganization of education departments in the provinces and districts continued; and records were not consistently kept of all the changes. Consequently the questions were refocused to ascertain impact. These questions are as follows: TLMP Evaluation Report Volume II – South Africa 215 • How was the DBE influenced by the strategy for developing supplemental readers by Ithuba? Was this methodology utilized in any other way by relevant education departments at the national or regional level? • How have READ and Molteno continued to use the storybooks and/or processes developed for Ithuba in other parts of the country, and what have been the results? • How have teachers continued to use the materials? • How has children’s reading ability in home and other languages increased? • What are the elements of sustainability for Ithuba? The range of stakeholders from whom data were collected included: officials of DBE and provincial DOEs; representatives of READ, Molteno and Uniprint; representatives of the University of Limpopo (UL) and the University of Pretoria (UP); and school principals, teachers of grades 4, 5, and 6, and students (see ANNEX B: Schedule of Stakeholder Interviews and Activities). The team conducted interviews and site visits in and around Pretoria and in Limpopo. Most interviews were done in person, but some were done by telephone or e-mail depending on the ability of the team to contact people who had been involved. Only a few teachers were observed using the storybooks to determine how they have adapted the materials to meet their own teaching and learning requirements since TLMP ended, and these presentations were “staged” to accommodate the time of the team’s arrival. 3.3 Limitations of the Study • Inability to contact some key TLMP stakeholders: Since TLMP had ended four years earlier, the team was not able to find a number of key stakeholders. In some cases they had moved on, in some cases there were scheduling conflicts, and in the case of the KZN MOE we were not able to identify anyone who remembered anything about Ithuba. Further, with respect to KZN, our team was not able to identify any schools known to be using Ithuba materials located within a day’s drive of Pietermaritzburg, the capital. • “Slow down” and strike called by teacher’s union: Teachers’ strikes and slow-downs that were taking place during the early part of our work in-country made it extremely difficult to schedule visits and meetings with education officials. The first two school visits were arranged via personal connections of a team member. • Limited ability to assess the storybooks: The team did not have copies of the readers in order to assess them formally. 4. FINDINGS AND CONCLUSIONS 4.1 Ithuba Achievements and Challenges The team visited three schools: one in a Pretoria township and two in rural Limpopo, each about 100 km in different directions (northeast and southeast) from Polokwane. As a general note, the three schools visited, including the small village school in Siseluselu located several kilometers off the main road north of Tohoyuando, were equipped with physical plant in good condition (including separate latrines for boys and girls) plus electricity and working computers, fax, printers, and copiers. Children and teachers seemed happy to be there, and it did not appear that the schools had been “primed” for our visits. The schools did have rooms intended for libraries and, in principle, had people – teachers or community volunteers – scheduled to serve as librarians. Ithuba can count among its achievements the following: TLMP Evaluation Report Volume II – South Africa 216 • Storybook/TLM development: Ithuba contributed significantly to the number of high-quality supplementary reading storybooks available in schools in the nine official African languages plus English and Afrikaans. Other books were available in libraries, but the Ithuba books were prominent on the shelves and invited interest with their illustrations and format. The Molteno representatives reported that schools without libraries set up reading corners or brought out Ithuba books from storage to use during reading time. • Use of TLMs: It appears that the storybooks are being used as they were intended, as supplementary readers—at least for Language Arts. Ithuba books were found on the Ithuba stands, in cupboards that could be rolled to classrooms as needed, on library shelves and in classrooms. While we could not tell how regularly the TLMs were used, from their physical appearance and from the classroom observations it was clear at each school visited that various titles were in active use, by groups of students and/or by individuals. Although intended for the higher grades, we were informed at different locations that Foundation phase (grades 1-3) class teachers also used the storybooks, mostly for reading to their students. • Contribution to improved reading ability: Virtually all of the students we observed in grades 4 through 6 who were asked to read, whether chorally or individually, were able to do so with accuracy and fluency. While there is no way to tie this directly to the Ithuba storybooks, there was evidence that they contributed: for example, a Molteno representative reported that she saw students reading technically difficult graphemes (like consonant trigraphs in Sepedi) in the Ithuba books with comprehension and fluency, and they were able to re-tell stories orally or in writing, as called for in the Ithuba teachers’ guides. The team noted that a “typical” behavior in choral reading was to read slowly, with brief pauses between syllables, ostensibly so that everyone can keep up. Even when this was done, the overall intonation and clear stops between sentences or ideas demonstrated that children understood what they were reading. When individuals were asked to read, they demonstrated their fluency and even tried to read quickly; at times it seemed competitive. It was common for teachers to ask basic comprehension questions, which most students answered, thus demonstrating they understood the passage. Some teachers also asked students questions which called on them to extrapolate from the passages and develop their own perspectives, including engaging in ad lib role playing. • Building teacher competencies to develop reading materials: Enlisting and training teachers to become authors was innovative. The Ithuba author workshops reportedly raised the capacity of teachers and trainers to be authors while motivating them to write actively. The teacher trainers/educators in particular, who were part of the piloting of the methodology, were reportedly inspired and their capacity raised, so much so that some went on to inspire others. The hope throughout the process was that educators’ own writing would inspire students to write, which will in turn develop literacy skills in all languages, and at the Onane school, students volunteered their desire to write their own materials. Some teachers continued to write stories for their classes; the Molteno representatives said that teachers would approach them saying “I’ve written another story; how can I publish it?” • Building the capacity of illustrators: The project also reportedly raised the capacity of a smaller group of people to be illustrators, and judging by the high quality appearance of the books we saw the illustrators did good work. We were not able to speak to any of the illustrators, but we were told that there was at least one illustrator who became highly motivated to train others and that there is now a growing body of South African illustrators capable of illustrating commercial textbooks. • Expanding the approaches used and competencies built by Molteno and READ: According to Molteno representatives, the writing workshops were highly successful in getting teachers motivated and active in their writing; however, the materials needed a lot of reworking. The process of “versioning” worked well from their perspective as well as from the UTSA TLMP Evaluation Report Volume II – South Africa 217 implementers; this involved having the teachers write an English version of their home language story, then re-working them with experienced professionals to shape the story by applying materials development standards. • Sustaining the ideas and practices of Ithuba: UTSA informed the team that writing centers were developed in Limpopo and Gauteng that adhered to the ideas and practices developed by Ithuba. At the end of the project, the laptops used were donated to these two centers to facilitate the writing process. The practice was also implemented in writing camps for students. This notwithstanding, the team found no evidence that these writing centers still exist, at least not in their original form. • Further extension of Ithuba storybook development: In one case, a library specialist at the Limpopo DOE convinced her colleagues to develop storybooks in Sepedi (with a plan to extend the program to the other official African languages) for the foundation phase. They gained the support of Irish Aid to develop mock-ups and field-test them in the schools, but the storybooks have not yet been printed for financial reasons. Room to Read was in the process of incorporating Ithuba into its library programs. At the University of Limpopo, members of the English Department were inspired by prior involvement in Ithuba and in the Writing Center on campus. They sought funding from Belgium and created the “Multiple Literacies Project,” which produces high quality storybooks in home languages using the Ithuba model. • Continued use of Ithuba TLMs under curriculum reform: The Curriculum and Assessment Policy Statement (CAPS), the new curriculum policy, emphasizes early literacy in the mother tongue and literacy across the curriculum. While we do not have “hard” evidence for the consistent use of Ithuba materials throughout the provinces, we do have anecdotal evidence that they are being used, e.g., a Molteno representative said she was pleased to visit a multigrade class in Mpumalanga for another reason and the teacher was using Ithuba books in Ndebele, and in a telephone interview, the READ provincial officer for Mpumalanga confirmed to the team that Ithuba materials are still actively in use in the province. The singular focus on developing reading materials in home languages also enhanced the relationship between READ and Molteno. TLMP also experienced some challenges: • Sustainability issues within the DBE and USAID/SA: The biggest challenge, with strong implications for sustainability, was raised first by the USAID/SA education team at the in-brief: Why hasn’t the DBE taken on TLMP/Ithuba as its own, and/or expanded on the model for developing these much-needed storybooks in the nine official African languages? The USAID/SA education representatives gave this as justification for not extending the project or using its models or processes, despite the great need for storybooks in African languages. When we raised the question with Molteno representatives, they said that they had worked with the DBE which “should have had ownership and kept it going… but there was a break.” By “break” they were referring to a period of reorganization of DBE/DOE departments; this was confirmed by the DBE representatives from the curriculum department with whom we spoke, who said they had been “consumed” by the reorganization until quite recently. They were very familiar with the project; in fact, there was an Ithuba stand in the office where we met, and they were quite positive about the storybooks based on written and anecdotal reports they had been given. They also recognize the Ithuba storybooks as consistent with current curricular aims, both for home language and for “first additional language.” Because commercial publishers are “not coming up with readers,” they would like to use Ithuba as a “springboard.” Their question to us was: “Now that we are organized to support early literacy, is there any chance of re-examining the project and/or using it as a model?” This suggests that USAID would do well to re-visit the project and facilitate DBE ownership along with the NGO partners. • Uneven provincial inclusion of Ithuba: We had planned to have a team visit to KwaZulu-Natal. However, despite significant efforts we were unable to identify anyone at the KZN DOE with TLMP Evaluation Report Volume II – South Africa 218 knowledge of Ithuba, nor were we able to identify any schools within a day’s drive of Pietermaritzburg where Ithuba materials were in use. The unevenness of Ithuba involvement in KZN was explained to us by an NGO representative as a result of the high turnover at the DOE during reorganization. • Lack of equity in distribution of titles across home languages: While each language had eight titles of “group readers,” there was significant variation in the number of “independent readers,” e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga. • Pedagogical issues: Another challenge relates to the pedagogy used in working with the storybooks in the classroom. “Working” is the key word here, because most of the sessions that we were shown with the storybooks were heavy on the “didactic” side and less on the “enjoyment” side. In at least two cases, the students were not allowed to read to the end of the stories, although we could see that some students were reading ahead on their own. There was little silent reading in the classes that we observed (possibly an artifact due to our presence), except in a lesson where the teacher (a head teacher who had stepped in to demonstrate use of the storybooks for us) had given them comprehension questions and asked them to write the answers in their notebooks. Teachers generally did have the idea of letting students read aloud independently as well as together, although there were occasional vestiges of traditional methods like reading first and asking students to repeat. It was not possible to determine if the less traditional approaches taken by teachers were a result of Ithuba or other inputs—or a combination. • Regular and consistent use of the TLMs: Adding to the challenge of methodology, it was difficult to see if teachers used the books regularly, because the sessions we saw were “staged” for us. We were told at each school that teachers used the books, but there was no instance where a regular schedule of storybook reading was reported. Apart from the classroom observations which indicated that students were familiar with the readers, the main evidence we found that the books were in use was where the school directors were positive about Ithuba and where the books were displayed prominently in teachers’ rooms or libraries. Finally, despite assurances from the DBE and NGOs, we do not know if teachers themselves perceive that the storybooks are relevant to the current curriculum (CAPS). Even if they are, the references to the curriculum in the manuals would require adapting/updating/re-editing. • So many books, too few copies: As supplemental readers, just 20 copies of each title were provided to each school in its LOLT. However, as the books were very attractive and the stories were well contextualized, teachers might have wanted to use them more often as part of their reading lessons. Teachers generally had students share the books in pairs during classroom time, which seemed to work – for the size classes that we observed.9 However, if classes were larger, there would not have been enough copies for effective sharing, and 20 copies per school for a reader is not enough to make it possible to let students bring them home, where they could be used to broaden household use10. • Teacher’s guides getting lost: The teacher’s guides were developed as separate sheets (in strong cardboard) for each storybook; they were not bound in a single volume. We did not see the guides at all of the schools we visited, so we imagine that some sheets were lost or remained with some teachers, leaving others without expert guidance in using the storybooks. • Language mismatches: In some instances about which we heard, the home language spoken by the teacher/author was not the same as the home languages spoken by learners at the schools. 9 South Africa Government policy is 40 learners per primary school class. SAQMEQ Policy Brief #2, The Quality of Primary School Inputs in South Africa (September 2011). We cannot speak as to the extent to which actual enrollments reflect the policy. 10 One “work-around” would be for teachers to be working with more than one reader at a time, with students alternating. We do not know if this had been proposed. TLMP Evaluation Report Volume II – South Africa 219 • Distribution issues associated with relative number of speakers of a language in a school: Generally, with respect to the schools that we visited, books, including Ithuba materials, were distributed in proper numbers for the languages of the students at a school, although there could have been changes in demographics taking place between the time that books were requested and the time they were delivered and needed. (We saw at one school a chart of the number of students speaking a particular language by grade.) While at one school, we saw that the DOE had not reclaimed surplus copies of regular textbooks, we were told by the local implementers – as evidenced by our observation at the Mamelodi township school – that they took efforts to reclaim and redistribute Ithuba materials that were not needed at particular schools. • Selection of Dialects: This is a general issue associated with use of home languages for instruction, and not linked specifically to Ithuba. Various local languages have different dialects, which in some cases include borrowings from other languages. There are not necessarily broadly accepted “standard” dialects of a language, and generally materials were developed in the dialect spoken by the author(s). This can cause a pedagogical problem for children (and teachers) from different locales unless teachers are informed to let their students know that there are dialectal differences and that one dialect is not a priori better or worse than a different dialect. • Leveling issues: “Leveling” refers to the use of vocabulary and syntax assumed to be appropriate for the age level and competencies of the children for whom the materials are designed. According to Molteno representatives, it was a challenge to develop readers at the appropriate levels for students in grades 4-6 in the nine official African languages, as little had been documented on which home language literacy skills were developed at each grade level. Stories were written and edited by experienced teachers, but ensuring that they could be read by students in grades 4-6 was problematic. Selected teachers/authors attempted to field test the materials in their own schools, but the literacy levels achieved in each school varied. Hence, some of the books hit the mark but others missed in terms of level. On the other hand, at two schools we heard that learners as young as grade 2 were also using the Ithuba materials – primarily because there were no local language materials for their grade levels, so their uneven levels may have been a benefit rather than a hindrance to literacy learning. • No exchange of materials across languages. Combined, the Ithuba authors prepared some 130 readers in one or another of the languages of instruction. However, we were told that the DBE had not permitted materials developed for one language to be adapted for use by speakers of other South African languages – even though the materials would have been far more relevant to the lives of South African children than materials from New Zealand, which were not subject to such constraints. • Writing Centers not sustained: Ithuba had helped to set up writing centers for Limpopo and Gauteng. However, we were not able to find any legacy of them. • Need to plan for scale-up: We were informed by the Mission that DBE is unwilling to permit new pilot projects unless there is funding already planned for meaningful scale-up. While this obviously represents a challenge, it should be noted that there is demonstrated potential for at least some leveraging from, e.g., Irish Aid. • Printing issues: According to UTSA, some of the responsibilities that Uniprint had to assume (building book stands, “kitting” (packaging) materials in specific languages to be delivered to specific provinces, responding to technical requirements, delivery of materials to deep rural areas, etc.) demanded particular attention as to the timeliness and correctness of delivery. As a result, some language-specific kits were inappropriately assembled because little quality control was exercised. There were also differences in understanding as to delivery of materials between the printer and education authorities. UTSA found that monitoring the printer was a major challenge due to the lack of consistent in-country presence. Despite substantial efforts, the team was not able to glean any information from current representatives of Uniprint regarding Ithuba, who claimed that they did not have records. TLMP Evaluation Report Volume II – South Africa 220 4.2 TLMP Output By the time the project ended, TLMP had produced 2.3 million storybooks, teacher’s guides and Overview guides, and distributed them to more than 2,500 schools in each of the nine provinces, providing slightly less than 1,000 volumes to each. A total of 140 titles was developed, some of which were subsequently translated into South African English and/or Afrikaans. 140 educators had received training in writing stories, and 8,156 teachers had been trained in the use of the TLMs (largely by READ and Molteno). 4.3 TLMP Impact Many of the items noted as achievements under Section 4.1 above provide the basis for the impacts observed and discussed in this assessment. • Lived experience as the basis of stories: Educators were taught how to write stories based on their own lived experiences; the stories were of significant interest to the students; the hope was that teachers would encourage students to write their own stories. Several informants representing different categories of respondent told us that this experience greatly empowered many authors, e.g., through increasing self-esteem and community and professional recognition, and various educators continued to prepare stories. There was also perceived sense of pride, almost “bragging rights,” by students that their teachers had developed materials that were actually being used in schools. • Reading is a life-changing activity: As the TLMs addressed cross-cutting issues, including various health topics, one educator reported having stopped smoking because she learned of the consequences of that activity. Many of the themes were real to students, including how children’s lives have been impacted by HIV/AIDS. • Broadening perspectives and determining alternatives to cultural stereotypes: A frequent response from students when we asked why they liked the Ithuba materials was that they gave the students perspectives on how people in contexts similar to their own faced issues. According to the implementers, there were a number of stories that challenged assumptions about age and wisdom, male and female roles, body image/old and fat vs. young and thin; and dress. We were not able to verify how teachers used the storybooks to engage learners in discussions, but if the storybooks are being read, we know children are being exposed to broadened perspectives as Ithuba aimed to do. 5. LESSONS LEARNED • Create good relationships with DBE from the outset: According to MSI implementers, many of the successes of the project emanated from the good relationships established between UTSA and the DBE. Skeptical at first about the methodology, DBE officials were ultimately convinced that it was sound for developing supplemental readers. Without the professional approach of TLMP leaders and their consistent professional interactions with DBE, TLMP may not have taken place. • Identify divisions of responsibility: Especially in the context of rapid and/or frequent changes in organizational leadership and structure, curricular philosophy and content, and the like, it is important to identify – and keep track of – “who does what?” In a system like South Africa’s, in which provincial DOEs currently seem to have a fair amount of autonomy but not necessarily much in the way of resources, this knowledge can be quite beneficial when planning for implementation, scale-up, replication, and sustainability. • Writers could also have been editors: From UTSA reports, it was noted that once the South African team members had developed skills in estimating reading levels and judging appropriateness of stories, they could have been relied on to do all of the editing, rather than sending materials to TLMP Evaluation Report Volume II – South Africa 221 UTSA. This increase in capacity was confirmed by a local implementer, indicating one specific area in which Ithuba achieved its goals. However, the ability of South Africa to benefit from this increase in capacity depends on the extent to which it makes use of teacher-developed materials. • Ithuba needed a workable M&E system: Representatives of local implementers noted the lack of adequate follow-up, which hampered ability to analyze if and how the materials were used, any apparent concerns and feedback from teachers on their own perceptions and those of their students as to the readers as a whole or individual readers, and apparent contributions to student learning. • “Champions” with likely continuity are needed. We were told by a local partner that senior level education officials were committed to Ithuba at inception. However, over the course of time and personnel changes and practical necessity, other activities took higher priority. Based on very limited evidence, it appears that where Ithuba found sustainability after the end of USAID funding, it was due in large part to the commitment of the local partners to the project and of individual educators who learned from their own participation how effective and empowering the activities were for themselves and for learners. This suggests the great desirability also of creating a “critical mass” of participants who can act as advocates. This appears definitely to have been the case in Limpopo and at least to a quite noticeable extent in Mpumalanga and the environs of Pretoria and appears not to have been the case in KwaZulu-Natal. 6. RECOMMENDATIONS The following recommendations are intended both to inform USAID/South Africa for its education efforts and to inform the Africa Bureau and USAID education officers planning activities involving the development of textbooks and learning materials for literacy development projects in basic education. These recommendations are based specifically on perspectives from the Ithuba evaluation team. While a number of issues have surfaced in various implementations of TLMP, we have not included recommendations that do not have some identifiable basis from Ithuba. (General observations from the range of TLMP implementations are found in the synthesis Project Report.) 1. (USAID/SA) Engage in dialogue for continued use of Ithuba materials. There is both a real and a perceived need for supplementary reading materials to promote literacy in the 11 languages of instruction in South Africa’s primary schools. The American taxpayer has already invested millions of dollars in the development and production of some 140 supplementary materials in the languages of instruction for grades 4-6. These materials are, we have been told, consistent with the new curriculum guidelines, they seem generally well developed and are well received by the South African educators and students whom we have met, and other entities have been willing to provide some leveraging. Further, they are branded as coming from the people of the United States and have the potential of being seen and used in almost every household with a school-age child in South Africa. We strongly recommend that the Mission engage in dialogue with the Department of Basic Education (DBE) with respect to updating the materials and having them incorporated as supplementary readers for current education initiatives. 2. (USAID/SA) Consider using the Ithuba materials development model to develop new storybooks for early grade literacy (in South Africa, the Foundation level): Involving teachers and other educators in the story writing process was designed to create long-term impacts in teachers’ approaches to reading and writing as well as a means of getting quality materials produced. Literacy (in home languages) has been identified as a priority by DBE in South Africa and by other national ministries of education. Having students also participate in the story writing process will keep them engaged in learning and contribute to the creation of a reading culture. Writing and reading are mutually-reinforcing skills, so students who are engaged in both will more likely learn more efficiently and reach higher degrees of literacy sooner. TLMP Evaluation Report Volume II – South Africa 222 • While each language had eight titles of “group readers”, there was significant variation in the number of “independent readers” in unequal numbers, e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga. • In addition to providing speed in production and economies of scale, we believe (from professional knowledge, not directly from TLMP) that having some familiarity with the lives of citizens from other parts of the country can help to instill a sense of national unity. 3. Identify the likelihood of curricular and/or policy changes taking place during the life of the project and attempt to act proactively: This is obviously easier said than done, but early engagement to the extent possible can reduce the possibility that materials may be obsolete soon after they will have been developed. 4. Selection of implementers for future projects like TLMP: Implementing teams for projects that include TLM development should have the following characteristics/ skills: knowledge of the process of TLM development;11 knowledge of the national (i.e., South African) context; understanding of how current materials satisfy (or not) policy or curriculum requirements; knowledge of how story writing and other activities promoting literacy can be integrated into existing teacher training mechanisms; understanding of the sociolinguistic context and the language capabilities of teachers; understanding of the teaching methodologies currently used to promote reading and writing in the classroom; knowledge of the printing and dissemination process utilized by public and private agencies; and knowledge of piloting or other testing of materials to determine the actual (as opposed to notional) capacity of learners. 5. Provide teacher training workshops on the use of materials produced to all teachers and administrators involved in early grade reading: Whenever materials are produced, regardless of the ultimate goal, all teachers and administrators involved in their use or in the support of their use must be oriented regarding how to use them. Inherent in teaching any subject is the methodology used to facilitate the acquisition of knowledge. The TLM storybooks and teacher’s guides exposed teachers to alternative teaching methodologies that were more student-centered and participatory than many of those currently in use. To make these methods concrete rather than suggestive, teachers and administrators must be trained on their use as well as the pedagogy involved. To ensure the use of the TLMs and the training of teachers to use them, TLMs should be incorporated into the teacher training college pre-service and in-service curricula as part of a systematic approach to reading and writing in the home language. 6. (USAID/SA) Encourage the adaptation of materials that have been proven valuable in one language for students who speak other languages. There will likely continue to be significant shortfalls in materials in local languages for students to read. While there will be some duplication in content across some of the Ithuba materials that have been prepared for different languages, there are nonetheless unique materials that can be readily adapted for children who speak other languages. • While each language had eight titles of “group readers”, there was significant variation in the number of “independent readers” in unequal numbers, e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga. • In addition to providing speed in production and economies of scale, we believe (from professional knowledge, not directly from TLMP) that having some familiarity with the lives of citizens from other parts of the country can help to instill a sense of national unity. 7. Bind teachers’ guides into single pamphlets: The teachers’ guides for the Ithuba readers seem to be about the right length – four-page leaflets. However, because they are leaflets, we found that in various cases at the schools we visited, they had “strayed.” We recommend that for Ithuba (and 11 We recognize that for TLMP, development of the skills, if needed, by the MSI was one of the objectives. TLMP Evaluation Report Volume II – South Africa 223 for comparable activities), these leaflets be bound into single booklets that also incorporates general guidance on how teachers can use supplementary readers in literacy/reading classes. 8. Consider the use of Annual Program Statements (APS) to complement ongoing education activities. It may not be workable for an education activity to incorporate the development of learning materials as part of its work; e.g., its primary focus might be on teacher training or educational administration. However, this does not obviate both the need and desirability for learning materials. An APS would give local entities, possibly in collaboration with U.S. partners, the opportunity to build its own capacity and the capacity of local educators to develop, produce, distribute, and use materials on, e.g., a provincial level with the possibility that they could develop approaches novel in the country to any of these aspects, maintaining appropriate interactions with other development partners; it could also provide opportunities for facilitating leveraging. 9. (USAID/SA) Support local reproduction of materials. It may not be feasible to launch large￾scale press runs to reproduce particular readers. However, the schools that we visited, even in remote areas, were equipped with functioning copiers, and presumably district offices also have them. The TLMP cooperative agreements allow for local reproduction and adaptation of materials. Provincial DOEs should be informed, via DBE, that if a school or district is short some copies of a particular reader, it is free to make needed copies on its own, in color if possible. 10. If consistent with national textbook dissemination policies, engage local private sector publishers/printers in the development, printing and distribution of materials: Given the possibility that some households might wish to buy their own copies of materials, private sector printers can help to ensure that there is a steady stream of materials to stock both the public and private sector. 11. (USAID/SA) Encourage more effective use of libraries: The schools that we visited had stocked libraries staffed by teachers and/or community volunteers. They were in use, and books seemed to be used. However, at least for Limpopo, each of the DOE personnel responsible for guiding libraries is responsible for well over 1,000 schools. It would be useful if the Mission could arrange for the development of training that would help school librarians learn how to promote use of the library as a lending library since improving the ability of students to take books home would provide for reinforcement of learning, the strengthening of a culture of reading, and the possible engagement of family members in improving their own ability to read. TLMP Evaluation Report Volume II – South Africa 224 ANNEX A: Map of Limpopo Province TLMP Evaluation Report Volume II – South Africa 225 ANNEX B: Schedule of Stakeholder Interviews and Activities FULL TEAM Mon 5/6 In-brief with USAID in Pretoria • Meredith Fox, Education Team Leader • Peter Cronin, Education Foreign Service Officer • Nalini Reddy, Project Management Specialist – Education Tue 5/7 Meeting with Molteno Institute for Language and Literacy at Sandton Paula Gains, Research & Development Manager Johanna Mogodiri, Provincial Coordinator Freddy Nambahe, National Training Manager Wed 5/8 School visit to Pfundzo nde Tshedza Primary School (Sepedi with some Venda, Zulu and Tsonga), Mamelodi township, Pretoria, Gauteng province: classroom observations, discussions with faculty, visit to library Afternoon travel to Polokwane Thu 5/9 School visit to Siseluselu Primary School in rural, Venda-speaking part of Thohoyandou district, Limpopo province: classroom observations, discussions with faculty, visit to library Fri 5/10 Meeting at Limpopo Provincial Department of Education • Ms Motlalepua Teffo, Provincial Coordinator, Media & Library Services • Head, Media & Library Services ED and Ms. Teffo Onane Primary School, Hlogotlou-b, Monsterlus, Limpopo Province http://www.onaneprimaryschool.co.za/ • classroom observation, discussions with students, library visit • Sihlangu Solomon, Deputy Principal Carol and Rakgadi University of Limpopo • Former writing center at Dept of Languages, School of Languages and Communication Studies: Mr Mpho Seerane, former student of Dr Leketi Makalela • Meeting at Dept of Languages: Dr Rose-Marie McCabe, Ms Lehlogonolo (Lelo) Mkola, Ms Bongiwe Nomatsorane, Mr Noel Manganye about Ithuba and follow-up project funded by Belgium • Sepedi Department: Dr Mamalatswa Maruma • Contemporary English and Multilingual Studies program: Dr Esther Ramani Final meeting with Ms Teffo back in Polokwane Sat 5/11 Travel back to Pretoria, discussion of findings WHOLE TEAM Mon 5/13 Meeting with University of Pretoria linguists • Prof Vic Webb • Prof Michel Lafon READ Educational Trust • Ellen Dlamini, Mpumalanga Provincial Director (via telephone) Discussion with education professor Lilli Pretorius (CB) Tue Work on notes and debrief TLMP Evaluation Report Volume II – South Africa 226 5/14 Phone interview with Carole Bloch and other colleagues involved with early literacy (CB) Wed 5/16 South Africa Department of Basic Education • Jennifer Joshua • Kahlula Manona Work on synthesis of South Africa visit Thu 5/16 Debrief with USAID/South Africa: Meredith, Peter and Nalini Meeting with READ Educational Trust • Berthus Matthee, National Director, Evening departure for U.S. Later Follow-up e-mails with Paula Gains, Misty Sailors, Leketi Makalela, Michel Lafon This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by James Wile (Team Leader), Carol Benson, Richard Chediel, and Cathleen Sekwao under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania Evaluation Report - Tanzania Intentionally left blank TLMP Evaluation Report Volume II – Tanzania 227 EVALUATION REPORT - TANZANIA EXECUTIVE SUMMARY Evaluation Purpose and Evaluation Questions The purpose of this evaluation is to assess the impact of the Teaching and Learning Materials Program (TLMP) as it was implemented in mainland Tanzania from 2009 to 2012. A research design was developed to explore the way textbooks and learning materials (TLM) were created, distributed and used in Tanzanian classrooms. The operational assumption was that the process of developing and disseminating quality teaching and learning materials for secondary school math and science would increase students’ access to texts, help to improve instruction and lead to improved student learning outcomes. It was also hypothesized that this activity would build capacity of the contractor—South Carolina State University (SCSU)—as well as the Ministry of Education and Vocational Training (MOEVT) in Tanzania, local publishers/printers, and classroom teachers. Project Background SCSU had facilitated the development of TLMs for secondary school math and science for Zanzibar from 2005 to 2008. The MOEVT determined that the TLMs should be adapted and revised for use on the Tanzanian mainland, where they were intended to be distributed to approximately 1500 government secondary schools. SCSU worked in collaboration with the MOEVT and its curriculum specialists at the Tanzanian Institute of Education (TIE), the Educational Materials Approvals Committee (EMAC), and local publishers to produce and distribute over 1.25 million textbooks and accompanying Teacher Guides to schools across mainland Tanzania, ultimately reducing the pupil-to-textbook ratio to 1:3 in some subject areas and 1:1 in some grade levels by the end of the project on December 31, 2012. Evaluation Questions, Design, Methods and Limitations The evaluation was framed by a series of research questions. These include the following: • How well did the intervention achieve its program goals and expected impacts? • What were the results achieved for each partner country in relation to intended program targets, as well as to standardized and variable indicators, by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based MSIs, in-country institutions, the MOEVT, student achievement and teacher performance? • To what extent were in-country institutions (with support from U.S.-based MSIs) able to deliver services effectively in terms of coordinating materials design, curriculum alignment, production, and distribution? • What was the overall efficiency of allocated USAID funding in terms of usage and overall cost effectiveness? • What were the specific program accomplishments/outcomes as a result of the partnership between SCSU and its counterparts in Tanzania? • What lessons can be learned from this intervention program, and what recommendations can be made for potential program scale-up and/or replication as related to the new USAID Education Strategy? The design of the evaluation study consisted of three main components: 1) an extensive review of documents including project formation, internal (SCSU) operational documents, formal and informal reports, and the TLMP products; 2) interviews conducted at SCSU; and 3) a two-week field study in which two teams of one international and one Tanzanian education expert reviewed the processes, products, and outcomes of the project. The logic of this approach was that by triangulating these perspectives the data that would emerge would be both accurate and nuanced. TLMP Evaluation Report Volume II – Tanzania 228 The methods used to collect data combined structured interviews and structured performance observations. The evaluation team followed a comprehensive set of interview questions tailored to the key informants at SCSU, the education institutions and schools in Tanzania, publishers, and other stakeholders. The performance observations consisted of structured classroom observations of teachers using the TLMs and an informal reading assessment of students’ ability to read and understand the materials. Other data collection techniques included focus group discussions with classroom teachers and students. The team visited regional and district education offices as well as school storage rooms and libraries to see how books were stored and distributed. Additional data were collected through a rigorous review of documents pertaining to the inception, implementation and outcomes of the TLMP activity. As with any research study, this evaluation was confronted by several limitations. Chief among these was the challenge of meeting with key people involved in the project. Some had died, and some were unavailable because they had retired or been relocated, or were simply unavailable at the time of the study. In such cases, the team collected data from deputies, assistants, or individuals who were new in their positions and had no personal experience with TLMP. A second limitation was that much of the data relied on personal recall of meetings, workshops, and events that had taken place 12-24 months earlier. The lack of agenda records, meeting notes, and similar documents proved challenging when attempting to re-create the process in which TLMs were developed. A third limitation was the lack of any pre-intervention assessments or baseline data to compare/contrast against the final project outcomes. Classrooms could be observed to see whether teachers were using the TLMs appropriately, but except for teachers’ self-reports there was no way to compare the teaching/learning dynamic before and after the introduction of TLMs. Finally, as educators in Tanzania pointed out, it was too soon to be able to inquire about the impact of TLMP textbooks on student learning, because the materials had not been in use long enough to conclusively support such findings. Findings and Conclusions: The team found that TLMP on the mainland was generally successful in producing good quality textbooks that were relatively well aligned with the curricula for secondary math and science and that these TLMs met Tanzanian standards for cultural and political appropriateness. Although budget cuts eventually prevented SCSU from reaching the stated production target, in many cases the intervention dramatically reduced the pupil-to-textbook ratios at the recipient schools, and in some instances brought about a 1:1 pupil-to-textbook result. The team also found that SCSU made good use of the educational systems in place in Tanzania. Relying on the MOEVT’s subject area and curriculum experts at TIE, along with the adoption of MOEVT’s procedures, the program was able to achieve its objective of producing approximately 1.25 million TLMs. This process was aided by engaging the expertise of local commercial publishers and printers, in this case Uhuru Media (UM) and Oxford University Press (OUP). However, USAID budget cuts in the last two years prevented the project from proceeding with its plans to put a textbook in every pupil’s hands. Data and self-reports confirm that capacities were developed by all participants involved in the production of the TLMs, including SCSU faculty, MOEVT staff, and the staff at the commercial publishers. The team found that the TLMP textbooks and teacher guides were not having the degree of impact on teaching and learning desired. Although it may be too soon to conclude how teachers will eventually use these resources, the results of observations, interviews, and focus group discussions suggest that many secondary school teachers consistently use traditional lecture methods to transmit content TLMP Evaluation Report Volume II – Tanzania 229 information to students. No orientation or training was given to teachers with regard to using the TLMs. The research indicates that intensive and continuous professional development would be required to help teachers manage the paradigm shift from traditional teaching methods to critical thinking, problem￾solving, and application (which TLMs appear well-designed to support). The team found that the USAID/W-based management strategy used for TLMP implementation was not the most efficient or effective. The centralization of management in Washington in tandem with the relative inexperience of SCSU in international development did not guarantee optimal outcomes. Cultural and geographic distances between Washington, Tanzania, and SCSU created insurmountable challenges. Inefficiencies in communication and monitoring led to misunderstandings and inefficiency, particularly in the case of materials development. There were missed opportunities for utilizing the flexible nature of the collaborative agreement structure to build new relationships, strengthen local capacity, and to promote mathematics and science knowledge on the part of authors and teachers. Had USAID/Tanzania (with its extensive knowledge of local context and its personal and professional relationships with potential partners) been bought into the planning and project design more significant outcomes could have been achieved. Lessons Learned • Politics, culture, and international development: Politics is endemic to the entire process, and this had to be negotiated all along the way. One SCSU interviewee commented, “A full-fledged country study was required before we moved ahead on the proposal so as to be prepared for what we would see when we actually visited. In preparation for such a trip, we needed cross-cultural training so that we could interact with people locally with a degree of cultural sensitivity. Some training in language would also have been good.” • SCSU Leadership: Instability in SCSU leadership over the life of a multi-year program created a context in which, as one informant responded, SCSU TLMP leadership was constantly explaining the purpose of the project and the rationale for the college to support it. Without this stability, working with USAID was not fully institutionalized and the close working relationships with the MOEVT on the mainland were not established as they had been in Zanzibar. • Confusion in Authorship and Copyright: TLM authorship was to have been a joint activity between designated individuals of the MOEVT/TIE and SCSU; however, the Project Coordinator’s name appears as the author on one of the textbooks, and the copyright was maintained by SCSU, making reproduction of the materials extremely difficult. • Operations: Having an on-site, in-country coordinator would have facilitated key relationships. In addition, according to one interviewee it “would have been better” to have created the original texts for the mainland and then move to Zanzibar. • Pedagogy: Teachers were not very prepared to teach their subjects at the assigned grade level using the TLMs as they called for a different approach to teaching. Students are now learning from 21st Century textbooks; however, science kits are needed, teacher evaluations of the use of the books are needed, and a plan for the local team needs to be developed to carry on with the process. • Language: Teachers and students do not have the English competence to work in the level of English required by each of the textbooks. An English as a second language (ESL) approach may have been more effective rather than the assumption that students would be able to work in English at the same level of native speakers. • The value of systems: TLMP in Tanzania made great use of the existing systems within the national education context. Rather than operating as a project outside the national system, the mainland portion of TLMP was managed and supported largely by well-established Tanzanian systems for textbook revision, approval and distribution. This resulted in a cost savings and provided a measure of ownership for MOEVT including at the regional and district levels. TLMP Evaluation Report Volume II – Tanzania 230 • The value of professional publishers: TLMP benefited from the contributions of professional commercial publishers. The success of the project seems in large part because of the appropriate use of publishers’ local technical expertise. In fact, had the publishers’ standard practices for piloting and training been followed, the TLMs might have been better adapted to students’ needs, and teachers better prepared to use the TLMs. • International publisher practices: OUP is a multinational corporation and as such can take advantage of resources available in other countries. OUP printed many of the TLMs in Kenya, where machinery, supplies, and electricity are more reliable. Thus, the Tanzanian printing industry, per se, did not benefit from TLMP. • The limitations of textbooks as change agents: Although SCSU and the MOEVT may have felt that the level of innovation incorporated in the TLMs was modest most teachers observed seemed reluctant to change teaching styles. In this case, teachers were familiar with lecture and recall of terms and concepts, and used the TLMs to highlight facts rather than to engage students in a discussion of the implications and applications of factual information. • The limitations of centrally-managed international development programs: USAID supervision from Washington was not the most efficient. The USAID/W office appeared to be too far removed from the SCSU and the MOEVT, despite the presence of the USAID office in Dar es Salaam. Critical oversight regarding budget, partnership development, and product review seemed to be lacking or poorly timed. Recommendations Materials Improvement • Address the lower levels of English language capability of secondary school students: Student difficulty in understanding the vocabulary and more complicated sentence structures used in the TLMs requires reconsideration of the level of English used in the textbooks. Future textbooks in any/all subjects must be solidly based on an English language assessment to guide the level of English competency required. • Conduct a more comprehensive review of the usability of TLMs: Since teachers themselves have varying levels of subject matter knowledge and know-how in building competencies in their students, a more comprehensive assessment of their ability to understand and teach the range of concepts incorporated in the TLMs should be undertaken to fulfill two goals: 1) improve instruction at TTCs; and 2) improve classroom delivery of lessons. As noted above, textbooks don’t teach; teachers do. Systems Improvement • Link TLMP to teacher professional development: To maximize use of the TLMs, an investment should be made to build on the MOEVT’s own initiatives to implement a professional development support program on teaching with textbooks. • Address copyright issues: An investigation should be undertaken to determine who actually owns the copyright to the TLMs so that, if so desired, they can be reprinted. • Support a large-scale evaluation of the impact of TLMs on student learning: USAID might consider supporting MOEVT and Tanzanian research universities to conduct a national assessment of the impact of TLMP and other textbooks on teaching and learning. • Create links between teacher education institutions and schools: To maintain continuous professional development and to further professionalize the teaching profession, a link between SCSU science and math educators and secondary school teachers in Tanzania via shared technological resources, particularly through USAID-sponsored Teacher Resource Centers in Tanzania, should be considered. This pairing might be extended to U.S. secondary schools with which SCSU has a relationship through student teaching and in-service programs. TLMP Evaluation Report Volume II – Tanzania 231 • Build school-based management of resources: USAID might consider developing support programs that build on the MOEVT’s school-based decision making and decentralization efforts. The formation of school-based textbook review committees with objective criteria for evaluating potential textbooks for adoption is one critical function. USAID’s support for secondary school directors and academic leaders might help the MOEVT to capture these exemplars and create leadership development models around them for future school-based decision making. • Develop a strategy to support secondary math and science: All TLMs should be provided to TTCs so that they might be included in the curriculum in training new teachers, thus providing input on how to teach using a 21st Century textbook. Without such an input it is likely that students will not elect to study the sciences at a higher level. • Link to other secondary education programs: The TLMP experience – processes and outcomes – should be linked to other donor secondary school interventions. • Insure quality and accountability of textbooks at the school level: The same level of quality and accountability in the choice of textbooks in evidence at EMAC should be required at the school level. This will mean that the appropriate MOEVT divisions should provide training at schools (including directors and teachers) on how to make textbook choices. • Systematize the allocation of textbooks to schools that need them: As part of the process of identifying the number of textbooks needed in a particular subject at a particular grade level, the MOEVT should determine whether the subject is taught at that school and at that grade level so as not to provide books to a school where they are not needed. • Share TLMP evaluation processes and results with early grade reading project awardees: The lessons learned by SCSU in developing the TLMs should be shared with early grade reading project leaders. Even though the subject matter is different, there are useful elements emerging from the project that can be learned by others (e.g., copyrights, publishing, textbook review, etc.). USAID Improvement • Consistently apply performance monitoring procedures: USAID should require that all funded projects prepare a performance monitoring plan (PMP) and consistently conduct monitoring on project progress. In reviewing monitoring reports, included in quarterly reports, USAID should determine if the project is “on track” or whether approaches and activities need to be adjusted to obtain the desired results. • Build institutional capacity for less experienced US-based implementers: If USAID/W intends to earmark funds to expand capacities at U.S. institutions that do not have extensive experience in international development, it should anticipate the need for even greater oversight of their activities and ensure that such institutions are mentored so that capacity is gradually built over time. TLMP Evaluation Report Volume II – Tanzania 232 1. EVALUATION PURPOSE, QUESTIONS AND OBJECTIVES 1.1 The Structure of TLMP The Textbooks and Learning Materials Program (TLMP) was launched by the United States Agency for International Development in Washington, DC (USAID/W), in 2005 as a part of the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort to improve the management capacity of education sector personnel in a number of countries in Sub-Saharan Africa. TLMP was extended for another three years in 2008/09 to 2012. The overall aim of TLMP was to provide quality textbooks to African students. According to a TLMP White Paper produced by USAID, well over 25 million children have gained access to the textbooks produced through the project, which consist of more than 500 titles in 13 languages. According to the Cooperative Agreements (CA) awarded to each MSI, the materials produced were to be fully aligned with national curricula, to focus on primary education, to be culturally relevant and to integrate important cross-cutting themes such as HIV/AIDS, gender sensitivity and equity, hygiene and youth leadership. Each Minority-Serving Institution (MSI) was responsible for managing and implementing the TLMP in a specific country and with achieving specific outputs and results in accordance with its respective CA (the University of Texas San Antonio had been awarded two CAs). Originally a program to develop TLMs for primary schools, initial needs assessments conducted by each MSI determined that in three countries MOE priorities were for middle and secondary school TLMs. South Carolina State University (SCSU) was chosen as the MSI to develop textbooks in secondary school math, biology, chemistry and physics in English in Zanzibar from 2005 to 2008 and for mainland Tanzania from 2009 to 2012. This evaluation covers the TLMP implemented only in mainland Tanzania. 1.2 Evaluation Objectives This performance evaluation was intended to satisfy the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency (USAID) Education Strategy. 2. TLMP PROJECT BACKGROUND 2.1 SCSU Responsibilities SCSU was the recipient of two TLMP awards, the first addressing the need for secondary school textbooks in biology, chemistry, physics and mathematics in Zanzibar (2005-2008), and the second (2009-2012) adapting the Zanzibar materials for use on the Tanzanian mainland because of differences in culture, level of vocabulary/language, selected changes in the curriculum/syllabus, and incorporation of a competency-based approach. It was originally planned that during this period the expansion of materials for early childhood education in Zanzibar and in Mainland Tanzania would also take place, but USAID/W budget cuts prevented the production of early childhood TLMs. SCSU was the only MSI of the five to focus solely on the development of secondary school textbooks. TLMP Evaluation Report Volume II – Tanzania 233 At SCSU, the team was comprised of a Program Director, a Program Coordinator and Assistant Coordinator, and subject matter specialists in language arts, biology, chemistry physics, math, and education assessment and evaluation. The team was supported by a fiscal analyst who worked with the different SCSU departments in training and making logistical arrangements. Programmatic responsibilities of the leadership team included the following: • To establish professional teams (at SCSU and in Tanzania) to guide and supervise the development and production of TLMs for mainland Tanzania. • To select and supervise the professional development teams and other personnel that will develop and produce the TLMs. • To engage, support and supervise relevant and appropriate program activities and other activities associated with the development, production and delivery of the TLMs. • To accomplish the development and production of the TLMs so that the printing and distribution can take place within the timeframe. • To be responsible for the quality assurance of the TLMs. • To obtain approval from MOEVT in Tanzania and USAID in Washington, DC for the printing of the TLMs for secondary schools in the United Republic of Tanzania. • To approve and supervise the printing of the TLMs developed. • To monitor the delivery, storage and distribution of the printed TLMs. • To provide the appropriate and timely documentation, periodic reports, invoices, etc. associated with the development, production, printing, and delivery of the textbooks and other learning materials, as well as requesting timely reimbursement of expenditures of funds. The mainland adaptation program was a collaborative activity in which Tanzanian systems and procedures guided and directed the development, printing, quality control and dissemination of TLMs rather than the activities of SCSU. SCSU’s role was mainly to organize a local advisory committee, co￾facilitate several adaptation workshops, and manage the program budget. Zanzibar (2005-2008) – We present the following achievements for the Zanzibar portion of TLMP to illustrate that the bulk of the production work was accomplished during this period: • Development, publishing and distribution of 1,140,000 books: • 10 secondary student textbooks (some covering two grades) • 10 teachers’ guides • 3 laboratory/practical manuals • 20 titles of supplementary materials (posters and wall charts) • Strengthening of US/Tanzania education development partnerships • Achievement of a textbook-to-student ratio of 1:1 in Zanzibar • Training of 1,250 Zanzibari science and mathematics teachers • Awarding of Presidential scholarships at SCSU to 2 deserving Zanzibari students Project documents produced by SCSU assert that national test score averages in Pemba and Unguja (in Zanzibar) increased by over 30% since 2005, with TLMP materials a presumed factor. Mainland Tanzania (2009-2012) - The MOEVT determined that the TLMs developed for Zanzibar could be of better use on the mainland if they could be adapted to conform to the competency-based curriculum that had recently been implemented in secondary schools, and reflected a more inclusive culture. TLMP Evaluation Report Volume II – Tanzania 234 Adaptations were completed through a series of workshops that were co-facilitated by SCSU, the Tanzanian Institute of Education (TIE) and the Educational Materials Approval Committee (EMAC). Editorially-oriented workshops were facilitated by MOEVT/EMAC and the commercial publishers, Oxford University Press (OUP) and Macmillan/Uhuru Media (UM). The nature of these collaborations and the contexts in which they took place are detailed below. 2.2 MOEVT and the Educational System in Mainland Tanzania The formal educational structure on mainland Tanzania consists of two years of pre-primary education (KG1-2); seven years of primary school; four years of junior secondary (“O” levels); two years of senior secondary (“A” levels); and three or more years of tertiary education. Education is classified in three levels: basic, secondary and tertiary. Two ministries manage and coordinate the education sector: the Ministry of Education and Vocational Training (MOEVT) and the Ministry of Regional Administration and Local Government (MRALG) (which manages basic education). At the primary and secondary levels quality assurance is the responsibility of school heads, Ward/District Education Offices, and School Inspectors. TIE, a quasi-independent education agency, was responsible for producing all instructional materials up until legislation in 1995 when its operating mandate changed. Prior to that time (and perhaps more recently), TIE developed detailed curricula based on the goals expressed in the national education improvement plan. These curricular goals now serve as content guidelines for commercial textbook publishers to develop all manner of textbooks. The MOEVT provides further publishing standards through the criteria developed by the EMAC. Tanzania has a robust and competitive education textbook market, particularly in secondary education, where English is the current language of instruction. International English-language publishers (OUP and UM) maintain sales representatives and offices in Tanzania and in neighboring Kenya. Both of these publishers were responsible for producing the TLMs for Zanzibar and mainland Tanzania. TLMP Evaluation Report Volume II – Tanzania 235 The MOEVT has been engaged in a change process that includes four critical features: • Move to Competency-Based Curriculum: In an effort to move instruction from a tradition of lecture, note-taking and content recall, the MOEVT, TIE and the National Examination Council of Tanzania (NECTA) organized a competency-based curriculum with performance objectives accompanying content. Students are expected to learn not only the content of science and mathematics, but also to understand how to apply subject-matter concepts to real-life situations. Student evaluation is based on performance criteria set by the curriculum. This dramatic shift in the teaching/learning dynamic has left many secondary school teachers in Tanzania unprepared, but the math and science departments at MOEVT are working on ways to help them, including developing a manual on how to use local materials for Chemistry and a DVD about using textbooks for teaching, both of which refer to the TLMs. Many of the existing textbooks in schools did not reflect this competency-based curriculum; the TLMs were specifically designed to fill this need to support activity-based opportunities in which students can demonstrate competencies. Teacher self-guides referring to TLMs DVD self-guide for teachers referring to TLMs • Decentralization of Decision-Making: An MOEVT priority is the decentralization of educational decision-making to the district and school levels. MOEVT no longer mandates the use of a specific textbook or instructional resource. Instead, the policy calls for school directors and classroom teachers to decide on a main and supplementary textbook(s) for each course at each grade level from among available materials (including TLMs and those developed by commercial publishers or by TIE). Schools are given budgets to buy their own materials, so the donated TLMs have been well appreciated in the designated schools. Secondary schools rarely have the funds to buy enough textbooks for all students, but TLMP tried to provide a sufficient number for each student to have one textbook each. For supplementary materials, some teachers use the TLMs, some use textbooks donated from the U.S. or other English-speaking countries, and some use materials they have created themselves. Others do not have any textbooks, relying on the lecture method to transmit key subject content. • Obtaining Funding for Materials Dissemination: The MOEVT does not have a budget for disseminating instructional materials to schools (via regional and district education offices) as this is now the responsibility of publishers (TLMP leadership assumed MOEVT would deliver what the publishers had produced). The geographic expanse of the mainland presents enormous logistical challenges, especially during the rainy season when school principals had to travel up to 100 kms to pick up their TLMs. • Commercial Professional Development: The MOEVT has limited financial capacity to provide in-service professional development programs for teachers (INSET). Typically, INSET is funded TLMP Evaluation Report Volume II – Tanzania 236 through external donors. Commercial publishers provide INSET on the use of the texts they have produced to those who purchase them. • No Education Portfolio at USAID/T: At the inception of TLMP USAID/T did not have an education portfolio or Education Director to coordinate potential linkages between other USAID￾supported education activities and TLMP. Hence, the MOEVT did not have an opportunity to discuss TLMP and its ramifications in other educational development activities supported by USAID/T. The MOEVT had prioritized the need to enhance services to nearly 2000 “new” government secondary schools (established since 2006). Many of these schools were operating with donated materials from the U.S. or Europe, outdated textbooks produced by TIE, or no textbooks at all. These needs provided the direction for TLMs. 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS 3.1 Research Conducted at SCSU The research design included initial data collection on the program from the TLMP leadership team and faculty textbook developers at SCSU (see ANNEX A – Questions Posed of SCSU Project Team). 3.2 Field Research Activities The evaluation team conducted site visits in the capital and three regions: Dar es Salaam (Temeke District), Coast (Kibaha Town), Arusha (Arusha, Arusha City, Meru and Monduli Districts), and Morogoro (Morogoro Town and Rural Councils). These regions were specifically selected to represent urban, peri-urban, and rural, remote schools. Data were collected from MOEVT officials at the national, regional and district levels and at schools from directors, teachers, and students. Teachers were also observed using the TLMs in the classroom, and an informal assessment of student reading ability was undertaken. The following is a summary of research activities (for a full schedule of research activities, see ANNEX B – Schedule of Data Collection Activities): • 18 secondary schools visited • 22 school administrators interviewed • 19 classroom observations • 33 students given an informal reading assessment • 56 students participated in focus group discussions • 49 teachers participated in structured focus group discussions • 5 teachers/writers participated in structured focus group (Dar es Salaam) • 3 Regional Education Officers (REO) interviewed • 10 District Education Officers (DEO) interviewed • 6 MOEVT officials interviewed • 2 Tanzanian Institute for Education (TIE) (Curriculum) officials interviewed • 3 USAID-Tanzania Education Officers interviewed • 4 Publisher’s representatives interviewed A final component called for a team review of the TLMs in terms of their structure, readability, and coverage of the cross-cutting issues (see ANNEX C. Data Collection Instruments). TLMP Evaluation Report Volume II – Tanzania 237 Strong room for safekeeping of TLMs During the second week of field research, the team divided to ensure that each unit had one national expert to manage local contacts and language (Kiswahili) and to provide valuable cultural/historical/educational contexts where appropriate. The Monitoring and Evaluation (M&E) expert from USAID accompanied the Morogoro unit, while the Arusha unit was accompanied by a member of the Arusha DOE. Their roles were largely to observe. In Dar es Salaam, the entire evaluation team met with the MOEVT and TIE representatives directly involved in the project, and with the senior representatives of OUP and UM, the two commercial publishers who produced the TLMs. Visiting a makeshift school library Observing a chemistry lesson in Arusha 3.3 Limitations of the Research A number of contextual factors presented certain limitations on data collection. These included: • Changing personnel: Many MOEVT and TIE officials (as well as those in OUP and UM) who had participated in TLMP were not accessible as they had been replaced due to death, retirement, or change in sector. • Textbooks recently introduced in classrooms: Several TLMs had not been in the schools long enough for the team to collect meaningful data on impact on student learning outcomes; materials had not always been delivered to schools in a timely manner, although most deliveries had been completed by 2011. • Lack of baseline assessment: No baseline studies had been undertaken on the status of teaching and learning prior to the introduction of the TLMs. The team thus engaged informants in recollecting their perceptions of the materials when they were introduced and how these compared with their previous texts. • Passage of time: Much of the information obtained from informants regarding the processes of developing, editing, reviewing/revising the TLMs was based on remembering events that happened in 2009-2010, so it is understandable that some data could be missing from their recollections. • Logistical challenges: The Morogoro unit lost a day of travel due to their vehicle breakdown. While these meetings were important in providing key information, because sub-teams had to visit the respective Regional Education Officer (REO), District Education Officer (DEO), and the school principal/head teacher before visiting the classrooms, the amount of time available for observations TLMP Evaluation Report Volume II – Tanzania 238 was limited. This was especially challenging because mathematics and science are normally taught during the first periods of the school day. • Time/Sampling limitations: In the time available, it was not possible for the evaluation team to make a comprehensive evaluation of the schools that received the TLMs. 4. FINDINGS AND CONCLUSIONS 4.1 SCSU Achievements and Challenges SCSU has faced many institutional leadership challenges over the past several years. Despite internal setbacks, the TLMP management team moved ahead in addressing changes in the Zanzibari materials to make them more suitable for use on the mainland. TLMP was challenged with ongoing communication problems related to social/cultural factors as well as technical misunderstandings with their Tanzanian partners. The inability of SCSU to have its own permanent in-country field manager contributed to the challenges it faced in program management. The program also suffered from changes in Tanzanian mainland counterparts. For example, the Director of the TIE, the Permanent Secretary of Education and Chair of the Project Advisory Committee (PAC), and the Director of EMAC were all replaced during TLM production. With the exception of a series of initial writer’s workshops co-facilitated with MOEVT and TIE, SCSU activities during the mainland phase appear to have been largely managerial and supervisory. Additionally, OUP and UM assumed greater responsibility for revising their respective textbooks. Despite the administrative and budgetary challenges, the following were important achievements identified by SCSU: • International procurement: A new wire transfer policy was developed and implemented by the Controller’s office, including a Wire Transfer Request Form that enabled disbursement of funds to faculty and to pay invoices in a timely manner. The state procurement system was also revised so as to allow procurements in Tanzania and to exceed $5,000 without major university review. • Travel: Forms, advances and approvals times were simplified and shortened to accommodate a quick turnaround. • Internationalization plan: The current board chair has requested that the Director of TLMP develop a plan for enhanced international activities beyond Fulbright research grants and a limited education abroad program currently funded under Title III. • Exposure to African culture: SCSU students have met Tanzanian students and learned from them, as has faculty. • Teaching and Learning: Faculty participants have incorporated different teaching and learning concerns in their classrooms and in their teaching practicums. 4.2 TLMP Output The original goal was to produce 2.25 million textbooks for distribution to over 1,500 government secondary schools in all regions of the mainland. USAID funding cuts resulted in a shortfall; the final publication tally was 1,560,400 according to MOEVT. Production numbers were reportedly affected by textbook configuration: the Zanzibar science materials combined Forms 1 and 2 into one book and Forms 3 and 4 into another, but the mainland science materials separated the Forms 3 and 4 textbooks, adding content according to the mainland curriculum. The creation of two titles from one should have raised the number of copies, but did not. Mainland production and distribution was as follows: Table 4.2.1 – Production and Distribution of TLMs TLMP Evaluation Report Volume II – Tanzania 239 Title Form # copies Year Publisher Mathematics 1 160,000 2010 UM Mathematics 2 160,000 2010 UM Mathematics 3 120,000 2011 UM Mathematics 4 120,000 2011 UM Biology 1&2 160,000 2010 OUP Biology 3 70,000 2012 UM Biology 4 70,000 2012 UM Chemistry 1&2 160,000 2010 OUP Chemistry 3 120,000 2011 OUP Chemistry 4 120,000 2011 OUP Physics 1&2 160,000 2010 OUP Physics 3 120,000 2011 OUP Physics 4 120,000 2011 OUP Teacher Guide Math 1 8,000 2010 UM Teacher Guide Math 2 8,000 2010 UM Teacher Guide Math 3 6,000 2011 UM Teacher Guide Math 4 6,000 2011 UM Teacher Guide Biology 1&2 6,700 2010 OUP Teacher Guide Biology 3 3,500 2012 UM Teacher Guide Biology 4 3,500 2012 UM Teacher Guide Physics 1&2 6,700 2010 OUP Teacher Guide Physics 3 6,000 2011 OUP Teacher Guide Physics 4 6,000 2011 OUP Total 1,560,400 4.3 Project Management and Partnerships TLMP was largely managed from SCSU. SCSU established a Program Advisory Committee (PAC) chaired by the Permanent Secretary of Education with membership including curriculum specialists from TIE and subject area teachers from universities and secondary schools. SCSU also established a relationship with the Director of EMAC who served as the local program coordinator, because revision of textbooks fell under the normal purview of that office. In this way, local structures were respected and utilized effectively by TLMP. USAID/T assigned a person on the PAC to monitor progress. Partnerships were established only with the publishers, OUP and UM, but several issues arose – one involving turnover and the other the ownership of the TLM copyright. Key figures at OUP had been replaced due to an alleged corruption scandal involving the World Bank,12 and the new Managing Director had been in office only two weeks prior to our meeting. The publisher of the Zanzibari materials, Macmillan, had sold off the TLMP contract to a mainland partner, Uhuru Media. Ownership of the copyrights was in dispute (although the CA required that ownership be retained by the MOEVT or its designate) up until the last visit of the SCSU TLMP leadership team in December 2012. At that time OUP reported that the SCSU team demanded the entire set of digital textbook files and required OUP to sign over the rights that they previously had to re-publish the TLMs for sale. 12 See http://www.worldbank.org/en/news/press-release/2012/07/03/world-bank-sanctions-oxford-university-press-corrupt￾practices-impacting-education-projects-east-africa) for the World Bank’s discussion of the action it took on debarment of OUP East Africa and OUP Tanzania. TLMP Evaluation Report Volume II – Tanzania 240 4.4 Project Implementation 4.4.1 Materials Development The two individuals most directly involved with the adaptation of the Zanzibari TLMs were the Director of Secondary Education Science and the Coordinator of EMAC. Their respective teams estimated that 50% of Zanzibar TLMs required revision or replacement due to cultural references (e.g., depictions of mosques or Muslim clothing), curriculum alignment, editorial modifications, and errors. Adaptation of the TLMs was largely led by the EMAC Coordinator who began with the organization of a Review Board to evaluate the Zanzibari materials. The Board consisted of subject matter specialists from the TIE and exemplary secondary school teachers from around the country. The Board used such review criteria as degree of alignment with the national curriculum, cultural and gender appropriateness, and paper and print quality. Board recommendations were submitted to EMAC. When all requests for revision/correction had been addressed and the materials were approved by EMAC, the TLMs were then sent to OUP and UM who conducted a final editing (with the assistance of subject matter specialists and editors) and then had the TLMs printed either in Tanzania (OUP) or Kenya (UM). For a review of the adaptation process followed for Basic Mathematics, Forms 3 and 4, see ANNEX D. Mathematics Example of Adaptation Process. Several authors/adapters felt that their contributions were not accepted by the publishers. For example, a mathematics specialist told the team that the context-appropriate suggestions he and his colleagues gave for the math book cover illustrations were not used, and there was no explanation from the publisher. They found the final design (an abstract head with numbers floating around in all directions) confusing and culturally inappropriate. One constraint in revising the TLMs was time, which placed a burden on the reviewers, writers, and editors. For example, the publication of the Forms 3 and 4 biology books by UM was delayed until February 2013, after our research was concluded. SCSU and USAID officials were aware of this delay, and the MOEVT confirmed that the books were awaiting distribution. The problem was a combination of timing and funding and the need to add content when the two textbooks were separated into two texts. Additionally, the books had apparently arrived during examination time and both funds and time were lacking to distribute them. Subsequent to the revision of the TLMs, the Director of Secondary Education Science supervised the development of a self-guided in-service DVD and manual for secondary school teachers on “teaching with textbooks.” While not specifically a training program in how to use the TLMs, these self-study resources are intended to support the transition from content-based instruction (lecture and memorization) to competency-based teaching—and they depict the TLMs among other available textbooks. These self-study professional development materials were waiting to be distributed to schools at the time of this evaluation. There were both positive and negative aspects in the development of the TLMs. According to representatives of UM, an important outcome of this process was the capacity development of its cadre of textbook writers and editors. UM also mentioned the positive impact the project had on enhancing their capacity to deliver high-quality products on a tight time schedule, which skill will be applied to future textbook production activities. Informants interviewed could not easily describe the role SCSU technical experts played in the revision and adaptation process. One Tanzanian mathematics specialist pointed to a (U.S.) name in the credits and said he never had contact with the man. There is no evidence of analytical input by SCSU to ensure achievement of program objectives; rather, the process followed parameters established by EMAC. TLMP Evaluation Report Volume II – Tanzania 241 Moreover, according to informants from both publishing companies, the piloting of new textbooks and orientation/training teachers to the new materials were never conducted due to budgetary restrictions. One workshop co-facilitated by EMAC and TIE (with representatives from SCSU present) was provided to teachers, but when the team asked for the curriculum, it was not provided. USAID/W funding was cut in the last two years of TLMP, resulting in a shortfall in the total printing of TLMs. This meant the project goal of providing one book for every student was compromised. The result was that USAID/T was seen as over-promising and under-delivering. Public assurances were made by the Mission Director to MOEVT, but the relationship between USAID/T and MOEVT has been strained. Copyright page Back cover The most critical finding pertaining to the development of the TLMs is the issue of copyright ownership and intellectual property. That SCSU and the MOEVT both claim copyright ownership is a matter of deep concern as without local ownership of the copyright, the materials cannot be reprinted. Ownership was apparently an issue beginning with the production of the Zanzibar TLMs, as the name of the Project Coordinator appears on some of the books as if he were the author. TLMs published by OUP for Zanzibar cannot be reprinted for the mainland until copyright ownership is clarified. This unfortunately means that some of the (inappropriate) Zanzibari books are being sold in the mainland market. UM did not experience these difficulties and to their knowledge they have the right to re￾publish and sell any TLMs after 2012. It is not clear whether the mainland versions of the textbooks should be considered new books or edits of the Zanzibar publications, a further complication. We left USAID/T with this information so that the legal issues could be clarified. 4.4.2 Materials Distribution and Storage The Director of Secondary Education Science also managed the TLM distribution. Upon receiving the printed books from the publishers, the different titles were broken into pre-assigned 80 unit blocks and repackaged for specific schools. These schools were targeted because they were “new” schools, i.e. started since 2006, most of them relatively rural and remote. A few “old” schools were added to the list because they had not received material assistance through other programs. The TLMs were distributed to REOs, then -sorted and sent to DEOs, and finally retrieved by school principals. TLMs were distributed in the following manner: TLMP Evaluation Report Volume II – Tanzania 242 • Notification was sent to the MOEVT and the Ministry of Regional Administration and Local Government (MRALG) in the Prime Minister’s office to alert them that the TLMs were printed. • The MRALG contacted the Regional Administrative Secretaries to say that the TLMs would be delivered to the MOEVT warehouse. • The REO at the Regional Administrative Office informed the target secondary schools through their principals to expect a specific number of science and math textbooks and teacher’s guides for forms 1-4 on a specific delivery schedule. Targeted schools included community schools registered in 2007-2010; all secondary schools designated by MOEVT as training centers for science and mathematics teachers; old established secondary schools with instruction ending at “O” level; and selected community schools registered in 2005/06 that had not received teaching materials from MOEVT. • The number of books distributed to schools was 160 copies per title for Forms 1 and 2 and 80 copies per title for Forms 3 and 4. (For biology, 1,500 schools received the books at an average of at least 46 copies per school). (This uniform distribution did not take into account the specific needs of individual schools or the number of students studying specific subjects in the upper forms, where sciences were elective. This meant that some TLMs occasionally ended up where there was little or no demand for them. There did not seem to be any mechanism that would have enabled DEOs or principals to take initiative to re-distribute the TLMs according to actual needs.) According to MOEVT officials, delivery of the final Form 3 and 4 biology books has been delayed because DEOs and school principals are waiting for the next quarterly tranche of budget funds to be released by MOEVT. Such TLMs are unlikely to be used until the 2013-14 academic year. There appeared to be a good accountability system at each distribution point as the numbers of books delivered to each school – and to each point along the channel - was documented. SCSU did not appear to have a mechanism for monitoring this distribution. When USAID/T conducted spot monitoring in Morogoro, cartons of TLMs were found in the DEO’s office that had been there for at least three months since their receipt from the REO. In other cases the team found TLMs that had been distributed to schools but not to teachers or students. Overall the situation should have been more carefully monitored. At the school level, the team saw a range of strategies for distributing or maintaining the TLMs. In most cases TLMs were logged into the school system and individual copies were labeled with a control number. Where there were enough textbooks to assign one copy per student, these were recorded. Most schools had a policy that required students to pay a replacement fine for any loss of books assigned to them. When students get the books for the whole term or school year, they are required to sign for them and cover them with protective book jackets. When there was an insufficient supply of textbooks to provide each student a book, they were kept in the library, a storage room, or were given to the subject head, who disbursed the books when teachers and/or students signed them out. In these cases, schools typically maintained sign-out logs. TLMP Evaluation Report Volume II – Tanzania 243 We saw many lists like this indicating delivery EMAC approval page and book no. 54/78 Library borrowing record at Kipok Girls’ TLMs were not distributed to students at several schools. The explanation given was that the school did not want to risk loss or damage of the materials. These materials were often kept in secure storage rooms, but it seemed unlikely that students would be able to access them. 4.5 Assessment of TLMs by Various Stakeholders The TLMs were generally felt to be useful and of high quality at all levels, including USAID/T. The publishers pointed out that the quality of the paper, ink, printing and binding make the TLMs attractive and durable and provide good value for cost. They appeared to be durable, even when (over)used or apparently soaked through. Unused math books, Ilboro Boys’ SS A very used math book Used biology books, some covered Studying at the library, Toangoma SS The most important observation was that the TLMs were adapted to be consistent with the 2005 competency-based curriculum currently in effect. Teachers like that the TLMs are sequenced, i.e. written to build on information that has previously been introduced so that content becomes gradually more complex. Teachers feel that TLMs are of good quality and comparable to the materials used in private schools. Members of the Education Team at USAID/T had a similar opinion, and Peace Corps Volunteers using the TLMs reportedly had high praise for the books. Students reported they liked the language in the texts, the photographs and illustrations, along with the chapter summaries, key vocabulary, exercises and activities. Students found that the activities and some examples reflected their daily lives. They also reported the format enabled them to use the TLMs for self-study, which is important given high teacher absenteeism, particularly in rural areas. Students TLMP Evaluation Report Volume II – Tanzania 244 commented on the list-like nature of the presentation of content. This made the identification of key information easier, especially for second-language users. Students generally perceived the quality of the textbooks produced by TLMP to be high. They commented that the color illustrations and use of graphics was an improvement over the older textbooks from TIE. Some teachers perceived that the TLMs were not uniform in quality in each of the subjects. Some preferred the materials developed by TIE, whether or not their departments had done a comparison. Some math teachers perceived the books as providing too shallow a treatment of some content areas, but an expert math teacher told us that those teachers probably do not know the books well because the sequential approach means delving more deeply into some topics later in the books. Teachers who lacked understanding of the competency-based curriculum were more comfortable with the older textbooks, with which they themselves had learned and which fit the thematic content of the previous curriculum. Others wished for designated sections in the TLMs on performance evaluations, which are required for a competency-based curriculum. Both authors and teachers mentioned that some small errors should be corrected if the books were to be reprinted. Some teachers commented positively on the structures used to present information. The textbooks highlighted key terms and concepts. Teachers frequently commented that even if they did not use the textbooks in their classes, the TLMs helped them develop their lesson plans. Few teachers seemed to be aware of the planning resources available in the teacher’s guides accompanying each textbook, and indeed, not all were aware the teacher’s guides existed—until colleagues went and found them. School principals and head teachers were generally positive in their perceptions of the quality of the textbooks. Typically, head teachers had reviewed the textbooks informally and made the decision to use them, either as supplementary texts or (more often) as main texts. All informants expressed the opinion that the materials were consistent with the subject area curriculum. A minority of head teachers seemed to be aware that the textbooks were meant to support the competency-based approach being implemented nationwide. Informants felt that using the textbooks would lead to improved student learning by virtue of color graphics, clarity of information, organization, and features that support student learning. Some principals and head teachers felt that increasing the number of textbooks in the classroom would have a positive impact on student learning. However, our observations indicated that simply putting textbooks in learners’ hands was only one part of the puzzle; teachers often copied text on the blackboard, even with the books in front of the pupils. 4.6 Use of TLMs and Teacher Observations 4.6.1 Teacher Observations The evaluation team observed over 60 teachers at 18 secondary schools. The use of the TLMs in these settings varied considerably. Approximately 15 percent of teachers observed appeared to use the TLMs as they were intended, i.e., they were the main instructional resource used during the lesson, teachers generally followed the teacher’s guide, and the teacher used the text to guide student reading during class. About 40 percent of the teachers observed used the TLMs as supplementary resources. This was also reinforced by comments from focus groups. For example, in one math classroom a teacher was observed assigning a particular math concept and then directing students to draw on resources in the school library (textbooks) for sample exercises. Teachers also assigned the TLMs as resources for independent study and especially for the use of graphs, diagrams, and charts. The efficacy of this approach seems questionable as the texts appeared to be generally beyond most students’ independent TLMP Evaluation Report Volume II – Tanzania 245 reading level. Interestingly, nearly all of the teachers who said they used the TLMs as a resource commented that they also used them as a resource for themselves. The textbooks helped identify key concepts, vocabulary and high value content. However, none of the teachers in this group appeared to use the TLMP resources as a guide to innovative pedagogy. Approximately 25 percent of teachers observed did not use the TLMs in any manner. Some used texts previously published by TIE. These teachers seemed to use a traditional lecture mode in which students copied information from the chalkboard. Approximately 20 percent of teachers observed appeared to use the materials in an inappropriate manner. For example, teachers were observed lecturing students using the content of the text (which the students had in front of them) or teachers reproduced diagrams and charts that were already produced in the students’ texts. The most significant observation on the use of TLMs is that teachers seemed to be unable to use these resources to stimulate critical thinking, scientific reasoning or opinion-formation. Instead all teachers observed, whether they used the TLMs or not, focused instruction on information that could be easily listed, memorized and repeated. The MOEVT is generating a professional development program that focuses on the topic of teaching with textbooks as teachers appear to be unable to incorporate instructional materials into their pedagogical repertoire. It may actually be the case that because textbooks convey information, the traditional role of the classroom teacher is becoming redundant. Teachers clearly need re-training on how to build on text-based information. The variation in the use of the textbooks may be attributable to several factors, none of which appeared to be related to the essential quality of the textbooks: • Textbook policy: The current decentralized policy empowers the leadership of each school to select commercially-published texts from an MOEVT/TIE-approved list, i.e. materials that have passed the EMAC approval process. Whereas materials from TLMP were once the only books in schools, and indeed still are in some rural areas, now varied instructional resources are available. In addition, school leadership is encouraged to use more than one resource for content information. Some books are used textbooks donated from English-speaking countries; others were published by TIE with copyrights from 1992. Other secondary science texts produced with USAID funding were also found in the schools. It is not at all clear why USAID/W provided the funding for yet another set of texts, essentially duplicating what USAID/T had already done in the recent past. • Competition from commercial publishers: Most schools had a mix of commercial materials either purchased or donated. Some of these commercially prepared materials were nearly 20 years old while others were apparently quite new (TIE had engaged in a partnership with Pearson Publishing in 2009 for the production of science textbooks). • Teacher understanding of TLMs: In most cases, it appeared that the TLMs arrived in schools without any formal review process. Some teachers were unaware that the TLMs reflected the new national competency-based curriculum in contrast to the older TIE materials. Others were unaware that there were teacher’s guides to accompany each TLM, although they were in school libraries or stores. Some principals commented that teachers were more likely to use the materials with which they were familiar. • TLMs as a resource: Teachers reported using the TLMs to prepare their lessons, especially to identify key terms and learning objectives. Others reported using the activities or exercises with their students. Both teachers and students said that the science books were useful for self-study TLMP Evaluation Report Volume II – Tanzania 246 during study time, when doing homework, or for supporting learning when teachers are absent. In many cases, students used TLMs on loan in the school library. Most schools had well-functioning check-out systems from libraries, subject department heads, or their teachers (who had signed them out from their departments). • Use of traditional teaching methods: When asked if the TLMs had changed their teaching methods, teachers often said yes. However, according to observations in 62 classrooms, teachers still appeared to rely heavily on traditional teaching methods of lecturing, note-taking and literal recall. Students are also instructed to copy content from the chalkboard, even when the same information was in the TLMs. One reason for this practice seems to be the strong tradition of fact learning and recitation, reinforced by an assessment tradition that rewards recall of factual information in flawless English (which is nearly impossible, as English is a foreign language for most learners). Related to this is the difficulty of comprehending complex technical content and critical analysis in a foreign language, so that remembering key terms and lists of attributes already taxes the linguistic proficiency of teachers as well as students. To meet their needs, not only would the materials need to be written at an appropriate level of English, but approaches to teaching and learning English across the curriculum would need to be built in, and strategies developed to support both linguistic and cognitive development. The point is that textbooks alone cannot transform the teaching/learning dynamic. • Lack of equipment: Officials at the Secondary Education Department at MOEVT indicated that it would be extremely useful to have adequate instructional materials and laboratory equipment (e.g., microscopes, voltmeters or oscilloscopes) referred to in the texts. Without appropriate equipment, teachers omit activities and focus on memorization at the expense of developing scientific reasoning, hypothesis, observation, and analytical thinking. • Lack of students/Lack of teachers: It is not uncommon in Tanzania for there to be no students studying a particular subject area at a given time, especially in forms 3 and 4 science. One school official commented that high-achieving students in his remote school typically transfer out to other schools after their second year, leaving a reduced demand for higher level textbooks. In such cases, textbooks are stored until such time as a class of students is formed. In addition, secondary schools have difficulty recruiting and retaining qualified instructors for upper-level sciences and mathematics, especially in remote and rural schools. In this case textbooks are unused and stored until appropriate teachers are hired. Even though they did not always use the materials to promote independent reading for knowledge, teachers still commented that they wished for a 1:1 pupil-to-textbook ratio. We observed teachers managing this situation by having students read in pairs or assigning group work in which students used the textbooks as resources. We observed one math teacher who used the textbooks as a resource along with all of the other textbook series in the school library. After presenting students with a particular math concept, he would assign students to identify similar math problems in the math resource books, solve these problems and present their solutions on the chalkboard. In general we observed very little instruction that corresponded to the suggestions in the accompanying teacher’s guides. In only one biology classroom observed did the teacher have students go outside to collect examples of living and non-living things and sort these according to the definitions in the textbook. 4.6.2 Reading Assessment One sub-team (including a Kiswahili speaker) did informal assessments of student reading. These were intended not as a measure of student reading performance but to determine whether the TLMs were written at students’ instructional level (meaning the ability to identify 90-95 percent of the words and comprehend 75-90 percent of the material). Students in forms 1-4 participated in individual informal TLMP Evaluation Report Volume II – Tanzania 247 assessments of reading using sample passages selected at random from their TLMs in a location outside the classroom. The results of these sample assessments gave cause for concern. Although most students were able to decode words with a high level of accuracy, they generally had a great deal of difficulty comprehending what they read. Students were unable to respond to specific prompts for literal or inferential thinking. Even when prompts or directions were provided in Kiswahili, students remained unable to respond. These findings are particularly troubling given that many teachers and students saw these TLMs as useful outside the classroom, in other words to be used without scaffolding or any other form of support. As these materials did not seem to be on students’ instructional level (and certainly not their independent reading level), it is unlikely that students would gain much information without considerable reading comprehension support provided by the teacher (such as pre-reading, guided reading, vocabulary development, etc.). 4.7 Textbook Review The TLMs (textbooks and teacher guides) were reviewed according to standards for reader-friendly instructional texts. This analysis has to do with specific elements that make instructional texts easy or difficult for students to use as learning materials. These elements include: • Sentence length • Passage organization and sentence cohesion • Use of headings and labels • Pre-reading and post-reading activities A more detailed analysis of the TLMs appears in ANNEX E. Textbook Review. 4.8 Conclusions TLMP in mainland Tanzania created a significant quantity of instructional materials for math and science for resource-poor secondary schools and clearly helped reduce the pupil-text ratio. The books were published with high quality materials. However, several troubling conclusions emerged from this research. • Limited capacity at SCSU: One of the main program objectives was to build capacity at the MSI. Despite its experiences in Zanzibar, SCSU demonstrated that it still has limited capacity to manage and implement a large-scale development project. Both SCSU and the Tanzanian educational system would have been better served if the MSI had partnered with an NGO or agency with experience in educational development, so that SCSU capacity would have been raised. • No innovations or insights applied: Very few individuals from SCSU were involved and no visibly new approaches, activities, or strategies were introduced. • Redundancy in textbook support: Although Tanzania has a great need for textbooks nationally, the need is not uniform. A cost-effective approach would demand a more nuanced understanding of the education context that goes far beyond simple textbook production to distributing them to the neediest areas and following up on their use. • Textbooks can’t teach: TLMP intended to improve learning outcomes for secondary school students in Tanzania. However, even well--designed instructional materials do not guarantee effective teaching nor do they automatically lead to improved learning outcomes. Textbooks do not teach; well-prepared professional teachers do. Along with providing materials, TLMP should have worked with teachers to help them understand the demands of competency-based teaching and learning—as well as teaching a second/foreign language across the curriculum. At the very least, there should have been an orientation to the teacher’s guides. TLMP Evaluation Report Volume II – Tanzania 248 5. LESSONS LEARNED Staff politics, culture, and international development: Politics is endemic to the entire process, and this had to be negotiated all along the way. One SCSU interviewee commented, “A full-fledged country study was required before we moved ahead on the proposal so as to be prepared for what we would see when we actually visited. In preparation for such a trip, we needed cross-cultural training so that we could interact with people locally with a degree of cultural sensitivity. Some training in language would also have been good.” SCSU Leadership: Instability in SCSU leadership over the life of a multi-year program created a context in which, as one informant responded, SCSU TLMP leadership was constantly explaining the purpose of the project and the rationale for the college to support it. Without this stability, working with USAID was not fully institutionalized and the close working relationships with the MOEVT on the mainland were not established as they had been in Zanzibar. Confusion in Authorship and Copyright: TLM authorship was to have been a joint activity between designated individuals of the MOEVT/TIE and SCSU; however, the Project Coordinator’s name appears as the author on one of the textbooks, and the copyright was maintained by SCSU, making reproduction of the materials extremely difficult. Operations: Having an on-site, in-country coordinator would have facilitated key relationships. In addition, according to one interviewee it “would have been better” to have created the original texts for the mainland and then move to Zanzibar. Pedagogy: Teachers were not very prepared to teach their subjects at the assigned grade level using the TLMs as they called for a different approach to teaching. Students are now learning from 21st Century textbooks; however, science kits are needed, teacher evaluations of the use of the books are needed, and a plan for the local team needs to be developed to carry on with the process. Language: Teachers and students do not have the English competence to work in the level of English required by each of the textbooks. An ESL approach may have been more effective rather than the assumption that students would be able to work in English at the same level of native speakers. The value of systems: TLMP in Tanzania made great use of the existing systems within the national education context. Rather than operating as a project outside the national system, the mainland portion of TLMP was managed and supported largely by well-established Tanzanian systems for textbook revision, approval and distribution. This resulted in a cost savings and provided a measure of ownership for MOEVT including at the regional and district levels. The value of professional publishers: TLMP benefited from the contributions of professional commercial publishers. The success of the project seems in large part because of the appropriate use of publishers’ local technical expertise. In fact, had the publishers’ standard practices for piloting and training been followed, the TLMs might have been better adapted to students’ needs, and teachers better prepared to use the TLMs. International publisher practices: OUP is a multinational corporation and as such can take advantage of resources available in other countries. OUP printed many of the TLMs in Kenya, where machinery, supplies, and electricity are more reliable. Thus, the Tanzanian printing industry, per se, did not benefit from TLMP. The limitations of textbooks as change agents: Although SCSU and the MOEVT may have felt that the level of innovation incorporated in the TLMs was modest most teachers observed seemed reluctant to change teaching styles. In this case, teachers were familiar with lecture and recall of terms and concepts, and used the TLMs to highlight facts rather than to engage students in a discussion of the implications and applications of factual information. The limitations of centrally-managed international development programs: USAID supervision from Washington was not the most efficient. The USAID/W office appeared to be too far removed from the SCSU and the MOEVT, despite the presence of USAID in Dar es Salaam. Critical oversight regarding budget, partnership development, and product review seemed to be lacking or poorly timed. TLMP Evaluation Report Volume II – Tanzania 249 6. RECOMMENDATIONS 6.1 Materials Improvement Address the lower levels of English language capability of secondary school students: Student difficulty in understanding the vocabulary and more complicated sentence structures used in the TLMs requires reconsideration of the level of English used in the textbooks. Future textbooks in any/all subjects must be solidly based on an English language assessment to guide the level of English competency required. Conduct a more comprehensive review of the usability of TLMs: Since teachers themselves have varying levels of subject matter knowledge and know-how in building competencies in their students, a more comprehensive assessment of their ability to understand and teach the range of concepts incorporated in the TLMs should be undertaken to fulfill two goals: 1) improve instruction at TTCs; and 2) improve classroom delivery of lessons. As noted above, textbooks don’t teach; teachers do. 6.2 Systems Improvement Link TLMP to teacher professional development: To maximize use of the TLMs, an investment should be made to build on the MOEVT’s own initiatives to implement a professional development support program on teaching with textbooks. Address copyright issues: An investigation should be undertaken to determine who actually owns the copyright to the TLMs so that, if so desired, they can be reprinted. Support a large-scale evaluation of the impact of TLMs on student learning: USAID might consider supporting MOEVT and Tanzanian research universities to conduct a national assessment of the impact of TLMP and other textbooks on teaching and learning. Create links between teacher education institutions and schools: To maintain continuous professional development and to further professionalize the teaching profession, a link between SCSU science and math educators and secondary school teachers in Tanzania via shared technological resources, particularly through USAID-sponsored Teacher Resource Centers in Tanzania, should be considered. This pairing might be extended to U.S. secondary schools with which SCSU has a relationship through student teaching and in-service programs. Build school-based management of resources: USAID might consider developing support programs that build on the MOEVT’s school-based decision making and decentralization efforts. The formation of school-based textbook review committees with objective criteria for evaluating potential textbooks for adoption is one critical function. USAID’s support for secondary school directors and academic leaders might help the MOEVT to capture these exemplars and create leadership development models around them for future school-based decision making. Develop a strategy to support secondary math and science: All TLMs should be provided to TTCs so that they might be included in the curriculum in training new teachers, thus providing input on how to teach using a 21st Century textbook. Without such an input it is likely that students will not elect to study the sciences at a higher level. Link to other secondary education programs: The TLMP experience – processes and outcomes – should be linked to other donor secondary school interventions. Insure quality and accountability of textbooks at the school level: The same level of quality and accountability in the choice of textbooks in evidence at EMAC should be required at the school TLMP Evaluation Report Volume II – Tanzania 250 level. This will mean that the appropriate MOEVT divisions should provide training at schools (including directors and teachers) on how to make textbook choices. Systematize the allocation of textbooks to schools that need them: As part of the process of identifying the number of textbooks needed in a particular subject at a particular grade level, the MOEVT should determine whether the subject is taught at that school and at that grade level so as not to provide books to a school where they are not needed. Share TLMP evaluation processes and results with early grade reading project awardees: The lessons learned by SCSU in developing the TLMs should be shared with early grade reading project leaders. Even though the subject matter is different, there are useful elements emerging from the project that can be learned by others (e.g., copyrights, publishing, textbook review, etc.) 6.3 USAID Improvement Consistently apply performance monitoring procedures: USAID should require that all funded projects prepare a PMP and consistently conduct monitoring on project progress. In reviewing monitoring reports, included in quarterly reports, USAID should determine if the project is “on track” or whether approaches and activities need to be adjusted to obtain the desired results. Build institutional capacity for less experienced US-based implementers: If USAID/W intends to earmark funds to expand capacities at U.S. institutions that do not have extensive experience in international development, it should anticipate the need for even greater oversight of their activities and ensure that such institutions are mentored so that capacity is gradually built over time. TLMP Evaluation Report Volume II – Tanzania 251 ANNEX A. Questions Posed of SCSU Project Team Ask re Following Documents • MOE report on the appropriateness of the textbooks developed for Zanzibar • Matrices of changes needed to textbooks to contextualize them to the Mainland • Mapping instrument to align textbooks sequentially and in scope with the curriculum of the Mainland Background • Initial MOU in 2005-2008 for secondary textbooks and learning materials in biology, chemistry, physics and math in Zanzibar ($5.5 million); what was accomplished under AEI? What were lessons learned from this first phase? How did SCSU build on what was done in Zanzibar to continue the TLM production and distribution process for the mainland in the second phase under CA of 2009- 2012 ($13 million)? How did the results of Phase1 influence Phase 2? • How did SCSU become involved in the development of TLMs at the pre-school level in Zanzibar? What happened when funding was not available? Was any attempt made to find other funding for the early childhood program? Administration • How did all the changes in senior leadership affect the ability of TLMP to get things done? What types of stumbling blocks were overcome over the period of Phases 1 and 2 of TLMP? • What was the role of the Ministry of Regional Administration and Local government in the basic education part of TLMP in Zanzibar? • What is the actual number of TLMs produced for Zanzibar? For the Mainland secondary schools? Project Implementation • How and who decided who would get the textbooks? • What was the problem with the legal office and the potential delay in publishing? • What was the nature of your relationship with the Peace Corps? The World Bank? • What criteria were used in school/classroom observation? • What research was conducted on the entire project process? What lessons were learned? What best practices? All noted are a repeat of the first/earlier reports; nothing is new. • What were the process and instruments used by EMAC in approving the textbooks that were produced in Zanzibar and adapted for use on the Mainland? • While the quarterly reports and other documents report on the outputs, please explain the actual work that the textbook developers did in the first phase and how the materials were changed for use on the Mainland? Who of the SCSU did what? Who of the MOE/TIE did what? What teachers were involved? How were they chosen to be a part of the writing team? Who were the subject matter specialists in each team? How did they work with the teachers? • What was the process used to include “culturally relevant” items in the textbooks? Who determined cultural relevance? • How did the Tanzanian and SCSU subject matter experts work together to produce the textbooks? What was the process from start to finish? • Why was the number of textbooks to be produced increased by 40,000? • What was the process used to develop the teachers’ guides? Who specifically was involved? How were curriculum standards incorporated/followed? • Who was responsible for writing the quarterly and annual reports? Why is there so much repetition and so little actual data/information presented? TLMP Evaluation Report Volume II – Tanzania 252 • What is the system of distributing textbooks at schools? How large are classrooms and how many students actually receive textbooks? Project Impact • What is SCSU able to do now that it is different from the outset of TLMP? What capacities were built by whom? What institutional changes? • What is the MOE/TIE able to do now that it is different from the outset of TLMP? • How have teaching practices changed as a result of having access to books? • How has learning by student improved as a result of using textbooks? TLMP Evaluation Report Volume II – Tanzania 253 ANNEX B. Schedule of Data Collection Activities Date Time Organization Person Interviewed/ Research Activity Title WHOLE TEAM DAR ES SALAAM 2/18 10:00 IBTCI Team Meeting 2/18 12:30 MOEVT Chris Kibanga Director￾EMAC Local TLMP coordinator 2/18 1:00 MOEVT Silvestina Mgisimba; Acting Permanent Secretary of Education 2/18 2:00 MOEVT Dorothy Mwaluko Director, Secondary Education￾Science & Mathematics 2/19 1:00 Uhuru Media Lela Abdala Managing Director 2/19 3:00 USAID Tom LeBlanc; Laura Kikuli and Abbas Nsanzugwanko Education Manager; Program Manager; M & E Specialist 2/20 2:00 TIE Leonard Akwilapo Acting Director 2/20 5:00 TIE Godson Lemme TIE Subject Matter Specialist (Physics 2/21 9:30 MOEVT M. Mukaruka Coast Regional Education Officer 2/21 10:30 MOEVT Omari Kisuda; Alice Msemwa Kibaha District Education Officer; Assistant District Education Officer 2/21 12:00 MOEVT Georges Ngonyani Academic Master Mwanalugali Secondary School TLMP Evaluation Report Volume II – Tanzania 254 2/21 1:30 MOEVT Teacher Focus Group Mwanalugali Secondary School WHOLE TEAM COAST 2/21 3:30 MOEVT Aaron Ndunguru Headmaster, Kibaha Town Council School 2/22 9:30 MOEVT Bashiry Shellimoh Temeke District Education Officer 2/22 10:30 MOEVT Godfrey Mchimbini Head Master; Tuangoma Secondary School 2/22 11:00 MOEVT Various Teachers Tuangoma Secondary School 2/22 12-2:00 MOEVT Various teachers Classroom observations Tuangoma Secondary School SUB TEAM 1 MORO￾GORO 2/25 8:30 MOEVT Michael Daffa Morogoro Rural Council District Education Officer 2/25 9:30 MOEVT Mtembeje Kingimali; Syvanu Konambi Academic Officer Morogoro Municipal District Education Office; District Education Officer 2/25 10:30 MOEVT Rukia Chembe Assistant Academic Officer, Morogoro Regional Education Office 2/25 11:00 MOEVT Tabitha Tusekelege; Various Teachers Head Mistress Kilakala Girls Boarding School 2/25 1:00— 4:00 MOEVT Berthea Kulwa Various Teachers Head Mistress TLMP Evaluation Report Volume II – Tanzania 255 Students Kingolwira Secondary School 2/26 2:00 MOEVT Theresa Kamugishu Sekwao Secondary School 2/27 9:00- 12:30 MOEVT Anna Mmbeya; Rosemary Sam; Various Teachers Head Mistress, Kirokala Secondary School; Assistant Academic Officer 2/27 1:30-4:30 MOEVT Ibrahim Janguella; Various Teachers Head Master, Mkuyuni Secondary School 2/27 8:45-9:15 MOEVT Damari Mchome Meru District Education Officer￾Secondary Education SUB￾TEAM 2 ARUSHA 2/27 9:30 MOEVT Alex Kitomari Head Master, Poli Secondary School 2/28 9:00-2:00 IBTCI Team Team Debrief 2/28 3:00 Oxford University Press Peter Crowder; Mussa Khamis; Frederick Sylvester Acting Managing Director; Program Manager; Assistant Program managers WHOLE TEAM DAR ES SALAAM 3/1 9:00 MOEVT Silvestina Mgisimba Acting Permanent Secretary of Education 3/1 10:00 USAID Debriefing: Tom LeBlanc; Laura Kikuli and Abbas Nsanzugwanko Education Officer; Program Manager; M & E Specialist TLMP Evaluation Report Volume II – Tanzania 256 ANNEX C. Data Collection Instruments In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? How were these different for Phase 1 in Zanzibar and Phase 2 on the mainland? • Have you supported any other book/learning materials development projects before? What indicators were used to track progress? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see TLMP leveraging other educational development activities in Malawi or elsewhere in the future? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? Were all the textbooks and teachers’ guides accepted as part of the national curriculum? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in this country? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outcomes of TLMP? What would you change? • Would you say that the model for TLM creation developed by SCSU can be utilized in other areas? If not, what would you change? • If a TLMP-type project emerges/is needed for the future, what would your recommendations be for improvement? • How are you working with the World Bank to extend the creation/publishing/ disseminating process of TLMs? • How are you working with the MOE to develop sustainability plans for the use and continuous development of TLMs? • Do you believe that the textbooks produced for the Mainland were in line with the national curriculum? Did they properly address gender, HIV/AIDs and other cross-cutting themes? What else should be included? • Was the level of expertise provided by SCSU and the MOE appropriate to the design of textbooks that have current and future applicability? What type of capacity building was conducted by SCSU? By the MOE? • Are you aware of any studies that have documented the changes in national examination outcomes that could be linked to the production and distribution of the TLMs? 2) MOEVT/TIE Administrators (triangulate with different administrators at each level) • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded or were the responsibilities added? Were salaries topped off for participation? If yes, by how much? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? How many teachers did the TLMP Evaluation Report Volume II – Tanzania 257 MOE train on the use of the TLMs? How did you train them? For how long? Could you please provide a list of the teachers who were trained and the schools they represented? • What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOE provide? How frequently? • How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • Did you find that the TLMs produced followed the MOE curricula appropriately? Did they incorporate gender? HIV/AIDS? Other cross-cutting themes? • What was the difference between the TLMs produced for Zanzibar and those produced for the Mainland? What process was used in producing the TLMs for the mainland? • Do you believe that the way the TLMs were structured helped teachers in becoming more learner￾centered? If yes, what, specifically, was included to do this? If not, what should be included? What other teacher training do you provide in learner-centered methodologies? • What did the MOE learn from this project about developing textbooks? Is the MOE able to reproduce this process? If not, what would the MOE need? Now that the project has ended, what is your plan to continue the production of TLMs? What new textbook policies have been developed as a result of TLMP? • How has the material presented in the TLMs been included in preparation for national exams? Has performance on exams improved with the use of TLMs? • How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOE benefited from TLMP? How has it been challenged? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? Why? If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with SCSU? What could be improved? • Have the textbooks become your national textbooks? If not, why not? What is preventing this adoption? If yes, do you have enough for all students and teachers in the country? If the answer is no, how many more are needed? What is the MOE plan to make up for this shortfall? 3) Material Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? • How did you become involved in TLMP? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? How many were male and how many females? Does this represent the gender distribution of teachers of science and math at the secondary level? • How often did the two teams meet? What were the results of these meetings? • What process was used in creating the TLMs? How long did it take for each subject and grade level? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? In terms of ensuring gender neutrality? In terms of including such cross￾cutting themes as HIV/AIDS? What other cross-cutting themes did you include? Were the materials culturally relevant? TLMP Evaluation Report Volume II – Tanzania 258 • In developing the TLMs, how was the methodology of learner-centered teaching included? If so, how did this help teachers improve their teaching? Improve learner outcomes? • How did the textbooks for the mainland differ from those produced in Zanzibar? Scope and sequence? English language complexity? Illustrations? Cultural issues? Other? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? • How satisfied are you with the collaborative production process between yourselves and SCSU? What would you change? • How did you field test what was written? With whom? Where? Did you have a rural/urban balance? What was the difference in the results? Did you have to make significant changes to allow for appropriate usage in rural areas? What were these? • How do you think the TLM development process can be improved in the future? 4) Printers/Publishers – Uhuru Media and Oxford University Press • Tell me about your operations before you were granted the TLMP contract and how they changed as a result of TLMP participation? Who were your clients and what was the nature of the work you did before the project and how did this change after the implementation of the project? • How did the contracting occur with the SCSU? What was the content of the contract, i.e., what were your responsibilities? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? Was the sequencing of the materials to be produced in accordance with your capacity, or were there any production difficulties that involved obtaining paper, skilled staff, etc.? • Where were the TLMs printed? What was your particular role in producing the TLMs? • What, if anything, did you learn through working on TLMP that enabled you to produce materials more efficiently and/or less expensively? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? Could you please share your distribution list with us? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have you hired? What new equipment have you purchased? How much has your income increased? What other inputs would you require to take on more textbook production projects? In Field 5) Regional Education Offices • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? TLMP Evaluation Report Volume II – Tanzania 259 How many of each textbook was made available to each school? Were there enough for all children in each class? • Are all teachers who received the TLM’s required to use them in their teaching? What other materials do they have/use? Is there a sufficient number of these materials for all children? If not, what does a teacher do? • What instruction did you give to each school about how the TLMs were to be used? Did the MOE provide a TOT? If so, how many teachers in your district attended the TOT? How were these teachers chosen? How many male? How many female? Is this division representative of the number of men and women teaching math and sciences at the high school level? How many of these teachers went on to teach others? How many others were trained? • How were inspectors instructed on how to evaluate teachers using TLMs? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 6) Principals/Directors • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender equity? If so, what have the results been? • What language(s) do children speak when they enter school? Is this language the language of instruction? Up until what grade? When does English become the language of instruction? Is the level of English used in the TLMs appropriate? If not, what needs to be done? • What is the average amount of time a teacher remains in your school? What factors cause them to leave for work elsewhere? Once teachers have training in different subjects, do they generally remain in the school or go elsewhere? • How many of each textbook did you request for the school? How many of each textbook did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers received the MOE training on the use of the TLMs? How did you choose these teachers? How many were males? How many were females? Does this distribution represent the gender division of teaching math and science at the high school level? What were the teachers’ reaction to/opinion of the training? Did you attend the training yourself? If so, what was your opinion of the training? • How are the teachers coping with the new materials? For those who have not been trained, how are they using the materials? Have the TLMs enabled the teachers to be more learner-centered in the methodologies they use? If not, what else do they need? • What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on what they have learned to their colleagues? • Do you believe the materials are culturally relevant? Address HIV/AIDs? Address issues of gender equity? Follow the curriculum for each subject appropriately? If not, what would you change? • Have the textbooks produced been adopted by the MOE as the core textbooks for each subject at each grade level? What other subject matter books do teachers and learners have? How would you compare/contrast the two? TLMP Evaluation Report Volume II – Tanzania 260 • Are the textbooks relevant for the different science labs you have in school? If not, how should they be changed? • Is the language level of the textbooks appropriate for the English language capability at each grade for your students? What needs to be changed? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 7) Classroom and Head Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree) • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching in English? • How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach your subject? What do you do when you don’t have enough textbooks for each learner? Do you have a teacher’s guide for each of the textbooks? If not, what do you use? • How would you compare the non-TLMP with TLMP textbooks in terms of usability? Level of English language used? Illustrations? Complexity of explanations? Other? How would you recommend that they be changed? • When did you receive training on the use of TLMs? How long did it last? Who delivered the training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice? If you did not attend any training related to the materials, how did you learn how to use them? • When did you receive the TLMs for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your learners? How many learners must share a textbook? • Were you able to use the textbooks after the training? Did you feel you needed more training? In what? Do you have the appropriate lab equipment to utilize the textbooks? What do you do if you do not? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? Are the texts culturally relevant? Do they address gender equity? Do they address HIV/AIDS? How might the presentation of these ideas be improved? Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? Have the materials increased your teaching skills? How? • What type of difficulties do the learners have in using the materials? Is there a difference between girls and boys? What would be needed to overcome these difficulties? TLMP Evaluation Report Volume II – Tanzania 261 • What changes have you observed and recorded in children’s achievement since the TLMs were introduced? Are more girls or boys performing at a higher level in math and sciences since using the textbooks? • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1-4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent girls and boys equally? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMs, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? 8) TEACHER OBSERVATIONS OF TLM USE IN THE CLASSROOM School __________________________ Village/Town/City ________________________ Country _________________________ Type of School: ___Primary; ___Jr. Sec; ___Sec Teacher Sex _____ M; _____F Grade Level: __________________________ No. of Students: ___ M; ___ F; ___Total No. of Books _________________________ Age range of Students: _____M; _____ F Mother Tongue of Students ________________________________________________ No. & type of learning materials _____________________________________________ Indicator Observed Not Observed Teaching Using TLMs Instruction 3. The teacher has prepared materials on the use of the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP 7. The teacher can read and explain TLM content to the students 8. The teacher uses TLMs throughout the class period 9. Students use TLMs throughout the class period TLMP Evaluation Report Volume II – Tanzania 262 Indicator Observed Not Observed 10. Students can read and understand the subject matter in the TLMs. 11. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment) 12. The teacher makes sure learners make connections of TLMP content to daily lives 13. The teacher gives and corrects homework using the TLMs. 14. The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson 15. Students and teachers use mother tongue/English (French) when asking and responding about TLMs (circle which language) TLMs/Artifact Inventory 16. Lesson objectives are written on the board in either local language or English (French) 17. Learning aids/materials are posted in the classroom (TLMP produced and others). 18. TLMs are locked up in the cupboard. 19. Word walls display key words in local languages and English 20. To demonstrate language experience, sentences appear on the chalkboard or on a chart in the classroom 21. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books) 22. There is evidence that teachers mark exercise books in a process of continuous assessment Comment_____________________________________________________________________ _____________________________________________________________________________ 9) Learners’ Reading Competency Note: Because the TLMP textbooks for secondary Mathematics and Science in Tanzania were not intended to improve students’ English language reading proficiency, the purpose of evaluating students’ reading performance was to evaluate the readability or fit between the students’ reading ability and the level of text difficulty presented by the TLMP textbook. In addition, because these were secondary school students there was no need to assess them on the elements of early grades reading—letter recognition, familiar word reading, and decoding. The protocol applied is provided below. EVALUATION OF LEARNER READING COMPETENCY School ___________________________ Village/Town/City ________________________ Country _________________________ Grade Level ______________________ Class Subject Matter _____________________ Student Age _____________ Sex ____M; ____F Home Language/Mother Tongue of Learner: ____________________________________ Language of Testing: _____________________________________________________ Title/Type of TLM used in Class: ______________________________________________ Reading Aloud: TLMP Evaluation Report Volume II – Tanzania 263 Instructions: Identify 2 short passages (3 to 5 sentences) from one of the TLMs for the appropriate grade level and make a clean copy of each passage. [Please save passages to accompany evaluation results. They should preferably be cut from within stories or texts so that memorization can be ruled out. One passage should be easier than the other, so one can be taken from the beginning of the book and one from the middle, depending on how far into the school year they are.] Have the student read the more advanced passage aloud and note the results below. If s/he is successful, stop after this. If the advanced passage is too difficult, provide the easier passage and conduct the exercise again, noting the results. Reading Fluency Passage 1: ____________________ (Rate and comment) 1) Fluent _______________________________________________________________ 2) Little difficulty (specify words) ___________________________________________ 3) Very haltingly (specify words) ____________________________________________ 4) Can read only a few words (specify words) __________________________________ 5) Cannot read at all ____________________________________________________ Passage 2: ____________________ (Rate and comment) 1) Fluent _______________________________________________________________ 2) Little difficulty (specify words) ___________________________________________ 3) Very haltingly (specify words) ____________________________________________ 4) Can read only a few words (specify words) __________________________________ 5) Cannot read at all ______________________________________________________ Reading Comprehension/Oral Reading Ability of learner to paraphrase what s/he has read silently: 1) Can put into own words all that has been read without difficulty _____________________ 2) Can put into own words only selected points __________________________________ 3) Has difficulty putting passage into own words __________________________________ 4) Cannot put passage into own words _________________________________________ 10) TLM Analysis Instrument – Criteria Used for Analysis of Textbooks Generated for Mainland Tanzania Subject _________________________ Grade Level _________________________ STUDENT TEXTBOOKS 1. Curriculum Coverage • How balanced is the presentation of topics as reflected by the table of contents? 2. Readability • What are the general features of sentence length and vocabulary? • What is the readability level as measured by the Flesch Readability formula? 3. Text Structure TLMP Evaluation Report Volume II – Tanzania 264 • How is content organized within paragraphs (e.g., main idea/details; lists, definitions, characteristics, etc.)? • How are paragraphs linked using linguistic markers and content information? 4. Text Elements • How does the text use bold-face and italic fonts to highlight key information? • How does the text use labels, captions, and subheadings to highlight important information? 5. Graphics • What kinds of graphics are used in the textbook (photographs, drawings, charts, maos, diagrams etc.)? • What is the quality of these graphic elements? 6. Additional Features • Does the textbook contain a chapter overview and review? • Does the textbook include a glossary or list of important formula? • Does the textbook include self-checking review questions? TEACHER’S GUIDES 1. Lesson Objective • Are lesson objectives clearly identified for each lesson? • Are objectives linked /cross-referenced to the national curriculum? 2. Support for Planning • Does the teacher’s guide suggest the amount of time needed to complete a unit? • Does the teacher’s guide identify recommended/required instructional resources and equipment? 3. Background information • Does the teacher’s guide include material that provides additional content material for the teacher? • Does the teacher’s guide contain suggestions for a variety of teaching techniques? 4. Meeting the Needs of Diverse Students • Does the teacher’s guide contain suggestions for adapting instruction and content to meet the needs of diverse learners? 5. Overall appearance • Does the appearance of the text seem dense and overwhelming? • Are key resources easily identified with text graphics? • Is there a table of contents, glossary and index to simplify usage? TLMP Evaluation Report Volume II – Tanzania 265 ANNEX D. Mathematics Example of Adaptation Process A report of the review process for Basic Mathematics Forms 3 and 4 indicates that a six-person Tanzanian team and the lead mathematics writer from SCSU adapted these materials. The books were examined to address concerns in the following areas: • Conformity to the syllabus • Organization and presentation of content • Correctness of facts, concepts and figures • Conformity to socio-cultural values of Tanzania • Integration of life skills and cross-cutting issues • Adaptability and applicability to local and global settings • Relevance and appropriateness to learners and teachers • Use of quality instruction • Ability to appeal to learners of different abilities • Presence of enjoyment index • Assessment of learners’ understanding • Appropriateness of language and communication aspects The findings of their review led to changes in the following areas: • Supplying introductions for some subtopics/content of chapters, where necessary • Reorganizing the content of some chapters • Adding examples/exercises that integrate cross-cutting issues and life skills • Revising/adding class activities to chapters, where necessary • Introducing/including elements to increase the enjoyment index of mathematics • Revamping/rewriting some concepts that were felt to be inadequately presented • Suggesting changes/improvements in figures, terminology, notation, position of figures, etc. • Proving a variety of examples and problems requiring critical thinking and applications of real-life situations throughout the text A similar review process was undertaken for all textbooks produced in Zanzibar for use on the mainland due to more clearly defined learning parameters of the curriculum. A similar process was undertaken for each of the teacher’s guides. TLMP Evaluation Report Volume II – Tanzania 266 ANNEX E. Textbook Reviews In general, the textbooks appeared well-designed, with a number of structural elements to support learning. Chapters were generally brief and were subdivided into content chunks of a half to a full page. This would make the reading task more manageable for a second/foreign language user. The subdivisions were clearly labeled with bold fonts to identify the main idea of the topic of the section. Much of the content within each section was presented as bulleted lists. This structure seemed to reflect the tradition of rote memorization and recall of factual information. Other study aids in the student textbook included chapter openers which identified key concepts and learning objectives and chapter closers that helped students review the critical content presented in the chapter. An abundant use of color photographs, charts and other graphics seemed especially helpful for guiding all students but especially valuable for second/foreign language learners. However, when the content was presented as connected text, readability formulas (based on sentence length and syllable counts) suggested that the writing was generally above grade level for native English speakers. This level of text difficulty would be extremely challenging for second/foreign language readers, as the informal reading assessment indicated. A review of the Teacher’s guides revealed distinct differences between the two publishers. The Guides prepared by Uhuru offered considerable “extra” information for the teacher. These extras included: • The identification of learning objectives for each lesson and explanation of how these objectives were linked to the national curriculum • A review of various teaching techniques • A list of recommended materials for each lesson (which would be especially useful when planning science activities) • Practice assessments modeled after the national assessment formats • Background content information providing additional facts related to lesson content • A suggested block plan detailing the number of days to allow for each chapter • Specific teaching suggestions for each lesson The Guides produced by Macmillan were adequate but less detailed. They focused on learning objectives and answers to questions in student textbooks. Overall, both types of Guide seemed to be an exceptional resource in a context that suffers from a dearth of professional materials. It is unfortunate that most teachers in the schools we visited were unaware of these resources or had not made use of them. Unfortunately there were no indications that the publishers asked classroom teachers to review these guides or pilot them at any time during the development/adaptation of the TLMs.