December 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. The principal authors are Edward Jay Allan and Nancy E. Horn under Task Order AFR-12-00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. FINAL EVALUATION Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania 1 EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA FINAL EVALUATION Principal Authors Edward Jay Allan, Project Director Nancy Horn, Team Leader Other International Team Members Eric Allemano Carol Benson Thomas Tilson James Wile National Team Members Erango Ersado (Ethiopia) Feleke Desta (Ethiopia) Habtamu Mammo Ethiopia) Kofi Marfo (Ghana) Kofi Mereku (Ghana) Denis Khasu (Malawi) Hannock Mateche (Malawi) Babacar Diouf (Senegal) Alhousseynou Sy (Senegal) Rakgadi Phatlane (South Africa) Richard Chediel (Tanzania) Cathleen Sekwao (Tanzania) December 7, 2013 The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP TABLE OF CONTENTS INTRODUCTION AND ACKNOWLEDGEMENTS.................................................... iii ACRONYMS...................................................................................................................... iv EXECUTIVE SUMMARY.................................................................................................. v Summary of Outcomes and Impacts.......................................................................................................vi Key Findings and Recommendations.....................................................................................................vii The Most Critical Findings and Recommendations........................................................................vii Status of the Curricula, Pedagogical Philosophies, and Intended Use of TLM.........................vii Development and Production of the Materials..............................................................................viii Lack of Pre-Service and In-Service Teacher Training in Use of TLM; Teachers Guides......viii Monitoring, Evaluation, and Assessment...........................................................................................ix 1. INTRODUCTION ......................................................................................................... 1 2. BACKGROUND ........................................................................................................... 2 2.1 TLMP Overview – USAID/Washington ........................................................................................... 2 2.2 MSIs......................................................................................................................................................... 3 2.3 MOEs...................................................................................................................................................... 3 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS................................. 4 3.1 The Evaluation Team............................................................................................................................ 4 3.2 Evaluation Questions............................................................................................................................ 4 3.3 Evaluation Methodology....................................................................................................................... 4 3.4 Limitations to the Study....................................................................................................................... 5 4. FINDINGS AND LESSONS LEARNED...................................................................... 6 4.1 Introduction............................................................................................................................................ 6 4.2 TLM Produced and Summary of Positive Impacts ......................................................................... 6 4.3 MSI Preparedness and Management.................................................................................................. 7 4.3.1 Discussion on MSIs........................................................................................................................ 7 4.3.2 Conclusions and Lessons Learned............................................................................................. 9 4.4 MOE Commitment and Participation ............................................................................................... 9 4.4.1 Mixed Processes and Outcomes................................................................................................ 9 4.4.2 Conclusions and Lessons Learned.......................................................................................... 10 4.5 Teacher Readiness and Participation ............................................................................................. 11 4.5.1 Teacher Readiness...................................................................................................................... 11 4.5.2 Teacher Participation................................................................................................................. 11 4.5.3 Conclusions and Lessons Learned.......................................................................................... 12 4.6 TLM Development, Printing and Distribution ............................................................................. 12 4.6.1 TLM Development ..................................................................................................................... 12 4.6.2 Printing and Physical Production............................................................................................. 13 4.6.3 Distribution.................................................................................................................................. 14 4.6.4 Marking, Copyright, and Submissions to DEC..................................................................... 14 4.6.5 Conclusions and Lessons Learned.......................................................................................... 14 4.7 Use of Materials by Teachers and Learners................................................................................. 15 i Evaluation of TLMP 4.7.1 Challenges Inherent in Teacher Training Practices............................................................. 15 4.7.2 Challenges Inherent in the TLM Themselves....................................................................... 16 4.7.3 Conclusions and Lessons Learned.......................................................................................... 17 4.8 Sustainability of TLMP ....................................................................................................................... 17 5. RECOMMENDATIONS.............................................................................................18 5.1 USAID Missions.................................................................................................................................. 18 5.2 MOEs and Systems Improvement................................................................................................... 18 5.3 TLM Development, Printing and Distribution ............................................................................. 19 5.4 TLM Use and Assessment................................................................................................................ 19 ANNEXES ........................................................................................................................20 ANNEX A. SCOPE OF WORK .....................................................................................20 ANNEX B. WORK PLAN...............................................................................................23 1. Introduction ................................................................................................................25 2. Project Overview........................................................................................................25 3. Objectives of the Performance Evaluation..............................................................26 4. Research Approach, Methodology and Process ......................................................27 4.1 Contacting and Scheduling Interviews with MSIs........................................................................ 28 4.2 Obtaining and Reviewing Project Documents............................................................................. 28 4.3 Assigning Members of the Team to Each Country..................................................................... 28 4.4 Identifying Target Populations from Whom to Gather Data................................................... 29 4.5 Developing Interview Schedules and Other Data Collecting Instruments ........................... 29 4.6 Scheduling In-country Research Activities.................................................................................... 29 4.7 Review of Cost-Effectiveness .......................................................................................................... 30 4.8 Developing and Editing Preliminary Country Reports............................................................... 30 4.9 Draft and Final Reports..................................................................................................................... 30 5. Management and Logistics ........................................................................................31 6. Draft Timeline..............................................................................................................31 ANNEX C. ETHIOPIA RESEARCH QUESTIONS .....................................................32 ANNEX D. SCHOOL-BASED RESEARCH ACTIVITIES...........................................39 ANNEX E. TLM PRODUCED........................................................................................41 ANNEX F. REPRESENTATIVE TLM ASSESSMENT .................................................43 ANNEX G. REPRESENTATIVE READING ASSESSMENT (Ethiopia, n=40 ...........46 ANNEX H. SELECT BIBLIOGRAPHY OF RECENT MATERIALS ..........................47 ANNEX I. CONFLICT OF INTEREST STATEMENTS..............................................49 ii Evaluation of TLMP INTRODUCTION AND ACKNOWLEDGEMENTS This evaluation report is comprised of two volumes. This main volume contains the evaluation report per se, which represents a synthesis of the evaluation team’s findings for each of the six cooperative agreements and which incorporates the annexes that relate to the evaluation as a whole. The companion volume contains the reports on the individual cooperative agreements. Unlike most evaluations, this assignment called for IBTCI to analyze the ways in which the project affected the implementer itself. IBTCI would first wish to express our appreciation to USAID’s Africa Bureau for giving our team the opportunity to learn firsthand how this important regional project functioned and to share our perspectives on how TLMP and analogous projects could be made more effective. Our most sincere thanks and appreciation to Joe Kitts, Sandy Oleksy-Ojikutu, and Megnote Belayneh for their solid support throughout the length of the project. We would also like to express our appreciation to the staff of the six USAID missions with which TLMP – and hence our evaluation- interacted. Their active interest, recommendations and suggestions, and support helped greatly, especially given that scheduling for the country visits did not always come at opportune times. It goes without saying – but it certainly needs to be said – that this evaluation could not have taken place nearly as effectively without the commitment, the competence, the flexibility, and the sense of common purpose of our team of international and national evaluators, led by Dr. Nancy Horn, who brought together the breadth of their experience in sub-Saharan Africa and the depth of their knowledge of their own systems. Our thanks go to the staff implementing TLMP at the various Minority Serving Institutions and to the university administrators. Unlike most evaluations, the TLMP evaluation called for study of ways in which involvement in the project had an impact on the implementer itself. The candor and responsiveness of the MSI personnel, to and including chancellor or president, was critical in giving us what we hope is an accurate and balanced set of perspectives on TLMP, especially for the expansion phase, the time frame for the evaluation. By no means to be bypassed are the host country personnel from multiple organizations in each country who were engaged in developing and producing the TLMP materials – a total of more than 30 million pieces, representing some 500 different titles in 13 languages – and other TLMP stakeholders – Ministry staff, printers, teacher trainers, etc. – with whom the teams met. And most definitely to be remembered are the beneficiaries of TLMP – the students and their teachers. The teams had the opportunity to observe 55 classes taking place in a wide range of settings and to talk with over 260 teachers and headteachers/principals. We are most grateful for having had the chance to learn from them, and to have learned from all involved with TLMP. Edward Jay Allan TLMP Evaluation Project Director International Business & Technical Consultants, Inc. iii Evaluation of TLMP ACRONYMS AAMU Alabama Agricultural & Mechanical University ADEA Association for the Development of Education in Africa AEI President’s Africa Education Initiative CA Cooperative Agreement CPD Continuous Professional Development CRECCOM Creative Center for Community Mobilization CSU Chicago State University DEC Development Experience Clearinghouse DBE (South Africa) national Department of Basic Education ECSU Elizabeth City State University EGRA Early Grade Reading Assessment EFL/ESL English as a Foreign Language/English as a Second Language HBCU Historically Black Colleges and Universities IBTCI International Business & Technical Consultants, Inc. KG kindergarten L1 First language, mother tongue L2 Second language MIE Malawi Institute of Education MLC (Ethiopia) Minimum Learning Competencies MOE Ministry of Education MOEST (Malawi) Ministry of Education, Science and Technology MOU Memorandum of Understanding MSI Minority Serving Institution MTPDS Malawi Teacher Professional Development Support (Program) NESP (Malawi) National Education Sector Plan 2008-2017 NPC (Malawi) National Primary Curriculum OUP Oxford University Press PAC Project Advisory Committee PCAR (Malawi) Primary Curriculum and Assessment Reform PEA (Malawi) Primary Education Advisors PMP Performance Monitoring Plan RM Read Malawi S1-S3 (Malawi) Standard 1-Standard 3 SOSA (Malawi) The name given to Book One, Standard 1, Chichewa SSA Sub-Saharan Africa TELL (Ethiopia) Teach English for Lifelong Learning TL Team Leader TLM Textbooks and Learning Materials TLMP Textbook and Learning Materials Program TOT Train the Trainer TPR (Malawi) Total Physical Response TT Teacher Training TTC Teacher Training College UNICEF United Nations Children’s Fund USAID United States Agency for International Development USAID/W United States Agency for International Development/Washington UTSA University of Texas, San Antonio iv EXECUTIVE SUMMARY “Textbooks are among the most cost-effective inputs to student learning.” - World Bank Education in Sub-Saharan Africa: A Comparative Analysis, 2012 The Textbooks and Learning Materials Program (TLMP) began in 2005 as part of the President’s Africa Education Initiative (AEI), which also contained components for teacher training, to provide secondary school scholarships to girls from rural areas, and to promote innovative activities. U.S. Minority-Serving Institutions (MSI) and host country partners in six countries joined to produce over 17 million copies of learning materials. In 2008 an extension/expansion phase of TLMP was implemented to enable five Minority-Service Institutions (MSI) to continue work in five of the original six countries of Sub-Saharan Africa – Chicago State University (CSU)(Ghana), Alabama A&M (AAMU)(Ethiopia), Elizabeth City [NC] State University (ECSU)(Senegal), South Carolina State University (SCSU)(Tanzania, shifting its work to the mainland from Zanzibar), and the University of Texas San Antonio (UTSA)(South Africa) and for UTSA to start work in Malawi. By the end of TLMP in December 2012, over 30,000,000 textbooks and learning materials (TLM) representing some 500 different titles in 13 languages had been produced for use in grades and subjects from kindergarten language arts to secondary school science. This represented nearly ten times the original target of a minimum of 600,000 for each of the six cooperative agreements (CAs). Primary level Language Arts South Africa (Phase I wrap-up) UTSA Malawi (Phase II only) UTSA Ghana Chicago State Ethiopia AAMU Secondary level History, Geography Senegal ECSU Math, Science Senegal ECSU Tanzania SCSU In 2012, USAID’s Africa Bureau contracted with International Business & Technical Consultants, Inc. (IBTCI) to conduct a final evaluation of the expansion phase of TLMP. The evaluation objectives were to • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based MSIs, in- country institutions, Ministries of Education (MOEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, curriculum alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy. A total of six U.S. and 12 national specialists in basic education in sub-Saharan Africa conducted the evaluation. The assignment called for evaluation of capacity-building, support and accomplishments of TLMP vis-à-vis the MSIs themselves as well as vis-à-vis the host countries. Consequently, the project v Evaluation of TLMP began with interviews by the project director or the team leader with TLMP staff at the five MSIs and, as possible, university administrators, generally on campus. Based on these meetings and a document review, a generic research design for country visits was prepared and subsequently adjusted to reflect the circumstances of each country and “lessons learned” from individual country visits as they took place. Fieldwork in each country lasted between 10 and 13 days. In each country other than South Africa (which was an outlier in several ways), fieldwork was conducted by two or three pairs of evaluators comprised of a U.S. and a national researcher. To maintain inter-country and inter-evaluator comparability and consistency, each of the U.S. specialists worked in at least two countries in different combinations. In addition to meetings and school visits in the national capitals, in each country team members conducted interviews and classroom observations in urban and rural locales at distances up to a day’s journey each way from the capital to provide some degree of geographic and demographic diversity. Key limitations included repeated re-scheduling by USAID, teacher and general strikes, weather, threats of disease outbreaks, frequent turnover of MOE personnel (one reason for the very limited institutionalization of TLMP), and distances and school timetables that limited the number of schools that could be visited in the available time. Nonetheless, in addition to meetings with developers and MOE personnel, the team met with over 260 school-based educators and observed 55 classes. Summary of Outcomes and Impacts Please note that these are generalizations, not necessarily applicable to each MSI’s CA and each country. 1. The expansion phase of TLMP far exceeded its target of 600,000 TLM per CA. By December 2012, the TLMP partnerships had produced an estimated 30,712,111 TLM representing some 500 different titles in 13 languages for use in grades and subjects from kindergarten language arts to secondary school science. 1.1. The primary grade teacher-developed materials in South Africa and in Malawi have attracted additional support from other donors for replication and/or duplication. Indeed, as of October 2013, the Malawi Institute of Education had reprinted two million more TLMP books and had issued a tender for seven million more. We understand that the materials for secondary level in Tanzania are to receive funding from other donors for duplication. 1.2. However, for the most part, the primary level TLM were being used as supplementary materials, not as primary textbooks. 1.3. The teacher-developed primary readers received great acceptability on the part of students and teachers. South Africa’s Limpopo Province had received funding from Irish Aid to assist in replication, and Room to Read had also evinced interest in providing support. However, USAID/ South Africa plans not to make any use of the Ithuba TLMP materials even though they satisfy current curricular standards. Because of changes in Ghana’s policy on languages of instruction and because they rely to a large extent on workbooks that are no longer available, TLMP in Ghana is realistically obsolete, although Chicago State University (CSU), the implementer, continues to work with USAID/ Ghana on education activities. We have no knowledge as to whether TLMP materials will be incorporated within USAID/Ethiopia’s new READ education project. We understand that the secondary level TLMP materials developed for Tanzania are to be duplicated. 1.4. The materials developed by TLMP are not available on the Development Experience Clearinghouse (DEC) and only a small number are available from implementers. Therefore, realistically, they can have negligible impact for other developers of instructional materials. 2. According to MSI staff, including top-level administrators, TLMP has had a positive influence on a number of aspects of the respective universities and their communities. It has led to increased awareness on campus of the global community, strengthened student exchange activities, improved the capacity of the university to serve as a community resource, and strengthened the ability of each MSI to conduct international development activities. Further, CSU and the University of Texas, San vi Evaluation of TLMP Antonio (UTSA) have continued to participate in USAID-assisted educational activities. 3. A very important aspect of expansion phase TLMP activities was the sharply increased transfer of responsibilities for all aspects of materials development to host country organizations and individuals. Because of the nature of mobility among education personnel in sub-Saharan African countries, this does not inherently represent an increase in the capacity of organizations. However, it does represent valuable increases in the capacity of individuals in specialty areas such as textbook and materials writing and illustration. MOEs are trying to discern how these scatted resources can be orchestrated to generate more materials to satisfy current reading requirements (in mother tongue, English, and French). Key Findings and Recommendations For ease in reference, key findings are followed by key recommendations in italics. The Most Critical Findings and Recommendations 1. There was one critical flaw in the design and implementation of TLMP – inadequate attention to the role and importance of teacher training. In the design of TLMP, the overwhelming focus was on the development and production of the textbooks and learning materials with relatively little attention paid to how teachers would make use of the materials developed. However, in their own learning experiences prior to teacher training, many – perhaps most – teachers would have had limited access to textbooks themselves, often new approaches to the teaching-learning process were incorporated in the materials, and training in the use of textbooks is not a normal part of pre-service teacher training. As a result, most teachers had difficulties in using the materials effectively. Adjustments were made, but only in-service training was addressed, but it is not clear whether what was offered was adequate. Significant attention should be paid to training teachers in how to use textbooks and supplementary materials as well as to the development of the materials themselves. This training should begin at the teacher training college level, preferably using the textbooks that students would use themselves in the classroom, since this is a skill all teachers need to have and, given that in-service training tends to be associated with particular projects, there is limited probability that it would be addressed in in-service training soon after a teacher enters service. 2. The materials produced were generally of high quality. However, a pervasive issue was that the level of language used was generally much higher than the competency of the learners and, often, higher than the competency of primary school teachers. This is partly the result of unrealistic expectations on the part of ministries of education (see discussions in materials from the Brookings Center for Universal Education and the Learning Metrics Task Force in the References) but also due to deficiencies in assessment of the capacities of the learners. (a) Assessments should be made of the linguistic capabilities of the users of materials and materials should be adjusted accordingly. (b) In developing second language (L2) materials for both learners and teachers, implementers should keep in mind that L2 skills at the primary level, even for teachers, are likely to be quite weak and ESL/EFL-type approaches should be used. (c) In piloting/field-testing materials and prior to duplication for broader use, teachers at “typical” (rather than “demonstration”) schools should be given a reasonable time to try out the materials in the classroom and then be debriefed about the challenges that arise. 3. While highly appropriate as a modality for teaching literacy in languages like English, which have irregular orthography, because African languages are written phonetically the “whole word” method is not nearly as effective as phonemic approaches for teaching literacy in African languages. In generating materials to increase literacy in any language, a mixed methods approach should be used to give children the opportunity to build reading skills. (See, e.g., Helen Abadzi, Literacy for All in 100 Days?, in References.) Status of the Curricula, Pedagogical Philosophies, and Intended Use of TLM 1. Several factors within each respective MOE affecting the use of materials developed by TLMP vii Evaluation of TLMP included changes in pedagogical philosophy, changes in curricula, and changes in policy with respect to languages of instruction. These changes, which particularly affected TLMP in Ghana and in Ethiopia, were beyond the control of the implementers but resulted in resources being spent on materials which could not be used. USAID should consider sharing whatever it learns from its interactions with MOEs about future policy so that projects can anticipate changes and adjust their projects accordingly. 2. In some instances, materials that implementers believed were going to be core texts turned out to be materials that the MOE wanted used as recommended or supplemental texts (to be used at the learner’s discretion to expand the culture of reading). To some extent, this was due to the fact that the materials did not yet exist. At project inception, USAID and implementers should discuss with the MOE what the intended use of materials is – e.g., as core texts, supplemental materials, recommended, etc. – and the nature and extent of associated teacher training. The MOE should also provide explicit direction to relevant district education offices, inspectorates, etc. that the TLM are approved for use in classrooms. Development and Production of the Materials 1. The approach used in initially developing stories by training teachers to write supplemental readers was effective in South Africa, Malawi and Ethiopia; they became problematic to teachers and learners when they were not appropriately leveled for the grade intended. Over 300 titles in first languages (L1) and second languages (L2) were publishable after editing and were used in classes. Based on classroom observations and discussion, learners can read the readers with understanding and find them interesting, relevant, and informative. Teachers (including teachers who were not involved in production of the materials) find them effective although many teachers found the teacher’s guides difficult to read. Other funders have shown active interest in replicating and/or reproducing supplementary readers, some teachers have continued to prepare readers on their own, and some students have also indicated an interest in preparing readers of their own, to be shared with others. Replication of an approach that involves teachers developing their own materials should be seriously considered for activities in various sectors – not just literacy – that involve TLM. 2. In some cases (e.g., Malawi), class periods for reading were 90 minutes or more (too long for first￾and second-graders to be attentive without a book to look at) with too much material to be covered. Adequate time should be allowed for pilot testing the amount of time it actually takes to cover a lesson, and dialogue with MOE on what is pedagogically realistic before making any changes to class time allocation. 3. Both physically and pedagogically, TLM must be durable so as to prevent the need for recurrent expenditure for book replacement. Ghana’s MOE, for example, planned to have student workbooks replaced every year, but this was not feasible due to budgetary constraints. TLM should be designed to last for at least two years. To help reduce wear and tear, and also to reduce the physical burden of carrying textbooks, especially for younger children, textbooks might be split according to terms, thus reducing the wear and tear on the books. Any expendable parts could be incorporated into teacher’s guides, with instructions on copying a master onto a blackboard and children then copying, then completing, into an exercise book or blank piece of paper. Lack of Pre-Service and In-Service Teacher Training in Use of TLM; Teachers Guides This is a general issue for most of the countries evaluated and is by no means limited just to TLMP. 1. For pre-service education, educators generally reported that very little, if any, time is devoted to the practical use of textbooks or to the use of supplementary materials in the classroom. Further, instructors do not generally incorporate the TLM that students will encounter when they start teaching, and were rarely used. Teacher trainers should be fully equipped to teach their students in the use of grade-specific TLM and promote their use in the classroom once students graduate. Learner￾centered methodologies were core to the production of TLM; this approach has now been adopted by all MOEs involved in TLMP. viii Evaluation of TLMP 2. Any in-service training (INSET) in the use of TLM tends to be limited to that provided in conjunction with specific projects or, as in the case of Tanzania, by publishers for their own products. Teachers not involved with relevant projects are effectively left on their own. INSET should focus on the use of TLM not only for their interdisciplinary content (many can be used in teaching a range of subjects), but also for the learner-center methodologies employed in teaching. 3. Although many teacher’s guides did provide guidance intended to introduce teachers to new teaching and learning approaches, teachers observed typically used the traditional “chalk and talk” teaching approaches because they were unsure how to use the new methods. Short guides (e.g., four pages long) for supplemental readers were useful, but because they were short, were often separated from the readers they belonged to. (a) Based on reports, teachers found teacher’s guides that were scripted to be more usable. Guidance for individual lessons was quite useful. (b) Even if individual lessons might be printed separately (e.g., for supplemental readers), to avoid being lost, associated teacher’s guides should be bound together rather than issued separately. Monitoring, Evaluation, and Assessment 1. MSIs did not maintain effective Monitoring & Evaluation (M&E) systems during the expansion phase, either for the CA as a whole or for assessing the effectiveness of materials produced as actually used, with the result that tracking progress toward objectives was challenging. Apart from greater attention to routine M&E, any project that produces TLM must include active follow-up to determine if challenges using them have been resolved, how teachers have adapted them for local use, and whether any other materials should be incorporated in future. Ithuba readers in South Africa help children in the rural Thohoyandou school in Limpopo learn. ix “Textbooks are among the most cost-effective inputs to student learning.” - World Bank Education in Sub-Saharan Africa: A Comparative Analysis, 2012 1. INTRODUCTION USAID has been involved in developing educational institutions and student learning possibilities in sub￾Saharan Africa for more than six decades, working with African educators and community members on all facets of education and improving the environment for greater participation in learning. In Ethiopia, as an example, in 1952 Emperor Haile Selassie invited the U.S. Government to begin education sector capacity development under USAID’s predecessor, the Point Four Program. Despite major efforts across the continent from national governments, bilateral and multilateral donors, and non-governmental organizations, major challenges remain. According to a 2012 World Bank report, primary school students in low-income Sub-Saharan African countries have, on average, learned less than half of what is expected of them. Across the continent, only three out of four adults who completed six years of schooling can read. In the best-performing countries, almost everyone reads after completing a six-year cycle. However, there are broad differences in the number of years of schooling needed to provide children with lifelong literacy skills. According to a 2006 ADEA (Association for the Development of Education in Africa) study, textbooks and other materials are among the most cost￾effective inputs for student learning – if they are used. The Textbook and Learning Materials Program (TLMP), as implemented in 2009-2012, was an extension of the TLMP component of the President’s Africa Education Initiative (AEI), implemented 2005-2008. AEI also included components for Teacher Training, the African Girls Scholarship Program, and Innovative Activities. The main objectives of the TLMP extension were to: 1) produce and distribute no less than 600,000 (per country) high quality, cost-effective textbooks and learning materials (TLM) and 2) strengthen the capacity of U.S.-based Minority Serving Institutions (MSIs) of higher education to build sustainable linkages with African education institutions to provide technical assistance after the completion of the program. As the third objective, materials were to be aligned with Ministry of Education (MOE) curriculum priorities and include cross-cutting themes (e.g., gender, health, etc.); These materials were to be developed and/or adapted under each respective Cooperative Agreement (CA) in collaboration with the host partner country’s MOE and other local entities. This evaluation covers only the expansion phase of TLMP (2009-2012), funded by USAID’s Africa Bureau. Three of the CAs awarded were to continue work on TLM begun under AEI in Ghana, Ethiopia, and Senegal; one was to build on work done in Zanzibar for schools on Tanzania’s mainland; and one was to implement a TLM teacher/author development process launched in South Africa and then reproduced in Malawi. Originally intended to develop TLM for primary schools, MSI needs assessments determined that in Senegal and Tanzania MOE priorities were for middle and secondary school TLM. The MSIs and their focus in each of the six countries were as follows: • Chicago State University (CSU) – English language TLM for kindergarten and grades 1-3 in Ghana. • Alabama Agricultural & Mechanical University (AAMU) – English language textbooks for grades 1-4 and supplementary reading TLM in Ethiopia. • University of Texas at San Antonio (UTSA) – Supplemental reading TLM in nine different home languages, Afrikaans and English for grades 4-6 in South Africa (2005 to 2008); supplemental reading TLM in Chichewa and English for grades 1-3 in Malawi from 2009 to 2012. • Elizabeth City State University in North Carolina (ECSU) – Textbooks in French for middle and high 1 Evaluation of TLMP school history, geography, mathematics and science in Senegal. • South Carolina State University (SCSU) – Textbooks in English for secondary school math, biology, chemistry and physics for mainland Tanzania. Primary level Language Arts South Africa (Phase I wrap-up) UTSA Malawi (Phase II only) UTSA Ghana Chicago State Ethiopia AAMU Secondary level History, Geography Senegal ECSU Math, Science Senegal ECSU Tanzania SCSU This report presents a synthesis of findings, lessons learned and recommendations from all six countries. Research conducted in all but South Africa (where TLMP had ended in 2008) was standardized to the extent feasible so as to be able to report consistently on project successes and challenges; the impact assessment conducted in South Africa posed different questions, focusing more on sustainability and how this was achieved. Where possible, information will be summarized, but where specific points need to be raised, the country will be identified and findings presented. The primary audience for this evaluation is: 1) staff of E3, the Africa Bureau of USAID and of Missions involved with implementing TLMP; 2) the implementing partners and other stakeholders; and 3) USAID education officers whose work involves the development, production, distribution, and use of textbooks and other learning materials. 2. BACKGROUND 2.1 TLMP Overview – USAID/Washington TLMP was an innovative project which had as an intended outcome strengthening the ability of MSIs to participate in and manage development activities in collaboration with host country organizations. The successes reported in the AEI evaluation conducted by the Aguirre Division of JBS International (17,726,000 TLM were produced) led USAID’s Africa Bureau to extend CAs to MSIs either to continue their collaborative work in their respective countries or to launch a new TLMP (UTSA in Malawi). Each CA under the expansion was valued at around U.S. $13 million, an amount that was reduced in 2010 to approximately U.S. $8 million per MSI. Over the course of the expansion phase, USAID/W continued to orient the MSIs on how to work with USAID and MOEs in a collaborative manner, and how their own institutions would likely need to change their policies and procedures to implement a large USAID project. No specific cross-cultural, historical, and development orientation on each of the TLMP countries was provided. Based on the findings presented in the AEI evaluation, USAID’s Africa Bureau determined that teacher training (TT) on the use of the TLM should be integrated into TLMP. Consequently, USAID allocated additional funds to each of the five Missions largely to provide some TT support to TLMP and to enhance Mission ability to play a monitoring role.1 However, USAID/W and the Missions were severely 1 USAID/Ethiopia was the mission that most extensively carried through on mission-based support to TLMP. The Education office funded a contract to provide teacher training, funded the purchase of a sufficient number of textbooks to improve the student: textbook ratio, and encouraged the Project Director to share her findings and reports so that the Mission could keep abreast of activities. 2 Evaluation of TLMP challenged to provide TLMP oversight after funding for TLMP was cut. As a result, both USAID and MSIs had limited opportunity to benefit from mutual feedback on reported activities. 2.2 MSIs MSIs were awarded their CAs under AEI for their potential to become effective USAID development partners and on the basis of their proposals. Awardees were all state institutions of higher education that had experience, in varying degrees, in conducting research and in publishing findings, all had a history of serving minority populations (four of the five MSIs are Historically Black Colleges and Universities (HBCUs) while one, UTSA directs its educational efforts to the Hispanic community in Southwest Texas), all had a very active college of education (some of which had earned accolades in teaching minority populations), and, for the most part, all had supportive university leadership that was willing to broaden the university’s role to include work in Africa. While each MSI had experience with individual Africans among its faculty and student population and some individual faculty may have had some work experience in Africa, no college of education had institutional experience in working in Africa. Through AEI, MSIs increased the student: textbook ratio and improved access to learning materials in their respective countries. With this AEI experience, USAID/W awarded the MSIs CA to continue their TLMP work. 2.3 MOEs The education contexts into which each MSI entered were largely ones of frequent change. MOEs in all TLMP countries were still building their internal, institutional processes while being strongly influenced by foreign (largely former colonial) countries and by donors. Additionally, while TLMP was designed on the assumption that materials would be developed in and for learning in English (or in French, in the case of Senegal), in the midst of TLMP implementation, at least three MOEs determined that their priority in literacy would be in learning how to read in regional mother tongues. Institutionally, MOE’s had to cope with ever-changing leadership and funding priorities. This meant changes in policies and practices; decentralization of educational decision-making; changes in curricula and pedagogical philosophies; very high staff turn-over at all levels; inefficient school and department financial allocation systems; and changing language skills required in teaching, among others. MOEs typically have little experience in the “nuts and bolts” of textbook production. Generally textbooks have been prepared in conjunction with technical assistance projects or have been adaptations of texts published by international firms. Relatively few MOE personnel have the responsibility to produce textbooks. In some countries, particular books are prescribed for school use; in others, schools and sometimes individual teachers can order off a list of approved texts. If texts are produced by the private sector, the private sector firm often takes on the tasks of storage and delivery to district offices and/or the actual schools that have chosen their books. Sometimes, the publishers provide training for teachers on how to use the books. Based on our observations, teacher competence in the countries visited is often way below what is required. With changing requirements for entrance and years of training at teacher training colleges (TTCs), along with changing curricula and language of instruction, teachers are often not prepared to teach the range of subjects for which they are assigned, and teachers of lower grades must be prepared to teach in self-contained classrooms. For the middle and upper grades, a shortage of teachers in one subject area will necessitate reassignment of a teacher who may not have the subject matter competence. Additionally, a teacher may be assigned to teach at a school where s/he has a weak or non￾existent command of the language of instruction. Also, very little time is allotted at TTCs either to the teaching of reading or to the effective use of textbooks. When Education for All (EFA) was adopted by most African countries and school fees at the primary level were in large part eliminated, classrooms – especially at the lower grades – became overcrowded (in many cases, with up to 200 students per class) as children, particularly girls, who had heretofore 3 Evaluation of TLMP been left out, were enrolled. This caused enormous strains on the ability of education systems to supply teachers with even minimal qualifications and to provide resources such as textbooks, let alone the major infrastructure needs for new schools and classrooms. 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS 3.1 The Evaluation Team The international team was comprised of a project director, a team leader, and four other U.S. education specialists with experience in sub-Saharan Africa. Apart from the project director, who participated in field work only in South Africa, each of the other team members conducted field work in at least two countries, an approach designed to promote comparability and consistency of the research. In each country other than South Africa, each U.S. specialist was paired with a local specialist. Because of the size of the country and of the activity, there were three U.S. and three local specialists on the Ethiopia team. In the other four active TLMP countries, there were two U.S. and two local specialists per team. In South Africa, where TLMP ended in 2008, one local specialist joined the two U.S. evaluators; because TLMP is in limited, if any, use outside of the northeastern part of the country, we were not able to make use of the services of the anticipated second local specialist. 3.2 Evaluation Questions The objectives of the evaluation were to: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions, MOEs, etc.), student achievement, teacher performance, amongst other criteria; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Review allocated USAID funding in terms of usage and overall cost effectiveness; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new Agency Education Strategy. An important aspect of this evaluation was to determine what impacts, if any, participation in TLMP had on MSI’s in their ability to be engaged in USAID projects. See Annex A – Scope of Work for a detailed presentation of the evaluation assignment. 3.3 Evaluation Methodology A mixed-methods approach was taken to conducting the research (see Annex B - Workplan). In designing the research, qualitative, open-ended questions were developed to collect data on and from the MSIs and stakeholders in each country. The methodology included review of project documents and materials produced; interviews with MSI administrators, U.S. and local TLMP implementers, USAID/W and Mission staff , MOE staff at various levels, educators and others collaborating on the TLMP materials, and other stakeholders; classroom observations; and informal student reading assessments. A TLM assessment instrument was also developed and utilized where sets of TLM were made available (full sets for Ethiopia and Ghana; partial sets for Senegal, South Africa and Tanzania). This exercise could not be undertaken in Malawi because representatives of the Malawi Institute of Education (MIE) said that the TLM were not available. Data collection for this evaluation began with on-campus interviews at the participating MSIs by the project director or the team leader. MSIs were both the subject of research and the origin of background research. Interviews included TLMP staff and campus officials, including one university president and one chancellor, and addressed the CA, how it was implemented, any effect that implementation of TLMP had on the MSI operations, and the impact of TLMP on the MSI community. 4 Evaluation of TLMP After analyzing relevant documents and the data from the MSI interviews, a generic research design was prepared and discussed with our COR and others at USAID in Washington. As the parameters of TLMP in each country became clear, country-specific adjustments were made as appropriate. See Annex C – Ethiopia Research Questions for an illustrative sample. Country visits were scheduled to accommodate the changing availabilities of each Mission. At the beginning and the end of each country visit, briefings were held with Mission staff. Country visits typically lasted from 10 to 13 days; following the first three country visits (Senegal, Tanzania, and Malawi), the team leaders from each of the teams conducting those evaluations joined in Ethiopia to compare notes and to conduct the evaluation of TLMP in Ethiopia. Drafts of the country reports were prepared as the teams went along, and at the conclusion of all six country visits, they were refined and used as the basis for this synthesis report. The country reports form Volume II of this Evaluation. Following is the final schedule of country visits and international team members: 1. Senegal Jan 26-Feb 9 Eric Allemano, Carol Benson 2. Tanzania Feb 17-March 1 Jim Wile, Carol Benson 3. Malawi Feb 17-March 2 Nancy Horn (Team Leader), Tom Tilson 4. Ethiopia March 3-19 Nancy Horn, Eric Allemano, Jim Wile 5. South Africa May 5-16 Ed Allan (Project Director), Carol Benson 6. Ghana June 9-20 Tom Tilson, Eric Allemano To reduce the possibility of inter-evaluator drift and inter-country drift, with the exception of the Project Director’s participation in the evaluation of Ithuba in South Africa, each member of the evaluation team participated in at least two country visits, and the teams had different compositions for each country. Country visits were structured to begin with meetings in the capital with TLMP partners, MOE officials, and other stakeholders, and to conduct pilot data collection in at least one school. This latter was done not only to obtain the data and identify potential issues but also to help assure that both international and national evaluators had a common understanding of the evaluation and its approaches. Teams then divided into sub-teams, each comprised of a U.S. and a national education specialist, with each sub-team visiting schools and education officials in locales in different areas and with different demographics located within a radius of about one day’s journey each way from the capital. On return to the capital, the teams conducted follow-up meetings with implementers and others and began writing up their observations. A summary of school-based research activities is included in Annex D – School-Based Research Activities. 3.4 Limitations to the Study • The team targeted schools with different demographics (e.g., urban-rural, different ethnicities and economic bases). However, largely due to the distances that had to be travelled (even including by air in Tanzania), only a limited number of schools could be visited. Therefore, we feel that we should characterize the schools visited as being illustrative rather than representative. Time constraints also precluded the collection of comparative data from schools that had not received TLM. • Several USAID missions postponed the evaluation a number of times, calling for frequent rescheduling and the need to juggle the work of team members. • Interviews with MOE officials were hampered by: strikes; staff turn-over; lack of awareness of TLMP and/or confusion between TLMP and other interventions; and difficulties in contacting and arranging meetings with appropriate personnel. • The ability to visit schools was hampered by: teacher strikes in South Africa, Senegal, and Malawi; warnings of a meningitis outbreak in southern Ethiopia; changes in school or regional leadership; involvement of schools and their leadership teams in other workshops; class scheduling (typically, lower grades begin class between 7:30 and 8:00 in the morning, and end between 10:30 and 12:00, 5 Evaluation of TLMP generally making it feasible to conduct classroom observations in only one school a day owing to the inability of reaching two schools at times when observation of TLMP-based classes would be possible); flooding; and vehicle breakdown in Tanzania; a shortage of schools still using TLMP materials. • Data on numbers of teachers trained and MOE staff members involved in TLM development were not consistently available. 4. FINDINGS AND LESSONS LEARNED 4.1 Introduction Distinctive patterns emerged when data collected from MSIs, in-country stakeholders and USAID Missions were analyzed. In this section, we present these patterns as part of a systems analysis of the elements of the TLMP. Before continuing with details of particular aspects of TLMP implementation, we wish to highlight a central design flaw, which is discussed in some detail in section 4.7.1. In the design of TLMP, the overwhelming focus was on the development and production of the textbooks and learning materials with relatively little attention paid to how teachers would make use of the materials developed. However, in their own learning experiences prior to teacher training, many – perhaps most – teachers would have had limited access to textbooks themselves, often new approaches to the teaching-learning process were incorporated in the materials, and training in the use of textbooks is not a normal part of pre-service teacher training. As a result, many teachers had difficulties in using the materials effectively. USAID did respond to the first of the TLMP recommendations in the JBS evaluation of AEI, “Expand resources dedicated to teacher training, in concert with Mission and MOE teacher training programs.”2 However, we feel that greater attention should have been paid to the actual implementation of the teacher training. 4.2 TLM Produced and Summary of Positive Impacts Each MSI facilitated the production of a very impressive number of TLM. For a full listing of materials produced, see Annex E – TLM Produced. COUNTRY LEARNER TLM TEACHER’S GUIDES TOTAL PRODUCTION Ghana 5,446,706 380,653 5,827,359 Ethiopia 12,200,284 145,000 12,345,284 Malawi 5,208,968 260,120 5,469,088 Senegal 3,050,000 N/A 3,050,000 South Africa N/A N/A 2,300,000 Tanzania 1,660,000 60,400 1,720,400 TOTAL 30,712,111 Evidencing post-project local interest in TLMP’s materials, as of October 2013, the Malawi Institute of Education had reprinted two million more TLMP books and had issued a tender for seven million more. Based on interviews and documentation, as a result of TLMP participation, MSIs: • Improved their administrative and financial operations to enable them to work with USAID and to implement projects that entailed procurements in and for developing countries – in fact, CSU and UTSA have been continuing to work on Mission-funded education projects in Africa; • Expanded their exchange programs (including expansion to universities outside of Africa); • Internationalized their curricula; 2 Aguirre Division of JBS International, Outcomes and Impacts Evaluation of the President’s Africa Education Initiative, Executive Summary, October 2009, page 14. 6 Evaluation of TLMP • Included international topics in faculty development and tenure requirements; • Established and/or strengthened other linkages to developing countries, including joint research activities in fields other than education such as health and agriculture; • Developed proposals, individually and/or jointly, to participate in international development projects; • Increased their participation in professional association meetings; • Increased their visibility in the international education field and/or in the community; • Increased the development of public-private partnerships to work in developing countries; and • Generated long-term relationships with educational institutions and USAID missions in each of their respective countries. As a result of TLMP participation, staff of MOEs: • Increased their capacity to develop textbooks and teaching materials; • Developed textbooks in the national languages of Malawi and South Africa; • Devised and/or streamlined the textbook production and approval process; • Developed the ability to edit other authors’ materials; • Developed the skills to field-test new materials; • Developed stronger inter-institutional practices that gave rise to better integrated operations; • Recognized the need for the inclusion of all new materials in pre- and in-service teacher training; and • In at least two countries (South Africa, Malawi), TLMP-based materials have become the impetus for other donors to provide support for analogous materials. Overall, TLMP increased the availability of high quality materials in line with the curriculum although addressing of cross-cutting issues (gender stereotypes, HIV/AIDs, and health) tended to be weak. Institutional success of those participating includes: • Printers generally had to hire more employees (although on a temporary basis) to address the production needs of the TLM contract. (Some printing was done in China by USAID/Ethiopia due to contracting regulations, the Oxford University Press (OUP) in Tanzania had some printing done in Kenya since OUP could access resources more easily, and by a French publisher for several runs in Senegal.) • Illustrators were either trained to create original drawings or were identified through their reputation or the appearance of their work in other published materials. Their work was appreciated by the MOE, teachers and learners as the illustrations helped in the learning process. • Distribution systems were developed and managed more effectively by both the MOE and printers. In general, pilot schools received enough TLM to give each student access to a book. • Teachers said they liked the TLM a lot, largely for the comprehensiveness of the teacher’s guides (the guides for Ethiopia and Ghana had fully-developed lessons included for teachers to follow), the colorful design and illustrations of the learners’ books, and the contextualization of the vocabulary and stories. • Learners in South Africa and Malawi liked the stories, the colorful pictures, the small physical size of the books, the materials in African languages, and the familiar “situations” presented in the stories. However, in our view, a major opportunity was missed. Only one MSI, Elizabeth City State University, consistently submitted materials developed to DEC, and these materials do not appear on the DEC search engine. Although the materials have great value as models for replication in other countries, realistically speaking, the TLMP materials are lost to the development community. 4.3 MSI Preparedness and Management 4.3.1 Discussion on MSIs The experience MSIs gained in implementing TLMP helped them learn a great deal about what was needed to manage a USAID project effectively. There was also some degreee of information-sharing and 7 Evaluation of TLMP cross-fertilization with respect to promotion of teacher-developed materials. However, certain significant issues remained: • University Leadership Participation: Many of the changes that MSIs needed to make in order to support TLMP (e.g., changes in procurement practices) required very high-level decision-making. Consequently, TLMP directors had to spend significant amounts of time addressing presidents, deans, and members of boards of trustees to convince them that USAID participation would benefit the university community. This was particularly critical when senior leadership changed. • Teaching and Travel Schedules: Most MSI faculty continued to teach and therefore could schedule country visits only during breaks or sabbaticals. The timing of their trips did not always match the needs of the project, sometimes causing hiccups in the flow of work. • Needs Assessments: While each MSI launched its program under AEI with a needs assessment, this was generally not updated for the expansion phase. As a result, when they prepared materials for the expansion, implementers lacked adequate or appropriate information on teacher preparedness, class size, teaching methodology, etc. • M&E System and Performance Monitoring Plans (PMP): MSIs did not maintain effective Monitoring & Evaluation (M&E) systems during the expansion phase, either for the CA as a whole or for assessing the effectiveness of materials produced as actually used, as was recommended by the AEI evaluation, with the result that tracking progress toward objectives was challenging. • In-Country Oversight: Three MSIs opened small offices during the expansion of TLMP to provide for smooth implementation (UTSA in Malawi, a new TLMP country; AAMU in Ethiopia as a teacher training project was added; and Ghana, where TLMP transitioned into another project). SCSU felt it did not need an office as previously developed materials from Zanzibar were being adapted for the mainland by the MOE; and ECSU had turned significant implementation activities over to the MOE. • Program Advisory Committees (PAC): PACs were established in each country. Their purpose was to facilitate local ownership, provide expert opinion on the process and output of TLM development, and to provide a forum for mutually beneficial discussions among MOEs, related stakeholders and MSIs. However, issues arose when local members expected to receive a “sitting fee,” reimbursement for travel expenses, and meals, all of which had not been included in the TLMP budget. Initially local members of PACs provided valuable linkages to key stakeholders, and performed a limited number of activities. In all cases, however, PAC activities declined to the point that they no longer meet. • Quarterly and Annual Reports: While each MSI did prepare its quarterly and annual reports, on reviewing these documents, several things became apparent: the quarterly report for one MSI had the same content over most of the duration of the project, changing only the number of TLM produced; two MSIs consistently under-reported their activities; and no MSI submitted a formal workplan or M&E system. MSIs made various erroneous assumptions about the development and use of TLM: • MSIs relied on faulty assumptions as to the processes by which MOEs approved TLM and by which schools decided which TLM they would use. In Senegal and Tanzania, publishers developed and published textbooks under the guidance of an MOE unit; in Ethiopia books were produced on bid by external publishers and were in competition with materials developed through donor-funded projects; in Malawi, the MIE was responsible for developing texts, but these were printed on bid by commercial printers. In South Africa, a curriculum unit was responsible. In some cases only one text was available; in others there were several. • In South Africa, Malawi, and Ethiopia, implementers assumed that the level of learner competence in mother tongues (L1) and in English (L2) was similar to that postulated by the curricula. It wasn’t similar, and so materials were too difficult for learners. (To a very large extent, this has been a factor even for very experienced implementers.) Apart from just getting these levels of actual 8 Evaluation of TLMP competency wrong, 3 this was exacerbated by the use of “whole word” methodology, which is generally not appropriate for teaching literacy in African languages. 4 • In Ghana and Ethiopia, many lower-grade primary school teachers proved to have only limited competency in English, making it extremely difficult for them to read and put into practice the guidelines in the teacher’s guides as to how the TLM were to be used. • MSIs assumed that teachers knew how to integrate the use of several different materials when, in practice, teachers knew only how to teach using only a single textbook. Implementers seem not to have realized that strategies for teaching reading, per se, had not been a part of teacher training and were not really a part of the early grade curricula. Further, the teaching methodology utilized in the TLM was not able to address very large size classes. 4.3.2 Conclusions and Lessons Learned • In conducting baseline or needs assessment research, the implementers should have asked critical questions on the local educational system, pre- and in-service teacher training practices, the level of learner and of teacher competency in English and mother tongues, textbook use, learning outcomes, and related topics. • TLMP CA managers learned that the support of senior MSI leadership was critical for the development of appropriate institutional policies to facilitate project implementation. • Small administrative offices established by MSIs assisted in making timely decisions to ease the TLM production process. • The absence of effective M&E systems meant the work was not being planned or monitored appropriately. • PACs were not as useful as they were expected to be, largely because MSIs assumed their work would be voluntary and not remunerated. 4.4 MOE Commitment and Participation 4.4.1 Mixed Processes and Outcomes MOEs in each country work in a resource-deficit environment in which donors negotiate with MOE to implement a certain project. Consequently, in each country visited, the MOE was involved in “piloting” a number of donor-funded projects, the evaluation of which would determine which program they would use to teach reading and other subjects. Our visits often took place where several projects were being piloted, and significant discussion had to take place with administrators and teachers as to which project we were there to evaluate. Therefore, at the MOE level it was difficult to ascertain whether they were fully committed to TLMP. The MOE experienced a number of other challenges in terms of making a decision on which approach was the most effective: • In Ghana, the leadership of the MOE changed three times in two years (making it necessary for CSU to get approval for its activities after each change), with subsequent policy changes. In 2007, one year after the launch of TLMP under AEI, the MOE reordered its priorities and began greater focus on early grade reading in home languages. • In Ethiopia, curricular revisions take place every five years, and the latest revisions were drawn up by an external consultant, who may not have been adequately informed about the regional variations in educational practices, i.e., the grade at which English becomes the language of instruction. 3 There are also important philosophical issues involved in deciding whether competencies should be realistic or aspirational, as discussed by the Learning Metrics Task Force (see Brookings Center for Universal Education materials listed in the References). But this question falls well outside the evaluation Scope of Work and, in any event, was not taken into consideration by the implementers. 4 See, e.g., Helen Abadzi, Literacy for All in 100 Days? A research-based strategy for fast progress in low-income countries, 2013, discussed in References. 9 Evaluation of TLMP Changes in textbooks follow the revisions in curricula. In Ethiopia, as in other countries, MOE expectations of English language capability of both teachers and students were not realistic. • The Tanzania MOE had been undergoing decentralization when TLMP was launched. Under the new organization and policies, schools can choose their own texts among those approved by the MOE. Hence TLMP materials were in competition with other books, which were the focus of marketing programs by publishers. Complicating this process is the MOE’s movement toward a competency￾based curriculum that was supposed to be incorporated in all secondary math and science books; only TLMP-produced texts followed this directive. • MOEs have developed criteria to review textbooks produced by publishers and projects, although the review process is very inconsistent. MOEs, in general, have a system for obtaining feedback on usability, but not all research conducted is acted upon. • When a project to develop and print materials is agreed to by the MOE, specific negotiations must outline who is responsible for delivery and what system will be used to track delivery to schools. Distribution to regional warehouses may be provided, but schools must then usually arrange to pick up their own allocations. Arrangements to address shortages/surpluses of materials tend to be haphazard. • Collaboration in the development of TLM was problematic. In South Africa and Malawi, the MOE had to be convinced that teachers could write stories in their national languages that could then be used as supplemental readers. In Ethiopia, the curriculum division of the MOE had bid internationally for English language books and had competitive materials developed; it was not until the final stage of materials development (Ethiopian folk tales) that the MOE regional curriculum developers were brought in. In Tanzania Mainland and Senegal, materials are developed by private publishers. And in Ghana, KG1 and 2 collaborative materials development with faculty and consultants at two universities tried to follow an ever-changing curriculum. • It is not at all clear how much MOE institutional capacity (as distinct from the capacity of individuals) was strengthened. Although MOE officials asserted that they “had had their capacities built” in textbook development, in fact not many MOE staff members participated. Further, when the team members tried to interview the developers, many could not be found because some were in schools, others were in teacher training colleges, some remained in the ministry but were not in positions related to curriculum/materials development, and some had left education altogether. The high degree of mobility, voluntary or otherwise, among educational personnel is a notorious challenge to institutional capacity-building • There was considerable confusion as to the adoption of TLM as the “official” texts in several countries. In Ethiopia, the approach, process and output were initially adopted, but once it was found that the level of English used was too high for teachers and students, the materials were relegated to a “supplemental” status. In Senegal, the arduous process of getting materials approved as textbooks impelled getting them adopted as “support materials” instead. On Tanzania’s mainland, schools choose which textbooks they use. On the other hand, the materials for South Africa and Malawi were intended from the beginning to be supplemental readers. 4.4.2 Conclusions and Lessons Learned • Textbook development projects work best when the various directorates of the MOE are working together in an overall learning improvement program; appropriate and consistent patterns of communication would facilitate the process between MOEs and MSIs. • The elements of a collaborative textbook development program need to be spelled out in the CA to prevent any confusion as to who is responsible for what, including delivery. • The presence of other donors implementing pilot literacy and teacher training programs confused teachers and evaluators. Stabilized MOE policies and procedures would assist in determining which intervention has the greatest efficacy. • Establishing and building the capacity of textbook review committees that have specific guidelines on 10 Evaluation of TLMP how to ascertain the appropriateness of a book for each grade level would assist the MOE in making textbook decisions. If schools choose their own texts, as in Senegal and Tanzania, these guidelines should also be shared at the school level so that appropriate choices can be made. • The commitment of MOE about the adoption of textbooks into the official curriculum should be made in the Memorandum of Understanding (MOU) so that donors and developers can ensure their use, at least for a period of time. • When adopted as official materials, TLM should be incorporated into pre- and in-service teacher training so that current and future teachers can become adept at using them in teaching and learning. 4.5 Teacher Readiness and Participation 4.5.1 Teacher Readiness In each of the TLMP countries, the long-standing deficit in the number of teachers trained was exacerbated by the enormous expansion of enrollments as the result of Education for All (EFA). Using Ethiopia as an example, at least ten years ago would-be teachers had to hold 12th grade leaving examination certificates, and attend a TTC for one year. As this system was not producing a sufficient number of teachers to fill demand, the entrance level was lowered to a 10th grade leaving certificate, with one year of TTC. Research determined that at TTC graduation students did not have enough subject matter knowledge, so the curriculum changed and graduation requirements now include passing a three-year diploma course. For already practicing teachers, the MOE required them to take supplementary courses to earn their diplomas. The MOE is now working on teacher certification that will allow teachers to obtain teaching jobs in other countries, and the curriculum is now being compartmentalized so that would-be teachers can focus on language arts, math and science, or moral training. Those with diplomas in language arts will have proficiency in teaching the mother tongue of the region in which the TTC is located and English. This is to satisfy the MOE priority to teach children how to read in their home languages. Other challenges inhibit teacher development: • In-service teacher training (INSET) is delivered in all TLMP countries, whether to bring teachers up to a certain minimum standard of qualification (e.g., a three-year program to bring Ghanaian and Ethiopian teachers to diploma level) or to strengthen skills in particular areas. In several countries, INSET that focuses on methodology and pedagogy is provided largely by donor-funded projects and, as a result, generally is limited to project-involved schools. In Ethiopia, TTC faculty provided INSET on a regular basis at cluster center schools to which they were affiliated and to which they send practice teachers, but when the USAID-funded teacher training project ended, training was provided only intermittently. • MOEs are increasingly prioritizing use of the mother tongue (L1) as the language of instruction, particularly for the lower grades. However, very little attention is paid at TTCs to the teaching of reading, for either L1 or L2, second and/or a European language. Even when curricula are in place, few L1 materials, either for language arts/literacy or for other subjects, are available. • Changing policies brought kindergarten (KG) teacher training in Ghana under the domain of the MOE (similar changes were documented for Ethiopia and Malawi). Prior to this policy decision, KG teaching and learning was in the hands of churches, community organizations, and individuals; there was no standard for training KG teachers. When TLMP was introduced in Ghana, the curriculum for training KG teachers had not yet been finalized. 4.5.2 Teacher Participation • Teachers in all TLMP countries played a critical role in TLM development. In South Africa and Malawi, teachers and other educators were trained to write stories based on their respective contexts and experiences, which were then edited, illustrated, and printed in 11 South African languages and in two Malawian ones as supplemental readers. In Ethiopia, teachers who had 11 Evaluation of TLMP participated in a British Council-sponsored intensive English language program (of approximately 220 hours) were selected to develop the TLM for grades 2-4; several of these were faculty members at TTCs. In Tanzania, teachers of secondary math and science were enlisted to review the Zanzibar materials and develop additional materials for use on the mainland. In Ghana, early childhood educators contributed to the type and content of TLM developed. In each country, the MOE exercised varying levels of oversight over the materials being developed by TLMP. • Many teachers took the skills they developed and integrated them into their own teaching. For example, they continued to develop stories and, in South Africa, they proposed these stories for publication. In some instances, the teachers had their students write stories about their lives and then had their fellow classmates read and discuss them. 4.5.3 Conclusions and Lessons Learned • Educational and textbook development projects work best when policy on pre- and in-service teacher training is well defined and well-supported. • Teacher involvement in TLMP increased demand for better teacher training, textbooks that are aligned with the curriculum, lessons that can be taught in a particular time frame, and the availability of textbooks on a 1:1 ratio of learners to textbooks. • Teacher training programs do not generally include the methodology of teaching reading in either L1 or L2 and across the curriculum; this gave rise to TLMP challenges in what assumptions to make in both teacher and learner literacy competency. 4.6 TLM Development, Printing and Distribution 4.6.1 TLM Development TLM development was different in each country. As noted above, UTSA involved teachers deeply in story development, field testing, and illustrations in South Africa and Malawi. In Malawi, textbook development typically comes under the MIE; for TLMP, the MIE played a supportive role, lending its expertise in working with teachers in story development. In other TLMP countries, a mix of external and internal consultants, internal MOE staff, university experts, and MSI faculty developed the TLM. A standard practice at project outset under AEI was that the MSI faculty would first develop materials, which the in-country developers would then review and provide their feedback to the MSI faculty, who then made the changes. This practice was generally reversed in the extension phase, especially in Senegal and Tanzania; typically, the MOE-designated experts took over the majority of the work, sending it to the MSI counterparts for their review and comment. Alignment. TLM were aligned with current national curriculum priorities; alignment with cross-cutting themes such as gender equity, HIV/AIDS and other health issues tended to be weak. All TLMPs followed the current curriculum priorities, including topics to be mastered in each subject at each grade level. However, in some instances, the curriculum itself was not well developed and, in many cases, the curricula assumed academic achievement and/or linguistic competences that had not been achieved. Design. Once the materials were produced in narrative form, illustrators were hired and/or teachers trained to draw pictures illustrating different points in a story or to teach vocabulary or grammar. For the most part, illustrations were done well in terms of the pictures drawn, but teachers interviewed reported that sometimes the pictures didn’t fit the picture/point, sometimes the illustrations were printed in colors that did not print well, and often pictures illustrating vocabulary or grammar points did not include any legend underneath explaining the point of the illustration. In two cases – illustrations for TLM in Ghana and Senegal – the nude anatomical illustrations were inappropriate. Methodology. While there is a place for “whole word/whole language” methodology for the teaching of reading for languages such as English, in which the orthography of the language does not closely match its phonology, and, as referred to on page 9, it is considerably less useful than phonemic-type approaches to teaching reading (and writing) in African languages. 12 Evaluation of TLMP Field Testing and Piloting. Field testing was a challenge, largely due to the limited time available and the expertise needed, and there was little actual pilot-testing of materials. In most countries, field testing consisted of a teacher/developer taking advanced copies of the TLM to a classroom and having learners respond to questions that teachers posed largely on the physical presentation of the materials. Teachers were not required to actually use them in the classroom. As a result, the field testing was inadequate for a number of reasons: • Many schools at which field testing was conducted were “demonstration” schools (e.g., schools associated with teacher training colleges) at which teachers and students had additional input from TLMP partners, who could give personal guidance to teachers who needed assistance; • The feedback was given by students who performed well (i.e., they were not the “average” student); • Specific guidelines were not provided to conduct rigorous field testing; and • “Average” teachers who would be responsible for teaching from the TLM were not asked to respond. This latter point is particularly troubling as teachers had not been given the opportunity to actually work with the TLM to determine any issues. Lack of teacher competence to field test the materials was a further complication: • Teachers frequently lacked the capacity to teach; • Generally, the teachers’ command of English was not at the level anticipated by the teacher’s guides; • Although there has been greater emphasis on mother tongue teaching, teachers may not have adequate pedagogical competence in their own language; • Teachers were not used to integrating several learning resources into a lesson; • Not all teachers knew how to prepare lesson plans – or how to use them; and • The pedagogy entailed in using the TLM was different from the way teachers were used to teaching (a more learner-centered methodology vs. didactic methods). Leveling. Leveling was also a problem. With MOEs overestimating the language and literacy capabilities of teachers and learners, and MSIs following the curriculum in developing the TLM for each grade level, the language used was frequently too advanced for the designated grade level. Teachers themselves had a lot of difficulty following the teachers’ guides and often had to consult dictionaries to understand what was written. Instructional Time Misjudged. When TLM were actually used by teachers, it was found that some TLM developers misjudged the amount of time needed to teach particular lessons. This took two forms, by no means mutually exclusive: it took longer to teach particular lessons than anticipated, and in some countries, the amount of time allocated for a class was too long and the children became tired. It is not clear from the research if the lessons were actually practiced before they were printed and distributed. Quality Review. Overall, MOEs did not really have a “quality review” process to determine if TLM were produced to meet teacher and learner requirements in terms of language, pedagogy and content. The only country that had this process institutionalized was Senegal, although the process generally entailed books published in France for French students; the MOE had little experience reviewing textbooks specifically for Senegal. 4.6.2 Printing and Physical Production Initially it was assumed that local printers would not have the full spectrum of skills and equipment to undertake contracts to produce the volume of TLM needed in each country and that the MSIs would expand their capacities. The former was not necessarily the case. All printers contracted had a fairly high degree of expertise and equipment and in just two cases, bookkeeping and “kitting” (packaging) required some input, which the MSIs provided. All printers had to hire additional staff to complete the printing, binding, and kitting of the TLM and teacher’s guides, although staff was generally hired on a temporary basis. As the result of having the TLMP contract, however, most printers gained in recognition for their ability to produce a large volume of books. 13 Evaluation of TLMP Specific printing issues arose in several countries. In Ethiopia, USAID/Ethiopia provided funding to print a significant number of TLM, but because of the size of the procurement, the Mission had to obtain international bids. The bid was awarded to a printer in China, so local Ethiopian capacity was not built. In Tanzania, Oxford University Press, the successful bidder, had the TLM produced in Kenya, where paper, equipment and electricity were in steady supply. In Senegal, ECSU contracted with local printers who had the technical capacity; however, they did not have the financial capacity to front the advance costs, e.g., importing paper. As a result, ECSU worked with its various partners to develop appropriate credit arrangements, which transformed this challenge into a win-win-win situation. In Ghana, CSU arranged for the procurement of printing equipment for the MOE, which proved not to be necessary because the MOE did not do its own printing. A specific issue in textbook production emerged in Ghana and Ethiopia. Student TLM were produced in the form of expendable workbooks for KG and grade 1, respectively. It was assumed that the MOE or “someone” would cover the recurrent expenditure of annual reprinting. In reality, materials have to be reusable year after year. Some teachers in both countries have developed “work-arounds,” e.g., trying to erase previously used workbooks or copying pages for children to write on. But in practical terms, once the available supply of workbooks is exhausted, schools stop using the TLM. When textbooks or other materials are prepared, there should be no assumption that workbooks or hand-outs would be available and that exercises would need to be done using plain paper. We should note that in South Africa, schools even in very remote areas were equipped with photocopy machines, computers, and printers, so in principle, local reproduction of materials is possible. 4.6.3 Distribution In some cases, confusion arose as to who was responsible for distributing the TLM: the MOE or the printer. For the most part, MOEs were responsible for delivery of materials to a district education office or a regional warehouse. However, shortfalls in funding and logistics (vehicles) prevented timely distribution to the schools themselves. Publishers whose books had been ordered by schools delivered their materials to those schools. When printers were asked to deliver TLM to schools, they incurred additional costs, some of which they decided to “donate” as an in-kind contribution. The systems for delivery varied, resulting in some TLM being “lost” along the way (initially the case in Malawi until the TLMP leadership team developed a “foolproof” system of delivery). The systems typically broke down when school representatives were responsible for picking up the number of books allocated for their school – if they were informed of their availability and if they had the transport. Once TLM were picked up, internal school distribution was not guaranteed as principals and/or teachers were not always prepared to receive and/or use them, also an endemic problem, especially where educators were held liable for replacement of lost books. 4.6.4 Marking, Copyright, and Submissions to DEC • The materials reviewed appeared to be marked appropriately for cooperative agreements. • In Senegal and in Tanzania, there were differences in understanding among stakeholders as to the rights and permissions to edit, update, revise, and/or duplicate materials produced by TLMP. This caused problems in revising and reprinting materials in Tanzania, where most of the responsibility for textbooks is in the hands of the private sector. • Training should be provided in intellectual property rights. In Senegal, content (text and pictures) developed by other entities was included without obtaining permission. • USAID should provide greater direction to implementers as to USAID’s expectations on the submission of TLM to DEC. As discussed on page 7, these are valuable resources that could be of great value to implementers of analogous projects – if they were available. 4.6.5 Conclusions and Lessons Learned • TLMPs functioned better once the roles and responsibilities of MOEs and MSIs for collaborative TLM development were spelled out. 14 Evaluation of TLMP • Involving teachers and educators in TLM production yielded significant TLM output as well as teachers who were very familiar with the materials and could use them effectively. It also increased the possibility of teachers repeating the process with their students. • Leveling any language materials is absolutely essential to creating positive learning outcomes among learners; appropriate review policies, procedures and staff must be available to carry out these tasks. • Field testing works best when adequate time and training are provided to all types of teachers from a broad array of schools (urban, peri-urban and rural; well-resourced vs. under-resourced, etc.) to learn what works best in what context. • A properly trained quality control unit at each of the MOEs might have identified some of the inconsistencies and errors in production, e.g., inappropriate illustrations, legends under pictures, and teacher’s guides written at a realistic level. • Procurement processes for reprinting materials should be decided upon with the collaborating partner so they are not cumbersome and other organizations that desire to purchase the materials can do so in a timely manner. • Greater clarity is needed with respect to the ownership and rights associated with copyrights • Implementers should not consider TLM, including workbooks, as expendable and should print them to last for at least a couple of years. 4.7 Use of Materials by Teachers and Learners 4.7.1 Challenges Inherent in Teacher Training Practices Producing textbooks does not necessarily equate with improving learning. The methodologies teachers use in presenting the materials affects learning, both in the classroom and at home (if the children are allowed to take the TLM home for their own study), and the didactic methodologies still in use do not produce the desired learning outcomes. The challenge to TLMP in providing training to teachers on the use of the TLM was that TT was not an inherent feature of the CA. Consequently, TLMPs had to rely heavily on the MOE to provide the training needed. Training teachers in using more learner-centered methods that were incorporated in TLM has been an MOE priority in many TLMP countries. However, TTCs have not fully embraced these approaches because they are often seen as “project related” rather than a reflection of MOE policy. As one faculty member of a TTC reported, “When the project is here, we are ‘hot.’ When the project ends, we ‘cool down.’” Since MOEs are testing multiple approaches, a number of issues related to training teachers on the use of the TLM emerged: • Teacher training curricula have been slow to adopt new materials largely because of the changing policy environment and the ability of TTC faculty to incorporate new materials in their lessons. One of the two TTCs the team visited in Ethiopia and both of the ones in Malawi included the TLM in their training, but in each instance it was a part of a larger topic of “supplementary materials” and then only for about an hour over the whole study period. The other TTC in Ethiopia incorporated the TLM in a major way, but we believe that this happened primarily because 1) a Peace Corps Volunteer had been recruited to teach solely on the TLM, and 2) several of the materials developers were on the faculty of the TTC so they were very familiar with them. • INSET on the use of the TLM occurred – in one form or another – in Ghana, Ethiopia, South Africa, and Malawi, where the focus was on primary level TLM. The cascade training-of-trainer (TOT) approach was generally taken, with selected educators receiving training and a training manual on the use of the TLM; these trainers then trained head teachers in a particular region or district, and head teachers were to cascade the training to their school-based colleagues. The approach and content did not produce the desired results, largely owing to the following: • The pronunciation, vocabulary and writing skills of the teachers were frequently not well developed. • The underlying pedagogy inherent in the materials was not a focus of the training, so many 15 Evaluation of TLMP teachers missed the point and just followed along with the lesson script without realizing that they were also supposed to be learning how to teach in a different way. • The training offered to teachers through the cascade model was “watered down” at each level, with the teachers actually teaching in the classroom receiving the least substantive training. • TLM did not address assessment of student learning. • Training needed to be conceptualized as a long-term process entailing more than a “one-off” workshop; except in Ghana (where there has been morphing of TLMP and other interventions), there was no real follow-up. • Training methods and techniques did not address the issues of teaching overcrowded classrooms, teaching with an insufficient numbers of books, and the use of supplementary readers. It was the expectation of USAID/W that each Mission would support these activities, but this was generally not the case, although USAID/Ethiopia negotiated a separate contract with another organization to train approximately 2,000 teachers and CSU is supporting USAID/Ghana with basic education activities. 4.7.2 Challenges Inherent in the TLM Themselves A full assessment of the TLM by this evaluation team, based on a standardized format addressing pedagogy and physical appearance, was conducted only in Ethiopia, Ghana and Tanzania because adequate sets of the materials were not available in the other countries. See the sample assessment in Annex F - Representative TLM Assessment. Teachers in all countries had a number of specific issues with the materials: • The flow of materials from one grade to the next was not even; for example, a “linguistic cliff” occurred between grades 1 and 2 TLM in Ethiopia. • Teacher’s guides do not give instruction on how to conduct continuous assessment. • Too many lessons are covered in each unit. • Teachers do not have experience in integrating other materials in their lessons. • Some of the teacher’s guides and learner books are too large, making them cumbersome to carry. • The “shelf life” of most TLM is only two years as cover materials are not very sturdy. • Teacher’s guides and learner books assume an ability to read at a level in English that is beyond the ability of teachers and students. • There is emphasis on vocabulary development or grammar, but there are no exercises. • The selection of words used is often not appropriate for the students’ grade level or competency. • Use of the materials frequently extends the class day. Aside from educator perspectives on having their work day extended, this can interfere with home responsibilities for the children and can also pose challenges when schools operate in shifts. • Books lost cannot be readily replaced. Further, for cost reasons, workbooks used in Ethiopia and Ghana cannot be reprinted. In South Africa, on the other hand, where schools even in remote areas are equipped with copiers, it may be effective and cost-effective to photocopy replacement or additional materials to meet needs. In interviews, teachers said that they use the TLM. However, when conducting classroom observations, effective use of the TLM was not always in evidence. Approximately 65 teachers were observed, most of whom were interviewed after observations. The challenges listed in the previous section address some of the reasons why the TLM were not always used appropriately. Learner use of TLM was also problematic. An assessment instrument was developed by the evaluation team to measure student performance in English – speaking, listening, reading and writing – and was administered by both nationals and Americans in Ethiopia, Malawi (the two countries that developed TLM for grades 1-4) and Ghana (a revised version implemented due to the KG focus). More informal 16 Evaluation of TLMP assessments were conducted in Senegal and Tanzania, where materials were produced in French and English, respectively, and for middle and secondary school. Assessments were not possible in South Africa owing to the “Limitations” noted in Section 3 above. For a sample of the results of this assessment at different schools in Ethiopia, please see Annex G - Representative Reading Assessment. A summary of results obtained from the informal learner assessment of the use of the TLM as students are promoted to higher grade levels includes: • Increased ability to identify letters • Increased sight word recognition with greater exposure to different words • Increased ability to understand the text • Increased ability to read more difficult texts • Increased ability to write as students are promoted to higher grade levels Owing to the often overlapping implementation of other reading and educational programs, the increased ability of learners to master the skills of reading cannot be attributed solely to TLMP. 4.7.3 Conclusions and Lessons Learned • Due to a shortage of teachers in each country, teachers can be assigned to teach subjects for which they are not trained. Moreover, the pedagogy still in use does not correspond to the newer approaches utilized in the TLM, which are more learner-centered. Hence, providing specific training on the use of the TLM is an integral factor in determining their use, whether by trained or untrained teachers. • Without a long-term strategy and policies to adopt the TLM produced, TTCs are reluctant to incorporate them directly into their curricula. • When MOEs agree to co-implement a project for textbook development, budget plans must be developed to ensure future reproduction of materials either directly for schools or for sale in commercial establishment • Integrating TLM training into the TTC and INSET curricula will help build the capacity of future and current teachers to use the materials. • While TLM have the capability of providing the basis for a different type of learning at all levels, depending on subject, more resources are needed at the middle and high school levels, such as science kits. • The ability of learners to enjoy the TLM is hampered by the level of English used (Malawi, Ethiopia, Tanzania), and most require a teacher’s guidance in reading them. 4.8 Sustainability of TLMP Given the number of new basic education projects, at least at the primary level, the influence of TLMP is not likely to be expressed directly, but rather to be incorporated through staffing within the new projects. It is, in general, too early to tell to what extent existing materials will be used, although USAID/South Africa intends to develop new materials for its new project. This notwithstanding, the Limpopo Province of South Africa has received funding from Irish Aid to plan new materials on its own, and Room to Read has expressed interest in providing support. Due in part to the change in Ghana’s policy on languages of instruction, to the fact that workbooks to accompany the TLMP texts are no longer available, and to the fact that there have been subsequent education interventions, TLMP is effectively obsolete; however, its implementer, Chicago State University, is working with Mission-based education interventions. As of the time of the Ethiopia country visit, it was not known to what extent, if at all, TLMP materials would be incorporated in the new Reading for Ethiopia's Achievement Developed (READ) Technical Assistance Project. 17 Evaluation of TLMP 5. RECOMMENDATIONS 5.1 USAID Missions • A critical starting point is the recognition that teacher/instructor training on how to use textbooks and supplementary materials must be incorporated into interventions that involve school-oriented materials development. It would be highly desirable for this to begin with pre-service teacher training. • In the course of project preparation, as is routine, and at project inception, Missions should determine whether other donors are pursuing a similar or related strategy so that there is no unintended overlap between a USAID-funded project and those supported by other donors and should determine in consultation with the MOE what the status is of the current curricula and pedagogical philosophy, whether plans for modifications are being contemplated, and what the anticipated timeframe is. While changes in government or of Minister can result in changes in MOE approaches with little notice, given that often curricula have anticipated life cycles, any advance notice would reduce the potential of wasted effort. Missions should communicate as appropriate with chiefs of party on contemplated MOE changes and the potential impact of such changes in USAID-funded projects. • When choosing an awardee for a textbook/TLM-type production activity, USAID should take into consideration: • Knowledge of the host country context, including the policies and procedures of the MOE. • Experience in developing curricula and TLM in the targeted subjects at the targeted levels. • A commitment to adapting university policies, practices and procedures to be supportive of the successful implementation of a USAID project (e.g., travel authorization, procurement, maintaining field operations, etc.). • Missions might wish to have implementers and the MOE develop a memorandum of understanding that lays out the terms of anticipated collaborations, including the terms of any Project Advisory Committee. • Missions should support the development and implementation of workplans that allow adequate time for pilot testing of TLM. 5.2 MOEs and Systems Improvement Presumably, future TLM activities will be conducted primarily within the context of Education Strategy Goal 1 and/or Goal 3. As part of project planning, Missions should call for MOEs to agree to: • A pre-service and in-service teacher training program on the use of the TLM and the supporting pedagogy within the context of a continuous professional development program, with MOE stakeholders (e.g., supervisors/inspectors and curriculum specialists) participating in the training so that they can be supportive of teachers in follow-up activities. • Identify MOE directorates to have co-responsibility for materials development, project implementation, pilot testing of materials, monitoring and evaluation all under specific guidelines. • Providing implementers with notice of potential changes in policies that could affect their projects. • Keep approved TLM in use for at least a minimum stated period, including in pre-service training, so that teachers can become comfortable with using materials and methodologies and/or adapt them to their own contexts. • A commitment by the MOE to strengthen its internal capacity in curriculum and syllabi development to reduce reliance on external consultants. Note that these commitments, especially those related to pre-service training, need not necessarily be part of the USAID-assisted activities, but should be a funded part of some planned activity. 18 Evaluation of TLMP 5.3 TLM Development, Printing and Distribution Implementers should: • Identify the specific roles and responsibilities that each member of the partnership has so that confusion does not arise at the time of implementation. • Conduct their own assessments of the language skills in relevant languages of learners and of teachers at relevant levels as part of project start-up; this can be done in conjunction with the baseline study called for by the Evaluation Policy. Bring to the attention of the MOE concerns should the actual skills level be substantially different from the assumed skills level. • Work with teachers and educators to empower them to write stories that can be used in classes, whether as “teacher-developed materials” or potentially as broadly distributed materials; and to work with curriculum specialists to develop materials for self-contained and subject matter classes. Consider the feasibility of replicating the methodology developed by UTSA in generating TLM through providing writing workshops to teachers and other educators, who then develop culturally relevant stories in home languages. The project would then take the materials through a range of leveling and pilot testing processes to ensure that children would be able to learn, in a step-wise manner, how to read in their home languages. • Develop teachers’ guides at appropriate levels of language that provide teachers with clear guidance, including notes on methodology, on how they can teach individual lessons. • Produce books for teachers and students that are durable and that are of a physical size appropriate for students in lower grades (e.g., consider dividing a student book into sections, with each to be used for a term rather than the whole year). • Develop a support system for TLM delivery to schools so that schools get materials in appropriate quantities for each grade and language in a timely manner. Implementers and/or MOEs could explore the potential of asking other government agencies with vehicles, such as the police, to assist in getting materials to the schools. • The rights for MOEs to reprint and/or update/revise TLM should be more explicit, especially in countries where textbooks are produced by commercial publishers. 5.4 TLM Use and Assessment Implementers should: • Develop a system within a quality control unit of the MOE for pilot testing in the classroom of materials produced so that they can be leveled for use at appropriate grade levels. Such a system might include: 1) teachers receiving advanced copies to use in the classroom at appropriate points in the curriculum, 2) teachers identifying difficulties in using the TLM, and 3) revisions to the TLM prior to full-scale printing. Such a system would also include quality control measures developed by MOEs and any specific, cross-cutting issues that would be included. • After piloting, periodically assess student ability to use, understand and enjoy the TLM and of teacher skills in using the TLM and associated methodologies. Have active follow-up to determine if the TLM need to be adjusted and/or if teacher guides need to include any other points. 19 Evaluation of TLMP ANNEXES ANNEX A. SCOPE OF WORK Section C – DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls ‘and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History o TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over 3 million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings. o TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over 6 million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English. o TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers guides and training materials and trained nearly four thousand teachers on methodological classroom usage. o TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 1.8 million materials in both French and English for grades 2-10 in science, mathematics, and language arts. Over 160 20 Evaluation of TLMP teachers were trained on utilizing the materials as part of their curriculum. o TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over 6,000 teachers. The work was completed in 2009. o TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over 1,200 teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection. Task 2 – Data Review. Task 3 – Coordination and Management. Task 4 – Site Visit. Task 5 – Data Analysis. IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives: • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based MSIs, in- country institutions, Ministries of Education (MOEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and 21 Evaluation of TLMP • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Strateg y_feb2011.pdf V. OPERATING CONSTRAINTS/ LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance can be found here: http://www.usaid.gov/policy/ads/200/203.pdf The new USAID Evaluation Policy can be found here: http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. END OF SECTION C F.2. DELIVERABLES 1. Instrument Design and Approval- five (5) weeks after Award of RFTOP 2. Draft Final Report- 20 Weeks after Award 3. Final Report- 22 Weeks after Award *Draft Report. The Contractor shall prepare a draft report for review by the COTR. The draft report shall be provided in hard copy and electronically to the COTR. *The final report shall document the analyses performed. The report shall be concise, clearly written, free from typographical and grammatical errors, and less than 20 pages in length excluding appendices. All documentation and products become the property of USAID. 22 Evaluation of TLMP ANNEX B. WORK PLAN International Business & Technical Consultants, Inc. 8618 WESTWOOD CENTER DRIVE  SUITE 220  VIENNA, VA 22182  USA TEL: 703-749-0100  FAX: 703-749-0110  EMAIL: EALLAN@IBTCI.COM Evaluation of the Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania Task Order AID-OAA-TO-12-000054 Draft Work Plan Submitted to Joe Kitts, COR, USAID/Africa Bureau Edward Jay Allan IBTCI Project Director (703) 749-0100 ext. 217 eallan@ibtci.com Nancy Horn, Team Leader Eric Allemano, Team Member Carol Benson, Team Member James Wile, Team Member Version of October 29, 2012 23 Evaluation of TLMP TABLE OF CONTENTS 1. Introduction ................................................................................................................25 2. Project Overview........................................................................................................25 3. Objectives of the Performance Evaluation..............................................................26 4. Research Approach, Methodology and Process ......................................................27 4.1 Contacting and Scheduling Interviews with MSIs........................................................................ 28 4.2 Obtaining and Reviewing Project Documents............................................................................. 28 4.3 Assigning Members of the Team to Each Country..................................................................... 28 4.4 Identifying Target Populations from Whom to Gather Data................................................... 29 4.5 Developing Interview Schedules and Other Data Collecting Instruments ........................... 29 4.6 Scheduling In-country Research Activities.................................................................................... 29 4.7 Review of Cost-Effectiveness .......................................................................................................... 30 4.8 Developing and Editing Preliminary Country Reports............................................................... 30 4.9 Draft and Final Reports..................................................................................................................... 30 5. Management and Logistics ........................................................................................31 6. Draft Timeline..............................................................................................................31 24 Evaluation of TLMP 1. INTRODUCTION The IBTCI team is pleased to present this work plan to USAID/Africa Bureau. This document outlines the work plan and research design IBTCI will utilize to evaluate the TLMP in six countries: Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. This work plan includes a brief overview of TLMP (for context), the approach taken to the research in each country, the activities to be undertaken before and after the field research, the data collection instruments, a preliminary division of roles and responsibilities among team members, and a draft timeline. 2. PROJECT OVERVIEW Begun in 2005 as part of the President’s African Education Initiative (AEI), the first phase of TLMP developed and disseminated over 10 million textbooks and other learning materials to schoolchildren in Ethiopia, Ghana, Senegal, South Africa, Tanzania (Zanzibar), and Zambia. These materials and associated teacher training and support for textbook design, development, and production was performed by five US Minority Serving Institutions (MSI), each paired with the Ministry of Education (MOE) and other relevant stakeholders in a specific country through the modality of cooperative agreements. Building on the success of TLM under the AEI, in September 2009 USAID’s Africa Bureau awarded four of the original MSIs – Alabama Agricultural and Mechanical University (AAMU) (Ethiopia), Chicago State University (CSU) (Ghana), Elizabeth City State University (ECSU) (Senegal), and South Carolina State University (SCSU) (Tanzania) cooperative agreements to continue their work with educators in their paired countries. In addition, the University of Texas at San Antonio (UTSA) continued working with South Africa and was awarded a cooperative agreement to work with Malawi. These cooperative agreements were generally for three years and collectively awarded the institutions approximately $13 million. The overall goal of TLMP was to provide quality textbooks for students. To date, well over 25 million African children have begun to utilize the textbooks produced (more than 500 distinct titles written in 13 languages). According to the Cooperative Agreements, the materials produced were supposed to fully align with national curricula, be culturally relevant, and incorporate significant cross-cutting themes, such as HIV/AIDS, gender sensitivity and equity, hygiene, and leadership. (The research will determine if these requirements have been met.) The materials were produced under institutional partnerships established between MSIs and host country MOEs and other stakeholders. The MSIs are responsible for management of the cooperative agreements, which have been implemented by an in-country team, augmented by technical assistance arranged by the MSI, to ensure program success. MSIs sought to build the capacity of private sector printers and publishers by awarding them printing and distribution contracts. MSIs also leveraged their awards to provide some degree of teacher training on the use of the teaching and learning materials (TLM), although teacher training per se is not a primary purpose of TLMP. In summary, MSIs saw to it that capacity was built at all levels of textbook development, production, distribution, and training teachers in their use so that the countries can continue to meet their needs in the future. The goal of providing textbooks to students was more than met by MSIs; however, based on findings from the Evaluation of the African Educational Initiative (AEI), their activities – as 25 Evaluation of TLMP presented in their respective business plans – produced outcomes and leveraged development in many other sectors: • Built the educational development institutional capacity of MSIs to enable them to be successful in obtaining grants to conduct work in African countries (and elsewhere) • Built the capability of MOEs to produce textbooks and educational materials • Developed a cadre of educators to teach using learner-centered methodologies to democratize the classroom • Impacted the literacy rates in each country • Increased the ability of children to do well in national exams, and thus go on for further education • Developed the capacity of private sector printers and publishers, thus increasing the number of jobs created and making it possible for MOEs to outsource their printing needs • Fostered good will and built relationships with future generations of leaders • Produced a more educated, literate population that can more effectively maintain control over their country’s economy and resources, improve prospects for democracy, and reduce conflict. This work plan will outline a process that will collect data to support findings on each of these outcomes and detail the many “ripple” effects TLMP has leveraged in each country. 3. OBJECTIVES OF THE PERFORMANCE EVALUATION The Performance Evaluation is intended to satisfy the following objectives: • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based MSIs, in￾country institutions (MOEs, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Strategy_feb20 11.pdf 26 Evaluation of TLMP Summarized, IBTCI sees the overall purpose of this Performance Evaluation as providing USAID/Africa Bureau with information on: • What impact, if any, does TLMP seem to have had on the way that students learn and teachers teach in the participating countries? What are the conditions that have facilitated or constrained the impact? • Do lessons from the TLMP experience seem to have been institutionalized or incorporated in other education-sector activities? Because of the time that has elapsed since the South Africa TLMP activities ended, a somewhat different approach will be applied to evaluating the impact of TLMP there. 4. RESEARCH APPROACH, METHODOLOGY AND PROCESS Five tasks have been posited for this evaluation: 1) Data Collection; 2) Data Review; 3) Coordination and Management; 4) Site Visits; and 5) Data Analysis. We will utilize a mixed methods approach to evaluating the current TLMP cooperative agreements, rather than TLMP overall. To incorporate the five tasks, we envision following the research design presented below: The research design entails several steps: • Working with USAID missions to determine optimum times for country visits • Contacting and scheduling interviews with MSIs • Obtaining and reviewing project documents • Assigning members of the team to each country • Identifying target populations from whom to gather data • Developing specific interview schedules and other data collecting instruments • Refining instruments based on experience and the parameters of each project • Developing and editing preliminary country reports • Writing overall report Overall, the process begins with communicating with the education officers at participating USAID missions to identify mutually convenient dates in each country and communicating with the MSIs to identify mutually convenient dates for the Team Leader and/or Project Director to make site visits to them. Our intention is to visit the MSIs before beginning the country visits. This is based partly on the fact that the cooperative agreements run out the end of this calendar year and partly on the fact that feedback received from missions so far indicates that site visits during this calendar year would not be convenient for them. After collecting data from the MSIs on the development of ideas for TLMP, challenges and successes in project implementation, and project management, the teams will conduct work in the field, where data will be collected from relevant stakeholders; ten calendar days in country are allocated for each country visit. Our level of interaction with Mission education officers will 27 Evaluation of TLMP be dependent on their own preferences and workloads, although we hope that they will help to facilitate meetings with MOE officials. In any event, IBTCI will provide Mission education officers with a draft country workplan and, if desired, draft instruments for review prior to arrival. After each country visit, the team will prepare a report on the individual TLMP activity and its associated cooperative agreement. 4.1 Contacting and Scheduling Interviews with MSIs Prior to in-country data collection, IBTCI will contact each MSI to schedule interviews with Project Directors and related staff. The purpose of these interviews is to ascertain how each project was envisioned, how the project built upon institutional and faculty capacities, how the project was managed, how partnerships were created, and how participation in the project further built the capacities of MSIs to engage in other educational development either as a single institution or in a consortium. 4.2 Obtaining and Reviewing Project Documents IBTCI will obtain all relevant project documents from each MSI and other sources, to include but not be limited to: the Cooperative Agreement, especially Section B and the budget; all quarterly and annual reports; any research undertaken or special reports developed on the project; all data related to the printing/publishing and dissemination of TLM; any assessments of children’s educational performance after using the TLM; any changes in personnel and the reasons therefor; data on partnerships created; challenges to implementation; reports on workshops held; lessons learned; outcomes; and a review of the materials themselves to determine whether the materials are culturally relevant, gender neutral, and include such cross￾cutting themes as HIV/AIDS. The data obtained from MSIs will be shared with the team conducting the evaluation in the selected country and reviewed for specific points to be included in the research. 4.3 Assigning Members of the Team to Each Country The Research Team is composed of four individuals: Nancy Horn, Ph.D., Team Leader; Eric Allemano, Ph.D., Jim Wile, Ph.D., and Carol Benson, Ph.D. The Ethiopia and South Africa team will include three people due to the scale, complexity and outreach of these projects, while the other four countries will include two key researchers. Each researcher will be supported by a local expert who can assist in data collection and translation. Team assignments will likely be as follows: • Ethiopia and South Africa – Nancy, Eric and Jim • Ghana – Nancy and Jim • Senegal (French) – Eric and Carol • Malawi – Nancy and Eric • Tanzania – Jim and Carol Each team member will be responsible for reading all relevant documents, making any adaptations to the core data collection instruments, scheduling appointments with relevant interviewees, collecting the data, analyzing it and producing the draft country report. 28 Evaluation of TLMP 4.4 Identifying Target Populations from Whom to Gather Data Beginning with our data collection from MSIs, IBTCI will endeavor to obtain an up-to-date list of partners and stakeholders in the MOE, schools, districts, and the like to identify who will be interviewed in each location. The teams will be supported by in-country coordinators, who would be able to help the team in making appointments and in translating when needed. It is anticipated that school-based research with principals, teachers, parents and children will take place in at least one rural and one peri-urban district, in addition to the capital. The purpose of these interviews is to determine the difference the TLMP has made to school administration, teaching methodology, parental satisfaction, and children’s learning. A check sheet will also be utilized in observing classroom use of TLM. Time permitting, children in selected classes will participate in a PRA drawing exercise to demonstrate the impact TLM are having on their learning. Interviews will also be conducted with a range of MOE officials, both at HQ and in regions/districts, and with curriculum and grade level/subject matter specialists responsible for designing curriculum and developing materials to determine how their capacities have been built throughout project implementation. The printers/publishers contracted to produce the TLM will also be interviewed to determine how their capacities have been built and how many jobs have been created as a result of the project. 4.5 Developing Interview Schedules and Other Data Collecting Instruments The research will use a mixed methods approach: qualitative (key informant interviews, focus group interviews, a PRA drawing exercise with the children, classroom observations, and Likert Scale ranking questions) and quantitative (teacher observation charts, analysis of learning assessments undertaken, etc.). See Annex A – Data Collection Instruments – for a draft of the core questions to be posed of each type of stakeholder. 4.6 Scheduling In-country Research Activities IBTCI will finalize the research schedule as USAID travel approvals are given. Due to holidays and school term schedules, a very tentative field research schedule is as follows: • Senegal and Ghana – January 9-22, 2013 • Ethiopia – January 28-February 8, 2013 • Malawi and Tanzania – February 18-March 1, 2013 • South Africa – March 6-20, 2013 IBTCI plans for document reviews to begin in October 2012 and continue throughout the period of research. After field data has been collected, between three and five days have been allotted to draft preliminary country reports before going on to the next country. To the extent feasible, country visits will be scheduled so that more than one country can be included in a single international itinerary. This schedule has been developed so as to limit the expenditure involved in international air fares from the US and other locations to the research sites. Subject to revision based on individual country circumstances, we anticipate that a typical in￾country field visit will have a schedule like the following: Predeparture Communications with USAID Education Officers 29 Evaluation of TLMP Day 0: Day of arrival. Meeting with local researchers, if feasible. Day 1: Meeting with USAID staff and Local TLMP Coordinators Day 2: Meeting with TLMP implementers and partners in capital Day 3: Meeting with MOE officials in the capital Day 4: Team split, with each sub-team (international plus local researcher) travelling to different locations for meetings with Provincial, district, and/or local education authorities Day 5: School visits and classroom observations: interviews with principal/vice principal, head teachers; focus group with teachers trained in TLMP; PRA exercise with children, and parent interviews Day 6: Continuation of Day 5 Day 7: Rest Day 8: Same as Day 5 but in a school district in the capital city Day 9: Follow-up meetings with TLMP Coordinators Day 10: Debriefing of USAID mission staff and other stakeholders as desired. Departure. 4.7 Review of Cost-Effectiveness Absent the existence of comparative data on reading results from students/teachers who have received (or not received) different degrees and types of interventions, “cost-effectiveness” as such can only be estimated impressionistically. To the best of its ability, based on available documentation, for each country the IBTCI team will determine the approximate direct costs per student of preparing and producing different types of TLMP materials and the costs of producing conventional materials. Based on interviews with teachers, the team will identify the types of materials that have proven most useful. 4.8 Developing and Editing Preliminary Country Reports A standardized format for writing country reports will be developed so as to ensure that comparable data is collected in each country and at each MSI within the USAID report guidelines. In turn, this format will be utilized to develop the overall report, to which the country reports will be appended. While preliminary country reports will be developed at the end of data collection in each country, they will be edited and finalized by the Team Leader upon completion of the field research and submitted to IBTCI for final editing. Although not a formal deliverable, IBTCI will submit each country report to USAID thereafter. 4.9 Draft and Final Reports The Team Leader will develop a draft report and submit it to IBTCI for review prior to submission to USAID for comment. The draft report will be revised based on USAID feedback. Once revised and finalized by the Team Leader, IBTCI will submit the final report to USAID. 30 Evaluation of TLMP 5. MANAGEMENT AND LOGISTICS In accordance with the Task Order, IBTCI will submit the following deliverables: 1) Instrument design and approval (appended to this work plan), due by November 2, 2012 2) Draft Final Report, due 20 weeks after award; however, because of Mission schedules that preclude field activities during calendar year 2012, an extension will be needed 3) Final Report, due two weeks after delivery of the Draft report 4) Monthly status reports, due by the last business day of every month. Quality control will be provided by Mr. Robert Van Heest, IBTCI Vice-President, who will review and comment on drafts prior to submission. He will also review the monthly reports submitted by the Team Leader to IBTCI at the end of each month before forwarding them to USAID. The Project Director, Ed Allan, Ph.D., will have direct oversight over the implementation of the evaluation, making sure that local support researchers are hired and that a local logistics person is hired and makes all local travel arrangements and assists in making appointments. 6. DRAFT TIMELINE Deleted as not being relevant given the substantial changes in actual implementation. 31 Evaluation of TLMP ANNEX C. ETHIOPIA RESEARCH QUESTIONS These questions are representative of the questions generally asked during each of the country visits, although there were variations based on individual country circumstances. DRAFT QUESTIONS TO BE POSED OF PARTNERS AND STAKEHOLDERS IN INTERVIEWS HELD IN ETHIOPIA In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the expertise developed in textbook production by the MOE being leveraged to obtain other, similar grants? What would prevent this from happening? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbook and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in Ethiopia? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • How satisfied were you with your relationships with Alabama A&M and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOE Administrators (triangulate with different administrators at each level) • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLM? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOE provide? How frequently? • How do the woreda inspectors assess teachers? Were they trained in the use of TLM? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLM included in teacher training (either pre-service or INSET)? • How was the decision made to include supplementary/complementary readers in the TLMP? Who made the decision that these materials should be based in folk tales? How did the process of developing these materials differ from that used in developing the TLM for grades 1-4 English 32 Evaluation of TLMP classes (textbooks)? Was there an advantage of one approach over the other? Please explain. • What will the MOE do to continue the production of TLM now that the project has ended? Have any other donors been found to continue this activity? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLM in the future? In the redevelopment and printing of the grades 6, 7, and 8 materials? What would prevent this from happening? • How has the material presented in the TLM been included in national exams? Has performance on exams improved with the use of TLM? How do you know? • How was TLMP monitored by the MOE? What indicators did you use? How often did you go to schools to observe the use of TLM? How was TLM production managed and monitored? • How has the MOE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with AAMU? How could they be improved? 3) Material Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How did the production process and personnel differ between the development of the textbooks and the development of the supplementary reading materials (folk tales)? Did one process have an advantage over the other? If so which one, and how were results different? Would you recommend one process over the other for future book production? • How often did the two US- and Ethiopia-based teams meet? What were the results of these meetings? • In developing TLM, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? What cross-cutting themes did you include? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLM? • How satisfied are you with the collaborative production process between yourselves and AAMU? What worked well/did not work well? What would you change to improve the process? • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other 33 Evaluation of TLMP stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLM? (describe both the textbooks and the supplementary reading materials) What did the MOE do (specify unit)? What did the project do? What challenges emerged in your work with the MOE? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOE play (specify unit)? What roles did your office play? What guidelines did the MOE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Who was responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who provided the training? For how long? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Gender distribution of those who attended? Did TTC faculty attend? From which TTCs? Did university faculty attend? How many? From which universities? • What other in-service teacher training is provided by the MOE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLM? Project Management and Outcomes • What was the TLMP management structure in Ethiopia? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Ethiopia PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of project work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with AAMU? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What instructions were you given on how to distribute the TLM? From whom? What kind of 34 Evaluation of TLMP difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLM after they were produced? To whom did you distribute them? How many TLM were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? • How satisfied were you with the relations established with AAMU? How could they be improved? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLM were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? • How were inspectors instructed on how to evaluate teachers using TLM? Do all children have/use the books produced? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLM produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparity in enrollment/attendance? If so, what have the results been? • What is the linguistic background of the learners at this school? What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which 35 Evaluation of TLMP grade? At what grade does English become the language of instruction? Do you believe your students are adequately prepared in English to learn entirely in English? What needs to be done to prepare students better? • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLM? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLM? Where was the work undertaken? For how long? • How many of each textbook and workbook did you request for the school? How many of each text/work books did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLM? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOE provide? How often? Are those who attend expected to pass on (cascade) what they have learned to their colleagues? Do they receive any incentive to do this? • How has the cluster center training and resource system enhanced the ability of teachers to be more learner-centered? How has the TLMP enhanced the adoption of learner-centered teaching practices? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? 8) Classroom and Head Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher? (certificate, diploma, degree) • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching English? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLM? Please explain. • How many students do you have in your classes? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP textbooks do you have to teach? What do you do when you don’t have enough TLM for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive the TLM for your classes? How many were you provided? From whom 36 Evaluation of TLMP did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? • When did you receive training on the use of TLM? How long did it last? Did someone from the TLMP project or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • Do you believe the TLM were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLM were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLM? • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLM? What evidence can you give for this? • What is your opinion of the TLM in so far as their attractiveness to students? On a scale of 1-4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLM in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLM that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLM? • What do you think is the overall impact of the program on your students? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the supplementary readers? 3) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? 37 Evaluation of TLMP • Why do you think you were chosen to be a trainer? • On a 1-4 scale, with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? • How familiar were you with the TLM before you delivered the training? What materials were you provided to be a trainer? What materials did you provide to your trainees? How confident were you after the TOT that you could teach others in how to use the TLM? What else did you need? • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any compensation for conducting this training? 38 Evaluation of TLMP ANNEX D. SCHOOL-BASED RESEARCH ACTIVITIES 1. Senegal Jan 26-Feb 9 Eric Allemano, Carol Benson 2. Tanzania Feb 17-March 1 Jim Wile, Carol Benson 3. Malawi Feb 17-March 2 Nancy Horn (Team Leader), Tom Tilson 4. Ethiopia March 3-19 Nancy Horn, Eric Allemano, Jim Wile 5. South Africa May 5-16 Ed Allan (Project Director), Carol Benson 6. Ghana June 9-20 Tom Tilson, Eric Allemano Country TTCs & Locations Schools and Locations # Princi￾pals Inter￾viewed # Teachers Inter￾viewed # Classes Observed Student Reading Assessment Ethiopia Addis Ababa: Kotabe TTC Amhara: Debre Berhan TTC Addis Ababa: Sefre Salam, Wondrad Oromiya: Mulugeta Gedlu, Buruyu, Dukem 1 9 22 55 Grade 1 – 20 Grade 2 – 20 Total 40 Ghana Cape Coast: University of Cape Coast Winneba: University of Education, Winneba Accra: Osu Presbterian, Osu Annex; North Legon Cape Coast: Abakam. Golden Treasure, Philip Quaque Effutu: ACM, Ebenezer, Effutu Ga East: Presbyterian Basic, Living Star Ho: Kpenoe, Mawuli Jasikan: Jasikan Demo Koforidua: Denguano, St. Mary’s 10 14+ 3 (staged) 24 Malawi Lilongwe TTC Blantyre TTC Lilongwe: Lilongwe Demonstration School Zomba: Mponda Primary; Police Primary; Domasi Demonstration School; Domasi Government; St. Anthony’s Boys; St. Anthony’s Girls Blantyre: Malavi Primary, St. Theresa Primary, Montford College; PIM Primary; Mafe Primary; South 40 90 19 S1 – 11G, 11B S2 – 11G, 11B S3 – 11G, 11B Total 66 39 Evaluation of TLMP Lunzu Primary; Namilungu Primary Senegal Saint-Louis Louga Thies Kaffrine Kaolack Fattick N/A South Africa Pretoria: University of Pretoria Limpopo: University of Limpopo Pretoria: Pfundzo nde Tshedza Primary Limpopo: Siseluselu Primary, Onane Primary 4 N/A Tanzania (secondary schools) n/a Dar es Salaam: Tuangoma, Mwanalugali Coast: Kibaha Girls, Kibaha Town Arusha: Poli, Kimaseki, Ilboro, Kipok Girls, Makuyuni Boys, Lowassa Morogoro: Kilakala Girls, Kingolwira, Sekwao, Kirokala, Mkuyuni 22 54 19 33 TOTAL 81 180 55 40 Evaluation of TLMP ANNEX E. TLM PRODUCED Country Student TLM Teacher’s Guides Total Production Ghana • Literacy KG1workbooks – 1,031,000 • Literacy KG2 workbooks – 785,040 • Numeracy KG1 workbooks – 1,031,000 • Numeracy KG2 workbooks – 784,333 • Environmental Science KG1 workbooks – 1,031,000 • Environmental Science KG2 workbooks – 784,333 Total: 5,446,706 • Literacy KG1 teacher’s guides – 29,151 • Literacy KG2 teacher’s guides – 24,401 • Numeracy KG1 teacher’s guides -29,148 • Numeracy KG2 teacher’s guides – 24,401 • Environmental Science KG1 teacher’s guides – 29,151 • Environmental Science KG2 teacher’s guides - 24,401 • Laminated wall charts – 220,000 Total: 380,653,653 5,827,359 Ethiopia Grade 1 – 4,300,000 Grade 2 – 1,700,284+ Grades 2, 3 & 4 – 5,500,000 Folktales – 700,000 Total – 12,200,284 Grade 2 – 50,000 Grades 3 & 4 – 45,000 Folktales – 50,000 Total – 145,000 12,345,284 Malawi 183 Titles (120 in Chichewa and 60 in English; two alphabet books (120 in Chichewa and 80 in English), six overview guides in English, 20 Chichewa and 10 English “big books,” alphabet posters (4 in Chichewa and 4 in English), posters on how to care for books (4 in English), and water resource posters (4 in English) x 1,272 educational centers. 120 in Chichewa and 90 in English x 1,272 educational centers 5,469,088 Senegal Elementary: 1,000,000 Middle: 1,400,000 Secondary: 250,000 Reproduction of elementary French textbooks: 400,000 N/A 3,050,000 South Africa Combination of 140 storybook titles for grades 4-6, teacher’s guides for each, and overview guides in 11 languages N/A 2,300,000 Tanzania Math 1-4: 560,000 Biology 1-4: 300,000 Chemistry 1-4: 400,000 Physics 1-4: 400,000 Total: 1,660,000 Math TG 1-4: 28,000 Biology TG 1-4: 13,700 Chemistry TG 1-4: Physics TG 1-4:18,700 Total: 60,400 1,720,400 TOTAL 30,712,111 41 Evaluation of TLMP 42 Evaluation of TLMP ANNEX F. REPRESENTATIVE TLM ASSESSMENT This form was used to review the pedagogical and physical components of student and teacher materials for Ethiopia. 1. Student Books A. Pedagogical Content of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Alignment with Syllabus All textbooks are aligned with the syllabus, and meet the Minimum Learning Competencies (MLCs) for each grade level Organization of Content Textbooks for grades 1, 3, and 4 appear to move from simple to complex, but the grade 2 text does not, with vocabulary and sentence structure too long and difficult. Correctness of Content/ Conformity with Ethiopian Culture “Lions live in the forest and prey on small animals” is erroneous, indicative of other passages. Content generally reflected urban/ westernized culture and did not depict all religious and ethnic groups. In an attempt to be gender sensitive, some tasks failed to recognize that they are a part of religious culture. Integration with other Subjects & Cross-cutting Issues Only minimal attention was paid to linking content to other cross￾cutting issues that would likely occur in other subject areas such as math and environmental science. Quality of Illustrations The textbooks make a good use of illustrations to guide vocabulary; the picture quality of the grade 2 text was often blurry because contrasting colors were not used. Relevance & Appropriateness for Learners and Teachers The topics were relevant and appropriate for children of typical age grades. However, the pedagogical content was not appropriate as a basic language development program (presentation seemed unstructured and unsystematic). Opportunities for Assessing Student Progress The texts provide ample opportunity for continuous performance assessment, but the format lacks opportunities for unit review and assessment. Language and Communication “While many stories and activities are innovative and engaging, they are too difficult for the English abilities of most learners and even many teachers at the third grade level.” There seems to be a mismatch between expectations of teachers’ capacity to read and model the level of English required at each grade level and unrealistic expectations about pupils’ cumulative language development. This mismatch may result in pedagogies that require the teacher to translate English content and directions into native languages, an overemphasis on the mechanical components of language (letter identification) and a learning strategy that emphasizes memorization of vocabulary and sentence frames. Enjoyment Index Textbooks are colorful and full of color illustrations; however, the grade 1 textbooks looks like a penmanship workbook, and the content of the other grades neglected opportunities to introduce jokes, riddles, songs, etc., that might stimulate learner interests. 43 Evaluation of TLMP B. Physical Design of TLMP Textbooks, Grades 1-4 Assessment Characteristic Team Assessment Cover and Paper Stock Durable and appropriate stock, but may be more durable if the textbooks were produced in two volumes rather than one to preserve the longevity of the books. Durability would be enhanced if children were provided book covers. Durability is projected as 2 years, but may be as much as 4. Bindings Durable and appropriate; should have a shelf life of 3-4 years, provided they are handled appropriately. Size and Dimension The grade 1 textbook is more of a workbook that provides children the opportunity to write letters and sound them out when a phonics approach is used. Other textbooks should be divided in two so that children and teachers can use each part each term. Quality of Print Good quality and legible; some illustrations lack sharp contrast. Typeface The sans serif font is good for grade 1 as it is clean and simple. However this typeface does not match frequently used fonts in texts (a, g, t). Structures Supportive materials to enhance children’s learning might have included a pupil picture dictionary, high frequency language phrases, and some grammar models/sentence frames. 2. Teacher Guides A. Pedagogical Quality of Teacher Guides Assessment Characteristic Team Assessment Objectives MLCs and unit objectives are presented at the opening of the unit, but are absent for each lesson and do not identify learning outcomes. The number of lessons to be covered per week is beyond the overall number of days a child is to be in the classroom over the school year. Objectives are not order, in sequence, and do not necessarily build on what has already been learned. Organization & Presentation The Guides contains lessons that are not in the student text, resulting in an odd labeling system that is clumsy and confusing (e.g., units in the student’s text may open with lesson three). Teaching & Learning Strategies Teaching activities are designed to be interactive rather than didactic. The format uses a scripted text for teachers; however, teacher English competency is not sufficient for them to read and understand the scripts. Moreover, the scripted format will not accommodate irregularities in the pupils’ texts nor will it help teachers address children’s questions as they arise. Teaching Ideas The guides introduce or model a number of high interest techniques to promote active learning and language skill development. Referencing The guides provide extra materials to reinforce or enrich lessons, such as crossword puzzles or suggestions for making flashcards. It also lists the answers to the activities in the student book. However, the guides do not orient the teacher to resources in other books, publications or the Internet. The teacher will need a 44 Evaluation of TLMP dictionary to teach the meaning of the vocabulary. B. Physical & Design Quality of Teacher Guides Assessment Characteristic Team Assessment Cover & Paper The guides are not attractive and have no picture on the front cover. The paper quality is appropriate, durable and appears water￾resistant. Binding Adequate Size & Dimension Foolscap paper was used for the guides; the number of pages is daunting, with one volume 274 pages. Page length is long due to the scripted nature of the lessons. Quality of Print Good, with a good mix of boldface and textboxes; however, there is simply too much print on each page. More white space is needed so that teachers can “bracket” some portions or write ideas/reminders in the text. Structural Elements There is a vocabulary list, but no definitions. Language patterns (sentence frames) do not provide the name of the pattern, e.g., irregular plurals, verb tenses, adjectives, etc., only an example is given. The front part of the guides do not provide useful background information for the teacher on issues such as language development, lesson planning, and assessment. This type of content assumes a language proficiency that is beyond the competency levels of most teachers. 45 Evaluation of TLMP ANNEX G. REPRESENTATIVE READING ASSESSMENT (ETHIOPIA, N=40) Methodology: • Speaking was assessed by asking the child to respond to a series of questions about himself/herself in English (e.g., What is your name? Are you a boy or a girl? How old are you? What grade are you in?). • Letter identification was conducted by asking children to name 10 letters in English (both upper and lower case). • Word recognition was conducted by children reading up to 10 words that had been drawn from the grade appropriate textbook. • Listening comprehension included a passage that one member of the team read aloud from a grade textbook, and then “wh” questions were posed to determine if the student understood the passage. • Oral reading fluency (word accuracy and phrasing) was assessed using a passage drawn from a grade appropriate textbook. • Writing competence was measured by asking the child to write three letters, write his/her name and/or one word of her/his choice. Results obtained in Ethiopia include the following: Characteristic First Graders Second Graders Comment Letter Identification 75% identified 70% 90% identified at least 70% Listening & Speaking 100% 100% One-word responses Word Recognition 20% recognized 70% 35% identified at least 70% Comprehension Not able 25% answered 60% of questions Reading Not able 50% read w/high accuracy 15% able to read One-word responses Writing 40% able to write 3 letters 100% able to write names 35% able to write a word in English 80% able to write 3 letters 100% able to write names 35% able to write a word in English 46 Evaluation of TLMP ANNEX H. SELECT BIBLIOGRAPHY OF RECENT MATERIALS Following is a bibliography of selected recent materials relevant to the production of textbooks and learning materials, not necessarily related to TLMP, and their use for literacy instruction that are available on the Web. (Few TLMP materials are available on the Web.) This builds on the very extensive body of references included in the Education Strategy Reference Materials. USAID Materials USAID Education Strategy: 2011­2015 http://pdf.usaid.gov/pdf_docs/PDACQ946.pdf -- Implementation Guidance (rev. April 2012) http://pdf.usaid.gov/pdf_docs/PDACT461.pdf -- Technical Notes (rev. April 2012) http://pdf.usaid.gov/pdf_docs/PDACT681.pdf For planning and implementation, please note the guidance on Collecting Baseline, Midline, and Endline Data on page 11. -- Reference Materials (rev. April 2012) http://pdf.usaid.gov/pdf_docs/PDACT680.pdf USAID Education Summit, August 2013 http://www.usaided2013.net/summit-agenda/ The first day of the Education Summit targeted Goal 1, the second day Goal 3, and the third day Goal 2. The website includes most of the PowerPoint presentations made, a number of which relate to TLM. JBS International, Checklist for Conflict Sensitivity in Education Programs (July 2013) (in draft form) It is our understanding that in the future all education programming, not just that planned for conflict settings, should pay attention to any potential that content of curricula, materials, etc. may contain language that could exacerbate the potential of conflict. JBS International, An Outcomes and Impact Evaluation of the President’s African Education Initiative. (2009) EdQual, Language of Instruction & Quality of Learning in Tanzania and Ghana, EdQual Policy Brief No. 2 – September 2010 http://www.edqual.org/publications/policy-briefs/pb2.pdf Provides discussion of comparative approaches to the teaching of first languages in Sub-Saharan Africa. World Bank Materials Helen Abadzi, Literacy for All in 100 Days? A research-based strategy for fast progress in low-income countries, Global Partnership for Education, draft of May 30, 2013. http://documents.worldbank.org/curated/en/2013/05/18042078/literacy-all-100-days-research-based￾strategy-fast-progress-low-income-countries A very useful and interesting paper on approaches that help classroom learners acquire literacy skills generally in an extremely short period of time. Makes numerous references to USAID-assisted education sector projects. Kirsten Majgaard and Alain Mingat. 2012. Education in Sub-Saharan Africa: A Comparative Analysis. Washington, D.C.: World Bank. https://openknowledge.worldbank.org/bitstream/handle/10986/13143/9780821388891.pdf?sequence= 1 47 Evaluation of TLMP This study provides very useful analyses of numerous aspects of education (including textbooks) in the region. It also includes discussion of relevant data from other regions. Other Materials Katharina Michaelowa and Annika Wechtler. 2006. “The Cost-Effectiveness of Inputs in Primary Education: Insights from the Literature and Recent Student Surveys for Sub-Saharan Africa.” Paper presented at the ADEA (Association for the Development of Education in Africa) Biennial Meeting, Libreville, Gabon, March 27–31, 2006. http://www.adeanet.org/adeaPortal/adea/biennial￾2006/doc/document/B1_2_michaelova_en.pdf Brookings Institute Center for Universal Education, 2013. Toward Universal Learning: What Every Child Should Learn. http://www.brookings.edu/research/reports/2013/02/learning-metrics -- Toward Universal Learning: A Global Framework for Measuring Learning. http://www.brookings.edu/research/reports/2013/07/global-framework-measuring-learning These discuss issues of realistically achievable curricula and measurements; they relate to the fact that so many materials were prepared for levels noticeably higher than those actually typical of the students for whom they were intended (“leveling”). 48 Evaluation of TLMP ANNEX I. CONFLICT OF INTEREST STATEMENTS 49 I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date August 03, 2013 Name THOMAS TILSON Title CONSULTANT Organization IBTCI Evaluation Position Team Leader Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date June 19, 2013 Name MEREKU, DAMIAN KOFI Title PROF. Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 19. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 20. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 21. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 22. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 23. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 24. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. √ √ I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 2/13/13 Name Denis Khasu Title Dr Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 2 August, 2013 Name Hannock Yamikani Mateche Title Evaluation Specialist Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number(contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated(Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Univ. of Texas San Antonio) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. x x I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 01, 2013 Name Dr. Richard W. Chediel Title Team Member Organization IBTCI Evaluation Position Education Evaluation Specialist Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523