October 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Edward Jay Allan (Project Director), Carol Benson, and Rakgadi Phatlane under Task Order AID-OAA-TO-12-00054 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania TLMP in South Africa Intentionally left blank EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA TLMP IN SOUTH AFRICA Prepared by: Edward Jay Allan, Project Director Carol Benson, Team Member Rakgadi Phatlane, Team Member October 6, 2013 The authors’ views expressed in this trip report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP – TLMP in South Africa i TABLE OF CONTENTS ACKNOWLEDGEMENTS .................................................................................................................ii Map of Limpopo Province ..................................................................................................................ii LIST OF ACRONYMS AND TERMS ...............................................................................................iii EXECUTIVE SUMMARY ................................................................................................................... 1 KEY RECOMMENDATIONS .......................................................................................................................................... 3 1. EVALUATION PURPOSE AND OBJECTIVES ........................................................................... 5 1.1 Background to TLMP ............................................................................................................................................... 5 1.2 Evaluation Objectives ............................................................................................................................................... 5 2. PROGRAM BACKGROUND ..................................................................................................... 6 2.1 TLMP Background in South Africa........................................................................................................................ 6 2.2 DBE Background and Priorities ............................................................................................................................. 8 3. EVALUATION METHODOLOGY AND LIMITATIONS .......................................................... 8 3.1 Overall Methodology .................................................................................................................................................. 8 3.2 Research Conducted at UTSA .............................................................................................................................. 9 3.2 Research Activities in South Africa ...................................................................................................................... 9 3.3 Limitations of the Study ........................................................................................................................................ 10 4. FINDINGS AND CONCLUSIONS .........................................................................................10 4.1 Ithuba Achievements and Challenges ................................................................................................................. 10 4.2 TLMP Output ........................................................................................................................................................... 14 4.3 TLMP Impact ............................................................................................................................................................ 14 5. LESSONS LEARNED ...............................................................................................................15 6. RECOMMENDATIONS ...........................................................................................................16 ANNEX A: SCOPE OF WORK ......................................................................................................19 ANNEX B: SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES ...................22 ANNEX C: CONFLICT OF INTEREST STATEMENTS ..............................................................24 Performance Evaluation of TLMP – South Africa ii ACKNOWLEDGEMENTS The team members would like to express our appreciation for the willingness of Department of Education officials, the NGOs Molteno and READ, USAID/South Africa, and UTSA project implementers to provide us with documents and information as well as ensure that we could access targeted schools, teachers, and students in the time we had available for the visits. The international members also note the extremely important contributions made by Dr. Thabile Mbatha and especially Dr. Rakgadi Phatlane, whose experience and contacts greatly facilitated our access to key informants and information. MAP OF LIMPOPO PROVINCE Evaluation of TLMP – TLMP in South Africa iii LIST OF ACRONYMS AND TERMS AEI President’s African Education Initiative APS Annual Program Statement CA Cooperative Agreement CAPS Curriculum and Assessment Policy Statement DBE Department of Basic Education (national) DOE Department of Education (provincial) EFA Education for All EFL English as a Foreign Language EGRA Early Grade Reading Assessment ESL English as a Second Language IBTCI International Business & Technical Consultants, Inc. Ithuba Ithuba [“Opportunity”] Writing Project KZN KwaZulu-Natal LEA Language Experience Approach LIEP Language-in-Education Policy LOLT Language of Learning and Teaching MOE Ministry of Education Molteno Molteno Institute for Language and Literacy MSI Minority Serving Institution NCS National Curriculum Statement NGO Non-governmental organization NQATF National Quality Assurance Task Force READ READ Educational Trust RFP Request for Proposal SSA Sub-Saharan Africa TLM Textbooks and Learning Materials TLMP Textbook and Learning Materials Program TOT Training of Trainers UL University of Limpopo (now the University of the North) UP University of Pretoria USAID/SA United States Agency for International Development/South Africa USAID/W United States Agency for International Development/Washington UTA University of Texas Austin UTSA University of Texas San Antonio Performance Evaluation of TLMP – South Africa 1 EXECUTIVE SUMMARY This evaluation differs from those conducted in the five other countries in which TLMP was implemented in that funding for Ithuba (the name given to the project in South Africa) was provided to complete work begun under the President’s African Education Initiative (AEI) and not to initiate continued activities; USAID-assisted activities ended by 2009. Unlike the other five projects funded during 2009-2012, which were being wrapped up in the course of the evaluation, the evaluation of Ithuba aimed to determine any lasting impacts that the project might have had on the South African educational system, administrators, teachers and students. The objectives of the study were as follows: OBJECTIVES  Determine if the planned outcome of facilitating the development of a “reading culture” in South Africa occurred.  Assess the results achieved and determine whether the South African TLMP had any spin-off effects on policies or practices of the Department of Basic Education (DBE), regional education offices, schools and implementing partners.  Seek evidence of how the practices and effects of TLMP may have continued since the end of TLMP in 2008.  Identity any long-term effects of teacher/educator participation in the development of supplemental readers.  Document challenges and lessons learned.  Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy http://www.usaid.gov/ our_work/education_and_universities/documents/USAID_ED_Strategy_feb2011.pdf PROGRAM BACKGROUND The focus of the Ithuba Writing Project (Ithuba) was the development and production of storybooks in South Africa’s nine official African languages, plus Afrikaans and English, by training teachers to become authors and some to become illustrators. The lead South African partner was READ Educational Trust (READ), joined by the Molteno Institute of Language and Literacy (Molteno). The writing workshops were designed and implemented by TLMP leadership at UTSA and its partner, the University of Texas Austin (UTA). The workshops were first delivered to trainers (using a training-of-trainers (TOT) model) and then rolled out with teachers from target areas. Assisting UTSA in delivering further workshops were five trainers from Molteno and 10 trainers from READ, all of whom had been trained in materials development processes. The goal was to create a model for materials development so that teachers would have the capacity to use storybooks to promote literacy learning along with the capacity to create future iterations of stories in appropriate languages to suit their needs. Teachers’ guides in the form of 4-page leaflets linked the stories to the national curriculum, in hopes that teachers would make use of the stories to teach literacy across the curriculum. The storybooks were printed at Uniprint in Durban and distributed to target schools through the provincial Departments of Education (DOEs). Two writing centers were created, but the team could find no evidence of their continued existence at the time of the evaluation. METHODOLOGY AND LIMITATIONS A “mixed methods” approach was used, comprised of:  Review of program documents and a sampling of TLMs produced Evaluation of TLMP – TLMP in South Africa 2  Key informant interviews with UTSA staff, in-country partners, DOE personnel at various levels, staff of other stakeholders, and school principals and teachers  Site visits, including classroom observations and, as feasible, getting perspectives from learners During the in-country phase (May 5-16, 2013, inclusive), a team of two Americans and one South African collected data from a range of stakeholders in Limpopo and Gauteng provinces and READ’s representative from Mpumalanga, including: officials of DBE and the provincial DOEs; representatives of READ, Molteno and Uniprint; representatives of the University of Limpopo (UL) and the University of Pretoria (UP); and school principals, teachers of grades 4, 5, and 6 (the “Intersen” years), and students in three schools – a township school in Pretoria and two rural schools in Limpopo. Data collection was challenging due to the team’s limited time in country, a teachers’ strike and go-slows called by the unions, the fact that the project had ended in 2008/2009 and the reorganization of education offices since that time. We saw individual storybooks and teachers’ guides in the appropriate languages in the schools. ACHIEVEMENTS OF ITHUBA There have been a number of achievements of Ithuba/TLMP in South Africa. The major achievements that could be verified are the following:  Storybook development: Ithuba contributed significantly to the number of high-quality supplementary reading storybooks available in schools in the nine official African languages.  Use of TLMs: At least some storybooks are being used in some classes in Gauteng, Limpopo, and Mpumalanga. We found Ithuba materials displayed on Ithuba stands, in rolling cupboards, on library shelves and in classrooms. Foundation phase (grade 1-3) teachers reported using the storybooks, along with teachers of grades 4-6, for whom they were intended.  Contribution to improved reading ability: Virtually all of the students we observed in grades 4 through 6 who were asked to read from the storybooks, whether chorally or individually, were able to do so with accuracy and fluency in their home languages. While there is no way to tie this directly to Ithuba, their presence and use would suggest that they have contributed significantly to children’s reading skills. A representative of Molteno reported that she saw students reading technically difficult graphemes (like consonant trigraphs in Sepedi) in the Ithuba books with comprehension and fluency, and the team saw that students were able to re-tell stories orally or in writing, as called for in the Ithuba teachers’ guides.  Contribution to teacher competence in developing literacy skills: Enlisting and training teachers to become authors was innovative and seems to have left lasting impressions. The Ithuba author workshops reportedly raised the capacity of teachers and trainers to be authors while motivating them to write actively, and it is hoped that they will also motivate their students to write.  Extension of Ithuba methods for storybook development: In one case, a library specialist at the Limpopo DOE convinced colleagues to help her develop storybooks in Sepedi (with a plan to extend the program to the other official African languages) for the Foundation phase. They gained the support of Irish Aid to develop mock-ups and field-test them in the schools. In another case, members of the English Department at UL were inspired by Ithuba so they sought funding from Belgium to create the “Multiple Literacies Project,” which produces high quality storybooks in home languages using the Ithuba model.  Ithuba storybooks are still appropriate under the new curriculum: The new Curriculum and Assessment Policy Statement (CAPS) emphasizes early literacy in the mother tongue and literacy across the curriculum, and the team was informed by South African educators that the Ithuba materials are congruent with the new curriculum standards. Evaluation of TLMP – TLMP in South Africa 3 CHALLENGES TO ITHUBA There were also some challenges to TLMP in South Africa.  Sustainability issues within the DBE and USAID/SA: In principle, the only obstacle to the basic sustainability of the Ithuba materials is funding for the reproduction of existing materials plus limited funding for revisions/updating and, if desired (and recommended), funding for adaptation of existing materials into other languages and for training. There is an ongoing need – both real and perceived – for supplementary reading materials in local languages. The Ithuba materials appear to be high quality and are consistent with current curriculum guidelines. The materials have great acceptance by the teachers and students and provincial DOE personnel whom the team met, and other agencies have been willing to support expansion, to at least a limited extent. However, we understand from Mission staff that USAID intends to work on its new education initiative from scratch and national DBE personnel advise that they are not in a position to take the Ithuba materials up at this time.  Pedagogical issues: Most of the classroom observation demonstrations we saw were heavy on the “didactic” side and less on the “enjoyment” side. Teachers generally did have the idea of letting students read aloud independently as well as together, although there were occasional vestiges of traditional methods like reading first and asking students to repeat, and students at the Onane Primary School were encouraged to develop ad hoc skits based on the reading.  Consistent use of the TLMs: Adding to the challenge of methodology, it was difficult to determine whether or not teachers used the books regularly, because most of the sessions we saw were “staged” for us. We were told at each school that teachers used the books, but except at the Onane Primary School there was no instance where we saw use of the materials in the course of a class conducted in its regular time period.  Some logistical difficulties: Because the readers were supplemental and not meant to be used as texts, only 20 copies of each reader was shipped to a school, enough for sharing if a class had the 40 students called for by DBE policy but not enough if classes were larger. The 4-page teacher’s guides, which were developed as separate sheets (in strong cardboard) for each storybook, were not found at all schools. There were some reports of mismatches between the languages of the stories and the home languages of the learners. It was difficult for teacher-authors to write consistently at grade 4-6 levels. LESSONS LEARNED Many constructive lessons were learned by the implementers, including the following:  Creating good relationships with education officials: According to UTSA implementers, many of the successes of the project emanated from the good relationships established between UTSA and the DBE.  Teachers as writers and editors: From UTSA reports, it was noted that once the South African team members developed skills in estimating reading levels and judging appropriateness of stories, they could have been relied on to do all of the editing, rather than sending materials to UTSA.  More evaluation and follow-up: Local implementers commented that there was inadequate M&E or follow-up.  Make TLMs available to parents: In the interest of creating a culture of reading, the storybooks should have been available to families, possibly through loaning schemes, promoting intergenerational activities around print. KEY RECOMMENDATIONS 1. USAID/SA should engage in dialogue for continued use of Ithuba materials: At the very least, the mission should initiate a dialogue with DBE to determine interest. The major pedagogical reasons are discussed above – the massive need for supplementary reading materials (into the Evaluation of TLMP – TLMP in South Africa 4 production of which the American taxpayer has already invested millions of dollars), their compliance with current curriculum guidelines, their high acceptability. In addition, the materials are branded as being supported by the American people and will be seen in most South African households with a child in elementary school and can be used also to promote adult literacy and to develop a culture of reading. 2. Consider using the materials development model of TLMP (Ithuba) to develop new storybooks for early grade literacy: Involving teachers and other educators in the story writing process was designed to create long-term impacts in teachers’ approaches to reading and writing. Literacy (in home languages) has been identified as a priority by DBE in South Africa as well as by USAID. 3. Selection of future implementers: For a project involving TLM development, implementers should have: knowledge of the context; understanding of how TLMs satisfy curriculum requirements; experience integrating literacy promotion strategies into existing teacher training mechanisms; understanding of the sociolinguistic context and the language capabilities of teachers; knowledge of the printing and dissemination process utilized by public and private agencies; and ability to consistently monitor and evaluate (along with partners) project activities. 4. Consider the use of Annual Program Statements (APS) to complement ongoing education activities. It may not be workable for an education activity to incorporate the development of learning materials as part of its work; e.g., its primary focus might be on teacher training or educational administration. However, this does not obviate both the need and desirability for learning materials. An APS would give local entities, possibly in collaboration with U.S. partners, the opportunity to build its own capacity and the capacity of local educators to develop, produce, distribute, and use materials on, e.g., a provincial level with the possibility that they could develop approaches novel in the country to any of these aspects, maintaining appropriate interactions with other development partners; it could also provide opportunities for facilitating leveraging. Performance Evaluation of TLMP – South Africa 5 1. EVALUATION PURPOSE AND OBJECTIVES 1.1 Background to TLMP The Textbook and Learning Materials Program (TLMP) was launched in 2005 by the Africa Bureau of the United States Agency for International Development (USAID/W) as a part of the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. TLMP was extended for another three years in 2009 to 2012. The funding mechanism was the use of Cooperative Agreements (CA) awarded to each of five Minority-Serving Institutions (MSI). The University of Texas San Antonio (UTSA) was awarded CAs for South Africa (2005-2008) and Malawi (2009-2012). Each MSI was responsible for generating (i.e., identifying, selecting, developing, adapting, printing, and assisting with distributing) a minimum of 600,000 copies of quality, cost-effective education materials produced in partnership with the national ministry of education and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 1) Produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA, 2) Strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) Ensure alignment with national curriculum and to include relevant cross-cutting themes (i.e. gender, health, etc.). Through an initial needs assessment and discussions with the national Department of Basic Education (DBE) in South Africa, it was determined that UTSA should develop supplemental reading materials for grades 4-6 in nine African languages, plus Afrikaans and English to be in alignment with the then current curriculum. To implement the project UTSA received approximately US$8 million. 1.2 Evaluation Objectives Since the South African TLMP (known as Ithuba) ended in 2008, this evaluation differs somewhat from those done on the other five TLMPs, the Africa Bureau was looking for an impact assessment. As such, the objectives of this evaluation include:  Determine if the planned outcome of facilitating the development of a “reading culture” in South Africa occurred.  Assess the results achieved and determine whether Ithuba had any spin-off effects on policies or practices of the DBE, regional education offices, schools and implementing partners.  Seek evidence of how the practices and effects of TLMP may have continued since the end of TLMP in 2008.  Identity any long-term effects of teacher/educator participation in the development of supplemental readers.  Make clear, explicit and actionable recommendations, suggesting options to expand the impacts achieved to date and determining the conditions under which scale-up and/or replication in other countries would be recommended in accordance with the new Agency Education Strategy http://www.usaid.gov/ our_work/education_and_universities/documents/USAID_ED_Strategy_feb2011.pdf Evaluation of TLMP – TLMP in South Africa 6 The evaluation will also discuss the impact that implementing TLMP in South Africa has had on UTSA, the MSI partner (see ANNEX A – SCOPE OF WORK). 2. PROGRAM BACKGROUND 2.1 TLMP Background in South Africa TLMP in South Africa took the form of the Ithuba Writing Project (Ithuba). The focus of its activities was the development and production of storybooks in the nine official African languages, plus Afrikaans and English, by training teachers to become authors and some to become illustrators. Teachers’ guides in the form of 4-page leaflets linked the stories to the national curriculum in hopes that teachers would make use of the stories to teach literacy across the curriculum. The storybooks were printed by Uniprint in Durban and distributed to target schools through the provincial DOEs. With the donation of the project laptops to two writing centers, one in Gauteng and one in Limpopo, it was planned that writing workshops would spin off to other teachers as well as learners. The idea was that if teachers were engaged in writing in their home languages they would engage learners in writing as well. UTSA and its partners developed a series of intensive writing workshops in Gauteng (largely urban) and Limpopo (largely rural) during which a total of 140 teachers from all nine language groups were taken through the steps of writing original children’s books in their own languages. Teachers were encouraged to find the intersection between their experiences/expertise and topics that children in grades 4, 5, and 6 (the “Intersen” years) would find interesting. Authoring teachers were asked to write their stories in the language in which they felt most comfortable. These teacher-authors were also provided a workshop on rendering a version of their story in South African English—not a translation, but a relevant adaptation of the story, accounting for differences between languages, cultures and experiences. They drafted their stories on laptop computers, giving them access to technology and increasing their skills sets to include word processing and formatting. The project implementers also designed an illustrators’ workshop to develop the skills of local artists, whose original works were then featured in the books. The writing workshops were designed and implemented by TLMP leadership at UTSA and its partner university in the U.S., the University of Texas Austin (UTA). The workshops were first delivered to trainers (using a training-of-trainers (TOT) model) and then rolled out with teachers from target areas. Assisting UTSA in delivering further workshops were five trainers from Molteno Institute of Language and Literacy (Molteno) and 10 trainers from READ Educational Trust (READ), all of whom had been trained in materials development processes. The goal was to create a model for materials development so that teachers would have the capacity to use storybooks to promote literacy learning along with the capacity to create future iterations of stories in appropriate languages to suit their needs. The separate, spiral-bound training manuals, which were provided in each trainer kit covered the following topics:  Drafting Our Stories  Revising and Illustrating Stories  Revising and Field-Testing our Stories  Developing, Revising and Editing Stories  Implementing Ithuba Materials The latter manual covered read-aloud, shared reading/guided reading, independent reading, and discussion topics for teachers as well as matrices linking the stories to “standards” (competencies) in the National Curriculum Statement (NCS) curriculum and referring to the individual teachers’ guides for each storybook. Evaluation of TLMP – TLMP in South Africa 7 As an intermediary step in the process, members of the development and editing team (which consisted of South African and U.S. literacy and content experts) worked on the English versions of the books to ensure their instructional quality. The team also worked to link the storybooks to the national curriculum. The newly-trained local illustrators drew pictures for mother tongue and English versions of the books, and the books were field tested with children in South African public schools by the authoring teachers. Acceptability criteria included: word choice (readability), comprehension levels, and story quality criteria (including relevant illustrations). Finally, a team of South African language experts edited the stories for appropriateness of word choice and conventionality. Eventually the entire effort resulted in the production of 140 storybooks in English and 140 storybooks among the nine official African languages. The books also had to include certain cross-cutting issues, in accordance with the CA: cultural relevance, gender roles, and HIV/AIDS. In keeping with this directive, the percentage of stories covering these and related topics are as follows: HIV/AIDS (26%), abuse and bullying (15%), crime, racism and classism (97%), and cultural traditions (25%). Based on the implementers’ own analysis, common themes the stories included were caring (31%), responsibility (23%), achieving goals (21%), earning respect (7.8%), traditional values (3 stories) and breaking gender barriers – all values that society and schools would want to instill in children. Each narrative was between 20 and 32 pages long, contained between 8 and 10 full-color illustrations and was approximately 4.5” X 5.5” in size. The stories were instructionally integrated with the NCS through teacher’s guides that served as an impetus for mathematics, natural science, or social studies lessons that followed the reading of the books. The contract to print the books was awarded to Uniprint. After print runs of all titles, kits were assembled that included storybooks and accompanying teachers’ guides in the language(s) chosen as the Language of Learning and Teaching (LOLT) at each target school. They also included an Ithuba Overview guide to assist teachers in integrating the teaching and learning materials (TLM) across the curriculum. As integral TLMP partners, Molteno and READ facilitated the teacher workshops on the use of the storybooks. These two non-governmental organizations (NGOs) reported that they conducted workshops of 8 and 16 hours that trained a total of 5,903 teachers and principals on the use of the storybooks. By the end of the project, approximately 8,000 teachers had participated in these trainings, which included an orientation to using the Ithuba teacher’s guides accompanying each storybook. After TLMs were delivered to the provincial DOE (and subsequently made available in schools), Ithuba organizers reported achieving the following outcomes:  Educators who worked in learning areas outside of language development learned to integrate the stories into their specialized curriculum.  Teachers were introduced to Shared Book Reading and Read Aloud methods.  Educators interviewed reported using the supplementary readers during daily lessons and that their students enjoyed them.  Teachers were so impacted by the books that one teacher in Limpopo reportedly quit smoking after reading one of the Sepedi stories, Le reng le kgoga, which discusses the dangers of smoking.  Authoring teachers received 10 hours of Continuing Education credit from the University of Pretoria that can be used toward a degree program. When TLMP ended, READ offered to conduct a final print run of Ithuba materials and to offer teacher training based on those storybooks. READ also proposed to work closely with provincial education staff to add value to these endeavors, and to bear all administrative costs of these efforts, including the costs of additional training. During this same activity, READ took responsibility for the re-kitting of the 227 Evaluation of TLMP – TLMP in South Africa 8 kits printed by Uniprint that were stripped out and repacked as some had not been appropriately packed for the regions to which they were to be delivered. Ithuba also helped to establish two writing centers, one in Limpopo and the other in Gauteng, with the idea of creating some sustainability in the process of training teachers to be storybook authors and the laptops used by Ithuba were donated to these centers. However, by the time that the evaluation team conducted its work in May 2013, the writing centers had become defunct. As a result of Ithuba, staffs at UTSA and the partner institutions have generated a number of publications and presentations at international meetings. The MSI implementers have expanded on lessons learned under Ithuba in implementing a subsequent TLMP project in Malawi. 2.2 DBE Background and Priorities The 1997 adoption by the South African government of its Language-in-Education Policy (LIEP) mandated that children be taught in their home languages throughout primary school. This was a challenge for South Africa as historically under Apartheid, several “Bantustans”/Homelands had been developed in each region, each constituting the separate homeland for a particular ethnic group that spoke a specific language. When the government changed, a decision had to be made in each province as to which African language would be the “official” language. For example, in Limpopo there had been approximately 14 homelands, with each speaking its own language and with each creating a homeland￾based educational system. The challenge for education became 1) deciding which language would be the “official” language of the province, and 2) how to dismantle the Apartheid-based homeland education departments and form one provincial department of education (DOE). DOE officials and different language communities were hard-pressed to make this decision, although they saw the reasoning behind it. The decision was ultimately made for each province, resulting in the nine “official” languages; however, because of the overwhelming number of languages spoken in several provinces, students must be fluent and be able to read in at least three languages: mother tongue, the language of learning and teaching (LOLT), and English or Afrikaans. This created huge problems in terms of providing learning materials to schools to satisfy the DBE requirement to teach in the “official” mother tongue of the province. At the time of TLMP implementation, there was no specific curriculum development unit within the DBE. When UTSA presented its approach, DBE officials were skeptical as the prevailing assumption was that teachers would not be able to develop the skills necessary to produce the materials. With careful explanations and evidence from similar activities in the U.S., UTSA convinced the DBE of the efficacy of the approach. Thus Ithuba was the first to involve teachers and administrators in developing learning materials. Because the approach was the first of its kind in South Africa, new ways of collaborating with the DBE had to be developed. This challenge had to be met in the context of USAID/SA closing out its educational programs. Despite this programmatic closure, USAID/SA assigned two members of its staff in Pretoria to provide limited assistance. With the closure of the USAID/SA education program, according to UTSA, when TLMP ended in 2008 it could not be renewed; instead, UTSA went on to conduct similar activities in Malawi for grades 1-4. 3. EVALUATION METHODOLOGY AND LIMITATIONS 3.1 Overall Methodology In collecting data for this impact assessment, a “mixed methods” approach was used, comprised of:  Review of program documents and a sampling of TLMs produced Evaluation of TLMP – TLMP in South Africa 9  Key informant interviews with UTSA staff, in-country partners, DOE personnel at various levels, staff of other stakeholders, and school principals and teachers  Site visits, including classroom observations and, as feasible, getting perspectives from learners as well as from educators Site visits were used, inter alia, to determine whether books actually had been distributed to schools and, if so, if they were still being used. Materials were reviewed to determine gender balance and appropriateness; apparent bias with respect to urban-rural issues; apparent relevance and appropriateness of content, including illustrations; language level; alignment with curricula; etc. 3.2 Research Conducted at UTSA Dr. Nancy Horn (team leader) conducted advance on-site and follow-up field work at UTSA to determine 1) the background to the project and how it was implemented, and 2) how MSI participation in a USAID-funded program built its capacity to engage in future development projects. 3.2 Research Activities in South Africa The field work in South Africa was conducted between May 5 and May 16, 2013, inclusive, by Dr. Edward Allan (project director), Dr. Carol Benson (a specialist in language and education, with particular familiarity with South Africa), and Dr. Rakgadi Phatlane, an educational researcher from the University of Pretoria. Dr. Thabile Mbatha also provided valuable assistance, although ultimately we were not able to visit KwaZulu-Natal. With the background information provided by UTSA, the team leader developed a background paper/research design that included questions to be posed to different stakeholders. However, finding individuals to answer these questions was highly problematic owing to the passage of four years’ time since the project ended. During this time, officials were reassigned or transferred; partner staff members resigned or passed away; leadership changed in all areas; the reorganization of education departments in the provinces and districts continued; and records were not consistently kept of all the changes. Consequently the questions were refocused to ascertain impact. These questions are as follows:  How was the DBE influenced by the strategy for developing supplemental readers by Ithuba? Was this methodology utilized in any other way by relevant education departments at the national or regional level?  How have READ and Molteno continued to use the storybooks and/or processes developed for Ithuba in other parts of the country, and what have been the results?  How have teachers continued to use the materials?  How has children’s reading ability in home and other languages increased?  What are the elements of sustainability for Ithuba? The range of stakeholders from whom data were collected included: officials of DBE and provincial DOEs; representatives of READ, Molteno and Uniprint; representatives of the University of Limpopo (UL) and the University of Pretoria (UP); and school principals, teachers of grades 4, 5, and 6, and students (see ANNEX B: Schedule of Stakeholder Interviews and Activities). The team conducted interviews and site visits in and around Pretoria and in Limpopo. Most interviews were done in person, but some were done by telephone or e-mail depending on the ability of the team to contact people who had been involved. Only a few teachers were observed using the storybooks to determine how they have adapted the materials to meet their own teaching and learning requirements Evaluation of TLMP – TLMP in South Africa 10 since TLMP ended, and these presentations were “staged” to accommodate the time of the team’s arrival. 3.3 Limitations of the Study  Inability to contact some key TLMP stakeholders: Since TLMP had ended four years earlier, the team was not able to find a number of key stakeholders. In some cases they had moved on, in some cases there were scheduling conflicts, and in the case of the KZN MOE we were not able to identify anyone who remembered anything about Ithuba. Further, with respect to KZN, our team was not able to identify any schools known to be using Ithuba materials located within a day’s drive of Pietermaritzburg, the capital.  “Slow down” and strike called by teacher’s union: Teachers’ strikes and slow-downs that were taking place during the early part of our work in-country made it extremely difficult to schedule visits and meetings with education officials. The first two school visits were arranged via personal connections of a team member.  Limited ability to assess the storybooks: The team did not have copies of the readers in order to assess them formally. 4. FINDINGS AND CONCLUSIONS 4.1 Ithuba Achievements and Challenges The team visited three schools: one in a Pretoria township and two in rural Limpopo, each about 100 km in different directions (northeast and southeast) from Polokwane. As a general note, the three schools visited, including the small village school in Siseluselu located several kilometers off the main road north of Tohoyuando, were equipped with physical plant in good condition (including separate latrines for boys and girls) plus electricity and working computers, fax, printers, and copiers. Children and teachers seemed happy to be there, and it did not appear that the schools had been “primed” for our visits. The schools did have rooms intended for libraries and, in principle, had people – teachers or community volunteers – scheduled to serve as librarians. Ithuba can count among its achievements the following:  Storybook/TLM development: Ithuba contributed significantly to the number of high-quality supplementary reading storybooks available in schools in the nine official African languages plus English and Afrikaans. Other books were available in libraries, but the Ithuba books were prominent on the shelves and invited interest with their illustrations and format. The Molteno representatives reported that schools without libraries set up reading corners or brought out Ithuba books from storage to use during reading time.  Use of TLMs: It appears that the storybooks are being used as they were intended, as supplementary readers—at least for Language Arts. Ithuba books were found on the Ithuba stands, in cupboards that could be rolled to classrooms as needed, on library shelves and in classrooms. While we could not tell how regularly the TLMs were used, from their physical appearance and from the classroom observations it was clear at each school visited that various titles were in active use, by groups of students and/or by individuals. Although intended for the higher grades, we were informed at different locations that Foundation phase (grades 1-3) class teachers also used the storybooks, mostly for reading to their students.  Contribution to improved reading ability: Virtually all of the students we observed in grades 4 through 6 who were asked to read, whether chorally or individually, were able to do so with accuracy and fluency. While there is no way to tie this directly to the Ithuba storybooks, there was evidence that they contributed: for example, a Molteno representative reported that she saw Evaluation of TLMP – TLMP in South Africa 11 students reading technically difficult graphemes (like consonant trigraphs in Sepedi) in the Ithuba books with comprehension and fluency, and they were able to re-tell stories orally or in writing, as called for in the Ithuba teachers’ guides. The team noted that a “typical” behavior in choral reading was to read slowly, with brief pauses between syllables, ostensibly so that everyone can keep up. Even when this was done, the overall intonation and clear stops between sentences or ideas demonstrated that children understood what they were reading. When individuals were asked to read, they demonstrated their fluency and even tried to read quickly; at times it seemed competitive. It was common for teachers to ask basic comprehension questions, which most students answered, thus demonstrating they understood the passage. Some teachers also asked students questions which called on them to extrapolate from the passages and develop their own perspectives, including engaging in ad lib role playing.  Building teacher competencies to develop reading materials: Enlisting and training teachers to become authors was innovative. The Ithuba author workshops reportedly raised the capacity of teachers and trainers to be authors while motivating them to write actively. The teacher trainers/educators in particular, who were part of the piloting of the methodology, were reportedly inspired and their capacity raised, so much so that some went on to inspire others. The hope throughout the process was that educators’ own writing would inspire students to write, which will in turn develop literacy skills in all languages, and at the Onane school, students volunteered their desire to write their own materials. Some teachers continued to write stories for their classes; the Molteno representatives said that teachers would approach them saying “I’ve written another story; how can I publish it?”  Building the capacity of illustrators: The project also reportedly raised the capacity of a smaller group of people to be illustrators, and judging by the high quality appearance of the books we saw the illustrators did good work. We were not able to speak to any of the illustrators, but we were told that there was at least one illustrator who became highly motivated to train others and that there is now a growing body of South African illustrators capable of illustrating commercial textbooks.  Expanding the approaches used and competencies built by Molteno and READ: According to Molteno representatives, the writing workshops were highly successful in getting teachers motivated and active in their writing; however, the materials needed a lot of reworking. The process of “versioning” worked well from their perspective as well as from the UTSA implementers; this involved having the teachers write an English version of their home language story, then re-working them with experienced professionals to shape the story by applying materials development standards.  Sustaining the ideas and practices of Ithuba: UTSA informed the team that writing centers were developed in Limpopo and Gauteng that adhered to the ideas and practices developed by Ithuba. At the end of the project, the laptops used were donated to these two centers to facilitate the writing process. The practice was also implemented in writing camps for students. This notwithstanding, the team found no evidence that these writing centers still exist, at least not in their original form.  Further extension of Ithuba storybook development: In one case, a library specialist at the Limpopo DOE convinced her colleagues to develop storybooks in Sepedi (with a plan to extend the program to the other official African languages) for the foundation phase. They gained the support of Irish Aid to develop mock-ups and field-test them in the schools, but the storybooks have not yet been printed for financial reasons. Room to Read was in the process of incorporating Ithuba into its library programs. At the University of Limpopo, members of the English Department were inspired by prior involvement in Ithuba and in the Writing Center on campus. They sought funding from Belgium and created the “Multiple Literacies Project,” which produces high quality storybooks in home languages using the Ithuba model. Evaluation of TLMP – TLMP in South Africa 12  Continued use of Ithuba TLMs under curriculum reform: The Curriculum and Assessment Policy Statement (CAPS), the new curriculum policy, emphasizes early literacy in the mother tongue and literacy across the curriculum. While we do not have “hard” evidence for the consistent use of Ithuba materials throughout the provinces, we do have anecdotal evidence that they are being used, e.g., a Molteno representative said she was pleased to visit a multigrade class in Mpumalanga for another reason and the teacher was using Ithuba books in Ndebele, and in a telephone interview, the READ provincial officer for Mpumalanga confirmed to the team that Ithuba materials are still actively in use in the province. The singular focus on developing reading materials in home languages also enhanced the relationship between READ and Molteno. TLMP also experienced some challenges:  Sustainability issues within the DBE and USAID/SA: The biggest challenge, with strong implications for sustainability, was raised first by the USAID/SA education team at the in-brief: Why hasn’t the DBE taken on TLMP/Ithuba as its own, and/or expanded on the model for developing these much-needed storybooks in the nine official African languages? The USAID/SA education representatives gave this as justification for not extending the project or using its models or processes, despite the great need for storybooks in African languages. When we raised the question with Molteno representatives, they said that they had worked with the DBE which “should have had ownership and kept it going… but there was a break.” By “break” they were referring to a period of reorganization of DBE/DOE departments; this was confirmed by the DBE representatives from the curriculum department with whom we spoke, who said they had been “consumed” by the reorganization until quite recently. They were very familiar with the project; in fact, there was an Ithuba stand in the office where we met, and they were quite positive about the storybooks based on written and anecdotal reports they had been given. They also recognize the Ithuba storybooks as consistent with current curricular aims, both for home language and for “first additional language.” Because commercial publishers are “not coming up with readers,” they would like to use Ithuba as a “springboard.” Their question to us was: “Now that we are organized to support early literacy, is there any chance of re-examining the project and/or using it as a model?” This suggests that USAID would do well to re-visit the project and facilitate DBE ownership along with the NGO partners.  Uneven provincial inclusion of Ithuba: We had planned to have a team visit to KwaZulu-Natal. However, despite significant efforts we were unable to identify anyone at the KZN DOE with knowledge of Ithuba, nor were we able to identify any schools within a day’s drive of Pietermaritzburg where Ithuba materials were in use. The unevenness of Ithuba involvement in KZN was explained to us by an NGO representative as a result of the high turnover at the DOE during reorganization.  Lack of equity in distribution of titles across home languages: While each language had eight titles of “group readers,” there was significant variation in the number of “independent readers,” e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga.  Pedagogical issues: Another challenge relates to the pedagogy used in working with the storybooks in the classroom. “Working” is the key word here, because most of the sessions that we were shown with the storybooks were heavy on the “didactic” side and less on the “enjoyment” side. In at least two cases, the students were not allowed to read to the end of the stories, although we could see that some students were reading ahead on their own. There was little silent reading in the classes that we observed (possibly an artifact due to our presence), except in a lesson where the teacher (a head teacher who had stepped in to demonstrate use of the storybooks for us) had given them comprehension questions and asked them to write the answers in their notebooks. Teachers generally did have the idea of letting students read aloud independently as well as together, although there were occasional vestiges of traditional methods like reading first and asking students to repeat. Evaluation of TLMP – TLMP in South Africa 13 It was not possible to determine if the less traditional approaches taken by teachers were a result of Ithuba or other inputs—or a combination.  Regular and consistent use of the TLMs: Adding to the challenge of methodology, it was difficult to see if teachers used the books regularly, because the sessions we saw were “staged” for us. We were told at each school that teachers used the books, but there was no instance where a regular schedule of storybook reading was reported. Apart from the classroom observations which indicated that students were familiar with the readers, the main evidence we found that the books were in use was where the school directors were positive about Ithuba and where the books were displayed prominently in teachers’ rooms or libraries. Finally, despite assurances from the DBE and NGOs, we do not know if teachers themselves perceive that the storybooks are relevant to the current curriculum (CAPS). Even if they are, the references to the curriculum in the manuals would require adapting/updating/re-editing.  So many books, too few copies: As supplemental readers, just 20 copies of each title were provided to each school in its LOLT. However, as the books were very attractive and the stories were well contextualized, teachers might have wanted to use them more often as part of their reading lessons. Teachers generally had students share the books in pairs during classroom time, which seemed to work – for the size classes that we observed.1 However, if classes were larger, there would not have been enough copies for effective sharing, and 20 copies per school for a reader is not enough to make it possible to let students bring them home, where they could be used to broaden household use2.  Teacher’s guides getting lost: The teacher’s guides were developed as separate sheets (in strong cardboard) for each storybook; they were not bound in a single volume. We did not see the guides at all of the schools we visited, so we imagine that some sheets were lost or remained with some teachers, leaving others without expert guidance in using the storybooks.  Language mismatches: In some instances about which we heard, the home language spoken by the teacher/author was not the same as the home languages spoken by learners at the schools.  Distribution issues associated with relative number of speakers of a language in a school: Generally, with respect to the schools that we visited, books, including Ithuba materials, were distributed in proper numbers for the languages of the students at a school, although there could have been changes in demographics taking place between the time that books were requested and the time they were delivered and needed. (We saw at one school a chart of the number of students speaking a particular language by grade.) While at one school, we saw that the DOE had not reclaimed surplus copies of regular textbooks, we were told by the local implementers – as evidenced by our observation at the Mamelodi township school – that they took efforts to reclaim and redistribute Ithuba materials that were not needed at particular schools.  Selection of Dialects: This is a general issue associated with use of home languages for instruction, and not linked specifically to Ithuba. Various local languages have different dialects, which in some cases include borrowings from other languages. There are not necessarily broadly accepted “standard” dialects of a language, and generally materials were developed in the dialect spoken by the author(s). This can cause a pedagogical problem for children (and teachers) from different locales unless teachers are informed to let their students know that there are dialectal differences and that one dialect is not a priori better or worse than a different dialect.  Leveling issues: “Leveling” refers to the use of vocabulary and syntax assumed to be appropriate for the age level and competencies of the children for whom the materials are designed. According to Molteno representatives, it was a challenge to develop readers at the appropriate levels for 1 South Africa Government policy is 40 learners per primary school class. SAQMEQ Policy Brief #2, The Quality of Primary School Inputs in South Africa (September 2011). We cannot speak as to the extent to which actual enrollments reflect the policy. 2 One “work-around” would be for teachers to be working with more than one reader at a time, with students alternating. We do not know if this had been proposed. Evaluation of TLMP – TLMP in South Africa 14 students in grades 4-6 in the nine official African languages, as little had been documented on which home language literacy skills were developed at each grade level. Stories were written and edited by experienced teachers, but ensuring that they could be read by students in grades 4-6 was problematic. Selected teachers/authors attempted to field test the materials in their own schools, but the literacy levels achieved in each school varied. Hence, some of the books hit the mark but others missed in terms of level. On the other hand, at two schools we heard that learners as young as grade 2 were also using the Ithuba materials – primarily because there were no local language materials for their grade levels, so their uneven levels may have been a benefit rather than a hindrance to literacy learning.  No exchange of materials across languages. Combined, the Ithuba authors prepared some 130 readers in one or another of the languages of instruction. However, we were told that the DBE had not permitted materials developed for one language to be adapted for use by speakers of other South African languages – even though the materials would have been far more relevant to the lives of South African children than materials from New Zealand, which were not subject to such constraints.  Writing Centers not sustained: Ithuba had helped to set up writing centers for Limpopo and Gauteng. However, we were not able to find any legacy of them.  Need to plan for scale-up: We were informed by the Mission that DBE is unwilling to permit new pilot projects unless there is funding already planned for meaningful scale-up. While this obviously represents a challenge, it should be noted that there is demonstrated potential for at least some leveraging from, e.g., Irish Aid.  Printing issues: According to UTSA, some of the responsibilities that Uniprint had to assume (building book stands, “kitting” (packaging) materials in specific languages to be delivered to specific provinces, responding to technical requirements, delivery of materials to deep rural areas, etc.) demanded particular attention as to the timeliness and correctness of delivery. As a result, some language-specific kits were inappropriately assembled because little quality control was exercised. There were also differences in understanding as to delivery of materials between the printer and education authorities. UTSA found that monitoring the printer was a major challenge due to the lack of consistent in-country presence. Despite substantial efforts, the team was not able to glean any information from current representatives of Uniprint regarding Ithuba, who claimed that they did not have records. 4.2 TLMP Output By the time the project ended, TLMP had produced 2.3 million storybooks, teacher’s guides and Overview guides, and distributed them to more than 2,500 schools in each of the nine provinces, providing slightly less than 1,000 volumes to each. A total of 140 titles was developed, some of which were subsequently translated into South African English and/or Afrikaans. 140 educators had received training in writing stories, and 8,156 teachers had been trained in the use of the TLMs (largely by READ and Molteno). 4.3 TLMP Impact Many of the items noted as achievements under Section 4.1 above provide the basis for the impacts observed and discussed in this assessment.  Lived experience as the basis of stories: Educators were taught how to write stories based on their own lived experiences; the stories were of significant interest to the students; the hope was that teachers would encourage students to write their own stories. Several informants representing different categories of respondent told us that this experience greatly empowered many authors, e.g., through increasing self-esteem and community and professional recognition, and various educators continued to prepare stories. There was also perceived sense of pride, almost “bragging Evaluation of TLMP – TLMP in South Africa 15 rights,” by students that their teachers had developed materials that were actually being used in schools.  Reading is a life-changing activity: As the TLMs addressed cross-cutting issues, including various health topics, one educator reported having stopped smoking because she learned of the consequences of that activity. Many of the themes were real to students, including how children’s lives have been impacted by HIV/AIDS.  Broadening perspectives and determining alternatives to cultural stereotypes: A frequent response from students when we asked why they liked the Ithuba materials was that they gave the students perspectives on how people in contexts similar to their own faced issues. According to the implementers, there were a number of stories that challenged assumptions about age and wisdom, male and female roles, body image/old and fat vs. young and thin; and dress. We were not able to verify how teachers used the storybooks to engage learners in discussions, but if the storybooks are being read, we know children are being exposed to broadened perspectives as Ithuba aimed to do. 5. LESSONS LEARNED  Create good relationships with DBE from the outset: According to MSI implementers, many of the successes of the project emanated from the good relationships established between UTSA and the DBE. Skeptical at first about the methodology, DBE officials were ultimately convinced that it was sound for developing supplemental readers. Without the professional approach of TLMP leaders and their consistent professional interactions with DBE, TLMP may not have taken place.  Identify divisions of responsibility: Especially in the context of rapid and/or frequent changes in organizational leadership and structure, curricular philosophy and content, and the like, it is important to identify – and keep track of – “who does what?” In a system like South Africa’s, in which provincial DOEs currently seem to have a fair amount of autonomy but not necessarily much in the way of resources, this knowledge can be quite beneficial when planning for implementation, scale-up, replication, and sustainability.  Writers could also have been editors: From UTSA reports, it was noted that once the South African team members had developed skills in estimating reading levels and judging appropriateness of stories, they could have been relied on to do all of the editing, rather than sending materials to UTSA. This increase in capacity was confirmed by a local implementer, indicating one specific area in which Ithuba achieved its goals. However, the ability of South Africa to benefit from this increase in capacity depends on the extent to which it makes use of teacher-developed materials.  Ithuba needed a workable M&E system: Representatives of local implementers noted the lack of adequate follow-up, which hampered ability to analyze if and how the materials were used, any apparent concerns, feedback from teachers on their own perceptions and those of their students as to the readers as a whole or individual readers, and apparent contributions to student learning.  “Champions” with likely continuity are needed. We were told by a local partner that senior level education officials were committed to Ithuba at inception. However, over the course of time and personnel changes and practical necessity, other activities took higher priority. Based on very limited evidence, it appears that where Ithuba found sustainability after the end of USAID funding, it was due in large part to the commitment of the local partners to the project and of individual educators who learned from their own participation how effective and empowering the activities were for themselves and for learners. This suggests the great desirability also of creating a “critical mass” of participants who can act as advocates. This appears definitely to have been the case in Limpopo and at least to a quite noticeable extent in Mpumalanga and the environs of Pretoria and appears not to have been the case in KwaZulu-Natal. Evaluation of TLMP – TLMP in South Africa 16 6. RECOMMENDATIONS The following recommendations are intended both to inform USAID/South Africa for its education efforts and to inform the Africa Bureau and USAID education officers planning activities involving the development of textbooks and learning materials for literacy development projects in basic education. These recommendations are based specifically on perspectives from the Ithuba evaluation team. While a number of issues have surfaced in various implementations of TLMP, we have not included recommendations that do not have some identifiable basis from Ithuba. (General observations from the range of TLMP implementations are found in the synthesis Project Report.) 1. (USAID/SA) Engage in dialogue for continued use of Ithuba materials. There is both a real and a perceived need for supplementary reading materials to promote literacy in the 11 languages of instruction in South Africa’s primary schools. The American taxpayer has already invested millions of dollars in the development and production of some 140 supplementary materials in the languages of instruction for grades 4-6. These materials are, we have been told, consistent with the new curriculum guidelines, they seem generally well developed and are well received by the South African educators and students whom we have met, and other entities have been willing to provide some leveraging. Further, they are branded as coming from the people of the United States and have the potential of being seen and used in almost every household with a school-age child in South Africa. We strongly recommend that the Mission engage in dialogue with the Department of Basic Education (DBE) with respect to updating the materials and having them incorporated as supplementary readers for current education initiatives. 2. (USAID/SA) Consider using the Ithuba materials development model to develop new storybooks for early grade literacy (in South Africa, the Foundation level): Involving teachers and other educators in the story writing process was designed to create long-term impacts in teachers’ approaches to reading and writing as well as a means of getting quality materials produced. Literacy (in home languages) has been identified as a priority by DBE in South Africa and by other national ministries of education. Having students also participate in the story writing process will keep them engaged in learning and contribute to the creation of a reading culture. Writing and reading are mutually-reinforcing skills, so students who are engaged in both will more likely learn more efficiently and reach higher degrees of literacy sooner.  While each language had eight titles of “group readers”, there was significant variation in the number of “independent readers” in unequal numbers, e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga.  In addition to providing speed in production and economies of scale, we believe (from professional knowledge, not directly from TLMP) that having some familiarity with the lives of citizens from other parts of the country can help to instill a sense of national unity. 3. Identify the likelihood of curricular and/or policy changes taking place during the life of the project and attempt to act proactively: This is obviously easier said than done, but early engagement to the extent possible can reduce the possibility that materials may be obsolete soon after they will have been developed. 4. Selection of implementers for future projects like TLMP: Implementing teams for projects that include TLM development should have the following characteristics/ skills: knowledge of the process of TLM development;3 knowledge of the national (i.e., South African) context; understanding of how current materials satisfy (or not) policy or curriculum requirements; knowledge of how story writing and other activities promoting literacy can be integrated into existing teacher training mechanisms; understanding of the sociolinguistic context and the language capabilities of teachers; understanding of the teaching methodologies currently used to promote reading and writing in the 3 We recognize that for TLMP, development of the skills, if needed, by the MSI was one of the objectives. Evaluation of TLMP – TLMP in South Africa 17 classroom; knowledge of the printing and dissemination process utilized by public and private agencies; and knowledge of piloting or other testing of materials to determine the actual (as opposed to notional) capacity of learners. 5. Provide teacher training workshops on the use of materials produced to all teachers and administrators involved in early grade reading: Whenever materials are produced, regardless of the ultimate goal, all teachers and administrators involved in their use or in the support of their use must be oriented regarding how to use them. Inherent in teaching any subject is the methodology used to facilitate the acquisition of knowledge. The TLM storybooks and teacher’s guides exposed teachers to alternative teaching methodologies that were more student-centered and participatory than many of those currently in use. To make these methods concrete rather than suggestive, teachers and administrators must be trained on their use as well as the pedagogy involved. To ensure the use of the TLMs and the training of teachers to use them, TLMs should be incorporated into the teacher training college pre-service and in-service curricula as part of a systematic approach to reading and writing in the home language. 6. (USAID/SA) Encourage the adaptation of materials that have been proven valuable in one language for students who speak other languages. There will likely continue to be significant shortfalls in materials in local languages for students to read. While there will be some duplication in content across some of the Ithuba materials that have been prepared for different languages, there are nonetheless unique materials that can be readily adapted for children who speak other languages.  While each language had eight titles of “group readers”, there was significant variation in the number of “independent readers” in unequal numbers, e.g. none in Ndebele, Setswana, Sesotho and Tshivenda but five in Zulu, 11 in Xhosa, 12 in Seswati, 14 in Sepedi, and 15 in Tsonga.  In addition to providing speed in production and economies of scale, we believe (from professional knowledge, not directly from TLMP) that having some familiarity with the lives of citizens from other parts of the country can help to instill a sense of national unity. 7. Bind teachers’ guides into single pamphlets: The teachers’ guides for the Ithuba readers seem to be about the right length – four-page leaflets. However, because they are leaflets, we found that in various cases at the schools we visited, they had “strayed.” We recommend that for Ithuba (and for comparable activities), these leaflets be bound into single booklets that also incorporates general guidance on how teachers can use supplementary readers in literacy/reading classes. 8. Consider the use of Annual Program Statements (APS) to complement ongoing education activities. It may not be workable for an education activity to incorporate the development of learning materials as part of its work; e.g., its primary focus might be on teacher training or educational administration. However, this does not obviate both the need and desirability for learning materials. An APS would give local entities, possibly in collaboration with U.S. partners, the opportunity to build its own capacity and the capacity of local educators to develop, produce, distribute, and use materials on, e.g., a provincial level with the possibility that they could develop approaches novel in the country to any of these aspects, maintaining appropriate interactions with other development partners; it could also provide opportunities for facilitating leveraging. 9. (USAID/SA) Support local reproduction of materials. It may not be feasible to launch large￾scale press runs to reproduce particular readers. However, the schools that we visited, even in remote areas, were equipped with functioning copiers, and presumably district offices also have them. The TLMP cooperative agreements allow for local reproduction and adaptation of materials. Provincial DOEs should be informed, via DBE, that if a school or district is short some copies of a particular reader, it is free to make needed copies on its own, in color if possible. 10. If consistent with national textbook dissemination policies, engage local private sector publishers/printers in the development, printing and distribution of materials: Given the Evaluation of TLMP – TLMP in South Africa 18 possibility that some households might wish to buy their own copies of materials, private sector printers can help to ensure that there is a steady stream of materials to stock both the public and private sector. 11. (USAID/SA) Encourage more effective use of libraries: The schools that we visited had stocked libraries staffed by teachers and/or community volunteers. They were in use, and books seemed to be used. However, at least for Limpopo, each of the DOE personnel responsible for guiding libraries is responsible for well over 1,000 schools. It would be useful if the Mission could arrange for the development of training that would help school librarians learn how to promote use of the library as a lending library since improving the ability of students to take books home would provide for reinforcement of learning, the strengthening of a culture of reading, and the possible engagement of family members in improving their own ability to read. Evaluation of TLMP – TLMP in South Africa 19 ANNEX A: SCOPE OF WORK DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners‟ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MoE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History  TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over 3 million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings.  TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over 6 million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English.  TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers guides and training materials and trained nearly four thousand teachers on methodological classroom usage.  TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 1.8 million materials in both French and English for grades 2-10 in Evaluation of TLMP – TLMP in South Africa 20 science, mathematics, and language arts. Over 160 teachers were trained on utilizing the materials as part of their curriculum.  TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over 6,000 teachers. The work was completed in 2009.  TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over 1,200 teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection. Task 2 – Data Review. Task 3 – Coordination and Management. Task 4 – Site Visit. Task 5 – Data Analysis. IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives:  validate stated program goals and impacts;  assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions, Ministries of Education (MoEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country;  determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution;  review allocated USAID funding in terms of usage and overall cost effectiveness;  highlight specific program accomplishments per MSI-host country partnership; and Evaluation of TLMP – TLMP in South Africa 21  document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Str ategy_feb2011.pdf 6 V. OPERATING CONSTRAINTS / LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance can be found here: http://www.usaid.gov/policy/ads/200/203.pdf The new USAID Evaluation Policy can be found here: ttp://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. Evaluation of TLMP – TLMP in South Africa 22 ANNEX B: SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES FULL TEAM Mon 5/6 In-brief with USAID in Pretoria  Meredith Fox, Education Team Leader  Peter Cronin, Education Foreign Service Officer  Nalini Reddy, Project Management Specialist – Education Tue 5/7 Meeting with Molteno Institute for Language and Literacy at Sandton Paula Gains, Research & Development Manager Johanna Mogodiri, Provincial Coordinator Freddy Nambahe, National Training Manager Wed 5/8 School visit to Pfundzo nde Tshedza Primary School (Sepedi with some Venda, Zulu and Tsonga), Mamelodi township, Pretoria, Gauteng province: classroom observations, discussions with faculty, visit to library Afternoon travel to Polokwane Thu 5/9 School visit to Siseluselu Primary School in rural, Venda-speaking part of Thohoyandou district, Limpopo province: classroom observations, discussions with faculty, visit to library Fri 5/10 Meeting at Limpopo Provincial Department of Education  Ms Motlalepua Teffo, Provincial Coordinator, Media & Library Services  Head, Media & Library Services ED and Ms. Teffo Onane Primary School, Hlogotlou-b, Monsterlus, Limpopo Province http://www.onaneprimaryschool.co.za/  classroom observation, discussions with students, library visit  Sihlangu Solomon, Deputy Principal Carol and Rakgadi University of Limpopo  Former writing center at Dept of Languages, School of Languages and Communication Studies: Mr Mpho Seerane, former student of Dr Leketi Makalela  Meeting at Dept of Languages: Dr Rose-Marie McCabe, Ms Lehlogonolo (Lelo) Mkola, Ms Bongiwe Nomatsorane, Mr Noel Manganye about Ithuba and follow-up project funded by Belgium  Sepedi Department: Dr Mamalatswa Maruma  Contemporary English and Multilingual Studies program: Dr Esther Ramani Final meeting with Ms Teffo back in Polokwane Sat 5/11 Travel back to Pretoria, discussion of findings WHOLE TEAM Mon 5/13 Meeting with University of Pretoria linguists  Prof Vic Webb  Prof Michel Lafon READ Educational Trust  Ellen Dlamini, Mpumalanga Provincial Director (via telephone) Discussion with education professor Lilli Pretorius (CB) Tue 5/14 Work on notes and debrief Evaluation of TLMP – TLMP in South Africa 23 Phone interview with Carole Bloch and other colleagues involved with early literacy (CB) Wed 5/16 South Africa Department of Basic Education  Jennifer Joshua  Kahlula Manona Work on synthesis of South Africa visit Thu 5/16 Debrief with USAID/South Africa: Meredith, Peter and Nalini Meeting with READ Educational Trust  Berthus Matthee, National Director, Evening departure for U.S. Later Follow-up e-mails with Paula Gains, Misty Sailors, Leketi Makalela, Michel Lafon Performance Evaluation of TLMP – South Africa 24 ANNEX C: CONFLICT OF INTEREST STATEMENTS Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Edward Jay Allan Title Project Director Organization IBTCI Evaluation Position [X]Team Leader Team member Evaluation Award Number (contract or other instrument) AID-OAA-TO-12-00054 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date October 10, 2012 X Evaluation of TLMP – TLMP in South Africa 25 Name Carolyn J. (Carol) Benson Title Consultant Organization IBTCI Evaluation Position Team Leader [X]Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date March 03, 2013 X Evaluation of TLMP – TLMP in South Africa 26 Name Rakgadi Phatlane Title Dr Organization IBTCI Evaluation Position Team Leader [X]Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No [X] If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 13. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 14. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 15. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 16. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 17. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 18. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date February 08, 2013 X