June 2013 This publication was produced at the request of the Africa Bureau of the United States Agency for International Development. It was prepared independently by Nancy E. Horn (Team Leader), Tom Tilson, Denis Khasu, and Hannock Mateche under Task Order AFR-12- 00001 awarded to International Business & Technical Consultants, Inc. (IBTCI). The views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. EVALUATION REPORT Evaluation of the USAID-Funded Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa and Tanzania TLMP in Malawi Intentionally left blank EVALUATION OF THE USAID-FUNDED TEXTBOOKS AND LEARNING MATERIALS PROGRAM (TLMP) IN ETHIOPIA, GHANA, MALAWI, SENEGAL, SOUTH AFRICA AND TANZANIA TLMP IN MALAWI Prepared by: Nancy E. Horn, Team Leader Tom Tilson, Team Member Denis Khasu, Team Member Hannock Mateche, Team Member June 28, 2013 The authors’ views expressed in this trip report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of TLMP – TLMP in Malawi i Table of Contents ACRONYMS...................................................................................................................... iii EXECUTIVE SUMMARY................................................................................................. iv 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES.................................. 1 1.1 The Background to TLMP.................................................................................................................. 1 1.2 Evaluation Objectives.......................................................................................................................... 1 2. READ MALAWI PROJECT BACKGROUND ........................................................... 2 2.1 UTSA Responsibilities.......................................................................................................................... 2 2.2 MOEST Context and the Educational System................................................................................ 3 3. EVALUATION QUESTIONS, METHODS & LIMITATIONS.................................. 4 3.1 Research Conducted at UTSA.......................................................................................................... 4 3.2 Research Activities in Malawi (Lilongwe, Zomba and Blantyre Environs)............................... 4 3.3 Limitations of the Study...................................................................................................................... 5 4. FINDINGS AND CONCLUSIONS ............................................................................ 6 4.1 UTSA Achievements and Challenges............................................................................................... 6 4.1.1 Administrative Adjustments....................................................................................................... 6 4.1.2 Beneficial Impacts on UTSA and its Community .................................................................. 7 4.2 TLMP Output......................................................................................................................................... 7 4.3 Project Management and Partnerships............................................................................................ 8 4.4 Project Implementation ................................................................................................................... 10 4.4.1 Materials Development............................................................................................................. 10 Guided Reading and Read Aloud Books..................................................................................... 10 Teacher’s Guides.............................................................................................................................. 11 4.4.2 Comments on the TLMs Made by Teachers, Directors and PEAs................................. 12 Positive Responses........................................................................................................................... 12 Negative Responses......................................................................................................................... 13 4.2.3 Conclusions................................................................................................................................ 14 4.5 Outputs and Outcomes................................................................................................................... 15 4.5.1 Teacher Observations............................................................................................................... 15 4.5.2 Learner Reading Assessments and Outcomes..................................................................... 15 Basic Reading..................................................................................................................................... 16 Comprehension ................................................................................................................................ 16 Gender................................................................................................................................................ 17 Evaluation of TLMP – TLMP in Malawi ii Conclusions ....................................................................................................................................... 17 4.6 Stakeholder Interest In and Use of Materials/Sustainability ................................................ 17 4.6.1 NGO Reading Programs.......................................................................................................... 17 4.6.2 TTCs............................................................................................................................................ 18 5. LESSONS LEARNED................................................................................................. 18 6. RECOMMENDATIONS.............................................................................................19 6.1 Materials Improvement.................................................................................................................... 19 6.2 Systems Improvement...................................................................................................................... 20 6.3 Towards a More Comprehensive Approach to Reading Development............................... 20 6.4 Linking TLMP to USAID Priorities in Early Grade Reading……………………………...20 APPENDIX A. SCOPE OF WORK ..............................................................................22 APPENDIX B1. QUESTIONS POSED IN VARIOUS INTERVIEWS CONDUCTED AT UTSA..........................................................................................................................25 APPENDIX B2. DATA COLLECTION INSTRUMENTS, MALAWI STAKEHOLDERS............................................................................................................29 APPENDIX C. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES ...........................................................................................................................................41 APPENDIX D. READING ASSESSMENT METHODOLOGY ...................................46 APPENDIX E. CONFLICT OF INTEREST STATEMENTS ………………………… 51 Evaluation of TLMP – TLMP in Malawi iii ACRONYMS ADP Area Development Program AEI President’s African Education Initiative CA Cooperative Agreement CRECCOM Creative Center for Community Mobilization CPD Continuous Professional Development EGRA Early Grade Reading Assessment HUB Historically Underutilized Businesses IBTCI International Business and Technical Consultants, Inc. MIE Malawi Institute of Education MoE Ministry of Education MOEST Ministry of Education, Science and Technology MOU Memorandum of Understanding MSI Minority Serving Institution MTPDS Malawi Teacher Professional Development Support (Program) NESP National Education Sector Plan 2008-2017 NPC National Primary Curriculum PAC Project Advisory Committee PCAR Primary Curriculum and Assessment Reform PE Program Evaluation PEA Primary Education Advisors RM Read Malawi S1-S3 Standard 1-Standard 3 SOSA The name given to Book One, Standard 1, Chichewa SSA Sub-Saharan Africa TDC Teacher Development Center TL Team Leader TLM Textbooks and Learning Materials TLMP Textbooks and Learning Materials Program TPR Total Physical Response TTC Teacher Training College UNICEF United Nations International Children’s Fund USAID United States Agency for International Development USAID/M United States Agency for International Development/Malawi USAID/W United States Agency for International Development/Washington, DC UTA University of Texas Austin UTSA University of Texas San Antonio EXECUTIVE SUMMARY The University of Texas San Antonio (UTSA) was awarded a Cooperative Agreement (CA) to implement a textbooks and learning materials program (TLMP) in Malawi in 2009 to generate supplemental reading materials for Standards (grades) 1-3. The evaluation of this program took place in February/March 2013, and had the following objectives: • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.-based Minority Serving Institutions [MSIs]), in-country institutions (ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. The program UTSA implemented included: • Writing workshops with teachers and others, as they were the original authors of the stories; • Follow-up editing that focused not only on the mechanics and content, but also on how the materials conformed to the curricula developed under the NPC; • Development of teachers’ guides for all volumes; • Illustrator workshops led by the South African illustrator who had done the graphic illustrations; • Capacity building workshops led by the Institute of Economic Development at UTSA (focus was on small business development) for Kriss Offset (the printer) to improve their management practices; • Community mobilization by the Creative Center for Community Mobilization (CRECCOM) to build awareness to the value of reading; • Workshops with lead teachers and primary education advisers (PEAs)on how to use the materials (cascaded to others in their respective clusters, then schools); • Continuous collaboration with the Malawi Institute of Education (MIE), English Curriculum Division (the lead unit working with UTSA) in research, outreach, workshop participation and leadership, etc.; and • Establish e-learning labs at the Lilongwe and Blantyre Teacher Training Colleges (TTC) (through a partnership with INTEL). The educational context in which the Read Malawi (RM) TLMP was implemented included a National Education Sector Plan 2008-2017 that sought to 1) equip students with basic knowledge and skills to enable them to function as competent and productive citizens in a free Evaluation of TLMP – TLMP in Malawi v society; 2) reduce absenteeism, increase enrollment and reduce the dropout rate; 3) improve learning outcomes; and 4) retain girls to complete the primary cycle. UTSA envisioned that the generation of supplementary reading materials would advance this Plan and create a reading culture among children in the early grades. Competing with Read Malawi, however, was a number of other Ministry of Education, Science and Technology (MOEST) or donor-sponsored reading programs and materials that took a more phonics-based approach. RM materials took a whole word/whole language approach and assumed that children could already read. This evaluation identified many stakeholders from whom to collect data. Protocols were developed for each group, and teacher observation forms as well as reading assessment instruments were implemented with teachers and students, respectively. Several limitations, however, prevented a comprehensive evaluation: 1) time allocated for research; 2) floods on roads that necessitated the choice of different schools; 3) a civil service strike that closed all the schools; and 4) the institutional failure of the Malawi Institute of Education (MIE) (the local implementing partner) to provide a full set of the TLMs for team review. Despite these challenges, the team was able to collect significant data on the different components of the program. 1. Impact of TLMP on UTSA – Several institutional changes were made as a result of project involvement: from local to international procurement; making sub-awards; streamlining travel policies to ensure a quick turn-around; creating a global reputation among faculty leading to increased enrollment in education abroad programs; an increased credit rating; and improving the university’s standing as a research university. 2. TLMP Output – A total of 5,260,548 materials were printed and distributed. On average, each of the 1,272 receiving institutions was given 3,700 books. 3. Management and Partnerships – UTSA developed several partnerships to implement RM, all of which contributed to the success of documents for the project. At the head of the list was the partnership with the MIE, the main implementing partner. A para-statal that is the curriculum development arm of the MOEST, the MIE – under the leadership of the English curriculum unit – spearheaded the process with UTSA in providing the writing workshops, and overseeing the development process. In developing the materials, RM had a major problem in leveling them so that they would be accessible to children in Standards (grades) S1-S3. Writers took the materials to demonstration schools and observed children using them; however, no attempt was made to involve teachers in using them and then providing feedback. The training provided to teachers was insufficient, and the methodology – cascading – led to different participants being trained for different durations with teachers themselves receiving a “watered down” version of the training. Evaluation of TLMP – TLMP in Malawi vi A partnership with a community mobilizing organization sought to increase appreciation for reading and conducted a number of innovative activities toward that end. Another partnership was created with UltiNets to install approximately 100 computers at the Lilongwe and Blantyre Teacher Training Colleges (TTC) for teachers to write the stories and for children in the adjacent demonstration school to learn about computers. Beyond this input, TTCs were not properly capacitated by UTSA to train teachers on the use of the TLMs either for pre-service or for in-service continuous professional development. 4. Project Implementation – Malawian teachers, curriculum specialists, artists, desktop designers, editors and printers were all identified by UTSA, the University of Texas, Austin (UTA), and MIE to develop and produce the TLMs. Through a series of writing workshops at two TTCs, authors of books (in-service and pre-service teachers, MOEST officials, TTC Lecturers and MIE curriculum specialists who were fluent in English) learned to write high quality complimentary reading materials that were situated in local contexts and took into account certain cross-cutting themes: gender; HIV/AIDS; and local concerns. These materials were authored in both Chichewa and English. Authors used a combination of their lived experiences and topics of interest to learners as a basis for the books. Teacher’s Guides were developed in a similar manner. Printing of the materials was at first undertaken by a South African printer, but when shipments were shorted, pages printed upside down, etc., the organization hired to repackage the TLMs was hired to do the printing. They not only accomplished this task, but was able to donate approximately $50,000 worth of TLMs to the project. Already a well-resourced printer, Kriss Offset benefited from the TLMP order, but did not expand its business in any appreciable way. Although the materials were welcomed by stakeholders and helped some children to learn how to read, many challenges emerged in how to use them: they were too difficult for both the teachers and the students; there was too much material and too many lessons to cover in the time allotted to teaching the books; a special time was set aside on the timetable for the use of the books (Tuesdays and Thursdays) that had deleterious effects on the ability of teachers to address other subjects and other language programs; teachers mistakenly used the books as texts and so having just 20 copies of a title for use by 120 students was clearly insufficient; 20 copies of all 180+ titles were provided to selected schools, leaving approximately 4,500 schools without any TLMs and; the training provided to teachers was inadequate to change their teaching habits and learn how to incorporate supplementary materials into the curriculum. 5. Outputs and Outcomes –Teacher observations demonstrated – in schools that were not demonstration schools or had not received special training – that teachers were not able to cope with all the requirements presented in the teacher’s guides. Students could not cope either with the long class period during which time they were meant to listen to a story being read by the teacher. Children’s reading ability was non-existent in S1 but increased through S2 to S3. Not surprisingly, in the informal assessments conducted, most of the strides in reading were measurable in Chichewa rather than in English. Evaluation of TLMP – TLMP in Malawi vii 6. Stakeholder Interest – Many organizations became interested in purchasing the TLMs for their own purposes. Among such organizations is UNICEF, Save the Children, Concern Worldwide and World Vision. Each has either placed an order or is considering placing and order with the MIE to purchase considerable numbers of several titles to be used in their own programs. TTCs remain “outside the loop” in terms of receiving and giving training on the use of the TLMs; hence, without pre- and in-service training, it is not clear whether the use of the TLMs is sustainable. 7. Lessons Learned – Foremost is the noted absence of a sustainable teacher training program that would “guarantee” the use of the TLMs in the future. Assumptions made about children’s ability to read and thus enjoy the TLMs were erroneous and defined the entire approach to the development of the TLMs (e.g., use of a whole word/language approach instead of an integrated approach using phonics). Field testing of the books was inadequate; teachers should have been provided the materials for up to three months to try them out and then be requested to explain how they were used and the challenges they faced. Disagreements in the MOEST about the practicability of different approaches to reading led to confusion in the classroom as so many reading programs were being piloted at the same time as RM. 8. Recommendations Materials Improvement • The books, especially S1 and S2, need to be simplified, placing greater emphasis on the alphabet and syllables. The book should be reviewed and revised according to the realities within each grade. • Encourage teachers to split up the long lessons into several shorter classes in which the value of each book can be fully mined before proceeding to the next book. The notion of “flexibility” is critical. • The read aloud books should be shortened and different activities integrated so that children do not get bored and lose interest (one teacher used Total Physical Response (TPR) in reading the stories, i.e., when an action was identified she had the students get up and perform the action, thus providing one more activity to help comprehend the story). • Teacher’s Guides should not be sheets of paper, but should be bound into a booklet so that they can easily be handled and cannot easily be lost. • The allocation of time for each activity noted in the Teacher’s Guides needs to be reconsidered given the number of students in the classroom, the amount of time it takes to perform the activity, and the amount of time it takes to assess what has been learned. Encourage teachers to be creative and allow them to adapt instruction to their specific classroom needs. Systems Improvement • Introduce and teach basic reading skills including phonological awareness and phonics in S1 through the use of the new Ministry Primary Curriculum and Assessment Reform (PCAR) materials, Malawi Teacher Professional Development Support Program/Early Grade Reading Assessment (MTPDS/EGRA) materials, Timawerenga and Tikwere programs, build on these Evaluation of TLMP – TLMP in Malawi viii in S2, and begin using the readers in S3 once children know how to read. Then continue to use the TLMs as designed, as supplementary readers to the PCAR curriculum through S8. • Encourage teachers to learn from each other in using the TLMs by meeting with teachers at neighboring schools (along with Primary Education Advisors [PEAs]) by sharing innovative practices. • Train teachers on how to integrate supplemental materials into their lesson plans. • To increase the number of participating schools, redistribute the materials, perhaps sharing the180 titles among three neighboring schools, rather than all materials being in one school. The schools could then rotate or share the materials on a regular basis. • Provide more guidance on how to manage the books so that teachers can access titles that they need without difficulty. • Train all teachers up through S8 for several reasons: teachers often change grade levels and, thus, need to be prepared for RM should they be placed in S1-S3; and the books can be used in all grades beginning in S3 and ending in S8. • In communities, a greater effort can be made to establish mother and/or father groups to promote a reading culture. Parents could come to school in the afternoons to teach their children to read, and at home parents, older siblings, neighbors or relatives should assist children in reading. Towards a More Comprehensive Approach to Reading Development • The MOEST should be encouraged to follow-up with the development of the new Reading Strategy, which, among other things, would determine the proper role of the current multiple reading programs (or their components) including Read Malawi, Tikwere, MTPDS, and others. Critical to this strategy would be the teaching of phonics in S1 and S2, the use of the PCAR core text for grammar in S1 and S2, and the introduction of the TLMs in S3 and their continued use through S8. • A less cumbersome system for reproducing the RM TLMs to satisfy the needs of NGOs must be developed. Additionally, the readers should be made available in bookshops on a commercial basis since books will be lost or damaged and there is no way to replace them. Also, many individuals stated that parents would be interested in procuring these books for their children. • Strengthen the e-Lab benefits for children by identifying instructional software (there’s much free software available on the Internet) and installing the software on the computers and providing guidance on the use. Linking TLMP to USAID Priorities in Early Grade Reading • The methodology used in creating the TLMs generated many culturally relevant, colorful, interesting, and contextualized stories. The strategy should be considered for producing future materials to enhance early grade reading in different mother tongue languages. Additions should include lessons on phonics/morphology, leveling, extensive piloting by teachers, and an agreement to produce the materials commercially so that other stakeholders can easily purchase the materials for their own use. • In rolling out the new early grade reading project, USAID should work with project leaders to determine how best to use the TLMs already produced in Chichewa and English to make Evaluation of TLMP – TLMP in Malawi ix them suitable as textbooks and/or to add certain sections or build upon other reading materials to strengthen the TLMs and make them more accessible to children and teachers. • The investment made in the production and distribution of the TLMs should not be wasted but should be built upon, taking the lessons learned from this report and adding other innovations to make the TLMs useful in building a culture of reading. In particular, teacher training should be provided on how to blend the phonics/whole language approach to assist children in learning how to read and to enjoy it. Evaluation of TLMP – TLMP in Malawi 1 1. EVALUATION PURPOSE, QUESTION AND OBJECTIVES 1.1 The Background to TLMP The Textbooks and Learning Materials Program (TLMP) was launched by the Africa Bureau of the United States Agency for International Development (USAID) in 2005 in conjunction with the President’s African Education Initiative (AEI). It contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. TLMP was extended for another three years in 2008/09 to 2012. Based upon the provisions noted in their respective Cooperative Agreements (CA), each of the five Minority-Serving Institutions (MSIs) was responsible for managing and implementing the TLMP in a specific country and with achieving specific outputs and results. After having been awarded a CA to work in South Africa in 2005- 2008/09, the University of Texas, San Antonio (UTSA), was awarded a second CA to implement TLMP in Malawi 2009-2012. Each MSI was responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MOE) and other local specialists. The main objectives of the TLMP (during both AEI and the TLMP extension) were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities within SSA; 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program; and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e., gender, health, etc.). Originally a program to develop TLMs for primary schools only, MSIs conducted initial needs assessments and determined that in three countries MOE priorities were for middle and secondary school TLMs. In Malawi, the Ministry of Education, Science and Technology (MOEST) determined that supplemental reading TLMs in Chichewa and English should be developed for standards (grades) 1-3. To implement the project, UTSA (and their fellow MSIs) received about the same level of funding - approximately US$8 million. 1.2 Evaluation Objectives This performance evaluation covered work led by UTSA in Malawi during the period 2009-2012 and was intended to satisfy the following objectives (see APPENDIX A – SCOPE OF WORK): • Validate stated program goals and impacts; • Assess the results achieved for each host partner country in relation to intended program targets measuring quantitative and qualitative impacts of TLMP in terms of local capacity Evaluation of TLMP – TLMP in Malawi 2 building (i.e., U.S.-based MSIs, in-country institutions, ministries of education, etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • Determine if in-country institutions (with support from U.S.-based MSIs) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • Highlight specific program accomplishments per MSI-host country partnership; and • Document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the new USAID Education Strategy. 2. READ MALAWI PROJECT BACKGROUND 2.1 UTSA Responsibilities The TLMP CA implemented by UTSA was called Read Malawi (RM) because it sought to establish a reading culture among the youngest learners in primary schools in both Chichewa (the national language) and English. RM produced books in several categories in accordance with different methods: read aloud, shared reading, and independent reading. TLMs produced for use in the first year were written in Chichewa and those produced for children in the second year in English (translations of many of the stories read in Chichewa in the first year). Building on the teacher-based story writing methodology developed and implemented in South Africa, UTSA included the following program activities in Malawi: • Writing workshops with teachers and others, as they were the original authors of the stories; • Follow-up editing that focused not only on the mechanics and content, but also on how the materials conformed to the curricula developed under the NPC; • Development of teachers’ guides for all volumes; • Illustrator workshops led by the South African illustrator who had done the graphic illustrations for TLMP (“Ithuba”) in South Africa; • Capacity building workshops led by the Institute of Economic Development at UTSA (focus was on small business development) for Kriss Offset (the printer) to improve their management practices; • Community mobilization by the Creative Center for Community Mobilization (CRECCOM) to build awareness to the value of reading; • Workshops with lead teachers and primary education advisers (PEAs) on how to use the materials (cascaded to others in their respective clusters, then schools); • Continuous collaboration with the Malawi Institute of Education (MIE), English Curriculum Division (the lead unit working with UTSA) in research, outreach, workshop participation and leadership, etc.; and • Establish e-learning labs at the Lilongwe and Blantyre Teacher Training Colleges (TTCs) through a partnership with Intel Corporation (Intel). TLMs were also produced in braille through a relationship with Montfort College (for the sight and hearing impaired). Montfort translated selected readers into braille and then distributed Evaluation of TLMP – TLMP in Malawi 3 them to schools that blind students attended, but no training was provided to teachers on how to use them. To enhance teacher training, UTSA created a partnership with the University of California, Berkeley, to establish e-learning centers at the TTCs in Lilongwe and Blantyre. Through another contract with Intel, computers for teachers and students were installed at each TTC. 2.2 MOEST Context and the Educational System The MOEST has aggressively sought to raise its educational standards as a way of improving literacy levels. The Malawi Growth and Development Strategy, the Ministry of Education Guiding Policy Document, and the National Education Sector Plan 2008-2017 (NESP) called for primary schools to 1) equip students with basic knowledge and skills to enable them to function as competent and productive citizens in a free society; 2) reduce absenteeism, increase enrollment and reduce the dropout rate; 3) improve learning outcomes; and 4) retain girls to complete the primary cycle. The NESP launched a sector-wide improvement strategy, including revision of the National Primary Curriculum (NPC). The new curriculum, among other things, focused on reducing the high rates of illiteracy in the early school years. With additional donor support, MIE designed, printed and distributed textbooks in each of the core subject areas for Standards 1 to 8, including Chichewa and English language textbooks. A national effort was undertaken to orient all primary teachers in the more than 5,000 schools to this new outcomes-based education. However, the government recognized the need for additional materials to develop a culture of reading in the primary schools. Different approaches were taken in the development of language textbooks, each of which emphasized good practices developed on the basis of language acquisition and reading theories in other countries and supported by different donors: • SOSA – The name given to Book One, Standard 1, Chichewa under the previous curriculum that focused on phonological approaches to reading development. • PCAR – Primary Curriculum and Assessment Reform – a reform movement in Chichewa reading that has recently moved from the whole word to a more phonics-based approach. • MTPDS/EGRA – Malawi Teacher Professional Development Support (Program)/Early Grade Reading Assessment – a project supporting in-service teacher training with a special emphasis on teaching reading in Chichewa in S1 in seven of the 34 education districts. The approach stresses five core skills including phonological awareness and, for S1 classes, MTPDSP developed scripted lessons and associated readers. The project measured learning gains through EGRA. • Tikwere – A radio program designed to support teachers in the teaching of various concepts, especially mathematics and the teaching literacy, using a phonics-based approach. TLMP was the first project the MIE actually worked on in such depth, although, according to senior MOEST officials, it was considered just another “pilot” activity. The MIE staff members were seconded to manage and participate in the project. Teachers who participated in the writing workshops were paid per diem and travel allowances, but did not have salaries topped Evaluation of TLMP – TLMP in Malawi 4 off by the project. No fees were paid for writing the different stories. A local Program Coordinator was hired to manage the different processes. The MIE worked closely with the Primary Education Advisors (PEAs) who support all education operations of between 10 and 15 schools in their respective zones. PEAs, after initial training provided by MIE and UTSA, cascaded the training to teachers locally, but with each level of cascading, training became “watered down.” In the words of one senior MOEST official: “We need to get rid of at least one level in the cascading process as when the training actually reaches the teachers it is not as effective as the training provided to the master trainers.” Moreover, delivery of the books was not closely monitored initially so there was significant slippage observed in the way books were distributed (e.g., in a cluster of 12 schools, perhaps only nine received them). Once a UTSA manager devised a more foolproof delivery system, the slippage stopped. Once the books were delivered to schools, UTSA lobbied the MOEST for dedicated time in the timetable for teachers to use them. Time was made for their use on Tuesdays and Thursdays, often by extending the class day. (In S1 and S2, classes are only held from 7:30-10:30.) This change edged out other reading programs already in schools, including the use of the English core text (which was recently updated to include more decoding and comprehension skills) and acted as a disincentive for teachers to integrate the TLMs with other materials. This dedicated time was allowed as part of the pilot orientation because teachers were not used to integrating “supplemental” materials into their lesson plans. Devoid of such materials, teachers largely followed the core text. Hence, training was needed on how to incorporate and use other materials to expand children’s ability to read. In other words, according to one MOEST official, the RM materials were seen as “add-ons” that made more work for teachers. Without training on the use of supplemental materials, teachers either followed the teachers’ guides to the letter or did not use them at all. 3. EVALUATION QUESTIONS, METHODS AND LIMITATIONS 3.1 Research Conducted at UTSA The overall evaluation research design called for research to be conducted at the MSIs to determine 1) the background to the project and how it was implemented, and 2) how participation in a USAID-funded program built the capacity of MSIs to lead future projects. The interviews conducted at UTSA were based on several sets of questions posed of leaders, project managers and participants (see APPENDIX B1 – QUESTIONS POSED IN VARIOUS INTERVIEWS AT UTSA). 3.2 Research Activities in Malawi (Lilongwe, Zomba and Blantyre Environs) With the background information provided by UTSA, the Team Leader (TL) developed a background paper/research design that included different categories of questions to be posed to different stakeholders (see APPENDIX B2 – DATA COLLECTION INSTRUMENTS, MALAWI STAKEHOLDERS). The range of stakeholders include MOEST and MIE officials, writers, printers, school directors, teachers, children, community mobilizers, and IT people who worked with the TTCs in establishing and maintaining the computer centers at the two TTCs in Lilongwe and Blantyre. Evaluation of TLMP – TLMP in Malawi 5 In Malawi, the team consisted of two American and two Malawian experts in education. The team conducted a number of interviews in Lilongwe with USAID/Malawi (USAID/M), MOEST officials, and various stakeholders, and then moved on to Zomba and Blantyre to collect data at MIE, schools and from other stakeholders (see APPENDIX C – SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES). In Lilongwe, the team piloted the data collection tools at the Lilongwe TTC and its respective demonstration school. Adjustments were made to the instruments prior to the team’s departure for Zomba, the central locus of project implementation (where the MIE is located). The RM Project Coordinator was hired to facilitate appointments to be made with officials and schools. Unfortunately, the schools chosen in Zomba were “high performing schools,” those that had received additional attention and were well resourced, and were those that had also benefitted from other projects (a concern which USAID/M cautioned us against). Hence, the results achieved for Zomba are not representative of what was achieved elsewhere. In Blantyre the team chose its own schools, following our own parameters and those of USAID/M, and was able to obtain information believed to be more generalizable to the broader school population. The Education Officer responsible for the project at USAID/M accompanied the team during the Blantyre data collection and was able to observe results for himself and corroborated other research conducted in Mission-based programs. At each school we interviewed the director and teachers who were using the TLMs and observed how teachers used the TLMs in the classroom. We also administered an informal reading assessment using both Chichewa and English TLMs that called for children to name certain letters, read certain sight words, and listen to a passage and answer questions in English about the passage. Assessment materials were taken directly from the grade-specific materials of the children tested, and with which the children ought to have been very familiar. The materials were from standards (grades) 1, 2 and 3 TLMs. When the PEAs were located at a school, we also interviewed them; otherwise we went to their offices. 3.3 Limitations of the Study Overall, there was insufficient time to conduct a full evaluation that was geographically inclusive and that included schools chosen on the basis of a random sample. We also did not have time to collect data on schools that had not received the books to determine what difference having the books made to the children’s ability to read. When the two American evaluators arrived, Malawi was experiencing heavy rains that interfered with our movements to schools. The road between Zomba and Blantyre (under construction) was flooded necessitating our making other plans (two days before moving to Blantyre, the road was re-opened). Two days after launching the research in Lilongwe, the Civil Service went on strike. The meetings we had scheduled with MOEST officials had to be rearranged as they were told not to come to the office. Two officials came to our hotel to be interviewed, while two others refused to do so. Another was addressing Parliament on the salary needs of teachers and so did not meet with us until we returned to Lilongwe from Blantyre. With teachers on strike, Evaluation of TLMP – TLMP in Malawi 6 data collection at several schools was inhibited and we could not conduct the research as extensively as planned. The full set of TLMs produced was not available at any time for team review. At MIE we were told that the books had all been distributed and none were available – a “fact” we found very strange since MIE was the implementing partner, held the copyright, and was responsible for going out on bid to reproduce any of the materials. 4. FINDINGS AND CONCLUSIONS We begin this section of the report with the data collected at UTSA in Texas so that the reader can understand how an MSI dealt with the challenges of being awarded a USAID CA to conduct work in an African country for the very first time. We then move on to the print output of the project so that the reader is aware of the number of items produced and to whom they were distributed. Findings and conclusions on the implementation of the project are then presented as are the results of teacher observations and student reading assessments. 4.1 UTSA Achievements and Challenges Although globalization is a primary focus for the 40-year-old university, some of its systems and procedures were not in sync to take on international development projects. Significant changes took place in Business Affairs and in Grants & Contracts: templates were redesigned for routing so as to reduce approval time frames, forms were generated to increase efficiency, travel disbursements were managed differently so as to allow for international/developing country expenditures and included per diem (no credit cards), etc. 4.1.1 Administrative Adjustments: Through implementation of the TLMP, the following changes were made: • Procurement (under VP) – An international bidding system was put in place to let out contracts for supplies in Malawi that emanated in Malawi, South Africa, or elsewhere. The university had to overcome the challenge of a legislative mandate to take bids from the historically underutilized businesses (HUB) roster in Texas. The ceiling for procurement was also lifted and now can go up to $1 million before needing board approval. • Consultants vs. Sub-awards – Initially, individuals from other institutions were to have been awarded consulting contracts by UTSA; with further inquiry, however, the Office of Research determined that sub-awards should be offered to the University of Texas, Austin (UTA) – the institution at which several TLMP consultants were based. This different approach triggered cost-sharing by several institutions so that UTSA did not have to provide total costs. • Changes in Per Diem and Travel Advance Practices – Payment of per diem, in accordance with USAID parameters, was instituted for UTSA faculty as well as MIE and other partner staff (when they came to the US). Faculty and staff also had to obtain travel advances as not many hotels/restaurants in Malawi would take credit cards. A system in compliance with IRS regulations for paying international visitors was also developed. Evaluation of TLMP – TLMP in Malawi 7 • Travel Policies – Because travel had to be arranged quickly at times, the approval process was streamlined and new operational guidelines developed. 4.1.2 Beneficial Impacts on UTSA and its Community - Participation in TLMP, both through Ithuba in South Africa and through Read Malawi, had the following beneficial impacts on UTSA: • Community Impacts – Faculty who have participated in TLMP now share what they have learned in their classes in which students are largely teachers themselves. Faculty reputations have been enhanced by their international participation, and now more students are signing up for education abroad. • Increased Credit Score – With the inflow of grant funding, the credit rating of UTSA advanced to AAA, thus allowing them to borrow money to build more dormitories and classrooms. Ten years ago, international revenue was $200 million; it is now $450 million, 50% of UTSA income. • Institutional Internationalization – UTSA has gone on to sign a Memorandum of Understanding (MOU) with China for students to study at UTSA, and to develop projects in Peru, Honduras, Mozambique, and Pakistan. The Institute for Economic Development is expanding its outreach beyond Mexico and South America and is looking forward to taking advantage of other small business development funding opportunities in other countries. Many departments, centers, and institutes are now exploring global initiatives. The Center for Vaccines has developed a vaccine against chlamydia and would like to explore how it could be used in developing countries. Collaborating schools have developed a cell phone solar charger and would like to explore how this could be used in developing countries. • International Reputation – UTSA was designated number 53 out of 100 top research universities under 50 years old. 4.2 TLMP Output Initially, in 2009, the project had a $13 million budget, but this was cut to $8,859,624 (USAID funding of $8,150,528 plus institutional cost-share) because of USAID overall budget cuts. With this level of funding, RM generated 183 titles (120 in Chichewa and 60 in English, two alphabet books and an overview guide) that were developed by teachers and other educators, illustrated, printed and distributed to 1,272 educational centers (945 primary schools, (three per zone) out of 5,300 primary schools, 6 TTCs, and 350 Teacher Development Centers (TDCs) at which PEAs were located). In addition, 90 Teacher’s Guides and training manuals were produced, all in accordance with NPC guidelines. The following is a summary of what was produced and distributed: • Read Aloud Books: Chichewa, 1,200; English, 600 • Guided Reading Books: Chichewa, 1,200; English 600 • Alphabet Books: Chichewa 120; English 80 • Overview Guide: English 6 • Teacher’s Guides: Chichewa 180; English 90 • Big Books: Chichewa 20; English 10 • Alphabet Posters: Chichewa 4; English 4 Evaluation of TLMP – TLMP in Malawi 8 • How to Care for Books Poster: English 4 • Water Sources Poster: English 4 A total of 5,260,548 materials were printed and distributed. Overall, each receiving institution was given 3,700 books. The average cost of production/unit in each category was: • Teachers’ Guides – Average cost of $0.18/unit (printer 1 $0.16/unit; printer 2 $0.21/unit) • Learners’ Books – Average cost $0.33/unit (printer 1 $0.31/unit; printer 2 $0.35/unit) Unit costs do not include delivery as this occurred across units and cannot be divided between the two. Costs also do not include transportation, custom fees and school delivery of books initially produced in South Africa and sent to Malawi. 4.3 Project Management and Partnerships As RM was a centrally-funded project, USAID/M assigned one person to maintain liaison with RM and to exercise some oversight over its activities. The relationship between UTSA and USAID/M was cordial and it appears that many joint decisions were made on how to proceed. Early in the project, however, the head of the Mission’s Education section was transferred to Ghana, and she was not replaced for approximately a year, leaving responsibility for RM in the hands of an assigned officer. When RM was launched, officials at the MOEST were skeptical about the ability of teachers and other educators to create stories as had been done in South Africa. Owing to the major reform movements taking place in Malawian education at the time, however, the MOEST decided to accept the program outlined by UTSA. That Malawians became authors of the supplemental readers became a source of pride not only for the MOEST but for the authors themselves. The MOEST delegated the task of partnering with UTSA in the development and implementation of RM to the MIE. The English curriculum unit was designated as the lead department, even though readers would be developed in both Chichewa and English. While several textbooks were being written (with other donor support) that focused on phonics, the focus of the supplementary readers was on whole word or whole language. This approach assumed that children beginning in Standard 1 were already able to read. Despite the expert input from several institutions, the ability of the project – according to an MOEST senior official – to accurately level the materials (that is, to assure that the level of the materials was appropriate for the learners’ age and knowledge base) was “missed.” Typical Bantu language morphology consists of syllables comprised of a vowel by itself at the beginning of a word or of a consonant followed by a vowel. However, this morphology was not characteristically followed in producing the books because the pedagogical approach did not reflect the use of phonics. In fact, frequently very long words and names were used that children could not decode regardless of how long or how hard they tried to figure out what Evaluation of TLMP – TLMP in Malawi 9 was written. In schools where other donor materials based on phonics were being used, alongside RM readers, children were better able to decode. Comprehension, however, remained a problem. For sustainability purposes, we inquired into how TTCs dealt with the RM materials. At the Lilongwe TTC, English language faculty members stated that a maximum of only one hour of instruction (out of a year of classroom instruction and a year of a practicum) was dedicated to the RM materials under the Language and Literacy Syllabus. At the Blantyre TTC, RM books are dealt with under the three hours allocated for the year under the curriculum topic “Dealing with Supplementary Readers.” Although teacher training on the use of the materials was not included in TLMP CAs, UTSA determined that such training was necessary. Selected educators were trained, including PEAs, who were then charged to cascade the model to teachers in the schools for which they were responsible. The model was watered down, and teachers trained by PEAs and others did not benefit in the same way as those trained originally by RM staff. PEAs did provide RM support in their respective zones. In one zone, however, the PEA had issued a directive to stop using the RM materials as “the project is over.” This decision appears to have been made unilaterally and affected only one zone. The hundreds of books that had been delivered to the three schools in that zone remained in unopened boxes in locked storage rooms. While a Program Advisory Committee (PAC) was established at the outset of the project, it was not called into session to make any major decisions or to provide any specific input. A partnership created to assist RM in creating a reading culture was with CRECCOM, an organization that sensitized the community on how to care for books and the pedagogy of the RM instructional model. Community based activities within the CRECCOM model included: theatre for development; awareness/sensitization meetings; mother groups (reading groups led by mothers); volunteer assistants; role modeling; and school incentive packages, to name a few. The shortcoming of CRECCOM’s participation was that it was in only one community per district and the organization’s ability to reach out further was limited by funding. A second partnership that was created was with UltiNets under the sub-project name of Intel Teach to establish e-Learning Labs at Lilongwe and Blantyre TTCs. Under this initiative, UTSA entered into an agreement with UC Berkeley and INTEL to establish computer labs (with 96 computers each, 48 for use by faculty and 48 for use by children at the demonstration school). The computers were used by teachers writing the RM stories. Beyond this initial use, it was unclear how RM used this resource. Additionally, there was no instructional software (except for one math program – Easy Learning) and no instructional syllabus available to teach teachers and students how to use the computers. A third partnership was created with Montfort College for the visually and hearing impaired. Under this agreement, Montfort transliterated 60 of the stories into braille, and then Evaluation of TLMP – TLMP in Malawi 10 distributed them to 30 special education centers. However, no orientation or training was provided to teachers or administrators, leaving them unclear on how to use the readers. A final partnership was created for the printing of the TLMs with Kriss Offset, an impressive family-owned printing company located in Blantyre. Printing of the first run of materials (developed prior to the launching of RM and not in accordance with RM processes) was undertaken in South Africa with a printer that was used during the implementation of UTSA’s TLMP in that country. When the order was received by Kriss Offset to repack for internal distribution, it was found to be short, pages were printed upside down, quality control measures were not taken, and importation of the materials was costly. Thereafter, UTSA entered into an agreement with Kris Offset to print the materials in Malawi. They did so successfully and economically. However, UTSA made some promises to Kris Offset about the number of books to be produced that never materialized owing to USAID budget cuts. Kriss Offset invested in building a warehouse to store books before they could be shipped to schools by the MOEST. The order was subsequently cut from 4,500 schools to 1,272 learning institutions. Kriss Offset also believed that other orders would be forthcoming, but the cumbersome bidding process required by MIE prevented Kriss Offset from winning any subsequent bids. Hence, Kriss Offset, while benefiting from the order they did receive, did not expand their operations to address this particular order. In conclusion, the management structure and partnerships created enabled UTSA to deliver most of what it promised. The major gap in all areas, however, was the training needed on the use of all materials and the integration of the partnerships with other administrative activities. 4.4 Project Implementation Because the MIE had not been a part of the early decision-making concerning the approach and operations of RM, UTSA acted on a number of assumptions about education in Malawi that were not valid: UTSA assumed that children acquired reading skills in the first grade and on this basis a “whole language” approach was used in developing the TLMs; there was no Malawian expert in early childhood education and early literacy on the project; the faculties of the TTCs were not consulted in terms of the type of training teachers were provided; and very little, if any, time was spent in the classroom to determine the challenges teachers faced nor were teachers asked how they cope. Once the materials were developed, writers themselves were asked to “field test” the TLMs on the basis of a check sheet rather than through actual use by teachers over time. Hence, teacher feedback was minimal. 4.4.1 Materials Development Guided Reading and Read Aloud Books Malawian teachers, curriculum specialists, artists, desktop designers, editors and printers were all identified by UTSA, UTA and MIE to develop and produce the TLMs. Through a series of writing workshops at two TTCs, authors of books (who were in-service and pre-service teachers, MOEST officials, TTC Lecturers and MIE curriculum specialists who were fluent in English) learned to write high quality complimentary reading materials that were situated in local contexts and took into account certain cross-cutting themes: gender; HIV/AIDS; and local concerns. These materials were authored in both Chichewa and English. Authors used a Evaluation of TLMP – TLMP in Malawi 11 combination of their lived experiences and topics of interest to learners as a basis for the books. The educators writing the stories participated in three workshops. In the first workshop, they learned to document and describe experiential events of interest to children. The books were authored in Chichewa and translated into English for further development; authors revised their Chichewa versions recursively. During the second workshop, authors continued to revise their books; they were encouraged to keep their audience in mind through crafting mini-lessons. Included in these mini-lessons were the topics of content, word choice and imaging decisions. In the third workshop, the stories were field-tested with learners in classrooms. The field testing was problematic as 1) there were no specific guidelines and authors merely observed what the children could manage, and 2) much of the field testing was conducted at TTC demonstration schools where children and teachers receive more attention. Leveling then became a more definitive problem. Such items as word length and difficulty, sentence structure and length, repeated phrases, picture support, etc., were partially addressed, especially in the guided reading books; but the read aloud books included many challenges for both teachers and students. The illustrations for the materials were developed and designed by local artists either working for the Nation Newspaper or employed by the MIE. Typically, each book has between seven and ten images. All images were painted by hand (it was deemed by UTSA that the color quality from computer-generated images was too poor). Illustrators reported that they had to do their work in a very short time frame (generally no more than three weeks) when the UTSA Project Director was in-country, and felt that this was not their best work. They were paid on the basis of their productivity, although some illustrations were rejected at the last minute, meaning that they were not fully paid for their output. Teacher’s Guides The Teacher’s Guides, developed in both Chichewa and English, were linked directly to the national curriculum and provided teachers with ways of developing literacy and subject area skills. Those written for English instruction supported second language acquisition practices and the teaching of reading strategies, including word identification/knowledge, fluency strategies and comprehension strategies. The Teacher’s Guides used a daily, patterned step-by￾step approach that included (on the guided reading day) a preview of difficult words, guided writing, shared and guided reading, word wall time, and life skills inquiry/home connections, each with a stated amount of time to accomplish each task (for a total of up to105 minutes, depending on the grade level). On the “read aloud” day, time was allotted to reporting/ gathering time, read aloud, response activity, independent reading time, individual assessment and support, and promoting a reading culture (for a total of 105 minutes). Because each lesson is so constructed, once teachers grow accustomed to the patterns they can implement them in ways that are helpful to their learners. However, teachers reported that the time allotted to each activity was insufficient. At well over an hour the full lessons were too long, and the children lost interest because they also had to share 20 books among as many as 120 children in a classroom. Moreover, the transition time between activities took time away from the substance of the activity, leaving the lesson Evaluation of TLMP – TLMP in Malawi 12 rushed and/or incomplete. The time allotted was inadequate for the number of words that had to be previewed and the types of activities that had to be accomplished, but too long for the comfort and attention level of the learners. The Overview Guide includes six key components or “big ideas”: 1) The teacher uses a variety of tools to scaffold understanding between oral and written language; 2) In Guided Reading, the teacher gradually releases responsibility to learners; 3) The teacher uses the literate environment to scaffold connections between home/school and life skills subject area; 4) The teacher uses the Read Aloud books as an instructional practice to purposefully scaffold comprehension of text; 5) The teacher provides learners with opportunities to respond to the Read Aloud and Guided Reading books in creative ways; and 6) The teacher promotes independent reading and a culture of reading. While these big ideas were useful in guiding the development of the TLMs and Teacher’s Guides for each lesson, teachers were presented with many challenges in implementing them. 4.4.2 Comments on the TLMs Made by Teachers, Directors and PEAs - During the two-week field visit, the evaluation team visited 13 schools in six districts, interviewed 40 head teachers/management staff, 13 groups of teachers totaling over 50 teachers, and eight PEAs (who provided additional information on three other schools). The team also met with about 100 learners as part of the reading assessment. We had anticipated providing separate findings for each of the stakeholders noted; however, the responses from each of these groups were so similar that we consolidated them, noting differences when reported. Responses to our interview questions on the materials and their use were both positive and negative. Positive Responses Children. The largest number of positive responses (12) indicated that the materials helped some children to read, learn new words, and construct sentences. The materials motivated learners (8) because the books were attractive with colored pictures (5), and had interesting stories and meaningful (culturally-based) pictures (5). A smaller number of responses included an improvement in the reading culture, using pictures to help learners predict stories, and the opportunity for children to take books home where they could practice reading with their parents or others. Other responses included children learning about their past from stories (when books were taken home and discussed, parents and grandparents recalled hearing similar stories and discussed them with the children), the ability to tell stories, developing good listening skills, and developing skills to think critically. Teachers. Teachers reported that the training was good, that they gained knowledge on how to teach reading, and boosted their interest in stories and reading. The lesson plans, charts and books helped teachers to be more effective. Evaluation of TLMP – TLMP in Malawi 13 Parents. Children and parents in certain areas borrowed books from the local school to take home, which helped to develop a reading culture, gave children an opportunity to practice reading at home, and increased communication between parents and teachers. Negative Responses Books Too Difficult. Stakeholders reported that the books were too difficult, especially for Standard 1 (S1, equivalent to first grade) and Standard 2 (S2, equivalent to second grade). Books for S1 had long sentences and difficult vocabulary rather than letters, syllables, and simple words and phrases. Teachers pointed out that the children were not able to read without an introduction to the alphabet and syllable construction, and that the whole word approach was not effective for beginners. They reported that children were interested in the pictures, but not in reading, and often just tried to memorize what the teacher was reading from the books. The Read Aloud books tended to be too long and difficult and the children lost interest. Class Periods Too Long/Too much Material. Class periods ranged from approximately 90 minutes in S1 to up to 120 minutes in S2 and S3. One teacher noted that the extended day actually increased absenteeism. There was just too much material and children became bored during the long lessons. This was also stressful for teachers. Several teachers noted that it was “impossible to teach one book in one day,” which was the expectation set by RM. Under this system, the teachers and children rushed through a book, typically, with the children learning very little and, in most cases, unable to read any of the text. As one teacher said, it seems like the program is “teaching titles and not reading.” The instructions for teachers resulted in one book being used for one day, and then never seen again in that class. Thus, there is a situation in which the lessons are too long, yet there is more material than can be taught even in this long period. Timetable Issues. RM lessons were scheduled on the timetable for Tuesdays and Thursdays (without MOEST approval, according to a senior official) and created conflicts with other subjects. The programs “collide, overload, and confuse students.” Also, many teachers were struggling with how to deal with TLMs, especially when the lessons take up so much time and the teachers also have to attend to other reading and English language programs, in addition to the official PCAR curriculum - the radio Tikwere programs - and, in some districts, the MTPDS/EGRA lessons. Also, one teacher noted that these three programs were not examinable; only subjects that are officially on the timetable are included in the exams. Too Few Copies of Each Title. An overarching misconception of the TLMs on the part of stakeholders was that they were textbooks and not supplementary materials. Teachers were not used to integrating supplementary materials in their lesson plans (nor did they have training on this in their year-long TTC program). Hence, teachers believed that the TLMs were their textbooks. With that perception, teachers reported that 20 copies of a given title were too few for classes with up to 120 students. Too Many Books. Paradoxically, while each school may have too few copies of each title, only 20, at the same time they had too many titles to use. The teachers were not sufficiently knowledgeable about how to integrate this range of materials into their daily lesson plans. While teachers struggled with the limited number of books for a given title, in the storeroom Evaluation of TLMP – TLMP in Malawi 14 of some schools, there were boxes of books that had never been opened. Also, while most schools have encouraged children to take books home, some of the schools have limited this practice because of books becoming lost. And, since the books are not available commercially or from the MOEST/MIE, there is no way to replace them. Training on the Use of the Books. Several teachers noted that the training was too short, and that the few teachers who attended the training did not always train their fellow teachers at the school level. Also the PEAs who were responsible for the training, were overloaded with too many responsibilities. There was limited school-based training, and some teachers felt that RM was imposed on them. Many teachers commented that the allowance money provided during the training was inadequate. 4.2.3 Conclusions: The number and type of books created by RM is commendable, especially in light of the methodology used to create them. The stories were written in Malawi and, thus, are relevant and of interest to children and teachers. The books are full of attractive pictures, with one illustration for each page of text in the guided readers. They are virtually the only additional print materials available to teachers and children. This is a major accomplishment. The children love to hold the books and enjoy the pictures, which serve as a jumping-off point to understanding the stories. The stories encourage children’s prediction and creative thinking skills. And the teachers report the program is helping to develop reading skills and a reading culture among both the children and parents. However, the full benefits of the program are constrained by several factors. First, the language of the books, especially in S1 and S2, is far too advanced for the children. In S1, for example, rather than starting with letters, syllables, and simple words, the books begin with full paragraphs, sentences and big words. There are claims that RM helps children learn to read, but the TLMs are not books for beginners. Once children have developed basic reading skills, these books can make a major contribution to develop further reading skills and create a reading culture. The TLMs produced by RM are not treated as “supplementary” but rather as a separate subject because they appear on the timetable every Tuesday and Thursday. The time allocated is very long for the early grades and children lose interest. Moreover, these long periods cut into and eliminate other subjects that are normally part of the curriculum. Another problem with the implementation of RM is that teachers are instructed to complete one book during a single class period, a virtual impossibility if the children are to get any significant reading skills out of the lesson. And the long lessons have forced many schools to extend the school day, which is not yet official policy. And while there are a large number of books, there are only 20 copies of each title, making RM difficult to use in even modest sized classrooms. There are too many titles for any one school, as evidenced by boxes of books remaining unopened, yet too few copies of each title to be used effectively in the classes. In summary, although RM was highly successful in developing a large number of attractive books, the books were not leveled properly, especially for S1 and S2, and the implementation Evaluation of TLMP – TLMP in Malawi 15 framework and expectations have created problems in the schools and classrooms, which have limited the effectiveness of the program and created problems in relation to other subjects on the timetable that are to be taught. 4.5 Outputs and Outcomes 4.5.1 Teacher Observations: One element of the evaluation was observing teachers using the TLMs in their classrooms. Because of the civil service strike, we were unable to do this at each school and, in some cases, a lesson was “staged” because we came on a day other than Tuesday or Thursday. With these caveats, we nevertheless present a summary of the classroom observations. Of the teachers observed, six were teaching at the Domasi Government School (part of the MIE “lab” approach); four at the Domasi Demonstration school, a school that also got additional attention; four were at Mponda “High Implementing School” in Zomba town; three in rural Blantyre at PIM Primary School, and two at Mafe School. The average number of learners in the classes observed was 72 and the mother tongue was largely Chichewa, with a few in the class speaking Yao. Four of the teachers were males and fifteen were females. The observation form identified items to be checked that were “observed” or “not observed.” The highest number of “observed” checks was achieved by teachers at Domasi Demonstration School, and the lowest was PIM Primary School. In only one instance (for standard/grade 3) was writing used to reinforce what was learned, and was continuous assessment of homework assignments in evidence (at Domasi Demo). In only six observations did the teacher explain the goals and purpose of the class, identify difficult words before undertaking the reading lesson, and explain the lesson in English (in other instances it was explained in Chichewa). In 50 percent of the lessons observed, children could not understand the English being used. In seven cases, the teacher did not write the lesson objectives on the board; and in at least five cases teachers did not demonstrate personal mastery of English. While we cannot draw strong conclusions based on this limited sample, we do take these findings as indicative of the ability of teachers to provide instruction in English, follow the teacher’s guides appropriately, and to fit the entire lesson into the time allotted. 4.5.2 Learner Reading Assessments and Outcomes: The range of reading/language development programs in each school made it difficult for us to attribute reading ability to the use of the TLMs. Nevertheless, we endeavored to ascertain children’s ability to decode and read the TLMs that they had recently focused on in their classes. The purpose of this assessment was not to carry out a rigorous impact evaluation, but to try to determine what the outcomes were of the lessons taught in the classroom. The results of these assessments need to be interpreted with caution for several reasons: • Schools were not selected at random. Because of the limited time for field activities, the evaluation team selected schools largely for their accessibility during periods of flood and teacher strikes. • Students were not always selected at random within a class. As a result of the strike, students who happened to be at the school in the appropriate grades were selected for the Evaluation of TLMP – TLMP in Malawi 16 assessment. In other cases, where school was in session, teachers chose the students to participate, and often this resulted in the “best” students being chosen. • To minimize possible influence on reading scores as a result of other reading programs in a school, the team tried to choose districts and zones that did not have other programs. The one exception was Zomba Rural that was also using MTPDS/EGRA. Although the test scores would be influenced by MTPDS as well as RM, it was helpful to talk to teachers about the relative advantages of each program. • There are no comparison scores. The evaluation team did not have baseline scores or the results of comparable schools that did not participate in the RM program For a presentation of schools selected, a description of the instruments, and how the reading assessments were undertaken, see APPENDIX D – READING ASSESSMENT METHODOLOGY. Basic Reading The results show a steady improvement in reading scores at each level. For example, in Chichewa, virtually no one could read in S1, by S2 about one-third of the children were able to read, and by S3 the children who could read increased to 45 percent . Regarding letter recognition, the S1 children could identify some letters – 42percent could identify M and 26 percent could identify S. By S3, the letter recognition scores increased to 85 percent and 90 percent respectively. As expected, word recognition also increased substantially from 21 percent in S1 to 67 percent in S 3. While these results show a significant improvement in reading scores from S1 to S3, there is no indication of RM’s influence in this improvement because there is no baseline or control group. Nevertheless, it is clear from these results, that many children are learning basic reading skills in the early primary grades. Yet, even by the second term in S3, only 45 percent could read fluently and 21 percent could not read at all. A summary of the findings by Standard/grade level is as follows: Standard Read Chichewa Fluently Can’t Read/ Identify Letters in Chichewa Read English Fluently Can’t Read/ Identify Letters in English 1 4% 96% 2 31% 59% 6% 88% 3 45% 21% 32% 47% Comprehension After reading the passages, the evaluation team members determined whether the child had enough reading ability to take the comprehension test. Those results are as follows: Evaluation of TLMP – TLMP in Malawi 17 Language and Grade Level Number of Children Percentage Able to Comprehend Passage Chichewa 2 2 out of 2 100% Chichewa 3 11 out of 26 82% English 3 4 out of 22 25% The few children who took the comprehension test did well in Chichewa, but not well in English. Gender When the test results were disaggregated by sex, in general, the boys scored higher than girls. When all boys and all girls across both languages and the three grade levels are calculated, 35 percent of the boys and 26 percent of the girls can read. Conclusions The results of the reading assessments show a steady improvement in reading skills across the three grades. Whereas there was virtually no ability to read in S1, by S3 45 percent of the children could read fluently in Chichewa and 32 percent in English. Few of the children had enough reading skills to take the comprehension test, but those who did take the test in Chichewa scored well. Finally, with the exception of S2 in Chichewa, boys significantly outperformed the girls on the reading tests. Nevertheless, these reading scores need to be treated with caution because of the lack of a rigorous procedure for selecting schools and children, and the lack of any comparison data. It is impossible to tell to what extent RM may have contributed to these gains in reading scores. Particularly given the poor leveling process for the books in S1 and S2 and the complete lack of any introduction of basic skills in recognizing letters, syllables, and words, it seems unlikely that RM would have contributed significantly to the development of initial reading skills. However, once those skills had been obtained through the use of other teaching methodologies, the TLMs could have contributed to developing further reading skills. 4.6 Stakeholder Interest in and Use of Materials/Sustainability 4.6.1 NGO Reading Programs: The development and publication of so many supplementary readers came to the attention of UNICEF and other NGOs implementing reading programs in Malawi, i.e., Save the Children, Concern Worldwide, and World Vision. UNICEF - Will purchase 30 sets of the RM books for libraries in district schools by the end of the project in 2016. By that time UNICEF will have stocked 300 libraries with RM materials. Save the Children - Has placed an order for 10 titles in English – 35,000 books – and plan to purchase the same in Chichewa to support their Literacy Boost program in Zomba that Evaluation of TLMP – TLMP in Malawi 18 includes Reading Camps with links to 40 primary schools in four zones. It is envisioned that 350 camps will be established with each school having 5-10 camps to support reading activities Concern Worldwide – Has placed an order for approximately 133,000 readers, teacher’s guides and TLMs in Chichewa and English for use in schools included in their field of operation. World Vision – Is considering purchasing an undetermined number of books for distribution in its Area Development Programs (ADP) throughout the country. A stumbling block to the fulfillment of these orders is the cumbersome bidding process that MIE, the owner of the copyright, must go through. Several of the NGOs consulted cited the long lead time needed for bids to go out, a winner to be chosen, and production to take place. While the MIE is looking into ways to cut down this time-consuming process, it has not achieved success, thus frustrating the NGOs who wish to use the materials. 4.6.2 TTCs: There are no current plans to include training on RM TLMs at TTCs. It is possible that when the Director of Inspectorate and Advisory Services (DIAS) completes his strategy paper on reading, literacy, and language instruction, a more expansive teacher training element will be included. There was no appropriate pre- and in-service teacher training on the use of supplementary reading materials. It is likely that the books not yet opened and stored in secure closets at receiving schools will not see the light of day. Teachers find them too much of a challenge to use in the way prescribed, and in self-contained classrooms take time away from studying other subjects. With regard to using the TLMs at TTCs, when it became clear that the TTCs needed assistance in developing Continuous Professional Development (CPD) modules, UTSA promised these would be provided. They were never developed. Hence, the TTCs are not prepared to provide training on the use of the TLMs to either pre-service or in-service teachers. It remains to be seen what the new strategy will be and how both core textbooks and supplementary readers will be integrated. 5. LESSONS LEARNED • In designing and implementing a textbooks and learning materials development program, teacher training on how to use the new materials must be included at both the pre- and in￾service levels so that teachers can become familiar with the materials, have an opportunity to adapt them to large class sizes, and to internalize their use, and for the textbooks and learning materials to be used over the long term. • When piloting newly-developed textbooks and learning materials, teachers should be given the textbooks to use in the classroom for at least a month (better, for three months) and then research should be undertaken to determine challenges teachers faced and/or overcame in using them so that they can be revised before final printing. • Before launching a TLM production project, thorough research needs to be undertaken into the educational system on how teachers are trained (and for how long), the frequency and substance of in-service teacher training programs, and classroom practices that foster the inclusion of materials other than the core textbook into lesson plans. Evaluation of TLMP – TLMP in Malawi 19 • Before launching a TLM production project, thorough research needs to be undertaken on the textbooks used prior to the project, the learning outcomes and results achieved in using those books, and the level of reading ability children have under past practices so as to avoid making assumptions about what children are able to read. • Launching an innovative TLM production project works best when the MOEST, MIE, and PEAs are working together in an overall learning improvement program that is not competing with other programs to create results. • “Leveling” any reading materials is absolutely essential to creating positive learning outcomes among children. • Having a management entity at MIE improved management, administrative and financial practices. In so doing, MIE not only had technical expertise in the subject matter, but also had management expertise, which led them to applying for grants on their own. • RM works well with learners that have already developed skills in identifying vowels and consonants, and are able to form basic syllables and simple words. • Delivery of TLMs worked best once a foolproof tracking method was developed. • For a U.S.-based university to work in an African country successfully, staff need to be culturally oriented not only to the country but also to the systems and procedures involved in working in that country (e.g., rolling blackouts preventing long-distance communications). 6. RECOMMENDATIONS While RM has experienced a number of successes, recommendations made below point to areas that need improvement: 6.1 Materials Improvement • The books, especially S1 and S2, need to be simplified, placing greater emphasis on the alphabet and syllables. The books should be reviewed and revised according to the realities within each grade. • Teachers should be encouraged to split the long lessons into several, shorter classes in which the value of each book can be fully mined before proceeding to the next book. The notion of “flexibility” is critical. • The read aloud books should be shortened and different activities integrated so that children do not get bored and lose interest (one teacher used Total Physical Response (TPR) in reading the stories, i.e., when an action was identified she had the students get up and perform the action, thus providing one more activity to help comprehend the story). • Teacher’s Guides should not be loose sheets of paper, but should be bound into a booklet so that they can easily be handled and cannot easily be lost. • The allocation of time for each activity noted in the Teacher’s Guides needs to be reconsidered, given the number of students in the classroom, the amount of time it takes to perform the activity, and the amount of time it takes to assess what has been learned. Encourage teachers to be creative and allow them to adapt instruction to their specific classroom needs. Evaluation of TLMP – TLMP in Malawi 20 6.2 Systems Improvement • Introduce and teach basic reading skills including phonological awareness and phonics in S1 through the use of the new Ministry PCAR materials, MTPDS/EGRA materials, and Tikwere programs, build on these in S2, and begin using the readers in S3 once children know how to read. Then continue to use the TLMs as designed, as supplementary readers to the PCAR curriculum through S8. • Encourage teachers to learn from each other in using the TLMs by meeting with teachers at neighboring schools (along with PEAs) by sharing innovative practices. • Train teachers on how to integrate supplemental materials into their lesson plans. • To increase the number of participating schools, redistribute the materials, perhaps sharing the 180 titles among three neighboring schools, rather than having all materials remain in one school. The schools could then rotate or share the materials on a regular basis. • Provide more guidance on how to manage the books so that teachers can access titles that they need without difficulty. • Train all teachers up through S8 for several reasons: teachers often change grade levels and, thus, need to be prepared for RM should they be placed in S1-S3; and the books can be used in all grades beginning in S3 and ending in S8. • In communities, a greater effort can be made to establish mother and/or father groups to promote a reading culture. Parents could come to school in the afternoons to teach their children to read, and at home parents, older siblings, neighbors or relatives should assist children in reading. 6.3 Towards a More Comprehensive Approach to Reading Development • The MOEST should be encouraged to follow-up with the development of the new Reading Strategy, which, among other things, would determine the proper role of the current multiple reading programs (or their components) including Read Malawi, Tikwere, MTPDS, and others. Critical to this strategy would be the teaching of phonics in S1 and S2, the use of the PCAR core text for grammar in S1 and S2, and the introduction of the TLMs in S3 and their continued use through S8. • A less cumbersome system for reproducing the RM TLMs to satisfy the needs of NGOs should be developed. Additionally, the readers should be made available in bookshops on a commercial basis since books will be lost or damaged and there is no way to replace them. Also, many individuals stated that parents would be interested in procuring these books for their children. • Strengthen the e-Lab benefits for children by identifying instructional software (there’s much free software available on the Internet) and installing the software on the computers and providing guidance on the use. 6.4 Linking TLMP to USAID Priorities in Early Grade Reading • Educators constitute a valuable resource in the development of culturally-relevant reading materials. In focusing on early grade reading in mother-tongue languages, it would be useful to utilize the method that UTSA developed in creating TLMs. Developing the stories in local languages by those who speak those languages and who were socialized into their use Evaluation of TLMP – TLMP in Malawi 21 will create reading opportunities of interest to children, their families and teachers. However, integrated into this methodology should be lessons in the phonics/morphology of local languages and English (if books will also be produced for early reading in English). Leveling must also be paid considerable attention, as does the rigorous testing of materials in local languages and English to determine their suitability, level of enjoyment, etc. Once produced, materials should be made available commercially so that other stakeholders might be able to purchase the materials for their own use in a timely manner. • USAID programs in early grade reading should consider how to use the TLMs produced, e.g., implementing many of the recommendations made in the above sub-sections. To make these TLMs suitable as textbooks, introductory workbooks or texts need to be developed in phonics/morphology and linked to the vocabulary, syntax, and style of the stories printed. Alternatively, materials developed for other reading programs can be used to introduce the TLMs in Chichewa and English. This strategy might require a “renumbering” of the TLMs to reflect the skill level developed in decoding and comprehension. • The reading resource developed through TLMP in Malawi utilized a fairly rigorous approach that produced TLMs that both children and teachers enjoy for their color, layout, content, cultural relevance, gender sensitivity, and contextual issues raised. That so many titles were produced in both languages is an attestation of UTSA’s and MIE’s desire to fulfill the project plan and provide students with opportunities to read they did not have before. This resource should not be wasted; on the contrary, a concerted effort should be made to link the early grade reading program focus of USAID with TLMP. That future reading programs will have an on-site management entity should smooth out any operational challenges as well as create a more integrated effort. APPENDIX A. SCOPE OF WORK DESCRIPTION/ RESULTS-ORIENTED STATEMENT OF OBJECTIVES (SOO) Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania I. BACKGROUND TLMP contributed directly to USAID’s effort in the development and distribution of learning materials to improve sub-Saharan African (SSA) host country partners’ management capacity in the education sector. Each Minority-Serving Institution (MSI), based upon the provisions noted in their Cooperative Agreement (CA), was responsible for managing and implementing the TLMP in a specific country and with achieving specific output results. Each MSI was also responsible for providing (i.e., identifying, selecting, developing, adapting, printing, assisting with distributing, and training users) a minimum of 600,000 copies of quality, cost-effective education materials for use in primary schools in its host partner country. These materials were to be developed and/or adapted under the CA in partnership with the host partner country’s Ministry of Education (MoE) and other local specialists. The main objectives of the TLMP were to: 1) produce and distribute high quality, cost-effective textbooks and learning materials, in support of USAID’s African Education Initiative (AEI) to enhance girls’ and boys’ access to learning opportunities in primary schools within SSA, 2) strengthen the capacity of U.S.-based MSIs to build sustainable linkages with African institutions, which would enable the latter to continue technical assistance after the completion of the program, and 3) ensure alignment with national curriculum to include relevant cross-cutting themes (i.e. gender, health, etc.). TLMP Cooperative Agreement History • TLMP Ethiopia: USAID Cooperative Agreement RLA-A-00-09-00035-00; In coordination with local entities, Alabama Agricultural and Mechanical University (AAMU), over three million English for Ethiopia textbooks were produced and disseminated for grades 1, 6, 7, and 8. Over 132 teachers were subsequently trained to use the materials in classroom settings. • TLMP Ghana: USAID Cooperative Agreement RLA-A-00-09-00036-00; In coordination with local entities, Chicago State University (CSU) has trained 260 teachers in using the developed materials. Over six million materials and textbooks have been created and distributed for students up to grade 3 in mathematics, environmental science, and English. • TLMP Malawi: USAID Cooperative Agreement RLA-A-00-09-00033-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over five million supplemental reading books, teachers guides and training materials and trained nearly four thousand teachers on methodological classroom usage. • TLMP Senegal: USAID Cooperative Agreement RLA A 00-09-00037-00; In coordination with local entities, Elizabeth City State University (ECSU) produced and distributed over 1.8 million materials in both French and English for grades 2-10 in Evaluation of TLMP – TLMP in Malawi 23 science, mathematics, and language arts. Over 160 teachers were trained on utilizing the materials as part of their curriculum. • TLMP South Africa: USAID Cooperative Agreement RLA-A-00-05-00079-00; In coordination with local entities, University of Texas, San Antonio (UTSA) developed and provided over 1.4 million materials in 11 languages for grades 4, 5, and 6, as well as trained over six thousand teachers. The work was completed in 2009. • TLMP Tanzania: USAID Cooperative Agreement RLA-A-00-09-00034; In coordination with local entities, South Carolina State University (SCSU) created and disseminated over 1.1 million materials for secondary level usage in the fields of science and mathematics. Over one thousand, two hundred teachers were trained. The Contractor will be provided with each institution’s Cooperative Agreement by each individual institution, which will include the relevant scope of work. The Contractor will be required to obtain other pertinent documents as necessary. II. PURPOSE The purpose of this SOO is to support the Evaluation of Textbooks and Learning Materials Program (TLMP) in Ethiopia, Ghana, Malawi, Senegal, South Africa, and Tanzania. III. SCOPE OR MISSION Task 1 – Data Collection. Task 2 – Data Review. Task 3 – Coordination and Management. Task 4 – Site Visit. Task 5 – Data Analysis. IV. PERFORMANCE OBJECTIVES / DESIRED OUTCOMES The Contractor shall provide all labor, equipment, supplies and materials, and travel necessary to conduct Textbooks and Learning Materials Program Evaluation (PE). The PE is intended to satisfy the following objectives: • validate stated program goals and impacts; • assess the results achieved for each host partner country in relation to intended program targets, as well as standardized and variable indicators by measuring quantitative and qualitative impacts of TLMP in terms of local capacity building (i.e. U.S.- based MSIs, in-country institutions, Ministries of Education (MoEs), etc.), student achievement, teacher performance, amongst other criteria, in each host partner country; • determine if in-country institutions (with support from U.S.-based Minority-Serving Institutions [MSIs]) were able to deliver services effectively in terms of coordinating material design, alignment, production, and distribution; • review allocated USAID funding in terms of usage and overall cost effectiveness; • highlight specific program accomplishments per MSI-host country partnership; and • document lessons learned and provide recommendations for potential program scale-up and/or replication as related to the New Agency Education Strategy Evaluation of TLMP – TLMP in Malawi 24 http://www.usaid.gov/our_work/education_and_universities/documents/USAID_ED_Str ategy_feb2011.pdf 6 . V. OPERATING CONSTRAINTS / LIMITATIONS We anticipate that Awardee would complete one site visit per country and that the site visits would take no longer than 10 days each. There is not a requirement for specific key personnel or a combination of key personnel to complete the site visits; however consistency in terms of personnel for the site visits is preferred. The Contractor shall perform the PE in accordance with USAID ADS 203 and the new USAID Evaluation Policy published in January 2011. The USAID ADS 203 Performance and Monitoring Guidance can be found here: http://www.usaid.gov/policy/ads/200/203.pdf . The new USAID Evaluation Policy can be found here: http://www.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf. Monthly Status Reports. The Contractor shall provide written reports to the USAID COTR or his/her designee on the progress of the work, contacts made, and problems encountered on a monthly basis. They should be submitted by the last business day of every month. Comment Reponses. Comments will be provided to the Contractor electronically. The Contractor shall prepare comment responses that clearly state the actions taken to incorporate the comment or show the changes in a redline and strikeout version of the revised report. The Contractor may contact the reviewers for clarification. Unresolved technical issues shall be coordinated with the COTR. Evaluation of TLMP – TLMP in Malawi 25 APPENDIX B1. QUESTIONS POSED IN VARIOUS INTERVIEWS CONDUCTED AT UTSA 1. What have you learned from implementing the project in South Africa that you are using in Malawi – process as well as support people, etc.? 2. Of whom does the team consist at the University of Texas/Austin under Jim Hoffman’s direction; was this expertise not available at UTSA? 3. What role did the Institute of International Development at UTSA play in the two projects? 4. How was the collaborative process of creating materials developed? Was this led by Jim Hoffman? Who was involved in development? What was the process of creation, review, revision, etc.? How were the books field tested? Where? Who? What documentation/reports? Who developed the Teacher’s Guides? What process was undertaken for these? 5. How was the decision made about where to create the e-learning labs? Who made this decision? 6. What was problematic about obtaining copyrights for materials? 7. What is the Alliance for the International Study of Texts and Literacy? How are they involved? Get report of meeting April 6-28, 2010. 8. Of what does your exit plan consist? Get a copy of exit plan. 9. What was the purpose of the CB workshop for local printers? Did any of those trained receive any/part of the printing contract? Are they capable of taking on these activities in the future to bring it to Malawi instead of Uniprint in South Africa? 10. How were the rules of procurement between UTSA and Malawi changed to adapt to Malawi requirements? 11. Who are the members of the PAC? What roles did they play? How often did they meet? What was their SOW? 12. How did you handle payment or topping off MIE/MOEST/PAC members? How much? What for? 13. How was training and then use of materials monitored? What instruments were developed? Who was monitored? 14. What was the agenda for the annual international conferences? Who was invited? What were the outcomes? 15. How different are the approaches in teaching literacy in Chichewa from English? How is the system working to introduce the Chichewa materials in grade 1 and then the English version in grade 2, etc.? 16. What problems were encountered in the delivery of books from printer in SA to kitting in Malawi to delivery to districts to delivery to schools to use by children? Discuss the value chain. 17. Who determined pilot schools? How? How many? Where? Who/how were demonstration schools determined? And six TTCs? 18. Capacity Building Report – get a copy. 19. What changes in Teachers Guide to require a second edition? Any lessons learned here? 20. Report on National Roll Out Evaluation of TLMP – TLMP in Malawi 26 21. What were all the CB activities regarding how to do research? Who attended? Where? What have been the results? How will research continue to be conducted as part of the exit strategy/sustainability? 22. What changes in training teachers between read aloud and guided reading? What was the problem? What lessons learned? 23. What is the USAID funded MTPDS project? How does TLMP work with them? 24. What are “Efficacy” schools (30)? Where are they? (all three regions) M&Edone there – any reports? 25. Continuous professional development manual – get a copy. 26. What parameters for the production of the video at Mponda school in Zomba? How is this to be used? Do schools have the equipment to play? Who will facilitate? What monitoring afterwards? 27. Zomba Pilot Study. 28. Monitoring Tool for Village Heads and SMCs. 29. How have the materials been institutionalized through incorporation in their use at pre￾service TTCs? 30. What difference did it make to your work when Malawi revised its national primary curriculum? Did you have to revise to make them conform? 31. What skill shortfalls did you encounter in your Malawi partners? How did you address these (besides hiring outside consultants)? What capacities were built for the future? 32. How were the 14 core trainers selected? What criteria were used? Get Read Malawi Training Manual for Complementary Books. How were 9 training videos used? Did training differ for all the various groups? National monitoring team, Divisional SEMAs, core team, PEAs, TTC lecturers, head teachers, teachers. 33. What indicators did you use for monitoring all training and implementation? 34. Given the range of training that was provided, and the number and type of challenges that still remained, what would you do differently in terms of timing and training? 35. What kind of outcomes have been achieved through community mobilization activities? By the community members? By the SCMs and PTAs? Were the needs of illiterate parents addressed at all? 36. Where were reading centers established? What is their purpose? Who manages them? What resources are there? How are they different from community/school libraries? 37. From what you have learned about teachers in the Malawi system, do they shift jobs often? What are the reasons for this? GENERIC QUESTIONS POSED OF ALL MSIs TLMP Program Administrators (MSIs and Field Offices) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? • Tell me about the Lessons Learned from the implementation of Phase 1? How did this affect your approach and the substance of Phase 2? What changes were made in personnel? Why? • How is phase 2 different from phase 1? Evaluation of TLMP – TLMP in Malawi 27 • What are the components of your TLMP agreement? How many TLMs in what subject area? How many volumes of each? Do you have any kind of results/outcomes report that identifies all the numbers? • In country, how did you go about assembling your team? Did you have any assistance doing this? From USAID? Other stakeholders? What skills/abilities did each person have? What were their responsibilities? • At your university, how did you go about assembling your team? What skills/abilities did each person have? What were their responsibilities? • How did you monitor your progress? Do you have a PMP? Materials Development Process • What process was used in the creation of these materials? What guidelines did you follow in creating the materials? How did you determine the appropriateness of vocabulary, readability, complexity of structure, etc.? When creating the materials in English, how did you factor in that the students were learning English as a second or third language? How did you insure that the materials conformed to national curriculum standards? • In implementing the project, what role did the MOE play (specify unit)? What roles did your university play? What guidelines did the MOE provide? What level of competency did the MOE/curriculum developers have? What level of expertise did your university provide? What challenges emerged in your work with the MOE? How were they resolved? • What process did you use to review and revise the materials? How were the materials, field tested? Who was involved? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? Are they now able to take on similar work for other projects or for the MOE? What would prevent them from being the designated printer for the continued printing and distribution of these materials? Was the printer just a printer or also a publisher? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? Could I have a copy of the teacher training curriculum with handouts? How were learner-centered teaching methods blended with training on the TLMs? Did you make a DVD of the process? • What other teacher training is provided by the MOU? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries topped off? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? Evaluation of TLMP – TLMP in Malawi 28 Project Management and Outcomes • How was the university strengthened as a result of Phase 1? As a result of Phase 2? What is the university now able to do that it was not before the program? How has it built the university’s capacity to develop ideas for, submit proposals for, and implement other USAID projects? • What other types of assistance did USAID provide to you, either in the Mission or in DC? What was the substance of that assistance? • What kind of networks and/or public-private partnerships did you create in country and/or in the US? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, how have teachers changed? Principals? District/Provincial administrators? The MOE itself? The printers and distributors of the materials, i.e., how did the project improve the national publishing/printing industry? • What types of policy changes have you observed as a result of your work? Have any new policies been created in admission of children to school About distributing books vs. keeping them locked in cupboards? About class size? About early literacy development? • What role will the MOE play in extending the whole idea of TLMP? Has anyone been appointed to conduct follow-up activities? • What other funding/projects have you leveraged to continue the work of TLMP or to expand it in other directions? • What accomplishments are you most proud of? What are the outstanding features of TLMP for you, your team, and your institution? Logistics • Which officials in country shall we interview? What are their contacts? • How shall we work with your in-country team? • From your point of view, which would be two of the “best” schools and two of the “worst” schools? What are your criteria? Where are these schools located? What are their contacts? Evaluation of TLMP – TLMP in Malawi 29 APPENDIX B2. DATA COLLECTION INSTRUMENTS, MALAWI STAKEHOLDERS In Capital and Adjacent Locations 1) USAID Mission – Education Team • What has been the Mission’s role in implementing TLMP? What types of support activities have you provided to the project? • How does TLMP fit in with other USAID education program/priorities in this country? How do you see the experience of TLMP leveraging or leading to UTSA working on other educational development activities in Malawi or elsewhere in the future? • Did the development of TLMP have any (beneficial) effect on the national curriculum? On educational language policy? Has any new emphasis been placed on textbooks and learning material development? • How does USAID support teacher training in this country? How has the TLMP been linked to these efforts? What would make these efforts sustainable? • What specific challenges has TLMP faced in this country? How were they addressed? • How satisfied are you with the way TLMP was managed? What would you change? How satisfied are you with the outputs and outcomes of TLMP? What would you change? • Would you say that the model for TLM creation developed by UTSA (working with teachers to write the materials) can be utilized in other areas? Or was this a process you feel can be used for only supplementary or complementary materials? How would it need to be improved to be useful in other contexts? • How satisfied were you with your relationships with UTSA and the work they did? What suggestions do you have for overall improvement? • What were the lessons learned for the mission in overseeing the TLMP? Would the mission support a similar project in the future? 2) MOEST/MIE Administrators • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? • What was your particular involvement in TLMP? During what period? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. • How did you decide which staff members/departments were to work on the TLMP? Were they seconded to the project or were project responsibilities added to their normal tasks? Were any incentives provided for participation? What? • How was it decided which schools would receive the TLMs? Which teachers would attend the TOT? Which teachers would receive the TLMP cascaded training? • What types of policy change has the MOEST/MIE instituted regarding textbooks and/or supplementary/complementary materials as a result of TLMP? Regarding teacher training? • What other types of teacher training does the MOEST/MIE provide? How frequently? Evaluation of TLMP – TLMP in Malawi 30 • How do PEAs assess teachers? How does the district work with teachers to improve their teaching? How was this changed after the TLMP teacher training was delivered? How were the TLMs included in teacher training (either pre-service or INSET)? • What will the MOEST/MIE do to continue the production of TLMs now that the project has ended? What new textbook policies have been developed as a result of TLMP? • Is the MOE ready to use its own funding in the creation of TLMs in the future? What would prevent this from happening? • How has the material presented in the TLMs been included in national exams? Has performance on exams improved with the use of TLMs? • How was TLMP monitored by the MOEST/MIE? What indicators did you use? How often did you go to schools to observe the use of TLMs? How was TLM production managed and monitored? • How has the MOEST/MIE benefited from TLMP? How has it been challenged? Were there any negative effects of TLMP? If so, what were they and how were they addressed? • How satisfied are you with TLMP? If you were to make recommendations to another country implementing TLMP, what would you suggest? (Why?) If you could change anything about TLMP, what would it be? Why? If you were to scale up the production and distribution of these books, what would you want to be different? • How satisfied are you with the collaborative relationships established with UTSA? What could be improved? 3) Materials Developers/Curriculum Specialists • What is your current position? For how long have you had this position? For how long have you been working in this area (e.g., subject matter, curriculum and instruction, grade level)? What is your educational/training background in this area? • How did you become involved in TLMP and at what point in the process? What was your specific role at the outset? At the end of the project? • What was the composition of the writing/production team? How many males? Females? What types of expertise was represented? What other expertise was needed, in your view? How were the members of the team compensated for their activities? • How often did the two teams meet? What were the results of these meetings? • In developing TLMs, how did you ensure conformity with the national curriculum in terms of subject matter and grade level? How did you include gender concerns? How did you include HIV/AIDs-related concerns? How did you include cultural/contextual relevance? What other cross-cutting themes did you include? • How was the decision made (and who made it) that materials would be produced only in Chichewa and English (rather than multiple mother-tongue languages)? How was the decision made (and who made it) that certain materials should only be produced in Chichewa while others were produced in both Chichewa and English? • What type of local and international review process did the production team have to go through? • How did you obtain illustrators for the TLMs? Evaluation of TLMP – TLMP in Malawi 31 • How satisfied are you with the collaborative production process between yourselves and UTSA? What worked well/did not work well? What would you change to improve the process? • How do you think the production process can be improved in the future? 4) TLMP Program Administrators (Zomba) Background Information • Tell me about how you got involved in TLMP? How you organized your team? The roles of each member on the team? Did you have any assistance doing this? From USAID? Other stakeholders? Materials Development and Distribution Process • What process was used in the creation of TLMs? What did the MOEST/MIE do (specify unit)? What did the project do? What challenges emerged in your work with the MOEST/MIE? How were they resolved? How did you liaise with all stakeholders? • In implementing the project, what role did the MOEST/MIE play (specify unit)? What roles did your office play? What guidelines did the MOEST/MIE provide? • How did you identify printers and distributers of these materials? What challenges emerged in your work with them? How did you build the capacity of the printers? What work are they now able to take on with other clients? What other services did the printer provide? • How was the decision made about which districts/schools would receive the materials? Was the printer responsible for distribution? What was the distribution chain? How did you monitor distribution? • How was teacher training conducted? Who and how was it decided which teachers to invite? How many sessions were held? How many teachers actually attended each session? What geographic distribution? • What other teacher training is provided by the MOEST/MIE? In what format? How did the TT for TLMP differ from the TT for other areas? • In conducting TOTs, were teacher salaries supplemented? By how much? Did those teachers attending the TOTs and then cascading the training have their salaries topped off? By how much? • How successful was the cascade training model? How many teachers did those who participated in the TOT actually train on the use of the TLMs? Project Management and Outcomes • What was the TLMP management structure in Malawi? What types of services did you provide to the MSI and other stakeholders? • What was the composition of the Malawi PAC? How often did they meet? What decisions did they make? How did these decisions affect the project? • How often did you visit project implementation sites? What types of monitoring did you perform? How frequently? • What types of assistance did USAID provide to you? • What kind of networks and/or public-private partnerships did you create? How are you collaborating with other stakeholders? • From your point of view, as a result of the TLMP project, what has changed either positively or negatively? How have teachers changed? Principals? District/Provincial Evaluation of TLMP – TLMP in Malawi 32 administrators? PEAs? The MOEST/MIE itself? The printers and distributors of the materials? • What types of policy changes, if any, have you observed as a result of your work? • What accomplishments are you most proud of? If a TLMP-type project were to be undertaken again, what would you do differently? 5) Printers/Publishers • Tell me about your operations before you were granted the TLMP contract and how they changed (either positively or negatively) as a result of TLMP participation? • How did the contracting occur with the UTSA? Did you have adequate personnel and technical resources to fill the order? What was lacking? How did you overcome these? • What is your opinion of the management training that was provided to you by UTSA? What key lessons did you learn that you applied to the management of your operations? • What instructions were you given on how to distribute the TLMs? From whom? What kind of difficulties did you encounter in keeping to the distribution schedule? • When/how did you distribute the TLMs after they were produced? To whom did you distribute them? How many TLMs were delivered to each receiver? What kind of tracking/delivery system did you establish? What kind of challenges did you have in distributing the materials? How were these overcome? • How did having the TLMP contract change the way you do business? Improve your capacity? What new work are you now able to do that you could not before TLMP? How many new employees have your hired? What new equipment have you purchased? What other inputs would you require to take on more textbook production projects? In Field 6) Regional Education Offices/Primary Education Advisors • What is your current position? How long have you been in this position? For how long have you been working in education? In what positions? What is your educational/ training background related to this work? • What was your particular involvement in TLMP? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the program? Please explain your reason for this choice. • How many of each TLM did you request for your district/region/province? (Subjects, languages, levels?) How did you calculate this number for appropriate grade level students? If you had any surplus, what did you do with the materials? If you had any shortfall, what did you do? • What instructions did you give for distribution to each school? How did you work with the distributor of the text and workbooks to ensure that they were properly delivered and received? • What instruction did you give to each school about how the TLMs were to be used? How many teachers in your district attended the TOT? How were these teachers chosen? How many were males? Females? How many of these teachers went on to teach others through the cascade model? How many others were trained? How would you rate the quality of the training they provided? Based on what evidence? Evaluation of TLMP – TLMP in Malawi 33 • How were inspectors instructed on how to evaluate teachers using TLMs? Do all children have/use the books produced? • From your point of view, on a scale of 1-4 with 1 being very effective and 4 being not effective at all, how would you rank the TLMs produced for this project? What do you recommend for materials improvement? Program improvement? In Schools: 7) Principals • What is your current position? How long have you been a principal at this school? How long have you been a principal? In how many schools? • What is the overall economic status of the people in this community? How do they generate income? What is the composition of most families/households? How big a problem is HIV/AIDS in this community? About what percentage of your students are Orphans or Vulnerable Children (OVC)? • Do families send their girls to school as often as their boys? What gender-based trends do you see in enrollment? Has your school done anything to make teachers or families more aware of gender disparities in enrollment/attendance? If so, what have the results been? • What language(s) do children speak when they enter school? Is this language the language of instruction? If yes, until which grade? Are the TLMs in the appropriate language(s) for this school? If not, how should they be changed? • In terms of teacher mobility, has there been any increase or decrease in the rate of teacher transfer since they attended a TOT or were trained in the use of the TLMs? What are the most common reasons why teachers request a transfer? [If appropriate, you can prompt, e.g., “Does this have to do with obtaining a higher salary, improving living conditions, or other factors?”] • How many of your teachers/administrators participated in the development of TLMs? Where was the work undertaken? For how long? • How many of each textbook did you request for the school? How did you calculate this number to include entering KG1 students? How many of each textbook did you actually receive per grade level? If you had any surplus, what did you do with the materials? If you had a shortage, what did you do? When during the term were the books received? • How many of your teachers attended the TOT in the use of the TLMs? How did you choose these teachers? How many of these teachers went on to teach others? How many other teachers received the training from a teacher who attended the TOT? What were the teachers’ reaction to/opinion of the training? Did you attend the TOT yourself? If so, what was your opinion of the training? • What other types of teacher training does the MOEST/MIE provide? How often? Are those who attend expected to pass on what they have learned to their colleagues? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice? What improvements would you make to the TLMP? Why? Evaluation of TLMP – TLMP in Malawi 34 8) Classroom Teachers • What is your current position? How long have you been teaching this subject at this grade at this school? How long have you been a teacher? What other classes have you taught before? At what grade level? What is the level of education you have achieved? What qualifications do you have to be a teacher (certificate, diploma, degree)? • In this Region, which languages are used for instruction at which grade levels? In which language is initial literacy (reading and writing) learned? At what grade do children start learning English? At what grade does English become the language of instruction? • What is your greatest challenge in teaching in English? In Chichewa? In any other Mother Tongue languages? What would you like to improve? • What role, if any, did you play in producing the TLMs? Please explain. • How many students do you have in your classes? Males? Females? Specify class and number of students. What is the age range of your students in each class? • What non-TLMP textbooks do you have to teach reading? What non-TLMP workbooks do you have to teach? Does every child have a textbook? Workbook? What do you do when you don’t have enough textbooks or workbooks for each child? Do you have a teacher’s guide for each of the textbooks/workbooks? If not, what do you use? • What TLMP supplementary readers and TLMs do you have to teach? What TLMs do you have to teach? What do you do when you don’t have enough TLMs for each child? Do you have a teacher’s guide for each of the books? If not, what do you use? • When did you receive training on the use of TLMs? How long did it last? Did someone from TLMP or another teacher deliver the training? What is your impression of the TLMP training? On a scale of 1-4 with 1 being very good and 4 being very bad, how would you rank the TLMP training you attended? Please explain your reason for this choice and identify areas where it could be improved. If you did not attend any training related to the materials, how did you learn how to use them? • When did you receive the TLMP materials for your classes? How many were you provided? From whom did you receive them? How did you distribute them to your students? How many students must share a textbook? A workbook? Are students allowed to write in their workbooks? • Were you able to use the textbooks/workbooks after the training? Did you feel you needed more training? In what? • From your point of view, how satisfied are you with the way gender issues are treated in the books? What can be done to improve this? From your point of view, how satisfied are you with the way HIV/AIDS issues are treated in the books? What can be done to improve this? From your point of view, how satisfied are you with the way cultural/ contextual issues are included in the books? What can be done to improve this? • Do you believe the TLMs were aligned with the curriculum? If not, how should the materials be changed? • Do you believe the TLMs were properly sequenced (go from easiest to hardest)? What would need to change if they were not? • For each class that you teach, how long per day/how many periods per day [per week, per month] do you use the TLMs materials? Evaluation of TLMP – TLMP in Malawi 35 • What, if anything, does “learner-centered teaching” mean to you? Do you think these materials help you to be more learner-centered in your teaching? Why/why not? • How “ready” were your students to use the materials distributed? Was the grammar and vocabulary at a level that could be understood by students? What type of difficulties do the students have in using the materials? How should the program overcome these difficulties? • What changes (either positive or negative) have you observed and recorded in girls’ and boys’ achievement on annual or national examinations since the TLMP workbooks/ materials were introduced? Do you think these changes are attributable to the use of the TLMs? What evidence can you give for this? • What is your opinion of the TLMs in so far as their attractiveness to students? On a scale of 1-4, with 1 being very attractive, and 4 being not very attractive, rank the materials. Please explain your reason for this choice. • What is your opinion of the TLMs in the ways that they depict girls and boys? Do they represent them in non-traditional /traditional roles? • Is there anything about the TLMs that you would change? What? Why? • In using the TLMP workbooks/materials, what changes have you made in your teaching? How useful is the Teacher’s Guide in planning and teaching your lessons? On a scale of 1-4 with 1 being extremely helpful and 4 being not helpful at all, please rank the Teacher’s Guide. Please explain your reason for this choice. • What is the greatest challenge your students experience in using the TLMs? • What do you think is the overall impact of the program on your students? Have you noticed any difference between males and females? What kind of difference does it make in learning for a child to have textbooks/workbooks? What do you think could be improved to have an even larger impact? • What is your impression of the TLMP? On a scale of 1-4 with 1 being very good and 4 being very bad, how would your rank the program? Please explain your reason for this choice. What changes would you make to improve the program: 1) in the textbooks and learning materials? 2) In the delivery of the program? IF ALSO PROVIDED TOT, ANSWER THE FOLLOWING ADDITIONAL QUESTIONS: • If you attended the TLMP TOT training, how many other teachers did you teach afterward? Where did you conduct this training? What worked well? What difficulties did you encounter in doing this? Were you provided with any follow-up support after you received the training? What type? How often? By whom? • Why do you think you were chosen to be a trainer? • On a scale of 1-4 , with 1 being very satisfied and 4 being not satisfied, how would you rank the training you received? Please explain your reason for this choice. What feedback, if any, did you receive from observers on your training style and approach? How did this feedback improve your own teaching? • How familiar were you with the TLMP materials before you delivered the training? What materials were you provided to be a trainer? What materials did you provide the trainees? How confident were you after the TOT that you could teacher others in how to use the TLMs? What else did you need? Evaluation of TLMP – TLMP in Malawi 36 • Was the length of training adequate for you to cover all topics well? What area required more time? • Did you receive any added compensation for conducting this training? 9) CLASSROOM OBSERVATIONS OF TEACHER USE OF TLMs School: ______________________________ Village/Town/City: ____________________ Country: _________________________ Type of School: ___Primary ___Jr. Sec ___Sec Teacher Sex: _____ M _____F Grade Level: __________________________ No. of Students: ___ M ___ F ___Total No. of repeaters/overage learners: ___M ___ F ___Total Languages spoken in this community: ___________________________________________ Home Language/Mother Tongue of Learners in the Class: ____________________________ Home Language/Mother Tongue of Teacher: _____________________________________ No. of books: _________ No. & type of learning materials: _________________________ Indicator Observed Not Observed Teaching Using TLMs Instruction 1. The teacher has prepared an authentic lesson that uses the TLMs for the class period. 2. Students have the appropriate TLMs and are ready to use them in class activities. (Note ratio of materials to learners.) 3. The teacher explains the goal and purpose of the class lesson to the students. 4. The teacher identifies, pronounces and defines any difficult vocabulary before teaching the lesson. 5. The teacher begins the class activity with questions that review previous activities using the TLMs and draws on the prior knowledge of the students. 6. The teachers uses learning aids/materials produced by TLMP. 7. The teacher can read and explain TLM content to the students. 8. The teacher uses TLMs throughout the class period. 9. Students use TLMs throughout the class period. 10. Students can read and understand the subject matter in the TLMs. 11. Students are actively and interactively engaged with the teacher in the use of TLMs (Q&A, group work, workbook practice, continuous assessment). Evaluation of TLMP – TLMP in Malawi 37 Indicator Observed Not Observed 12. The teacher makes sure learners make connections of TLMP content to daily lives. 13. The teacher gives and corrects homework using the TLMs. 14. The teacher shows evidence of having used the Teacher’s Guide in presenting the lesson. 15. Students and teachers use mother tongue/English when asking and responding about TLMs (circle which language). TLMs/Artifact Inventory 16. Lesson objectives are written on the board in either local language or English. 17. Learning aids/materials are posted in the classroom (TLMP produced and others). 18. TLMs are locked up in the cupboard. 19. Word walls display key words in local languages and English. 20. To demonstrate language experience, sentences appear on the chalkboard or on a chart in the classroom. 21. Students write words and sentences in their exercise books (demonstrating evidence of having pencils/pens and exercise books). 22. There is evidence that teachers (or peers) mark exercise books in a process of continuous assessment. Comment_______________________________________________________________ _______________________________________________________________________ Evaluation of TLMP – TLMP in Malawi 38 9) Learners Reading Competency - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read a few sentences aloud, and then rank the performance in the following manner: 1) fluent; 2) little difficulty; 3) very haltingly; 4) can read only a few words; 5) cannot read at all. If a learner tries to read a word, identify what strategy he/she is using to read: 1) sound it out – phonics; 2) sight reading; 3) other strategy (TBD). Reading Comprehension - In each country, identify words and/or a short passage appearing in a book produced by the project. Have learners read the passage silently and after they have finished, have learners explain what they have just read. In the case of KG and grade 1 where pre-reading text and workbooks have been developed, identify letters, ask what sound the letter makes, and then have the learner point out words that include this letter. Then have the child write the letter (or a word). Rank learners in the following manner: 1) instant recognition; 2) uses other cues to identify the letter; 3) does not associate the graphic presentation of a letter with the sound it makes; 4) cannot recognize a letter in a word; 5) cannot write a letter or a word. Use the following instrument for all grades: EVALUATION OF LEARNER READING COMPETENCY Note: Try to evaluate 12 learners per grade: 3 girls and 3 boys that the teacher has indicated are advanced learners, and 3 girls and 3 boys that the teacher has indicated are weak. School: __________________________ Village/Town/City: ________________________ Country: _________________________ Type of School: ___Primary ___Jr. Sec ___Sec Grade Level: ______________________ Class Subject Matter _____________________ Student Age: _____________ Sex: ____M ____F Repeater: ___ Yes ___No Home Language/Mother Tongue of Learner: ___________________________________ Language of Testing: _____________________________________________________ Title/Type of TLM used in Class: _____________________________________________________________________ Reading Aloud: Instructions: Identify 2 short passages (3 to 5 sentences) from one of the TLMs for the appropriate grade level and make a clean copy of each passage. [Please save passages to accompany evaluation results. They should preferably be cut from within stories or texts so that memorization can be ruled out. One passage should be easier than the other, so one can be taken from the beginning of the book and one from the middle, depending on how far into the school year they are.] Have the student read the more advanced passage aloud and note the results below. If s/he is successful, stop after this. Evaluation of TLMP – TLMP in Malawi 39 If the advanced passage is too difficult, provide the easier passage and conduct the exercise again, noting the results. Reading Fluency Passage 1: ____________________ (Rate and comment) 1) Fluent _______________________________________________________________ 2) Little difficulty (specify words) _____________________________________________ 3) Very haltingly (specify words) ______________________________________________ 4) Can read only a few words (specify words) ____________________________________ 5) Cannot read at all ______________________________________________________ Passage 2: ____________________ 1) Fluent _______________________________________________________________ 2) Little difficulty (specify words) ______________________________________________ 3) Very haltingly (specify words) ______________________________________________ 4) Can read only a few words (specify words) ____________________________________ 5) Cannot read at all ______________________________________________________ Reading Comprehension/Silent Reading [Note that you will need the teacher or another local person to translate what each student says in lower grades.] Instructions: Identify two short passages (3-5 sentences) from one of the TLMs for the appropriate grade level and make a clean copy of each passage. [Please save passages to accompany evaluation results. These can come from the beginning of stories or texts so that the context is clear. As above, one passage should be easier than the other.] Have the student read the more advanced passage silently and then re-tell what the passage is about using his/her own words. Note the results and comment below. If s/he is successful, stop after this. If the advanced passage is too difficult, provide the easier passage and conduct the exercise again, noting the results. Ability of learner to paraphrase what s/he has read silently: 1) Can put into own words all that has been read without difficulty ____________________ 2) Can put into own words only selected points __________________________________ 3) Has difficulty putting passage into own words __________________________________ 4) Cannot put passage into own words _________________________________________ 10) Community Mobilizers/CRECCOM (Zomba) • What is the core business of your organization? How did you become involved in TLMP? • What did you do in communities to help people understand the importance of reading? Which strategies were the most effective? Do you believe that community mobilization was an appropriate activity for TLMP? Why? Evaluation of TLMP – TLMP in Malawi 40 • What results did you achieve? What challenges did you face? How were these overcome? • What is your opinion of the supplementary reading TLMs produced by TLMP? What was the community’s/parents’ opinion? • What actions did community members take to demonstrate their value of reading had changed? (e.g., is there a demand for adult literacy classes; do parents read with/to their children? Etc.) • How satisfied were you with your relationship with UTSA? How could this relationship be improved? • If you could restart your involvement with TLMP, what would you do differently? Evaluation of TLMP – TLMP in Malawi 41 APPENDIX C. SCHEDULE OF STAKEHOLDER INTERVIEWS AND ACTIVITIES DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE LILONGWE 17/2 2:30 TLMP team at Wendal’s 18/2 8:00 Lilongwe Demonstration School Carolyn Majiga Head teacher 18/2 2:00 Lilongwe TTC Elick Kwanda. Goodson Jona Acting Director eLab 18/2 3:30 MOEST Meeting at Wendal’s Patrick Themu Residential Course Manager DTED 19/2 7:30 USAID Ramsey Sosola John Collins Chikondi Maleta Program Management Specialist Education Officer Program Management Specialist 19/2 10:30 MOEST Mrs. Chikondano Mussa Director of Secondary Education (formerly Deputy Director, Basic Education) 19/2 2:00 Ultinets Stuart Winga Head of Operations DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE 19/2 3:30 Save the Children Lexon Ndalama Acting Senior Manager for Education 20/2 7:00 DEPART FOR ZOMBA ZOMBA 20/2 11:30 UTSA Project Coordinators Ms. Sellina Kanyerere Evaluation of TLMP – TLMP in Malawi 42 Mr. Henri Chilora 20/2 3:00 CRECCOM Madalo Samati Director of Programs 20/2 4:00 Illustrator -MIE Heath Kathawere 21/2 7:00 Schools Team 1 (Zomba) Mponda Primary school Police Primary School Dorothy Kalta M.M. Chagoma Head teacher Head teacher 21/2 7:00 Schools Team 2 Domasi Demonstration School Domasi Government School James Kalomgonda Nehru Banda Head teacher Head teacher 21/2 2:30 Malawi Institute of Education (Zomba) Dr. William Susuwele-Banda Executive Director DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE 22/2 6:30 School Visits (Zomba, St. Anthony’s Zone) St. Anthony Girl's Primary School St. Anthony Boy’s Primary School Sr. Catherine Bulla Louis Matekenya Gregory Kamwendo Head Teacher DHT HT 22/2 10:00 St. Anthony’s Zone TDC Ms. Mphatso Makhumula PEA 22/2 1:00 Zomba Rural Education Office Mac Owen Ligomeha Hamilton Hon Saiti Alifo Ussi Paul Chindamda PEAs 2:30 Writers at MIE Foster Gama Evaluation of TLMP – TLMP in Malawi 43 23/2 8:30 TRAVEL TO BLANTYRE Evaluation of TLMP – TLMP in Malawi 44 DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE BLANTYRE 10:30 Kriss Offset Sabri Gani Ashraf Patel MacDonld Lambole General Manager Finance Controller Sales and Marketing Manager 25/2 7:00 Schools Team 1 (Chiradzulu District) Malavi Primary School St. Theresa Primary School Maston Angiston Sitima Patron Alexdandef Makwinja Head teacher Head teacher 25/2 11:30 Montford College for the Blind Hastings Magombo Deputy Principal and Head Visual Impairment Department 25/2 2:30 MOEST Alex Chipungu Logistics Officer, Supplies 25/2 2:30 Blantyre TTC Mr. Magelegele Principal, Blantyre TTC 26/2 7:00 Schools Team 1 (Chiradzulu District) PIM Primary School Mafe Primary School C.T.K. Mbewe Raison Chidziwe Head teacher Head teacher 26/2 7:00 Schools Team 2 (Blantyre Urban District) South Lunzu Primary School Namilangu Primary School DATE TIME ORGANIZATION PERSON(S) TO BE INTERVIEWED TITLE 26/2 2:30 Illustrators - Nation Newspaper James Kazemba Ralph Mwara Illustrators 2:30 PEAs at Ryalls Hotel Monica Kainja Luke Mamboya PEA – Ndirande Zone, BT Urban PEA – Bangwe Zone- BT Urban 27/2 9:00 DEPART FOR LILONGWE LILONGWE Evaluation of TLMP – TLMP in Malawi 45 28/2 8:30 Team Meeting at Crossroads 28/2 Write Findings & PPT for USAID Debriefing Arrange meetings with UNICEF (Panjee 0888-384194 28/2 2:00 MoEST Mr. R. Agabu Director, DIAS 1/3 8:00 Debriefing at USAID 1/3 Finalization of work with team 2/3 10:00 Tom and Nancy Leave APPENDIX D. READING ASSESSMENT METHODOLOGY Introduction The data collection instrument appearing in Appendix B2 became very cumbersome; consequently the streamlined methodology appearing below was the one used. Schools and Children Selected District Zone School Blantyre Urban South Lunzu Namilango South Lunzu Chiradzulu Malavi Malavi St. Theresa PIM Mafe PIM Zomba Rural Mchengawede Domasi Demonstration Domasi Government St. Anthony St. Anthony Boys St. Anthony Girls Zomba Urban Mponda Mponda Within each school, two boys and two girls were selected from S1, S2 and S3 giving a total of 66 children. The children in S1 were tested in Chichewa and those in S2 and S3 were tested in both Chichewa and English using the Guided Readers that had recently been the subject of classroom lessons. Theoretically, the children should have been able to read the selected texts. Instruments and Testing One important shortcoming of the RM books, especially in S1, is that they did not include an introduction to letters, syllables, or words. Yet, as will be seen by the test results, almost no S1 children could actually read; thus, the evaluation team included in the testing process both letter and word recognition. Only then would the team ask the children to read a passage from the appropriate level TLMs. The team followed the same routine for all three grade levels. The test was administered to children individually, often with a teacher from the school present (especially when the team member was not from Malawi). After a brief introduction to each child, the test began by asking the child to point to the letter “M” anywhere on the page shown. Typically, the page inside the cover of the book was shown to the child where there were several examples of M as in Malawi. The second letter asked was S. In the English tests for S2 Evaluation of TLMP – TLMP in Malawi 47 and S3, other letters were used such as d, g, k and u. The team member recorded whether the child could identify each letter and, if so, the number of attempts before actually identifying the correct letter. A correct answer was coded as a 1, and an incorrect answer as a 0. The second component of the test included the identification of words. The team member would say a word, and the child was then asked to point to that word on the page. In general, each child was asked to identify two words. The team member tried to identify simple words, but even this was difficult in some of the books. Typical words in Chichewa included akuti (they say), agogo (grand pa/ma), anthus, and ndi (and). In English, words included fruit, village, fun, good, today, and brother. Depending on the number of words identified correctly, the child would receive a score from 1 (can identify all words without a problem) to a 5 (unable to read any words). Then, the child was asked to read one or two short passages selected at random within the reader. A reference to each passage was recorded by the team member. Each child would receive a score from 1-5 according to the following scale: 1= fluent, 2 = little difficulty, 3 = very haltingly, 4 = can read only a few words, 5 = cannot read at all. Finally, if a child were successful in reading the passage, s/he was asked to read another passage silently and then describe in his or her own words what the passage said. The response for each child was coded as follows: 1 = can put into own words all that has been read without difficulty, 2 = can put into own words only selected parts, 3 = has difficulty putting passage into own words, or 4 = cannot put passage into own words. All of these exercises were completed in both Chichewa and English at all three grade levels, with the exception in S1 where only Chichewa was tested. During the analyses, the five scale responses were reduced to three. For example, the responses to reading a passage were coded as follows: the "fluent" and "little difficulty" were combined into a single score indicating ability to read. The middle category "very haltingly" was left as is. And the bottom two categories "can read only a few words" or "cannot read it all" were combined as “cannot read” measure. Evaluation of TLMP – TLMP in Malawi 48 APPENDIX E. CONFLICT OF INTEREST STATEMENTS Evaluation of TLMP – TLMP in Malawi 49 Evaluation of TLMP – TLMP in Malawi 50 Name Denis Khasu Title Dr Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number (contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Chicago State University, Alabama A&M, South Carolina State Univ., Elizabeth City State University, Univ. of Texas San Antonio I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. Signature Date 2/13/13 Evaluation of TLMP – TLMP in Malawi 51 Name Hannock Yamikani Mateche Title Evaluation Specialist Organization IBTCI Evaluation Position Team Leader Team member Evaluation Award Number(contract or other instrument) AFR-12-000001 USAID Project(s) Evaluated(Include project name(s), implementer name(s) and award number(s), if applicable) Textbooks & Learning Materials Project (Univ. of Texas San Antonio) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 2 August, 2013 x x