MALAWI TEACHER TRAINING ACTIVITY SUMMATIVE EVALUATION REPORT July 2008 This report was produced for review by the United States Agency for International Development Contract No: GS 10F-0112J Order No: 690-M-04-00260-00. It was prepared by David Chapman, Suzanne Miric, and Shirley Miske for Miske Witt & Associates, under a subcontract to the American Institutes for Research. The Malawi Teacher Training Activity is implemented by the American Institutes for Research in partnership with Malawi Institute of Education, Miske Witt & Associates, and Save the Children, US. MALAWI TEACHER TRAINING ACTIVITY SUMMATIVE EVALUATION REPORT Part A: In-service Teacher Training Component Evaluation by David Chapman and Suzanne Miric Part B: Life Skills Pre-Service Teacher Training Evaluation by Shirley Miske for Miske Witt & Associates July 2008 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. TABLE OF CONTENTS List of Acronyms ........................................................................................................................................................ iii Foreword ..................................................................................................................................................................... iv Acknowledgments ...................................................................................................................................................... v Part A: MTTA In-Service Teacher Training Component Evaluation ............................ A-1 Executive Summary ............................................................................................................................................... A-2 Summary of Key Findings ..................................................................................................................................... A-3 Introduction ............................................................................................................................................................ A-4 Methodology ........................................................................................................................................................... A-4 Background: Malawi and MTTA ......................................................................................................................... A-5 Overview of MTTA ............................................................................................................................................... A-6 Cycle training .............................................................................................................................................. A-6 Teacher Professional Development Conferences ............................................................................. A-7 Development of instructional supervision ........................................................................................... A-8 ZINFA facilitated teacher-training workshops ................................................................................... A-8 Head teachers as instructional leaders ................................................................................................. A-8 Resource materials and books ............................................................................................................... A-9 Mobile Teacher Training Troupe ........................................................................................................... A-9 Main Findings ......................................................................................................................................................... A-10 Changes in Pupil Achievement Levels ............................................................................................................. A-21 A View of MTTA through the Galal Model ................................................................................................... A-22 Cross Cutting Issues in MTTA ......................................................................................................................... A-23 Teacher achievement gains .......................................................................................................................... A-23 Cascade training ............................................................................................................................................. A-23 The utility of monitoring and evaluation data ......................................................................................... A-24 Costs and benefits of volunteerism ........................................................................................................... A-24 The efficacy of demonstration projects .................................................................................................... A-24 Sustainability .................................................................................................................................................... A-25 A bottom-up approach ................................................................................................................................. A-25 System-level adaptability to decentralization .......................................................................................... A-25 USAID project management ....................................................................................................................... A-25 Insights for future projects .......................................................................................................................... A-28 Conclusions ........................................................................................................................................................... A-28 Annexes Annex A: A statistical picture of MTTA ................................................................................................. A-29 Annex B: Summary of MTTA inputs ........................................................................................................ A-32 Annex C: List of people contacted .......................................................................................................... A-34 Annex D: Documents reviewed ............................................................................................................... A-35 Part B: MTTA Life Skills Pre-Service Teacher Training Component Evaluation ......... B-1 Executive Summary ................................................................................................................................................ B-2 Introduction ............................................................................................................................................................. B-4 Methodology ............................................................................................................................................................ B-4 Background: Life skills and pre-service teacher training in Malawi ............................................................. B-5 Overview of the Life Skills pre-service teacher training component ......................................................... B-6 Evaluation findings ................................................................................................................................................... B-9 Summary ................................................................................................................................................................ B-14 Conclusions ........................................................................................................................................................... B-17 References ............................................................................................................................................................. B-19 i ii Annexes Annex E: MTTA Pre-Service Life Skills Evaluation Schedule ............................................................ B-20 Annex F: Questions for TTC Lecturers ................................................................................................ B-21 Annex G: Responses from Group Interviews with TTC Lecturers ............................................... B-22 Annex H: Questions for Student Teachers .......................................................................................... B-25 Annex I: Schools, Teachers, and Topics of Classroom Observations ........................................... B-26 LIST OF ACRONYMS AIR American Institutes for Research DEF District Education Facilitator DEM District Education Manager DTED Department of Teacher Education and Development GoM Government of Malawi IBB International Book Bank IEQ Improving Educational Quality JCE Junior Certificate of Education MANEB Malawi National Examination Board MESA Malawi Education Support Activity MIE Malawi Institute of Education MIITEP Malawi Integrated In-service Teacher Education Program MoEST Ministry of Education, Science, and Technology MSCE Malawi Secondary Certificate of Education MTTA Malawi Teacher Training Activity MTTT Mobile Teacher Training Troupe MW&A Miske Witt & Associates PCAR Primary Curriculum Assessment Reform PEA Primary Education Advisor PSLE Primary School Leaving Exam STI Sexually Transmitted Infection(s) TDC Teacher Development Center TTC Teacher Training College TOT Trainer of trainers TPDC Teacher Professional Development Conference USAID United States Agency for International Development ZINFA Zonal In-service Facilitator iii FOREWORD The Malawi Teacher Training Activity, a three-year and 11-month initiative, was funded by the United States Agency for International Development (USAID)/Malawi in collaboration with the Malawi Government in response to the need to improve the quality of education in Malawi. MTTA was implemented by the American Institutes for Research (AIR), in partnership with Save the Children, US, Malawi Institute of Education (MIE), and Miske Witt and Associates. From its launch in September 2004 until December 2007, and then for an additional 7-month extension period through July 2008, MTTA worked to improve the quality of education in Malawi through improving professional skills of primary school teachers at the pre-service and in-service levels. In four target districts (Kasungu, Machinga, Mzimba South and Phalombe, see map of Malawi below), MTTA strengthened teachers’ content knowledge in mathematics, science and English by building on AIR’s former initiative in Malawi, Malawi Education Support Activity (MESA), which included in￾service training on pupil-centered, gender-fair, and active-learning teaching methods. MTTA worked to train over 6,300 teachers in every primary school in each target district; develop and disseminate learning resources to every teacher and school; and train head teachers, primary education advisors and mentor teachers to support these classroom teachers through field- based supervision. iv ere As a complement to its intensive in￾service teacher training, MTTA also worked at the pre-service level, orienting pre-service lecturers to Life Skills Education in five government and grant￾aided Teacher Training Colleges (TTCs) and two private colleges. The main objectives of the pre-service activity w to facilitate the development of knowledge and skills among teachers that would help them avoid HIV and AIDS infection; to enhance teachers’ capacity to facilitate open discussions in the classroom about HIV and AIDS and sexuality; and to develop teachers’ pedagogical competences in student￾centered approaches. This summative evaluation report, made possible with USAID/Malawi’s support from the 7-month extension funding, comprises two distinct evaluations. Part A of this report includes an evaluation of MTTA’s in-service teacher training efforts, which represented the largest component of the project’s activities and, as such, of the two parts of this report the evaluation in Part A demanded the greatest level of effort. Part B includes the findings from a separate, but complementary evaluation of MTTA’s pre-service teacher training activities. Together, these two evaluations provide a summative look at the impact of MTTA on the quality of teaching in Malawi through improvements to teachers’ content knowledge and instructional abilities. Maching Phalombe Kasungu Mzimba South Simeon Mawindo, MTTA Chief of Party, American Institutes for Research Jennifer Anderson, MTTA Project Manager, American Institutes for Research ACKNOWLEDGMENTS The Evaluation Report on the in-service component of the Malawi Teacher Training Activity (MTTA) was conducted in collaboration with a number of people who made this study possible, including: Dr. Shirley Miske, Miske Witt & Associates, for her support and advice throughout this consultancy; MTTA Chief of Party, Mr. Simeon Mawindo, and his staff, especially Dr. Hartford Mchazime, MTTA Deputy Chief of Party, who provided valuable guidance and background information; MTTA staff member Rath Kathewera, who assisted with providing teacher in-service training information; MTTA staff member Douglas Mbingwa, who provided field information, and the following Ministry of Education personnel and MTTA District Education Facilitators, who assisted with providing information and facilitating school visits: Mr. Christopher Khoropa Mzimba South District Education Facilitator Ms. Eneret Santhe Kasungu District Education Facilitator Ms. Emylda Bongwe Machinga District Education Facilitator Ms. Nerissa Boatman Phalombe District Education Facilitator Ms. Lucy Chiwala Kasungu PEA Mr. A.G. Louhanga Kamgawa Zone PEA Ms. Bernadette Sekeleza Machinga PEA Mr. Charles M. Gunsaru Zomba Malawi Institute of Education Mr. McKnight Kalanda Lilongwe Director of Basic Education, MoEST Ms. Darles Mbewe Lilongwe Acting Director, DTED Mr. Pathon Nyirongo Mzimba South District Education Manager Mr. Dudley Chiala Central East Education Division Manager A special thanks also goes out to the head teachers, teachers, and students who participated in interviews for this study. David Chapman and Suzanne Miric The Evaluation Report on the MTTA Life Skills pre-service teacher training component was conducted in collaboration with Mr. Sydonio Matope, Save the Children US, and Ms. Joy Mueller, MW&A Consulting Associate, who participated in data instrument development and data collection. Other individuals to whom thanks are due for making this study possible are Mr. Simeon Mawindo, MTTA Chief of Party, Dr. Hartford Mchazime, MTTA Deputy Chief of Party, and the MTTA staff, all of whom offered support and encouragement; and MTTA Learning Materials Coordinators Ms. Kara Janigan (2004 – 2006) and Ms. Laura Ivey (2006 – 2008) who provided reflections on their involvement in the project. Special thanks to the principals and Life Skills tutors of Blantyre Teacher Training College, Lilongwe Teacher Training College, and St. Joseph’s Teacher Training College, Dedza and to the students of these colleges, their pupils, and the school principals who participated in this study. Finally, our sincere thanks to Ms. Marisol Perez, USAID/Malawi Education Team Leader, and Mr. Ramsey Sosola, USAID/Malawi Program Management Specialist, for their interest and encouragement in undertaking this summative evaluation. Shirley Miske v Part A: MTTA In-Service Teacher Training Component Evaluation By David Chapman and Suzanne Miric A-1 EXECUTIVE SUMMARY MTTA has been remarkably successful in promoting teachers’ professional engagement, enthusiasm for teaching, and desire to upgrade their teaching practice. It has energized the teaching force in the four districts in which it has worked by building in-service training around content needs identified by teachers, by giving teachers sustained attention and support via supervision visits and on-going cluster and school level training, and by providing recognition and status for those teacher with higher levels of competence. It has provided tangible support to weak teachers while utilizing the skills of stronger teachers in ways that have recognized and motivated both groups. MTTA infused schools with training opportunities, on-going instructional supervision and attention, books and resource materials. It created professional stimulation and novelty in an environment in which those are often lacking. It gave teachers a sense that their work was important, that they were valued, and that help was available. It introduced a structure of local resource people based on competence and merit, rather than seniority (on which the Ministry of Education, Science, and Technology [MoEST] system is largely based). It has done these things at a remarkably low cost, which increases the feasibility that the Malawi MoEST could sustain and extend this model to other districts, if it chooses to do so. MTTA appears to have made a substantial positive impact on how Malawi teachers, headteachers, and district education personnel view their own responsibility for the quality of primary education. At the same time, it provided them with tools with which to improve the quality of that education. Evidence of this increased teacher engagement and professionalism comes largely through observations and interviews with MTTA project district, zonal, cluster, and school level personnel and with central, divisional, district level MoEST personnel. Enthusiasm for MTTA was unanimous at all levels. Literally thousands of teachers have volunteered their time as Zonal In-service Facilitators (ZINFAs), mentor teachers, trainer heads, troupers, teacher facilitators, and participants in MTTA activities, a level of volunteerism that is extremely impressive. Teachers report that they do this because they value the professional development opportunities available through MTTA. Thousands of teachers have literally “voted with their feet.” The extent to which three years of MTTA activity has had an impact on teacher content mastery or student learning is less clear. Results of a longitudinal reanalysis of a sample of teacher content mastery data indicates that gains in teacher mastery were small at best. Possible reasons for the lack of notable change in teacher content mastery are that the assessment test was only somewhat aligned with the content provided by MTTA, and the five cycles of in-service teacher training (a primary intervention of the MTTA project), were not fully delivered to the teachers. Teachers were intended to complete five three-day cycles of in-service training by the end of year 2, during which they would have received about 15 days of primarily content training in each of the three subject areas (science, math, English). After the Malawi Education Support Activity (MESA) project was terminated prematurely, pedagogical teaching strategies were also incorporated into MTTA’s in-service training. Based on instructions from one of the international partners, the intended three days of training in each cycle were reduced to two days during each of the last two workshops, effectively reducing teachers’ content instruction to 75% percent of the intended treatment. This meant that either teachers did not cover all topics in the training handbook or that they rushed through them with little or no time for clarification of difficult topics. In short, MTTA teachers did not receive the full amount of intended training as originally envisioned, in part because of the need to devote time to teaching methodologies, and in part because of the decision to shorten the training days. A further complicating factor was the absence of a comparison group, A-2 needed to determine if the changes in teachers’ content master could be attributed to the project or whether they would have occurred anyway. Pupil achievement in MTTA schools increased, but the lack of longitudinal analysis, absence of a comparison group, and lack of national assessments of pupil achievement at grade levels prior to Standard 8 make it impossible to attribute these gains to MTTA interventions. However, while pupil achievement gain is the ultimate goal of this project, it may be unreasonable to expect measurable gains to emerge within the three-year time frame of MTTA. Teacher Professional Development Conferences (Cycles 6 and 7), which introduced peer-teaching among teachers, were widely regarded as successful. The MoEST’s Continuing Professional Development Technical Manual that orients teachers to school-based professional development as part of the roll-out of the new national curriculum Primary Curriculum Assessment Reform (PCAR), draws heavily on MTTA’s use of Teacher Professional Development Conferences. This is a testament to the MTTA project’s success in many respects at embedding good practices within the wider educational system and achieving high levels of sustainability. Similarly, headteachers reported that the training of headteachers as instructional leaders was well received and considered effective. There are other measures of progress among teachers in the four MTTA districts that MoEST and project staff thought might reflect project impact. In particular, many teachers saw MTTA as helping them in preparing them to take or improve their scores on the Malawi Secondary Certificate of Education (MSCE), which in turn helps them move toward college admission or obtaining a more lucrative job in the banking or other sectors. In the current context of Malawi this is particularly important since education is one of the few routes to upward professional mobility. However, an analysis of the change in MSCE holders in MTTA and non-MTTA districts indicated a lower rate of increase in MTTA districts. Previous research suggests, however, that data on the professional qualifications of teachers may be more valid in some districts than in others (Miske Witt & Associates, 2007). SUMMARY OF KEY FINDINGS 1. MTTA has been successful in promoting teachers’ professional engagement, enthusiasm for teaching, and desire to upgrade their teaching practice. It has energized the teaching force in the provinces in which it has worked. 2. Given the remarkable level of voluntary commitment to professional development within this project, the MTTA represents a low cost model for upgrading teachers and facilitating a sense of professionalism that builds on and strengthens local talent. 3. MTTA has contributed to strengthening the content mastery and pedagogical knowledge of some teachers. However, overall increases in teachers’ content knowledge, as measured by the MTTA test of teacher content mastery, appear to have been modest. A possible reason for the lack of greater gains in content mastery is that only 75% of the intended in￾service training of teachers appears to have been delivered. Other reasons may pertain to test validity, and lack of teacher interest in taking the test. Teachers report that the training in content and pedagogical methods that was delivered has been helpful, and in at least some cases was enthusiastically adopted in a self-sustainable way at the school level. 4. Pupil achievement has increased over the period of this project but the extent that these gains exceed those that would have occurred without MTTA cannot be determined from available data. However, it is not reasonable to expect measurable increases in pupil achievement by the third year of this project. Teachers will need several years to consolidate and practice teaching the new content before pupil achievement can be considered a valid measure of project success. A-3 INTRODUCTION The Malawi Teacher Training Activity (MTTA), which operated from September 2004 to July 2008 (three years followed by a seven-month extension), was aimed at improving the content knowledge of Malawi primary teachers through a set of pre- and in-service training and a variety of teacher support activities, including the provision of training manuals and other resource materials. The project was implemented in four of the 33 districts across Malawi. With the early termination of MESA, the MTTA scope was modified to also include some training in pedagogical methods that had been intended for delivery through MESA. This evaluation was commissioned to provide an end-of-project assessment of the effectiveness and value of the in-service teacher training component of MTTA. A two-person evaluation team was in Malawi from May 24 – June 9, 2008 to conduct this study. It should be noted that two other components of MTTA, the Life skills pre-service teacher training component and the HIV/AIDS Club Initiative (HASCI), are addressed in separate evaluations and are not addressed in this report. This evaluation is framed around three key questions: 1. How and to what extent were the inputs delivered? 2. How and to what extent were the inputs used? To what extent did beneficiaries believe the inputs were relevant, appropriately delivered, and effective? 3. To what extent did these inputs make a difference for teachers and pupils? Specifically, to what extent did in-service teacher training result in improved content knowledge and pedagogical practices on the part of teachers and greater learning on the part of pupils? The evaluation is framed within two models. Initially MTTA will be examined from a goal attainment model, in which the concern is the extent to which intended activities were undertaken and objectives achieved. Subsequently, the project will be examined from the perspective of Galal’s (2002, 2008) model of educational reform. From this perspective, improving an education system requires attention to three dimensions. First, there needs to be sufficient inputs to the education system and those inputs need to be combined in ways that constitute effective instructional processes. Second, there needs to be attention to teacher motivation, e.g., there needs to be sufficient incentives and an accountability system that ensures that those incentives go to individuals who deserve them. Finally, there needs to be a professionalization of the teaching force, a way to build pride among educators and encourage them to take greater responsibility for the outcomes of schooling. The following sections will present the methodology used in this evaluation and a brief overview of the MTTA project, summarize the extent inputs have been delivered and used, the response of Malawi educators to this project, and evidence related to project impacts. The final section of the report will summarize the evaluation findings and offer cross-cutting observations regarding the success of MTTA. METHODOLOGY This evaluation was conducted from 20 May – 11 June 2008. The evaluation employed multiple measures that included (a) a review of project documentation, (b) a summary of pupil- and school￾level achievement data, (c) site visits to two schools in each of the four districts in which MTTA was working, and (d) interviews with project staff and divisional, district, zonal, cluster and school-level educators. The purpose of the interviews was to assess the perceptions of project relevance, appropriateness and effectiveness held by multiple stakeholder groups. In each district the team also met with the District Education Manager (DEM) or Assistant DEM, Primary Education Advisors (PEAs), Teacher Development Center (TDC) staff, ZINFAs, cluster leaders, and/or training heads. A-4 This evaluation was based primarily on a review and synthesis of data already available from the project, the Government of Malawi (GoM), and the schools. The team also undertook limited original analysis and selective re-analysis of existing MTTA monitoring and evaluation data. MTTA project analyses of teacher and pupil knowledge gains were undertaken as comparisons of teachers at different points in time and of students at different points in time, but pre- and post-data were not linked at the individual level that would have enabled a longitudinal analysis for either teachers or for students. Hence, re-analyses were conducted to assess teachers’ and pupils’ knowledge gains within a longitudinal framework. Original analysis was conducted on teacher interview data that had been collected in 2007 but not yet analyzed (due to turnover in project staff). BACKGROUND: MALAWI AND MTTA [Readers familiar with the structure of the Malawi education system and the design of the MTTA project may wish to skip the next two sections. Readers new to the MTTA will need this information in order to interpret subsequent findings.] Like many other countries in the region, Malawi follows an eight-four-four formal education system. The first eight years are primary education, followed by four years of secondary and four years of tertiary each. Each year in primary education is referred to as a standard, with primary education ranging from Standard 1 to Standard 8. The official entry age for primary education is six. In 2004, Malawi had 5,321 primary schools, with 43,197 primary school teachers and 3,280,714 primary school pupils. About half of the almost one million school-aged orphans in Malawi have been orphaned by AIDS (UNICEF, 2008). The MoEST oversees the education system. Primary education headquarters are located at the ministry, and are divided into six education divisions. Within the education divisions there are a total of 33 district education offices, headed by a DEM. Districts are in turn divided into education zones. Each education zone, which normally includes 10 to 15 primary schools, has a TDC and is headed by a PEA. The number of zones in the four MTTA districts range from 26 in Mzimba South, to eight in Phalombe. Within zones there are clusters, a concept introduced by MTTA. Clusters usually consist of two to four schools, and teachers can usually walk to other schools within their cluster within a few hours. Both pre-service and in-service training of teachers are overseen by the Department of Teacher Education and Development (DTED). Approximately 50% of the GoM education budget is provided through donor assistance. In 1995, the government abolished school fees in all government primary schools. This resulted in high enrollment rates, with overall enrollment by October 1994 swelling from 1.5 to 3.2 million students (Mchazime, 2008). Abolishing school fees also led to a shortage of 22,000 teachers. The government responded by hiring 22,000 untrained teachers, most of them secondary school leavers (Mchazime, 2008). These teachers were given 10 days of basic teaching skills before being deployed. In 1996, the government launched an emergency teacher preparation program, the Malawi Integrated In-service Teacher Education Program (MIITEP). The MIITEP provided a 16-week in￾service training to newly recruited teachers along with distance learning materials, considered to be equal to the two-year normal period of training for primary school teachers. Most teachers, however, found it difficult to cope with distance learning materials. Consequently, MIITEP was perceived as failing to meet professional standards of teaching (Mchazime, 2008). MTTA was a response to the continuing sub-standard teaching skills. Begun in September 2004, MTTA was intended to improve the quality of education in Malawi through improvement of primary school teachers’ professional skills. In particular, MTTA’s focus was on building teacher content knowledge in English, mathematics, and science. Improving content knowledge was intended to complement Malawi’s historical focus on pedagogy in its pre-service teacher training programs. It also complemented another USAID funded project, MESA, which had begun two years earlier in 2002. MESA assisted communities in supporting schools and provided in-service teaching methodologies training. Because MESA was already operating in four districts, MTTA also targeted A-5 the same districts – Mzimba South, Kasungu, Machinga, and Phalombe. When MESA was phased out prematurely, MTTA picked up key aspects of MESA by incorporating pedagogy into the MTTA training. OVERVIEW OF MTTA MTTA is a multi-component project. The components complement each other, yet are distinct enough so that each could be replicated or sustained independent of each other. The seven components within MTTA’s in-service teacher training program that particularly add value to this project are described below. Again, readers familiar with the project may wish to skip to “Main Findings” section below. Cycle training: A series of five in-service workshops were held for teachers in four districts during school holidays in 2005 and 2006. The workshops utilized a cascade model and were originally scheduled to last three days. Early workshops were based on teachers’ identification of what they wanted to learn in three content areas: English, science, and mathematics, and on what the National Core Trainers believed teachers needed to know. Later workshops also incorporated content based on a needs assessment (e.g., the 2004 baseline study used for gauging teacher mastery of content) of teacher knowledge in English, science, and mathematics, a mathematics curriculum analysis, and a pupil assessment that drew items from the Malawi Improving Educational Quality (IEQ). At each workshop teachers received a cycle training handbook, which included topics covered in the workshops. Trainers also received a 50-page resource guide for each of the subjects, which compiled key content areas from the trainings and from the sources used to compile the training guides. The cascade model utilized a top-down approach with three basic levels. 1. The top level consisted of the National Core Group of trainers. Its 35 participants included lectures from the Teachers Training Colleges and MTTA staff who received training as facilitators during a National Core Group Training Workshop. The teacher training handbooks were also developed at this level. 2. The second level consisted of the trainer of trainers (TOTs). The TOTs were trained by the national trainers. The number of TOT participants ranged from about 300 to 500, and included PEAs, ZINFAs, head teachers, trainer heads, and cluster- and school-level mentors. The majority of TOTs were practicing teachers. PEAs, typically the only position with no teaching responsibilities, accounted for only 9% of 582 participants in Cycle 3 training. TOT trainings were supposed to be held at the zonal level for two days at the beginning of every cycle. The training included content knowledge and pedagogy, and TOTs were given the opportunity to practice teaching. 3. The third level of training occurred at the cluster level, which usually consisted of two to five schools. TOTs delivered workshops to teachers within a cluster. Workshops were held at a cluster center (e.g., school). Although the cascade model has drawbacks, this component of the MTTA project was of value in the manner in which it empowered local personnel. Ordinary teachers could be selected as mentors and trained at the zonal level in facilitation and supervision skills. It also demonstrated the feasibility of identifying and developing training materials around high-interest, high-need teacher content learning. Figure 1 provides an overview of the project organization. A-6 District Education Manager (33 districts in Malawi) District Education Facilitator (MTTA created position) Coordinating Primary Education Advisor Primary Education Advisor (one PEA per zone), heads the Teacher Development Center (TDC)S ZINFA (in most cases a TDC Assistant Coordinator; in some cases a teacher); assists PEA administratively – MTTA created volunteer role as teacher supervisor/ professional School￾based teacher development facilitator Cluster-based mentor (teacher within cluster recommended by PEA, approved by DEM – MTTA created role) Head teacher (MTTA created instructional leader role) School-based mentor (teacher within school recommended by PEA, approved by DEM – MTTA created role) Trainer head works at school & cluster level (teacher within school recommended by PEA, approved by DEM – MTTA created role) Mobile Teacher Troupe Trainer (PEAs, ZINFAs, mentor teachers, retired teachers); MTTA created role – visits zones or clusters School-based teacher District Level Zonal Level Cluster Level School Level Figure 1 Key: red boxes = government roles with added MTTA professional responsibilities; blue = roles created by MTTA Teacher Professional Development Conferences: Teacher Professional Development Conferences (TPDCs) replaced workshops in Cycles 6 and 7. The two- to three-day conferences were held in November-December 2006, and April 2007. TPDC marked a dramatic change in MTTA’s training approach, in that it flattened the cascade level and shifted training responsibilities and expertise to the teacher. Practicing teachers were asked to identify topics for training and to submit them, through a head teacher, to a Cluster Committee. Teachers could both suggest topics and volunteer to peer teach for an hour. The Cluster Committee selected topics and facilitators, and TPDC conferences were then held at the cluster level. Because ordinary teachers and TOTs (mentor teachers, head teachers, and ZINFAs) co-facilitated at the same level, teachers realized they sometimes knew more than the TOTs. Teachers were also able to recognize expertise among their colleagues, which varied according to topic and subject area. Teachers reported finding the peer teaching model extremely useful and engaging. Not only did the TPDCs increase their content knowledge and confidence in their teaching skills, but it A-7 encouraged a sense of camaraderie among teachers with different skill and motivational levels. Most topics covered in the TPDCs were drawn from senior primary school class lessons, according to MTTA staff who attended some of the conferences. Development of instructional supervision: Many new layers of instructional supervision were introduced through the MTTA project. These included District Education Facilitators (DEFs), ZINFAs, head teachers, mentor teachers at the cluster level, and mentor teachers at the school level. Only the DEFs are MTTA staff and thus in non-sustainable roles – the remainder of the positions were created as unpaid, volunteer responsibilities tied to existing positions. These positions include assistant coordinators at TDCs (e.g., ZINFAs). ZINFAs and DEMs report that the ZINFA role often was created by adding supervision and professional development to an existing position as assistant coordinators to PEAs, who also operate at the zonal level. The development of a multiple and overlapping supervision model was a valuable component of this project. It both introduced local levels of supervision that did not previously exist, and demonstrated the value of redundancy. Having multiple people able to supervise the same school and teachers meant unexpected events, such as illness or problems at another school, created less likelihood of a suspension of supervision. Teachers and supervisors reported that supervision is generally welcomed by teachers, as it is done in a collegial manner and offers a chance for teachers to gain recognition and receive assistance. Teachers in the MTTA project districts could expect supervision at least twice or three times a year, compared to almost non-existent instructional supervision previously. ZINFA facilitated teacher-training workshops: This initiative proved valuable in several ways. One, by organizing trainings at the zonal level, ZINFAs could be more responsive to the needs of teachers in their zone and increase the amount of in-service trainings available to teachers, sometimes significantly. Two, ZINFA-facilitated workshops introduced a new understanding of the value of workshops to teachers. Because teachers were not paid for attending ZINFA-facilitated workshops but did so anyway, a new workshop model was introduced that did not involve an “allowance syndrome” (e.g., teachers attending workshops because they are paid). ZINFA-facilitated workshops began in 2005, and were held at TDCs. They supplemented cluster trainings and the TPDCs. In 2006, 71 such workshops were held, with 2,964 participating teachers. In some cases, long distances from a school to a TDC hampered the involvement of teachers. Head teachers as instructional leaders: Until the MTTA project began, headteachers in Malawi had not received instructional leadership training since 2000. Rather, new headteachers, who are usually recommended by PEAs, receive administrative training on topics such as how to make official requests to the government. MTTA staff became aware of the need for headteacher instructional leadership while collecting data in 2005 and 2006. After recognizing this gap existed between administrative and instructional skills, the MTTA staff developed a headteacher training handbook in 2007. Unlike the more comprehensive resource guide developed by the Malawi Institute of Education (MIE) for headteachers in 2004, the MTTA headteacher training handbook targeted topics such as teaching, learning, and assessment in a condensed, easy-to-read format. The MTTA project produced and distributed an initial 1,000 copies of the handbook to headteachers, DEMs, DEFs, PEAs, and ZINFAs throughout the four districts in early 2007. A revised version was produced and distributed to deputy headteachers later in the year. Two training sessions were conducted to introduce the handbook to the school leaders. The first training was geared toward PEAs, ZINFAs, and others who had previously participated in TOT trainings during Cycles 1-5. The second directly involved headteachers. Follow-up interviews in Phalombe District indicated that headteachers found the information interesting and useful, and that both headteachers and teachers thought the behavior of headteachers in schools had changed. This has led to a value-added role for headteachers as not only administrative leaders, but instructional leaders. (See Box A). A-8 Resource materials and books: The development and provision of learning materials for teachers to study on their own time and at their own pace was a key component in this project. Two types of material were available to teachers. First, resource guides in English, science, and mathematics were distributed to every trainer in the MTTA project districts before or during the time of the Cycle 3 training, in July-August 2005. Called “In-Service Resource Manuals,” these manuals include concepts and topics that were identified earlier by teachers through interviews, assessments, and classroom observations. These topics and concepts, such as factors and fractions, were also covered during MTTA’s cluster-level teacher training workshops. The resource guides compiled some of the information to which teachers had previously been introduced at trainings, along with pupil-centered teaching strategies for each subject area (e.g., English, science, and mathematics). Second, books from the International Book Bank (IBB) were distributed to the 54 Teacher Development Centers (TDCs) in the four project districts. Headteachers were able to borrow and return books on behalf of their staff, thereby making books available at schools. Teachers and pupils were also able to borrow books directly from the TDC libraries. Teachers reported that both types of material were useful, and that they frequently made use of them to improve their content knowledge, improve their teaching practice, and to upgrade their professional qualifications. For at least some teachers, resource guides provided critical knowledge which in turn enabled them to use and understand the books. Box A: A Teacher’s Perspective: Instructional Supervision by a Headteacher When Moliday Ulanda Nderi, 41, learned her headteacher would visit her classroom to observe her last month, she didn’t become nervous or anxious. Instead, the 41-year-old teacher had become accustomed to viewing her headteacher as someone who could help her teach better. She planned the lesson for her Standard 8 learners carefully. The headteacher arrived at 7:35 a.m. and stayed to observe until 8:10 a.m. Afterward, the headteacher and Nderi discussed how Nderi could improve her teaching in composition. Then, as the discussion continued, Nderi raised a question that had been on her mind. “How do you make pupils study on the weekends?” she recalled asking her headteacher. “He told me, ‘You have to group them at the place where they live’. That made sense to me,” said Nderi, who lives in the largely rural district of Mzimba South. “I tried it, and it worked better.” Mobile Teacher Training Troupe: The Mobile Teacher Training Troupe (MTTT) proved a valuable component of this project through its targeting of poorly functioning schools to increase content knowledge and pedagogical skills among teachers. Drawing on an innovative intervention tailored to fit the needs of educators, MTTT sent teams of retired teachers and PEAs to schools for a week. The teams spent two days observing classes, one day giving feedback, one day demonstrating teaching methods and overseeing peer teaching, and one day supervising a volunteer teacher demonstration. Thirty schools were originally selected for visits by PEAs throughout the four districts. Later, the number of targeted schools was increased to more than 120, with the number of troupers increased from six to 31. MTTA staff reported that this intervention was so popular that all the districts initiated their own local versions of the MTTT. Further statistics describing MTTA are presented in Annex A. The project has four particularly noteworthy characteristics: 1. During the first two years, the project organized in-service teacher training in a rather top￾down manner, in which the training plan and materials were formulated by project staff in consultation with the Malawi Institute of Education and the Ministry of Education (MoEST). Specifically, content was based on teacher identification of high-priority topics, a teacher needs assessment (e.g., the 2004 baseline study), and item analysis of a pupil achievement test. During year three, the project employed a bottom-up approach in which teachers A-9 identified their own in-service training needs, organized training at the level of school clusters, and teachers themselves designed and delivered that training. Although both approaches adopted a user-oriented design, they differed in scope (e.g., teacher needs aggregated across four districts, versus teacher needs aggregated within an individual cluster). 2. Teachers’ participation in both top down and bottom up training was intrinsically motivated. Except for meals, they were not compensated or rewarded for their participation by the project. Likewise, mentor teachers did not receive rewards for their participation other than the training and resource guides. ZINFAs, who were sometimes practicing teachers, received bicycles, which enabled them to visit other schools in their cluster in a timely manner. Bicycles are equivalent to about one month’s salary for a new teacher. This kept costs low and increased the feasibility (if not the likelihood) of sustainability. 3. Local teachers were used as instructors in both the top-down and bottom-up training. This approach recognized and honored teachers based on merit in a system that is largely seniority based. 4. The project attempted to address wide-ranging and dramatic knowledge gaps among teachers, by following a non-sequential and priority-driven approach to covering topics in English, science, and mathematics. For example, teachers who attended “second chance” community day secondary schools instead of government secondary schools in many cases never had any science courses themselves. MAIN FINDINGS How and to what extent were the inputs delivered? MTTA reports that it met and exceeded nearly all its targets for inputs in support of in-service teacher training and on-going support to those teachers. This included the delivery of training handbooks for each cycle of training; instructional handbooks for head teachers and deputy head teachers in 2007, and resource manuals for English, science, and math (which were delivered around the third cycle of training). The core deliverable, in the sense that all other inputs were designed around it, was the sequential series of in-service teacher trainings. By the end of the project, MTTA had held seven cycles of in￾service training (the last two of which were teacher professional development conferences) for essentially the same teachers in the four target districts. Participation in the training ranged from 5,925 to 6,389 teachers across the seven cycles, as detailed in Table 1. This fluctuation was due to teacher transfers, attrition, and absenteeism. Table 1: Number of teachers participating in each training cycle Cycle Female Male Total 1 1768 4621 6389 2 1654 4304 5955 3 1568 4664 6232 4 1618 4407 6025 5 1495 4602 6097 6 1486 4154 5640 7 1430 4495 5925 Source: MTTA monitoring and evaluation reports. A-10 However, actual implementation of the cycle training differed considerably from the initial plan. The project intention was that teachers would each participate in five three-day cycles of in-service training. Content upgrading would have occurred in each of the three subject areas in each of the cycles, which would have resulted in teachers having competed approximately 5 days of in-service training in each subject of the first two years of MTTA. Teachers actually received 25% less. In response to a decision of one of the international partners, the length of the teacher training in each cycle was cut to 2 days for some cycles, ostensibly for financial reasons. This meant that some classroom teachers received less training than anticipated. Table 2 illustrates length of training as reported by school personnel. Consequently, one of the primary project interventions was not fully delivered. This later would make it difficult to determine whether the lack of meaningful increases in teacher content mastery was due to the original design of the project, the project’s adaptability in incorporating pedagogy when USAID terminated the MESA project prematurely (thereby reducing the time spent on content upgrading), or the reduced number of training days. Table 2: Number of days of in-service training as reported by selected school personnel Cycle 1 2 3 4 5 Core training 5 5 5 5 4 TOT training A 3 3 3 3 2 TOT training B 5 5 5 4 4 Cluster A training 2 2 2 2 1 Cluster B training 3 3 3 3 3 Note: Different TOT trainings and clusters are designated as A or B. Source: School visits Teachers report that cluster trainers responded differently to the need to deliver the training in a shorter amount of time. In some clusters, trainers increased their pace in an effort cover all the material and did not take time to answer teacher questions. This left some teachers confused about things they did not understand. In other clusters, trainers tried to answer teacher questions but were only able to cover part of the material. Compounding this issue posed by shortening the training was that the teacher trainers themselves did not always have full mastery of the content they were teaching. Core training was conducted for 5 days in Cycles 1-4, and reduced to 4 days in Cycle 5. However, information collected during school visits indicated that the length of TOT training may have varied by location, ranging from 2-5 days per cycle. To familiarize TOTs with the tests teachers would be given, TOTs were given pre￾and post-tests – which also had the unintended consequence of being able to assess their level of content mastery, although individual scores were not disseminated. The tests were based on items from the Primary School Leaving Exam (PSLE) administered at the end of Standard 8, and included English, mathematics, and science. Post-tests suggest that the majority of TOTs had not yet mastered content subject areas even after a week of training, although their scores are considerably higher than those of teachers. After the third cycle of training, for example, only 26% of TOTs had achieved full mastery of the content. After the fifth cycle of training, 21% of TOTs had achieved full mastery of the content. Although the training of TOTs resulted in learning for a significant number of TOTs (e.g., 14% of participants moved from partial to full mastery of content by the fifth cycle), only approximately a quarter of TOTs had fully mastered the content delivered during TOT training and which they were to subsequently deliver to teachers. Teacher professional development conferences (Cycles 6 and 7) were conducted in all the clusters in the site visit sample. Teachers liked these sessions because they so directly addressed their needs. Along with this training, teachers were supported with on-going school and classroom visits by PEAs, ZINFAs, cluster mentors, training heads, and headteachers. Based on information from the school visits, most teachers received about three instructional supervision visits each academic year, A-11 though some reported as many as six visits. Teachers also remarked on the importance to them of the reference books provided by MTTA and the IBB. Teachers commented that the books and reference materials by themselves would not have been as useful without the cycle training, because the training provided them with skills they needed to make the best use of the print materials. A summary of project inputs is presented in Annex B. In summary, MTTA in-service training appears to have reached the intended number of teachers, but (a) the amount of training delivered to those teachers was somewhat less than initially intended and (b) those delivering the training had not necessarily mastered the content themselves. Nonetheless, teachers, headteachers and other stakeholders were positive about the training and other project activities that were delivered. How and to what extent were the inputs used? To what extent did beneficiaries believe the inputs were relevant, appropriately delivered, and effective? Educators at every level of the education system indicated that the cycle training, zonal and cluster initiated training activities, follow-up supervision visits, and resource materials were widely implemented and greatly appreciated. Particularly in rural areas, teachers suffer from professional isolation. The instructional supervision visits were widely regarded as stimulating and helpful, and were positively received. During September 2007, MTTA staff conducted interviews with 341 teachers to assess their level of involvement in of MTTA activities. While this data has not previously been analyzed, the evaluation team, in collaboration with the monitoring and evaluation specialist, summarized responses four items of particular relevance to this evaluation. In one question, teachers indicated the number of MTTA training sessions they had attended. Of the 187 teachers responding to the item, 81% indicated that they had participated in five or more training sessions over the three years of the project (Table 3). These included the cycle training, teacher professional conferences, and training sessions organized at the zonal, cluster and school levels. The largest number of teachers reported having participated in 6-7 training sessions. Since teacher mobility in Malawi may be as high as 40%, according to an MTTA assessment, the number of training sessions attended by teachers who did transfer into the district after MTTA had started could be higher. Table 3: How many in-service training sessions organized by MTTA have you attended so far? (187 respondents to this item) Number of training sessions Number of teachers attending this number of training sessions 1 6 2 4 3 8 4 17 5 15 6 33 7 75 8 8 9 11 10 8 11 0 12 2 Number of teachers responding to this question 187 Source: Raw data obtained from the MTTA Monitoring and Evaluation Unit A-12 A key aspect of the MTTA treatment is that teachers receive ongoing support, guidance and encouragement through a series of instructional supervision visits by PEAs, ZINFAs, Cluster teachers, and headteachers. Of the 223 teachers responding to this item, 165 (about 80%) indicated they had been observed three of more times over the course of the 2007 school year (Table 4). At the same time, the data suggest considerable variability in the frequency teachers get observed, with 25% of the teachers reporting they had been observed only once or not at all. Visits to eight schools indicated that ideas from the MTTA training were being implemented. Several classrooms had science corners, shopping corners, weather stations, and word trees demonstrating key concepts associated with science, math and English. Teachers were able to recount the content of zonal and cluster level training sessions they had attended. At the same time, they report these sessions tended to be short (about 30-40 minutes) and that transportation to training sessions at venues away from their school sometimes posed a problem, as MTTA did not provide transportation funds. While the cycle training concentrated on content upgrading, the element most frequently identified as useful by teachers was the training in how to get large classes working in small groups. With class sizes regularly exceeding 100 per class in the lower primary grades, this reaction is understandable. In general, teachers seemed more attuned to the pedagogical practices they had learned than the content lessons in which they had participated, although several teachers noted that female teachers in their schools were now more comfortable teaching the content of upper grade levels. Historically, upper primary grade levels have been the preserve of male teachers. Table 4: How many times have you been observed teaching in your school this year? Number of times observed Number of teachers reporting this number of observations 0 7 1 51 2 63 3 45 4 31 5 8 6 10 7 4 8 4 Number of teachers responding to this question 223 Source: Raw data obtained from the MTTA Monitoring and Evaluation Unit In every school visit, teachers and headteachers mentioned the importance of the books and materials provided by MTTA and IBB (see Box B). These reference materials helped reduce the complexity of teachers’ work by giving them a further source of information on topics they were teaching. It also helped teachers in studying for the MSCE exam. The importance of these materials is reinforced in results from the interview study, as nearly 60% of the teachers reported that they personally borrowed a book from the TDC or cluster center libraries (Table 5). A-13 Box B: How Reference Materials Are Used at a School Fanki Chikapa, a Standard 1 teacher, studied MTTA reference materials and books every day. Within a short time, he succeeded to pass the MSCE. But it wasn’t a simple process. Without face-to-face trainings offered through MTTA cycle trainings, he wouldn’t know how to use the reference guides, he said. Without reference guides, his English would not have been good enough to understand the books. So Mr. Chikapa found it impossible to say which type of assistance (e.g., the training, guides, or books) was the most important. “I learned English from those books,” he said proudly, referring to the MTTA training handbooks and reference guides. “If you go to the training, using the books is easy.” Mr Chikapa said if books weren’t available through MTTA, he would bike about 100 kilometers to reach a book shop. The trip by bike, along mostly dirt roads in Phalombe District, takes three days. Books cost from 2,000-5,000K each. He appreciates the availability of books through MTTA. “Add more books, if possible,” says Mr Chikapa, who adds he is glad he can now teach higher level classes. “Those books for grammatical English. Math books. I read everything!” Table 5: Have you personally borrowed a book donated by MTTA from the TDC or cluster center? Borrowed a book Number of teachers Yes 119 No 82 Number of teachers responding to this question 201 Source: Raw data obtained from the MTTA Monitoring and Evaluation Unit Teacher interviews also suggest that the value of reference materials can be further leveraged by providing the right mix of training, beginning-level reference materials, and more advanced reference materials. The degree of accessibility is also factor. At least one district had decentralized the distribution of IBB books, allowing them to be kept at the school cluster level. This proved to be popular and successful decision. Providing reference materials may also reduce costs associated with teaching, both in terms of spending money on the private acquisition of learning materials, and in terms of time spent away from school. One of the strengths of the MTTA project is that it honors local expertise. MTTA has created roles for particularly strong teachers, as ZINFAs, cluster coordinators, trainer heads, and mentor teachers. This has been motivating and energizing to a system in which, previously, there were few formal means through which effective teacher could be recognized on the basis of their performance. Asked who they felt was the greatest support in helping them with their teaching, teachers overwhelmingly identified headteachers and PEAs (Table 6). The helpfulness attributed to headteachers may reflect their proximity and their role in signing off on lesson plans, class assignments, etc. Likewise, PEAs, who report to DEMs and visit schools on a regular basis, are in a position to recommend teachers for promotion, changes in deployment, etc. In addition to providing classroom supervision, headteachers and PEAs are able to provide a broad range of assistance to teachers. A-14 Table 6: Who do you feel is the great support in helping you with your teaching? Source of support Number of teachers seeing this source as most important Headteacher 108 PEA 62 Mentor teacher 26 Others -- Other teachers 9 -- MTTA troopers 1 -- ZINFAs 1 -- All of them 2 Number of teachers responding to this question 209 Source: Evaluation interviews Mentor teachers, a new role created through the MTTA project, were the third most frequently cited source of support by teachers. Mentor teachers were chosen about half as often as Primary Education Advisors, and about a fourth as often as headteachers. This suggests at least some mentor teachers effectively engage teachers. Since mentor teachers do not have formal supervisory powers, the form of assistance they offer may be of a ‘softer,’ more collegial, variety. (See Box C). ZINFAs are another role created through the MTTA project. While teachers did not appear to view ZINFAs as their greatest source of support, a number of PEAs reported that the greater professional role of ZINFAs in organizing trainings and visiting schools in a professional development capacity was helpful. Lucy Chiwala, a PEA in the district of Kasungu, said managing 225 teachers on her own is challenging. . MTTA’s training of ZINFAs and headteachers as instructional assistants has been helpful.It has increased the amount of instructional supervision occurring at any one time in her zone. It has also facilitated a more forceful leadership presence, in that sometimes the PEA, the ZINFA, and headteachers visit schools together as troupers. Ms. Chiwala believes the success of MTTA’s approach to encouraging ZINFAs to assume roles as professional development facilitators was that MTTA clarified the different roles for PEAs and ZINFAs at the beginning of the project. ZINFAs always check with her before proposing activities, she said. “We were told the roles of the PEA and ZINFA were not the same – that is how MTTA introduced it,” Ms. Chiwala said.. “The introduction was straightforward – that is the key. They don’t do anything without us knowing.” Another way teachers used the training they received through MTTA was to further their own career advancement. The MTTA project notes that over the period of the MTTA project there was an increase in the number of primary school teachers who held a Junior Certificate of Education (JCE) awarded after ten years of schooling and who completed secondary education and passed the exit examination and were awarded a Malawi School Certificate of Education (MSCE) (Table 7). While it is not possible to determine the role played by MTTA activities in fostering this upgrade in credentials, MTTA actively encouraged JCE teachers to complete their MSCE and it is likely that MTTA had some role in this increase in academic qualifications. Nonetheless, when the change in the proportion of MSCE holders in MTTA districts is compared to non-MTTA districts, it appears that MTTA districts are actually lagging behind other districts (Table 8). It should be noted, however, that these rates of change were influenced by the Ministry’s allocation of new teachers as well as by the rate at which existing teachers upgraded their credentials. A-15 Box C: A Mentor Teacher’s Perspective on Joint Lesson Planning Barnard James Nywongo was unsure what to expect when he arrived at 7 am to supervise a math teacher at his school. Would the teacher perform well? Would the teacher achieve the objectives of the lesson? When a teacher has significant trouble implementing topics covered in the MTTA trainings, Nywongo, a mentor teacher in the district of Mzimba South, suggests jointly preparaing a lesson plan. Some teachers continue to “go in the wrong direction” even after a workshop or initial supervision. They may be sick, or have other issues, Nywongo explained. Joint planning is a more intensive form of mentoring. It “roots out problems” and allows mentors and teachers to share skills and experiences. So Nywongo was pleased to find the teacher had significantly improved. “This time, it was so encouraging,” said Nywongo, who had discussed the teacher’s strengths and weaknesses while jointly preparing the lesson. “With mathematics, you can easily look in learner’s notebooks and check the results. But after I saw the improvement, I knew I would need to continue to encourage the teacher.” Nywongo believes joint planning is particularly useful for working with weaker teachers. As a Mobile Teacher Training Troupe member, Nywongo often recommends the use of joint planning to headteachers in schools he visits. In cases when more than a few teachers demonstrate similar weaknesses, Nywongo recommends school-based inset training. Some teachers are slower than others. That is natural, he says matter-of-factly. Table 7: Shift in academic qualifications of MTTA teachers between October 2004 and July 2007 (reported as percent of teachers) Academic qualifications October 2004 October 2005 October 2006 July 2007 PSLCE 0.5 0.5 0.5 0.2 JCE 58.1 48.1 44.6 37.4 MSCE 41.4 51.4 54.9 62.4 Source: MTTA monitoring and evaluation reports Table 8: Change in proportion of teachers holding JCE and MSCE credentials, 2004-2007 2004 JCE 2007 JCE Change in % of teachers with JCE 2004 MSCE 2007 MSCE Change in % of teachers with MSCE MTTA TOTAL 3084 2334 -32% 2945 3225 +9.5% Non MTTA Districts TOTAL 20887 16417 -27% 16526 19954 +21% Source: Computed by authors, using MoEST data for 2004 and 2007 A-16 Teacher interviews during eight school visits indicated a remarkable level of positive regard and support for MTTA activities. Prior to MTTA, teachers in rural schools report receiving limited instructional supervision. The infusion of resources and personal attention in MTTA signaled to teachers that their work was important, that there were people to whom they could turn for help, and that people outside of their own school cared about their success. Teachers remarked on the value of the content training – particularly the value of the TPDCs. Many teachers skip over class lessons covering materials which they themselves do not understand. Teachers and mentor teachers report that the content coverage provided in the training encouraged them to sometimes cover lessons that they might otherwise have skipped. Of particular importance, teachers said, was the newfound freedom to demonstrate and address weaknesses and strengths in their content knowledge within a peer teaching context. Even mentor teachers were surprised at the extent teachers were able to collectively foster self-learning (See Box D). While they appreciated the content boost, they particularly liked the pedagogical strategies introduced by MTTA. In some respects, pedagogy and content were closely intertwined. For example, setting up a science corner, a shopping corner or a weather station involved pupils in practical problem solving activities in a way that advanced both content and pedagogy in the classroom. These strategies had the advantage of being clear, observable actions that teachers could take to help deliver content in large-enrollment classrooms. Working in small groups was a particular strategy that teachers often mentioned as helpful. Teachers suggested the project could be improved through the provision of more books, more training, and a longer life for MTTA. While topics covered in the training were identified through a survey of teachers themselves, not all teachers found all topics to be relevant. Nonetheless, there was an overall positive regard for the idea and the implementation of MTTA. To what extent did in-service teacher training result in increased content knowledge and improved pedagogical practices on the part of teachers and greater learning on the part of pupils? Changes in teacher mastery over duration of project: MTTA assessed teacher content mastery each October (August in 2007) by having teachers complete a short test using items drawn from the PSLCE. Mastery was defined as correctly answering 80%-100% of the items correctly; partial master was defined as correctly answering 31%-79% of the items; non-mastery was defined as correctly answering 0%-30% of the items. Table 9 summarizes the percent of teachers achieving mastery in each of the subject areas, by year, over the course of the project, as reported by MTTA. Since the teachers taking the test each year may have differed, these data need to be interpreted with caution. Table 9: Percent of teachers correctly answering 80% of items correct on a content test in their subject area Year Mathematics (13)* Science (31 ) English (9 with multiple parts ) 2004 3.5% 1.4% 10.7% 2005 6.3% 5.4% 13.8% 2006 8.9% 6.5% 17.1% 2007 13.0% 9.1% 21.5% Source: MTTA monitoring and evaluation reports * Number of items on content test indicated in parenthesis. A-17 Box D: A Mentor Teacher’s Perspective Charity Kamanga Chilinda, a 38-year-old mentor teacher, arrived at a Teacher Professional Development Conference in April with trepidation. Teachers in her school cluster had not been trained as facilitators. How would they handle teaching their own colleagues? “I was a bit worried how they were going to perform,” Chilinda recalled thinking as she waited for the two-day conference to begin at 8 am. “Sometimes, when it’s their first time to perform, teachers become shy. They lack confidence – how should they be in front of their friends, will they express themselves freely? With learners it works, because teachers are at a higher level.” But teachers that day performed as well as any mentor teacher, Chilinda said. In her own mind, Chilinda attributed the success of teachers to the fact they had sufficient training materials, and were fully prepared. “Some teachers had a bit of a problem at first,” explained Chilinda, who along with a ZINFA, a training head, and nine ordinary teachers facilitated the conference within the Kaphuta Cluster. “But after another day, they were used to it. Any questions that came in (that they couldn’t answer), they felt comfortable to throw back to their friends.” Chilinda believes teacher professional development conferences are useful because teachers can share topics and identify future sources of support. Sometimes teachers are afraid of people who have more ideas and concepts than they do and keep questions to themselves, explained Chilinda, who has been teaching since 1994 and is pursuing a diploma in theology via distance education. The experience of facilitating the conference also gave Chilinda and many other teachers the idea that they could also be facilitators. Teacher Professional Development Conferences were also more successful than regular trainings because of their structure. Cycles 1-5 often ended abruptly. Teachers had to rush home to cook after 5 pm.TOTs such as Chilinda were not able to cover every topic in the MTTA training guides in detail. By contrast, the April conference has led to follow￾up training. Since the conference, three trainings have been held at all four cluster schools, Chilinda said. Results show a clear progression in teacher content knowledge over the course of the project. Gains were small but teacher content knowledge increased to some extent in each of the subject areas each year. While a positive finding, five limitations need to be considered: 1. Psychometric properties of the tests used each year are not available, so the extent that variations in scores may be due to differences in item weighting or test reliability cannot be ruled out. For example, while the 13 test items on the mathematics test were the same each year, items were assigned different point values from year to year. The total number of points a teacher could score on the test varied from 38 to 42, depending on the year. 2. The content mastery tests given to teachers were designed to show improvement on target USAID indicators. Therefore, the extent to which the test was a valid measure of content knowledge gain is questionable. The topics covered in the trainings and resource manuals, although drawn from the 2004 Malawi National Examination Board (MANEB) PSLE, may not have been the same items tested by MTTA, although the MTTA test also used items drawn from the 2004 MANEB PSLE. This is because only a small section of the PSLE was covered A-18 in trainings. For example, of the 31 questions in the MTTA’s mathematics test, only nine questions directly corresponded to material in MTTA’s mathematics resource guide. Topics such as agroforestry, animal classifications, and white blood cells were not included in the resource guide, but were included in the test. While it can be argued that the test was a fair assessment of domain knowledge, the limited number and specificity of the items highlight the question of test validity. 3. MTTA analysis of change in teacher content mastery was based on an annual comparison of the groups of MTTA teachers who took the test each year (a cohort analysis). These teachers were based at the same 126 schools. Due to teacher mobility and absences, teachers taking the test somewhat varied from year to year and no individual identifier was assigned, making it difficult to conduct any longitudinal analysis of the same individuals over time. 4. No comparison group was used, making it impossible to determine if the changes were due to MTTA or would have occurred anyway. (This could have happened if there was an influx of newly trained teachers into the sample.) 5. Ancedotal evidence suggests that some teachers may not have applied themselves in taking the test, as they did not see its value since it would not affect their MSCE scores or likelihood of personal advancement. Individual scores were not reported back to teachers or schools. To the extent that the tests were measuring pupil learning, these data suggest that MTTA teachers, as a group, were increasing their capacity to teach the content in these subject areas. However, given the limitations, results should be regarded as indicative, at best. A central question, then, was the extent that the gains in mastery level reflected in the MTTA cohort analyses provided results similar to what would have been indicated had a longitudinal analysis of teacher knowledge gain been conducted over the course of the project. To examine this issue, a longitudinal sample of 123 teachers taking the teacher mathematics test in both 2004 and 2007 was constructed by matching based on school, gender, and professional qualifications (these had been frozen during this time). The sample was drawn from 11 of 33 schools in Mzimba South, and from three or four schools in the remaining three districts. This is not a random sample of all MTTA teachers but a convenience sample of the first 123 teachers for which a 2004/2007 match could be identified. While Mzimba South had a high teacher turnover rate during this time, the matching criteria provide a reasonable basis for estimating that the same teachers are being examined pre-post within a longitudinal framework. The teacher math competency test was composed of 13 items, where items were assigned different point values, for a total of 40 points. The comparison of teacher scores in 2004 and 2007 is reported in Table 10. Over the three years of the MTTA project, these teachers gained 2.95 points (out of 40). While this was statistically significant (e.g., the gain was more than might be due to just sampling error), the magnitude of gain was small. At least in mathematics, MTTA appeared to have only a small impact on teachers’ content mastery over three years. The most important story may not be the proportion of teachers achieving full mastery but, rather, the impact of the training on those teachers in the “partial mastery” group. Across the three subjects in 2007, 60%-80% of teachers demonstrated only partial mastery, but that could mean they scored anywhere from 31% to 79% of the items correct (Table 11). Such a broad middle category might have masked some of the benefits of MTTA training. This might have occurred if teachers in the low range of partial mastery meaningfully increased their content knowledge during the project but did still did not achieve full mastery. The longitudinal subsample of 123 teachers (described above) provided a means of testing this possibility. This analysis indicates that, within this full subset of 123 teachers, the percentage teachers demonstrating mastery increased from 2.3% to 12.2% while there was a small decrease in A-19 those teachers demonstrating no mastery (Table 12). When the 2007 mastery levels of those teachers demonstrating partial mastery in 2004 are examined more closely, results indicate that 13.9% of these teachers moved up to full mastery while a roughly equivalent number lost ground and ended up in the non-mastery group. This suggests that while MTTA appears to have strengthened the content mastery of some teachers, that impact is not uniform and that some teachers dropped in content mastery over the duration of the project. Table 10: Longitudinal comparison of mathematics test scores for teachers between 2004 and 2007 (N=123) Paired differences Mean Std. Dev. Corr. t. df p. > 2004 test 15.67 7.51 2007 test 18.62 9.44 Increase 2.95 2004-2007 test .21 -3.04 122 .003 Table 11: Percent of teachers by level of content mastery by subject area, 2007 Mastery Level Mathematics Science English Full 13.0 9.1 21.5 Partial 59.7 79.3 65.3 Non 27.3 11.6 13.2 Source: MTTA monitoring and evaluation reports Table 12: Longitudinal analysis of mathematics scores for paired subsample of teachers, 2004 and 2007 with 2007 content mastery levels of teachers who demonstrated partial mastery in 2004 Mastery level Longitudinal teacher sample N=123 Teachers demonstrating partial mastery in 2004 only N=101 2004 2007 2007 Full mastery 2.3% 12.2% 13.9% (N=14) Partial mastery 80.5% 75.7% → 71.2% (N=72) Non-mastery 17.1% 15.4% 14.9% (N=15) Source: Computation by authors At the same time, these results highlight the challenge that faced MTTA. Overall, teacher content knowledge is weak. At the beginning of MTTA in mathematics and science, more than 95% of the teachers failed to master the content (e.g. scored below 80% on the teacher mastery test). At the end of the project, over 90% were still unable to do so in science, 85% in math and 78% in English. Did male and female teachers show similar gains in math scores between 2004 and 2007? A comparison of mean scores by gender (Table 13) indicates no significant difference by gender of teacher in scores at either test period. A-20 Table 13: Comparison of mean differences between male and female teachers, 2004 and 2007 on longitudinal subsample of teachers Gender of teachers Mean math test scores 2004 2007 Male 15.9 18.7 Female 15.3 18.5 Significance of mean differences by gender within year N.S.* N.S. Source: Computation by authors * Test of mean differences indicates no significant difference. CHANGES IN PUPIL ACHIEVEMENT LEVELS While pupil achievement gain is the ultimate goal of this project, its use as an indicator of MTTA success needs to be treated with caution. Even if optimally effective, it is unlikely that MTTA teacher in-service training and follow-on support would have had time to percolate down to classroom instruction at a level substantial enough to have produced a measurable impact on pupil achievement scores. Using pupil achievement as an indicator of project success would be an unfair test of project accomplishments. Teachers completed the five cycles of in-service training by the end of year 2, during which they would have received content training in each of the three subject areas (science, math, English). The extent that 15 days of teacher upgrading can impact pupil scores is highly problematic. Even if the training was highly successful, teachers would need several years to practice and integrate the training into their classroom lessons. Further, teachers’ content knowledge and pedagogical skill are only a partial predictor of pupil achievement. Pupil test scores are influenced by multiple factors beyond just teachers’ content of knowledge. Together these reasons suggest that pupil achievement can be a useful indicator but should not be over-interpreted. Nonetheless, one would expect that if MTTA were successful in influencing pupil learning, it would be manifest most strongly in the third year of the project, when teachers had completed the five cycles of training. Data from MTTA assessments of pupil learning, using specially designed tests linked to the subject area curricula in each of the three subject areas, suggest this did indeed occur. For Standard 3 learners, 50 mathematics items from a scope of Standard 3 curricula were administered. The items were put into seven groups of beginning mathematics, numeration, subtraction, addition, multiplication and division, money and measurement. Pupil performance, 2004 to 2007, is reported in Table 14. Comparable data for Standard 6 is reported in Table 15. While these results are indicative of MTTA impact, the findings must be treated with considerable caution. Pupil achievement in MTTA schools increased, but the lack of longitudinal analysis, absence of a comparison group, and lack of any national assessment of pupil achievement prior to the end of Standard 8 makes it impossible to attribute these gains to MTTA interventions. A-21 Table 14. Percentage of Standard 3 learners correctly answering 80% of test items, by subject areas, 2004 to 2007 MATHEMATICS Mastery Level 2004 2005 2006 2007 Boys Girls Overall Boys Girls Overall Boys Girls Overall Boys Girls Overall Full 5.6 4.4 5.0 7.9 3.8 5.9 10.2 10.4 10.3 14.4 13 13.8 Partial 81.7 79.7 80.7 79.7 84 81.8 89.6 89.2 79.3 77.8 76 77.1 Non 12.8 15.8 14.3 12.4 12 12.3 10.2 10.4 10.3 7.8 10 9.1 SCIENCE Full 6.3 2.3 4.4 9.2 7.5 8.4 18.1 19.9 19.3 31.1 27 28.8 Partial 67.6 68.2 67.9 79.1 77 78.1 76.9 74.2 75.5 66.9 69 68.1 Non 26.1 29.4 27.7 11.7 15 13.7 4.9 5.9 5.2 2 4.2 3.1 ENGLISH Full 5.7 5.1 5.4 4.5 4.3 4.4 8.8 11 9.4 10.8 11.8 11.3 Partial 11.7 10.4 11.1 10.2 10.8 10.5 17.6 17.8 17.1 16.2 15.6 15.9 Non 82.6 84.4 83.5 85.3 84.9 85.1 73.6 71.2 73.5 73 72.5 72.8 Source: MTTA monitoring and evaluation reports. Table 15. Percentage of Standard 6 learners correctly answering 80% of test items, by subject areas, 2004 to 2007 MATHEMATICS Mastery Level 2004 2005 2006 2007 Boys Girls Overall Boys Girls Overall Boys Girls Overall Boys Girls Overall Full 0.6 0 0.3 0.3 0 0.1 0 1.4 0.7 3.2 1.6 2.4 Partial 11.9 8.1 10 10.2 6.4 8.3 14.1 12 13.2 15.8 10.1 12.9 Non 87.9 81.9 89.7 89.5 93.6 91.5 85.9 86 86.1 81 88.3 84.6 SCIENCE Full 0.3 0 0.1 0 0 0.3 1.4 2.8 2.1 4.9 2.8 3.8 Partial 13.4 16.7 15.1 22.5 1.4 13.4 24.2 18 21.3 23.5 21.9 22.6 Non 86.3 83.3 84.8 77.5 98.6 86.4 74.5 79 76.6 71.7 75.3 73.5 ENGLISH Full 74.5 74.1 74.3 73.9 72.5 74.2 76.6 75 75.6 80.2 82.2 81.2 Partial 16.6 18.6 17.6 16.8 22.8 20.1 17.7 18 17.7 13.8 11.3 12.5 Non 8.9 7.3 8.1 9.4 4.6 5.7 6 7.4 6.6 6.1 6.5 6.3 Source: MTTA monitoring and evaluation reports. A VIEW OF MTTA THROUGH THE GALAL MODEL Galal’s (2002, 2008) framework for analyzing educational reform offer a useful lens in which to examine MTTA. From this perspective, successful education reform needs to have sufficient inputs wisely combined into effective instructional processes. There need to be incentives to encourage relevant stakeholders to engage in the reform effort and utilize the inputs properly. A consistent and fair accountability system is necessary to ensure that incentives are fairly allocated. But even inputs and incentives are not enough. Teachers must have a sense of professionalism and take greater responsibility for the outcomes of schooling. Through this lens, MTTA must be regarded as quite successful. It was able to deliver a substantial amount of training (though perhaps less than intended). It established a teacher supervision system that teachers found helpful and which moved far beyond what was previously available. Perhaps A-22 most impressive, it created a level of professionalism among teachers that evoked an enormous level of volunteerism and commitment. It did much to move responsibility for the quality of primary school instruction to the primary schools themselves. While it is possible to criticize MTTA for specifics in shortfalls in delivery or in meeting achievement targets, the more important perspective is in its overall impact on Malawi primary education. In its totality, MTTA appears to have made a deep and substantial positive impact on how Malawi teachers, headteachers, and district education personnel view their own responsibility for the quality of primary education and, at the same time, provided them with tools with which to improve the quality of that education. CROSS CUTTING ISSUES IN MTTA Teacher achievement gains: Why were teacher achievement gains low? One reason may be that those who were training the teachers were themselves weak in the content and were unable to deliver effective instruction. Another possible reason is that the magnitude of the treatment, e.g., considerably less than the intended 5 days of training in each subject over two years, was not a sufficient amount to lead to the intended result. A third possibility is that the training was adequate, but that teachers started from such a low knowledge base that they were unable to incorporate the new material into their own teaching. A fourth possibility is that the training was adequate but that teachers lacked sufficient motivation to implement the new information into their classroom instruction. A fifth possibility is that some combination of these factors operated together to limit what teachers were able to learn and use. How one might design an MTTA-like project to more effectively raise teacher content mastery would depend heavily on which of these factors was the most important factor constraining their mastery. However, teachers appeared to demonstrate considerable enthusiasm, suggesting that the more salient cause of low achievement may have been the weak content mastery of trainers in combination with reduced instructional time for teachers. Overall, however, it may be unrealistic to expect measurable gains in teacher content knowledge, based on such limited amounts of in-service training. As discussed earlier, teachers only received a total of 3-4 days of training in content and pedagogy in each subject area during the first two years of the project. Since the topics had been identified by teachers as ones on which they particularly needed help, it is likely that the training was relevant. However, the extent that such a limited amount of training can make a dramatic difference in teacher content mastery is questionable. While the value of this content training is not being questioned, the notion that it will lead to measurable changes in teacher mastery is. USAID needs stronger empirical bases for estimating how much achievement gain is reasonable to expect. There is a risk that, due to initial overpromising of results, even successful projects are appearing to have fallen short. This occurs not because a project is poorly designed or implemented, but because targets were set too high. Cascade training: As previously discussed, MTTA in-service teacher training employed a cascade model in which MTTA staff and local partners (e.g. MIE, teacher training colleges) trained a group of 180 trainers (i.e., TOTs) who were drawn from ZINFAs, DEAs, Cluster Coordinators and exceptional teachers. These TOTs, in turn, conducted the five cycles of training. However, cascade training is widely viewed as controversial. There is a tendency for message accuracy and fidelity to erode with each step down the cascade. This was a particular issue in MTTA, since only 25% of those receiving the initial training and who were to then serve as the teacher trainers demonstrated mastery of the content they were expected to teach and the length of the training they were to deliver to teachers was significantly shortened. While, overall, this group (the TOTs) had a higher level of content knowledge than the teachers, their overall low level of content mastery may have limited the quality and effectiveness of the training they were able to subsequently deliver. It is possible that the low levels of teacher gain A-23 in content mastery over the course of the project were due, in considerable part, to the limited levels of content mastery among those who trained them and the limited time TOTs were given to conduct that training. On the other hand, one outgrowth of the cascade approach was to make educators at the school, cluster, zonal, and district levels responsible for the identification of training topics and direct delivery of subsequent training. This contributed to participants’ view that MTTA activities were relevant to their needs and to building the sense of professionalism discussed earlier. The utility of monitoring and evaluation data: MTTA is to be complimented on the extensiveness of its monitoring and evaluation activities. In particular, the project collected massive amounts of teacher and pupil achievement data. In many respects this is appropriate, given USAID increasing emphasis on assessing outcomes rather than merely documenting inputs. However, two factors limited the utility of the MTTA teacher and pupil achievement data. The lack of unique identification codes for each teacher and pupil that remained with them through the project meant that data from different test points could not be linked within or across years. This limited meaningful longitudinal analysis. Secondly, the lack of a comparison group made it impossible to assess the extent that gains observed among project teachers and pupils could be attributed to MTTA activities and would not have happened anyway. It should be noted that part of the explanation for this is that MTTA inherited its data collection design from previous projects that used a similar approach, and MTTA shared pupil and some teacher baseline data with MESA. Costs and benefits of volunteerism: A possible drawback to MTTA’s model of teacher professionalization has been the trade-off in instructional time by relying heavily on practicing teachers to assume greater responsibilities without pay. Many schools, especially in rural areas, have fewer teachers than the number of standards at the school (e.g., normally 1-8). A large percentage of schools also have extremely high teacher-pupil ratios, with some classes containing more than 300 pupils. One consequence is that headteachers, ZINFAs, and mentor teachers leave their pupils without supervision on days they supervise other teachers or, alternatively, turn their classes over to colleagues who are already teaching their own classes. According to a July 2007 survey, teachers were supervised an average of 2.75 times during the year, with teachers turning primarily to headteachers and PEAs for support. If each of 6000 teachers were to receive supervision that entailed two hours (overall) of a supervisors time, and that occurred an average of 2.75 times per year, it would entail 33,000 hours during which instructional supervisors being away from their own classes. If the instructional supervision leads to meaningful improvements in the classroom practice of those observed, this may be a reasonable trade-off. However, the price in lost instructional time is considerable. This is a tradeoff that needs to be examined more carefully. A large mitigating factor could be that MTTA discouraged the selection of ZINFAs and cluster-level mentors from under-staffed schools. Generally, schools with headteachers without classes of their own were targeted for filling such positions. Additionally, MTTA staff assert mentor teachers generally achieve higher learning outcomes in their classrooms, a possibility that would suggest the skill-level of teachers is such that a trade-off does not occur.Another possible “cost” associated with MTTA’s approach lies in its very success. As teachers gain the opportunity to improve their MSCE scores through trainings and access to resource materials, teachers are in a better position to purse teaching careers at the secondary level or in better-paying non-educational fields. Teacher interviews suggest at least a segment of primary teachers aspire to move to the secondary level, gain admittance to a college, or find a job with NGOs or in banking. This could be an issue, insofar as Malawi is not yet training enough primary teachers to meet its needs. Increasing teacher mobility contributes to the development of Malawi, but not necessarily to primary education. The efficacy of demonstration projects: Effective projects sometimes die once external funding ends, not because the projects are flawed, but because the right people do not recognize their success. One intention underlying MTTA was that success in the four target districts would serve as a demonstration of an effective approach to teacher upgrading, and this success would lead the A-24 MoEST to adopt this approach. However, both school, district, division, and central MoEST educators generally felt that project activities and success were not widely visible outside of the four districts or in the MoEST. A number of MoEST personnel were familiar with the project and thought it had been successful but, overall, the MTTA approach seemed to lack strong champions at the central level. At the same time, MTTA staff, through attendance at MoEST events and through meetings, have been remarkably successful at facilitating the adoption of a number of its practices and interventions within other MoEST projects. The MoEST’s Continuing Professional Development Technical Manual, a part of the PCAR roll-out that orients teachers to school-based professional development, draws heavily on MTTA’s use of Teacher Professional Development Conferences. Sustainability: School and district personnel unanimously thought MTTA had been successful. Asked what would be sustained, virtually all thought of sustainability in terms of individual MTTA elements that they believed that schools would continue to use. Most frequently mentioned were: • Mentor teachers • Use of locally available materials in instruction • Headteacher supervision • Use of books and resource manuals already in the schools • Troupers to provide instructional supervision • School generates school (or cluster) level in-service training (i.e., TPDC) • Use of small group instruction in high enrollment classrooms Sustainability in the view of participants was viewed in terms of successful elements, not overall program. The common element is that these items could be continued by school personnel themselves without any additional MoEST support. (See Box E). A bottom-up approach: Does the bottom-up approach work? Or, put another way: can teachers within a weak education system be a source of content assistance to other teachers? Evidence from the MTTA project suggests teachers weak in content can still be a source of content assistance and support to other teachers. It may not be as efficient as international assistance agencies would like, but there is a trade-off in ownership that may make the approach worth it. Still, the bottom-up approach worked, in large part, because MTTA had already infused a great deal of training and resource materials into the local school environment. These allowed local school personnel access to ideas and information as they sought to help each other. System-level adaptability to decentralization: Beyond the narrower aims of improving content and pedagogical knowledge among teachers, the MTTA project inadvertently demonstrated the responsiveness of the educational system to a comparatively decentralized approach to school management and professional development. By devolving new and different responsibilities to ZINFAs, headteachers, trainer heads, mentor teachers, and teachers themselves, MTTA demonstrated that educational personnel at various points in the system were able to collaboratively assume greater responsibility as part of their roles without creating role competition or confusion. Even more interestingly, increased responsibility and access to others appeared to create conditions of greater openness to new ideas, as evidenced by apparent teacher receptivity to learner-centered pedagogies. (See Box F). USAID project management: There was a clear intention that MTTA be an integrated extension of MESA, the previous USAID funded primary education project. More recently, USAID’s plan was that MTTA would lead directly into and support PCAR, the new national curriculum. However, when MESA was terminated early, MTTA adjusted its training to provide training in the pedagogical skills that had MESA had intended to introduce. This shifted and diluted some of the intended A-25 MTTA focus on content, though it should be noted that at least some teachers liked the pedagogical dimension of the training. Toward the end of the project, MTTA had intended to provide content training designed to introduce teachers to the new curriculum being introduced by PCAR. When the implementation of PCAR was delayed, MTTA ended up training teachers on the old curriculum rather than the new one. Again, this shift jeopardized some of the intended synergies. In both cases, MTTA made sensible adjustments to the shifting circumstances. Project management is to be complimented for its flexibility and creativity in responding to changing situations. However, USAID should be aware that such changes can lead to mission drift and lost synergy. Box E: Sustainability of Mentor Teacher Position: A Mentor Teacher’s Perspective Ablaham Llongu felt honored when he learned he had been chosen as a teacher mentor. He didn’t know what it meant. But he knew the PEA had selected him from among teachers at his cluster’s four schools, with the approval of the District Education Manager. “When it was announced, I had no objection,” recalled Llongu, who believes he was chosen because of his teaching abilities and ability to help other teachers professionally. “I accepted to take the challenge, because they saw ability in me.” Llongu, one of five teachers in Standard 8, as well as a Standard 3 teacher, knew mentoring was a largely volunteer position. Except for a bicycle and allowances for a week’s worth of TOT training, mentors receive no monetary compensation. For Llongu, however, this was not a problem. “Training is enough,” said Llongu, sitting in an empty classroom at Chankhozi Free Primary School in the District of Kasungu. “We are after knowledge and skills as to how we can help the young ones…even help other teachers.” But while Llongu says he would be glad to continue mentoring indefinitely even after the MTTA program ends, financial support for teachers is important. Sometimes after-school insets suffer from lack of attendance. There is a Malawian saying, ‘An empty stomach cannot hear,’ Llongu pointed out. On hot days when some teachers must walk long distances to attend a training, offering drinks is important. In most cases, Llonge thinks providing drinks and a training manual would probably be enough to sustain a cluster-level mentoring system indefinitely. It is also important to provide mentors with bicycles. Some cluster schools are separated by 15 kilometers. In his cluster, the furthest school is 12 kilometers away. If he tried to walk, school would be over by the time he arrived, Llongu said, laughing. With a bike he can reach schools with enough time and energy to do day-long supervision visits. Llongu believes the mentor model is valuable. He is normally welcomed by teachers throughout the cluster, he says, and has seen many improvements since MTTA began. For example, some teachers used to be afraid to teach upper classes such as Standards 5 through 8, he said. Now more of them are willing to teach those classes. Another example is the number of pupils speaking English. When he arrived at Chankhozi Free Primary School about five years ago, learners didn’t speak a word of English, even at Standard 8. Now, as a result of teacher training, teachers have “vast knowledge.” As a result, learners even in Standard 1 are able to speak English, Llongu said with pride. A-26 Box F: A Teacher’s Perspective: Professional Networking at the School Level Lovelace’s professional network has expanded greatly as a result of MTTA. Where before her network included her learners, the other teachers at her school, health workers, and the community and its chiefs, now it also includes two trainer heads from her school cluster, and colleagues she meets from 13 other schools at zonal trainings. When prompted, Lovelace, a slim 32-year-old Standard 6 teacher, also includes the PEA, the ZINFA, the headteacher, and the school’s mentor teacher. (See Map #1). Map #1: A Standard 6 Primary Teacher’s Mental Map of Professional Contacts Health workers (monthly) Learners (every day) Colleagues at school (every day) Colleagues from 13 zonal schools (monthly) Trainer heads (once or twice a term) Community & chiefs (once or twice a term) Mentor teacher & ZINFA (once or twice a term) PEA (once per term) Note: Italics indicate prompts by interviewer. The map suggests that MTTA has significantly expanded Lovelace’s teaching network, even when many positions associated with MTTA (e.g., PEA, ZINFA, head teacher, and mentor teacher) are not counted. According to social networking research, networks not too tightly linked through geographical or other types of proximity and which maintain outside channels of communication are more likely to adapt new ideas and engage in information exchange (Bathelt, Mamlberg, & Maskell, 2004). Lovelace notes her relationships have become more nurturing and positive as a result of watching and meeting other teachers at zonal and cluster workshops. She now sees learners and fellow teachers – a key part of her network – as her most important sources of help. Instead of trying to instruct 95 pupils at once, Lovelace now asks for pupil volunteers to facilitate small groups. She also encourages pupils to find answers on their own. This makes teaching easier, and sometimes leads to mutual learning, she explains with a smile. Her relationship with fellow teachers has also changed. Lovelace has learned it is OK to seek and give professional assistance, a new skill she saw demonstrated through knowledge sharing at MTTA conferences and workshops. “Before MTTA, we used to jump over difficult topics – we were shy to call for help,” Lovelace explained. “Now, when you have a topic you are not comfortable with, you can ask a colleague, ‘Can you help me with that?’ And they helped me. Before MTTA, there was not that kind of help.” A-27 Insights for future projects: Are there insights from MTTA experience that might inform the design or operation of future USAID projects? MTTA has demonstrated that: • Even in resource-poor communities, it is possible to develop spirit and enthusiasm for upgrading quality of education right down to the school level. • Even in poor communities, it is possible to evoke a level of personal contribution, professionalism, and volunteerism among teachers and headteachers. • Educators at the local level can be important sources of assistance in helping their peers upgrade both content and pedagogy. • Failure to properly assign individual identification codes and provide for comparison groups seriously limits the claims that can be made about project outputs and impacts. • Properly designed tests are necessary if project staff want to be able to critically assess project success CONCLUSIONS The preceding analysis supports four recommendations for the design of future development assistance projects in education: 1. MTTA was grounded consistently in local teachers’ own assessment of their content and pedagogical needs. Both training and ongoing school level support was delivered by Malawian educators. This conveyed a respect for local teachers and appears to have increased the level of local ownership. Teachers report that it enhanced the relevance of the training they received. MTTA demonstrated that local teachers, with proper support, can, in turn, be a key source of content and pedagogical support for their peers. The MTTA experience demonstrates that significant local involvement can be achieved. It serves as a model for other development assistance projects. 2. The level of volunteerism elicited by this project was extraordinary. Zonal, cluster and school level coordinators, trainers, and instructional supervisors received no compensation and no release time for their activities and often were expected to pay their own transportation expenses to undertake their MTTA duties. This kept local costs low, increasing the feasibility (though the not certainty) of the MoEST sustaining this model and implementing it on a wider scale without outside assistance. At the same time, the professional recognition conferred on those who assumed these responsibilities appeared to serve as a source of pride and motivation. The MTTA experience demonstrates that significant local volunteerism can be achieved. In this respect MTTA serves as a model for other development assistance projects. 3. Good project design is not enough. To test the effectiveness of the design, a project needs to be fully implemented. While, on one hand, this is obvious, MTTA stands as an example of a project in which a primary intervention was not fully implemented. The teacher training was cut short. This makes it difficult to determine whether small gains in teacher content master were due to a flaw in the project design or the failure to fully implement the project. 4. Assessment of project success depends on relevant data collected within a proper design. More attention needs to be given throughout a project to collecting data in a way that allows for longitudinal analysis and to have a comparison group against which to compare project results. The importance of these factors is not always widely understood within USAID (Chapman, 2008) and among all actors in the international organizations implementing this project, which may have resulted in insufficient resources devoted to evaluation impact design and implementation. Although greater attention is being given to impact studies, there is a need to become more intentional in building these components into the project design. A-28 ANNEX A: A statistical picture of MTTA At the inception of the project, there were 824 schools in the four target districts. By 2007, there were 833 (Table A-1). Table A-1: Number of zones, clusters and schools by 2007 District Zones Clusters Schools Mzimba South 26 75 275 Kasungu 25 108 319 Machinga 12 49 155 Phalombe 7 29 84 Total 70 261 833 Table A-2: Pupil enrolment in Malawi primary schools from 2004 to 2007 Year Boys Girls Total 2004 235,614 230,991 466,605 2005 232,314 231,398 463,712 2006 234,267 237,050 471,317 2007 236,009 239,238 475,247 Table A-3: Number of teachers trained in MTTA cycle and TPC training sessions Cycle Female Male Total 1 1768 4621 6389 2 1654 4304 5955 3 1568 4664 6232 4 1618 4407 6025 5 1495 4602 6097 6 1486 4154 5640 7 1430 4495 5925 A-29 Table A-4: Change in proportion of teachers holding JCE and MSCE credentials, 2004-2007 2004 JCE F 2004 JCE M 2004 MSCE F 2004 MSCE M 2007 JCE F 2007 JCE M 2007 MSCE F 2007 MSCE M MTTA Districts Kasungu 483 834 301 951 361 647 356 1011 Phalombe 96 295 32 292 65 180 40 317 Machinga 196 377 126 428 118 281 151 436 Mzimba South 246 557 174 641 212 470 198 716 MTTA TOTAL 1021 2063 633 2312 756 1578 745 2480 Non-MTTA Districts Central Eastern Dowa 413 794 161 543 311 624 211 608 Nkhotakota 197 385 147 334 137 298 178 377 Ntchisi 144 368 60 233 120 303 104 355 Salima 233 292 113 250 201 227 153 338 Central Western Dedza 297 516 161 497 249 411 230 587 Lilongwe Rural East 465 577 212 559 311 394 285 596 Lilongwe Rural West 579 854 251 678 454 619 321 776 Lilongwe City 1327 138 853 151 930 127 344 1174 Mchinji 268 492 170 559 217 411 178 594 Ntcheu 319 640 147 531 262 489 191 659 Northern Chitipa 138 483 56 310 120 464 63 342 Karonga 212 420 121 377 203 414 130 418 Likoma 13 25 10 19 13 20 11 21 Mzimba North 262 543 119 397 234 479 145 501 Mzuzu City 364 87 278 128 281 60 365 136 Nkhata Bay 139 284 83 361 133 299 92 397 Rumphi 209 371 92 286 206 354 126 388 Shire Highlands Chiradzulu 199 360 110 313 180 258 148 385 Mulanje 280 492 133 484 211 354 150 538 Thyolo 343 530 203 659 256 371 241 695 Southern Eastern Balaka 229 332 152 443 183 277 187 483 Mangochi 311 499 159 477 274 431 197 568 Zomba Rural 319 618 148 433 281 516 205 576 Zomba Urban 238 66 180 100 152 47 254 141 A-30 A-31 Table A-4: Change in proportion of teachers holding JCE and MSCE credentials, 2004-2007, continued 2004 JCE F 2004 JCE M 2004 MSCE F 2004 MSCE M 2007 JCE F 2007 JCE M 2007 MSCE F 2007 MSCE M Southern Western 0 0 0 0 0 0 0 0 Blantyre City 1070 163 952 296 707 116 1167 317 Rural Blantyre 482 357 349 446 306 249 385 497 Chikwawa 106 373 91 405 83 271 100 473 Mwanza 119 267 84 301 58 80 71 194 Nsanje 61 225 54 307 48 209 51 351 Neno 0 0 0 0 41 83 34 152 Non MTTA Districts TOTAL 9336 11551 5649 10877 7162 9255 6317 13637 2004 JCE F 2004 JCE M 2004 MSCE F 2004 MSCE M 2007 JCE F 2007 JCE M 2007 MSCE F 2007 MSCE M MTTA TOTAL 1021 2063 633 2312 756 1578 745 2480 Non MTTA Districts TOTAL 9336 11551 5649 10877 7162 9255 6317 13637 ANNEX B: Summary of MTTA inputs No. Indicator Means of verifica￾tion Year 2004 2005 2006 2007 Baseline Tar￾get Actual Tar- get Actual Tar- get Actual M F Total Total M F Total Total M F Total Total M F Total 1. Number of teachers trained in Mathematics, English and Science (in target districts). December 2004 (Cycle 1) Training Reports 2004 0 0 0 6,000 4,621 1,768 6,389 6,000 N/A Number of teachers trained in Mathematics, English and Science (in target districts). April 2005 (Cycle 2) Training Reports 2004 0 0 0 6,000 4,304 1,654 5,958 6,000 Number of teachers trained in Mathematics, English and Science (in target districts). August 2005 (Cycle 3) Training Reports 2004 0 0 0 6,000 4,664 1,568 6,232 6,000 Number of teachers trained in Mathematics, English and Science (in target districts). December 2005 (Cycle 4) Training Reports 2004 0 0 0 6,000 4,407 1,618 6,025 6,000 Number of teachers trained in Mathematics, English and Science (in target districts). August 2006 (Cycle 5) Training Reports 2004 0 0 0 N/A 6,000 4,602 1,495 6,097 Number of teachers trained in Mathematics, English and Science (in target districts). December 2006 (Cycle 6) Training Reports 2004 0 0 0 6000 4,154 1,486 5,640 Number of teachers trained in Mathematics, English and Science (in target districts). April 2007 (Cycle 7) Training Reports 2004 0 0 0 6,000 4,495 1,430 5,925 2. Percentage of teachers using participatory teaching methods during instruction of mathematics, science or English (in target districts). Classroom Observation 2003 35.6 42.4 36.9 50 61.8 62.2 62.0 65 70.8 71.2 71.0 70 77.9 78.3 78.1 3. Percentage of teachers demonstrating full mastery in using mathematics concepts (in target districts). Teacher Assessment 2004 4.6 1.0 3.5 6.0 6.1 6.6 6.3 8.0 9.3 8.0 8.9 10.0 14.6 9.9 13.0 4. Percentage of teachers demonstrating full mastery in using science concepts (in target districts). Teacher Assessment 2004 1.0 2.3 1.4 4.0 7.8 0.3 5.4 6.0 6.5 6.6 6.5 8.0 9.7 7.9 9.1 A-32 A-33 No. Indicator Means of verifica￾tion Year 2004 2005 2006 2007 Baseline Tar￾get Actual Tar- get Actual Tar- get Actual M F Total Total M F Total Total M F Total Total M F Total 5. Percentage of teachers demonstrating mastery in English comprehension (in target districts). Teacher Assessment 2004 10.5 11.1 10.7 14.0 13.8 13.8 13.8 16.0 18.3 14.6 17.1 18.0 24.4 15.5 21.5 6. Percentage of pupil increasing individual mastery levels in mathematics in Standard 3 (in target districts). Pupil Assessment 2003 0.5 0.4 0.5 5.0 7.9 3.8 5.9 8.0 10.2 10.4 10.3 N/A 7. Percentage of pupil increasing individual mastery levels in science in Standard 3 (in target districts). Pupil Assessment 2005 6.3 2.4 4.4 6.0 9.2 7.5 8.4 8.0 18.1 19.9 19.3 8. Percentage of pupil increasing individual mastery levels in English in Standard 3 (in target districts). Pupil Assessment 2003 0.4 0.2 0.3 5.0 4.5 4.3 4.4 8.0 8.7 10.0 9.4 9. Number of pre-service teachers trained in Life Skills for HIV/AIDS curriculum Training Reports 2005 0 0 0 0 1,901 348 2,249 2,500 1,099 794 1,893 2,500 1,584 923 2,507 10. Number of school-based pilot clubs created (in target schools). Training Reports 2006 N/A 80 80 N/A 11. Number of primary school head teachers selected and trained in club leadership skills (in target schools). Training Reports 2006 160 109 51 160 12. Number of youth mentors selected and trained to assist club leaders (in target schools). Training Reports 2006 160 80 80 160 13. Number of Chichewa version of Sara Comic Book Series disseminated to the school-based pilot clubs (in target schools). Delivery Reports 2006 12,80 0 12,800 14. Percentage of school-based pilot clubs supervised and/or supported in a month. Supervision Report 2006 100% 100% Source: MTTA Performance Monitoring Plan Chart of Indicators, 2008. Note: Indicator 1: The number of teachers trained in English, mathematics and science in Cycle 2 is lower than the target because during that period some teachers who were undergoing the MIITEP training program were attending residential courses at teacher training colleges. A total of 406 teachers did not attend Cycle 6 workshops because most of them had travelled out of their duty stations lowering the number of teachers trained to 5,640 instead of 6,046. Description of indicators and how they were measured: Number of teachers trained in Mathematics, English and Science (in target districts). (Cycles 1 to 7): Number of teachers (disaggregated by gender) who successfully completed an in-service teacher training program. In-service teacher training was defined as training for existing teachers in the three content areas (mathematics, science and English). The indicator was calculated by finding out the number of teachers trained in the three content areas through the training registration forms which teachers filled during the in-service training. ANNEX C: List of people contacted • Mr. Fahim Akbar, AED/EQUIP2 EMIS advisor in the MoEST • Mr. Inok Alli, District Education Manager, Phalombe District (missed meeting) • Mr. Harold Banda, Headteacher, Chanhozi Primary School, Kasunga District • Ms. Emylda Bongwe, District Education Facilitator, Machinga District • Mr. Dudley Chiala, Education Division Manager, Central East Education Division • Ms. Charity Kamanga Chilinda, Mentor Teacher, Davey Primary School, Mzimba South • Mr. Franki Chikapa,Teacher, Namenjeriam Primary School, Phalombe District • Ms. Lucy Chiwala, Primary Education Advisor, Kasunga District • Mr. Charles M. Gunsaru, Director, Malawi Institute of Education • Mr. Fritz Kadyoma, MTTA staff • Mr. McKnight Kalanda, Director of Basic Education, MoEST • Mr. Rath Kathewera, MTTA staff • Mr. Chaplain Katumbi, former MTTA staff • Mr. Richard Kimbal, General Development Officer, USAID • Mr. Christopher Khoropa, District Education Facilitator, Mzimba South District • Mr. Christopher Kumikundi, District Education Manager, Kasunga District • Mr. Rodney Longwe, Teacher, Chaima Primary School, Kasunga District • Mr. Mr. Ablaham Lungu, Cluster Mentor Teacher, Chanhozi Primary School • A.G. Louhanga, PEA, Kamgawa Zone • Mr. Rexford Manjaalera, Headteacher, Chaima Primary School, Kasunga District • Mr. Simeon Mawindo, Chief of Party, MTTA • Ms. Darles Mbewe, Acting Director, Department of Teacher Education & Development • Mr. Douglas Mbingwa, MTTA staff • Dr. Hartford Mchazime, Deputy Chief of Party, MTTA • Mr. Morundo, Cluster Mentor Teacher, Namenjeriam Primary School, Phalombe District • Mr. Harold Mwangala, Head Teacher, Mombe Primary School, Phalombe District • Ms. Moliday Ulanda Ndovi, Teacher, Mzimba L.E.A., Mzimba South • Ms. Florence Nkosi, Education Assistant, USAID • Mr. Orton Nyirenda, data analyst, MTTA • Mr. Pathon Nyirongo, District Education Manager, Mzimba South District • Mr. Bernard James Nyuongo, Mobile Troupe Trainer and Mentor Teacher, Mzimba L.E.A. • Ms. Marisol Perez, Education Officer, USAID • Mr. Isaac M.C. Phiri, Coordinating Primary Education Advisor, Kasunga District • Ms. Nerissa Boatman Phiri, District Education Facilitator, Phalombe District • Mr. Curt Reintsma, Director, USAID • Ms. Eneret Santhe, District Education Facilitator, Kasunga District • Ms. Bernadette Sekeleza, Primary Education Advisor, Machinga District • Mr. Ramsey Sosola , Education Office, USAID • Mr. P. K. Zimpita, ZINFA, Chaima Primary School, Kasunga District A-34 ANNEX D: Documents Reviewed • Bathelt, H., Mamlberg, A. & Maskell, P, (2004). Clusters and knowledge: Local buzz, global pipelines and the process of knowledge creation. Progress in Human Geography, Vol. 28, No. 1, 31-56. • Creative Centre for Community Mobilization (2006). 2006 Survey on IBB Books. American Institutes for Research (AIR): Washington, DC • Kendall, N. (2008). Achieving our Goals and Transforming our Schools: Best Practices in the Malawi Teacher Training Activity. USAID: Washington, DC • Malawi Institute of Education (2008). Continuous Professional Development of Teachers: Technical Manual for the Organization of CPD. Malawi: Domasi • Miske Witt & Associates (2006). Trip Report: October 19-November 10, 2006. USA: Roseville. • MTTA (2008). Innovative Strategies & Best Practices of MTTA. Malawi: Zomba • MTTA (2008). Education Strategic Objective Performance Implementation Review. Malawi: Zomba • MTTA (2008). Work Plan and Performance Monitoring Plan Relative to PCAR: January-July 2008. Malawi: Zomba • MTTA (2008). Malawi Teacher Training Activity: January-March 2008 Quarterly Technical Report. Malawi: Zomba • MTTA (2007). Headteacher Training Handbook. An In-Service Training and Resource Handbook for Primary Headteachers. Zomba • MTTA (2007). Monitoring and Evaluation: Draft Final Technical Report, September 2004 to December 2007. Malawi: Zomba • MTTA (2007). Becoming an Instructional Leader: An In-Service Training and Resource Handbook for Primary Headteachers and Deputies on Instructional Leadership. Malawi: Zomba • MTTA (2007). July-September 2007: Quarterly Report. Malawi: Zomba • MTTA (2007). April-June 2007: Quarterly Report. Malawi: Zomba • MTTA (2007). January-March 2007: Quarterly Report. Malawi: Zomba • MTTA (2007). Malawi Teacher Training Activity: January-December 2007, Annual Report. Malawi: Zomba • MTTA (2007). MTTA Forum. Vol 2(1) Malawi: Zomba • MTTA (2006). January-December 2006 Annual Report. Malawi: Zomba • MTTA (2006). July-September 2006: Quarterly Report. Malawi: Zomba • MTTA (2006). April-June 2006: Quarterly Report. Malawi: Zomba • MTTA (2006). January-March 2006: Quarterly Report. Malawi: Zomba • MTTA (2006). Mobile Teacher Training Troupers’ Orientation Handbook. Malawi: Zomba • MTTA (2006). MTTA Forum. June. Malawi: Zomba • MTTA (2006). Teaching Mathematics in Malawian Primary Schools: Reading and Writing. An In-Service Resource Manual for Primary Teachers. Malawi: Zomba • MTTA (2006). Teaching Science in Malawian Primary Schools: Reading and Writing. An In-Service Resource Manual for Primary Teachers. Malawi: Zomba • MTTA (2006). Fifth Cycle Training Handbook. Malawi: Zomba • MTTA (2005). Teaching English in Malawian Primary Schools: Reading and Writing. An In-Service Resource Manual for Primary Teachers. Malawi: Zomba • MTTA (2005). Teaching Science in Malawian Primary Schools: Practical Activities for Effective Pupil Learning. An In-Service Resource Manual for Primary Teachers. Malawi: Zomba A-35 A-36 • MTTA (2005). Mid-Term Performance Report: Successes, Challenges, and the Way Forward. Malawi: Zomba • MTTA (2005). Annual Report: January-December 2005. Malawi: Zomba • MTTA (2005). July-September 2005: Quarterly Report. Malawi: Zomba • MTTA (2005). April-June 2005: Quarterly Report. Malawi: Zomba • MTTA (2005). January-March 2005: Quarterly Report. Malawi: Zomba • MTTA (2005). Annual Report: January-December 2005. Malawi: Zomba • MTTA (2005). Second Cycle Training Handbook. Malawi: Zomba • MTTA (2005). Third Cycle Training Handbook. Malawi: Zomba • MTTA (2005). Fourth Cycle Training Handbook. Malawi: Zomba • MTTA (2004). Annual Report: September-December 2004. Malawi: Zomba • MTTA (2004). MTTA Technical Proposal. Malawi: Zomba • MTTA (2004). First Cycle Training Handbook. Malawi: Zomba Part B: MTTA Life Skills Pre-Service Teacher Training Component Evaluation By Shirley Miske B-1 EXECUTIVE SUMMARY Malawi has one of the highest HIV/AIDS prevalence rates in the world. To help respond to this urgent situation, this initiative addressed a critical need – training lecturers (i.e., tutors) of Teacher Training Colleges (TTCs) and pre-service teachers in Malawi’s new ‘Life Skills for HIV/AIDS Education’ curriculum. The initiative, which took place in Malawi’s TTCs, had three goals: (1) facilitate the development of knowledge and skills that allow teachers to avoid HIV/AIDS infection; (2) enable teachers to facilitate open discussions in the classroom about HIV/AIDS and sexuality; and (3) ensure teachers have practiced and mastered participatory methodologies to facilitate skills development and to effectively implement the Life Skills for HIV/AIDS Education curriculum (USAID, RFQ, 2004). This initiative was one component of the Malawi Teacher Training Activity (MTTA), a USAID-funded project that was implemented from August 2004 through July 2008. It took place in partnership with Malawi’s Ministry of Education, Science, and Technology (MoEST) and was a continuation of work by other development partners, who had invested also significant resources in the development of Life Skills curricula and related HIV/AIDS￾awareness resources in Malawi. An end-of-project evaluation found this initiative successfully accomplished the following activities: (1) Materials – A variety of materials were produced, delivered, and integrated into TTC activities. The materials included an HIV and AIDS information booklet, a Life Skills resource manual for tutors and TTC students, entitled Life Skills for HIV and Aids Education: Resource Manual for Teachers; and videotaped Life Skills lessons on DVDs. Data suggested that all materials were received enthusiastically. (2) MTTA provided training for TTC Life Skills tutors, along with follow-up monitoring and supervision. Data from MTTA monitoring reports and from self-reports in group interviews with tutors indicated that tutors’ professional knowledge and skill development had increased. Their improved lesson delivery also increased pre￾service teachers’ effective delivery of Life Skills in the primary school classroom. For example, they began to use participatory methods such as role play in their Life Skills lessons. (3) Each TTC organized guest lectures on topics related to HIV and AIDS. Data suggested that tutors and student teachers appreciated these events, which occurred once per school term. The evaluation also found the following challenges: (1) Replacement DVD players, which were supposed to accompany DVDs containing Life Skills lessons, had not been delivered to tutors at the time of the evaluation. Resourceful tutors were able to play the DVDs using personal equipment, but the delay in delivery impacted the usage of the DVDs. B-2 (2) The appropriateness of some Life Skills topics at the primary school level was a source of discussion and some disagreement among teachers and some members of the community. (3) The sustainability of HIV/AIDS-awareness guest lectures at the TTCs and the sustained impact of learning resource materials is uncertain. Some TTCs are unlikely to use their own funds or to issue invitations to invite guest lecturers to speak without offering remuneration. And unless the newly developed resource materials for tutors and TTC students are reprinted and staff development for tutors continued, new Life Skills tutors and new cohorts of TTC students will not have access to the same materials and training that had an impact on teaching and learning in this study. B-3 Life Skills is one of those subject areas that is answering a need. Life skills is meeting needs not only of the learners in the primary school but also learners at the college level. It is a subject that is double-edged – all of us are getting something from it. Principal, Lilongwe Teacher Training College INTRODUCTION The purpose of this end-of-project evaluation is to assess the effectiveness and value of the Malawi Teacher Training Activity’s (MTTA) pre-service Life Skills teacher training component.1 The evaluation takes the approach of the goal attainment model, exploring the extent to which intended activities were carried out and project objectives achieved. It is framed around three questions: (1) Were the inputs delivered and used? (2) To what extent did beneficiaries (i.e., tutors and pre￾service teachers) believe the inputs were relevant, appropriately delivered, and effective? (3) To what extent did these inputs make a difference for tutors and for pre-service teacher training students? The sections of the study present the following information: the methodology used in the evaluation; a brief description of the background of Life Skills and pre-service teacher training in Malawi; an overview of the MTTA Life Skills pre-service teacher training component; and findings of the evaluation at three levels of inputs – MTTA staff to Teacher Training College (TTC) lecturers, TTC lecturers to TTC students, and TTC students to primary school pupils in Standards 2 through 7. The last two sections are a summary of the extent to which inputs have been delivered, used, and have had an impact on Life Skills teaching and learning; and conclusions and recommendations. METHODOLOGY A three-person team led by Dr. Shirley Miske, Miske Witt & Associates, conducted this evaluation. Mr. Sydonio Matope, co-implementer of MTTA Life Skills training from Save the Children Malawi, and Mrs. Joy Mueller, Consulting Associate, Miske Witt & Associates, also participated. The team conducted group interviews with TTC tutors, interviewed headteachers and teachers, and conducted classroom observations. Mr. Matope served as Chichewa language interpreter for the Standard 2, 3, and 4 lesson observations. The evaluation was scheduled for a one-week period, Friday, April 25 through Friday, May 2, 2008 (see Annex B-1). The MTTA project administrators determined that three TTCs in relatively close proximity could be visited during this period, and they suggested two of five government TTCs and one of two private colleges be included. The institutions selected were Lilongwe Teacher Training College and Blantyre Teacher Training College (government), and St. Joseph’s Teacher Training College (private). Evaluators conducted a group interview (see Annex B-2 for tutor group interview questions) for 60 to 90 minutes with all Life Skills tutors or lecturers who were present at the TTC (i.e., those who were not out in schools observing student teachers). The St. Joseph’s lecturers were interviewed off-site, since they were participating in a workshop in another location at the time of the evaluation. Thirteen tutors who teach Life Skills were interviewed at the three TTCS. (Note: TTC tutors who teach Life Skills also teach other subjects (e.g., agriculture, social studies, science and technology, expressive arts). Six Life Skills tutors from Lilongwe TTC participated in the group interviews, together with four from Blantyre TTC and three from St. Joseph’s. 1 This evaluation is one of three end-of-project summative evaluations of various components of the Malawi Teacher Training Activity conducted under the auspices of Miske Witt & Associates during April-June 2008, prior to the July 31, 2008 closing of the MTTA. Dr. Lynn Evans conducted a summative evaluation of the HIV and AIDS School Clubs Initiative (HASCI), published under separate cover, and Dr. David Chapman and Ms. Suzanne Miric evaluated the in-service training component (see Part A). B-4 Morning visits to two area primary schools preceded each afternoon visit to a TTC. The evaluators observed two student teachers in each school teach a Life Skills class in Standards 2 through 7. At each school the evaluation team asked to observe equal numbers of male and female teachers. Since St. Joseph’s is a single-sex college for women, the four pre-service teachers observed from this TTC were women. Headteachers arranged for evaluators to observe a total of four male and eight female teachers teach 12 lessons at the six schools. Interviews with the teachers whose lessons had been observed (see Appendix B-4) and with the headteachers followed the classroom observations. (See Appendix B-5 for the complete list of schools, student teachers, and standards taught.) Classroom observations were documented through extensive field notes taken by evaluation team members. MTTA monitoring instruments for pre-service teachers, which had been used by the MTTA monitoring teams, were adapted for student teacher interviews. The pre-service teachers and headteacher were interviewed using the protocol in Annex B-3. Evaluators also reviewed Life Skills resource materials and other reports in order to become familiar with their content. The research design utilized a qualitative methodological framework. Focus group discussions with TTC tutors, classroom observations of pre-service teachers, and interviews with pre-service teachers were used to triangulate data. Limitations of the evaluation included the amount of time available to conduct the evaluation, which did not allow for pupil interviews of primary school pupils about the Life Skills lessons. In addition, although it was not essential, observing the tutors teach the Life Skills classes would have enriched the data set. The two Learning Resource Coordinators who were responsible for the development of the TTC Life Skills resource manuals were no longer living in Malawi at the time of the evaluation and thus were not available to discuss the materials in context. However, they were contacted following data collection in the field. BACKGROUND: LIFE SKILLS AND PRE-SERVICE TEACHER TRAINING IN MALAWI Note: This report presumes some familiarity with the MTTA based on Part A of the report. Readers who are not familiar with the structure of the Malawi education system and the design of the MTTA project should read “Background: Malawi and MTTA” in Part A. Malawi has one of the highest HIV/AIDS prevalence rates in the world. It is estimated that in 2008, 14% of the adult population is living with HIV/AIDS and one-half million children have been orphaned by AIDS (The Global Fund, 2008). The Government of Malawi (GoM) has developed a National HIV/AID Strategic Framework (2000-2004) to coordinate the country’s response to the HIV/AIDS epidemic. Although awareness of HIV/AIDS is high among secondary students and teachers, it is not as high among primary school pupils (MTTA, 2007). Comprehensive knowledge of HIV and AIDS also is low and behavior change has been slow, which has resulted in a high mortality rate. To build a comprehensive knowledge base and to promote behavior change, the Ministry of Education, Science, and Technology (MoEST) introduced Life Skills into the school curriculum from Standard 2. Life Skills refers to a large group of psycho-social and interpersonal skills that can enable children and adults to make informed decisions, communicate effectively, and develop coping and self￾management skills that can help them lead a healthy and productive life. Agencies such as UNICEF worked with the MoEST to develop its first Life Skills curriculum and materials. The UNICEF Life Skills textbooks included simple messages and knowledge about the human body, sexual behavior, violence, and exploitation (UNICEF, no date). With the Primary Curriculum Assessment and Reform initiative that introduced a new primary school curriculum to Malawi, the MoEST developed its own comprehensive Life Skills curriculum. In 2005 the Malawi Institute of Education (MIE) also developed a Life Skills textbook for TTC lecturers to use for new teacher preparation. TTC B-5 lecturers are now expected to prepare all prospective primary school teachers to implement the new Life Skills PCAR curriculum. Prior to 1994, prospective primary school teachers enrolled in a two-year training program. As noted in Part A of this Evaluation Report, following the 1994 declaration of Free Primary Education, the GoM launched an emergency teacher preparation program, the Malawi Integrated In-service Teacher Education Program (MIITEP). MIITEP provided a 16-week residential in-service training to newly recruited teachers along with distance learning materials. Of necessity, this was considered to be equal to the usual two-year training period. Once the urgent need for all new teachers to have some training had been addressed, a new “One plus One” model was initiated in 2005. In this model, pre-service teacher training was expanded to one full year (three terms) at a TTC followed by one year of teaching in a classroom. During this second year of the “One plus One” program they are supervised by tutors, mentor teachers, headteachers and/or a Primary Education Advisor on all subjects, including Life Skills. Although second-year students technically are considered student teachers in this model, two student teachers per class now plan and deliver all lessons to the class and they conduct pupil assessments. The 12 student teachers observed for this evaluation were responsible for classes of 16 to 96 pupils. In support of the MoEST’s goals to introduce Life Skills into the education system, in 2004 USAID included in its MTTA request for proposals (specifically, “Request for Quotations”) the request for an intervention that would support implementation of the new pre-service Life Skills curriculum in the following ways: (1) facilitate the development of knowledge and skills that allow teachers to avoid HIV/AIDS infection; (2) enable teachers to facilitate open discussions in the classroom about HIV/AIDS and sexuality; and (3) ensure that teachers have practiced and mastered participatory methodologies to facilitate skills development and to implement effectively the Life Skills for HIV/AIDS Education curriculum. Through the Malawi Teacher Training Activity (MTTA) begun in August 2004, the American Institutes for Research and its partners, Save the Children-US, the Malawi Institute of Education, and Miske Witt & Associates proposed to “map the terrain of Life Skills” related to best practices, available materials, and training needs at the Teacher Training Colleges (TTCs) and then to develop an appropriate support strategy. This strategy would involve training for tutors, the production of resource tools and materials, and – an innovative and previously untried strategy – the production of instructional DVDs on Life Skills lessons. The DVDs would include video clips of classroom teaching that would model both good practices of teaching Life Skills in all eight standards as well as facilitate interactive teacher training. Various development partners (e.g., UNICEF and GTZ), had already invested significant resources in the development of Life Skills and anti-HIV/AIDS curriculum in Malawi.2 The MTTA project design and staff acknowledged from the outset that this USAID-supported pre-service teacher intervention would be developed in partnership with the MoEST and with other development partners. For example, early in the project the Learning Resources Coordinator and AIR consultant visited the offices of numerous agencies across the country in order to “map the terrain of Life Skills” in Malawi. OVERVIEW OF THE MTTA LIFE SKILLS PRE-SERVICE TEACHER TRAINING COMPONENT The MTTA Learning Resource Coordinator and Health Specialist took responsibility for this pre￾service component early in the project. They began by conducting a needs assessment to understand the state of Life Skills teaching in Malawi in 2004. Working with international and 2 Several student teachers who were interviewed noted that the GoM had provided sufficient teachers’ guides and pupils’ textbooks, which they appreciated. They also mentioned that additional resources had been provided by Plan International, the World Bank, and others. B-6 Malawian HIV and AIDS education experts, they collected data from tutors, TTC students, MoEST officials, and others. On the basis of the needs assessment, MTTA, together with its partners at MIE and at the TTCs, decided to prepare the following project inputs for pre-service teacher training: (1) print materials (i.e., an HIV and AIDS information booklet; a Life Skills resource manual for tutors and TTC students); (2) videotaped Life Skills lessons on DVDs, with accompanying booklets on how to use the DVD, and DVD players for the TTCs; (3) training workshops for Life Skills lecturers on content and participatory methods, with follow-up monitoring of lecturers and student teachers, as well as training for TTC lecturers in general (since all lecturers participate in student teacher supervision); and (4) working with TTCs to organize a series of guest lectures by local experts on topics related to HIV and AIDS that would be organized by each TTC. Additional information on these inputs is given below. Print Materials. A short informational booklet, Basic Facts about HIV and AIDS, was printed and distributed to TTC lecturers and students. This was printed to meet the need of accurate, basic information reaching all beneficiaries of this MTTA component (i.e., lecturers, TTC students, and primary school pupils). Nearly 2,700 copies were distributed to five TTCs (Blantyre, Karonga, Kasungu, Lilongwe, and St. Joseph). MIE already had developed a Life Skills guide for lecturers to use in teaching Life Skills lessons at the TTC. However, the MTTA needs assessment revealed that the guide lacked information in certain areas (e.g., on sex and sexuality) and included only two lessons on how to teach Life Skills in primary school. Therefore, the MTTA Learning Resource Coordinator and the MIE Life Skills Specialist decided to develop a resource manual to supplement the MIE guide tutors were using. The resulting 106-page resource manual is entitled Life Skills for HIV and Aids Education: Resource Manual for Teachers. It contains three sections: (1) Teaching Life Skills; (2) Life Skills topics (see Box A); and (3) Additional Activities for Young Children, Pre-Adolescents, and Adolescents. The manual incorporates information and activities from earlier resource manuals published by the World Health Organization and Education International. Nearly 3,100 copies were distributed to all TTCs. Box A. Unit topics in Life Skills for HIV and AIDS Education: Resource Manual for Teachers. Malawi Teacher Training Activity (August 2006). Unit 1. Sex and sexuality Unit 2. Decision-making and problem solving Unit 3. Change and decision-making Unit 4. Communication Unit 5. Self-esteem Unit 6. Assertiveness Unit 7. Stress and anxiety management Unit 8. Peaceful conflict resolution Unit 9. Introduction to morals and values Unit 10. Importance of morals and values Unit 11. Prevention of crime and corruption Unit 12. Interpersonal relationships Unit 13. Roles and responsibilities in relationships Unit 14. Guidance and counseling in sexual relationships Unit 15. Planning and entrepreneurship Unit 16. Practicing good health habits Unit 17. Sanitation Unit 18. Sexually Transmitted Infections (STIs) including HIV and AIDS B-7 Life Skills DVD. Seward Inc. of Minneapolis, Minnesota (a sub-contractor to MTTA partner Miske Witt & Associates), developed the DVD for lecturers to use with their Life Skills classes. Seward Inc. specializes in all forms of instructional design and has developed Information and Communication Technology (ICT) materials for teachers and students internationally. The DVD prepared for teacher training in Malawi contains short videos of teachers in Standards 1 through 4 teaching a Life Skills lesson in Chichewa and of teachers in Standards 5 through 9 teaching a Life Skills lesson in English. MTTA staff identified teachers in area primary schools who used “best practices” in teaching Life Skills to be video-taped teaching their exemplary Life Skills lessons. Audio-visual department technicians from MIE then videotaped the Life Skills lessons under the guidance of Seward, and Seward then produced the DVDs with interactive instructional segments that could be used to teach both Life Skills content and pedagogy. (See Box B for DVD lesson topics.) Box. B DVD Lesson Topics from Malawi Pre-Service Life Skills Education for HIV and AIDS. Disc 1 Chichewa Disc 2 English Std. 1 What is HIV & AIDS? Std 5. Discrimination Std. 2 AIDS Std 6. Communication Myths Std. 3 “Sugar Daddy” & Decision-Making Std 7. Cultural Practices & Health Std. 4 Living Positively with HIV/AIDS Std 8. Drug & Substance Abuse Trainings for Life Skills Tutors. Between December 2005 and April 2007 MTTA held five trainings for Life Skills tutors. At the beginning tutors were not able to come together at the same time, so MTTA staff took the training to the TTCs and met with tutors in their own institutions. In the earliest trainings, participatory methods for teaching Life Skills lessons from the MIE guide were the focus of trainings. The resource manual and DVDs were introduced in the last two trainings. MTTA staff ensured that tutors understood the content and could use the methods properly by monitoring tutors teach the Life Skills lessons to TTC students, and by observing student teachers deliver the lessons to pupils in their classrooms. Guest Lectures. MTTA supported each TTC in setting up guest lecture series with one lecture per term on topics related to HIV and AIDS. Local or national experts presented on topics such as “ARV and Psycho-social Support: Referral to and from Counseling and HIV Testing Services” (Lilongwe TTC); “Opportunistic Infections and Care of a Person with HIV/AIDS” (Blantyre TTC); “Empowering People with HIV/AIDS to Live Positively” (St. Joseph TTC); and “Counselling People Living with HIV and Those Living with AIDS” (Kasungu TTC). Each speaker prepared a paper and was expected to distribute it to the lecturers and the students. B-8 EVALUATION FINDINGS The successes and challenges of the MTTA Life Skills pre-service teacher training component are described below according to the level of inputs delivered – TTC Lecturers, TTC students, and student teachers in primary schools. Lecturers learned new content and methods to teach Life Skills Training Workshops. All tutors who had attended the training workshops reported that they had found them to be extremely useful. They learned new content and practiced new participatory methods that they in turn could use with their students. In 2005 all of us [tutors] started teaching Life Skills without training, so with the MTTA training we feel so confident and prepared. The sessions from ‘05 to ‘07 helped us to gain experience on how to give these topics. We received lots of updates on HIV/AIDS. We have terms society may call “obscene language.” We may have had those feelings too – but having been oriented we understand. It’s the way it has to be. The courses enriched the different methods – not only new methods but for example research. We were given seven topics to [go out into] the community and to find out information. Also we were exposed to real life situations. We went to the hospital and visit the people. In entrepreneurship we were given topics. I went to the market to find out how fish were being bought and sold, the challenges [market sellers] meet. At first they were reluctant to talk to us, but then they were open. It’s not all that easy [for them]. Monitoring Life Skills Teaching at TTCs. TTC lecturers also valued MTTA staff monitoring their teaching after the workshops. One noted, After teaching we went to my office to discuss strengths and weaknesses. It was a useful conversation. She [Laura] also urged me to introduce something I could try, such as involvement of the learners. Resource Manuals. All tutors reported using the resource manual regularly in their lesson preparation. “It has more information – like morals and values, also sex and sexuality – than the MIE handbook [textbook],” noted one Lilongwe TTC tutor. “The [MIE] handbook does not have content on assertiveness,” added another. Another reported, “The [MIE] textbook doesn’t have enough information; this [resource manual] has case studies and values clarification.” Tutors found these topics in the resource manual to be particularly useful: messages on STIs and HIV/AIDS; counseling on HIV/AIDS; sex and sexuality (“the sexuality activities have details and get positive feedback from students”); effective communication skills; and morals and values. They noted that all topics were relevant to the current curriculum and that methods were well explained and participatory. They noted that the case studies were particularly helpful. Tutors and teachers alike suggested topics they would like to see treated in greater depth in the future if the resource manual were to be revised and reprinted. These included more in-depth information on each topic; more on entrepreneurship and research activities to empower the students; more on how to use profits appropriately; on communicable and non-communicable diseases and on goals and goal setting. If it is reprinted, suggestions were given to add more participatory methods, include more details on “potential and talents,” and to provide more detail on positive traditional practices that can help prevent HIV. Life Skills DVDs. Tutors who used the Life Skills DVD also reported that it was extremely useful. They described it in this way: B-9 They [students] looked at teaching skills and looked at the methodology of teaching [in the video clip] and then critiqued it. They were positive and constructive in their ideas and they gave their suggestions about the teaching that they observed. They [the students] really enjoyed it. They knew what was expected of them. A lot of debate came out of this viewing. Teachers were ready to use lesson plans [when they went out to the schools]; they were not shy for talking or getting students into groups; they used learner knowledge and they were well prepared for Life Skills teaching. [The Learning Resources Coordinator] observed me with my own DVD player. She enjoyed particularly the way students were prepared and discussions carried out after the DVDs. After that . . . it [the DVD player] was taken away on the basis that it would be returned, but unfortunately it was not. I showed all of it. It was very good; they enjoyed it, it prepared them. They were to look for the strengths and weaknesses in the lessons. They knew what was expected of them. It triggered lots of discussion and debate. DVD has also been an excellent tool to teach our class. Unfortunately, it worked for the first few days and then was broken. The students really enjoyed the lesson because . . . hearing together with seeing gives them a lot of information that they could interpret and make connections with the reailty. We used the DVD according to the topic. The lecturers’ comments about using the DVDs in class could not be supported by any of the 12 student teachers who were interviewed, since none of them had seen the MTTA DVD at the TTC. One student teacher reported that his friends had seen the DVD the first term – the term before he arrived at TTC. Several students said that they had seen a video or a film on sexually transmitted infections, but not the DVD. The Learning Resources Coordinator also noted that “even though the lecturers said they were excited to use the DVDs, most did not—and had not, each time we monitored the lecturers—mainly because it was too demanding to set up, the DVD players were not working properly, or because electricity was intermittent” (Personal communication, Laura Ivey, August 2008). Guest Lectures. Tutors also were enthusiastic about the guest lectures from area experts. They noted that the students also appreciated the lectures very much, since students and teachers alike were able to submit questions to the experts ahead of time and hear the answers during the lecture. The guest lectures gave students an opportunity to come up with controversial questions they needed to know from an expert – they wrote down the questions and compiled them for the guest speaker who answered them. It probably changed the mindsets of the students, for example, from NAPAM, he explained the background of how he got the virus how long living with it and what doing. One of the students said, “I’m happy just to have more information so I could ask information in advance.” Speakers were answering the questions they had received the day before. The guest lecturers at St. Joseph’s TTC included a doctor from Dedza Hospital; a woman from Lilongwe who was living with HIV; and a “role model” from the American embassy (i.e., a woman who talked about STIs). One of the tutors noted, “The students benefit a lot. There are certain questions they answer that we cannot. It is complementing what we do.” TTC students learn about and learn to teach Life Skills The student teachers reported in interviews that they had attended Life Skills lessons for three to five times per week over two or three terms at the TTCs. The majority of student teachers interviewed had also received the MTTA resource materials. Student teachers had found the course content and resources to be very useful; they reported that Life Skills lessons helped them B-10 to understand more about themselves and to develop important skills and knowledge that enabled them to teach the subject with greater confidence. Student teachers mentioned various Life Skills topics that were helpful to them personally as well as in their teaching. Personally beneficial topics included self-esteem, being assertive (“Now I don’t always rely upon someone else,” said one student teacher), learning about their bodies, critical thinking, effective communication (including writing), and decision-making. Other student teachers noted I have learned to know my strengths and weaknesses. I learned assertiveness, self-esteem, how to prevent how to [get] HIV/AIDS; how to take care of infected peoples. We like it because we know how our bodies are functioning. Student teachers also mentioned particular topics they had studied at the TTC that enabled them to teach Life Skills, and I learned how to teach pupils about STIs and activities that are bad among youth today in society. They also reported changes they observed in their pupils as a result of Life Skills lessons. Pupils learn about their bodies and their lives. Life Skills education is good because it is changing the behavior of the learners at home and at school. Some learners were easily taken up by others; now they are standing on their own. Student teachers noted that not all topics had been covered in their TTC Life Skills classes. This is not surprising, since the MIE handbooks originally were designed for a two-year training program. Since MoEST made the decision to have a One plus One program after the development of the handbooks, tutors had one year to cover material that had been intended for two years. One student teacher requested that enough time be added to Life Skills teaching at TTC so that all topics would be covered and so that she could then “handle all topics” in the classroom. She noted, “We did not study all topics. We learned sex and sexuality, effects of cultural practices, issues, morals and values. But it is difficult for me to teach the students sex and sexuality. I don’t have the language in Chichewa to help explain it to the students when they ask me questions.” Although student teachers had studied Life Skills at the TTC, they studied in English. Since Chichewa is not the first language for all TTC students, and since Standard 1 to 4 lessons are taught in Chichewa, other students are likely to share her assessment: “It is difficult to translate each and every word.” Now that they were in the classroom teaching, student teachers listed the following topics they wished the courses had included or about which they wanted to have more information: critical thinking, coping with stress and anxiety management; planning and entrepreneurship; and more skills in decision-making. A number of these topics were included in the resources manuals provided by MTTA, but not all student teachers had their own materials. Those who had attended Lilongwe and Blantyre Teacher Training Colleges the previous year had received their own resource manuals and reported that they used them for lesson preparation from once to several times per week. One said, “I use the MTTA book whenever I have the lessons – there are four per week . . . I want more books like this.” Student teachers from St. Joseph’s TTC did not have the resource manual. Since there were no plans to reprint the manual, St. Joseph’s TTC tutors decided to keep their set of manuals at the TTC for incoming students’ use only. Student teachers teach Life Skills in primary school Did the Life Skills classes at the TTC make a difference in the ways in which the student teachers taught? One tutor observed, “The taste of the pudding is in the eating. If the students are doing well B-11 it is because of these activities.” Indeed, since the Life Skills subject had been added to the curriculum in recent years, these student teachers quite likely would not have studied Life Skills in primary school. Therefore, it would be expected that the TTC Life Skills classes would have made a difference for student teachers by introducing them to the topics and materials, at the very least. Evaluators observed student teachers’ lessons on the following topics: entrepreneurship; coping with stress; values and staying in school; morals, values, and corruption; ways of dealing with adolescent problems; confidence and self-esteem; causes of conflict; and planning and sexual and reproductive health. Based on the primary school Life Skills curriculum and the workshops and resource materials MTTA had provided, evaluators expected to see student teachers teaching key concepts from the Life Skills curriculum, using teaching and learning materials, and discussing the impact of HIV and AIDS on children’s lives (e.g., caring for sick relatives) or prevention. The vignettes in Box C and Box D below illustrate some of the Life Skills content and pedagogy these student teachers used. Box C. Illustrative Life Skills Lesson - Standard 3 At 11:40 a.m. a female student teacher of Standard 3 in a Blantyre Rural District primary school writes “Life Skills” on the board in Chichewa and then faces the 98 pupils seated at wooden desks in a very warm classroom. To the teacher’s question, “What do you want to be when you grow up?” the pupils say “a teacher,” “a nurse,” “a doctor.” The teacher recalls that when she was a Standard 2 pupil she worked hard so that she could excel and continue on to Standard 3. She asks them to do pair work and discuss what they want to do now that they are in Standard 3. After conversing in pairs, they answer that they want to work hard to go on to Standard 4. She calls on two boys; one’s goal is to become a lawyer, the other wants to be a pilot. “How do you get there?” the teacher asks. “By being obedient to the teacher and doing our exercises in class.” The pupils do not have Life Skills books, so the teacher reads a story to the class from her copy of the pupil Life Skills textbook. The story is about about a 15-year-old boy named Blessings who is in Standard 3. When Blessings was seven years old he decided to leave school and start selling scones. When he realized the importance of school, he went back, but by then he was 12 years old. After she finishes the story, the teacher tells pupils to get into their groups. They move quietly and quickly. She brings out a poster on which she has drawn figures of three children talking and asks the pupils, “Who is listening?” How do you know?” She then asks them to discuss what Blessings thought was useful – selling scones or going to school. They spend several minutes in their groups. One pupil writes down her or his own and other pupils’ answers in an exercise book. The teacher reconvenes the whole class and asks the pupils about Blessings’s decision to sell scones. “Was he right?” she asks. “No,” the pupils reply in chorus. “What do you learn from this?” the teacher asks. A Standard 3 male pupil answers, “If you don’t go further with your schooling you’ll be a mini-bus driver, but if you continue your education you’ll end up a lawyer.” The teacher probes further: “If you don’t go on to school, what problems will you find?” Other children answer, “You will be the envy of your friends.” “Your parents may decide not to support you.” “You won’t be able to do simple calculations or read.” “You may end up a thief and stealing.” The teacher concludes the lesson by asking several recall questions: “How old was Dalitso?” “What was he selling?” “When he went back to school, how old was he?” B-12 Box D. Illustrative Life Skills Lesson - Standard 2 The lesson topic written on the board in Chichewa reads, “Good health habits.” At 11:40 a.m. the student teachers turns to face the 20 boys and 26 girls in the class and asks, “What did we learn about HIV and AIDS?” Pupils answer, “You can get it by sharing needles and razor blades.” The children are sitting in small groups. The teacher writes on the board in Chichewa, “Ways of spreading HIV and AIDS” and then explains the following terms to the class: 1. kulowa kufa. “If a husband dies and a brother takes over and the wife is forced to sleep with anyone to ‘clear her from spirits and misfortunes’ the wife can get AIDS.” 2. fisa. “Hiring someone to bear children – a hyena.” 3. mitala – polygamy. “If this man has AIDS and marries another wife, what does she get?” The pupils answer, “AIDS.” “And if he marries a third one?” “AIDS!” “And spreads it to a fourth?” “AIDS!” 4. chokolo – inheritance of a woman from your elder brother. The teacher reviews each of the terms with the Standard 2 pupils. She has a poster with the terms on it, which is identical to what she has written on the chalkboard. When she asks the question, a pupil stands and answers. When she reaches chokolo, they do not reply, so the teacher repeats the answer and then explains it further. The teacher then moves to a new part of the lesson. “Let’s sing a song about HIV and AIDS.” (The words to the song are: “There is AIDS, be careful of the way you conduct yourself. Mind your behaviors.”) The boys and girls sing and clap. The teacher asks: What are the ways of spreading HIV and AIDS? Pupils answer, “unprotected sex” and “sex.” The teacher adds “polygamy, and other things you have been learning.” She asks, “Is this ‘hyena’ a real hyena? No – it’s a human being.” “We also can revise those four points about ways of spreading HIV and AIDS.” She asks them to take out their exercise books. Each child has and takes out an exercise book, and opens it. Each one begins writing in her or his book. The teacher walks around and checks the books. Pupils appear to be copying the information correctly. The Standard 3 lesson (Box C) illustrates the psycho-social skills, such as making decisions and being assertive, which are included in Life Skills. The skills addressed in this lesson are the importance of persisting in school in order to achieve and ultimately be able to pursue one’s desired career. In the second vignette (Box D), a Standard 2 teacher tells students about various ways in which men can transmit HIV to women. As anticipated, student teachers were teaching key concepts from the Life Skills curriculum. While some teachers and community members questioned the appropriateness of the topics for the age of the pupils (e.g., explaining methods of adult HIV transmission to the infant level in Standard 2), teachers were indeed following the curriculum as was expected of them. The student teachers used visual materials that they had prepared in advance, which was true of each teacher evaluators observed. Teachers in all 12 classrooms observed had prepared a lesson B-13 plan and teaching and learning materials (e.g., charts, posters, pupil role play). In most classrooms pupils had a Life Skills textbook to read; however, several classrooms lacked pupil textbooks and teachers had to compensate, as did the Standard 3 teacher in the vignette above. A teacher in Blantyre Rural District reported they had “only 10 (Life Skills) books for 121 pupils,” while the second school visited in the same district had sufficient textbooks. As is illustrated in the vignettes, participatory teaching methods were used in every class observed. On one or more occasions during each lesson, pupils were asked to work in small groups; and one person from the group reported the small group’s answers to the entire class. With regard to discussing the impact of HIV and AIDS on children’s lives (e.g., caring for sick relatives) as well as prevention, HIV and AIDS or sexual behaviors were mentioned in three of the 12 lessons evaluators observed. Why was this observed so infrequently? Several probable reasons emerged. In some lesson topics (e.g., entrepreneurship), discussing HIV and/or AIDS was not part of the lesson in the student textbook; therefore, student teachers followed the lesson and did not mention it. Knowing the appropriate words to use was another reason given. As one student teacher noted above, she did not have the Chichewa vocabulary for teaching some of the Life Skills concepts. Level of comfort in teaching sensitive information is a third reason. A headteacher noted that not all the student teachers in his school were comfortable teaching this subject to the learners. The Standard 2 student teacher in Box D appeared to be very comfortable teaching the lesson – that is, giving students definitions for the terms under discussion. Her pupils participated in the lesson, echoing back the definitions of the terms to the teacher at the end of the lesson that she had given them earlier. However, the extent to which the learners actually understood the topics is not clear, since there was no true discussion or conversation about HIV and AIDS or a related topic. It should be noted that the headteachers reported (and student teachers confirmed) a range of frequency for Life Skills lesson delivery at the different levels, and there was not uniformity across schools. Life Skills was offered from two times per week to every day, depending on the level (i.e., infant, junior, senior) and the school. While the introduction of the new curriculum and MoEST schedule expectations appeared to be a partial reason for the variation in the timetable, this did not account for all of the variation. SUMMARY Delivery and use of Life Skills component inputs Were the inputs delivered and used? MTTA successfully delivered the resource manual and HIV and AIDS facts booklets and the training workshops for tutors with follow-up monitoring. All Life Skills tutors had received and were using the resource manual and HIV and AIDS information booklet, and the majority had attended all five training workshops. Two tutors did not attend two of the five workshops; another had not attended any since she was not employed by the TTC when the trainings were being offered. MTTA had also facilitated the delivery of guest lectures at each of the TTCs. The Life Skills DVDs also were delivered and used. However, the DVD players delivered to TTCs in 2006 with the Life Skills DVDs were removed in 2007 and had not been replaced by the time of this evaluation in May 2008. The DVDs apparently were received enthusiastically; however, tutors did not always follow through on using them, and the DVD players did not always work. Resourceful tutors continued to use the DVDs with their classes, showing the Life Skills DVD on personal DVD players brought from home. The DVDs and accompanying handbook, How to Use the Malawi Life Skills DVD, were distributed in October 2006 to all TTCs along with a DVD player (and an LCD projector and screen). Tutors were trained to use the DVDs and equipment during the Life Skills training workshop at MIE in September 2006 and again in December 2006. Mid-project monitoring determined that tutors who were using B-14 the DVDs found them to be very useful to their teaching (MTTA Annual Report, 2007). Those tutors not using the DVDs did not use them for various reasons. The reasons tutors gave included not having electricity in the classrooms and not being able to operate the equipment. MTTA reports also noted that some tutors were not able to make linkages between the objectives of the lessons taught on the DVDs and TTC Life Skills lessons. The Learning Resources Coordinator also recalled from her monitoring visits that some DVD players were in their original boxes unopened when she came to observe how the DVDs were being used. As a result, in April 2007 tutors were retrained on how to use and maintain the equipment and sessions were held on how to make connections between the lessons on the DVDs and their teaching objectives in the syllabus. It is not clear why tutors were still waiting for the DVD players to be delivered in May 2008, although TTC classes were not in session and all MTTA personnel were in the field when this evaluation was conducted, so it is likely that DVD player distribution was being delayed until other end-of-project tasks had been completed. The change of personnel in the Washington office appeared to have interrupted the continuity in plans for rolling out the use of the DVDs overall, but MTTA personnel in the field did everything possible (e.g., repeated trainings, on-site monitoring) to try to ensure the success of using the new technology. On at least one occasion the Learning Resource Coordinator tried to observe a tutor when the electricity failed, which prevented the tutor from using the DVD with his class as planned. The reality that some tutors were enthusiastic about the concept of having new technology but were reluctant to handle the hardware in practice, suggests that additional technology supports are needed. The tutors who used the DVD technology both in Life Skills and in Foundations classes, and who found it so compelling for use in class with students, could be assigned roles as “technology mentors” who would work with all other tutors. Assuming that this new hardware works well, these mentors could be the key individuals to assist with the rollout of the use of this technology over time. Relevance, Effectiveness, and Appropriate Delivery of Life Skills Pre-service Inputs To what extent did beneficiaries (i.e., tutors and pre-service teachers) believe the inputs were relevant, appropriately delivered, and effective? Beneficiaries (i.e., tutors and pre-service teachers) found the inputs to be relevant, effective, and, aside from the DVD players, appropriately delivered. Beneficiaries unilaterally deemed the Life Skills materials, training, and supervision or monitoring to be relevant. Classroom observations and teacher interviews indicated that the Life Skills lessons taught in the TTC were effective. For example, MTTA trained tutors to use various participatory methods, the tutors used these methods with their students, and the evaluation team observed the student teachers using these methods (e.g., role play) in their classrooms. The question of appropriateness was raised, but not with regard to delivery of materials or Life Skills topics in the TTC. Rather, at the primary school level, the appropriateness of particular topics and the level at which they should be taught continued to be a source of discussion and some disagreement. This Life Skills is telling children what they are not supposed to understand. Some [community members] are not happy – they say it is taboo (kulaula). In the junior courses they must remove topics to the other classes. For example, we have Mr. Banda’s story where they want to have sex with a Standard 3 learner – “You can get STIs, pregnant, and so on. Learners laugh. “What is ‘pregnant’?” It is not age appropriate, not according to the level of the pupils. Previously learners were shy but now they have opened up. They can discuss things concerning body parts. They like the subject. Some colleagues feel the subject discusses things that are taboo, but on the other hand it helps them to learn skills which are essential to life. B-15 Learners expressed shyness at first due to vocabulary used pertaining to the bodies but [show] full participation at a later stage. Frequently interviewees would indicate that members of the community disagreed with the teaching of Life Skills. The community thinks it is a subject which teaches pupils obscene language. The parents feel offended because they [pupils] know things that they are not supposed to know by that age. The subject is good. Learners are able to know what to do when things happen to them – how to behave themselves. The community sometimes fears to talk to the pupils, but we teachers talk freely. Several interviewees indicated that community members had been opposed to the teaching of Life Skills, but they were coming to understand its importance. At first some of the community (e.g., at a school in Chiragu), when the teacher was talking about parts of the body, the children would tell the parents and parents would say, “that is obscene language” I don’t want you to say that.” Now they hear it [language about body parts] on the radio and TV – not from the teacher only. It has changed. These comments suggest that stakeholders’ understandings of which Life Skills topics are “appropriate” and “relevant” in Malawi may indeed be changing. Impact To what extent did the Life Skills pre-service education inputs make a difference for tutors, for TTC students, and for their pupils? In 2004, USAID Malawi had called for an intervention in pre-service teacher training that would (1) facilitate the development of knowledge and skills that allow teachers to avoid HIV/AIDS infection; (2) enable teachers to facilitate open discussions in the classroom about HIV/AIDS and sexuality; and (3) ensure that teachers have practiced and mastered participatory methodologies to facilitate skills development and to implement effectively the Life Skills for HIV/AIDS Education curriculum. Triangulated data from TTC tutor focus group discussions, from pre-service teacher classroom observations, and from interviews with teachers indicate that the MTTA intervention in pre-service Life Skills teaching was successful in terms of tutors’ professional and personal knowledge and professional skill development, student teachers’ professional and personal development, and tutors and student teachers’ effective delivery of Life Skills lessons in the classroom. MTTA clearly met the three program objectives requested of the intervention. Tutors and students alike described the acquisition of new knowledge in various areas. Several tutors and students said they had learned from Life Skills teaching and workshops how to avoid being infected with and transmitting HIV. One tutor noted that the materials and training raised tutors’ awareness about not having affairs with their students and thus avoiding the possibility of infecting their students with HIV and AIDS. He said, Not only students are benefiting quite a lot from LS but also ourselves. . . . . For mainly the female students, lecturers can be tempted, but we now have something to assist us to [change our behavior] and also to have empathy. They are also children of other people. What if other people are doing something with our children? Tutors noted that the MTTA Life Skills training modeled and allowed them to practice participatory methods that they could use with their TTC students, who would in turn be able to use with their primary school pupils. With regard to “facilitating open discussions in the classroom about HIV/AIDS and sexuality,” these discussions took place in the MTTA trainings where MTTA staff modeled how to facilitate open discussions in the classroom about HIV/AIDS and sexuality. However, these open discussions were not observed at the TTC (since classes were not in session B-16 and could not be observed) nor in the primary schools. Student teachers were observed using participatory methods (pair work, small group work, role play, teacher-pupil interaction even in large classes) rather than only whole–group choral response. However, an “open discussion” with pupils was observed in only one Standard 7 class, and the topic was “entrepreneurship.” HIV/AIDS was not mentioned in that class although when questioned the teacher said he planned to make the link between entrepreneurship and avoiding HIV/AIDS during the next lesson. MTTA indicators and monitoring and evaluation plan were designed to measure how particular agreed-upon targets were reached rather than to measure extent of impact. A pre- and post-test of TTC students’ understanding of Life Skills’ topics and use of appropriate pedagogy was not administered; therefore, it is not possible to conclude that the MTTA materials and workshops in which tutors participated were influenced definitively or exclusively by the MTTA inputs. Student teachers who had received the resource manuals to keep did report that they referred to them in their lesson preparation. Both the student teachers and tutors mentioned helpful topics that were not elaborated on in the MIE materials but were written up in detail in the MTTA resource materials. Hence, one can infer that the materials and inputs to tutors did influence student teachers’ teaching at least to some extent. More exact measurement of impact would be an important contribution to the field in the future. CONCLUSIONS 1. MTTA Life Skills pre-service interventions were successful. The materials and training enabled tutors and teachers to acquire new knowledge and skills. The inputs made an important impact on tutors and on at least one cohort of teachers, professionally and personally. The tutors who were trained now have increased knowledge about HIV and AIDS; they learned new methods and strategies for teaching Life Skills to their pre-service teacher training students; and, they are teaching these strategies to their students, who are using them in the classroom. The majority of TTC students whom evaluators observed have copies of the resource manual and reported they use it regularly for lesson preparation. 2. Strategies for sustainability are needed. Whether the impact of MTTA in Life Skills teaching at the TTCs is able to be sustained is uncertain, since the MTTA project trainings have been completed and all printed resource manuals have been distributed. New tutors and incoming cohorts of students at Government TTCs have received neither training nor materials and must depend on their colleagues or TTC libraries for access to these resources. For the influence to not only continue at the current level but also to be intensified and extended, the materials will need to be reprinted for all new TTC tutors as well as for all new students to carry with them to the field. Guest lectures appear to be sustainable in some TTCs, but not in others. Some TTCs continued to invite guest lecturers either without funds or with their own funds. Others said guest lecturers required some remuneration, and it was not possible to continue inviting one person per term. This is a valuable project input, which appears to be sustainable on a larger scale only if TTC budget or project funds can pay or if speakers are willing to give lectures with little or no remuneration. St. Joseph’s TTC continued to invite guest speakers from organizations that wanted to share their messages about HIV and AIDS. Tutors at other TTCs noted that speakers they had invited “expect something for preparation and travel and materials.” The value of the input needs to be weighed against limited resources, and administrative decisions made about the extent to which the lectures can be continued. One strategy for sustainability suggested by tutors at two TTCs is to introduce a regular (annual or semi-annual) one-day workshop for all Life Skills tutors to come together to share experiences. This strategy worked well in MTTA Professional Development Conferences for teachers (see in-service component evaluation, Part A). It would be valuable to initiate the professional conference at the teacher training level as well, beginning with a focus on content and methods of teaching Life Skills. B-17 3. ICT in Pre-service Teacher Training. MTTA staff solicited feedback from tutors about the use of DVDs and DVD players at the TTCs, and they observed tutors teach using the DVD on one of the monitoring visits. They observed some Life Skills tutors use the DVDs skillfully to prepare students for what they would see in the classroom when they went out to teach. Some tutors who taught Foundations classes also used the DVDs to prepare their students for general classroom teaching, and observed that this enhanced students’ ability to think critically about the teaching and learning process. Other tutors, however, indicated that they were having trouble making connections between the Life Skills lesson objectives in their MIE guides and the DVD classroom lessons, or figuring out how to use the technology. Subsequently, MTTA staff reviewed key concepts and how to use the technology in the TTC training workshops. MTTA demonstrated the potential of including this kind of technology in pre-service teacher training. It also highlighted the importance of developing appropriate training strategies and to consider providing mentoring to those who are less comfortable using new technology by those who have enthusiastically embraced its possibilities. 4. Impact evaluation. Some USAID-funded projects have encountered difficulty delivering inputs as promised. Concern for accountability has necessitated a focus on monitoring project inputs and measuring whether inputs have been delivered and targets met. However, the entire MTTA project, this component included, encountered no such problem. Inputs were delivered and targets were met and exceeded. Therefore, as USAID and other international development partner agencies, researchers, and evaluators advocate for evaluating and measuring carefully the impact of interventions to try to answer impact-related questions (e.g., see Part A of this report), it will be important to consider incorporating an impact measurement design into the next phase of delivering Life Skills pre-service training to tutors, TTC students and primary school learners staff development in Malawi. B-18 REFERENCES The Global Fund (2008). Malawi and the Global Fund. Switzerland: Geneva. Retrieved on July 14, 2008 from http://www.theglobalfund.org/programs/countrysats.aspx?CountryId+MLW&lang=. Malawi Teacher Training Activity (2007). January-December 2007 Annual Report. Malawi: Zomba United States Agency for International Development. (2004). Request for Quotations; Malawi Teacher Training Activity (MTTA) – A part of the Africa Education Initiative. B-19 Annex E: MTTA Pre-Service Life Skills Evaluation Schedule Date Day Location/comment Activity Thur, Apr 24 1 Lilongwe Met with Sydonio in Lilongwe to discuss the evaluation design and instruments. Fri, Apr 25 2 Lilongwe Visited schools in Lilongwe Rural where pre-service teachers were teaching Life Skills lessons (a.m.); conducted focus group interview with Lilongwe TTC tutors (p.m.). Sat, Apr 26 3 Lilongwe to Blantyre Traveled from Lilongwe to Blantyre. Overnight in Blantyre. Sun, Apr 27 4 Blantyre Analysis of manual and interview notes. (Overnight in Blantyre) Mon, Apr 28 5 Blantyre Visited schools where pre-service teachers were teaching Life Skills lessons; conducted focus group interview with Blantyre TTC tutors. Tues, Apr 29 6 To Zomba Met with Simeon Mawindo, MTTA team, and HASCI Club evaluator. Wed, Apr 30 7 To Dedza Visited two primary schools where pre-service teachers were teaching Life Skills lessons; interviewed one Dedza TTC tutor (a Life Skills manual author). Thur, May 1 (Nat’l holiday) 8 Zomba Interviewed two St. Joseph’s tutors attending a workshop in Ntcheu. Returned to Zomba. Analyzed interview notes. Fri, May 2 9 Blantyre Finalized interview notes and began analysis. Sat, May 3 10 Blantyre Debriefed with COP Simeon Mawindo and MTTA team. Departed Blantyre for USA. B-20 Annex F: Questions for Lecturers 1. Manuals. Did you receive the Life Skills manual and HIV and AIDS resource guide produced by MTTA? Did all your students? Do you have it now? How often do you use it? How do you use it? What two things in the manual were most useful for you? Is there anything that is not in the manual that you would like to see? 2. Life Skills trainings. Did you attend all the Life Skills manual trainings? (How many?) Were the trainings useful? In which areas of the curriculum do you think the training was especially helpful? 3. Monitoring of tutors after the trainings. Did an MTTA person observe you teach and give you feedback? Was it useful? In what ways? 4. Life Skills DVDs and DVD players. Did you receive the DVD on Life Skills? the DVD player? Did you use the DVD? How frequently and in what ways? Did you find it useful? Did the students enjoy it? What were the students’ responses to the DVD? 5. Guest lectures. How frequently did you have guest lectures at the TTC? What were tutors’ reactions the lectures? the students’ reactions? Were the lectures useful? In what ways? Do you have a record of the lectures? Do you have copies of the summary papers that were delivered? Where are they kept now? Have you continued this lecture series? Why or why not? B-21 Annex G: Responses from Group Interviews with TTC Lecturers Code: SJ = St. Joseph’s Teacher Training College BTT = Blantyre Teacher Training College LLW = Lilongwe Teacher Training College Questions 1. Life Skills Manuals 1.1 Did you receive a manual? SJ Yes; I was one of the authors. BTT All tutors received it as did all students in the first group. LLW All tutors and students received manuals. 1.2 Do you have it now? Do student teachers have this manual? SJ Yes, I have it. The first group of students was given books. The students were not allowed to take them but instead we spared copies for the library and leave them in the library. Some tutors have extra copies to lend out. We want to increase more by reprint from USAID. We need to review and revise then add or subtract. Aspects of budgeting are hard for students so we need to include how to do personal budgeting. BTT Yes. The students would like some for this year. LLW We do but students currently do not; only students in the field have them at present. 1.3 How often do you use it? SJ Use it with every topic taught from the curriculum. BTT All the time since it has more information – like morals and values and sexuality – than in the MIE book, so use it as a resource. LLW As a reference to supplement the curriculum from MIE which we use. 1.4 How do you use it? SJ I use it as a resource with the topics that are schemed with MIE. BTT Resource and pick from both. When we use the MIE book we always have this one alongside us. LLW As a reference book and use it frequently when preparing. Use it twice a week because we teach it [Life Skills] twice a week. The textbook does not have enough information. 1.5 What two things in the manual were most useful for you? SJ Messages on STI, HIV/AIDS and counseling on the HIV/AIDS. These topics are great benefits as these students are coming out of Form 4 and are youth. They are still very young and do not have good lifestyle habits. BTT Information on sex and sexuality; communication skills give us more information that in the MIE book; all topics are relevant to current curriculum LLW Effective communication, morals and values, methods are well explained and participatory. Case studies were very good examples and helped me; Life Skills are essential; sexuality activities have details and get positive feedback. 1.6 Is there anything that is not in the manual that you would like to see? SJ Add more in each topic for in-depth information. I would love to see more on entrepreneurship and do research activities to empowering the students. Also how to use profits appropriately. We have also put it in the first term because we think it is important. BTT Communicable diseases need more information; goals and goal setting to help give skills for learners. LLW If it is reprinted, add more participatory methods; more details in “potential and talents”; more detail on traditional practices with positive traditional practices that help prevent HIV; heterosexual practice needs to be stressed. 2. Life Skills trainings. 2.1 Did you attend all the Life Skills manual trainings? (How many?) SJ Yes; all; I was a facilitator. BTT Missed 2 out of 5 lessons; one person attended all. We all meet when we start a new topic and discuss the topic. We give information to each other and can receive more information. LLW Five out of six present today attended. The one tutor who was not trained was not employed by the college when it was offered. 2.2 Were the trainings useful? B-22 SJ Yes and we would like more as knowledge is dynamic. BTT We received training so our learners get LS better. We became confident and well prepared to teach with this training. Courses helped us use different methods for teaching. One method was to move out to the community. Exposed to real life situation. Example we went to the hospital and talked to people living with AIDS. Entrepreneurship was very interesting and to find out how it works. Found out the market people are happy but they have challenges, too. We have not taken our students to market but plan to do so. LLW VERY useful! 2.3 In which areas of the curriculum do you think the training was especially helpful? SJ All useful, wonderful and great. BTT - LLW Methods on how we can face different topics. Some taught (modeled) and we could see it. Plan team invited specialist in the area to come as guest speakers. Shared challenges during session. How we teach gives understanding to learners. How we teach learners in different ways was helpful. Terms like obscene language in our culture…we were orientated on them and how to use them which was helpful. 3. Monitoring of tutors. 3.1 Did an MTTA person observe you teach and give you feedback? SJ Yes BTT Yes LLW Yes 3.2 Was it useful? If so, in what ways? SJ Yes. At the end of the lesson we shared ideas and it helped with my delivery of lessons BTT Yes. We discussed strengths and weaknesses, and it was very useful - and she introduced something for me to try. LLW Yes, they gave us feedback. Immediately both strong teaching and information on LS. In sequential lesson we could share these methods. We could use those methods across other subjects. Very helpful. We could try learner-generated ideas. 3.3 Things you would like to see added? (BTT only) We would like to know the feedback from the students which they (the monitors) did not give us. Not everyone was given the student feedback. 4. DVDs and DVD players 4.1 Were the DVDs and DVD players delivered? SJ Yes. BTT Yes. The DVD did not work and we would like to add it. The DVD players did not work with the DVD. Not compatible. We used our own to show to students. LLW Yes. Each college received the DVD but the player did not work. We were trained and we did not use it since it would not work. It [the DVD] could not be read by player (not compatible). Other DVD subject areas we did not see. 4.2 Did you use it? How frequently and in what ways? SJ Every term BTT Laura observed me with teaching with DVD using my own player and she liked the questions and responses and how I taught. She tried to refine the manual and train with the DVD. The player was taken away and it has not been returned. The DVD worked at the house but not at the school on the school player. LLW No. We did not have the DVD players. 4.5 Did you find it useful? SJ Yes the methods were there for the students to see how it can be done and give diversification to their teaching. BTT It was very helpful LLW - 4.6 Did the students enjoy it? What were the students’ responses to the DVD? SJ They looked at teaching skills and looked at the methodology of teaching and then critiqued it. They were positive and constructive in their ideas and gave suggests to the teaching that was observed. BTT They really enjoyed it. They knew what was expected of them. A lot of debate came out of this viewing. B-23 Strength: Students could watch. Weakness: DVD player not working. There is a new set – Math, science, English; have not seen them. College does not have DVD player. Teachers were ready to use lesson plans [when they went out to the schools]; they were not shy for talking or getting students into groups; they used learner knowledge and they were well prepared for Life Skills teaching. LLW (Did not use it.) 5. Guest Lectures 5.1 Were the guest lectures delivered? If so, how frequently? SJ Yes; once per term BTT One person per term came in. Timothy (8) learners, Queens’s central hospital came and gave talk on AIDS Day. Controversial questions could be written up and given to the guest speaker; lecture came up with topics that were learner given from their questions. LLW Yes. Once per term Types: Guest center talked about HIV/AIDS, Group BLM – sexuality, Welfare Office, Light House every month (VCT and ARVs), National Association for People Living with Aids – How to live positive with HIV/AIDS. We were grateful for the funding for guest speakers. 5.2 Do you have a record of the lectures or copies of the summary papers that were delivered? SJ I have copies but the college did not save any. BTT Yes, sent to Mr. Matope. Copies are not easily gotten. Some were given. Not in library. LLW Yes 5.3 Have you continued this lecture series? SJ Yes, Impact of HIV/AIDS on community; orphan support; basics facts of H/A; It’s very important for all of us to learn. BTT No; they [the speakers] expect something for their work. LLW Yes, Lighthouse right now comes as a guest lecture 5.4 What were tutors’ reactions to the lectures? SJ It was wonderful because they could give comments and to ask questions to learn more. The lectures were specialist and tutors could ask more question. BTT LLW 5.5 What were students’ reactions? SJ About the same as tutors as they could ask questions BTT BTT Learners liked it since it changed attitudes with what they learn. Helped their mind set when they shared in the room. Happy to have more information from guest speaker and the paper questions were great since no one knew what students asked what questions. Students liked the lectures and one day they called all the tutors together to talk to each other and hear a speaker. LLW Very good 5.6 Were the lectures useful? SJ Yes. They were experts. Living documents and copies were given to all the participants to have in hand. BTT LLW Very helpful B-24 Annex H: Questions for Student Teachers 1. Manuals. Did you receive the Life Skills manual and HIV and AIDS resource guide produced by MTTA? Do you have it now? If so, how often do you use it? How do you use it? What two things in the manual were most useful for you? Is there anything that is not in the manual that you would like to see? 2. Life Skills trainings. Did you attend a Life Skills class at the TTC? How frequently? Was it useful? If so, in what ways was it training helpful for your teaching? Was it helpful for you personally? If so, how? 3. Monitoring of Life Skills teaching. Has a head teacher, TTC tutor, or mentor teacher observed you teach a Life Skills lessons and given you feedback? Was it useful? In what ways? 4. Life Skills DVDs and DVD players. Did you see the DVD on Life Skills in your TTC Life Skills class? Did you enjoy it? What were other students’ responses to the DVD? 5. Other resources. What other resources did you receive to assist you with teaching Life Skills? Did you attend guest lectures related to HIV and AIDS at the TTC? Were the lectures useful? In what ways? Do you have a copy of the paper that was delivered? What other resources do you need to teach Life Skills effectively in the classroom? B-25 B-26 Annex I: Schools, Teachers, and Topics of Classroom Observations Primary School Pre-service Teacher Lesson Topic Lilongwe Rural School 1 Njewa T1- M Std 7 T2- M Std 3 Entrepreneurship Coping with Stress Lilongwe Rural School 2 Mbwatalika T1- F Std 4 T2- F Std 7 Values, aspirations, & staying in school Morals & values; corruption Blantyre Rural School 1 Lunzu T1- F Std 6 T2- F Std 3 Ways of dealing with problems adolescents experience Values, aspirations, & staying in school Blantyre Rural School 2 Likulu T1- F Std 7 T2- M Std 3 Basic facts about STIs, including HIV and AIDS Enhancing self-awareness Dedza School 1 Monokera T1- F Std 2 T2 – F Std 5 Dealing with conflicts Characteristics of low and high self-esteem Dedza School 2 Kantchito FP T1- F Std 7 T2 – F 2 Planning & Sexual & reproductive health Good health habits T = Teacher F = Female; M = Male