i Republic of Zambia Ministry of Education _________________________________________________ Learning at Taonga Market: IRI Learning Centres and Community Schools An Evaluation of Interactive Radio Instruction at Grade One in 2005 _________________________________________________________ _ Grade 1 Learners at Leopards Hill IRI Centre doing the dictation segment of the achievement test, February 2005 CONTACT INFORMATION Learning at Taonga Market in IRLCs and Community Schools: An Evaluation of Interactive Radio Instruction at Grade One in 2005 was conducted by the Department of Open and Distance Learning of the Ministry of Education with technical assistance and financial support from QUESTT, a USAID-funded project. Department of Open and Distance Education Ministry of Education P.O. Box 50093 Lusaka Zambia Office phone: +260-1-254 330 QUESTT (Quality Education Services Through Technology) Project Education Development Centre 4102 Mutenguleni Road Private Bag 542x, RW Lusaka, Zambia Office phone: +260-1-266 829 TABLE OF CONTENTS EXECUTIVE SUMMARY.............................................................................................................. H ...... 1 H H1.0 INTRODUCTION ..................................................................................................................... 4 H H1.1 BACKGROUND ......................................................................................................................... 4 H H1.2 ADAPTATION OF NEW BREAKTHROUGH TO LITERACY (NBTL) TO IRI.............................. 5 H H1.3 PURPOSE OF THE EVALUATION .............................................................................................. 6 H H2.0 EVALUATION METHODOLOGY......................................................................................... 8 H H2.1 SAMPLING DESIGN .................................................................................................................. 8 H H2.2 ASSESSMENT OF LEARNING ACHIEVEMENT.......................................................................... 9 H H2.2.1 Test Planning.................................................................................................................... 9H H2.2.2 Task Selection and Item Writing .................................................................................... 10H 2.2.3 H Piloting the Test.............................................................................................................. 10H 2.2.4 H Review by a Panel of Experts .......................................................................................... 10H 2.2.5 H Live Test Administration................................................................................................ 11H 2.3 A H SSESSMENT OF THE IMPACT OF SUPPORT .......................................................................... 11 H 3.0 FINDINGS AND DISCUSSION............................................................................................ H 12 H 3.1 D H EMAND FOR IRI .................................................................................................................. 12 H 3.2 P H ROFILE OF IRI LEARNERS IN IRLCS AND COMMUNITY SCHOOLS.................................... 14 H 3.2.1 H Sex and age of learners.................................................................................................... 14H 3.2.2 H Orphan Status................................................................................................................. 16H 3.3 A H TTENDANCE ........................................................................................................................ 17 H 3.4 P H ERFORMANCE IN ZAMBIAN LANGUAGES, ENGLISH LANGUAGE, AND NUMERACY ..... 18 H 3.4.1 H Overall and subtest performance .................................................................................... 18H 3.4.2 H Performance by sex.......................................................................................................... 20H 3.4.3 H Performance by age ........................................................................................................ H20 3.4.4 H Performance by attendance ............................................................................................ H21 3.5 E H FFECTIVENESS OF IRI AND NBTL: ZAMBIAN LANGUAGES LITERACY........................... H23 3.5.1 H Performance by type of school ........................................................................................ H23 3.5.2 H NBTL assumptions: pacing learners and assessing reading ability............................... H24 3.5.3 H NBTL assumptions: availability of learning materials .................................................. H27 3.5.4 H NBTL assumptions: time spent on pre/post-broadcast activities................................... H28 3.5.5 H NBTL assumptions: local language is also the children’s language of play................... H29 3.6 S H UPPORT AND EFFECTIVENESS OF IRI ................................................................................. H30 3.6.1 H Impact of Partner Support ............................................................................................. H30 3.6.2 H Partner support to mentors: Taonga Cares.................................................................... H32 4.0 CONCLUSIONS AND RECOMMENDATIONS............................................................... H H33 4.1 S H UMMARY ............................................................................................................................. H33 4.1.1 H Demand.......................................................................................................................... H33 4.1.2 H IRI learner profile and performance ............................................................................... H33 4.1.3 H Other factors affecting learning ..................................................................................... H35 4.2 R H ECOMMENDATIONS ........................................................................................................... H36 HREFERENCES.................................................................................................................................... H38 HAPPENDICES.................................................................................................................................... H39 AH PPENDIX A: TEST TASKS, QUESTIONS, POINT VALUES AND SCORING RUBRICS ................... H39 AH PPENDIX B: MENTOR QUESTIONNAIRE ................................................................................... H44 AH PPENDIX C: COMMUNITY FOCUS GROUP QUESTIONNAIRE ................................................... H49 AH PPENDIX D: DESCRIPTORS FOR MASTERY LEVELS BY GROUP................................................... H51 AH PPENDIX E: SPECIFIC RECOMMENDATIONS ABOUT READERS FOR IRI .................................. H52 LIST OF TABLES HTable 1: Number of Learners IRLCs and community schools, 2005........................................... H4 HTable 2: Population and sampling plan, by province.................................................................... H8 HTable 3: Pre-test sampling plan, by province and sex ................................................................... H8 HTable 4: Post-test population and sample, by province and sex .................................................. H9 HTable 5: Literacy, Numeracy and English subtests and skills areas .......................................... H10 HTable 6: Centres and community schools, and learners enrolled .............................................. H12 HTable 7: Number of IRLCs and community schools using IRI, by year and province ........... H13 HTable 8: Grade 1 learners in 2000, 2001, 2003, 2004 and 2005, by sex ........................................ H13 HTable 9: Number of IRI learners from 2000 to 2005, by sex ........................................................ H14 HTable 10: Number of Learners IRLCs and Community Schools, 2005 ..................................... H14 HTable 11: Number of Learners in GRZ schools, 2004................................................................... H15 HTable 12: Recommended age enrolment by grade in 2005.......................................................... H15 HTable 13: Percentage of orphans in IRI centre and GRZ schools, by province and year ........ H16 HTable 14: Grade 1 attendance, by province and sex (in percent)................................................ H17 HTable 15: Post-tested learners aged 7 years and older with 60 percent attendance or more.. H18 HTable 16: Means scores for the overall test and subtests, all learners........................................ H19 HTable 17: Performance by province, (in percent).......................................................................... H19 HTable 18: Numbers of schools above and below overall mean, by province and location..... H19 HTable 19: Means scores for male and female learners.................................................................. H20 HTable 20: Post-test means of IRI learners, by age ......................................................................... H21 HTable 21: Post-test means of IRI learners, by attendance ............................................................ H21 HTable 22: Post-test means of IRI learners, by weeks not taught ................................................. H22 HTable 23: Overall performance, by type of school (in percent) .................................................. H24 HTable 24: Mean scores for the Zambian Language Literacy skills, all learners........................ H24 HTable 25: NBTL Methodology......................................................................................................... H26 HTable 26: Predicted reading ability in Zambian Language literacy (by group membership) H26 HTable 27: Learning materials in GRZ schools and IRLCs/community schools,, 2005............ H27 HTable 28: Amount of time learners spent on pre- and post-broadcast activities ..................... H28 HTable 29: Support from partners..................................................................................................... H30 HTable 30: Post-test means for Nangombe IRI and Luano IRI ..................................................... H31 ABBREVIATIONS AND ACRONYMS DEBS .......................................................................... District Education Board Secretary DODE........................................................ Directorate of Open and Distance Education DRCC .....................................................................District Resource Centre Coordinator EBS ..............................................................................Educational Broadcasting Services EDC ..................................................................................Education Development Centre GRZ .................................................................... Government of the Republic of Zambia IRI ..........................................................................................Interactive Radio Instruction LTM......................................................................................... Learning at Taonga Market MOE ..................................................................................................Ministry of Education OVC............................................................................. Orphans and Vulnerable Learners PDP................................................................ Programme Development and Production POC ............................................................................... Provincial Outreach Coordinator QAA .......................................................................Quality Assurance and Accreditation QUESTT.............................................Quality Education Services Through Technology SEO-ODL.................................Senior Education Officer-Open and Distance Learning TED .................................................................................. Teacher Education Department ZIP .............................................................................................. Zonal In-service Provider Learning at Taonga Market in IRLCs and Community Schools Page 1 EXECUTIVE SUMMARY The Zambian National Broadcast Corporation broadcast Grade 1 radio programmes in 2000 in a pilot exercise. Since then, EBS has been developing programmes for additional grades. In 2005 Learning at Taonga Market was broadcast to approximately 56 159 learners in Grades 1 to 5. The ratio of boys to girls was 49.5 percent to 50.5 percent, respectively. The purpose of this evaluation was to determine whether there still is demand for IRI, who the learners are, whether IRI centres are utilised, and whether children who attend such centres do achieve learning targets as stipulated in the Zambian curriculum. It is also an evaluation that features a new component, that of incorporating NBTL methodologies in IRI to deliver the new Zambian Languages Literacy curriculum. With all 72 districts in Zambia using it, IRI is in high demand in Zambia. Programmes are been used by an increasing number of learners in IRLCs, community schools, and recently in GRZ schools. The increase in demand for IRI is established first from the pattern of establishing IRLCs (from 516 in 2003 to 893 in 2005), the number of students enrolled in IRLCs and community who participate in IRI (from 22 763 in 2003 to approximately 56 233 in 2005), and in that IRI programmes had been developed and were being broadcast for all primary grades. The demand for IRI in community schools just about quadrupled in three years (from 88 in 2003 to 338 in 2005). There is equal participation of boys and girls participate equally in IRI centres. In 2005, overall participation of boys and girls was at 50.5 percent for girls, and 49.5 percent for boys. Enrolment at subsequent grade levels shows that the attrition rates for boys and girls were about equal (50.9 percent of boys compared to 49.1 percent of girls in Grade 5, 2005). Other characteristics of IRI learners are a high proportion of above-age learners, which reflects the difficulty that children have experienced in gaining access to education; hence the IRI program does provide a second chance for such learners. There was a relatively high proportion of Grade 1 learners that were 6 years or younger in 2005, 16.4 percent, compared to 10 percent in 2003. The evaluation also highlights the fact that the proportion of orphans is, in some provinces, much higher for IRI learners compared to GRZ schools. For instance, there is a difference of 13 percent in the proportion of orphans in North Western province. Attendance and actual participation in IRI lessons also depends on whether centres and schools receive daily broadcasts. Mentors reported that radio reception was acceptable in most cases (more than 90 percent in IRLCs and community schools, and in GRZ pilot schools). It was also reported that radios were in good working condition throughout the school year. Data showed that 51.7 percent of the learners attended 80 percent of the lessons or more (120 or more out of 150 lessons); 29.9 percent attended between 60 and 79 percent (90 to 119 lessons), while 18.4 percent attended 59 percent or fewer lessons (90 lessons or less). Attendance of teachers and mentors was also recorded. Eighty-nine (89) percent of the who reported that the learners had not been taught for 1 week at the most when they [mentors] were absent, while 10.6 percent of the learners were not taught for two weeks or more. The most disappointing finding of the evaluation was that performance as reflected by post￾test mean score for 2005 Grade 1 learners was low. The overall mean for the composite score of the Zambian Language Literacy, English language, and numeracy mean for the post-test was 26.1 out of 87 possible points on the whole test (a mean of only 30.0 percent). The mean score for the Zambian Language Literacy component was 5.5 out of 43 (12.8 percent), 6.4 out of 16 (40.0 percent) for the English language component, while the mean for the numeracy component was 14.2 out of 28 possible points (50.7 percent). While the evaluation strongly Learning at Taonga Market in IRLCs and Community Schools Page 2 discourages norm-referenced interpretations and comparisons, we note that IRI learners in IRLCs and community schools performed much lower than their GRZ counterparts in Zambian language literacy and English language, but better than them in numeracy. Several reasons are proffered for poor performance in Zambian literacy subtest, the most important being that learners in GRZ school get more than twice as much time on learning activities, and that GRZ are resourced in a much superior way, in terms of the teachers and learning materials. The evaluation suggests that due to this differences in the learning contexts, performance standards should be set for learners in the three contexts (GRZ schools, community schools, and IRI learning centres). The results of the evaluation indicate that mentors and communities remain motivated and committed to IRI activities that take place in the learning centres. This is due in part, to a successful outreach strategy which includes, among others, providing education and assistance to mentors who experience debilitating health problems. While many suggestions for improvement were made in the report, recommendations for discussion and follow-up action are as follows: 1. That in view of the volume of under-aged learners in IRI centre, attendance policy should accommodate 6 year olds, and provide the option of allowing them to repeat Grade 1, while completely disallowing all learners that are 5 years and below. Repeating the grade level will provide the opportunity to attain the necessary literacy skills. 2. That an IRI programme be developed for the pre-school or pre-primary level. Such programmes would find utility in at least two settings. First, if the opportunity presents itself, community schools (and GRZ schools) will be able to use the programmes for the pre-primary grade. Second, parents who work from home will be able to listen in with their younger children, and in the process prepare children for school and cultivate a culture of participating in their children’s learning activities. 3. That print-rich learning environments be created for learners where centre facilities and other resources permit, readers be recommended and procured for use in pre/post broadcast activities. This will provide an opportunity for further time on learning tasks outside the prescribed activities. 4. Establish Grade level standards for language performance: A standard setting exercise should be conducted for Zambian language literacy and English language skills. The outcome of the exercise would be to describe typical grade level performance, grade level expectations and curriculum standards. The standards should be based on performance data collected from learners. 5. That EDC should create a test of basic skills for Zambian Language Literacy to be administered at the end of Grade 1 to determine if learners obtained basic reading and writing skills in the Zambian language. The test could also be used t assess for progression or retention. 6. That IRI methodology should be part of the certificate and diploma programmes at the Colleges of Education to would allow new graduates to readily meet the demand for teachers who can use IRI in GRZ schools and community schools. 7. That mentors be provide with training on NBTL methodologies through the distance learning mode, or be packaged in CDs and tapes to allow IRI mentors to study while Learning at Taonga Market in IRLCs and Community Schools Page 3 they work and attend face-to-face sessions with lecturers during school breaks. The training program should emphasise the basic tenets of NBTL. It should also teach mentors how to lead reading activities, how to care for readers, and how to use assessment strategies to gauge if children are achieving the literacy goals as set by the curriculum. 8. Design and conduct a study to identify critical success factors for IRI in three contexts: DODE should select a manageable sample of community schools and IRLCs for close monitoring so as to identify critical success factors for IRI, and to investigate the impact of community and partner support in IRI efficiency and effectiveness. Learning at Taonga Market in IRLCs and Community Schools Page 4 1.0 INTRODUCTION 1.1 Background Under the auspices of the QUESTT Project, the present IRI program in Zambia is founded on a mandate to provide an alternative delivery system for the basic education curriculum. Interactive Radio Instruction (IRI) is a teaching methodology in which a radio broadcast guides a teacher and learners through the activities of a lesson. While listening to the radio, learners actively participate in the lesson by singing, reading, writing, answering questions and solving problems in ways that ensure active learning. “Learning at Taonga Market” is the name of the IRI program that was developed for Zambia. Learning at Taonga Market is a delivery of the Zambian primary school curriculum that infuses methodologies such as New Breakthrough to Literacy (NBTL) and Step Into English (SITE) with the IRI methodology. Learning at Taonga Market lessons are written and recorded by Educational Broadcasting Services (EBS) of the Directorate of Open and Distance Education (DODE) of the Ministry of Education (MOE) in collaboration with the QUESTT Project of Education Development Centre, a USAID-funded project. Each lesson consists of a 30-minute broadcast, along with activities that the class completes before and after the broadcast. The activities for each lesson are described in a mentor’s guide. The programme follows the national curriculum and MOE’s calendar of three terms. There are 150 lessons at each grade level with 50 lessons in each term, plus five teacher training broadcasts at the beginning of each term. The Zambian National Broadcast Corporation broadcast Grade 1 radio programmes in 2000 in a pilot exercise. Since then, EBS has been developing programmes for additional grades. In 2005 Learning at Taonga Market was broadcast to Grades 1 through 5. Table 1 shows the number of learners enrolled in Interactive Radio Learning Centres (IRLCs) and Community Schools in 2005, by sex and province. Table 1: Number of Learners IRLCs and community schools, 2005 Province Male Female Total Central 3 959 4 019 7 978 Copperbelt 2 794 3 033 5 827 Eastern 5 072 5 188 10 260 Luapula 1 375 1 263 2 638 Lusaka 6 020 6 538 12 558 North-Western 1 723 1 774 3 497 Northern 3 332 3 128 6 460 Southern 2 283 2 217 4 500 Western 1 224 1 216 2 440 Total 27 782 (49.5) 28 376 (50.5) 56 159 Grade 6 and 7 lessons will be offered for the first time beginning in 2006 and 2007, respectively. The programme is designed to let learners complete seven years of education through the radio lessons and write the Primary School Leaving Certificate Examination. Those learners who pass the grade seven examination can attend upper basic grades in the government schools or enrol in the DODE Alternative Upper Basic Education Programme at distance learning centres. This evaluation focused on the assessment of learning gains and monitoring visits at the Grade 1 level. A sample of learners was tested twice in 2005 to assess their progress. Baseline testing was performed at the beginning of the school year in February, 2005 while Learning at Taonga Market in IRLCs and Community Schools Page 5 achievement testing was completed during the end of October and early November. The gains of learners in IRLCs were compared with learners from community schools. Teachers and community groups were interviewed to determine the types of support they receive. This report describes the results of the interviews and the testing. 1.2 Adaptation of New Breakthrough To Literacy (NBTL) to IRI IRI was designed to provide an educational programme for youth who would not be able to attend regular schools. Community schools that use IRI may provide instruction through other programmes adopted by the school in addition to IRI. However, for learners attending IRLCs , IRI is the only formal programme of instruction. To provide learners with relevant instruction, the programme needs to use the content and methods proscribed by the Ministry of Education curriculum. One imperative of the Ministry of Education has been for primary schools to apply new and more effective approaches to reading. From the 1960s through to the late 1990s, the language of instruction and of initial literacy at Grade 1 was English. Aware that children were having difficulty obtaining literacy, the Ministry of Education redefined the language policy in Educating Our Future (1996). Policy 8 under “Basic Education Quality and Curriculum” states: Officially English will be used as the language of instruction, but the language used for initial literacy learning in Grades 1–4 will be one that seems best suited to promote meaningful learning by children. (Republic of Zambia Ministry of Education, 1996:45) In accordance with its new policy, the Ministry of Education developed the Primary Reading Programme to help children develop desirable levels of literacy in both a Zambian language and English. As part of the Primary Reading Programme, the New Breakthrough To Literacy (NBTL) programme was developed to teach learners to read and write in a familiar Zambian language during Grade 1. NBTL was first introduced into Zambian government schools in 2000 and by January 2004 it was being used in all government primary schools. The initial version of Learning at Taonga Market for Grade 1 provided very little instruction in reading and writing—all of it in the English language. DODE wished to pass on the benefits of the NBTL methodology to IRI learners, so the Grade 1 lessons were revised in 2003. The revised version of the Grade 1 programmes is significantly differently from the initial version in that reading and writing skills are emphasized and the target language is the local Zambian language. Although the lessons are broadcast in English, mentors and teacher tailor lessons to the local audiences by translating the content during pauses in the broadcasts. The mentor’s guide provides lesson plans for each day. Each lesson includes translations of target sentences in each of the seven Zambian languages. The revised Grade 1 lessons include other aspects of the NBTL methodology such as a focus on phonics and use of the language experience approach. The focus on phonics provides practice in associating the sounds of the local language with written script. When introducing children to a phoneme, the teacher uses a picture or object to elicit a word with the sound. Children then tell the teacher other words that contain that sound. These words are written on the board and children are asked to underline the letter that represents that sound. In terms one and two the sounds of letters are used rather than letter names, although children will pick up these names as they are used in instructions to the teacher. Letter names and the alphabet are formally introduced in term three where alphabetical order exercises are undertaken. The language experience approach is used to introduce Learning at Taonga Market in IRLCs and Community Schools Page 6 target vocabulary and sentences. The teacher uses a poster and/or sound effects to elicit the target word or sentence from the children. After children say the target word or sentence, the mentor writes it on the board and the children read it. Children will then be asked to find specific letters or words in the sentence, and play games with these words to ensure that they become very familiar with them. NBTL methodology provides for moving learners from Stage 1 through Stage 3. At Stage 1 learners become accustomed to the daily literacy routine and are introduced to the first set of core vocabulary and target sentences. They develop basic literacy skills and learn to recognize that what is spoken can be written down and read. By the end of Stage 1, learners should be able to read at least half of the 13 words used in the initial set of nine sentences. During Stage 2, learners practice reading and writing additional target sentences. They develop the ability to recognize the sounds that go along with letters and to read and write sentences that describe scenes in the conversation posters. A learner who is able to read the words and sentences with ease and write a short story about one of the conversation posters is said to have “broken through” to literacy. A child who has broken through to literacy can begin to do independent reading and writing activities at Stage 3. The radio lessons lead classes through reading and writing activities for each set of core vocabulary and target sentences. Beginning in term 1, mentors and teachers assess each child’s ability to read the core vocabulary that is being covered. Extra help is given to learners who are unable to read the vocabulary. During the course of the first year, children are introduced to six sets of core vocabulary and target sentences that go along with the conversation posters. By the end of the year, children are expected to write several sentences in their first language about a picture in a conversation poster. Given that 2005 was the first year in which the revised version of Grade 1 Learning at Taonga Market was broadcast, a decision was made to evaluate IRI at the Grade 1 level. The evaluation will, among others aspects, investigate whether the newly incorporated aspects of NBTL have resulted in an increase in efficiency and effectiveness in the teaching and learning of literacy in a Zambian language. 1.3 Purpose of the Evaluation The overall goal of this evaluation is to determine if IRI is an effective resource at Grade 1 in IRLCs and community schools. The evaluation questions are as follows: 1. Is there a continuing demand for IRI in Zambia? 2. What are the characteristics of the children who participate in IRI? 3. Do learners attend daily broadcasts at the learning centres? 4. Are learners achieving functional literacy skills in a Zambian language? 5. Are learners achieving English language, and numeracy skills as expected at Grade 1 level? 6. How does mentor, community support and partnership support affect effectiveness of IRI? The first evaluation question addresses the issue of demand in terms of new IRI learning centres being established and learners enrolled in them, as well as the use of IRI in community schools. Parallels will be drawn with the demand for IRI in GRZ schools. While there is evidence that school age children who are in search of a learning opportunity participate in IRI, the second evaluation will investigate whether the programs continue to Learning at Taonga Market in IRLCs and Community Schools Page 7 attract and benefit out-of-school children experiencing socio-economic hardships, the population for whom they were originally intended. In order to receive the full benefit of IRI programs, learners are expected to attend a learning centre or community school where there is a mentor or teacher that can guide them through the lessons. Our hypothesis is that learners who attend and receive guidance from teachers and mentors will have more success in attaining learning targets than those who do not. One of these learning targets is attainment of functional literacy skills in one of the seven Zambian languages stipulated in the curriculum. The evaluation will probe whether or not the methodology for teaching such skills is achieving the intended results. The last evaluation question addresses attainment of learning targets. A curriculum-based test was used to assess learner performance in Zambian language literacy, numeracy and English language. The test was administered to a sample of Grade 1 learners at the beginning and the end of the school year. The pre-test scores are compared with the post-test scores to assess learning gains achieved during Grade 1. For information relating to effectiveness of support systems for IRI centres, interviews were conducted with mentors, community school teachers, and other who are in partnership with community schools and IRLCs in the provision of education. The assessment results are analyzed according to levels of support to determine the impact of each type of support. Learning at Taonga Market in IRLCs and Community Schools Page 8 2.0 EVALUATION METHODOLOGY 2.1 Sampling Design A pre-test-post-test design was employed to estimate learner characteristics and to assess learning gains over the course of the year. A two step sampling design was used, the first step being a sample of IRLCs drawn from five of the nine provinces. Two urban provinces were selected for evaluation (Lusaka and Copperbelt), two rural provinces (Eastern and Western) as well as one province along the line of rail (Central). Table 2 below indicates that of the 620 IRLCs and community schools in the five provinces, 45 were included in the pretest sample, while 71 were included in the post-test sample. The post-test sample was 18.3 percent of the population. Table 2: Population and sampling plan, by province Province Total number of IRLCs and CSs Pretest sample of IRLCs and CSs Post-test sample of IRLCs and CSs Central 132 10 14 Copperbelt 106 9 13 Eastern 155 10 14 Lusaka 151 10 15 Western 76 6 15 Total 620 45 71 At the second stage of sampling, Grade 1 learners were selected. The total number of Grade 1 learners in the selected schools was determined by figures from the beginning of 2004 with a proportionate number of learners coming from each province. Secondly, testing among boys and girls was to be done proportionately with an equal number of learners from IRLCs and community schools. Table 3 presents details of the planned pre-test sample size along with the numbers of learners who were tested. Table 3: Pre-test sampling plan, by province and sex Province Total 2004 G1 Actual Sample Size Pop. Planned Sample Size Female Male Total Central 4 129 210 108 106 214 Copperbelt 3 048 171 95 77 172 Eastern 4 301 220 110 110 220 Lusaka 3 741 190 86 84 170 Western 1 337 75 39 43 82 Total 16 556 866 438 420 858 The actual sample included a good balance of learners from community schools and IRI centres, with 50.7 percent of the learners coming from community schools and 49.3 percent from IRLCs. In all, 858 learners were tested, making 5.2% of the population. In all provinces but Lusaka, the actual numbers of learners tested met or exceeded the sampling plan. Sampling among boys and girls was done proportionately at each school. This yielded a total of 438 girls and 420 boys. An achievement test was administered to a random sample of Grade 1 learners at the beginning of the school year and near the end of the Grade 1 broadcasts. Learners from both IRLCs and community schools were selected for the evaluation. Baseline testing was conducted in February, 2005. The post-test plan was to sample 1 242 learners (7.5 percent of the population) in the same 45 community schools and IRLCs along with 26 additional schools in the same provinces (See Learning at Taonga Market in IRLCs and Community Schools Page 9 Table 3). Table 4 shows the size of the post-test sample, by province and sex. The number of learners that were tested was approximately 1 310, which was 7.9 percent of the 2004 Grade 1 population. The total of 16 556 reflected below is the population of Grade 1 learners in only five selected provinces. The population of Grade 1 learners in all nine provinces was 24 600 in 2004. Table 4: Post-test population and sample, by province and sex Province Total 2004 G1 Actual Sample Size Population Female Male Total Central 4 129 158 122 280 Copperbelt 3 048 127 112 239 Eastern 4 301 154 111 265 Lusaka 3 741 135 128 263 Western 1 337 127 136 263 Total 16 556 701 (54) 609 (46) 1 310 Test administrators were instructed to administer the test at centres that had good radio reception throughout the year and to select learners who had high or medium attendance. Attendance was rated high if learners attended 120 or more radio lessons, medium if learners attended between 90-119 lessons, or low for less than 90 lessons. Administrators were advised to select learners with low attendance only in situations where it was not possible to select a sample of learners who had attended 90 or more radio lessons. The reason for not testing at centres with poor radio reception and for disfavouring learners with low attendance was that the children would have missed a significant portion of the radio lessons, making it difficult to attribute learning or the lack thereof to IRI programs. In order to avoid biasing the sample in favour of high achievers, teachers were not allowed to select the learners. 2.2 Assessment of Learning Achievement During previous evaluations in 2001 and 2003, a Grade 1 test integrating literacy and numeracy skills was used in IRLCs and community schools. This test was based on the old Zambia Basic Education Curriculum. In 2004, the new Basic Education Curriculum was implemented in Grade 1. The new curriculum places emphasis on Zambian language literacy as taught through the New Breakthrough To Literacy (NBTL) methodology. Many of the NBTL techniques used to teach literacy in a Zambian language were incorporated into the new radio programmes. Learning achievement tests in Zambian language literacy, numeracy and English language skills were administered to learners. A team consisting of EDC consultants, the EDC Literacy Advisor, the EDC Research Advisor and the EBS Grade 1 writers was assembled to conduct test development activities. 2.2.1 Test Planning The first task in test planning was to review the contents of the new curriculum. This included selecting terminal objectives that should be mastered by learners during Grade 1, as well as important developmental objectives that are emphasised during the first year. A test plan was developed to determine skills to be assessed, and the level at which they would be assessed. As in the government curriculum for Grade 1, the Zambian language literacy section of the test focuses on reading and writing skills, while the English language assessment focuses on speaking skills. Skills selected for assessment in Grade 1 numeracy are counting, addition, subtraction and the ability to identify and draw shapes. Learning at Taonga Market in IRLCs and Community Schools Page 10 2.2.2 Task Selection and Item Writing The team revised the Grade 1 test to reflect changes in the curriculum, the most significant change being the inclusion of a Zambian language literacy section based on the fusion of IRI and NBTL methodologies. IRI methodologies used in the Learning at Taonga Market radio lessons emphasise a participatory learning approach, where learners acquire knowledge and skills through activities. When selecting tasks for the test, the goal was to use the types of activities practised during lessons to assess learning objectives. The Grade 1 Mentor’s Guide was useful in providing vocabulary covered during radio lessons. This was used to select tasks for English and Zambian Language Literacy. The participatory approach of the lessons, combined with the young age of the learners, lent itself to a test that consists mainly of questions to be answered orally by learners during individual interviews; hence only 4 of the 33 items in the test require written answers. Table 5 below shows the skills covered in each subtest. Table 5: Literacy, Numeracy and English subtests and skills areas Zambian Language Literacy [43 points] Numeracy [28 points] English Language [16 points] • Write from dictation • Write one’s own sentences • Identify words and sentences • Read sentences aloud • Read comprehension 6 6 8 16 7 • Counting aloud • Writing numbers • Addition • Subtraction • Shapes 8 4 4 4 8 • Name common items • Talk about time of day • Talk about actions • Count objects in picture • Use plural forms • State days of the week • Follow commands 4 2 2 2 2 2 2 2.2.3 Piloting the Test The test was piloted at five IRI centres/community schools in Lusaka District in June, 2004. Test administrators included the testing consultants, Senior Education Officers for Open and Distance Learning, Provincial Outreach Coordinators from EDC, members of EBS and the EDC Research Advisor. Pilot testing was preceded by one day of training in which the test administrators read and practised administering the tests. Pilot data revealed that the Zambian language section was too difficult as many of the Grade 1 learners could not complete any of the reading or the writing tasks. Two easier tasks were added to the section: one in which the learner listens to a word and then selects it from a group of written words, and another in which the learner listens to a sentence and then picks the sentence from a group of written sentences. Pilot testing also revealed that some learners who were unable to read the reading comprehension story aloud were nevertheless able to answer questions about the story correctly because of what they remembered from lessons. Thus, the original reading comprehension items were inadvertently testing the ability to recall answers from prior lessons instead of reading comprehension. A new comprehension passage was developed from vocabulary covered during Grade 1. 2.2.4 Review by a Panel of Experts The last step in the test development process was to have the Grade 1 test reviewed by a panel of testing experts. Copies of the test were sent to members of the Examinations Council of Zambia, the Directorate of Standards, the Directorate of Open and Distance Education and the University of Zambia. The panel made observations and recommendations that led to additional revisions. These included administering group tasks Learning at Taonga Market in IRLCs and Community Schools Page 11 before administering items to individuals, placing easier items at the beginning of the numeracy section to help learners feel confident and to capture information about easier skills, and replacing the item on naming colours with naming common objects in the classroom to avoid biasing the test against colour blind learners. Revisions were made to one of the rubrics used to rate sentences as a result of the comments made by the panel. Revisions were made to one of the rubrics used to rate sentences as a result of the comments made by the panel. Panellists’ views converged that the test was a satisfactory tool for measuring the achievement of learners who had completed Grade 1. The final version of the Grade 1 test comprising of 33 items and scoring rubrics is presented in Appendix A. The three subtests of the achievement test battery had different total point values. The Zambian language literacy section had a total of 43 points, while Mathematics and English language each had 28 and 16 points, respectively. All items allowed for partial credit. 2.2.5 Live Test Administration The pre-test was administered from 31st January to 14th February, while the post-test was administered from 25th October to 11th November. Five testing teams of four members each conducted the testing in Central, Copperbelt, Eastern, Lusaka, and Western provinces. The teams consisted of SEO-ODLs, POCs and DRCCs from each of the five provinces. Team members attended two days of training in Lusaka prior to pre-testing. All of the post-test teams included members from the pre-test teams. Test administrators were provided with guidelines for sampling learners, administering tests, and interviewing teachers and community groups. Group administered items were scored differently from individual tasks. Test administrators collected the learners’ written sentences for the first four items. These were later scored by the same test administrators, according to a rubric and entered on a scoring grid. For the remainder of the items, administrators scored items directly on the scoring grid as they administered the questions to each individual learner, using scoring rubrics as provided in Appendix A. 2.3 Assessment of the Impact of Support Two questionnaires were used to collect information about the types of support given to mentors and community schools. Data from these questionnaires was analyzed in conjunction with test results to determine the impact of each type of support on learning achievement. The first was a mentor questionnaire that was filled by the Grade 1 mentor at each of the sampled IRLCs and community schools (See Appendix B). The instrument poses questions about the mentor’s background, learning materials at the school, community participation and support from organisations such as MOE, churches and NGOs. The community focus group questionnaire was filled in by a member of the testing team during a group interview with members of the centre support committee, parents or other involved community members who are present (See Appendix C). Two of the questions had to do with the types of assistance that the community gave to the mentor and to the centre during the year. The responses to these two questions were compiled and coded to see if the assistance had an impact on the achievement of learners. Learning at Taonga Market in IRLCs and Community Schools Page 12 3.0 FINDINGS AND DISCUSSION This chapter presents findings of the evaluation, organised around the five evaluation questions: whether or not there continues to be a demand for IRI in Zambia; the characteristics of the children who participate in IRI; whether or not learners attend daily broadcasts at the learning centres; whether learners are achieving Zambian language literacy, numeracy & English skills as expected at the Grade 1 level; and, the effect of mentor, community support and other partnerships on the effectiveness of IRI. 3.1 Demand for IRI The Learning at Taonga Market programme was initially piloted for Grade 1 with 1,254 learners at 21 centres in Lusaka and Southern provinces in 2000. IRLCs were established in only 35 of the country’s 72 districts, even though the programme was broadcast in a substantial number of districts during the initial nationwide rollout in 2001. In 2003, the number of districts using IRI had grown to 62, and by 2005, IRI all 72 districts in Zambia had IRI learning centres. Since then, programmes have been used by an increasing number of learners in IRLCs, community schools, and recently in GRZ schools. In 2005, IRI programmes were being broadcast for Grades 1 to 5 to 56,233 learners. The demand for IRI can be traced first from the pattern of establishing IRLCs as shown in Table 6. Table 6: Centres and community schools, and learners enrolled Province Total centres, 2004 Closed centres 2004/5 New centres, 2005 Expected number of centres, 2005 Centres that submitted data, 2005 Number of learners 2005 Central 135 18 23 140 132 7 988 Copperbelt 85 4 41 122 106 5 859 Eastern 94 8 97 183 155 10 260 Luapula 71 9 14 76 38 2 638 Lusaka 86 7 90 169 151 12 558 Northern 77 4 44 117 107 6 460 North Western 27 1 7 33 37 3 498 Southern 51 4 54 101 76 4 532 Western 44 2 14 56 55 2 440 Total 670 57 384 997 857 56 233 From the total number of centres, new centres and centres that closed down, the expected number of centres is 997. Some centres were not able to submit data, which means the total number of learners could be much higher than 56 233. Table 6 reveals, also, that 57 centres closed down in 2004 and 2005 (about 8.5 percent of the centres in 2004). Closing down of centres is usually as a result of lack of support from communities, and not a statement of a lack of demand. In fact, use of IRI in community schools, and most recently piloting of IRI in GRZ schools suggests that the demand for IRI was, in 2005, greater that in the past. Table 7 presents the total number of IRLCs, community schools, and GRZ schools using IRI. It shows that the demand for IRI in community schools just about quadrupled in three years (from 88 in 2003 to 338 in 2005) as more community schools found it to be a useful learning resource. In 2005, the Ministry of Education introduced IRI pilot in 7 districts and 36 government schools. Learning at Taonga Market in IRLCs and Community Schools Page 13 Table 7: Number of IRLCs and community schools using IRI, by year and province Province/IRI Centres/ Community Schools 2003 2004 2005 Central Province 96 127 132 Copperbelt Province 63 81 106 Eastern Province 61 91 155 Luapula Province 54 64 38 Lusaka Province 46 86 151 North-Western Province 82 76 37 Northern Province 53 27 107 Southern Province 26 51 76 Western Province 35 44 55 Community Schools 88 188 338 IRI Centres 428 459 519 GRZ Schools N/A N/A 36 GRAND TOTAL 516 647 893 (From DODE and QUESTT, 2005:7) At the initial rollout when more communities were embracing IRI, EDC and EBS were also developing and increasing capacity to deliver IRI programs. Increasing capacity included, among others, developing programmes and learning materials, liaising with relevant national and district education structures; recruiting of mentors, and working with communities establish IRLCs and set up IRI committees. The increase in the number of IRLCs is due to outreach efforts to establish new sites where out-of-school youth can obtain basic education. Despite these efforts to increase the use of IRI, there was a decline in the number of centres in two provinces, Luapula and North Western. These also happen to be provinces with a high proportion of learners who are orphans (47 and 41 percent respectively). The main reason for the decline was lack of support, due to the fact that the POC was not appointed until halfway through the year, in one province, and the ineffectiveness on the available POc in another. The demand of IRI can also be illustrated through Grade 1 enrolments in Table 8, and subsequently, enrolment in other grade levels. The 2005 Grade 1 enrolment of 28,905 learners is over three and a half times the number of learners registered during the initial nationwide rollout in 2001 Table 8: Grade 1 learners in 2000, 2001, 2003, 2004 and 2005, by sex 2000 2001 2002 2003 2004 2005 Male 3 994 6 303 12 033 14 794 Female 3 788 6 368 12 567 14 111 Total enrolment 1 254 7 782 12 671 24 600 28 905 Growth 6528 11 929 4 305 Percentage growth Pilot Baseline 94.1 17.5 The large increase in enrolment between 2000 and 2001 was due to a nationwide rollout of the programme. The increases from 2001 to 2003 and onward represent an increase in demand for the programme throughout Zambia from learners who could not gain access to education in government schools. At 24 600 compared to 12 671 in the previous year, about twice as many learners enrolled as IRI learners in 2004 as a result of use of IRI in community schools. Table 9 presents the total number of learners receiving IRI programmes in community schools and IRLCs. It also indicates the grades that were offered in each year. Learning at Taonga Market in IRLCs and Community Schools Page 14 The highest percentage increase in enrolment (81.1 percent) occurred between 2001 and 2002, following the nationwide rollout of the programme, when many Zambian communities first became aware of IRI and embraced the opportunity for their children. The increase from 2002 onward represents participation at subsequent grade levels, and demand for learners who could not gain access to education in government schools. Table 9: Number of IRI learners from 2000 to 2005, by sex 2000 Grade 1 2001 Grade 1 and 2 2002 Grade 1 to 4 2003 Grade 1 to 5 2004 Grade 1 to 5 2005 Grade 1 to 5 Male 3 994 7,104 11,561 19,412 27,819 Female 3 788 6,989 11,202 19,101 28,414 Total enrolment 1,254 7 782 14,093 22,763 38,513 56,233 Growth 6528 6311 8670 15750 17720 Percentage growth Pilot Baseline 81.1 61.5 69.2 46.1 (From DODE and QUESTT, 2005:12) A comparison between Tables 8 and 9 indicates that demand was generally higher for the Grade 1 level compared to total enrolment for grades being offered. However, the uptake of IRI at community schools in 2003 did affect enrolments. While the Grade 1 cohort grew by 11,929 learners in 2004, a total of 3 821F 1 additional learners participated in radio lessons in Grades 2 to 5. This trend continued in 2005, but with greater growth in Grades 2 to 5: enrolment in Grades 2 to 5 increased by 13,415F 2, while enrolment in Grade 1 increased by 4,305 learners. The manifold increase of IRI learners is mainly due to the fact that 100 community schools participated in IRI in 2004, with the number increasing by an additional 150 community school in 2005. 3.2 Profile of IRI learners in IRLCs and Community Schools The IRI project in Zambia was founded on a mandate to provide an alternative mode of delivery for the existing Zambian curriculum, and to meet the demand for education for vulnerable groups such as out-of-school youth and orphans. This section of the evaluation examines the profile of IRI learners, their sex, age, and orphan status. 3.2.1 Sex and age of learners Girls are systematically excluded from learning opportunities in most developing countries, hence the need to disaggregate enrolment by sex in order to determine whether or not there is equal participation of boys and girls. Table 10 shows that 50.5 percent of the learners were girls, while 49.5 percent were boys in 2005. Table 10: Number of Learners IRLCs and Community Schools, 2005 Grade Male Female Total G1 14 111 (48.8) 14 794 (51.2) 28 905 G2 5 702 (49.4) 5 830 (50.6) 11 532 G3 5 118 (50.6) 4 991 (49.4) 10 109 G4 178 (50.0) 178 (50.0) 356 G5 2 674 (50.9) 2 583 (49.1) 5 257 Total 27 783 (49.5) 28 376 (50.5) 56 159 1 3 821 was obtained by subtracting growth in the number of Grade 1 learners (Table 8) from growth in the total number of learners (Table 9). 2 13,415 was obtained by subtracting growth in the number of Grade 1 learners (Table 8) from growth in the total number of learners (Table 9). Learning at Taonga Market in IRLCs and Community Schools Page 15 Since the inception of IRI, the overall enrolment ratio between boys and girls has always fallen within one percentage point, an indication that IRI provides equal opportunity and access to education to girls and boys (DODE and QUESTT, 2005:12). This result is in direct contrast with data from GRZ schools, where there were consistently more boys participating in Grades 1 through 9 than girls (a range of 2.4 - 4.3 percentF 3 registered in the period 2000 to 2005). A similar ratio of male and female learners is maintained within each Grade level, which suggests also, that attrition rate for boys and girls was about equal. Zambia IRI, and indeed IRI in general, has a good track record of providing equal access and participation opportunities, unlike in the formal school system where dropout rates for girls tend to be higher than those for boys. Table 11 shows that the percentage of girls in Zambian government schools declines steadily with increasing grade levels. Table 11: Number of Learners in GRZ schools, 2004 Grade Male Female Total G1 148,033 (50.0%) 148,125 (50.0%) 296,158 G2 142,803 (50.3%) 141,230 (49.7%) 284,033 G3 146,562 (50.6%) 143,301 (49.4%) 289,863 G4 144,966 (51.6%) 135,986 (48.4%) 280,952 G5 128,292 (52.4%) 116,573 (47.6%) 244,865 G6 114,233 (52.8%) 102,060 (47.2%) 216,293 G7 112,417 (55.3%) 90,716 (44.6%) 203,133 (From Republic of Zambia Ministry of Education, 2004:M102) The proportions of girls to boys are similar in GRZ schools and IRI schools from Grade 1 to 3. Grade 4 is not a strong point of comparison in that due to a change in programme scheduling, few learners used IRI at Grade 4 in 2005. However, the figures for Grade 5 show a contrast between the percentage of girls in IRI schools (49.1%) and in GRZ schools (47.6%). The percentage of girls in GRZ schools continues to decline at Grades 6 and 7. It will be useful to trace whether IRI schools continue to attract equal proportions of girls and boys when the programme is offered at Grades 6 and 7. Because of its flexible delivery mode IRI provides an opportunity for learners, especially girls, to attend lessons, as well as attend to other personal matters in their lives such as caring for younger siblings and participating in economic activities. IRI has also typically attracted a significant number of learners that are 6 years or younger, partly because of the “fun” nature of learning activities, and also because younger children tag along wit older siblings. Table 12 indicates the number of IRI above, below, and at recommended ages. Table 12: Recommended age enrolment by grade in 2005 Below recommended age At recommended age Above recommended age Mean age Grade Level and age Total N percent N percent N percent G1 (7 yrs) 28 936 4 852 17 8 789 30 14 652 51 G2 (8 yrs) 11 534 1 115 10 2 278 20 7 599 66 G3 (9 yrs) 10 111 660 7 1 359 13 7 777 77 G4 (10 yrs) 356 7 2 28 8 321 90 G5 (11 yrs) 5 296 472 9 440 8 4 230 80 Total 56 233 7 106 13 12,894 23 34 579 61 (From DODE and QUESTT, 2005:32) 3 Ministry of Education Annual School Census (ASC), 2005 Learning at Taonga Market in IRLCs and Community Schools Page 16 The mean age for Grade 1 learners was 8 yrs old. Twenty-three (23) percent of the learners were of recommended age, while the majority of IRI learners (61 percent) were above the age recommended for their grade level. The high proportion of above-age learners reflects the difficulty that children have experienced in gaining access to education; hence the IRI program provides a second chance for such learners. Overall, 13 percent of the learners were younger than the recommended age. The proportion of Grade 1 learners that were 6 years or younger in 2005 was considerably higher at 16.4 percent, compared to 10 percent in 2003. Being a system that makes special considerations for disadvantaged learners, some of whom are responsible for the upkeep of their younger siblings, Learning at Taonga Market has allowed a higher degree of flexibility to learners. Rather than miss the opportunity to attend, learners are generally allowed to bring their younger brothers and sister along to the IRLCs. It could also be that parents are using IRLCs in to fill a need for early childhood care facilities (crèches, pre-school or pre-primary). While having under-age learners cannot be discouraged categorically, the fact that the content of IRI programmes is not well suited for their level was mentioned in previous evaluations. In order to derive any benefit from the IRI programme, under-age learners do require specialised teaching approaches that are not part of the preparation and training of mentors, hence the IRI system should cannot afford to be burdened with younger learners. As will be reported in the subsequent sections, the performance of under-age learners tends to be lower than that of age-appropriate and older learners, hence overloading the system with under-age learners may render IRI ineffective as a learning tool. It may be possible at this stage in IRI programming to develop a programme for the pre￾school or pre-primary level to be used in at least two contexts. First, as the opportunity presents itself, community schools (and GRZ schools) will be able to use the programmes for the pre-primary grade. Second, parents who work from home will be able to listen in with their younger children, a highly desirable practice in terms of preparing children for school and cultivating a culture of being part of children’s learning activities. 3.2.2 Orphan Status Since 2003 IRI learners comprised a higher proportion of orphans compared to the proportion of orphans in GRZ basic schools. Overall, 36 percent of the IRI learners were orphans in 2005, compared to 20 percent in GRZ basic schools as indicated in Table 13. Table 13: Percentage of orphans in IRI centre and GRZ schools, by province and year IRLCs 2004 IRLCs 2005 Percent change from 2004 to 2005 GRZ schools 2005 Central 34 36 2 20 Copperbelt 38 45 7 20 Eastern 26 24 -2 20 Luapula 45 47 2 19 Lusaka 41 46 5 22 North Western 28 41 13 16 Northern 20 24 4 17 Southern 35 30 5 20 Western 44 44 0 23 All Provinces 35 36 1 20 Learning at Taonga Market in IRLCs and Community Schools Page 17 Twenty-four percent of the IRI learners were single orphans, while 12 percent were double orphans. In 2005, differences between provinces in the number of orphans varied between 24 percent in Eastern and Northern Provinces to as high as 47 per cent in Luapula Province. Northern and Luapula Provinces had the lowest and highest percentages, respectively, in 2004. An important finding is that the number of orphans continues to increase, in some provinces (Copperbelt, Lusaka and North Western, and Southern) registering increases of 5 percent or more in a space of only one year. It is assumed that many of the orphans lost their parents to HIV/AIDS and related illnesses. A logical response to this problem would be for Learning at Taonga Market to strengthen curricular activities that address issues relating to orphanhood. Also, special attention should be paid to the North Western Province which registered a 13 percent increase of orphans between 2004 and 2005. For instance, while it could be expected that there would be more demand for IRI centres to cater for the increasing number of orphans, it was reported that the number of centres in North Western province was declining (data presented in Table 7). Monitoring in North Western province should be designed to investigate some of these issues so as to come up with answers, first to ascertain that the decline in number of centres does not disadvantage current and prospective learners, and that teachers and mentors are aware of the volume of orphans that they have to work with. 3.3 Attendance Learners are expected to attend a learning centre or community school where there is a mentor or teacher that can guide them through the lessons for them to receive the full benefit of IRI programs. Indeed, learners who attended and received guidance from teachers and mentors in the past have had more success in attaining learning targets than those who attended less. Attendance and actual participation in IRI lessons also depends on whether centres and schools receive daily broadcasts. From the IRLCs and community schools that were sampled, 51.7 percent of the learners attended 80 percent of the lessons or more (120 or more out of 150 lessons); 29.9 percent attended between 60 and 79 percent (90 to 119 lessons), while 18.4 percent attended 59 percent or fewer lessons (90 lessons or less). Table 14 disaggregates attendance data by province and sex. Table 14: Grade 1 attendance, by province and sex (in percent) Attendance Attendance Total Male Female 120 lessons or more 23.2 24.0 22.2 Central 90 to 119 lessons 41.2 40.0 42.4 90 lessons or less 35.6 36.0 35.4 120 lessons or more 61.9 56.7 63.2 Copperbelt 90 to 119 lessons 23.4 26.0 23.2 90 lessons or less 14.8 17.3 13.6 120 lessons or more 79.9 82.1 78.2 Eastern 90 to 119 lessons 12.8 9.7 15.3 90 lessons or less 7.2 8.2 6.5 120 lessons or more 39.3 37.9 41.9 Lusaka 90 to 119 lessons 47.9 50.8 45.7 90 lessons or less 12.8 11.3 12.4 120 lessons or more 52.4 48.2 57.0 Western 90 to 119 lessons 26.2 25.9 26.6 90 lessons or less 21.3 25.9 16.4 Learning at Taonga Market in IRLCs and Community Schools Page 18 The highest attendance rate was recorded by Eastern Province, where 79.9 percent of the learners attended 80 percent of the lessons or more (120 lessons or more), compared to the Central Province where 23.2 percent of learners attended 80 percent of the lessons or more lessons, and more than a third (35.6 percent) attended 59 or fewer lessons. Table 15 indicates that there are more females than males in the group with the highest attendance rate of 80 percent attendance or more or more. If we exclude learners under the age of 7 and learners who had less than 60 percent attendance, the number drops from 1 310 to 909 (69 percent of the original number) as shown in Table 15. Table 15: Post-tested learners aged 7 years and older with 60 percent attendance or more Total Number of Post-tested Learners Post-tested Learners ≥7 Years Old with Attendance ≥60 percent Province Female Male Total Female Male Total Central 158 122 280 71 78 149 (53.2) Copperbelt 127 112 239 70 83 153 (64.0) Eastern 154 111 265 101 134 235 (88.7) Lusaka 135 128 263 103 106 209 (79.5) Western 127 136 263 83 80 163 (61.9) Total 701 609 1 310 428 (61.1) 481 (78.9) 909 (69.4) A comparison of Tables 14 and 15 introduces two important findings on age, sex and attendance. First, under-aged learners contributed to the high attendance figures (81.6 percent of all learners, including 6 years and under, compared to 69.4 percent of only those who were 7 year and older). This result was replicated in all 5 provinces. Second, 78.9 percent of the boys who were 7 years or older attended 60 percent of the lessons or more, compared to 61.1 percent of the girls who are at or above the age recommended for Grade 1. Of the learners for which the programme is intended, there is a higher rate of absenteeism for girls than boys. 3.4 Performance in Zambian Languages, English Language, and Numeracy This section presents the pre-test and post-test scores of all learners and an analysis of the impact of age, learner attendance and teacher attendance on learning achievement. Comparisons of the learners’ pre-test and post-test performance are made to determine learning gains achieved by Grade 1 learners at IRLCs and community schools. First, the scores of learners in IRLCs and community schools are compared, looking at the overall test means and at means for the numeracy, English language and Zambian language literacy subtests. Secondly, the learning gains of girls and boys are compared. Analyses indicate whether there are statistically significant differences between the gains made by learners in IRLCs and community schools and between the gains made by girls and boys. 3.4.1 Overall and subtest performance The overall post-test mean score for 2005 Grade 1 learners was 26.1 out of 87 possible points on the whole test (a composite score of the Zambian Language Literacy, English language, and numeracy), a mean of 30.0 percent. The lowest score was 0, and the highest, 86. The mean score for the Zambian Language Literacy component was 5.5 out of 43 (12.8 percent), 6.4 out of 16 (40.0 percent) for the English language component, while the mean for the numeracy component was 14.2 out of 28 possible points (50.7 percent) as shown in Table 16. Learning at Taonga Market in IRLCs and Community Schools Page 19 Table 16: Means scores for the overall test and subtests, all learners Subtest Means Possible Score Pretest (percent) Posttest (percent) Learning Gain (%) Zambian Language Literacy 43 1.3 (03.0) 5.5 (12.8) 9.8 English Language 16 3.4 (21.3) 6.4 (40.0) 18.7 Numeracy 28 7.6 (27.1) 14.2 (50.7) 23.7 Overall Test 87 12.3 (14.1%) 26.1 (30) 15.9 Pre-test score are baseline scores, an indication of learners’ prior learning in each skill area at the beginning of Grade 1, prior to their exposure to IRI. This information is important in order to determine learning gains. At 23.7 percent, the highest gain was in numeracy and the lowest was in Zambian Language Literacy (9.8 percent). Differences in pretest and posttest score were all significant.F 4 There were no differences in learning gains between learners from IRLCs and those attending community schools. Table 17: Performance by province, (in percent) Subtest Possible score Central Copperbelt Eastern Lusaka Western Zambian Language Literacy 43 04.5 18.1 14.0 13.3 14.7 English Language 16 37.1 56.1 29.9 55.0 21.9 Numeracy 28 50.2 41.5 48.5 62.3 51.4 Overall Test 87 25.3 32.5 28.0 36.6 27.7 Table 17 presents mean scores for the overall test and subtests, by province. Lusaka province had the highest mean, while Central province had the lowest mean. Table 18 shows the numbers of schools in each district that performed above and below the overall mean of 30 percent. Table 18: Numbers of schools above and below overall mean, by province and location Locality Central Copperbelt Eastern Lusaka Western Totals Above the mean Urban Peri-urban Rural 0 2 0 2 1 3 1 1 3 8 1 2 1 0 2 12 5 10 Total 2 6 5 11 3 27 Below the mean Urban Peri-urban Rural 3 3 6 0 1 6 1 0 8 0 0 4 1 0 11 5 4 35 Total 12 7 9 4 12 44 Of the 71 IRLCs and community schools where testing took place, 27 performed above the overall mean. Learners at urban locations tended to perform better than learners at peri￾urban and rural locations: 71 percent (12) of the urban centres scored above the mean, while 56 percent (5) of the peri-urban centres and 22 percent (10) of the rural centres scored above the mean. Lusaka Province, which had the highest mean, also had the greatest proportion of urban centres. 4 Zambian Language Literacy, t = 10.6, p = .00; English Language, t = 18.7, p = .00; Numeracy, t = 17.7, p = .00; Overall Test, t = 20.9, p = .00; Learning at Taonga Market in IRLCs and Community Schools Page 20 3.4.2 Performance by sex In the discussion on the profile of learners in the previous section, we noted that boys and girls had equal access and participated equally in IRI programmes at all grade levels in 2005. Past evaluations revealed, also, that there were no differences on the overall test and subtests (English Language and numeracy at the time), suggesting that the IRI approach was working as well for both sexes. Table 19 below presents means for boys and girls. A significant difference between means of boys and girls was found in the Zambian Language Literacy subtest (t = 2.7, p = .01). There were no significant differences in the other subtests, and in the overall score. Table 19: Means scores for male and female learners Subtest Possible Means Score Male Female Total Significance Zambian Language Literacy 43 6.1 (14.2) 4.9 (11.4) 5.5 (12.8) .01 English Language 16 6.2 (38.8) 6.4 (40.0) 6.4 (40.0) .49 Numeracy 28 14.7 (52.5) 14.0 (50.0) 14.2 (50.7) .49 Overall Test 87 27.1 (31.1) 25.1 (28.9) 26.1 (30.0) .02 Zambian Language Literacy was being tested for the first time in 2005, hence there is no data to compare these results with. However, a significant difference between boys and girls suggests that there is differential learning in favour of boys. A possible explanation is that a higher percentage of boys (78.9 percent, compared to 61.1 percent for girls) was reported to have attended 60 or more IRI lessons as shown in Table 14 above. Since IRI has worked well before, we can also note that literacy in Zambian languages is taught using a combination of IRI and NBTL, hence further investigation should be made into the methodology used in NBTL. It could also be that there are other factors in the teaching of first language literacy that are more enabling for boys than girls. 3.4.3 Performance by age It was estimated in this evaluation that 16.4 percent of all Grade 1 IRI learners are 6 years old or younger. Education policy places such children in pre-school since they have an additional year to go before their turn comes to enrol in Grade 1. Learners of ages 7 through 13 are those who are officially of primary school-going age, while those who are 14 are at an age when they should be entering secondary school. Like other IRI evaluations, Table 20 confirms the strong positive relationship between age of the learners and their performance. Under-age learners are shown to perform lower than those who are 7 years and older. While the overall test mean for those who are 6 years or younger is 19.4 percent, the overall test mean for learners who were 7 or older was 31.7 percent, a difference of 12.3 percent. Another finding, highlighted in Section 3.2.1, is that under-aged learners are increasing in number (16.4 percent of the sampled learners compared to 10 percent in 2003). It was recommended in the 2001 evaluation that since IRI was in high demand, there was need to “delineate a profile for the IRI learner, the ideal “out-of-school” learner for which the program was initially designed, such that the IRI service should be made available to those who need it the most”. Ideally, such profiling would have excluded under-aged learners. However, the current data shows that there is a larger proportion of 4 and 5 year olds, which suggests that the system has so far accommodated under-aged learners, and inadvertently encouraged them to participate in the IRI. Unfortunately this has compromised the efficiency and effectiveness of IRI programme. A policy position on this matter is long overdue. Learning at Taonga Market in IRLCs and Community Schools Page 21 Table 20: Post-test means of IRI learners, by age Age Category Test N Total Score Mean Score Std. Dev. Mean as % Zambian Literacy 159 43 3.4 4.6 7.9 Numeracy 160 28 9.4 6.5 33.4 English 159 16 3.9 3.0 24.3 Below 7 years (16.4 percent) Overall Test 155 87 16.9 9.5 19.4 Zambian Literacy 325 43 3.7 5.6 8.6 Numeracy 324 28 12.4 7.3 44.3 English 327 16 5.7 3.4 35.6 7 years (26.3 percent) Overall Test 316 87 21.7 12.0 24.9 Zambian Literacy 506 43 5.6 8.4 13.0 Numeracy 504 28 15.5 7.8 55.4 English 508 16 6.9 4.0 43.1 8 to 9 years (39.6 percent) Overall Test 494 87 27.9 15.6 32.1 Zambian Literacy 206 43 8.2 10.8 19.1 Numeracy 202 28 18.6 7.2 66.4 English 206 16 7.7 4.1 48.1 10 to 11 years (14..7 percent) Overall Test 201 87 34.3 17.7 39.4 Zambian Literacy 42 43 8.3 10.3 19.3 Numeracy 41 28 19.0 7.0 67.9 English 42 16 8.3 3.6 51.9 12 to 13 years (2.7 percent) Overall Test 41 87 35.4 16.9 40.7 Zambian Literacy 5 43 20.4 19.0 47.4 Numeracy 5 28 21.0 2.7 75.0 English 5 16 10.6 1.5 66.3 14 years and above (0.4 percent) Overall Test 5 87 52.0 21.6 59.8 We suggest that IRI should accommodate 6 year olds, and provide the option of allowing them to repeat Grade 1, while completely disallowing all prospective learners that are 5 years and below. 3.4.4 Performance by attendance The breakdown of post-test scores by the attendance of learners is given in Table 21. Of the number that reported on attendance, 51 percent of the learners attended 120 or more lessons, 30 percent attended between 90 and 119 lessons, while 18 percent had low attendance (less than 90 lessons) . Table 21: Post-test means of IRI learners, by attendance Attendance Test N Possible Score Mean Score Mean as % Std. Dev. High (120 -150 lessons) (51.7 percent) Literacy Numeracy English Overall 670 666 672 654 43 28 16 87 6.95 15.18 6.72 28.75 16.1 54.2 42.0 33.0 9.24 8.23 4.04 16.96 Medium (90-119 lessons) (29.9 percent) Literacy Numeracy English Overall 388 387 390 380 43 28 16 87 4.04 13.95 6.27 24.18 09.4 50.0 39.2 27.8 6.79 7.55 4.02 13.96 Low (less than 90 lessons) (18.4 percent) Literacy Numeracy English Overall 219 217 219 212 43 28 16 87 3.50 11.94 5.08 20.81 08.1 42.6 31.8 23.9 5.41 7.66 3.12 11.83 Learning at Taonga Market in IRLCs and Community Schools Page 22 As expected, learners who had high attendance achieved the highest post-test means, while those who had low attendance earned the lowest post-test means. Learners cannot expect to benefit from the IRI methodology unless they attend lessons regularly. For this reason, results from learners who had low attendance are removed from subsequent analyses of learning gains. Absenteeism by teachers and mentors affects learning. Table 22 shows the post-test scores according to the number of weeks that teachers or mentors were not available to teach and/or direct lessons. Eighty-nine (89) percent of the learners had Grade 1 teachers or mentors who reported that the learners had not been taught for 1 week at the most. According to the same teachers, 10.6 percent of the learners were not taught for two weeks or more. Table 22: Post-test means of IRI learners, by weeks not taught Weeks not Taught Test N Percent of learners Possible Score Mean Score Std. Dev. Mean as % Zambian Lang. Literacy 852 43 5.5 8.0 12.8 Numeracy 846 28 15.6 7.9 55.7 English 855 16 6.3 3.9 39.4 Zero to one week (89.4 percent of the learners) Overall Test 836 89.4 87 27.3 15.7 31.4 Zambian Lang. Literacy 40 43 1.3 3.9 3.0 Numeracy 39 28 13.9 6.9 49.6 English 40 16 5.1 2.5 31.9 Two weeks or more (10.6 percent of the learners) Overall Test 39 10.6 87 20.3 10.6 23.3 There is a significant difference between the overall test means for learners who were not taught for 0-1 week and learners who were not taught for 2 weeks or more.F 5 As shown in a similar evaluation of GRZ school using IRI, comparable data from GRZ schools indicates that the problem of teacher absenteeism does not affect learning (no significant differences). This is may be due to a number of factors, one of which is that it is easier to make alternative arrangements for teaching in a school, unlike in IRLCs which function like one-teacher schools. Also, a classroom environment provides opportunity for learners to engage in independent learning activities even when the teacher is not in attendance. While regulating attendance of volunteer mentors will always be a problem with IRI as it is currently practised, centres should where possible, attempt to channel whatever support they get towards building permanent shelters where some learning materials can be kept. In this way learners can be encouraged to come in everyday and interact with the materials even when the mentor is absent. Also, since a few of the learners in every class are usually over-aged, a formal system of identifying and developing “mentor aides” who will be given the responsibility over the radio and other learning materials can be tried out. This will also equip learners for independent learning as they advance through the grade levels. 5 Overall Test, t = 3.94, p = 0.00 Learning at Taonga Market in IRLCs and Community Schools Page 23 3.5 Effectiveness of IRI and NBTL: Zambian Languages Literacy As mentioned in Section 1.2, one of the important features Grade 1 IRI is that it was recently revised to incorporate Zambian languages literacy, presumably first language literacy for most learners. New Breakthrough To Literacy (NBTL), the MOE prescribed methodology for teaching and learning of first language literacy was for the first time communicated through radio broadcasts using IRI methodology. One of the important questions in this evaluation is to investigate the extent to which learners have been able to acquire Zambian Language literacy, and to examine whether the fusion of IRI and NBTL has increased efficiency and effectiveness in learning. In order to investigate these questions, the evaluation examined learning gains from three sets of learners; IRI learners in IRLCs and community schools, IRI learners in thirty-six (36) GRZ pilot schools, and a control group of non-IRI learners in seven (7) GRZ schools. A condition that needs to be satisfied when learning is communicated through radio broadcasts is that the radio broadcast should be of sufficiently good quality so that both the teachers and the learners are able to follow the content of the radio lesson. Indeed, one of the criteria used by the District Education Board Secretaries (DEBs) to select pilot schools in each of the 7 districts was that schools had good radio reception. Teachers who used IRI were asked, as part of the questions about the school and learning environment, how many days a week they had acceptable radio reception. Teachers reported that radio reception was acceptable in most cases (more than 90 percent in GRZ pilot schools, IRLCs and community schools). It was also reported that radios were in good working condition throughout the school year. Except in cases where learners were forced to sit too far away from the radio because of large classes sizes, schools were adequately provided for. Secondly, IRI is most effective if teachers use the teacher’s guide that accompanies and supplements the content of the radio broadcast for all lessons. The teacher’s guide contains information about the daily lesson, what will be taught during the lesson, and what and how teachers need to prepare for the lesson, as well as exercises to practise with the learners before and after the daily broadcast. Each teacher and mentor was provided with a guide during training. Proper use of teacher’s guides was also monitored at every visit to the schools during the year. Thirdly, IRI methodology is a learning methodology that can be used effectively by certified teachers as well as uncertified teachers who have received training in IRI methodology. Without face-to-face training, learning-by-doing will provide the necessary exposure for the teacher to work with songs and games, as well as the opportunity to manage time in a situation where activities and timing is predetermined. In the case of the present pilot, most teachers had no prior knowledge or experience with IRI before the first training. Therefore, the IRI methodology would produce measurable change if the trained teacher (or a trained replacement) were to teach the class for the whole school year. As a result both learner and teacher attendance was monitored and data was collected. 3.5.1 Performance by type of school Test administrators were advised to apply the afore-mentioned conditions in selecting learners for post-testing. Table 23 below presents mean scores for learners who had good radio reception, who had good attendance of 80 percent of the lessons or more; and were 7 years or older. Learners in IRLCs and community school outperformed learners in GRZ Learning at Taonga Market in IRLCs and Community Schools Page 24 schools (both IRI and control) in English Language, while learners in GRZ schools performed better in Zambian Language Literacy and numeracy. Table 23: Overall performance, by type of school (in percent) Tests Possible score IRLCs/ Community Schools Mean IRI GRZ Pilot Schools Mean Non-IRI GRZ Control Schools Mean Zambian Lang. Literacy 43 17.0 30.5 29.1 Numeracy 28 43.7 66.4 58.2 English 16 57.9 49.4 38.1 Given the emphasis that is placed on first language literacy by the Zambian education policy and the prominence that Zambian Language Literacy enjoys in terms of methodology, a post-test mean of only 17 percent for IRI and community schools as shown in Table 21 was far below the expected performance. A further examination of performance into the skills required for first language literacy in Table 24 reveals that with a mean of only 30.0 percent, learners did not master the most basic of literacy skill, reading discrete words. This, in NBTL terminology, means that the learners did not progress beyond Stage 1. Table 24: Mean scores for the Zambian Language Literacy skills, all learners Zambian Literacy skills Possible points Pre-test Mean (percent) Post-test Mean (percent) Gain (percent) Identifying words and sentences 8 1.3 (16.3) 2.4 (30.0) 1.1 (13.8) Reading comprehension 7 0.4 (05.7) 1.2 (17.1) 0.8 (11.4) Reading aloud 16 0.3 (01.9) 1.6 (10.0) 1.3 (08.1) Dictation 6 0.1 (01.7) 0.6 (10.0) 0.5 (08.3) Writing sentences 6 0.1 (01.7) 0.5 (08.3) 0.4 (06.7) This result is not borne out by the observations of mentors, who believed that the majority of learners had progressed beyond the basics (See Table 24). Another point to note from the table is that distribution of scores for each task favoured the more difficult skills of reading comprehension and writing (19 out of 43, or 44 percent of the test). Since these are developmental rather that terminal skills at Grade 1, the test may have been more difficult than was intended. 3.5.2 NBTL assumptions: pacing learners and assessing reading ability Other reasons for weak performance in Zambian language literacy are explored through the following discussion which examines important NBTL assumptions and actual classroom experiences in executing the NBTL methodology through IRI techniques. The first assumption is that there was close monitoring of reading for each learner, and that like teachers, mentors are capable of assessing readiness of learners to tackle varying levels of reading difficulty (reading stages in NBTL language). In fact, reality is that while NBTL is taught in ability groups, there was no intention nor opportunity to classify learners by ability groups since the IRI methodology is whole group learning approach that organises learning activities around average ability learners. Hence, the expectations when executing NBTL in a formal classroom and an IRI learning should, of necessity, differ. For instance, in an average GRZ class using NBTL, the expectation was that a significant proportion of learners would “break through” in the first year (about 25 percent of the learners), two other groups could come close, while the last group would be somewhere in the middle of Stage 2. Learning at Taonga Market in IRLCs and Community Schools Page 25 Conversely, the IRI environment did not provide adequate support for pupils to break through, hence an inherent expectation of the adapted NBTL/IRI approach was that only a small number of very bright children would “break through”. Most learners were expected learners to get through most of the vocabulary of Stage 2 if they attended ALL broadcasts, while a good number were expected to perform below average. In fact, this could be explain earlier findings, that both learner and mentor absenteeism had a greater effect on performance in the Zambian language literacy than in other subtests. Hence if carried through, the suggestion made earlier, that of retaining under-age learners will, among others, give children another opportunity to come out of Grade 1 having acquired the necessary literacy skills. These are the realities against which performance of IRI learners should be judged. Other methodological differences between GRZ schools and the IRI version of NBTL presented in Table 25 may explain why schools had better performance in Zambian language literacy. Learning at Taonga Market in IRLCs and Community Schools Page 26 Table 25: NBTL Methodology GRZ Schools IRI Learning Centres This is what happens in GRZ schools: ƒ GRZ teachers are taught the difference between a Stage 1, a Stage 2 and a Stage 3 learner. ƒ GRZ teachers assess each learner’s ability to read core vocabulary and use the information to place learners into four pace groups. ƒ Individual learners progress to the next set of core vocabulary and target sentences once they have demonstrated their ability to read the current set. ƒ In addition to assessing each learner’s ability to read core vocabulary, GRZ teachers are trained to test Stage 2 learners in reading, writing dictation and free writing to see if the learner has broken through to literacy. Such a learner is ready to move on to Stage 3 activities, which include independent reading and writing. This is what happens in IRI centres/community schools: ƒ Teachers/mentors are taught how to conduct lessons that allow children to progress through the three NBTL stages. The training does not provide teachers/mentors with explicit information about the 3 NBTL stages. ƒ Teachers/mentors are directed to use assessment information to provide extra assistance to individuals who are having trouble reading the vocabulary. There are no pace groups. ƒ The teacher/mentor needs to ensure that learners have mastered core vocabulary before the next set is introduced via radio lessons. Learners who have not mastered earlier vocabulary may be introduced to new vocabulary during radio lessons. ƒ Learners are given dictation and free writing activities, but assessment focuses on the ability to read core vocabulary. To determine if mentors could make accurate judgements about learner performance at different levels, teachers and mentors in IRLCs and community schools were asked to classify learners as high, medium or low, using their knowledge of how well the children can read in the Zambian language, and the descriptors in Appendix D. A similar exercise in NBTL terms would have been to ask teachers to classify learners as Stage 1, 2 and 3 readers. The presentation of Zambian Language Literacy mean scores in Table 26 below indicates that mentors and teachers classified 356 learners (28.2 percent) has having high reading ability, 362 (28.6 percent) as medium, and 546 (43.2 percent) as having low reading ability. To the extent that learners who were predicted to be of high reading ability registered the highest mean, and those in the low group had significantly lower reading scores, mentors were not arbitrary in their judgements. Table 26: Predicted reading ability in Zambian Language literacy (by group membership) Learners N Mean Pos = 43 Dictation Pos = 6 Writing Sentences Pos = 6 Identifying Words Pos = 8 Reading Aloud Pos = 16 Reading Comprehen Pos = 7 High Group 356 8.9 (20.7) 0.83 0.75 2.62 2.62 2.04 Medium Group 362 6.0 (14.0) 0.85 0.33 2.60 1.30 0.93 Low Group 546 2.8 (06.5) 0.26 0.19 1.39 0.49 0.51 However, the descriptors of medium ability readers stipulate that they should be able to do the following; x write their names legibly, without any spelling errors; x read about 20 vocabulary words; x they attempt to read new words using word attack techniques; x make sentences with familiar words and copy sentences into their books legibly; Learning at Taonga Market in IRLCs and Community Schools Page 27 x construct new sentences using familiar words, even though they have difficulty with this task. A logical expectation would be that at the time of testing, at least half of those that are classified as medium ability readers would be able to identify words, read words and sentences aloud, and answer simple questions about a reading passage. Such a level of achievement would have brought the overall mean for the medium ability readers at 15 points on the Zambian Language Literacy test (34.9 percent), or more. All of those who are identified as high readers would have mastered the same tasks to attain a mean score of over 31 points (at least 70.0 percent), while a good number of them would be attempting more difficult skills of constructing sentences and writing simple passages. The means in Table 26 clearly indicate that learners have fallen short of these standards and expectations; hence their classification by their mentors into these groups must have been based on comparing students to each other – a norm-referenced comparison instead of the criterion based classification espoused by NBTL. A possible area of improvement in the programming would be to be explicit about the milestones in reading ability, and to strengthen the mentor’s ability to discriminate between different reading levels through a systematic training protocol on assessment of reading. 3.5.3 NBTL assumptions: availability of learning materials Apart from the EBS produced mentor guide, teachers need additional print materials to conduct lessons effectively. Indeed, NBTL thrives in a print-rich learning environment. One reason for the low performance of IRI learners in Zambian language literacy may be shortages of learning materials, such as pencils and exercise books. Data on learning materials was compiled for 152 IRI centres and community schools that were monitored from September to December 2005 by POCs, Outreach Assistants and MOE staff in 2005. This was compared to data from 35 GRZ pilot schools, also collected during September 2005. Table 27 shows the supply of learning materials at the GRZ schools and at the IRI centres/community schools. Table 27: Learning materials in GRZ schools and IRLCs/community schools,, 2005 GRZ Schools IRI centres/ community schools Materials Difference N (35) percent N (152) percent percent Chalk 32 91 82 54 -37 Pencil 19 54 40 26.0 -28 Exercise books 12 34 30 20 -14 Readers No data No data 48 32 -- Mentors Guide 33 94 98 64 -30 IRI centres and community schools had fewer materials than the GRZ schools. Having adequate supplies of pencils and books would have made it more difficult for mentors to do writing tasks as directed by the radio teacher, or for learners to practice the writing skills needed to perform well in Zambian language literacy. Although no data was collected during testing, all GRZ basic school were supplied with readers as part of the NBTL kit. In contrast, it was found that only 32 percent of the monitored IRI centres and community schools had readers. While they are not the only reading materials that can provide exposure to reading, a shortage of readers would have made it difficult for learners in IRLCs and community schools to practice reading skills beyond reading tasks generated during lessons. Learning at Taonga Market in IRLCs and Community Schools Page 28 3.5.4 NBTL assumptions: time spent on pre/post-broadcast activities The third assumption is that mentors do spend time on pre-broadcast and post-broadcast activities. Mentors were asked how much time they usually spend on learning tasks before and after the Learning at Taonga Market broadcasts. Sixty-three of the mentors reported on time spent before and after the broadcasts, and all of them indicated that they spent some time on pre-broadcast and post-broadcast activities. Table 28 shows the number of mentors who selected each response, how much time their learners spent on pre- and post-broadcast activities, along with the learners’ mean scores in Zambian language literacy. Table 28: Amount of time learners spent on pre- and post-broadcast activities Pre-broadcast Post-broadcast Time learners spent on activities N Mean for Zambian Language Literacy N Mean for Zambian Language Literacy 5 – 10 minutes 283 4.88 139 4.65 11 – 30 minutes 312 5.22 319 4.69 More than 30 minutes 566 6.10 703 6.14 Total 1,161* 1,161* * There was no data for the amount of time spent by 145 learners All mentors who responded to the questionnaire reported that they spent some time on pre and post-broadcast activities. Mentors reported to have spent more time with learners on post-broadcast activities, as compared to pre-broadcast activities. Thirty mentors (47.6 percent) indicated that they typically spend more than thirty minutes on pre-broadcast activities, compared to 38 mentors (60.3 percent) spending the same amount of time on post￾broadcast activities. The table also shows the Zambian Language Literacy mean post-test scores according to the amount of time spent on pre-broadcast and post-broadcast activities for the learners who were taught by those mentors. As the time spent on pre-broadcast and post-broadcast activities increased, the mean post￾test score increased. Learners who typically spent more than 30 minutes on pre-broadcast activities registered a mean of 6.10 points, compared to those who spent 11 to 30 minutes on pre-broadcast activities (mean of 5.22 points) and those who spent 5 to 10 minutes (4.88 points). Although the means indicate a clear trend, an ANOVA indicates no significant difference among the means for time spent on pre-broadcast activities.F 6 However, the time spent on post-broadcast activities did have a statistically significant difference on achievement in Zambian language literacy. Learners who typically spent more than 30 minutes on post-broadcast activities registered a mean of 6.14 points, compared to those who spent 11 to 30 minutes on pre-broadcast activities (mean of 4.69 points) and those who spent 5 to 10 minutes (4.65 points). An ANOVA procedure indicates that these are significant differences.F 7 On the whole, these data shows that the amount of time spent on activities begins to have an impact on learning and mean post-test scores when teachers and/or mentors spend 30 minutes or more on pre-broadcast and post-broadcast activities. Indeed, GRZ schools spend one hour per day directly on NBTL while additional time may be spent on other learning activities that are designed to reinforce literacy skills. In contrast, learners at a typical IRI centre get total of 40 minutes, approximately 10 minutes on pre-broadcast activities, 10 6 Pre-broadcast: F = 2.35, significance value = 0.10 7 Post-broadcast: F = 4.27, significance value = 0.01 Learning at Taonga Market in IRLCs and Community Schools Page 29 minutes during the broadcast, followed by 10-15 minutes after broadcast. In essence, learners in IRLCs get about half the instruction time needed for optimal learning in the NBTL aaproach. While it is difficult to negotiate additional time for learners and mentors in IRLCs, pre, during, or post broadcast, it should be possible to select a few centres for close monitoring, where monitors can use a checklist to determne whether mentors take learners through all the prescribed pre/post-broadcast activities, at the very least. 3.5.5 NBTL assumptions: local language is also the children’s language of play The NBTL methodology is the Zambian adaptation of Breakthrough to Literacy (BTL), a methodology which was originally conceived as an approach for teaching of English as a first language. BTL is an integrated reading and writing and approach based on language￾experience approach (Mckay, Thompson, & Schaub, 1970). The philosophical basis for BTL is that by age six or seven when children come to a formal school, they have already developed their listening and speaking capabilities in the home language. Literacy learning then takes the form of translating spoken language into a written code and reading the written word. Even though it was first used in the teaching of English, it has achieved great success with African languages, particularly the Bantu languages because of their highly phonetic and syllabic structure. Used for teaching the local languages curriculum to learners who are predominantly first language speakers in Zambia, NBTL was perceived to be a teaching and learning approach that has the potential to provide the necessary foundation in increasing retention at the lower levels. Its emphasis is to promote achievement of learning outcomes for boys and girls by raising reading and writing proficiency in their first language at Grade 1 in regular schools, and more recently in community schools and IRLCs. NBTL as delivered via IRI would have its greatest impact when the language of instruction is the children’s language of play. Instruction should be provided in the children’s’ language of communication and play, and by a mentor who has a high proficiency in language that the children speak. Unfortunately, this was not always the case. In some instances, a teacher/mentor may choose to teach literacy in a language other than the children’s language of play because the language of play is not one of the seven Zambian languages of education. In other instances, a teacher might be teaching literacy in a language that is not his/her first language. To add to the problem, test administrators noted that the language used during testing was not always the language of instruction, as the test administrators were only prepared to test using the Zambian language of education that was predominant in the area. As a result there was a lot of translation, both at the level of the IRI programmes, and on the test, which may have introduced some “noise” in the lessons and testing. This would tend to affect learners in IRLCs more than learners in GRZ schools, as IRLC learners tend to be more heterogenous than those in GRZ schools. In the future, data should be collected about the language of play used by the children, the first language of the mentor/teacher and the language of instruction. This data should be used to analyze the variance due to mismatch between the language of play and the language of instruction as well as the mismatch between the language of instruction and the first language of the mentor. In addition, the assessment of Zambian language literacy should only be administered in instances when it can be conducted in the same language as the language of instruction. Learning at Taonga Market in IRLCs and Community Schools Page 30 3.6 Support and effectiveness of IRI This section describes the effect of various types of support on the achievement of learners in IRLCs and community schools. Sixty-eight (68) mentors/teachers responded to the self￾administered mentor questionnaire (see Appendix B), these being mentors in the sampled IRLCs and community schools. Also, test administrators collected data from 68 community focus groups using a script in Appendix C. Community groups consisted of members of the centre/school committee, parents or other community members who were present at the time of testing. 54.4 percent of the communities indicated that they gave some type of support to the mentor. Committees tended to offer three types of support: food, money or labour in the mentor’s farm field. Most centres offered no support, while only four centres offered three types of support 3.6.1 Impact of Partner Support IRLCs and community schools receive assistance from a variety of partners, including the Ministry of Education, churches, non-governmental organizations (NGOs) and community￾based organizations (CBOs). The Mentor Questionnaire contained questions about the types of support received from these partners and the helpfulness of that support. Table 29 summarises the responses of the 68 IRLCs and community schools that provided information about partner support. Table 29: Support from partners Source of Support Helpful Not Helpful No Support Support from MOE 64 1 3 Support from Churches 27 4 36 Support from NGOs 29 1 37 Support from CBO’s 18 3 45 Support from Other Sources 27 4 36 Providing assistance to 94.1 percent of the sampled centres, the Ministry of Education was found to be most helpful as a source of support. This outcome was expected since provision of IRI is partly facilitated by MOE. This was followed by NGOs who provided helpful assistance to 29 of the centres and schools and churches and other sources that each provided helpful support to 27 centres and schools. Mentors reported that CBOs gave helpful support to 18 of the centre and schools. The impact of support given to centres and community schools remains inconclusive. Two cases described here illustrate why the data concerning centre support were inconclusive. Nangombe IRI Centre is an example of a centre that received substantial centre support and some mentor support, but had low post-test scores. On the other hand, Luano IRI Centre is an illustration of a centre that reportedly received no support, but obtained exceptionally high post-test scores. Nangombe IRI Centre is located in a rural area of Chongwe District in Lusaka Province. It received three types of centre support in 2005: building materials and labour to construct a classroom, as well as food for a school feeding programme. The Grade 1 mentor also reported receiving financial support. Despite substantial centre support and the financial support for the mentor, the post-test results were among the lowest, as shown in Table 30. Learning at Taonga Market in IRLCs and Community Schools Page 31 Table 30: Post-test means for Nangombe IRI and Luano IRI Test Possible Points Means for Nangonmbe (N=28) Means for Luano (N=20) Means for all learners (N=1,310) Numeracy 28 7.3 (26%) 18.6 (66%) 14.2 (51%) English 16 4.4 (28%) 11.7 (73%) 6.4 (40%) Zambian Lang. Lit. 43 2.1 (5%) 26.7 (62%) 5.5 (13%) Total 87 13.8 (16%) 56.7 (68%) 26.0 (30%) One reason for Nangombe IRI Centre’s low results is the manner in which instruction was being conducted. The test administrator noted that in an effort to find time to earn a living the mentors had adopted a system of rotating—teaching at the centre on certain weeks and taking off during others. This system had two negative consequences: it made it difficult for the mentor on duty to conduct pre- and post-broadcast activities adequately and it prevented learners from receiving the guidance and attention they would have gained from having one mentor on a regular basis. Luano IRI Centre is at the other extreme. The centre is located in a rural area of Copperbelt Province. Neither the centre nor the mentor reported receiving any support, yet the results were the best among all of the post-tested centres and schools. One reason for the high Grade 1 results is that some of the learners had repeated Grade 1. At the end of 2004, the mentor decided that some of the learners had not achieved sufficient literacy or numeracy skills to proceed to Grade 2, so she had them repeat Grade 1. The example of Nangombe IRI Centre illustrates the insufficiency of the system used to collect information about mentor and centre support. Although the mentor reported receiving financial support from the centre, it was apparently inadequate or the mentor would not have adopted a rotating schedule to find other sources of income. Although building materials and labour were provided for a classroom block, the structure had yet to be completed, so the centre had not yet received the full benefit of the support. Such support probably would have translated into better assessment results only after it had been completed for at least a year, giving learners access to lessons throughout the cold dry season and the rainy season. It is also not clear whether the feeding programme had been in place long enough to have a substantial impact on learner attendance. To measure the impact of community and partner support, it would be better to trace any benefits of the assistance during the course of a longitudinal study. For instance, the 2005 assessment results for Nangombe could be recorded as a baseline and data could be collected over the course of the following years to trace the impact of the support received. The Zambian language literacy results of Luano IRI Centre highlight the idea that one year may not be enough to achieve the literacy skills that are described in the Grade 1 curriculum. Some of the learners of Luano IRI Centre succeeded at reading and writing in their first language, but it took them two years to learn those skills. It is not surprising that it would take two years to learn these skills, especially in the Zambian context where there is little reinforcement of first language literacy skills outside school. Two generations of adults have not been formally educated to read and write in their first language. In addition, there are few if any newspaper or magazine articles written in the seven Zambian languages of education. Almost all first language reading materials are produced by teachers and learners at schools or are provided by the Ministry of Education. In this context, it is important that learners have access to educational readers published in their first language. It would also be advisable to give learners two years to acquire literacy skills in their first language before they begin learning to read and write in English. Learning at Taonga Market in IRLCs and Community Schools Page 32 3.6.2 Partner support to mentors: Taonga Cares Zambia has a unique programme that supports mentors with personal circumstances to promote their health and wellbeing. Taonga Cares was initiated to support IRI mentors and all community school teachers to undergo volunteer counseling and testing (VCT) for HIV through a local partner, Kara Counseling. Borne out of necessity to assist mentors who teach daily with little or no remuneration, Taonga Cares came to being when one mentor who found it increasingly difficult to teach due to frequent illness discovered that she was HIV positive. Since the Ministry of Education was in the process of making free treatment available to teachers, she approached them for inclusion in the programme, only to find that volunteer teachers were not eligible. Since then, an increasing number of mentors have accessed Kara Counseling or other VCT facilities in their communities. The initial target communities were Kabwe, Choma and Lusaka, with plans to expand throughout Zambia by the end of 2006. In addition to VCT, Taonga Cares will eventually include identifying mentors and teachers as focal point people and training them in peer education, starting and maintaining Post Test Clubs and support groups, preparing to serve as role models in the fight against HIV and to help all teachers and mentors to be an example to their learners and communities regardless of their HIV status so that they can have safe and healthy lifestyles. With the high HIV prevalence in Zambia, Taonga Cares has become a motivation to mentors, and a necessary factor in the sustainability the Learning at Taonga Market system. It would be necessary to find ways to sustain it, even if it means seeking additional funding and support. Learning at Taonga Market in IRLCs and Community Schools Page 33 4.0 CONCLUSIONS AND RECOMMENDATIONS 4.1 Summary This is the third evaluation of IRI at Grade 1 since the nationwide rollout in 2001. The evaluation investigated the original set of questions that have been the subject of earlier investigations – whether there is a demand for IRI, who the learners are, whether they attend at IRLCs, and whether they are learning. It is also an evaluation that features a new component, that of incorporating NBTL methodologies in IRI to deliver the new Zambian Languages Literacy curriculum. Where possible, trends dating from previous evaluations were observed, while comparisons are made between the IRI learners in the GRZ schools and IRLCs/community schools. 4.1.1 Demand The demand for IRI in Zambia was, in 2005, greater that in the past. There are now three main types of IRI users IRI learners in IRLCs, IRI learners in community schools, and learners in GRZ schools. The increase is manifest in the number of districts that have IRI centres, (all 72 districts) the number of community schools and centres that use IRI (from 516 in 2003 to 893 in 2005), the number of students enrolled in IRLCs and community who participate in IRI (from 22 763 in 2003 to approximately 56 233 in 2005), and from the fact that IRI programmes are being developed for all primary grades and were being broadcast for lower basic and middle basic grades. While communities continue to establish new IRLCs, a definite area of growth for IRI will, within the next few years, be community schools. The demand for IRI in community schools just about quadrupled in three years (from 88 in 2003 to 338 in 2005). In view of the fact that the role of EBS/EDC can only be advisory, a response of the Learning at Taonga Market system would be to enhance the capacity of the outreach section to provide the basic resources that would be necessary to kick-start IRI activities in community schools. More importantly, a manageable sample of schools can be selected for close monitoring so as to isolate critical success factors for IRI in the context of community schools. 4.1.2 IRI learner profile and performance In 2005, overall participation of boys and girls in IRI centres was at 50.5 percent for girls, and 49.5 percent for boys. Since the inception of IRI, the overall enrolment ratio between boys and girls has always fallen within one percentage point, an indication that IRI provides equal opportunity and access to education to girls and boys. In contrast to GRZ schools where attrition rates are much higher for girls (52.4 percent of boys compared to 47.6 percent of girls in Grade 5, 2004, with a 10 percent difference between attrition rates of Grade 7 boys and girls in the same year), enrolment at subsequent grade levels shows that the attrition rates for boys and girls were about equal (50.9 percent of boys compared to 49.1 percent of girls in Grade 5, 2005. Equal participation in IRI opportunities is due to flexible learning arrangements that accommodate the needs of girls who would have otherwise been excluded from schooling opportunities because of the role they discharge as care-givers to the younger siblings. It also occurs because the IRI methodology has a deliberate strategy of including boys and girls equally in learning activities. It will be possible, after a few years of using IRI in community schools, to test the veracity of this claim. The expectation is that if the IRI Learning at Taonga Market in IRLCs and Community Schools Page 34 methodology works as expected the gap between attrition rates of boys and in community schools will be narrowed over the years. The high proportion of above-age learners reflects the difficulty that children have experienced in gaining access to education; hence the IRI program does provide a second chance for such learners. On the other hand, the proportion of Grade 1 learners that were 6 years or younger in 2005 was considerably higher at 16.4 percent, compared to 10 percent in 2003. Performance of under-age learners tends to be lower than that of age-appropriate and older learners. While overloading the system with under-age learners may in the long run dilute the effectiveness of IRI, under-aged learners cannot be completely disbarred from attending IRI centres with their older siblings. We suggest rather, that IRI should accommodate 6 year olds, and provide the option of allowing them to repeat Grade 1, while completely disallowing all prospective learners that are 5 years and below. The evaluation further identifies the constituency of under-aged IRI learners as a possible growth area which might need special attention in the form of developing a new pre￾primary level programme to be used by school audiences, early childhood education providers, and under the direction of parents and/or older siblings in the home. Granted, this may not completely eliminate the concern of having younger siblings tagging along to IRI centres; however, it will definitely fill a need for education at that level. Overall, 36 percent of the IRI learners were orphans in 2005, compared to 20 percent in GRZ basic schools. With Copperbelt, Lusaka and North Western, and Southern provinces registering 5 percent increase or more in a space of only one year, a disheartening yet important finding is that the number of orphans is increasing at an alarming rate. It is assumed that many of the orphans lost their parents to HIV/AIDS and related illnesses. A logical response to this problem would be for Learning at Taonga Market to strengthen curricular activities that address issues relating to orphanhood. Sampling of centres for the study proposed above should definitely include the North Western Province which registered a 13 percent increase of orphans between 2004 and 2005. The relationship between orphanhood and learning should be explored, which means that the orphan status of learners should recorded during testing. A statistically significant difference in performance of boys and girls was found for the Zambian Language Literacy mean scores. A more desirable outcome of a non-significant sex differences was recorded in the other subtests. Other than that, strong positive and statistically significant relationships were found between the age of the learners, attendance of learners and attendance of mentors and mean performance in the overall test. As expected, the oldest learners achieved the highest post-test means, while those who had low attendance earned the lowest post-test means. Similarly, learners who had high attendance achieved the highest post-test means, while those who had low attendance earned the lowest post-test means. There was a significant difference in mean scores among the learners who were not taught for 0-1 week and the mean for learners who were not taught for 2 weeks or more. While only 10.6 percent of mentors reported that the missed 2 weeks of school or more, a loss of two weeks for one mentor affects all the learners in his/her class. An immediate need would be to encourage centre committees to mobilise support towards building permanent shelters where some learning materials can be kept safe. In this way learners can be encouraged to come in everyday and interact with the materials even when the mentor is absent. Also, since a few of the learners in every class are usually over-aged, a formal system of identifying and developing “mentor aides” who will be given the responsibility over the Learning at Taonga Market in IRLCs and Community Schools Page 35 radio and other learning materials can be tried out. This will also equip learners for independent learning as they advance through the grade levels. Such a system would have yet another advantage. Data revealed that the amount of time spent on activities begins to have an impact on learning and mean post-test scores when teachers and/or mentors spend 30 minutes or more on pre and post-broadcast activities. This would allow learners the opportunity to invest more time on learning tasks. 4.1.3 Other factors affecting learning Teachers reported that radio reception was acceptable in most cases (more than 90 percent in GRZ pilot schools, IRLCs and community schools). It was also reported that radios were in good working condition throughout the school year. This was due mostly to successful monitoring and response by outreach programme. While learners in IRLCs and community school outperformed their counterparts in GRZ schools (both IRI and control) in English Language, learners in GRZ schools performed better in Zambian Language Literacy and numeracy. However, a majority of learners did not master the most basic of literacy skill [in Zambian Language], reading discrete vocabulary words. This, in NBTL terminology, means that the learners did not progress beyond Stage 1. This was not borne out by the observations of mentors, who believed that the majority of learners had progressed beyond reading vocabulary. An improvement in programming would be to be explicit about the milestones in reading ability, and to strengthen the mentor’s ability to discriminate between various stages of reading. Also, in order to assess if all learners are meeting specified learning targets at the end of the grade, it would benefit schools and centres if a parallel version of the Grade 1 test is availed to all centres to be administered by mentors. Other improvements can be made on the test, the Zambian Language Literacy test in particular. While efforts that were made in the trial testing of instruments during the test development phase are noted, administration of the test to a larger more representative sample of learners provides another opportunity to evaluate the instrument. Such an evaluation has identified a need to rationalize and revalidate the existing assessment Zambian Language Literacy test. An important modification before creating parallel versions or using the test again would be to increase the weighting of items that are likely to give more information (those of medium difficulty and high discrimination value), items which test terminal objective for the Grade 1 level. Conversely, the weighting of items which are associated with more developmental skills should be reduced. The IRI programme made critical assumptions about the NBTL methodology. These included the assumption that mentors are capable of identifying learners at varying levels of reading difficulty, that basic learning materials such as chalk, pencils, exercise books, readers and mentor’s guides will be available, that considerable time will be spent on activities, pre and post broadcast, and that children were being taught in their “language of play”. These assumptions did not hold for a significant number of cases. For instance, most centres had chalk and a mentors guide - only about 30 percent had readers, which means they did not spend adequate time on reading. Also, in a significant number of centres the language used to communicate IRI learning activities and to assess attainment of literacy skills was not the children’s language of play, the mentor’s first language, or the language of the test. Some “noise” may have been introduced during translation of both activities. It would be necessary in the future to collect data on the mentor’s competence in the language of the learners in order to make necessary adjustments in programming. Learning at Taonga Market in IRLCs and Community Schools Page 36 4.2 Recommendations A number of suggestions for improvement were made in the discussion of findings. Some of these are recommended for further actions. 1. Allow under-age learners the option to repeat Grade 1: In view of the volume of under￾aged learners in IRI centre, attendance policy should accommodate 6 year olds, and provide the option of allowing them to repeat Grade 1, while completely disallowing all learners that are 5 years and below. Repeating the grade level will provide the opportunity to attain the necessary literacy skills. 2. Develop an IRI programme for the pre-school or pre-primary level: First, if the opportunity presents itself, community schools (and GRZ schools) will be able to use the programmes for the pre-primary grade. Second, parents who work from home will be able to listen in with their younger children, and in the process prepare children for school and cultivate a culture of participating in their children’s learning activities. 3. Distribute readers for learners or find other ways of creating print-rich learning environments in order to increase time on learning task: Sets of Zambian language readers and English language readers should be selected for each grade level, produced and distributed to community schools and IRI centres (See Appendix E for specific recommendations about readers to be selected for each grade level). This will provide an opportunity for further time on learning tasks outside the prescribed activities. 4. Establish Grade level standards for language performance: It was established before in the discussion about assumptions of the NBTL methodology, that expectations about performance in literacy are different for learners in GRZ schools compared to those in IRLCs and community schools. A standard setting exercise should be conducted for Zambian language literacy and English language skills. The outcome of the exercise would be to describe typical grade level performance, grade level expectations and curriculum standards under the three contexts. The standards should be based on performance data collected from learners. 5. Create a Grade 1 Test of Basic Skills for Zambian Language Literacy: A test is necessary to determine if learners obtained basic reading and writing skills in the Zambian language as stipulated in the curriculum. This test ca be distributed to teachers towards the end of the year, for them to administer it at the end of Grade 1. The same test can be used, in the case of under-aged learners, help mentors determine whether the learner should repeat Grade 1 or proceed to Grade 2. 6. Include IRI in the Teacher Education Curriculum: Include Interactive Radio Instruction as part of the certificate and diploma programmes at the colleges of education. This would allow new graduates to readily meet the demand for teachers who can use IRI in GRZ schools and community schools. 7. Provide mentors with training on NBTL methodologies by distance learning: Develop a training program on delivering NBTL through IRI and package it in CDs and tapes allow IRI mentors to study while they work and attend face-to-face sessions with lecturers during school breaks. The training program should emphasise the basic tenets of NBTL. It should also teach mentors how to lead reading activities, how to care for readers, and how to use assessment strategies to gauge if children are achieving the literacy goals as set by the curriculum. Learning at Taonga Market in IRLCs and Community Schools Page 37 8. Design and conduct a study to identify critical success factors for IRI in three contexts: DODE should select a manageable sample of GRZ schools, community schools and IRLCs for close monitoring so as to identify critical success factors for IRI that are mentioned in other sections of the report, such as the actual content of activities that mentors and learners engage during pre/post broadcast segments, and the whether the children’s language of play coincides with the language of the mentor and the medium through which testing is delivered. The study should also investigate the impact of community and partner support in IRI efficiency and effectiveness. Learning at Taonga Market in IRLCs and Community Schools Page 38 REFERENCES DODE and QUESTT. 2005. 2005 IRI Enrolment Bulletin. Lusaka: Directorate of Open and Distance Education, QUESTT Project and USAID. Kariuki, S., Letshabo, K., Laflin, M., Mutale, K., Phiri, M. and Sakala, G. 2000. October 2000 Evaluation of the Interactive Radio Pilot for Out-of-School Audiences. Lusaka: USAID-Zambia. Letshabo, K. 2003. Interactive Radio Instruction for Out-of-School Children and Youth in Zambia: An Evaluation Report for Grade One Learners, 2003. Lusaka: Educational Broadcasting Services, Ministry of Education. Letshabo, K. and Kariuki, S. 2001. Interactive Radio Instruction for Out-of-School Children and Youth in Zambia: An Evaluation Report for Grade One Learners ,2001. Lusaka: Educational Broadcasting Services, Ministry of Education. Educational Broadcasting Services, Ministry of Education. Mackay, D., Thompson, B. and Schaub, P. 1970. Breakthrough To Literacy. London: Longman for the Schools Council. Republic of Zambia Ministry of Education. 2004. Educational Management Information System. Lusaka: Ministry of Education. Republic of Zambia Ministry of Education. 1996. Educating Our Future. Lusaka: Ministry of Education. Siaciwena, R., Thomas, M., Luangala, J. and Haambokoma, C. 2002. Report on the Rapid Appraisal of the Ministry of Education’s Interactive Radio Instruction Basic Education Delivery System. Lusaka: USAID-Zambia. Learning at Taonga Market in IRLCs and Community School Page 39 APPENDICES Appendix A: Test Tasks, Questions, Point Values and Scoring Rubrics Subject Item No. Task Test Question Point Value Scoring Rubrics 1 Write a sentence from dictation based on a picture from story. In the first picture, mother is carrying a bucket. [In the Zambian language] Write this sentence on your paper: Mother is carrying a bucket. 3 2 Write a sentence from dictation based on a picture from story. In the second picture, mother is drawing water into a bucket. [In the Zambian language] Write this sentence on your paper: Mother is drawing water. 3 0 points: Writes nothing at all, or writing not legible/readable, or the sentence is not related to the dictation. 1 point: Writes one or two words of the dictated sentence legibly, but not the full sentence. 2 points: Writes the dictated sentence fully, correctly and legibly, but with one or more spelling errors. 3 points: Writes the dictated sentence fully, correctly and legibly, with no spelling errors. 3 Write a sentence of your own based on a picture from story. In the third picture, mother is washing clothes. [In the Zambian language] Write your own sentence about picture 3. 3 4 Write a sentence of your own based on a picture from story. In the fourth picture, mother is hanging clothes to dry. [In the Zambian language] Write your own sentence about picture 4. 3 0 points: Writes nothing at all, or writing not legible/readable, or the sentence is not related to the picture. 1 point: Writes one or two words that are related to the picture, but not a full sentence. 2 points: Writes a legible sentence about the picture, but with fewer than three words spelled correctly. 3 points: Writes a legible sentence about the picture, with three or more words spelled correctly. 5 After hearing a word, select it from a group of written words [In the Zambian language] Point to the word “baby”. Point to the word “maize”. Point to the word “chickens”. Point to the word “bananas”. 4 One point for every word identified correctly. Zambian Language Literacy (43 points) 6 After hearing a sentence, select it from a group of written sentences [In the Zambian language] Point to the words “Uncle is drinking tea”. Point to the words “The boy is cooking”. Point to the words “The girl is writing”. Point to “The cow is eating grass”. 4 One point for every sentence identified correctly. Learning at Taonga Market in IRLCs and Community School Page 40 7 to 10 Read the sentences of a story aloud [In the Zambian language] Please read the story aloud: 7. A boy is selling maize at the market. 8. The boy is playing with a ball. 9. Goats are eating the maize. 10. The boy chases the goats. [The story is presented with one sentence per line without the question numbers.] 16 0 points: Not able to read any words at all, or mumbling incomprehensibly. 1 point: Reads sounds or syllables, but cannot read a complete word. 2 points: Reads one or more complete words, but cannot read the complete sentence. 3 points: Reads all of the words of the sentence, but with hesitancy or has to repeat certain words. 4 points: Reads all words of the sentence fluently. 11 Answer a question about a story What is the boy selling? 1 0 points: Incoherent or mumbling response, or wrong answer 1 point: Correct answer: maize 12 Answer a question about a story Why did the boy chase the goats? 3 0 points: Incoherent or mumbling response, or wrong answer 1 point: Correct answer using only one word (for example, eating/maize) 2 points: Correct answer using only a phrase (for example: goats eating/eating maize) 3 points: Correct answer using a clause or a full sentence, with an explanation (For example: Because they are eating maize. /He chased them because they are eating maize.) 13 Answer a question about a story Why do you think the boy was playing with the ball? 3 0 points: Incoherent or mumbling response, or wrong answer 1 point: Correct answer using only one word (for example, training/practicing/fun) 2 points: Correct answer using only a phrase (for example: to train/to practice/for fun) 3 points: Correct answer using a clause or a full sentence, with an explanation (For example: He is training. /He wants to have fun. /He is playing for _____ team.) Learning at Taonga Market in IRLCs and Community School Page 41 14 Say the numbers from 1 to 10 aloud Count from 1 to 10 2 15 Write the numbers from 1 to 10 Write the numbers from 1 to 10. 2 16 Count by ones Count from 48 to 68. [The test administrator can select from a variety of number ranges: 21–41; 37–57; 41–51; 73–93; 35-55; 70-90;29-49; 48-68; 25-55; 79-99; 62-82] 2 17 Write a 2-digit numeral Now write 79. [The test administrator asks the child to write a specific number between 12 and 98 that has two different digits. That is, not 11, 22, 33, etc.] 2 18 Count by twos Count in twos from 2 to 10. 2 19 Count by tens Count in tens from 10 to 100. 2 NR = No Response: The learner has not tried to give an answer. 0 points = Non-mastery: The learner has given an answer that is completely incorrect. 1 point = Partial Mastery: The learner has given an answer that is partially correct. 2 points = Full Mastery: The learner has given an answer that is completely correct. 20 Complete a number sentence [Present on a flashcard: 6 + 7 = 13 ____ + ____ = 13 ____ + ____ = 13 ] 6 + 7 add up to 13. What are two other numbers that add up to 13? Again, what are two different numbers that add up to 13? 2 Maths (28 points) 21 Add two numbers Add these numbers: 4 + 7 [Two of the following addition problems are presented on flash cards: 4+7; 9+8; 7+8; 5+9; 7+5; 7+9] 2 NR = No Response: The learner has not tried to give an answer. 0 points = Non-mastery: The learner has given answers that are incorrect. 1 point = Partial Mastery: The learner has given one correct answer. 2 points = Full Mastery: The learner has given two correct answers. Learning at Taonga Market in IRLCs and Community School Page 42 22 Subtract two numbers Subtract these numbers: 9 – 4 [Two of the following subtraction problems are presented on flash cards: 9- 4; 7-5; 8-3; 6-1; 9-2; 8-7; 7-3] 2 23 Subtract money to calculate change A man has K___ and buys tomatoes for K___. How much will you give back? [Two situations are presented using two of the following pairs of numbers: K500-K200; K700-K300; K900-K600; K400-K200; K600-K100; K200-K100 ] 2 24 Name shapes What is the name of this shape? / What shape is this? [The test administrator presents a circle, a square, a triangle and a rectangle in any order.] 4 One point for each correct response. 25 Draw shapes Now, draw a circle; a square; a triangle; a rectangle [The test administrator conceals the shapes used in the previous question.] 4 One point for each correct response. 26 Name objects in the school [The test administrator asks about two objects from the following: chair, door, window, floor, roof, chalk, pen, ruler, cup spoon.] What is this/that? English 2 Language (16 points) 27 Name items of clothing [The test administrator asks about two pieces of clothing from the following set: dress, hat, shoes, socks, shirt.] What is this? /What am I wearing? /What are you wearing? 2 NR = No Response: The learner has not tried to give an answer. 0 points = Non-mastery: The learner has given answers that are incorrect. 1 point = Partial Mastery: The learner has given one correct answer. 2 points = Full Mastery: The learner has given two correct answers. Learning at Taonga Market in IRLCs and Community School Page 43 28 Give the time of day when you go to sleep and wake up [The test administrator asks two questions.] a. What time of the day do you wake up? b. What time of the day do you go to sleep? 2 29 Talk about common actions (e.g., The teacher is pointing.) [The test administrator presents the grade 1 literacy poster called “At the Centre” and asks two questions.] a. What is the boy doing? b. What is the teacher doing? 2 30 Count the number of learners and the number of boys in a picture [The test administrator presents the picture on the cover of the Grade 1 Mentor’s Guide and asks two questions.] a. How many learners can you see? b. How many boys can you see? 2 31 Give the plural of common words [The test administrator presents two sets of objects (books, hands, pencils or legs), gives the singular form and has the learner give the plural form.] Here is one _book_, and here are two . . 2 32 State what day was yesterday and what day is tomorrow [The test administrator states what today is and asks two questions.] Today is _____. a. What day is tomorrow? b. What day was yesterday? 2 33 Follow a command The test administrator tells the learner to do two of the following things: Open the book. Give me the pencil. Touch your nose. Point to the chalkboard. 2 Learning at Taonga Market in IRLCs and Community School Page 44 Appendix B: Mentor Questionnaire Centre Name___________________District___________________ Province_________________ To be filled by the mentor teaching Grade 1 learners. Tick or write the appropriate answer in the space provided. SECTION A – BACKGROUND CHARACTERISTICS QUESTIONS Instructions/ Coding Q101. Your centre is situated in: 1. Urban area 2. Peri urban area 3. Rural) area __________ Q102. Your sex: 1. Female 2. Male __________ Q103. How old are you? ___________________years? __________ Q104. How long have you been a mentor? ___________________years? __________ Q105. Which grades have you taught as a mentor? Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 __________ Q106. What Grade level (s) are you currently teaching? __________ Q107. Have you been teaching this Grade 1 for the whole of 2005? 1. All year 2. Six months or more 3. Less than six months __________ Q108. How many days this year due to illness or other personal responsibilities? 1. None 2. 1 – 5 days 3. 6 – 10 days 4. More than 10 days __________ Q109. How many days of IRI mentor training have you attended? ___________ days __________ SECTION B – LEARNING AND LEARNING MATERIALS Q201. How long have you had the Grade 1 mentor’s guide? Do not have it 2. In term 1 3. In term 2 4. In term 3 __________ Q202. What reference materials do you have apart from the mentor’s guide? None. A variety of reference material donated to the centre. Some of reference materials / borrowed / improvised. __________ Q203. How do you find the content in the mentor’s guide? Most of the content is too difficult for me. Some of the content is too difficult for me. Some of the content is easy to teach Most of the content is easy to teach __________ Learning at Taonga Market in IRLCs and Community School Page 45 Q204. What reading materials do you have for the students? 1. Taonga Market posters 2. Ministry of Education NBTL materials 3. A variety of other reading materials donated to the centre __________ Q205. How much time do you usually spend with children on their learning tasks before the broadcast? 1. No time at all 2. 5 – 10 minutes 3. 11 – 30 minutes 4. More than 30 minutes __________ Q206. How much time do you usually spend with children on their learning tasks after the broadcast? 1. No time at all 2. 5 – 10 minutes 3. 11 – 30 minutes 4. More than 30 minutes __________ Q207. Is your radio in good working condition? 1. Yes 2. No (Explain at the under Item G) __________ Q208. How many broadcasts have you missed in the period between January to October 2005, due to your radio not working well? 1. None 2. One to ten 3. Eleven to twenty 4. Twenty or more __________ SECTION C – ATTENDANCE AND ACHIEVEMENT Q301. How many children have dropped out of the Grade since January, 2005? 1. None 2. One to three 3. Four to eight 4. Ten or more __________ Q302. What is the most common reason for the dropping out of school? 1. Most of those who dropped out had to go to work 2. Most of those who dropped out had transferred to a regular school. 3. Health reasons 4. Chores in the home __________ Q303. How much do you think the children are learning from the radio program? 1. Very little, far less than those in a government school 2. They learn a lot, but not as much as those that are in a government school 3. They learn a lot, just as much as those that are in a government school 4. They learn a lot, more than those that are in a government school __________ Q304. What do parents say, about their children’s learning? 1. Parents say that children learn very little, far less than those in a government school 2. Parents say that children learn a lot, but not as much as those that are in a government school 3. Parents say that children learn a lot, just as much as those that are in a government school 4. Parents say that children learn a lot, more than those that are in a government school __________ Learning at Taonga Market in IRLCs and Community School Page 46 SECTION D – COMMUNITY PARTICIPATION Q401. Do community members assist in the running of the centre? 1. No, they do not assist at all. 2. Yes, they do assist, but only a few times, after they have been asked 3. Yes, they participate satisfactorily 4. Yes, they participate a lot, and are very useful __________ Q402. Do you think your centre is successful? 1. Not at all 2. Yes, but only mildly successful. It needs a lot of improvement 3. Yes, it is performing adequately. It needs some improvement 4. Yes, it is extremely successful __________ Q403. What will happen to most of the children if the centre closed down? 1. They will transfer to local school 2. They will go to live with a relative in another town so they can attend school. 3. It would be the end of their education, unless government builds a school in the community __________ Q404. Would you recommend the IRI program to any of your relatives? 1. No 2. Yes, if there was no government or community school to attend 3. Yes, even if there is government or community school __________ Q405. Would you recommend your centre to any of your relatives? 1. No 2. Yes, if there was no government or community school to attend 3. Yes, even if there is government or community school __________ SECTION E – SUPPORT FROM THE MINISTRY OF EDUCATION Do you get other support from any of these? (write the name of the organizations) What type of support do you receive? Circle all that applies How helpful has the support been? Q501. Zonal In-service Provider (ZIP) 1. Yes 2. No 1. Visits 2. Training 3. Materials 4. Other 1. Not helpful 2. Helpful 3. Very helpful Q502. District Resource Centre Coordinator (DRCC) 1. Yes 2. No 1. Visits 2. Training 3. Materials 4. Other 1. Not helpful 2. Helpful 3. Very helpful Q503. District Education Board Secretary (DABS) 1. Yes 2. No 1. Visits 2. Training 3. Materials 4. Other 1. Not helpful 2. Helpful 3. Very helpful Q503. Other 1. Yes 2. No 1. Visits 2. Training 3. Materials 4. Other 1. Not helpful 2. Helpful 3. Very helpful Learning at Taonga Market in IRLCs and Community School Page 47 SECTION F – PARTNERSHIPS WITH OTHER ORGANIZATIONS Do you get other support from any of these? (write the name of the organizations) What type of support do you receive? For how long? How helpful has the support been? Church 1. Not helpful 2. Helpful 3. Very helpful Non-governmental Organisations 1. Not helpful 2. Helpful 3. Very helpful Community Based-Organisations 1. Not helpful 2. Helpful 3. Very helpful Individuals/Other 1. Not helpful 2. Helpful 3. Very helpful G. What has been the major success of the Taonga IRI programme in your community? _________________________________________________________________________ _________ _________________________________________________________________________ _________ _________________________________________________________________________ _________ H. What has been the major weakness of the Taonga IRI program in your community? _________________________________________________________________________ _________________________________________________________________________ _________________________________________________________________________ ___________________________ I. What is the greatest threat to the IRI program your community? _________________________________________________________________________ _________ _________________________________________________________________________ _________ _________________________________________________________________________ _________ Learning at Taonga Market in IRLCs and Community School Page 48 Learning at Taonga Market in IRLCs and Community School Page 49 Appendix C: Community Focus Group Questionnaire Name of centre/school: ____________________ District: _________________________ Number of centre/school committee members present: _____ Positions of committee members: _____________________________________________ Numbers of other community members present: _____ Question 1: How did you receive information about the interactive radio instruction program? (Did the community radio station provide any information on Taonga)? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 2: What prompted you to open a learning centre? (why do you send your children to Taonga Market)? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 3: What support is your community giving to the mentor? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 4: What support is your community giving to the centre? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 5: Where will your children go after they finish Grade 5? ___________________________________________________________________________ ___________________________________________________________________________ Learning at Taonga Market in IRLCs and Community School Page 50 ___________________________________________________________________________ Question 5: Have any Taonga Market children go on to write the Grade 7 exams? How well did they do? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 6: What are your major expectations about the Taonga Market program? What has been its major successes? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 7: What is the greatest threat to the IRI program in this community? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Question 8: Do you have any questions or concerns about the Taonga Market Program? ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ ___________________________________________________________________________ Learning at Taonga Market in IRLCs and Community School Page 51 Appendix D: Descriptors for mastery levels by group Group Label Descriptor Group L - Low performing group Learners in this group are able to: x write their names in a recognizable form, even though they may not be able to write legibly, or may make spelling errors; x read some of the targeted vocabulary words, even though they can only read less than twenty words Group M - Middle performing Learners in this group are able to: x write their names legibly, without any spelling errors; x read about 20 vocabulary words; x attempt to read new words using word attack techniques x make sentences with familiar words and copy sentences into their books legibly x construct new sentences using familiar words, even though they have difficulty with this task Group H - High performing group Learners in this group are able to: x write their names legibly, without any spelling errors; x read all vocabulary words and new words using word attack techniques x make sentences with familiar words and copy sentences into their books legibly x construct new sentences using new words x construct picture stories using familiar and new words, even though they have difficulty with this task Learning at Taonga Market in IRLCs and Community School Page 52 Appendix E: Specific Recommendations about Readers for IRI Recommended Readers for IRI Learners IRLCs are usually not permanent, lockable structures. It is pointless giving mentors large amounts of expensive reading materials as they will have nowhere to store them. It would be better to print slim cheap readers and deliver them to centres in amounts that are small enough so that each class set can be carried by the mentor in a box or a carrier bag. It would also be sensible if a box or carrier bag were supplied along with each set of books. The following readers should be provided to each class: Grade 1: A set of red-level and yellow-level PRP readers (one copy of each title) in the appropriate Zambian language. (Distribute one set per class of 40 learners.) Grade 2: A set of red-level and yellow-level PRP readers (one copy of each title) in English. (Distribute one set per class of 40 learners.) Grade 3: Finalize negotiations with Cambridge University Press to produce a low-cost compilation of the PRP yellow-level and green-level readers. The readers should be compiled into a single volume with one story per page. (Distribute ten copies to each centre that has Grade 3 learners.) Grade 4: Finalize production of the PDP anthology of stories. This should be produced as a 32-page reader with one story on each page. Black-and-white illustrations from the PDP artist can be included. (Distribute ten copies to each centre that has Grade 4 learners.) Grade 5: The Macmillan Zambia Basic English, Pupil’s Book for Grade 5 , which is being distributed as a textbook for IRI, should also be used as a reader. (Distribute ten copies to each centre that has Grade 5 learners.) Grade 6: The Read On Orange Activity Book and a Longman Reader called The Market Dentist. (Distribute ten copies to each centre that has Grade 6 learners.) Grade 7: Select a set of readers from the PRP graded readers. (Distribute ten copies to each centre that has Grade 7 learners.) Recommended Training for IRI Mentors The mentor training should provide instructions in organizing reading activities for groups of learners, pairs of learners and individual learners. Mentors also need to be trained to look after reading materials and care for them appropriately. Mentors should be guided in the use of readers through: x Distribution of a guide to go along with the readers x Amendments to face-to-face mentor training x Amendments to the mentors’ guides when future prints are made x Training via the radio during the broadcasts provided at the beginning of each term