Zambia Quality Education Services Through 
Technology (QUESTT) Project
Interactive Radio Instruction Program - Evaluation Report
Grade 1
Prepared by: 
Education Development Center, Inc.
USAID Award No.: 690-A-00-03-00096-00
December 2003
 - ii – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
TABLE OF CONTENTS 
EXECUTIVE SUMMARY...........................................................................................................iv
1.0 BACKGROUND ............................................................................................................... 1
Table 1: Number of IRI Learners in 2003, by grade and sex .................................................. 1
Table 2: Grade 4 mean scores for IRI and GRZ schools - mathematics (2002).....................2 
 1.1 Purpose of the Evaluation…………………………………………………………... 
4
2.0 FINDINGS......................................................................................................................... 5
2.1 Demand for IRI ............................................................................................................. 5
2.1.1 Enrollment and new centers............................................................................... 5
Table 3: 2003 IRI Grade 1 Learners, by sex ............................................................................... 5
Table 4: Number of Learning Centers in 2003, by Province and District............................ 5
Figure 1: Demand for IRI in the Eastern province…………………………………………...8 
2.1.2 New Audiences .................................................................................................... 9
2.2 Attendance .................................................................................................................. 11
2.3 Profile of IRI Learners............................................................................................... 12
2.3.1 Sex and age of learners ...................................................................................... 12
Table 5: Age of IRI learners in years, by province………………………………………….14 
2.3.2 Parent Status and Guardianship……………………………………………...13
Table 6: States of learners parents by province……………………………………………..14 
2.3.3 Prior school attendance ..................................................................................... 14
 2.3.4 What then, are the characteristics of the children who attend IRLCs?.......15 
 2.4 Performance in basic literacy……………………………………………………..16 
Table 7: Test means for literacy, numeracy and the overall test (2003) ………………….16 
Table 7: Mastery by item and (skill area) domain (2003)……………………………….….16 
Table 9: Mean comparisons for female and male learners……………………………..…..17 
2.4.1 Mean scores for Grade 1, 2003 by district and center………………………16
Table 10: Test means for literacy, numeracy and the overall test (2003), by district……18 
Figure 2: Centres, by locality and name, performing above and below 50 percent…….19 
2.4.2 Performance by age……………………………………………………………20
Table 11: Performance by age category………………………………………………….… .21 
2.4.2 Comparison with the 2001 cohort……………………………………………
20
Table 12: Test means for literacy, numeracy for 2001 and 2003………………………… 22 
 - iii – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
2.5 Impression on Effectiveness of IRI: Mentors, parents and other stakeholders. 21
 2.5.1 Mentors………………………………………………………………………… 22 
Table 13: Mentors' years of service and mean days of IRI training……………...………. 23 
2.5.2 Community/parent participation.................................................................... 22
3.0 CONCLUSION AND RECOMMENDATIONS........................................................ 24
3.1 Previous recommendations…………………………………………………. .25 
3.1.1 Demand for IRI ................................................................................................... 24
Figure 3: Recommendations from 2001 Grade 1 Evaluation…………………………… 26 
3.1.2 Attendance .......................................................................................................... 26
3.1.3 IRI Learner Profile.............................................................................................. 26
3.1.4 Student Learning……………………………………………………………… 27 
3.1.5 Mentor Profile..................................................................................................... 26
3.1.6 Monitoring and Evaluation Framework......................................................... 27
3.2 Siaciwena (2002) Recommendations ....................................................................... 27
3.2.1 Lesson development .......................................................................................... 27
3.2.2 Broadcast production......................................................................................... 27
Figure 4: Additional recommendations of August, 2002…………………………………..29 
3.2.3 IRI Management ................................................................................................. 28
3.3 Summary Recommendation....................................................................................... 28
4.0 APPENDICES .............................................................................................................................. 29
Appendix A: Methodology ................................................................................................................ 30
Evaluation Sample: Grade 1 IRI Learners in 2003 ............................................................................... 30
Appendix B: Data collection sheet for students’ profile................................................................. 35
Appendix C: Content Mapping for Grade 1 - English ................................................................... 36
Appendix D: Content Mapping for Grade 1 - Mathematics.......................................................... 37
Appendix E: Test Blueprint for Grade 1 - English .......................................................................... 38
Appendix F: Test Blueprint for Grade 1 - Mathematics ................................................................. 39
Appendix G: Test Plan: Basic Literacy and Numeracy .................................................................. 40
Appendix H : Test of Mathematics and English Language Skills ................................................ 41
Appendix I: Test Administration Training Notes........................................................................... 44
Appendix J: Guidelines for Test Administrators ............................................................................ 50
Appendix K: Mentor Questionnaire ................................................................................................. 51
Appendix L: Focus Group Questions ............................................................................................... 54
 - iv – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
EXECUTIVE SUMMARY 
Prior to the current evaluation, EBS commissioned two IRI evaluations at the Grade 1, 
in 2000 and 2001. These evaluations sought to investigate whether there was demand 
for IRI and regular attendance at the IRI learning centers, and how much learning was 
actually taking place. Conducted in August-September 2003, the current evaluation is 
also for the Grade 1 level. Data from this evaluation were collected from a sample of 55 
of 591 learning centers in five of nine provinces, with the sample size being 992 of 12 641 
learners. Findings provide insights into demand for IRI as manifested in the enrollment 
and daily attendance, learner and mentor characteristics, and student mastery of basic 
numeracy and literacy skills using a curriculum based achievement test. 
With EBS recording a total of 516 organized learning centers in all nine provinces in 
2003, the demand for IRI is still high, both in terms of the number of learners who enroll 
in Grade 1, and those who have continued on to subsequent grades. Central, Luapula, 
Northern and Western provinces have established more than three times the number 
they started with in 2001. The total enrollment has grown from 7 782 learners when the 
programme went to scale in 2001 to 22 773 learners in 2003. Enrollment at Grade 1 has 
experienced a 66.6 percent growth, from 7 782 in 2001 to 12 641 in 2003. 
Since 2001, EBS has produced and broadcast 855 programmes for Grades 1 to Grade 5. 
Most communities have expressed the desire to have programs for the entire primary 
phase, hence the Ministry of Education has decided to investigate the feasibility of 
adding programs for Grade 6 and Grade 7. 
The current cohort of Grade 1 learners is estimated to be 50.2 percent female and 49.7 
percent male. The overall mean age is 9.0 years. There are significant differences in the 
mean age when mean age is disaggregated by province (e.g. 8.2 years for Central 
province, compared to 10.3 years in Southern province). The population composition 
includes the school-going age group (7 – 13 years old), younger learners who should be 
in early childhood programs, and a small percentage from specialized groups such as 
in-mates, adult learners and street kids. The number of orphans recorded is 28.8 
percent. The heterogeneity of the population is confirmation that there is a growing 
interest in IRI. 
While learners continue to demonstrate that they are acquiring important skills of 
literacy and numeracy, performance in 2003 has dropped compared to 2001. This may 
be due to a larger and more heterogeneous sample in 2003. The mean score for 
numeracy is 63.0 percent (compared to 71.5% mean score in 2001), and 48.8 for literacy 
skills (compared to 56.6% mean score in 2001). The area in which learners showed 
weakness was in production of language, and writing of language in particular. As in 
the previous evaluations, learner performance increased as the age of the learner 
increased, while sex differences in performance were virtually nonexistent. There were 
significant differences in the performance of urban versus rural centers in that all urban 
 - v – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
centres had a mean score of over 50 percent, while 10 of 24 rural centres had a mean 
score below 50 percent. However, 3 of the top 6 centres (that had a mean score of 70 or 
above) are situated in urban Lusaka, while the other three are in rural localities. 
While students continue to come regularly to the centers for broadcasts, mentors 
indicated that centers do miss broadcasts, sometimes due to reasons beyond their 
control. It is estimated that 67.3 percent of the centers missed only one program, while 
19.1 percent of the centers missed 10 or more broadcasts. This figure is unacceptably 
high. Approximately 14 percent of the mentors also reported that they had lost 10 or 
more learners who had dropped out for a variety of reasons. 
The challenges faced by the IRI program are similar to those identified in the 2001 
evaluation. These include difficulties in the learning environment such as lack of shelter 
and malfunctioning radios in some centres, as well as inadequate mentor training 
and/or support from the communities. In contrast, centres with more community 
support were noted as positively impacting test performance scores. The majority of 
mentors (54.6 percent) deemed their centers to be a success, while a small number (3.6 
percent) admitted that their centers were not successful. A much higher majority (more 
than 80 percent) assessed that IRI centres to be doing as well as government schools in 
providing education to children, while the focus group meetings with community 
members revealed that they want to be assured of the continuity of IRI, both in terms of 
being there for as long as it is needed, and also extending the programs to Grades 6 and 
7. 
In addition to following-up on recommendations made in a number of IRI evaluation 
reports, this evaluation recommends that EBS should take advantage of the restructured 
departments and positions within the Ministry of Education (MoE) to streamline IRI 
activities such as outreach, monitoring and evaluation. This and other activities will 
ultimately help define the role of IRI in the overall provision of basic education in 
Zambia. 
 - 1 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
1.0 BACKGROUND 
The Interactive Radio Instruction (IRI) program in Zambia is founded on a mandate to 
provide an alternative basic education delivery system for the existing national 
curriculum. The IRI initiative was launched in July 2000 when Grade 1 programs went 
on air for pilot testing in 21 centres in Lusaka and Southern provinces. Since then, the 
program has been implemented in all provinces. 855 lessons have been developed, 
produced and broadcast for Grade 1 through to Grade 5. Thirty-two teachers have been 
trained to write lessons and produce radio programs1, while more than 800 mentors 
have been recruited and trained to manage the learning and the centers. A total of 169 
IRI learning centers were established in 2001 in all the nine provinces. That number had 
grown to 516 centers in 2003, operating in 62 of the 73 districts in Zambia. 
Since its inception, the IRI program has had three cohorts of Grades 1 and 2 learners, 
two cohorts of Grades 3 and 4 learners, and one cohort of Grade 5 learners. Table 1 
below indicates the number of IRI learners, by grade and sex in 2003. A total of 22 763
learners were enrolled in IRI programs as shown in Table 1. Grade 1 learners continued 
onto Grade 2 during the second half of the year. Similarly, Grade 3 learners continued 
onto Grade 4, while Grade 5 learners attended at this grade level throughout the year. 
Table 1: Number of IRI Learners in 2003, by grade and sex
Province Grade 1/2 Grade 3/4 Grade 5
Female Male Female Male Female Male
Total
Central 1068 1139 845 873 354 385 4664
Copperbelt 431 397 289 312 53 88 1570
Eastern 861 974 315 361 123 183 2817
Luapula 722 564 174 159 23 44 1686
Lusaka 986 818 697 539 505 447 3992
Northern 1164 1200 751 986 140 178 4419
N/Western 391 410 114 117 23 49 1104
Southern 495 513 140 180 106 138 1572
Western 250 288 144 180 38 39 939
TOTAL 6368 6303 3469 3707 1365 1551 22763
 
Since it admitted its first non-pilot learners in 2001, IRI has expanded to a total of 6 
curriculum subjects. English and Mathematics are offered for Grades 1 and 2. Two 
additional subjects, Science and Social Studies, are offered at Grade 3 and 4, while 
Religion and Home Economics are included in the Grade 5 curriculum. A non￾examinable Life-skills and HIV/AIDS education segment is incorporated into each half 
hour lesson. The IRI learning program for the foundation phase spans 2 years, with 
Grades 1 and 3 being aired for 100 week days during the first half of the year, while 
 
1 Twelve of the scriptwriters have since gone back to take teaching responsibilities in GRZ schools.
 - 2 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Grade 2 and 4 programs are aired for 100 week days during the second half. The Grade 
5 learning program is aired for 160 days, spanning the whole year. Programs for each 
grade are aired twice a day during working days. 
Several evaluation activities have been undertaken, one during the pilot and others 
during the IRI implementation period. These include formative evaluation of programs, 
where writers take turns to test their programs at a learning center, collecting baseline 
data on how much knowledge the children had at the onset of the IRI program, an 
appraisal of their attendance and how well they had learnt after 50 programs. 
Two evaluations (mid-pilot and summative) were conducted at Grade 1 in 2000 on the 
pilot sample, (Chondoka, 2000; EDC, 2000). A Grade 1 summative evaluation was again 
conducted in 2001 (EDC, 2001). Other evaluations of the IRI program were conducted in 
2002 at the Grade 2 and Grade 4 levels by the Examination Council of Zambia (ECZ). 
Also in 2002, an appraisal of the IRI system was conducted by an external evaluator 
from the University of Zambia (Siaciwena, 2002). 
The ECZ evaluation compared IRI learners with a corresponding sample of learners in 
government schools. The Grade 4 evaluation was particularly important because it 
became part of the country-wide competency testing that is conducted by government 
at the end of the foundation phase. In both Grades 2 and 4, IRI learners were found to 
be performing just as well as GRZ learners, especially in numeracy. For instance, the 
results of the Grade 4 evaluation revealed small non-significant differences in mean 
scores of IRI learners as compared to those in GRZ schools in Mathematics. Table 2 
presents the Mathematics results as reported by ECZ. 
Table 2. Grade 4 mean scores for IRI and GRZ schools – Mathematics (2002)
 
DOMAIN IRI GRZ
Notation 83.0 73.0
Numeracy 50.0 43.0
Addition and subtraction 67.4 67.2
Multiplication 52.0 44.4
Division 57.0 46.0
Number patterns 55.0 46.0
Decimal measures 37.0 34.0
Shapes and measurements 39.0 18.0
Fractions 59.0 38.0
TOTAL MEAN SCORE 57.0 46.6
Summative evaluation at the Grade 1 level in 2000 was conducted by EBS/EDC staff, 
under the supervision of an external consultant. The evaluation confirmed a high 
demand for IRI and growing interest in establishing more centres. Also, learners were 
attending daily broadcasts as expected. The evaluation also revealed that IRI learners 
were meeting learning targets as stipulated in the MOE primary school curriculum. No 
significant differences were reported between boys and girls, neither were there 
differences between learners from centers in urban and rural localities. 
 - 3 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
The August 2001 Grade 1 evaluation revealed, also, that children were learning and that 
the IRI learning program was providing a viable alternative to out-of-school learners . 
However, the evaluation also revealed that the programme was facing a number of 
challenges. It was recommended that the MOE should articulate a regulatory IRI 
strategy that provides the minimum requirements that should be met in order to ensure 
that the program reaches those who need it most; that mentors be encouraged to keep 
attendance information and a journal documenting critical incidents; that EBS consider 
a standard setting exercise in order to provide consistent descriptions of learning 
achievement and for year to year comparisons, and that a database of centers, learners 
and mentor profiles be established in order to improve efficiency and allow greater 
manipulation of data, as well as easier linkage to the MOE’s Planning Unit (Letshabo, 
2001). 
The August 2001 evaluation made additional recommendations, meant to make IRI 
more sustainable. The most critical was to find measures to prevent mentor attrition in 
view of the critical service that mentors provide. It was recommended that MOE/EBS 
should delineate the profile of an ideal mentor for the IRI program, her/his training 
needs, as well as explore options for mentor incentives such as preferential admission to 
teacher training. Most of the recommendations were not yet implemented at the end of 
2003, mainly because of resource constraints, both human and material. 
In August 2002, USAID commissioned an evaluation of the systemic aspects of the IRI 
program, where evaluators set out to investigate whether the major components of the 
system were functioning as expected, the strengths and limitations of the IRI delivery 
system and how delivery and other aspects could be improved. Findings of the 
evaluation were presented in a report entitled “Rapid Appraisal of the Ministry of 
Education’s Interactive Radio Instruction Basic Education Delivery System” (Siaciwena, 
2002). The findings of the report were presented under two general headings, that is, 
the instructional system and the support system. The findings on the instructional 
system confirmed that there was a high demand for IRI in many communities, and that 
the demand was growing. It also confirmed the centrality of the mentor in the whole IRI 
process, and the need to devise extraordinary measures that would ensure that mentors 
remain motivated to perform their roles in IRI. 
However, this evaluation also addressed, for the first time, issues pertaining to the 
production of lessons and the quality of the broadcasts, working relations between EBS 
and other MOE departments, strategic partnerships with others outside the MOE 
system, and MOE driven professional development for mentors and producers. It 
presented a list of twenty-seven recommendations, some of which are already being 
implemented. Figure 2 in Section 3 provides a summary of actions so far undertaken in 
response to the recommendations made in August 2001 (Kariuki & Letshabo, 2001), and 
in August 2002, (Siaciwena, et. al., 2002). 
 - 4 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
EBS has taken other definite steps to strengthen the IRI programme. In particular, IRI 
Provincial Outreach Coordinators (POC) were recruited in June 2003 to support the 
MoE offices at the provincial and district levels. The terms of reference for the POCs 
include the following: 
1) Collection of accurate data on enrolment, attendance and retention; 
2) Act as a link between the IRI centres at district and provincial MOE levels, and 
EBS for purposes of information gathering and sharing; 
3) Support the MOE staff in provincial, district and zonal levels to facilitate the 
provision of conducive learning environments; 
4) Assist provincial IRI Focal Point Persons in co-ordinating IRI activities (e.g. 
mentor training, monitoring centres, supervising and coaching mentors) 
5) Supporting the Outreach and Monitoring advisors in soliciting and exploring 
support from NGOs, churches, individuals and business houses and 
6) Working with MOE in expanding and strengthening the support to IRI centres 
and community schools through professional and logistical support. 
These and other measures were taken to systematize IRI monitoring processes within 
the MoE, and to create engagement and ownership of the programme at the community 
level. 
1.1. Purpose of the Evaluation 
The purpose of this evaluation was to assess whether Grade 1 learners that receive IRI 
programs continue to meet learning targets in numeracy and literacy as stipulated in 
the curriculum objectives, and whether the programs continue to benefit the population 
for which they were originally intended, that is, out-of-school children experiencing 
socio-economic hardships. Just as the previous evaluations, this evaluation assessed if 
there still is a demand for IRI programs at Grade 1, and whether the learners attend 
daily broadcasts as expected. Other evaluation questions addressed the effectiveness of 
IRI as a means of imparting basic literacy and numeracy skills. 
Data on enrollment, background characteristics of the learners, and attendance was 
collected at the selected 55 of 516 centers. The evaluation strategy stipulated the use of 
population parameters to describe the profile of learners in the centers, in particular 
their sex, age, who their guardians are, and whether they have any living parents 
(orphan status), and sample statistics for the achievement scores. The sample size was 
992, 11.5 percent of the population. The sample was drawn from 5 of the 9 provinces. 
A curriculum-based achievement test was used to assess performance in literacy and 
numeracy skills. Since programs are delivered through the medium of radio, some 
adjustments were made to the existing curriculum to make it suitable for radio.2 These 
differences were accommodated when developing the achievement test. A detailed 
discussion of the methodology is presented in Appendix A. 
 
2
 Adjustments include restating some objectives and sequencing of the curriculum. 
 - 5 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
2.0 FINDINGS
2.1 Demand for IRI 
2.1.1 Enrollment and new centers 
The demand for IRI has continued to grow. IRI was rolled out to all the nine provinces, 
initially with the instruction to establish only a few centres in each province. Out of a 
total of 22 763 learners who were enrolled in IRI programs in 2003, the population of 
Grade 1 learners was 12 641 in 2003, compared to 7782 in 2001 (61.6 percent growth) as 
shown in Table 3. 
Table 3: 2003 Grade 1 IRI Learners, by sex
Province 2001 2003
Female Male Female Male
Central 465 553 1068 1139
Copperbelt 445 485 431 397
Eastern 426 400 861 974
Luapula 117 134 722 564
Lusaka 1257 1252 986 818
Northern 306 365 1164 1200
N/Western 186 178 391 410
Southern 486 503 495 513
Western 100 124 250 288
TOTAL 3788 3994 6368 6303
The number of learning centers has increased from 251 (169 verified) in 2001, to 516 in 
2003, with the Central, Luapula, Northern and Western provinces having established 
more than three times the number that they had in 2001 as shown in Table 5. At least 
eighty-eight of the Interactive Radio Learning Centres (IRLCs) are community schools3. 
Table 4: Number of Learning Centers in 2003, by Province and District
Province No. of Districts 
with IRI centres
No. of centers in 
operation
2001 2003 2001 2003
Central 05 06 32 96
Copperbelt 06 10 30 63
Eastern 07 08 22 61
Luapula 01 07 05 54
Lusaka 04 04 61 46
Northern 04 12 22 82
Northwestern 03 06 37 53
Southern 04 04 33 26
Western 01 05 09 35
TOTAL 35 62 251 516
 
3
 See Section 3.1.2 on new audiences. 
 - 6 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Communities have continued to open new centers and operate under very difficult 
circumstances. For example, in Lusitu, a drought prone area in the Southern province, a 
community school started using IRI since 2003. Due to the inability to grow crops, 
people eat wild grass, wild vegetables and wild roots to survive. Still, the community 
has found ways to support the mentor, e.g. by providing him with some of the few 
drought resistant crops that are grown in the area, such as finger millet and sorghum. 
In Mongu district in the Western province, typically thought of as the ‘sandiest district 
in Zambia’, people mainly grow cassava and rice on a small scale. Though they get fish 
from the Zambezi river, this is seasonal. However, the community, through the Catholic 
church, supports the Mutwiwamba IRI Centre mentor by providing a 25kg bag of maize 
and cooking oil every other week. Community support is evident and particularly 
critical where mentors have died (e.g. in Chilanga at Linda center and Chongwe at 
Ndashika center), communities have managed to quickly mobilize and re-organize their 
centre operations to keep them going. 
In Southern province, Chikuni Community Radio, with the support of the Catholic 
church, continues to be not only a strong IRI pioneer in the province, but also the 
strongest among EBS partners. Chikuni has added 4 new centres, from 19 centers in 
2001 to 23 centers in 2003, with their enrolment growing from 640 in 2001 to 1 200 
learners in 2003. The IRI programme in Chikuni covers Grades 1 through 5. 
Apart from the growth in enrolment and opening new centers, other evidence of 
demand and growth of IRI, indeed its role as the alternative avenue through which 
access to affordable basic education is provided in Zambia, is its expansion to 6 
curriculum subjects and to five grade levels. Also, the resources that the MoE has 
committed to it as measured by the hours and cost of airtime/broadcasting time, 
printing of mentor’s guides for each grade, confirmation of teachers as producers at 
EBS, accommodating IRI outreach coordinators in provincial offices, budgeting for 
monitoring of IRI activities at district level, etc. 
The IRI programme currently takes up a good part of the day on Radio 2, of one of the 
most popular channels of Zambia National Broadcasting Corporation (ZNBC). The IRI 
broadcasts start at 0930hrs to 1630hrs during weekdays, breaking only for news 
segments. It is highly likely that IRI will continue to grow at least in the near future as 
can be seen below in Figure 1 on the next page, in an exchange between MOE officials, a 
chief and a local councilor in the Easter Province. 
 - 7 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Figure 1: Demand for IRI in the Eastern Province
 - 8 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
However, IRI has not always fared well. A total of 22 centres, 15 in Lusaka and 7 in the 
Southern province (Kalomo, Livingstone, Monze and Mazabuka districts) have had to 
close down for numerous reasons, including: 
• Mentor attrition, where mentors leave in pursuit of other opportunities such as 
employment (e.g. Mapepe, Lusaka) and personal development (the mentor in 
Misisi (Lusaka) opted to attend teacher training college); 
• Free education policy (e.g. Nchute, Chongwe - Lusaka);
• Misunderstanding over IRI vs. Community school learning. For example, in 
George (Lusaka), the centre was turned into a community school with a view to 
benefit from MOE school community grant. It seems that people did not 
understand that they could ‘graduate’ into becoming community schools as well 
as continue using the radio lessons;
• Lack of community support, commitment and understanding (for example, in 
Kalomo, Southern province). In several places, it was reported that children 
attended broadcasts out of sheer enjoyment rather than parental encouragement. 
In fact, while parents did not oppose the children’s learning, they seemed totally 
disinterested in supporting the mentors or the centres in any way. This resulted 
in mentors being discouraged, especially when they compared their situation to 
the kind of support that Chikuni mentors got. In Livingstone, communities also 
failed to replace one mentor who showed up drunk every day and the centre 
eventually closed; 
• Lack of support from some MOE provincial offices (hence, information regarding 
mentor training, need for radios, sensitization, etc. does not reach EBS); 
Enrollment and demand for IRI should be understood against a backdrop of a number 
of factors. First, Zambia has a lot of children of school going age who are not in school 
(an estimated 700,000 in 20004) Secondly, the government started implementing its Free 
Primary Education (FPE) policy in 2002. An increase from 7782 Grade 1 IRI learners in 
2001 to 12 641 in 2003 still indicates that IRI is filling a definite need, and is also 
emerging as an alternative way to providing access to schooling. However, there might 
be other lessons to learn on progression from one grade to the next. 
The 2003 Grade 5 learners, estimated to be 2916 in number, is the same cohort that 
started in Grade 1 in 2001 with an enrollment of 7782. On the surface, it may seem that 
IRI has high attrition and/or wastage rates (about 62 percent). Without a proper tracing 
mechanism, it is not possible to conclude that the children who have not continued with 
the IRI programme are “wastage”. 
First, government started implementing FPE in 2002 when the cohort was entering 
Grade 3, which means that a significant number of learners may have opted for “free” 
education in the school system. Second, a sizeable number of IRLCs are located in urban 
 
4
 Basic Education Sub-Sector Investment Programme (BESSIP): 2000 Programme Performance Indicators," 
Republic of Zambia, Ministry of Education, November 30, 2001. 
 - 9 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
and/or peri-ubran areas where there are schools, which means that learners may have 
opted to continue their education in the school system after taking advantage of the IRI 
accelerated approach. Thirdly, the 2001 Grade 1 cohort had a significant population of 
under-aged learners (6.4 percent). These may have joined the school system in 2001 just 
by virtue of the fact that schools were ready to accept them, and that their education 
was going to be free and/or subsidized. 
Indeed, in Samfya and Mansa districts in Luapula province, as well as in Chadiza and 
other parts of Eastern province, monitoring visits have observed that some communities 
do not take IRI seriously, or perhaps do not understand the intention of IRI and are 
using it more as a nursery school. One would expect that such children would go into 
the school system when they reach school-going age. Fourth, some of the over-aged 
learners, estimated at 20.8 percent of the 2001 cohort, may have decided that they had 
attained functional literacy, which might have been their goal for attending IRI in the 
first place. 
On the other hand, it is quite possible that the demand for IRI at the higher level is not 
as high as at the lower levels, in which case it may not be cost effective to deploy 
resources beyond the lower primary cycle (ending at Grade 4). Whatever the case, it 
will be necessary for EBS to monitor the 2002 and 2003 cohorts closely as they approach 
Grade 5 and, if possible, conduct a tracer study to provide the MoE with the evidence 
that they need to make data-driven decisions on how best to use IRI. 
2.1.2 New Audiences 
It has been demonstrated before that in addition to the original population of out-of￾school and vulnerable children, IRI has attracted adult and under-aged learners (EBS, 
2001). Exposing under-aged children that are not yet of school going age to organized 
learning has been passed as a positive development for IRI. However, this population, 
estimated at 500 of the 7782 learners in 2001 (6.4 percent) compared to 1298 of 12 641 
(10.3 percent) in 2003, is continuing to grow. Reports from centers continue to indicate 
that having learners who belong to early childhood programs in IRI programs does 
interfere with the smooth running of the learning process. As in 2001, it has been found 
to be necessary either to retrict enrollment to a certain age attendance, or to articulate a 
strategy for handling younger learners. 
Adult learners have persisted to the Grade 5 level in Chikuni Parish in Monze district. 
Other new audiences that have started to use IRI are community schools, while plans 
are underway to make IRI available to in-school learners, at the request of the 
Permanent Secretary. The intention is to air half hour segments in the morning as a 
means of supplementing classroom learning activities, while maintaining the focus on 
out-of-school audiences. 
Evidence also shows that IRI is meeting an important social and educational need, with 
its attraction of new and non-conventional audiences. EBS has received requests to 
 - 10 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
D , 
open new IRI centers to complement the education services in the prisons systems, with 
preparations to open a prison center in the Copperbelt already being underway. Even 
though this plan has not yet materialized, there have been talks to open a center for 
street kids in Kitwe. 
As part of its outreach effort, IRI has gone into partnership with a number of 
communities and organizations in 2003. One of the important is the Zambia 
Community Schools Secretariat (ZCSS), an umbrella organization for all community 
schools, with a total of 1,340 schools affiliated to it in 2002. In 2003, new partnerships 
were formed with the community school movement through a variety of players: 
• With support from UNICEF Zambia, 200 community school teachers were 
trained by MOE officials in IRI methodology. UNICEF subsequently gave 200 
radios to ZCSS to be distributed to the schools where teachers had been trained, 
while EBS provided 200 mentors guides. As a follow-up action, EBS organized 
IRI sensitization training for ZCSS management in March 2003. 
• ZOCS (Zambia Open Community Schools) – an affiliate of ZCSS, operates 17 
schools mainly in Lusaka (1 in Kabwe, Central province) with support from a 
variety of donors. ZOCS has always been interested in using IRI and distributed 
radios to all of their schools as early as 2001. However, monitoring has been 
inconsistent over the years. In 2003, there was renewed effort to encourage 
ZOCS teachers to introduce IRI in their schools with a request for EBS to train 
teachers and supervisors. This came about as a result of a few teachers who had 
been using IRI sharing positive reports about their experiences. Other teachers 
felt it was very important for supervisors to understand IRI so that they can 
support them more. Hence, in February 2003, EBS held a two day training, one 
for teachers and another for supervisors. All ZOCS schools have been sensitized 
and should be using IRI. 
• In 2003, the Reformed Open Community Schools (ROCS) expressed an interest in 
having IRI in some of their affiliates, and requested training for teachers. A total 
of 65 schools benefited from this training, 11 in the Copperbelt (5 in Kitwe and 6 
in Ndola), 11 in the Eastern province (Chipata district), 43 in Lusaka (21 in 
Chongwe and 22 in Lusaka urban. 
It is anticipated that many parents, some of which have expressed the feeling that the 
school system is shortchanging them by not allowing their children to receive IRI 
programmes, will welcome this move. On the whole, while the situation at Grade 5 
level needs to be studied closely, there is overwhelming evidence that the demand for 
IRI continues to grow, especially at lower primary (Grade 1 through Grade 4). A 
number of those who extol the virtues of IRI have boasted that it has so far been a 
reliable learning avenue for their children as can be seen in the quotation below. 
“This programme Madam PS, has got an advantage, because we mentors, 
we don’t go for strike” 
IRI Mentor, Webster Haamonga, Spokesperson for 60 Chikuni Mentors, 28 October, 2003
 - 11 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
However, it may be necessary to reexamine whether the original audience of out-of￾school and vulnerable children are not being shortchanged by other demands on the 
program. For instance, data has shown that orphaned children constitute only 28.8 
percent of learners who participate in IRI compared to 32.2 in the GRZ school 
population, which makes orphans somewhat underrepresented in this program. Is that 
acceptable, given that there is evidence that many more children are orphaned? If not, 
would it be desirable to extend EBS outreach efforts to seek out orphaned and other 
vulnerable children, and to ascertain that they are extended first priority towards this 
learning opportunity? These and other such questions need further investigation and 
resolution. 
The other concern that needs to be addressed is that of spontaneous expansion of the 
program, as has been demonstrated by opening of more centers than anticipated (516 in 
2003, as opposed to the projection of 350 by 2005 in the EBS Five-Year Strategic Plan), 
and the attraction of new audiences such as street kids and prisoners. While we 
acknowledge the fact that IRI has become an important strategy/alternative in 
extending access to basic education in Zambia, there is a potential to overwhelm the IRI 
support system to a point where in might cease to be effective (e.g. in provision of 
training and materials). A measure that can be used to safeguard the effectiveness of 
IRI could be to classify IRI centers into the categories of support that they will receive 
from EBS. 
2.2 Attendance 
Apart from managing the learning environment, one of the important tasks that 
mentors are expected to carry out is to take daily attendance. This was more 
manageable during the piloting phase when one of the pilot activities was to take 
attendance and submit registers to EBS on a monthly basis. Since then capturing 
attendance data has continued to be a problem. For most mentors, especially at centers 
who enroll 100 learners or more, the cost for taking daily attendance is loss of time for 
meaningful learning activities. As a result, most mentors neglected taking attendance 
with the result that attendance data was rudimentary. 
It is estimated, from available data, that on average learners attend 75 of the 100 days 
and receive IRI daily programmes at IRLCs. However, while children may show up for 
the daily programmes, learning is sometimes interrupted due to factors that are beyond 
the control of the mentor and/or the children, such as mentor absence, or radios that 
are not working. For instance, additional data from mentor questionnaires reveal that 
only 32. 7 percent of the mentors reported that their centres had received all 100 days of 
broadcasting. The remaining 67.3 percent reported that they missed at least one day of 
broadcasting due to a malfunctioning radio in the period between January and July 
2003, with 19.1 percent of those missing 10 days or more. In another example Mbulwe 
 - 12 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
centre in Serenje district lost more than 10 days of IRI programs when their center 
shelter collapsed while centers in Mongu lost a full week of broadcasting due to poor 
reception. Mentors also reported high learner attrition rates. While only 15.7 percent of 
mentors had not had any learner drop out of school in the period between January and 
July 2003, 14.8 percent reported a dropout rate of 10 or more learners. 
From the data that is available, there seems to be a need to monitor attendance more 
closely than has been done in the past, and to ensure that those who attend the IRLCs 
do receive the lessons. A recommendation that was made in the past was to develop a 
mentor-friendly strategy for tracking daily attendance, and avail official MOE registers 
that can be reclaimed at the end of the grade. An additional strategy would be to train 
mentors to only make a marking against the children that are absent in any given day, 
and to complete marking of the register at the end of each week. 
The mentor could also involve the children in boosting attendance rates by making 
them part of the “register marking” exercise by having them report who is absent, and 
tasking them to show up on the next day with a friend that was absent. This could 
actually interest them in attending the daily broadcasts. Needless to say, it is important 
for learners to attend daily broadcasts as their performance is usually judged against all 
the lessons. 
2.3 Profile of IRI Learners 
Learner profiles are necessary to ensure the best possible intervention for IRI recipients. 
Data collected on learner characteristics included sex, age, and whether they are 
orphans. Data was also collected on whether or not learners had formal schooling prior 
to enrolling at the center. Once more, sample data was used, with the unit of analysis 
being the individual learner. Generalizations on the profile of the learners are being 
made from the sample data to the population. 
2.3.1 Sex and age of learners 
The total number of learners enrolled in IRLCs indicates that there are 49.7 and 50.2 
percent male and female learners, respectively. The mean age of the learners was 9.0 
years, whereas the modal age was 8 years old (21.0 percent of the learners). The mean 
age for girls is 8.9 years compared to 9.2 years for the boys. 
There were differences in the mean age by province as shown in Table 5. The mean age 
of learners from Southern province is approximately 2 years more that the mean age in 
Central and Luapula provinces. The current sample of learners did not include any 
adult IRLCs, hence the age variation was relatively narrow, compared to the 2001 
evaluation. However, the number that is beyond the official age for lower primary has 
remained at 21.1 percent of the learners. 
Table 5: Age of IRI learners in years, by province
 - 13 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Northern Central Luapula Lusaka Southern Overall
2003
Overall
2001
Mean age 9.2 8.2 8.6 9.0 10.3 9.0 9.7
Modal age 10.0 8.0 8.0 10.0 8.0 8.0 7.0
Minimum 5.0 5.0 6.0 6.0 6.0 5.0 5.0
Maximum 16.0 14.0 13.0 13.0 16.0 16.0 51.0
Having younger learners seems to suggest that IRI centres are being used in the place of 
regular schools. It will be interesting to note, also, that learners from Central and 
Luapula provinces performed significantly lower than learners from the other three 
provinces. 
2.3.2 Parent Status and Guardianship 
Of the number reporting (579 of the 828 learners) 71.1 percent of the children had 
parents, while 28.8 percent were orphans as indicated in Table 8. Double orphans 
constituted 7.5 percent, while 21.4 percent learners had only one living parent. 
Compared to the 2001 evaluation where 33.2 percent of the children were reported to be 
orphans, this finding goes contrary to the trend as recorded by government and NGOs. 
This could be a confirmation of the suspicion voiced in the previous evaluation, that the 
number of orphans is being under-reported. 
Table 6: Status of learners’ parents by province
Living Parents Northern Central Luapula Lusaka Southern All 
Learners
Both alive 74.5 80.6 44.7 85.7 59.2 71.1
Only mother alive 18.6 7.4 31.6 6.6 22.5 17.6
Only father alive 2.2 6.0 9.2 1.1 5.6 3.8
No living parent 4.7 6.0 14.5 6.6 12.7 7.5
Orphans 25.5 19.4 55.3 14.3 40.8 28.8
One reason that was proffered for under-reporting of orphans is the inconsistency 
between the official definition and cultural definition of orphans, where children are not 
considered orphans if they are adopted into the family of a close relative, albeit 
unofficially. Mentors have confirmed, from their personal knowledge of the children in 
their communities, that significantly more children in their centers were orphans, even 
though the adult guardians did not want to disclose that information about their 
protégées. 
Table 6 indicates, also, that there are significantly more orphans in Southern and 
Luapula provinces and much less in Lusaka province. By virtue of the fact that urban 
centers in Zambia have a higher prevalence of HIV/AIDS and HIV related deaths, it 
was expected that there would be comparatively more orphans in the urban areas, 
particularly in the case of HIV/AIDS orphans. However, it is possible that orphaned 
children are increasingly being sent to the countryside to live with their grandparents, 
especially in cases where both parents are deceased. Another explanation for the low 
 - 14 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
prevalence of orphaned children in urban centers such as Lusaka is that most centers 
are situated in high poverty areas, such as Bauleni, Chainda, Kamanga, and Kanyama 
compounds. While children in those areas may be having both parents, their 
vulnerability stems from the overwhelmingly high poverty levels that their families live 
in. They are therefore, a legitimate population for IRI programs. More attention should 
be paid to the issue of the orphan status of children and other types of vulnerability, 
especially with the ever looming need for regulating attendance.
2.3.3 Prior school attendance 
Only 5.7 percent of the learners had some schooling experience prior to enrolling in the 
learning centers, compared to more than 16 percent in the previous cohorts. A possible 
explanation for this may be that the IRI program is increasingly meeting the needs of 
children for which it was originally started, those who were totally left out of the school 
system. 
2.3.4 What then, are the characteristics of the children who attend IRLCs? 
Whereas IRI was intended for disadvantaged children, it is not clear whether IRI 
learners are significantly different from the typical school learners. The mean age and 
the overall age distribution indicates that IRI learners are older than learners in the 
primary school system. This does suggest that IRI is catering for learners who were “left 
out”, referred to as the ‘over-age’ in the official school system. But apart from the age, 
learners in IRLCs and government schools are comparable in all the other 
characteristics under investigation. For instance, the orphan status of the IRI learners is 
comparable to that of the school population, which seems to suggest that the learners 
are not necessarily “disadvantaged”. 
There is a need, therefore, to revisit the original intent of IRI, that of reaching out to 
“out-of-school” children and to assess the extent to which it is being met within current 
arrangements. Empirical evidence seems to suggest that IRI is an alternative avenue 
that is also addressing the issue of access to primary education for a variety of 
audiences. Is this an acceptable development? If so, does the MoE want to drive it? 
These and other similar issues need, also, to be addressed by policy positions.
 
 - 15 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
2.4 Performance in basic Literacy and Numeracy 
Grade 1 learners attained a mean score of 24.2 out of 44 possible points on the whole 
test (a composite score of the numeracy and language score), a mean of 55.0 percent. 
The lowest score was 0, and the highest, 44. The mean score for the numeracy 
component was 12.6 out of 20 (63.0 percent), while the mean for the language 
component was 11.7 out of 24 possible points (48.8 percent). 
Table 7: Test Means for Literacy, Numeracy and the Overall Test (2003)
Content N Minimum Maximum Mean Std. Dev.
Numeracy 826 .00 20.00 12.6 4.7
Literacy 821 .00 24.00 11.7 5.7
Total Test 828 .00 44.00 24.2 9.1
As with previous evaluations, learners performed better on the numeracy component 
than on literacy. This is to be expected, particularly that they get the opportunity to 
practice numeracy skills in their daily lives. Table 8 below presents information on the 
percentage of learners who mastered each item, with items grouped under the different 
domain. It also provides, for comparison, the subtest mean for each domain (skill area). 
Table 8: Mastery by item and (skill area) domain (2003)
Skill Area Items Non￾Masters
Partial 
Masters
Masters Domain Mean
Production of language Item 16 20.5 7.6 71.9
Item 17 28.6 12.5 58.8
Item 18 43.2 25.5 31.3 4.7 of 12
Item 20 31.4 39.8 28.8 (39.2 percent)
Item 21 67.1 22.4 10.5
Item 22 68.5 12.9 18.7
Item 13 38.8 11.0 50.2
Item 14 14.9 22.6 62.5 4.3 of 8
Item 15 24.4 15.1 60.5 (53.8 percent)
Comprehension of language
Item 19 36.2 25.5 38.2
Recall of names Item 11 10.1 23.7 66.2 2.9 of 4
Item 12 16.5 23.3 60.1 (72.5 percent)
Counting and writing numbers Item 01 1.6 17.6 80.8
Item 02 19.9 14.9 65.2 5.6 of 8
Item 03 32.2 32.2 35.6 (70.0 percent)
Item 04 13.3 16.2 70.5
Shapes Item 09 34.3 26.6 38.9 2.1 of 4
Item 10 27.5 26.5 46.0 (52.5 percent)
Item 05 48.1 4.7 47.3
Item 06 24.3 3.5 72.3 4.9 of 8
Item 07 25.9 2.1 72.0 (61.3 percent)
Number operation
Item 08 38.0 2.0 60.0
In the literacy component, learners had the highest competence in recalling names of 
colors and items of clothing (at 72.5 percent). The least performance was registered in 
the skill area of production of language, where the mean for the domain was 39.2 
 - 16 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
percent. In particular, most learners were not able to read any of the simple two syllable 
words (Item 21, with 67.1 non-masters), nor spell them (Item 22, with 68.5 non-masters). 
Even though items on reading and spelling were included in the test, reading is not a 
terminal skill at Grade 1 as has been mentioned in previous evaluation reports. 
Assessment of this skill is important in as far as we can document the progress that 
learners are making towards being able to read. Also, learners have had very little 
opportunity to practice reading in the current version of the IRI program in that they 
did not have reading material at their disposal. 
As in the previous Grade 1 evaluation, all skill areas of the numeracy component were 
performed at a mastery level of over 50 percent as indicated by the domain means. 
However, this cohort was much weaker on the item on counting in twos (35.6 percent of 
the learners being able to execute the task perfectly compared to 53.8 percent in the 2001 
cohort), and on shapes where more than a quarter of the learners could neither recall 
the name of any of the four basic shapes, nor draw it. The difficulty in this item was 
partly because of the manner in which the item was scored. Table 9 presents a 
comparison of performance by sex. 
Table 9: Mean comparisons for female and male learners
 
Skill Area Female Male
Production of language 4.65 4.79
Comprehension of language 4.35 4.32
Recall of names 2.97 2.82
Counting and writing numbers 5.61 5.70
Shapes 2.09 2.22
Number operations 4.76 4.98
Overall Test Mean 24.05 24.29
Performance differences by sex were negligible in all skill areas. This is a known 
strength of IRI programs in many countries where the EDC version of IRI is 
implemented (source). However, it is important because it has been borne out in all IRI 
evaluations in Zambia. 
2.4.1 Mean scores for Grade 1, 2003 by district and center 
Mean scores were calculated by district and locality. Kafue disctrict, comprising only 
one center (Mimosa/Linda) has the highest mean score of 36.5 of 44 (70.5 percent), 
followed by Lusaka district comprising of 5 centres as reflected on Table 10. All of the 5 
centres performed above the mean even in the previous testing. Lusaka and Kafue 
districts have an advantage of being close to EBS, hence they may have an advantage of 
 - 17 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Table 10: Test Means for Literacy, Numeracy and the Overall Test (2003), by district
 
District Test N Minimum Maximum Mean
Kasama Numeracy 163 1.0 20.0 12.6
Literacy 164 0.0 24.0 12.1
Overall 164 3.0 44.0 24.7
Luwingu Numeracy 44 4.0 17.0 12.3
Literacy 44 0.0 20.0 11.5
Overall 44 4.0 36.0 23.8
Mungwi Numeracy 92 3.0 20.0 12.6
Literacy 92 0.0 23.0 11.8
Overall 92 10.0 42.0 24.4
Mbala Numeracy 29 3.0 15.0 9.3
Literacy 29 7.0 17.0 13.6
Overall 29 10.0 31.0 22.9
Mpulungu Numeracy 32 7.0 20.0 15.1
Literacy 32 1.0 23.0 10.3
Overall 32 8.0 43.0 25.5
Mkushi Numeracy 57 5.0 20.0 15.2
Literacy 57 0.0 24.0 12.1
Overall 57 10.0 43.0 27.2
Kabwe Numeracy 32 0.0 20.0 10.0
Literacy 32 0.0 20.0 7.3
Overall 32 1.0 38.0 17.3
Serenje Numeracy 56 0.0 20.0 9.8
Literacy 51 0.0 17.0 7.0
Overall 57 1.0 35.0 15.9
Chisekesi Numeracy 74 1.0 20.0 13.1
Literacy 73 5.0 24.0 14.7
Overall 74 8.0 42.0 27.6
Siavonga Numeracy 16 6.0 19.0 12.8
Literacy 16 4.0 16.0 9.6
Overall 16 10.0 32.0 22.4
Lusaka Numeracy 95 6.0 20.0 14.8
Literacy 95 3.0 23.0 14.9
Overall 95 13.0 43.0 29.8
Kafue Numeracy 46 3.0 20.0 16.1
Literacy 46 1.0 24.0 20.4
Overall 46 9.0 40.0 36.5
Chongwe Numeracy 46 3.0 20.0 13.3
Literacy 46 1.0 24.0 14.3
Overall 46 9.0 40.0 27.6
Mansa Numeracy 30 2.0 19.0 11.6
Literacy 30 0.0 20.0 8.8
Overall 30 3.0 39.0 20.4
Samfya Numeracy 60 1.0 20.0 9.4
Literacy 60 1.0 22.0 6.6
Overall 60 3.0 33.0 16.0
 - 18 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
getting extra support for their IRI centres. Also, their mentors are some of which 
received training during the pilot phase, who have continued to show enthusiasm and 
superior performance as IRI mentors. Differences in performance both by locality and 
by center are captured in Figure 2 below. 
Figure 2: Centers, by locality and name, performing above and below 50 percent
12 urban centers 
above 50 percent 
10 of 19 peri-urban 
centers 
above 50 percent
14 of 24 rural centers 
above 50 percent
1. Bauleni 
2. Garden
3. Kanyama
4. Kamanga
5. Chainda 
6. Musenga 
7. Luyeye
8. Mulyambike
9. Buyantashi
10. Tazara
11. Kasama
12. Chiba
1. Mulambe
2. Mwalushi
3. Paul Kalemba
4. Chimbele
5. Chisekesi
6. Linda
7. Mponda
8. Chibeka
9. Mulilo
10. Mweenda
1. Kamanampemba
2. Kapoli
3. Chipushi
4. Lwenge
5. Muzizi
6. Mishusha
7. Neverest
8. Cheelo
9. Kasikili
10. Kanchomba
11. Nangombe
12. Mwachilele
13. Nsemba
14. Lusitu
0 urban centers 
below 50 percent
9 of 19 peri-urban centers 
below 50 percent
10 of 24 rural centers 
below 50 percent
1. Langiboy
2. Chipalila
3. David Ramushu
4. Kasoma Bengweulu
5. Moomba
6. Chitondo
7. Malandu
8. Molombola
9. Mwafuli
1. Kabwenko
2. Mbulwe
3. Kalombo
4. Mubanga
5. Chechemu
6. Chifunde
7. Misundu
8. Mpandwe
9. Chipembele
10. Chilyabale
When disaggregating data by locality, three categories of urban, peri-urban and rural 
were used. Peri-urban areas in this case are those that fall within 15 kms of an 
administrative district. A typical peri-urban community will have at least one 
government school, in some cases a community school, and other amenities such as 
electricity and accessible roads. Poverty and/or income levels in peri-urban areas are 
not as high as in rural areas. 
As was expected, the highest mean score was for urban centers 28.1 (63.9%), followed 
by that of the rural centers at 23.5 (53.4). Peri-urban centers had the lowest mean score 
at 22.5 (51.1 percent). Figure 2 below presents centers that are performing above or 
below 50 percent in each of the three localities. All 12 urban centers performed above 50 
 - 19 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
percent, compared to only 9 of 19 peri-urban centers, contrary to expectations. The rural 
centers performed well, with 14 out of 24 performing above the 50 percent score. 
Centers which performed highest, with a mean of 70 percent or higher were Linda, 
Garden, Musenga, Nangombe, Tazara Farm, and Chisekesi. Three of these are in 
Lusaka province, one in Northern, one in Central, and one in Southern province. The 
spread of high performers throughout the districts suggests that IRI can achieve high 
results under all different localities, some of which are far from Lusaka where the best 
education support services are. Center specific factors are usually responsible for high 
performance, or the lack thereof, as was shown in the previous evaluation. For instance, 
Tazara Farm in Mkushi is one place where the farm owner has built a shelter for the 
center, and is paying the mentor K250 000 per month, and is also providing books and 
uniforms for learners. 
To further illustrate the influence of center specific factors on mean performance, four of 
five centers which were sampled in Southern Province have performed well (Chisekesi, 
71.9 percent; Cheelo, 69.0 percent; Kasikili, 68.0 percent; and Kanchomba, 67.3 percent). 
At 37.3 percent, Chipembele is the only center in this province that performed poorly. 
Once more, good performance in this province is due to the support given by the 
church through Fr. Tadeusz of Chikuni Community Radio Station in close partnership 
with the MoE through the Zonal Inservice provider (ZIP). Centres that have performed 
well in two or more evaluations are Linda, Garden, and Cheelo. There is sufficient 
evidence that these are functioning as expected, hence they will not be included in the 
next evaluation sample. 
On the very low end, eight (8) centers performed at 39 percent or below. Four (4) of 
these are in Central Province (Mbulwe, 22.1 percent; Chifunde, 24.4 percent; Mpandwe, 
33.9 percent; and, Moomba, 37.8), two are in Luapula Province, (Kasoma Bengweulu, 
25.5 percent; Mwamfuli, 31.5 percent; Chipembele at 37.3 percent, in Southern Province, 
and Mubanga at 39.9 percent, in Northen Province. In Serenje, Central Province, 
children did not receive daily broadcasts for a significant amount of time, when they 
had no shelter under which to conduct classes, hence they missed a sizeable chunk of 
the lessons. 
Some of the problems which may be impacting on performance in Luapula province 
were noted by test administrators during the IRI assessment visits in August. These 
included examples where mentors had decided to use only the mentors guide, and not 
the radio broadcast as required for IRI, and appeared somewhat disinterested in IRI or 
the learners. In addition, the community itself was ill informed on how the program 
should be running, and part of this was blamed of the fact that one mentor was not a 
resident of the community. 
 - 20 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
2.4.2 Performance by age 
The ages of learners in this sample ranged between 5 and 16 years old. However, with 
the mean age in urban areas being 8.9, compared to 10.6 in the rural areas, a significant5
difference was observed when age was disaggregated by locality. The first age category 
which accounts for about 10 percent of the learners was that of children who are yet to 
reach the official school-going age and could be considered too young to be in IRI 
centers. The second age category of 7 to 13 year olds captured those who are the official 
age for primary education. The third and last category, about 3.1 percent, was that of 14 
to 16 year olds, learners who would under normal circumstances be pursuing 
secondary education. Significant differences were also observed when the age 
categories were used to disaggregate performance as reflected in Table 11. 
Table 11: Performance by age category
Age 
Categories
Test N Minimum Maximum Mean Std. 
Deviation
6 years and under Numeracy 85 1.0 19.0 8.1 4.6
Literacy 81 0.0 22.0 8.2 5.8
Composite 85 1.0 37.0 16.0 9.0
7 thru 13 years Numeracy 708 0.0 20.0 13.0 4.5
Literacy 707 0.0 24.0 12.1 5.6
Composite 710 3.0 44.0 25.0 8.7
14 and above Numeracy 25 6.0 20.0 14.7 3.4
Literacy 25 3.0 24.0 12.6 5.7
Composite 25 10.0 41.0 27.3 8.1
As with the 2001 scores, the results showed an increase in performance as age increased. 
The mean differences were significant6. The standard deviations were also increasingly 
narrower, which means that with increasing age, learners were becoming more 
competent on the skills that were tested. Another conclusion that can be inferred from 
the fact that standard deviations were narrower for the numeracy subtest, in 
comparison to those of literacy skills subtest within each age group, is that examinees 
were more variant in literacy than in numeracy. The data seems to suggest that age has 
to be taken into consideration in IRI if returns are to be maximized. This finding will be 
important for future enrolment policy and practice as more IRI centers are organized. 
2.4.3 Comparison with the 2001 cohort 
Table 12 below includes, for comparison, performance scores for test compared to the 
2001 performance. Even though the scores show internal consistency, performance in 
2001 was superior to performance on the current test. This is partly due to the fact that 
there were a lot more IRI centers listening to Grade 1 broadcasts in 2003, which means 
 
5
 t = 3.89, p = .00 
6
 Composite: F = 14.3, p = 00; Numeracy: F = 23.5, p =.00; Literacy: F=5.8, p = .01, all at df = 341 
 - 21 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
that standardization of procedures, and supervision of the learning center was, at best, 
minimal. 
Table 12: Test Means for Literacy, Numeracy for 2001 and 2003
Sample size Mean
Content 2001 2003 2001 2003
Numeracy 405 826 14.3 12.6
Literacy 405 821 13.6 11.7
Total Test 405 828 27.9 24.2
Differences between the test scores from one year to the next are to be expected, at least 
in the formative years of the program when systems have not yet stabilized. However, 
there is need to have performance standards against which comparisons from year to 
year can be made. An opportune time to articulate these would be with the newly 
revised Grade 1 curriculum and/or programmes, there should be benchmarking 
against which performance from year to year can be compared. To this end, cut-off 
points that define different levels of performance should be defined. A psychometric 
approach that is a hybrid of mentors and teachers’ judgments on individual item 
content, and judgments on examinees test scores would be preferable. More 
experienced teachers from community schools which tune in to the IRI programs would 
be invaluable to this exercise since judgments of mentors alone could be challenged on 
the basis that they are not qualified teachers. 
2.5 Impression on Effectiveness of IRI: Mentors and parents and other 
stakeholders 
Data on views and impressions of mentors and parents were collected in a variety of 
ways. First, EBS receives reports from Provincial Outreach Coordinators on a monthly 
basis, and from various EBS/EDC monitoring activities. Views of mentors, parents and 
other stakeholders are expressed in many of those reports. Secondly, data was collected 
through a mentor self-administered questionnaire during testing time. A total of 110 
mentors responded to the questionnaire. Thirdly, focus group meetings were also held 
for the parents. Where possible, mentors were asked to join the focus group meetings. 
2.5.1 Mentors 
About 23 percent of mentors were female compared to 77.3 percent male. The age of the 
mentors ranged from 22 to 66 years old. On average, mentors had given 2.1 years of 
service. About 7.5 percent of mentors had served IRI since its first cohort in 2000, while 
34.7 percent were in their first year. Two of 102 mentors had received 19 days of 
training, while 26.5 percent had received no training at all. On average, mentors 
received 4.6 days of training as shown in Table 13. 
Table 13: Mentors’ years of service and mean days of IRI training 
 - 22 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Years as mentor N (%) Minimum Maximum Mean No training (%)
1 Year 32 (34.6) 0.0 07 2.3 50.0
2 Years 35 (32.7) 0.0 10 3.5 22.9
3 Years 24 (25.2) 0.0 19 7.6 04.2
4 Years 08 (07.5) 6.0 15 10.3 00.0
Overall 102 0.0 19.0 4.6 26.5
With an average of 10.2 IRI training days, the longest serving mentors are also the best 
prepared. As the table indicates, all of the veteran mentors received at least a full week 
of training at the beginning of the IRI program in 2000. With about 50 percent of 
mentors having received no training at all, training could partly explain the drop in 
performance scorers over the years (overall mean of 24.2 in 2003, compared to 27.9 in 
2001, and 32.3 in 2000). In addition to the age of the learner, mentor training seems to be 
the only other variable that introduces significant variation in performance. These data 
demonstrate two points; namely, the worth of mentors in the success of the IRI 
programme, and the value that can be added to the learning process by exposing the 
mentor to thorough training. Indeed, the centrality of mentors to the success of IRI has 
been confirmed in all evaluations, with recommendations made to curb mentor 
attrition. 
Apart from the training issues, mentors were asked about materials and support that is 
available to her/him through learning materials and the support given to her/him by 
the community, as well as the effectiveness of IRI. The majority of mentors thought 
their centers were successful centers (54.6 percent), while 3.6 thought their IRI program 
was not successful. More than 80 percent of the mentors believed IRI centers were doing 
just as well as GRZ schools, if not better. 
2.5.2 Community/parent participation 
Parents are overwhelmingly in favour of the IRI program for a variety of reasons, the 
most compelling being that it provides access to education, and an opportunity to poor 
and vulnerable children. Other reasons were the low cost implication of learning 
through IRI, cutting on the distances that children have to walk to regular schools, and 
community ownership of the program. For instance, in Siavonga the community wants 
to be in-charge of IRI themselves, and does not want government to interfere with 
“unreliable” assistance. In fact, community members cited “unending teacher strikes” 
as one of the reason for sending children to IRI learning centers, even in some places 
where there is a government school. 
However, parent participation and/or support in IRI is low, with only 30 percent of the 
parents assisting the mentor directly through offering money, food, and some 
household items. What emerged though is that in many cases, communities agreed in 
principle that they should make a contribution. However, they ended up not managing 
to do so. In a number of cases, it turned out that they stopped making cash 
contributions after government declared free education. 
 - 23 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
There were centers with systematic assistance for mentors. In Mkushi, where IRI centres 
were opened in farm blocks centers and mentors were supported by farm owners, both 
financially and materially. Some mentors were being paid between K200, 000 and K250, 
000 per month. One farmer in the farm block has put a permanent structure running 
Grades 1 through 4. The learners are expected to go to the nearest government school at 
the completion of the foundation phase. The farm owner is also providing uniforms for 
learners. Assistance from farm owners was also reported in Kalambo, Northern 
province where a farmer supports by providing transport, exercise books, radio and 
pays two teachers. Interestingly, this appears to be both a strength and a weakness as it 
was also identified as a threat due to the fact that the white farmer bears all payments. 
A finding of this evaluation from community focus groups confirms Siaciwena (2002) 
that community participation is best coordinated through unambiguous structures such 
as IRI committees. Where committees exist, IRI support activities are easily coordinated 
as in the case of Chikuni, where all centres have committees that were functioning well 
(Siaciwena, 2002). To the majority of parents and community members, the strengths of 
IRI included having their mentor on government payroll and community provides free 
housing as is the case in Mbala; provision of ‘uniform learning’ (from the radio 
teacher); the ability to read, write and speak in English; children that are well behaved 
and respectful; improved hygiene; reduction in the number of potential street children, 
and in incidents of stealing the absence of teacher/mentor strikes. 
The greatest threat to the IRI program was deemed to be minimal support offered to 
mentors by communities. Most parents and community members expressed a fear that 
mentors would get discouraged and stop teaching, thereby resulting in closure of 
centres and the end of opportunities for their children to learn. As one person put it, 
“that will be the end of the children’s education” Other threats included lack of 
ownership; lack of structures, “a place to call our own”, and hence, constant fear of 
eviction, inclement weather; malfunctioning radios; and early marriages of the girls, as 
reported in Muzizi and Nchecama (where there were only 4 girls in G4). Some 
parents/communities expressed uncertainty about the future of IRI, as in Government 
running out of money to pay ZNBC. Hence they were reluctant to support it. 
 - 24 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
3.0 CONCLUSION AND RECOMMENDATIONS 
EBS has systematically conducted evaluations of the IRI program at the Grade 1 level in 
order to track the demand and/or growth of IRI. The current evaluation was especially 
important since the Zambian context has changed to include restructuring and 
reoganisation of the MoE, and introduction of free primary education (FPE). With the 
introduction of FPE, IRI is emerging as a necessary partner and alternative to the school 
system in communities where the Zambian education system is overwhelmed, and the 
only available opportunity for accessing basic education in others. 
The evaluation is also important because it marks the end of the funding phase that saw 
the birth of IRI (its pilot and the rolling out of IRI to the rest of the country), and ushers 
in a new phase. Most of evaluation findings speak to the same issues that were raised in 
earlier reports, issues on which recommendations for action were made. Hence in this 
section we consolidate the previous recommendations, examine which ones have been 
acted upon, whether or not those that are yet to be acted upon have not been overtaken 
by events, and recommend accordingly. We also raise new issues to be considered in 
the planning of the next phase of IRI. 
 3.1 Previous Recommendations 
The table below indicates the overlaps in the recommendations, with respect to issues 
such as the development of regulatory measures for communities and/or organizations 
that want to start IRLCs; development of grade level performance standards of 
achievement; delineating mentor profiles and putting developing incentives for 
mentors; and developing a monitoring and evaluation framework which stipulates 
roles and responsibilities of the different monitors. 
3.1.1 Demand of IRI 
The August 2001 evaluation identified the need to introduce some regulation in 
establishing new centers to relieve mentors of the burden of having to attend to highly 
heterogeneous groups of learners. The criteria that has been stipulated in the IRI 
sensitization literature by EBS include measures such as recruiting a mentor and 
devising means to support him/her, constituting a village IRI committee, having a 
shelter, buying a radio and a chalkboard, and pledging support for the center. This 
criteria does not address the issue of regulating admission to a stipulated age group of 
learners, such that IRI is not watered down as mentors struggle to accommodate 
unintended audiences. 
We reiterate our earlier recommendation that there needs to be some regulation in 
establishing new centers. The regulatory framework should address issues such as the 
minimum number of learners that should be assembled before a center can start 
operating, the age of the learners, and the type of support that centers will qualify for,
 - 25 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Figure 3. Recommendations from 2001 Grade 1 Evaluation
RECOMMENDATIONS,
LETSHABO & KARIUKI, 2001
CORRESPONDING RECOMMENDATIONS 
SIACIWENA, 2002
Demand of IRI: Articulate a simple regulatory IRI strategy that provides 
the minimum requirements and criteria that should be met for a 
community to organize a formal learning center.
Recommendation 6: Develop criteria for certification process for the interactive 
radio instruction learning centres. Once these are established small grants 
should be made available to IRLCs. 
Attendance: Articulate a mentor-friendly strategy for tracking attendance 
of learners at centers, which should include providing an official register to 
be collected at the end of each grade. 
IRI Learner Profile: Delineate a profile for an IRI learner, the ideal “out￾of-school” learner for which the program was initially designed, such that 
the IRI service should be made available to those who need it the most.
Student Learning: Design a standard setting exercise that would 
determine cut-off scores for different levels of mastery (and/or 
proficiency) to provide consistent descriptions of learning achievement, 
and yearly comparisons.
Recommendation 13: Decide whether the program will expand to Grade 7, and 
if so, appoint working group to make proposals on the assessment and 
integration of IRI Grade 7 learners into the formal school system.
Mentor Profile: A deliberate attempt should be made to delineate the 
profile of an ideal mentor for the IRI program, his/her training needs, and a 
mechanism to improve the chances that mentors are retained. 
Recommendation 21: Make mentoring an advantage in admission to teacher 
training colleges. 
Recommendation 22: Improve all aspects of mentor training and award MOE 
certificates
Effectiveness of IRI: EBS and EDC should train mentors on how to keep 
a journal of critical events (aspects of the program that work well, and 
those that do not, as well as certain occurrences that demonstrate that 
learning has taken place), to be used in monitoring, formative, and 
summative evaluations of the program.
M & E Framework: Develop a cost-effective and sustainable monitoring 
and evaluation framework, possibly one that is sustainable, and that can be 
institutionalized into the operations of EBS.
Recommendation 7: Define monitoring and evaluation roles and responsibilities 
in light of decentralization of education services; create clear channel for 
communicating and utilizing data. EBS should not take leading role in M&E.
Recommendation 8: MOE should build capacity for monitoring IRI learning 
centres by allocating money to it in the budget; and by building appropriate 
expertise in monitors.
IRI Database: A database of center, student and mentor profile be 
developed. It will be more desirable and efficient than the current spread 
sheet that is used.
 - 26 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
depending on their circumstances. Siaciwena (2002) recommended that centers could then 
qualify for a small grant. This is desirable. One of the functions of the EBS based POCs could 
be to solicit support from various stakeholders, for specific centers, in cash or kind. 
Another issue of demand is that of the higher grades. While it has been demonstrated in this 
evaluation that participation at Grade 5 had dropped considerably, a new challenge for IRI is 
whether or not to develop new programs for Grades 6 and 7. Data that is collected from this 
evaluation seems to suggest that it will not be cost-effective move for EBS. However, a policy 
decision about the envisaged role of IRI in the overall education system could shape the 
direction to take, both for EBS and other MOE departments. 
3.1.2 Attendance 
With the appointment of POCs to assist MoE to supervise center activities more closely, there 
will be increased monitoring of attendance. EBS has had to provide attendance registers to 
IRLCs. However, in order to ensure that mentors are not spending valuable instructional 
time on an administrative task of taking attendance, a strategy on taking attendance has to be 
negotiated with mentors. One strategy would be to involve the children in higher grades in 
this task; that is, assign them days of the week they have to count their classmates, and then 
approach the teacher after the lesson to report who was absent. 
3.1.3 IRI Learner Profile 
The IRI system is now being overwhelmed by new audiences, sometimes at the expense of 
the out-of-school children. While it may be counter productive to discourage other users to 
take advantage of the IRI airwaves, EBS should rationalize the assistance and support that 
they can offer vis-a-vis their capacity. Policy positions and other consideration on this issue 
should give clear priority to out-of-school disadvantaged learners, however. 
3.1.4 Student Learning
Learners have progressed well through the five grades for which there are IRI programs. 
Comparative studies have shown that IRI learners were performing just as well as regular 
school learners, if not better at Grade 2 and 4 (ECZ, 2002). In order to earn IRI more support 
and credibility as a necessary alternative for about 600 000 Zambian children who are still left 
out of the school system, EBS should work more closely with the relevant MOE department 
to articulate grade level stands and/or competencies that can be used to ensure that the 
education opportunity of IRI learners is comparable to that of school learners. This effort will 
naturally culminate into integration procedures for all grade levels, school learners who want 
to join the school system. This is an extension and more comprehensive approach than 
suggested in Siaciwena, 2002. 
3.1.5 Mentor Profile 
As shown in Figure 3, an earlier evaluation had recommended that a deliberate attempt 
should be made to delineate the profile of an ideal mentor for the IRI program, his/her 
training needs, and a mechanism to improve the chances that mentors are retained. An 
increased number of mentors who have gone on to teachers training colleges, some with the 
 - 27 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
assistance of DIPs. EBS needs to track information on training opportunities for mentors, and 
make a concerted effort to support applications of mentors. In addition to this, all mentors 
should be exposed to yearly in-service training, and be awarded a MOE certificate in 
recognition of the training and their experience as suggested by Siaciwena, 2002. Moreover, 
the policy position on the role of IRI should articulate profile and preparation, as well as 
devise strategies aimed at retaining mentors. 
3.1.6 Monitoring and Evaluation Framework 
While there are several loosely coordinated monitoring and evaluation activities going on, M 
& E has to be institutionalized into EBS operations. This should be done within the overall 
task of developing a management system for the different EBS functions. 
3.2 Siaciwena (2002) Recommendations 
The USAID commissioned evaluation of August 2002 focused on different EBS processes as 
they impact on IRI delivery. In addition to the recommendations on IRI instructional systems, 
the recommendations below focused on support systems. We isolate a few for emphasis and 
discuss them broadly under lesson development, broadcast production, and IRI management 
as shown in Figure 4 on the next page. 
3.2.1 Lesson development 
Lesson development activities have so far been undertaken by EBS staff supported through 
technical assistants. EBS staff have done a commendable job, though the process continues to 
evolve and improve. While it is necessary to have on the job training to perform certain 
specific tasks it is even more desirable to present candidates for long-term training, people 
who will be empowered to direct the EBS IRI program in the future. Selected EBS staff could 
undergo in-depth training in aspects of curriculum and instructional design to the make the 
IRI lessons more effective. Alternatively, qualified curriculum specialist should be relocated 
to EBS where they can receive in-house and/or professional training in delivering the 
curriculum through the medium of radio. 
3.2.2 Broadcast production 
EBS has recently acquired equipment through USAID funding. With proper training, the 
available equipment is sufficient for the production activities. The weakest link in the 
broadcast production process is two pronged. First, the studio technicians need to be trained 
so as to update their skills. Human resource issues need to be complemented by appointing 
relevant personnel in supervisory roles as was suggested by Siaciwena (2002). Secondly, 
manuals for the production processes need to be updated, with sufficient time being allowed 
for quality assurance. 
Figure 4: Additional recommendations of August, 2002 
 - 28 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Improving Lesson Development
Recommendation 1, 2 and 27: EBS should train writers to be more creative and to produce 
interactive radio lessons; review work schedules for writers; avail more opportunities for 
formative evaluation; identify CDC curriculum specialists to work with writers. EBS should also 
train or use people who speak clearly as radio teachers; improve the Lifeskills component by 
increasing time for the segment, providing lifeskills training for mentors, and more teaching aids
Improving Broadcast Production
Recommendation 3 and 4: Provide necessary equipment for studio production and maintenance 
services; develop a workable production schedule, and develop studio procedures to ensure 
systematic handling of scripts and tapes; hire a studio manager to supervise technicians and other 
studio staff. 
Community participation
Recommendation 5: Continue to sensitize communities on IRI and their responsibilities; create and 
clear communication channels between EBS, MOE and the community
Capacity to manage the IRI program
Recommendation 9: Develop and institutionalize within EBS, an IRI management system.
Recommendation 10: Create conditions to enhance job satisfaction and retention among EBS staff.
IRI and other MOE Departments
Recommendation 12 and 16: Institutionalize IRI within MOE, identify roles for MOE departments 
who are strategic partners; office of PS should define roles of MOE departments in promoting the 
IRI, and provide guidelines
 
3.2.3 IRI Management 
Siaciwena (2002) has recommended the institutionalization of IRI within MOE. This process 
can logically begin by articulating a policy position on IRI as already mentioned above. Once 
the policy is articulated, the other recommendation would fall into place; that is, to identify 
roles for MOE departments who are strategic partners, and for the office of PS to define roles 
of MOE departments in promoting the IRI. 
3.3 Summary Recommendation 
In light of a number of actions that still need to be followed up from previous evaluation 
studies, this evaluation makes only one recommendation; that EBS should take advantage of the 
MOE restructuring exercise to streamline IRI activities, and to devolve some of the functions assumed 
in the pilot and roll-out phase to the relevant MOE departments. These functions include training of 
mentors, outreach, assessment of achievement, and monitoring and evaluation. These activities will 
create an enabling environment for the MOE to develop a responsive policy position on the role of IRI 
in the overall provision of basic education in Zambia. 
 - 29 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
4.0 APPENDICES 
Appendix A: Methodology for evaluation 
Appendix B: Data collection sheet for students’ profile 
Appendix C: Content Map for Grade 1 English 
Appendix D: Content Map for Grade 1 Mathematics 
Appendix E: Test Blueprint for Grade 1 English 
Appendix F: Test Blueprint for Grade 1 Mathematics 
Appendix G: Test Plan for Numeracy and English Literacy for Grade 1 
Appendix H: Grade 1 Numeracy and English Literacy Test 
Appendix I: Training Notes for the Administrators 
Appendix J: Guidelines for Test Administrators 
Appendix K: Mentor Questionnaire 
Appendix L: Focus Group Questions 
 - 30 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix A: Methodology 
Sampling 
In August 2003, at the time of the testing exercise, the reported population of Grade 1 
IRI learners was 12 461 learners, attending 516 centers in all nine provinces. The 
evaluation strategy stipulated the use of population parameters to describe the profile 
of learners in the centers, in particular their sex, age, who their guardians are, and 
whether they have any living parents (orphan status), and sample statistics for the 
achievement scores. While population parameters will be used where they are available, 
most of the variables will report sample statistics. 
A three step sampling design was used. First, a purposive sample was drawn from 5 of 
9 provinces, with provinces that had more centers being preferred over those that had 
less centers. Secondly, 55 centres were sampled randomly from a total of 516. Twelve 
centres are located in urban areas , 24 in rural communities, and 19 in peri-urban areas7. 
These centers had a total of 992 learners. Thirdly, 16 learners were sample from The 
table below shows the population of learners and sampled size per province. 
Evaluation Sample: Grade 1 IRI Learners in 2003 
 
Population 
of centers 
# of centers 
Sampled
Population
of learners per 
province
# of learners
sampled per 
province
Northern 82 24 2 364 384
Luapula 54 06 1 286 96
Central 96 10 2 207 160
Southern 26 06 978 160
Lusaka 46 09 1 804 192
TOTALS 304 55 8639 992
The sample of 992 was 11.5 percent of the population in the 5 provinces, and 7 per cent 
of the total Grade 1 population. The number that actually got tested was 828. This is 
because in some places, there were fewer than 16 learners, the number that was to be 
tested per centre. 
Test Development 
IRI methodologies emphasize a communicative learning approach, where instead of 
using the traditional learning approach of compartmentalizing into subject areas, 
presentation of new knowledge and skills is integrated. This has made it possible to 
develop a test of literacy and numeracy skills, that was packaged in the manner that 
 
7 Peri-urban areas in this case are those that fall within 15 kms of an administrative district. A 
typical peri-urban community will have at least one government school, in some cases a 
community school, and other amenities such as electricity and accessible roads. 
 - 31 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
learners are familiar with from their daily lessons, and presented to them orally. This 
section describes the rationale for developing a literacy and numeracy achievement test 
for Grade 1. The mastery test that is used in this evaluation was developed for the 
evaluation in 2000, and revised in 2001. The test development process for its 
development included content mapping, test blueprinting, developing a test plan, item 
writing and pilot testing. The test administration procedure is described, followed by a 
presentation and discussion of test results. 
Content Mapping 
The test development process commenced by a content analysis for Grade 1, performed 
by the Senior Research Officer with the Examinations Council of Zambia and a test 
development specialist. Documents that were analyzed are the Mathematics and 
English syllabi which specify terminal as well as instructional objectives for the primary 
level, and the mentors’ guide for the lesson broadcasts. Two content maps were 
developed, one for English and another for Mathematics. In these, instructional 
objectives that specify the content for the grade level were classified in a two-way 
matrix of terminal objectives by topic or skill area. In the absence of grade-level reading 
lists that usually indicate the reading levels of learners, the mentors’ guide was 
particularly useful in that it specifies new vocabulary and the numeracy skills that are 
presented in each lesson. It also formally specifies the life skills content, a deliberate 
effort to advance attitudinal learning alongside the cognitive skills that children have to 
master at this formative stage of being introduced to formal learning. 
In the content map for English instructional objectives are classified into a two-way 
matrix of twelve Grade 1 terminal objectives by the skill areas of reading, writing, 
speaking, listening, phonics and spelling (see Appendix B). Out of the 106 instructional 
objectives that make up the content for Grade 1 English, 82 are terminal. These are all in 
the areas of listening and speaking. The remaining 24 that fall in the areas of reading 
and writing are developmental objectives to be mastered in later years. 
The first three terminal objectives for the primary Mathematics course were deemed 
terminal for Grade 1. Learners at this level are expected to master the skills of naming 
and classifying objects according to given conditions, demonstrate understanding of the 
concept of number and numeration, and perform the basic operations on numbers and 
other simple measures. All the 22 instructional objectives which are specified in the 
content map are terminal, 10 of which require learners to demonstrate understanding of 
the concept of numbers and numeration (see Appendix C). 
Subject Blueprinting 
Test blueprints were developed for English and Mathematics (Appendices D and E). In 
these, instructional objectives were mapped into a two-way matrix of content topics by 
level of cognitive complexity using Bloom’s taxonomy8. Even though the application of 
 
8 Bloom’s taxonomy was the classification scheme that is used by the Ministry of Education. 
 - 32 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
what was learnt in both English and Mathematics is expected at this level, greater 
emphasis was placed on acquisition of new knowledge and comprehension of concepts 
for both subjects. The higher cognitive levels of Bloom’s taxonomy (analysis, synthesis, 
and evaluation) were not featured at all. 
Test Planning 
One test plan of literacy and numeracy knowledge and skills was developed. Once 
more, a two-way matrix of topic/skill area by cognitive level was used. Entries in the 
body of the test plan were instructional objectives from both the Mathematics and 
English syllabi (see Appendix F). The skills area of the test plan (first column) presented 
terminal objectives, where the first four skills was a summary of the 13 terminal 
objectives of Grade 1 English. The last three skills (in italics) reflect the mathematics 
terminal objectives that are suitable for Grade 1. 
The optimal test length that was decided upon by the curriculum and test development 
specialist was about 20 items, taking into consideration the method of administration, 
and the time that it would take to administer the test. The plan reflected a total of 22-
item test consisting of 12 literacy items and 10 numeracy items, sampled from 20 
instructional objectives. Four items were planned at the knowledge level, 13 at 
comprehension, and 5 at the application level. The last column of the test plan 
stipulated the number of questions to be tested per skill, and the different levels (using 
Bloom’s taxonomy) at which the items for each skill would be tested. Instructional 
objectives that were sampled for testing are presented in bold in the body of the test 
plan. 
Test Construction 
The purpose of the test was to assess and evaluate if learners have mastered basic 
literacy and numeracy skills, and whether they could understand simple 
communication at the end of the Grade 1 course. The guiding principle during test 
development was that assessment procedures should match the intentions of each 
learning target, hence the behaviors which were elicited from the learners included 
recalling certain facts, as well as performing certain tasks. For instance, the intention of 
the learning targets on language during the early stages of learning is that learners 
should comprehend language and begin to produce simple language. Their 
comprehension of language in the lessons is demonstrated by the acting out simple 
instructions, hence the assessment of language skills comprised mainly of requesting 
them to perform actions when given simple instructions. 
At least one item was constructed for the 20 objectives that were sampled from the test 
plan. The test development specialist was mainly responsible for generating the test 
items. The ECZ Senior Research an individual with a thorough knowledge of the 
intended curriculum, reviewed these for content coverage and readability. Members of 
 
 - 33 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
the IRI team who are most conversant with the way the radio lessons were structured 
and delivered also participated in the item review. 
For most objectives it was possible to construct a set of parallel items from which the 
test administrator would select the item to present to the learner. These items were pilot 
tested by 4 test administrators in 2 centers in Lusaka. Pilot testing assessed whether the 
questions elicited the intended behavior/skills, whether the correct difficulty levels in 
terms of content and language were maintained, the amount of time it took to 
administer the test, and whether the proposed administration procedure was 
reasonable and adequate. 
Modifications that were made after pilot testing included simplifying content for some 
of the questions, recasting other questions in simpler language, clarifying certain tasks, 
and inserting instructions, also to increase clarity on the administration procedure. 
Table 4 below produces a summary of intended learning targets (also reflected in the 
test plan) and the corresponding test items in the final version of the test. The final 
version of the test consisted of 22-items (see Appendix G). 
Skill areas and corresponding test items for Grade 1 Test, 2001 
Skill Area Intended Learning Target Test Items
1. Recalling names
2. Simple comprehension of language
11, 12, 
13, 14, 15, 19
3. Production of language (speaking)
4. Production of language (reading)
16, 17, 18
20, 21
Language
5. Production of language (writing ) 22
1. Counting and writing numbers 1, 2, 3, 4
2. Naming/classification/comparisons 9, 10
Numeracy
3. Number operations 5, 6, 7, 8
Twelve (12) items assessed language skills, while 10 items assessed numeracy skills. 
Translation into local languages was allowed for the 10 items that tested numeracy. In 
the interest of keeping the test short and simple, no items were included for the lifeskills 
component. Also, the objectives for the lifeskills component are not expressed explicitly 
in the curriculum, which means that a different strategy will have to be used to assess 
whether children are learning in this area. 
Other instruments 
Other instruments that were designed for this evaluation are mentor and focus group 
questionnaires. The mentor questionnaire was a self-administered instrument, while 
the focus group was filled by a member of the testing team 
Test Administration 
Testing was conducted in August, 2003 in 55 centers, 24 of them located rural areas, 12 
in urban areas, while 19 were located in peri-urban areas. 10 of the centers were part of 
 - 34 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
the sample in the 2001 evaluation. Test administration began with the identification of 
test administrators both at national and provincial levels. The officers identified were 
Outreach Advisor, Monitoring and Research Advisor, IRI Focal Points Persons, District 
Inservice Providers (DIPs), Assistant DIPs, POCs, Senior Producers at EBS and Peace 
Corps volunteers. Test administrators underwent a one-day training in test 
administration so as to equip them with knowledge and skills and also to have a 
common understanding of the evaluation exercise and its importance. Each test 
administrator was required to arrive at the center in time for the broadcast, test the 
learners and record the responses accordingly, as well as submit a report for the testing 
visit 
Interpretation of test scores 
The purpose of this test was to monitor if learning is taking place and describe the 
learning patterns, first, for the whole group of Grade 1 learners, and secondly, for 
subgroups. Subgroups that have been identified for analysis include males, females, 
rural learners, urban learners, learners in specific centers, among others. The test is 
therefore suitable and valid for monitoring learning at a group level, but not equally 
valid for describing achievement for individual learners. 
In order to enhance the validity of the test as a monitoring tool, content domains or skill 
areas have been clearly delineated, hence it is possible to calculate means for domain 
“subtests”. This is the most valuable information for our purposes because conclusions 
can be made confidently about skill areas where students seem to have achieved the 
most, and where they have been found to be weakest. Data on individual items on the 
test is only useful to the extent that it provides an indication of what learning 
difficulties learners could be having on specific content matter that each item is testing. 
However, it has also been possible to draw conclusions about mastery for the group of 
learners at the item level. The test was built such that for most items, administrators 
could randomly sample the item to be presented from a list of parallel stimuli (see Items 
1, 2, 5, 6, 7, 8, 11, 12, 13, 17, 18, 19, 20, 21, 22). The implication of this practice is that 
conclusions on content as presented in individual items were made not only on one 
item, but on a number of such items. The other strength was that there were different 
combinations of items that were presented to the students, which practice increased the 
validity of the test for making mastery decisions. 
As a tool for monitoring learning and making summative evaluations about the 
learning environment, the test has performed its intended purpose adequately. 
Conclusive evidence on performance in content specific items at the individual student 
level would be derived from a different process, which necessitates that there be more 
than just one item on each topic area, that sampling of items be adequate, that the test 
be of a suitable length, and that testing time be more suited to learners at the Grade 1 
level. The usual practice for such individual testing is that the achievement test for the 
whole curriculum is broken down into manageable subtests that are administered in 
several sittings. 
 - 35 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix B: Data collection sheet for students’ profile 
Center:
C# ID Name Age Sex SchAtt Reside Live With? Sibling? Parent Alive?
__________________________________________ 
 - 36 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix C: Content Mapping for Grade 1 - English 
Terminal Objectives Listening and
Speaking
Reading Phonics Writing Spelling
1. Understand and respond to teacher’s 
instructions
7.1
2. Give instructions 7.2 7.3
3. Use language in social interaction, 
classroom organization, correction, and 
simple explanations
1.1
4.1 4.2
8.1 – 8.5
4. Answer questions
5. Make statements 2.4
5.1 5.2 5.4 5.6 – 5.10
6.2 – 6.4
6. Ask questions 3.1 – 3.16
5.3 5.5
7. Ask for information 6.1
8.8
8. Use simple descriptive language 1.2 1.4 – 1.9
2.1 – 2.3 2.5 – 2.8 
9. Listen and demonstrate 
understanding
10. Use language simple in interactive 
activities 
6.2 – 6.4
9.1 – 9.9 10.1 – 10.7
11. Demonstrate understanding
12. Pronounce words 11.1
13.1 – 13.4
C1
C2
 - 37 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix D: Content Mapping for Grade 1 - Mathematics 
Topics /Terminal Objectives/ Naming Classifying 
objects and numbers 
according to given 
condition
Demonstrate 
understanding of concept 
of number and 
numeration
Perform basic operations 
on numbers and other 
measures
Sets 1.1-1.2 1.3–1.5
Numbers 2.1; 3.1 2.2–2.3 2.4
Completing addition and 
subtraction of number sentences up 
to 100
4.1 4.1; 5.1
Shopping Activities 6.1
Counting 7.1; 10.1 10.1
Telling time 8.1
Comparisons 9.1 – 9.2
Shapes 11.1 11.2
 - 38 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix E: Test Blueprint for Grade 1 - English 
Topics Knowledge Comprehension Application
1. Identifying and classifying 1.1 – 1.2 1.3 - 1.19 1.20
2. Describing narrating and reporting 2.1 – 2.2 2.4 2.3 2.5 – 2.7
3. Finding out (inquiring) 3.1 – 3.16
4. Denying, confirming and correcting 4.1 – 4.2
5. Expressing and finding out 
intellectual attitudes
5.1 – 5.4 5.6 5.8 – 5.10 5.5 5.7 
6. Expression personal meaning 6.2 6.1
7. Getting things done 7.1 – 7.3
8. Using language socially 8.1 8.3 8.4 8.2 8.5
9. Expressing concepts and 
relationships
9.1 – 9.9
10. Talking about quality and quantity 10.1 10.2 10.6 – 10.7 10.3 – 10.5
11. Learning about language 11.1 – 11.3
12. Speechwork and phonics 13.1 – 13.2 13.4 13.3 13.5C1-C2 (P)
13. Reading 1 2 – 5
14. Writing and spelling 1 – 7 (W)
1 (SP)
 - 39 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix F: Test Blueprint for Grade 1 - Mathematics 
Terminal Objectives/ Topics Knowledge Comprehension Application
1. Sets 1.1 – 1.5
2. Numbers 2.1 – 2.3 2.4
3. Naming numbers (0 – 10) 3.1
4. Completing addition and 
subtraction of number sentences up 
to 100
4.1
5. Addition and subtraction 5.1
6. Shopping Activities 6.1
7. Counting in tens and ones 7.1
8. Telling time 8.1
9. Comparisons 9.1 – 9.2
10. Counting in twos up to 100 10.1
11. Shapes 11.1 11.2
 - 40 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix G: Test Plan: Basic Literacy and Numeracy 
Skill Area Knowledge Comprehension Application
Number of 
items
Recall of names 1.1–1.2 1.4 - 1.9; 1.10-1.11; 
1.12-1.16; 1.18-1.19
1.17; 1.20 3 (2K, 1C)
Production of Language 
(Speaking)
2.1-2.2; 2.4; 3.1-3.16; 8.1; 
8.3; 8.4;10.1-10.2
2.3; 2.5-2.7; 4.1-4.2; 5.1-5.4; 5.8-
5.10; 7.1-7.3; 8.2; 8.5; 10.6-
10.7;13.1-13.2; 13.4
5.5; 5.7; 6.1; 9.1-9.3; 
9.4-9.9; 10.3-10.5; 
11.1-11.3;13.3;13.5;
4 (2C, 2A)
Production of Language 
(Reading)
1 2; 3; 4; 5 2 (1K, 1C)
Production of Language 
(Writing)
1; 2; 3; 4; 5; 6; 7 1 (A)
Comprehension of Language 6.2; 10.6; 10.7 2 (C)
Counting 7.1; 10.1 3 (C)
Number Operations (addition, 
subtraction and multiplication)
4.1; 5.1; 6.1 3 (A)
Classification and Comparisons 3.1; 11.1 1.1-1.5; 2.1; 2.2; 2.3; 9.1-9.2 2.4; 11.2 3 (1K, 1C, 1A)
Explanatory Notes: 
1. The skills area of this test plan (first column) presents a summary of terminal objectives. In the first four skills the 13 terminal objectives of 
Grade 1 English are restated in summary form. The last three skills (in italics) reflect the mathematics terminal objectives that are suited 
for Grade 1. The entries in the body of the test plan are instructional objectives from both the Mathematics and English syllabi. 
2. The last column of the test plan stipulates the number of questions to be tested per skill, and the different levels (using Bloom’s taxonomy) 
at which the items for each skill will be tested. This is a 22-item test consisting of 12 literacy items and 10 numeracy items. There are 4 
items at the knowledge level, 13 at comprehension, and 5 at the application level. 
3. Instructional objectives that are sampled for testing are presented in bold in the body of the test plan. From these, several items were set, 
The administrators will typically present one item to the learner, and a second item if the administrator is not certain about mastery/non￾mastery of the objective. 
 - 41 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix H : Test of Mathematics and English Language Skills 
GRADE 1 
TEST ADMINISTRATION SCRIPT 
Instructions: 
• Please read this information before the test and seek clarification where 
necessary. 
• Everything that appears in italics in the test is for the information or 
direction of the mentor/test administrator. Words that are not in italics are 
to be read out to the learner. 
• Administer the whole test (all questions) to each learner. The test is to be 
administered to one learner at a time, away from other learners. 
• The test item is to be presented to the learner a maximum of two times. 
• Translation is allowed on all numeracy items except 9 and 10.
• Use the test administration grid. Record a capital C against the objective to 
indicate that the correct response was given. Write PC for a partially 
correct response, or W for a incorrect response, or a NR for non response. 
 
 - 42 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Hello, What is your name? 
How are you ____________? I am well too. How old are you? Good. 
Now, I am going to ask you a number of questions. Sometimes I will ask you to just tell 
me the answer, and other times I will ask you to write your answer down, OK? Now, 
here the first question: 
Question Stem Selection Instructions
1. Count from _____ 13-33; 21–41; 37–57; 41–51; 73–93; 35-
55; 70-90;29-49; 48-68; 11-31;
14-34; 25-55; 79-99; 62-82; 12-32
Select and present one of the 
items. Allow a second 
chance if necessary
2. Now write _____ 10 –99 Ask them to write any two 
digit numeral from 10-99 
(inclusive).
3. Count in twos from 2 to 10 Allow a second chance if 
necessary
4. Count in tens from 10 to 
100
Allow a second chance if 
necessary
5. ____ is the same as ___
e.g., 7+6 = 6+7 
7+6; 9+4; 6+7; 4+7; 5+5; 6+4; 9+8
9-4; 7-5; 8-3; 6-1; 9-2; 8-7; 7-3
 Ask them to complete the 
statement, presenting both 
addition and subtraction
6. Add the following 
numbers
7+6; 9+4; 6+7; 4+7; 5+5; 6+4; 9+8 Present in horizontal 
orientation on flash cards
7. Subtract ____ from ____ 9-4; 7-5; 8-3; 6-1; 9-2; 8-7; 7-3 Present in horizontal 
orientation on flash cards
8. A man has K___ and buys 
____ for K___ . How much 
will you give back?
K500-K200; K700-K300; K900-K600
K400-K200; K600-K100; K200-K100
Present a story involving 
money and giving change. 
Allow a second chance if 
necessary
9. What is the name this 
shape?
Present a circle, square, 
triangle, and rectangle in 
any order
10. Now, draw a circle; a 
square; a triangle; a rectangle
Conceal the shape and ask 
learners to draw them from 
memory
 - 43 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
I am now going to ask the remaining questions in English. Please don’t be worried 
because it is simple English, just like you always hear in the radio, when the Mrs 
Musonda tells you what to do. OK ? Here the first question: 
Question Stem Selection Instructions
11. What colour is this? red, white, blue, green, black, 
brown, yellow, and orange
Present all colours in any 
order.
12. What is this?/What is _____ 
wearing?
dress; hat; shoes; socks; shirt Allow a second chance if 
necessary
13. What time of the day do you 
get up/go to bed
morning; evening; night, or any 
specific time
Allow a second chance if 
necessary
14. What can you see in this 
picture? What are they doing?
Present the picture on the 
learning center scene
15. Can you see ________?
Is __________________?
Ask several questions about the 
picture to elicit “yes” or “no” 
responses.
16. How many children can you 
see? women? radios?
Use language associated with 
counting.
17. Which animal do you 
like/do you not like?
animal, food, friends, colour Ask both the “like” and “do 
not like” questions about any 
of the items
18. Where is the book? Under, in, on top of, here, there, 
on, in front of, 
Present question involving 
prepositions or adverbs of place
19. Here is one stick, and here 
are two _____.
book; hand; pen; leg Present two words involving 
plurals. Allow a second chance 
if necessary.
20. Read these letters A – Z; a - z Present any 5 letters of 
alphabet randomly
21. Read this word 
What are these letters?
box; man; boy; pen; pot; dog; 
cat; table; shoe; hand; pen; car, 
leg; door; pot; spoon
Present two words and ask 
them to read. If not, ask them 
which letters make up the word
22. Spell _______ man; boy; pen; pot; dog; cat; 
pen; car, leg; door; pot; spoon, 
red
Present two of these. Allow a 
second chance at reading if 
necessary.
 - 44 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix I: Test Administration Training Notes 
Numeracy Items (1 – 10) 
The numeracy items may be presented in English, or the learners’ local language, just as it has 
been done in the teaching of these skills. Proficiency in English language should not be allowed 
to inhibit performance of the task at hand. Each of the items is described below. 
Question 1 
In this item, we want to test the learner’s counting skill. In order to increase the chances 
of beginning the sequences where indicated, tell the learner to repeat the number range. 
But, if the child starts at one still and covers the sequence that you have presented, give 
full credit for this item. 
Question 2 
This item is meant to solicit information on whether the students can write a two digit 
numeral correctly. Some learners will reverse the numerals, for example, writing 15 
instead of 51. In that case, ask the learner to write 15. Then ask her/him again to write 
51. If he/she still reverses the numerals at this point, score the reversed item as an 
incorrect. 
Question 3 
This is a counting item, which also exhibits basic multiplication skill. Present the item 
using the language that is understandable to the learners. You can ask them to count in 
twos, or ask them to count “2 by 2”. Present the first 2 number of the sequence, and let 
them supply you with the remaining three number of the sequence. 
Question 4 
This is another counting item that exhibits basic multiplication skill. Present the item 
using the language that is understandable to the learners. You can ask them to count in 
tens, or ask them to count “10 by 10”. Present the first 3 number of the sequence, and let 
them supply y the remaining seven. 
Question 5 
This item test their knowledge of number facts, and also, basic reasoning using 
numbers. Present two items from the set, each item being presented horizontally. Ask 
the student to read the number sentence, and then solve it. While some learners will 
add abstractly, those who need to perform this task concretely should be allowed to do 
so. 
 - 45 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Question 6 
This item tests addition of single digit, which yields answers is in double digits. Present 
two items from the set, each item being presented horizontally. Ask the student to read 
the number sentence, and then solve it. Some learners will add abstractly, while others 
will perform the task using concrete objects. This is acceptable. If it is not quite clear to 
you whether the learner has mastered addition, present a third item. 
Question 7 
The item tests subtraction of one single digit number form the other, with answers in 
single digits. Present two items from the set, each item being presented horizontally. 
Ask the student to read the number sentence, and then solve it. Some learners will 
subtract one number from the other abstractly, while others will perform the task 
concretely. This is acceptable. Present a third item if you need to convince yourself 
whether learner has mastered addition. 
Question 8 
The item tests application of subtraction in a commercial setting. The item should be 
presented in a number story involving denominations of 100. Present two items from 
the set. Most learners will perform the subtraction problem abstractly, while others will 
perform the task concretely. This is acceptable. Again, you may present a third item if 
you need to convince yourself of mastery.. 
Question 9 
The item tests simple recall of names of shapes. All four shapes should be presented in 
any order. Please do not interpret the names of the shapes. 
Question 10 
The item tests a first level operation with properties of the four shapes. Conceal the 
shapes that you used for the previous item, and ask the learner to draw the shapes as 
you call them out. Precision in drawing is not important. However, the children need to 
clearly differentiate a square from a rectangle. Please do not interpret the names of the 
shapes. 
 - 46 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Literacy Items (11 – 22) 
All literacy items are to be presented in English, without any translation. Functional literacy at 
a very basic level of the English language is expected. 
Question 11 
The item tests recall of names of colors. All the 8 colors in the list should be presented. 
Question 12 
The item tests understanding of language and use of language to recall and name items 
of clothing. Present an opportunity to name at least two items, and also be sure that the 
learner understands what you were asking. 
Question 13 
The item tests understanding of language and use of language in a conversation about 
different time periods in a day. Present another opportunity if you need to be ascertain 
whether or not the learner understands what you were asking. 
Question 14 
The item tests understanding of language and whether learners can use the language to 
describe what they see in pictures. Use the picture from the Taonga Market scene. 
Present several questions so as to have a good indication of the level of mastery (none, 
partial, or full mastery). 
Question 15 
The item tests understanding of language and use of language to confirm or deny 
certain facts. Using the picture from the Taonga Market scene, present several questions 
so as to have a good indication of the level of mastery (none, partial, or full mastery). 
Question 16 
The item tests understanding of language and use of language associated with 
counting. Using the picture from the Taonga Market scene, ask several questions that 
would require the to count objects in context. 
Question 17 
 - 47 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
The item tests understanding of language and its use to express personal meaning using 
reference to the first person, and possessive pronouns. Present several questions so as to 
have a good indication of the level of mastery (none, partial, or full mastery). 
Question 18 
The item tests understanding of language and use prepositions and adverbs of position 
or place. Present several questions so as to have a good indication of the level of 
mastery (none, partial, or full mastery). 
Question 19 
The item tests turning singular words into plurals. Present several words from the list 
so as to have a good indication of the level of mastery (none, partial, or full mastery). 
Question 20 
The item tests recognition of letters of the alphabet. Present the letters at random first, 
and then consecutively if the learner is not able to recognize them at random. Present 
several strings, so as to have a good indication of the level of mastery (none, partial, or 
full mastery). 
Question 21 
The item tests reading of words with single syllables. Present several so as to have a 
good indication of the level of mastery (none, partial, or full mastery). 
Question 22 
The item tests spelling of words with single syllables. Present several so as to have a 
good indication of the level of mastery (none, partial, or full mastery). 
 - 48 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix J: Guidelines for Test Administrators
 - 49 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
Appendix K: Mentor Questionnaire 
Tick or write the appropriate answer in the space provided. 
SECTION A – BACKGROUND CHARACTERISTICS
Questions Instructions/
Coding
Q101. Your centre is situated in:
 1. Urban area 2. Rural area 3. Extremely 
remote (rural) area
__________
Q102. Your sex:
 1. Female 2. Male __________
Q103. How old are you? ___________________years? __________
Q104. How long have you been a mentor? ___________________years? __________
Q105. Which grades have you taught as a mentor?
 Grade 1 Grade 2 Grade 3 Grade 4 Grade 
5
__________
__________
Q106. a) What Grade level (s) are you currently teaching? _________________years?
 
__________
Q107. How many days of IRI mentor training have you attended? ___________ days __________
SECTION B – LEARNING MATERIALS
Q201. How helpful is the mentors’ guide to you?
1. Not helpful at all
2. It is helpful only for certain topics
3. It is helpful in the majority of topics
__________
Q202. What reference materials do you have apart from the mentor’s guide?
1 None.
2 A variety of reference material donated to the centre.
3 Some of reference materials / borrowed / improvised.
__________
Q203. What reading materials do you have for the students?
1. None.
2. Word charts and posters
3. A variety of reading materials donated to the centre
__________
Q204. How do you find the content in the mentor’s guide?
1. Most of the content is too difficult for me.
2. Some of the content is too difficult for me.
3. Some of the content is easy to teach
4 Most of the content is easy to teach
__________
Q205. Is your radio in good working condition? __________
 - 50 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
 1. Yes 2. No
Q206. How many broadcasts have you missed in the period between January to July 
2003, due to your radio not working well?
 1. None 2. One to three 3. Four to eight 4. Ten or 
more 
__________
SECTION C – ATTENDANCE AND ACHIEVEMENT
Q301. How many children have dropped out of the Grade since January, 2003? 
 1. None 2. One to three 3. Four to eight 4. Ten or more __________
Q302. What is the most common reason for the dropping out of school?
1. Most of those who dropped out had to go to work 
2. Most of those who dropped out had transferred to a regular school. __________
Q303. How much do you think the children are learning from the radio program?
1. Very little, far less than those in a government school
2. They learn a lot, but not as much as those that are in a government school
3. They learn a lot, just as much as those that are in a government school
4. They learn a lot, more than those that are in a government school
__________
Q304. How many of the children in Grade 3 can read and write in their local language? 
1. About 20% 2. About 40% 3. About 60% 4. About 80% 5. Close to 
100%
__________
Q305. What do parents say, about their children’s learning?
1. Parents say that children learn very little, far less than those in a government 
school
2. Parents say that children learn a lot, but not as much as those that are in a 
government school
3. Parents say that children learn a lot, just as much as those that are in a 
government school
4. Parents say that children learn a lot, more than those that are in a government 
school
__________
SECTION D – COMMUNITY PARTICIPATION
Q401. Do community members assist in the running of the centre? 
1. No, they do not assist at all. 
2. Yes, they do assist, but only a few times, after they have been asked 
3. Yes, they participate satisfactorily
4. Yes, they participate a lot, and are very useful
__________
Q402. Do you think your centre is successful? 
1. Not at all
2. Yes, but only mildly successful. It needs a lot of improvement 
3. Yes, it is performing adequately. It needs some improvement
4. Yes, it is extremely successful
__________
Q403. What will happen to the children if the centre closed down?
1. They will transfer to local school
2. Most of them would go to live with a relative in another town so they can 
attend school
__________
 - 51 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
3. It would be the end of their education, unless government builds a school in 
the community
Q404. Would you recommend the IRI program to any of your relatives?
1. No 
2. Yes, if there was no government or community school to attend
3. Yes, even if there is government or community school 
__________
Q405. Would you recommend your centre to any of your relatives?
1. No 
2. Yes, if there was no government or community school to attend
3. Yes, even if there is government or community school
__________
E. What has been the major success of the Taonga IRI programme in your community? 
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_______________________________________________________________________________
F. What has been the major weakness of the Taonga IRI programme in your community? 
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
_______________________________________________________________________________
G. What is the greatest threat to the IRI program your community? 
_____________________________________________________________________________________
_____________________________________________________________________________________
__________________________________________________________________
Appendix L: Focus Group Questions (for Community Members) 
 
Question 1: How did you receive information about the interactive radio instruction 
 - 52 – 
Evaluation Report, Grade 1, 2003 Interactive Radio Instruction (IRI) Programme
December, 2003 
 Program?
Question 2 a: What prompted you to open a learning centre? 
Question 2 b: What preparations did the community make for IRI? 
Question 3: How soon are you hoping to have a government school in your 
community, now that the government has declared free education? 
Question 4 How active have you been in IRI activities? 
Question 5: What personal support are you giving to the mentor? 
Question 6a: What are your major expectations about the IRI program? 
Question 6 b: What has been its major success? 
Question 7: What is the greatest threat to the IRI program in this community?