Southern Sudan Interactive Radio Instruction (SSIRI) IMPROVING ENGLISH LANGUAGE AND NUMERACY SKILLS OF PEOPLE IN SOUTHERN SUDAN AN EVALUATION OF THE PERFORMANCE OF RADIO-BASED EDUCATION FOR ALL (Rabea) PROGRAM IN SOUTHERN SUDAN FINAL REPORT February, 2011 2 ACKNOWLEDGMENTS The consultant acknowledges the support and assistance received in planning and organizing this assessment from the Education Development Center, Inc., and GoSS Ministry of Education, Directorate of Alternative Education System (AES) and all the SSIRI Program management team, notably from the program’s Monitoring and Evaluation Specialist, Mr. Isaac Musoke. I recognize the constructive comments on the first draft of the report received from Mr. Richard Trewby, SSIRI Chief of Party, and the project management team at the EDC Home Office in USA. Last but not least, I also recognize in a special way a select group of Sudanese young men who took part in this exercise and formed a strong team of research assistants without whom this exercise would have been a daunting task. The views expressed in this document are a summary of the field findings with the author’s analysis hence errors and omissions are expected. The presentation of the material in this work does not imply the expression of any opinion whatsoever on the part of EDC or the GoSS Ministry of Education, Directorate of Alternative Education System concerning the legal status of any area in Southern Sudan or of its people or authorities. Androga Avelino Said, Consultant February 10, 2010 This report was produced by the Education Development Center, Inc and was made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of EDC and do not necessarily reflect the views of USAID or the United States Government. Copyright EDC 2011. All rights reserved 3 ACRONYMS AES Alternative Education System ALP Accelerated Learning Program ANOVA Analysis of Variance BALP Basic Adult Literacy Program CGS Community Girls Schools CPA Comprehensive Peace Agreement DDR Disarmament, Demobilization and Rehabilitation Program EDC Education Development Center FM Frequency Modulation GoSS Government of Southern Sudan IRI Interactive Radio Instruction LV Learning Village NGO Non-government Organization Rabea Radio-based Education for All SPLA Sudan People’s Liberation Army SPSS Statistical Package for Social Scientists SSIRI Southern Sudan Interactive Radio Instruction TRG Technical reference Group 4 CONTENTS EXECUTIVE SUMMARY .......................................................................................................................... 5 1 INTRODUCTION ................................................................................................................................ 7 1.1 Rationale for the Performance Evaluation .................................................................................... 7 1.2 Purpose of the Evaluation ............................................................................................................. 7 1.3 Organization of this report ............................................................................................................ 8 1.4 Socio-political context of the Evaluation ...................................................................................... 8 1.5 The Southern Sudan Interactive Radio Instruction Program ........................................................ 9 2 METHODOLOGY ............................................................................................................................. 10 2.1 Study Design ............................................................................................................................... 10 2.2 Sampling strategy for learners participating in achievement tests .............................................. 10 2.3 Test Construction ........................................................................................................................ 11 2.4 Methodology for analysis of test scores ...................................................................................... 12 2.5 Other Data Collection Methods .................................................................................................. 12 2.6 Participation of Stakeholders ...................................................................................................... 13 2.7 Training of MoE Staff ................................................................................................................. 14 2.8 Limitations of the evaluation approach adopted ......................................................................... 14 3 EVALUATION FINDINGS ............................................................................................................... 15 3.1 Program coverage in Southern Sudan ......................................................................................... 15 3.2 Results of the English language and Mathematics testing .......................................................... 16 3.3 Reported benefits of Rabea ......................................................................................................... 18 3.3.1 Rabea learners report improved skills in English language and Mathematics .................... 19 3.3.2 Rabea facilitators report improved teaching skills .............................................................. 23 3.4 Attitudes towards Rabea ............................................................................................................. 26 3.4.1 Likes and dislikes about Rabea ........................................................................................... 26 3.5 Awareness of Rabea B program among MoE and Education NGO partners ............................. 30 3.6 Participation of MoE and NGO partners ..................................................................................... 31 3.7 Issues of concern regarding the implementation of Rabea ......................................................... 33 3.8 Factors contributing to success of the Rabea program ................................................................ 36 3 CONCLUSIONS ................................................................................................................................. 38 4 RECOMMENDATIONS .................................................................................................................... 40 5 EXECUTIVE SUMMARY Purpose and Methodology The purpose of this evaluation is to enable SSIRI obtain indicators on the performance of Rabea B, and any promising practices related to the program which could be used to improve the quality of its implementation. The evaluation utilized a mixed-methods approach employing English language and Mathematics achievement tests, focus group discussions, personal profile interviews with beneficiary adult learners and facilitators, and in-depth interviews with education officials and representatives from partner NGOs. The evaluation of the performance of Rabea was carried out in five purposefully selected counties: Juba, Torit, Wau, Jur River, Yei and KajoKeji. However, the learners who did the English language and Mathematics achievement tests were selected from Juba, Wau and Jur River. Key Evaluation Highlights 1. 306 randomly selected learners from 7 ALP centers in 3 counties did two sets of achievement tests in Mathematics and English language. A comparison of the learners’ scores by level of exposure to Rabea (or number of lessons completed) shows that the mean scores significantly improved with increasing degree of exposure to Rabea lessons. Learners who attended more than 7 lessons (categorized as highly exposed in this evaluation) registered statistically better mean scores in English language (90.1%) and Mathematics (99.5%) compared to individuals who had a relatively low exposure. 2. The introduction of Rabea at the ALP centers in Southern Sudan seems to be helpful to both the learners and their teachers. From the learners’ experiences, it is evident that individuals who have attended Rabea lessons are improving their English language and Mathematics skills. The skills improvement in English language relate to improved ability to use the right tenses, better pronunciation of English words, reading English, writing in English language, and improved ability and confidence to communicate with others in English. There were also reports on improved ability to apply simple Mathematics operations such as addition, subtraction, multiplication and division in daily life. 3. The report has evidence suggesting that Rabea is a popular program in the areas where this evaluation was conducted. Most of those interviewed indicated that they like the program because of the approach employed during class instruction which makes learning practical, fun and enjoyable. Learners and others interviewed said that they are willing to recommend the program to those individuals who are not currently exposed to it or attending Rabea lessons. 6 4. Although SSIRI has designated staff to monitor the implementation of IRI programs in each of the states in Southern Sudan, it ensures that AES and other education officials at the state Ministry of Education are at the forefront of the delivery of these programs. This evaluation reveals that the Education Officials have taken on a number of roles as far as the implementation of the program is concerned: (a) teaching of adults at ALP centers using Rabea; (b) participating in State and County SSIRI planning and review meetings; (c) distribution of Rabea materials to the Rabea centers; (d) identification and selection of teachers for training; (e) co-facilitation of SSIRI training workshops; (f) supervision of other SSIRI programs; (g) mobilization of adult learners to attend Rabea or other IRI lessons; and (h) ensuring coordination between EDC/SSIRI staff and the state education officials during the implementation of IRI programs. 5. The participants raised issues which they felt are affecting the successful implementation of Rabea in some areas, and therefore need to be brought to the attention of the program’s management. The issues emphasized most during the interviews in order of frequency of mention by the participants are: (a) Poor quality of radio signal - commonly mentioned during interviews in Kajo Keji, Yei and Wau County; (b) Downsizing and transfer of teachers (including Rabea facilitators); (c) Irregular attendance of classes by the adult learners. 6. A synthesis and analysis of the interviews indicates 4 factors responsible for the seemingly successful community acceptance and implementation of the program: (a) The time at which the program is broadcast on radio is convenient for adult learners; (b) The civic education topics or content covered; (c) The interactive nature of the program and use of radio as a teaching aid; and (d) Training offered to the facilitators 7. The evaluation team sought input from all individuals who participated in the interviews and focus group discussions on how to strengthen the implementation of the Rabea program. There were varied suggestions, but in this report we present those proposals which were emphasized most, and are closely linked to the most pertinent issues of concern in the program’s implementation. These include: (a) address the issue of poor radio signal; (b) train more teachers to cater for the gaps created as a result of teacher transfer and downsizing; (c) ensure better storage of radios by the facilitators and schools; and (d) set a program implementation approach which helps individuals who have problems attending regular lessons due to social or other problems to cope and benefit from the program. This might address the issue of drop-outs cited by the learners. 7 1 INTRODUCTION 1.1 Rationale for the Performance Evaluation Evidence-based planning is at the core of SSIRI’s strategy of improving the quality and implementation of its programs. Because of this philosophy, SSIRI has commissioned a number of studies employing one or more data collection methods to provide critical insights into the implementation of its programs throughout Southern Sudan. In this regard, SSIRI commissioned an internally-led assessment aimed at evaluating the performance of one its programs called Radio-based Education for All (Rabea) 1 which is the basis of this report. Secondly the SSIRI program wanted to obtain the story behind its activity monitoring numbers. More specifically, the program’s statistics show that about 10,000 individuals are currently enrolled in and attend Rabea lessons in adult learning (ALP) centers. A population-based survey carried out in 2008 established that about 350,000 adults consistently listen to Rabea programs at home. These figures however do not represent the whole story behind the Rabea program. For example, the numbers do not exactly show if there are indications that the primary objective of Rabea of improving English language and Mathematics skills of people in Southern Sudan is likely to be attained. 1.2 Purpose of the Evaluation This is a utilization-focused performance evaluation. Its purpose is to enable SSIRI obtain evidence on what is working as far as the Rabea B program is concerned. This evaluation should also enable the program know more about the factors facilitating or hindering the implementation of Rabea, as well as any promising practices related to the Rabea program which could be used to improve the quality of its implementation. The following objectives guided this evaluation: 1. Assess any changes in speaking, reading, comprehension, writing skills in English language of those who have attended Rabea B lessons; 2. Find out if the people who attended Rabea B lessons have attained basic Mathematics skills and are able to use them in daily life; 3. Obtain feedback from stakeholders on the implementation and effects of Rabea B; and 4. Collect information on what the participants (learners and Rabea B facilitators) think about the IRI Methodology used during the Rabea lessons                                                              1 USAID’s new evaluation policy (2011) explains that a “Performance Evaluation’ focuses on descriptive and normative questions; what a particular project has achieved (either at an intermediate point in the execution of a project or at the conclusion of an implementation period); how the project is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making”. 8 This report is a synthesis of findings obtained through an achievement test composed of English language and Mathematics test items, interviews and focus group discussions. The findings have been presented in alignment with the specific objectives set out above. However, the report covers other subjects in this introductory section including a brief on SSIRI focusing on the Rabea program. This information has been added to provide the reader with an overview of the current trends in the education sector in Southern Sudan which are relevant and justify the implementation of the Rabea program. 1.3 Organization of this report Section one, provides explanation for conducting this evaluation, spells out the purpose of the evaluation, and the context in which the evaluation has been carried out. Section two provides details of the methods and data analysis followed, a description of evaluation participants, and the limitations of the evaluation approach adopted. Section three presents the evaluation findings, and highlights some activity monitoring statistics related to the implementation of the Rabea program. Section three also provides the learners’ achievement tests results, self-reported benefits by an array of individuals participating in the Rabea program, their attitudes towards the program, issues of concern regarding the Rabea program, and factors perceived as being responsible for the success of the program. Section four provides the evaluation team’s deductive judgment of the performance of the program based on the findings. The final section of provides a set of actionable recommendations based on the evaluation findings and conclusions drawn by the evaluation team. 1.4 Socio-political context of the Evaluation In Southern Sudan as in any other post conflict state, education is seen as an important building block in rebuilding and consolidating peace. At the time of the Comprehensive Peace Agreement (CPA), Southern Sudan had one of the highest illiteracy rates in the world, modestly put at an average of 85% illiteracy rate (92% among women and 80% among men).2 Although there have been substantial efforts from the NGO sector to support improvement of access to quality education, UNICEF in its recent report on the status of education in Southern Sudan indicates that many children of school going age especially in the pastoralist communities do not have access to education. In recognition of education as an invaluable vehicle for peace and development, from the very start after the CPA in 2005, Southern Sudan Ministry of Education embarked on an Alternative Education System (AES) to provide learning opportunities for learners who have missed the opportunity to complete their basic education and those who have never joined basic education including SPLA and other security forces.                                                              2 Government of South Sudan Ministry of Education Science and Technology, July 2006. 9 The design of AES is to provide a foundation for a highly responsive and flexible non-formal system that should provide quality education and training for specific target groups who have had no access to quality basic education particularly as results of the war in South Sudan. The various programs under AES include: (a) Alternative Learning Program (ALP); (b) Basic Adult Literacy Program (BALP); (c) Southern Sudan Interactive Radio Instruction (SSIRI); (d) Community Girls Education (CGE); (e) Intensive English Course (IEC); (e) Agro-forestry (AF); and (f) Pastoral Education (PE). Rabea B which is the focus of this evaluation is on the IRI programs implemented under SSIRI. 1.5 The Southern Sudan Interactive Radio Instruction Program Started in 2004, SSIRI is a collaborative initiative implemented by Education Development Center, Inc (EDC) through the Ministry of Education with financial assistance from the United States Agency for International Development (USAID). The goal of SSIRI is to support efforts to improve access and quality of education in Southern Sudan and the three areas. The intended beneficiaries of SSIRI are primary school learners in Grade 1-4, primary school teachers, and individuals- youths and adults who dropped out of schools and have limited access to formal education. Three programs have been developed to meet the education needs of each of the groups: Learning Village, Radio-based Education for All or Rabea and Professional Studies for Teachers. Rabea is a series of interactive radio-based programs intended to teach English, improve literacy and numeracy skills of adult learners and youths who have dropped out of school. SSIRI together with the GoSS Ministry of Education produces and disseminates three Rabea programs: Rabea for Beginners (or Rabea B), Rabea Intermediate, Rabea Advanced. Rabea for Beginners 1 and 2 which was the focus of this evaluation is designed to help the people in Southern Sudan speak, read and write in English, and improve their Mathematics skills. Rabea also includes Civic Education on issues pertinent to the people in Southern Sudan such as the Comprehensive Peace Agreement (CPA), Demobilization, Re-integration, and HIV/AIDS. Rabea is based on the syllabus of the Adult Basic Education program of the Ministry of Education of Southern Sudan. Rabea lessons are transmitted via live FM broadcasts or pre￾recorded (onto CDs or in mp3 format) and are delivered with accompanying audio players (Digital Devices) to groups of out-of-school learners both youth and adults. Rabea, like any other IRI program is delivered in a participatory learning arrangement where groups of learners are led by a facilitator who is guided by a Facilitator’s Guide and a radio teacher. Rabea B also suggests civic education topics for the facilitator and learners to discuss after the broadcast. Rabea B1 (60 lessons) targets Accelerated Learning Program (ALP) of the Ministry of Education Alternative Education System level 1 learners, and Rabea B2 (60 lessons) targets ALP level 2 learners. 10 2 METHODOLOGY 2.1 Study Design The rationale for the study design adopted comes from the specific objectives of the performance evaluation of Rabea B provided by SSIRI and EDC, in consultation with the Directorate of Alternative Education Systems (AES) and a Technical Reference Group set up for the purpose of moderating this evaluation. A mixed-method approach was adopted for this evaluation, using a learning achievement measurement and qualitative data collection design. More specifically, the design followed entailed administration of a learner achievement test using a criterion-based sample of Rabea learners, conduct of focus group discussions with the same learners, developing personal profiles of selected learners and facilitators at six (6) Accelerated Learning Program￾ALP (or Rabea) centers in the counties of Juba, Torit, Yei, Kajo Keji, Wau and Jur River. As a part of the study design, the team conducted one-on-one in-depth interviews with selected state and county education officials, and representatives from partner NGOs. 2.2 Sampling strategy for learners participating in achievement tests At the ALP centers, the evaluation team only selected and administered English language and Mathematics achievement tests to individuals (female and male) who had attended at least one (1) Rabea B lesson. The selection process majorly targeted individuals in level one (1) or two (2) under the ALP program. It was however difficult to identify individuals in level 2 to participate in the tests. Therefore, the team opted to test only ALP level one learners. An individual was deemed eligible for the testing if they had attended at least one Rabea B lesson in the past six months. While selecting the individuals to take part in the achievement tests, the team screened for level of exposure (attendance) to Rabea lessons (for screening form see appendix 1). After a rapid analysis of the completed screening forms, the individuals were put in three categories of level of exposure: (a) low exposure (attended 1-3); moderate exposure (attended 4-6 lessons); and high exposure (attended over 7 lessons). It is worth noting that by the time of testing (September 2010), SSIRI had broadcast about 20 Rabea B lessons. The team aimed at identifying 300 individuals with a gender ratio of 1:1 for testing. However due to high interest expressed by the program participants, 318 individual qualified for testing but 306 completed the mandatory test sets of English language and Mathematics. Over one half (54.9%) of the individuals who completed the test were men. Most (60%) of them were of the age category 20-29 years. Most (61.3%) of the learners completing both tests had what the evaluation team classified as moderate exposure. 41.2% of the learners tested were from Torit County in Eastern Equatoria. 11 2.3 Test Construction The tests consist of group and individual tests. The group test (also called class test) had 10 questions for Mathematics (Section A) and five for English Language (Section B). Section A aimed to test computational skills, knowledge and application of Mathematics operations. Section B aimed to assess comprehension of instructions and writing skills in English. It also aimed to assess the general civic knowledge of the learner in regards to areas included in Rabea lessons. The group or class tests are all written tests Individual tests involved face to face interaction of the participant with the test administrator. The test consisted of two sections: Section A (Mathematics): consisted of 3 questions and Section B (English Language) had ten questions. The objective of Section A was to assess computational skills, knowledge and application of Mathematics operations in daily life while that of Section B was to assess speaking, reading and comprehension skills in English language. The overall purpose of both the class and individual tests was to assess and evaluate if learners have mastered basic skills in English language, as well as basic numeracy skills, and whether they could understand simple communication in English at the end of Rabea B. The guiding principle during test development was that assessment procedures should match the objectives of Rabea B hence the behaviors elicited from learners included recalling certain facts, as well as performing certain tasks. The two tables below present a conceptual map of the skills assessed. Group Test Subject Skills tested Tasks/questions Mathematics Computational Skills and knowledge 1,2,3, 4, 8, 9, Application of operations skills and knowledge 5,6,7,10 English Language Comprehension and understanding of Instructions 1,4, 5 Writing skills 1, General Civic knowledge 2,3, Individual Tests Subject Skills tested Tasks/questions Mathematics Computational Skills and knowledge 3 Application of knowledge and operational skills 1, 2 English Language Speaking 1,2,3,4,5 Reading 6,7,8,9 Comprehension 10 12 2.4 Methodology for analysis of test scores The scheme for awarding scores for the correct answers to each of the items in both Mathematics and English language tests is included alongside the testing instructions in the test papers appended to this report. The first step of analysis involved scoring of the learners papers by two Rabea facilitators in Juba. The individual item scores for each type of test (group and individual tests) were entered into an Excel Spreadsheet and exported to SPSS for analysis. The maximum score for each of the tests (English and Mathematics respectively) was 40 marks. During analysis, learner’s performance in each subject was assessed using score obtained in individual test, group (class) test and total score. The total score of each learner for each subject was obtained by adding their score in the individual test to the group test score and finally computed into a percentage score. Mean scores for each subject were computed and one-way ANOVA was used to determine significant differences in mean scores according to background characteristics and level of exposure (number of lessons attended). 2.5 Other Data Collection Methods Focus Group Discussions As indicated above, focus group discussions were a vital method in this evaluation. This method was only applied to selected individuals who had participated in the achievement tests. These individuals therefore had also attended and completed at least one Rabea lesson. The team developed a focus group discussion guide (appendix 2), which was used as a data collection tool. The evaluation team had planned to hold at least sixteen (16) focus group discussions with groups of women, men, and special groups like demobilized soldiers who are or have previously benefited from the Rabea programs selected from pastoralists’, predominantly Arabic pattern, urban and rural communities. However, due to time and other logistical constraints, the team managed only six (6) mixed focus group discussions, four in Juba and two in Wau. Despite the limitations in attaining the required sample of focus group interviews, the team was satisfied with the level and depth of discussion generated through the six groups. By and large, evidence from the six groups and other data sources shows consensus on certain aspects of the implementation and benefits of Rabea program. In-depth Interviews The assessment team conducted one-on-one in-depth interviews with selected state, county and payam education officials, at least one representative from partner a NGO in the state, and at least two Rabea facilitators- one center head and an actual facilitator. For these interviews, the team developed a standard interview guide (see appendix 3) and shared it with SSIRI and the TRG for input on the method to be used in administering them and analyzing data collected. A total of 30 key in-depth interviews were conducted for this exercise. 12 Education Officials, 4 NGO partners and 7 center heads and7 Rabea facilitators participated in this exercise. 13 The individuals were selected to participate in the interviews only if they had ever heard of and/or participated in the supervision and monitoring of the Rabea program, or training of Rabea facilitators. Personal Profiles The Technical Reference Group (TRG) agreed with the assessment team to document “success stories” of individuals who have attended or facilitated Rabea lessons. The learners were selected in consultation with the respective Rabea center facilitators. The team also used a snowball approach to identify individuals for profiling exercise. The team was also able to select suitable participants through peer recommendation on the most active and popular learners or facilitators. The evaluation team developed personal profiling interview guidelines for each of the two categories of individuals (see appendix 4). A total of 20 personal profiles were compiled including 13 Rabea learners and 7 facilitators. The purpose of the personal profiles was to help the team obtain, through self-reports, a clear picture of how being a part of the implementation of the Rabea program is changing or has changed the livelihoods of the selected participants. Personal profiles can be a strong indicator of the impact of an intervention at personal level and serve to show how best practices can be utilized in improving other people’s lives. 2.6 Participation of Stakeholders One important aspect which this study sought to achieve is the active involvement of education officials from the Ministry of Education at GoSS and State level. For this matter, SSIRI set up Technical Reference Group (TRG) under the guidance of GoSS MoE, Directorate of Alternative Education Systems (AES).The TRG comprised of staff from the SSIRI program, MoE staff with representation from AES, Departments of Examinations, Assessment and Evaluation, and Curriculum Development, and selected Rabea facilitators. The TRG under the guidance of a consultant developed the framework for the qualitative assessment of Rabea B. The things considered by the TRG framework included: adequacy of the study objectives; selection of data collection methods and analytical tools; study sites and data sources; and the study design, sampling and sample size necessary to meet two of the objectives of this assessment focusing on assessing their learning gains. The TRG held only one meeting- at the inception of this exercise. However the consultant for this exercise has shared the findings of the evaluation and the draft report with individual members of the TRG. This process helped the consultant develop and refine the thematic areas for the final report. The experience of involving education officials and Rabea facilitators was enriching as it helped the team appreciate better the context of implementation of Rabea, ask appropriate questions. While the TRG was a good experience, it was a little challenging maintaining the group throughout the entire time of the exercise. Consequently, the TRG had little input in the analysis of the data obtained during the assessment exercise. 14 2.7 Training of MoE Staff SSIRI also wanted to utilize this assessment to build the capacity of MoE staff in research. All those on the evaluation team including members of the TRG were trained on how to conduct Focus Group Discussions, one-on-one interviews using standard guidelines. The team also received additional training and guidance on qualitative data analysis and report writing. The evaluation team also used the training exercise to refine the draft data collection tools by receiving input from the participants. Overall, the training took three (3) days. On the last day of the training the team carried out a pre-test of the FGD Guides and In-depth Interview Guides in Munuki Payam with a selected group of adult learners. The purpose of the pre-testing exercise was to ensure that the tools are clear to the audience, but more importantly, to ensure that any cultural sensitivity and other issues pertaining to the wording, content, and meaning of the questions in the tools are addressed prior to data collection. 2.8 Limitations of the evaluation  The counties (Juba, Torit, Yei, Kajo Keji, Wau and Jur River) included in this evaluation were selected because of convenience. Therefore the findings presented in this report may not necessarily reflect the status of Rabea program in other states or counties where the program is implemented.  The evaluation team had planned to hold at least sixteen (16) focus group discussions with groups of women, men, and special groups like demobilized soldiers who are or have previously benefited from the Rabea programs selected from pastoralists’, predominantly Arabic pattern, urban and rural communities. However, due to logistical constraints, the team managed only six (6) mixed focus group discussions, four in Juba and two in Wau. Because of this limitation, the experiences and views expressed in this report may not apply to certain groups of individuals attending Rabea lessons.  Because of lack of a comparison group, it is not possible for the evaluation team to reliably infer the effects of Rabea programs through the English language and Mathematics achievement tests. Secondly, the tools used did not capture information on the education history of the learners. Therefore it was not possible to control for any confounding factors, and assess the actual contribution of the program towards the learners’ gains in English and Mathematics skills. 15 3 EVALUATION FINDINGS 3.1 Program coverage in Southern Sudan Rabea is broadcasted on fifteen (15) radio stations with coverage of thirty (30) counties in Southern Sudan (Appendix 5). Results from a 2010 radio listenership household survey commissioned by EDC and carried out in fourteen counties where SSIRI programs are implemented show that about 9 percent of the adult population (15 + years)3 or just over 100,000 individuals in these counties consistently listen to Rabea outside the formal classroom setting. SSIRI program activity monitoring data also shows that there are a considerable number of individuals who attend Rabea programs in a formal school setting with a facilitator. In 2010, the program recorded an enrollment of 9,885 adult learners registered in 139 ALP centers in 8 states in Southern Sudan and one state (Southern Kordofan) in the three areas. SSIRI expects to register 10,000 adult learners in 2011 and over 50,000 by the end of the project in 2012. These targets should be attainable because SSIRI has now rolled out to all the 10 states and expanded to more counties in states where the program was already under implementation. SSIRI records show a mix of results for the past three (Table 1). The number of ALP centers implementing Rabea B grew from 78 to 138 over the past two years. Consequently, the number of adult learners registered in groups grew from 7,990 in 2009 to 9,885 in 2010. However, as shown in the statistics, in 2009 the program registered a drop in the number of enrolled learners compared to 2008. In 2009SSIRI did not register individuals listening to Rabea B in informal radio listening groups as was the case in the previous years. The program has since ceased registration of these individuals because the activities of the groups were unpredictable and it was not feasible for the program to make reasonable follow-up of individuals who benefited from the IRI program through this type of setting. Table 1: Number of Rabea centers and registered learners 2008-2010, source: SSIRI PMP Sta te Ce ntre s Enrolm e nt Ce ntre s Enrolm e nt Ce ntre s Enrolm e nt Central Equatoria 3,544 57 27 3,979 6,771 61 Eastern Equatoria 2,426 34 2 116 5 258 Jonglei 18 945 Unity 1,136 12 - - - Upper Nile 1,136 8 9 1,271 20 971 W arrap 1,136 6 - - - W estern Equatoria 356 9 22 979 3,815 71 W estern Bahr el Ghazal 13 1,235 18 1,645 1,916 18 S outhern K ordofan - - - - 1 75 Gra nd Tota l 139 9,885 78 7,990 14,751 194 2010 2009 2008                                                              3 Estimated adult population in the 14 counties is 1,149,319 (Southern Sudan Population Census, 2008) 16 3.2 Results of the English language and Mathematics testing Learner characteristics 306 randomly selected learners from 7 ALP centers did two sets of achievement tests in Mathematics and English language (Table 2). The random sample was made up of slightly more males (54.9%) and individuals aged 20-29 years (60%). Slightly less than one third (28%) of those who participated in the tests were above 30 years old. These statistics suggest that individuals who attend ALP centers and Rabea lessons could be mainly males and individuals aged 20-29 years old. The learners who did the achievement tests were selected from 3 purposefully selected states namely: Central Equatoria (99), Eastern Equatoria (126) and Western Bahr el Ghazal (81). As indicated in Table 1 below, the learners in Central Equatoria were selected from ALP centers in Juba County while for Eastern Equatoria they were selected from Torit County. In Western Bahr el Ghazal the learners were selected from ALP centers in Wau and Jur River counties respectively. The relatively bigger (30%, n=93) proportion of learners was selected from Police ALP Center in Juba. Table 2: Participating ALP Centers and number of learners tested ALP Centre Location (County) Num ber of te ste d le a rne rs Pe rce nt (%) Baryar ALP Center W au 33 10.8 Dr John Garang W au 48 15.7 E ast B ank A LP Center Jur River 18 5.9 Juba central Prison Juba 33 10.8 Munuki Adult Centre Juba 48 15.7 Police ALP center Juba 93 30.4 Torit one Torit 33 10.7 Tota l 306 100 Figure 1 Figure 1 shows that just over one half (53%) of the individuals who did the achievement tests (N=306) had attended 4-6 Rabea B lessons, and categorized in analysis as moderately exposed. Over one out of four (26%) of the individuals tested had completed seven or more lessons out of the 20 lessons which had been broadcast by the time of the evaluation. This finding could either indicate that there is inconsistent use of the radio lessons in the ALP centers or that many of the learners do not attend Rabea B lessons on a regular basis. 17 Performance of learners by background characteristics Table 3 below shows that the male learners’ mean score (83 percent) in English language test was significantly (p<0.05) better than that of their female counterparts (76 percent). However, the females (83 percent) registered a slightly better performance than their male counterparts (77 percent) in the Mathematics test. The results summarized in the table also show that the learners’ performance in Mathematics and English language respectively was not statistically different (p>0.05) across the three states of Southern Sudan from where the learners who participated in the tests were selected. There was also no significant difference in English language and Mathematics mean scores among learners in the various age categories stated in the table. Table 3: Performance of learners by background characteristics Learner's profile N Mathematics English Mean score (%) Std. Error Std. Deviation Mean score (%) Std. Error Std. Deviation Gender Female 138 82.971 1.8657 21.9181 76.087 1.6858 19.8040 Male 168 77.083 2.2676 29.3921 83.441 1.5566 20.1769 F=3.8 p=0.052 F=10.2 p=0.002 Age in complete years <20 36 70.833 5.2861 31.7167 81.818 3.8671 23.2028 20-24 90 80.555 3.0627 29.0561 80.909 1.8510 17.5609 25-29 93 78.494 2.6916 25.9575 77.419 2.1915 21.1344 30-34 81 84.567 2.3199 20.8795 81.144 2.3623 21.2607 35+ 6 75.000 3.7267 9.1287 86.363 6.0983 14.9378 F=1.8 p=0.121 F=0.689 p=0.594 State Central Equatoria 99 83.838 2.1330 21.2231 87.327 2.0712 20.6088 Eastern Equatoria 126 84.127 2.1566 24.2080 77.705 1.5590 17.5002 Western Bahr el Ghazal 81 67.901 3.5109 31.5983 75.084 2.4209 21.7887 F=11.9 p=0.000 F=10.2 p=0.000 Performance of learners by level of exposure to Rabea When a comparison of the learners’ scores is made by level of exposure to Rabea (or number of lessons completed) it is clear from table 4 below that the performance in Mathematics and English language significantly improved with increasing degree of exposure to Rabea lessons. Learners who attended more than 7 lessons (categorized as highly exposed) registered statistically (p<0.05) better mean scores in both English language (90.1 percent) and Mathematics (99.5 percent) than individuals who had a relatively low exposure. This suggests that the more lessons a learner attended, the better is his/her performance in Mathematics. This finding is consistent with other IRI studies which show that learners who attend more IRI lessons perform better than their counterparts who attend less. 18 Table 4: Performance of learners according to level of exposure to Rabea lessons Number of lessons attended N Mathematics English Mean percentage scores (%) Std. Error Std. Deviation Mean percentage scores (%) Std. Error Std. Deviation 1-3 (Low exposure) 64 40.625 3.4197 27.3579 62.642 2.7547 22.0381 4-6 (Moderate exposure) 161 85.300 1.0829 13.7410 82.044 1.3639 17.3070 7+ (High exposure) 56 99.588 .2891 2.6024 90.123 1.7141 15.4276 F=262.2 p=0.000 F=43.8 p=0.000 3.3 Reported benefits of Rabea The results of the achievement tests in English language and Mathematics show that the more Rabea lessons a learner attended the better they performed, suggesting that Rabea B lessons can have an effect on the performance of a learner. However because of the limitations of the study design followed in conducting the achievement tests, such as lack of a treatment group and possibility of learner ‘contamination’ through exposure to similar education programs, it was necessary to use additional data collection methods to try and triangulate the results of the achievement tests. The methods employed for this purpose as already mentioned in the methodology section included: focus group discussions with a selected group of learners who did the achievement tests; in-depth interviews with individuals who trained to facilitate Rabea B lessons on a regular basis, education officials and other individuals from partnering organizations who participate in the implementation and monitoring of SSIRI programs especially Rabea; and personal profile interviews with selected learners and Rabea facilitators . The qualitative evidence obtained through these additional data collection methods to a large extent corroborates the results of the English language and Mathematics achievement tests which indicate that when an individual attends Rabea lessons, they improve their English language and Mathematics skills. Through these additional data collection methods, the evaluation team also obtained evidence suggesting that the training provided by SSIRI on how to use IRI has improved the skills of the Rabea facilitators. The self-reports provided by the facilitators show that they have specifically improved the way they teach, interact with and involve the adult learners during Rabea lessons. 19 3.3.1 Rabea learners report improved skills in English language and Mathematics Interview and focus group discussion items were developed and used to elicit responses from Rabea learners, facilitators, and education officials on whether the Rabea B program has made a difference among individuals and the community. The reports obtained through the interviews and focus group discussions suggest that the program has majorly made a difference among individuals who attend lessons or listen to radio broadcasts. The experiences shared suggest that as a result of being exposed to the program, the learners have improved their English language skills. The skills improvement in English relate to improved ability to use the right tenses, better pronunciation of English words, reading English, writing in English language, and improved ability and confidence to address public gathering in English or communicate with individuals in English. On the other hand, some also shared experiences suggesting that they have improved their Mathematics skills. Most prominently, there were reports on improved ability to apply simple Mathematics operations such as addition, subtraction, multiplication and division in daily life for example when making a purchase in the market and taking correct measurement when doing tailoring. Below we share some of the experiences in the narrative and interview excerpts below. Despite being in level 4 (four) at Leikor ALP Centre in Kangapo I Payam, KajoKeji County, Samuel Alhai a 48 year old adult learner told the evaluation team during a personal profile interview that he still gets time to listen to or attend Rabea lessons on the three days when the radio (Rabea) lessons are broadcast and taught at the centre. Samuel told us that his life has changed since he began to attend the radio lessons two years ago. When sharing his experiences about the program he told the team that apart from developing the ability to correctly use the tenses in English language, he is also able to hold formal conversations in English. “My life has changed since I started attending the radio lessons. I have known many new things from the radio. English is what I like most from the radio lessons. I have learnt how to speak English and how to use the present, past and continuous tenses and now I can use them properly. I am proud my English [language skills] has improved and I am now able to hold conversations in English. If I had not attended these lessons I think my life would have remained the same like in the past. But now I can speak better English”. Just like Samuel above, Rebecca Keji Felix, a 40 year old mother of five who attends the Accelerated Learning Program at Mundari SPC centre in Lire Payam, KajoKeji County of Central Equatoria state, told the evaluation team during a personal profile interview that attending Rabea B lessons has helped her improve her use of tenses in the English language. Rebecca said that because she attends Rabea lessons she has learnt not to mix tenses when she speaks in English. 20 “… One of the things that I have learnt from Rabea that I will never forget is how to use the right tenses. I used to mix tenses when speaking English. I could mix present and future tenses. Nowadays I know that “was” is used in the past and “is” is used in the present. ... I liked the English language segment of Rabea. I knew some few English words and realized that I could learn more English from the radio [Rabea] lessons”. Furthermore, during the personal profile interview, Rebecca also said that she is now able to read her letter written in English, and speaks to her children and husband in English and engages in English conversations with her neighbors. “I usually tell my friends to look at me as an example of the advantages of radio lessons. I tell them that in the past I did not know how to read but now I can read. I used to ask my husband or neighbors to read my letters but now I can read and write my own letters.” She also said that the radio lessons (Rabea) have also made it easier for her to learn Mathematics, and explains how the Mathematics segments of Rabea have made her market days easier and enjoyable. “On the radio we are taught with examples. For example in today’s lesson in the Mathematics segment we learnt how to multiply. We were told that 20 X 2 is 40. Because I was hearing it from the radio, I learnt it easily. If anybody asks me the same question next time, I will remember easily. Because of the Mathematics lessons, I now feel confident that I cannot be cheated when I go to the market to sell vegetables over the weekend or when I buy from the shop”. Buga James is Payam Education Supervisor in Lire Payam, KajoKeji County. Buga is a very active education official who has been actively involved in the monitoring and supervision of the program. Buga shared his personal experience on how Rabea has benefited the learners in his County. “Rabea has improved the learners’ level of understanding particularly, they are able to speak English and they are able also to add numbers. The best example is my wife who is one of the learners at one of the centers using Rabea. When she reaches home she is able now to speak some words in English and able to add some numbers and willing to ask me questions about Mathematics. And I think this is a very big achievement”. Just like Buga above, other education officials who have actively participated in the monitoring and supervision of the implementation of AES programs including Rabea also reported observing changes in English language skills among individuals who attend Rabea lessons. As one of them indicated during an in-depth interview, government officials have been one of the prime beneficiaries of the program. They have improved their skill in speaking of the English language because of the time they have devoted to the Rabea lessons. 21 “…… spoken English has improved especially among the working [government] officials who attend Rabea lessons. They speak good simple English. Before the introduction of Rabea in the ALP centers those working [government] officials of Arabic Language background could not speak in English but now they do”. [In-depth interview with education official, AES Department, State Ministry of Education, Torit County] The experiences gathered by the evaluation team through the personal profiles interviews with Rabea facilitators suggest that their role as facilitators (the teachers of Rabea lessons) has improved their English language and Mathematics skills. For example, Arkanjelo Kur Majok a facilitator at Baryar ALP center, Jur River County, Western Bahr el Ghazal says that the process of preparing for and instructing a Rabea lesson has had a positive effect on his skills as he explained during a personal profile interview: “It [Rabea] has improved my English and Mathematics skills. Before going to class, I have to prepare well with the teachers guide and know what I am going to teach. This has helped me master good skills in Mathematics and English. Also as I practice with the learners I have become better”. The learners who participated in the focus group discussions in Juba, Wau and Jur River County seemed to collectively agree that the Rabea lessons have helped members like them improve their English language and Mathematics skills. Evidence collected during the discussions in the focus groups suggests that there are promising indicators among learners and the community at large that the program is on track in terms of improving the English language and Mathematics skills of people in Southern Sudan. “We like the radio program [Rabea] and the teachers teach well. It has helped many of us learn many things in Mathematics and how to say certain words in English. Although the subjects here in Level 1 are taught in Arabic and our local language, we are lucky to start learning English with the radio before Level 2. In level 2 we do not trouble learning in English. But I have to say, people in this Boma [village] see that those who attend radio lessons here speak English, read newspapers in English, and we also try to teach [English] to our family and friends. Although you do not get everything perfect, but you are better than those who do not attend the radio lessons”. [FGD with mixed learners, John Garang, ALP Center, Wau] However, there were comments during the focus group discussions which suggest that only individuals who are consistent in attending Rabea lessons at the ALP centers actually benefit from the program. Many, they added, dropped out of the program due to family and community obligation. These they said, find it difficult to cope with the lesson schedule and have to start all over again. And, as one of them indicated, “…at the centers the teachers and learners tend to ignore radio lessons and focus on those subjects that have text books for English and Mathematics, and teachers give us notes”. 22 Evidence from the key informant interviews with education officials suggests that the changes in English language and Mathematics skills reported by the learners and facilitators above are occurring and are discernable. “Yes, exposure to Rabea learning series has improved the Mathematics and English Language skills of very many people [in Wau, Western Bahr el Ghazal]. Before attending the program they could not calculate numbers well. But now they do addition, subtraction, multiplication and division very well. They also make good communications in simple English. Some of the learners are farmers, others are traders and others are messengers in offices. The messengers understand instructions in English better that before they enroll in the Rabea program. Some people could not sign using the pen before the program, but now most of them sign on the payment sheets using pens other than using figure prints. I think this is an achievement.” [Key informant interview with AES Director, State Ministry of Education, Wau, Western Bahr el Ghazal] On top of being able to make simple Mathematics operations and read, write and speak English, there are indications from the interviews conducted that Rabea learners are now also able to observe simple etiquette such as exchanging greetings (and greeting others in a humble way), and are more knowledgeable about pertinent civic issues in Southern Sudan such as the CPA and DDR issues. “…some of the learners in the Rabea centers are now able to add, multiply and subtract numbers. Because of attending the radio lessons people’s skills have improved in terms of counting piaster, pounds, and knowing how to use place values etc. They are also able to read, write and speak simple English. They can express themselves in simple terms such as exchanging simple greetings”. Because of attending Rabea classes, learners in accelerated learning program (ALP) centers are now good listeners and speakers. Sometimes the learners hear the instructions better than the facilitators and even correct the facilitators. Some of the learners have also adopted some of the language of communication taught in the Rabea lessons such as how to greet humbly. Also learners in Rabea centers now know more about current affairs issues, which they did not know and had never heard about previously. For example, they are aware of the CPA, DDR issues”. [Key informant interview with Payam Education Supervisor, Kangapo II, Kajo Keji County] One the education officials interviewed also added that on top of being able to improve the learners’ awareness of the pertinent civic issues in Southern Sudan, the Rabea program has influenced the level of community dialogue around these issues among people in his community. During the interview he specifically pointed out that the program encourages individuals to seek more knowledge on the civic issues discussed during the lessons. “The civics lessons in Rabea have helped the learners because when they go home they share anything they have learnt. Rabea also encourages people to seek more knowledge when they leave the centre. Now they have that ability to exercise political rights in society because of the civic education through Rabea”. [Key informant interview with Payam Education Supervisor, Kangapo II, Kajo Keji County] 23 Another education official gave his experience with the program and explains its benefits to the learners: “I can say exposure of people to Rabea learning series has changed their Mathematics and English language skills. Some people did not know how to write or speak in English but now most of them do. Where I was teaching in Jur River Rabea center there were no any other ALP lessons being conducted. But now these learners can read and write English. One of the learners who were in my former center is now working in the market. She is selling food items such as dried fish, vegetables etc. She is managing her money well and she gives balances of money correctly without difficulties. Some workers in the Jur River County were signing for their salaries using figure prints. But today the situation is different. They sign for their salaries on the payment sheet using pens like other educated officials. This is because Rabea program has impacted on them positively. I consider this as a success. Furthermore, other learners talk about HIV/AIDS, the right of women, human right, the CPA, the role of a citizen in the community etc. These are topics which are covered in the Rabea B1 lessons”. [Key informant interview with County Education Inspector, Jur River County, Western Bahr el Ghazal] 3.3.2 Rabea facilitators report improved teaching skills The Rabea facilitators receive training on how to use radio instruction, and are expected to teach at adult learning centers employing the interactive radio instruction skills obtained. The evaluation team obtained feedback on the benefits acquired by the Rabea facilitators as a result of their participation in the implementation of the radio-based instruction program. Those who provided feedback on the benefits of Rabea to the facilitators included Rabea facilitators, education officials and individuals from partner NGOs. The Rabea facilitators told the evaluation team about their personal experiences with the program, while the other category of respondents provided more of anecdotal evidence on the benefits of IRI training to the trainees (or Rabea facilitators). All the ten (10) facilitators interviewed told the evaluation team that they had completed SSIRI’s Rabea facilitators’ three-day training. All indicated that they had also been invited for at least one refresher training during the past two years. When asked to share their experiences and what they thought about the training provided by SSIRI, all said that they liked the training program. Asked why they liked the training, all indicated that they liked the training majorly because of the active-learning instruction approach employed by SSIRI staff during the IRI training as one of them explains: “The facilitators [SSIRI trainers] are very practical; I like their role play and involvement of the learners. By doing a practical work during the training it made it easy for me to [develop] the skills of operating the radio and how to follow the radio instructions”. [Personal profile interview with Rabea facilitator Munuki Central, female, 62 years, Juba County] 24 There are indications from the interviews that training on how to facilitate Rabea lessons and teaching experience accorded to the Rabea facilitators is making a positive change in their teaching skills. Because of the training offered by SSIRI, most of the individuals interviewed told the evaluation team that they are benefiting from the training. Notably, they said that they now speak better English, have better classroom management skills especially during lessons with adult learners, and they have improved their listening skills. “If I compare my teaching skills now and before I was trained as Rabea facilitator, I see much difference. As I mentioned before my English has improved, I can now manage and control the class without problem”. [Personal profile interview with Rabea facilitator Torit County] “It [the training on how to use radio during classroom instruction] has actually put me in a position to handle the adult learners very well. It has changed my teaching skills. My listening skills have improved too and the program is good.” [Personal profile interview with Rabea facilitator Torit County] Joseph Diardit 22, a facilitator at Nazareth ALP centre in Wau South Payam, Wau County explains how the IRI training has been of help to him. Joseph’s experience briefly highlighted in the excerpt below suggests that the training on how to use IRI (Rabea) has on top of improving his skills as a teacher of adult learners, enabled him to gain other skills such as: the ability to use a “learner-centered and participatory approach” during classroom instruction; ability to use teaching aids including the radio itself; and improved his listening skills. “Indeed the training conducted by SSIRI has improved my skills as a facilitator of adult learners. Before this training I only knew the teacher-centered method of teaching. [In this method] the teacher does everything and it is sometimes difficult to know whether the learners have understood the lesson or not. Through the SSIRI training I came to understand the learner￾centered method of teaching where the facilitator and the learners are actively involved in the learning process. I enjoy this participatory teaching method. I also learned about many teaching aids including the radio itself. Before I did not know that a radio can be a teaching aid but now I do. My listening skills have also improved”. The skill which was reportedly gained and mentioned most by majority of the facilitators is the ability to use a “learner-centered and participatory approach” during classroom instruction. This finding corroborates the IRI training objectives of SSIRI which largely empower Rabea facilitators to create an effective adult learning situation in the classroom. A “learner-centered and participatory approach” during classroom instruction is a vital part of this classroom environment. By using this approach, the Rabea facilitators allow active involvement, interaction, and socialization of the learners during the lessons. In support of the above assertion, the Outreach Advisor and trainer for SSIRI in Eastern Equatoria, Mr. Ale Peter Michael, emphasizes that Rabea lessons should serve their purpose and be provided in ways that allow learners to do more than just receive information. 25 Additionally, the teaching and classroom experience of the Rabea facilitators has had an effect on their adult teaching, listening and writing skills, as well as their class control and management ability as indicated in the following interview excerpts. “Yes, it has really improved my teaching skills as a facilitator of adult learners. The learners participate actively in the lessons. In the ordinary class the pupils do not take part in the lesson actively and they are not very much involved. The learners become very attentive and they do not want to miss an instruction from the radio teacher. Because of this even class control and management have become much easy.” [Rabea facilitator, Wau] “I like the program because it has developed my listening skills and writing skills. It even saves time. It makes the lessons participatory and enjoyable. It actually put me in a position to handle the adult learners very well. It has changed my teaching skills. My listening skills have improved too and the program is good.” [Rabea facilitator, Torit] For most s, as explained by one of them during a personal profile interview, being a facilitator of Rabea at ALP centers has enabled them to develop better lesson preparation and review skills since they are required to this as a part of their Rabea lesson preparation exercise: “My skills have been changed as a facilitator of Adult learners. We do not use to revise what we teach in the ordinary lessons. But the radio training has taught me how to revise the lesson after the broadcast each time I teach.” [Rabea facilitator, Torit] For some, being facilitators of Rabea lesson has improved their English language skills, as one of them indicated; “…because of my role as a Rabea teacher, my English is improving daily. Someone may think that I am a secondary school leaver but I am not. I only finished P.8”. Michael Adier Kuol, 27 a Rabea facilitator at Lomuku Adult Learning Centre in Yei County of Central Equatoria state was trained on how to facilitate Rabea lessons in 2008. He told the evaluation team during a personal profile interview that the IRI training provided by SSIRI added to his teaching skills, and he is a more confident teacher compared to how he was before the training. “Today I am able to teach freely without any fear. I can interact freely with the learners. I usually ask the learners questions and they reply by giving answers. Without the IRI training I would not be a confident teacher. Now I am a better teacher than I used to be in the past. In the past I was just gambling. I was not sure about my presentation skills. Now I am confident when I am teaching both radio lessons and normal lessons. I have now become a more active teacher than I used to be when teaching in class because of the training on IRI methodology than what I used to be. Secondly, the content of the training was really up to date on how to implement IRI lessons”. 26 Key learning point The introduction of IRI (Rabea) in adult learning at the ALP centers seems to be helpful to both the learners and the teachers who facilitate the lessons. Among the learners, we see patterns of reported experiences suggesting that individuals who have attended Rabea lessons are improving their English language and Mathematics. However, this evaluation presents findings which suggest that the more lessons the learners attend the better they are likely to perform. In the findings we also see that there is need for some flexibility in the delivery of the Rabea lessons. There is need to set an approach which helps individuals who have problems attending regular lessons due to social or other problems to cope and benefit from the program. This might address the issue of drop-outs cited by the learners during one of the focus group discussions. 3.4 Attitudes towards Rabea Three thematic areas emerged from the data obtained through interview and focus group discussions on the attitudes among learners, education officials and other stakeholders towards Rabea. The emergent themes explored here are: (a) Likes or dislikes about Rabea; (b) whether individuals who are currently attending or teaching Rabea would recommend the program to others especially their peers; and (c) whether individuals who are currently attending or teaching Rabea ever talk about the topics discussed in the program or the program itself to others in their community. 3.4.1 Likes and dislikes about Rabea Individuals who are currently attending or teaching Rabea either like or dislike certain aspects related to either the actual programming or technical characteristics of Rabea or the way the teachers (or facilitators) use interactive radio instruction in the classroom. Most importantly, evidence from the interviews suggests that Rabea is liked because it has the ability to arouse learning interest among learners and it improves listening skills among learners. Comments on the technical aspects of Rabea Evidence from the focus group discussions with Rabea learners and in-depth interviews with Rabea facilitators and education officials suggests that the learners who attend Rabea lessons, facilitators who use Rabea during classroom instruction and the education officials like Rabea and find the program interesting. Most of them indicated that they like the program because of “I also think the interactive nature of the radio lessons has facilitated the success of Rabea… Rabea encourages learners to participate equally in class. The use of radio as a teaching aid is also good because it motivates learners. The radio broadcasts attract learners and motivates them to be attentive in class and present for every lesson”. [In-depth interview, Payam Supervisor, Kajokeji] 27 the activities embedded in the program as explained by one of the participants during a focus group discussion in Wau County. “This program is excellent both for learners and the community members. The program has got very interesting activities and fun, that is why we like it very much and I like it myself too”. [Focus group discussion, mixed group, Wau County, Western Bahr el Ghazal] Even some individuals who listen to Rabea from their homes like the program because, as one male focus group discussion participant in Wau County said “…the lessons are presented in a very simple way, uses simple English during the radio instruction sessions and anyone can easily follow the lessons without a teacher”. There was also evidence that some respondents like Rabea because it is broadcast in the evening and the lessons take place at a favorable time when people are back from their “work places”, a timing which makes the program readily accessible to the working class in the community. Most said that they like Rabea majorly because of the content covered during the lessons especially the civic education topics which they find beneficial to the learners. “In fact it [Rabea] is very interesting especially that part of civic education, human rights, and land ownership, rule of law, Mathematics and of course not forgetting English Language. If we put into practice whatsoever we learn from the Rabea Program, we will benefit a lot. People will get to appreciate sending the children to school because they would have understood the importance education. They will send all their kids to school both boys and girls. They will also know the rights as citizens and know their obligations are in regards to the development of their nation.” [Focus group discussion, mixed group, Juba County, Central Equatoria] “I like the civic education especially those topics which cover HIV/AIDS, Demobilization, Re￾integration into the communities, and most importantly the Comprehensive Peace Agreement that brought peace and hope for the people.” [Key informant interview with County Education Inspector, Jur River County, Western Bahr el Ghazal] “I like the Rabea program because it brings other topics like civic education whereby they sensitize the community and it helps the learners to know the environment and society”. [Focus group discussion participant, female, mixed group, Wau County, Western Bahr el Ghazal] Others indicated that they like the program because of other reasons including the fact that the topics covered in the program are contemporary and relevant to the people of Southern Sudan, as Joseph Maring, a project manager with Windle Trust International in Western Bahr el Ghazal indicated: “I like the topics covered in this program because they reflect the context of Southern Sudan”. Some of those interviewed indicated that they like the program because it is easy to follow and has the potential to attract individuals who dropped out of schools to pick up lessons at home or better still join formal classroom teaching at the ALP centers; 28 “What I like about the Rabea for Beginners series is that when the program is put on and when the learners listen and when a teacher teaches the same topic the learners will understand easily. Again the RABEA for Beginners’ series helps those who are not at school but are listening at home to learn something. The series also draws the attention of those who are at home to come to school”. [Key informant, Education official, Kajo Keji County] “I like the RABEA for Beginners series because it is a good program. We see that the lessons are encouraging other learners who had dropped out of schools to return to school. The radio lessons motivate dropouts to return to school”. [Key informant, Rabea facilitator, Jur River County] Comments on classroom instruction by facilitators SSIRI ensures that all individuals who intend to use Rabea during classroom instruction are trained as IRI instructors before they start using the radio-based lessons. They undergo a four days training. The IRI training programs help teachers improve their teaching by exposing them to the principles and practices of learner-centered, learner-friendly education, and use of radio and related paraphernalia as learning aid. The IRI aspects emphasized during the IRI teacher training even for the Rabea facilitators should reflect in the way the trained teachers conduct their lessons, the overall participation of learners during the lessons as well as discernable improvements in learner performances. Overall, as indicated in the previous sections, IRI teachers or Rabea facilitators are expected to stimulate learner interest and participation during classroom lessons. Evidence collected suggests that learners like the way the Rabea facilitators conduct their lessons. The evidence emphasizes that the most likeable aspect about the facilitators is their ability to involve all learners present and make Rabea lessons as interactive as possible. “What I like about the facilitators is that though they are not qualified teachers they deliver well the information to the adult learners, one good thing is that they reinforce what the learners have heard on the radio. One of the things I have seen with facilitators is that they are friendly to the learners. This encourages the learners to participate in the classes. Secondly, most of them are always ready for the lesson because they read the Teacher’s Guide before the lesson. Thirdly if there are any gaps at the end of the lesson they always fill up these after the broadcast (by doing the after broadcast activities)”. [Key informant, Education Official, Yei County] Others, like one participant in a focus group discussion in Juba said that they like the facilitators because of the way they conduct Rabea lessons. More specifically, the evidence collected shows that the facilitators make sure that they are audible enough for every learner in the class to hear what they are saying, are always well prepared for the lessons, have good classroom management skills, and give equal opportunity to all learners to participate during the lesson. 29 “I like the way facilitators teach Rabea. They make sure they talk at the top of their voice very loudly so that the learners will understand all that is said. The facilitator would like to make sure that the learners get him correctly because there is no time for repeating statements. The facilitator who repeats instructions misses out on the instruction given by the radio teacher”. [Focus Group Discussion, Juba] “The facilitators make themselves ready. They make sure that the class is quiet and no unnecessary movement or noises are made. After the radio lesson is finished, the facilitator gives chance for the learners to ask questions about what they have not understood during the broadcast to know the meanings of the new words. This does not happen in the ordinary class room teaching. I like this very much.” [Focus Group Discussion, Juba] “What I like most is the introduction. It is spelt out clearly and also if there is a new word, the facilitator defines it clearly and relevant examples are given”. [Focus Group Discussion, Wau] Despite being recognized for their ability to live up to the expectations of the Rabea facilitators’ trainings, a few of the individuals who were interviewed mentioned some shortcoming of some of the facilitators which could be of concern to the program. For example, one of the Rabea learners during a focus group discussion in Juba was not happy that some facilitators do not keep within the pause time during the lessons, and in due course miss out on some instructions and lesson segments. “Some of the facilitators want to talk when the radio teacher is talking. Some facilitators laugh too long when there is something funny. So that way they miss out some points because the radio will not stop to wait for the teacher to finish laughing”. [Focus group discussion participant, Juba County] Similarly, four (4) of the education inspectors interviewed indicated that they noticed during their Rabea program supervision that some facilitators find it difficult to follow the radio instruction during the Rabea lessons. One of them explains: “… there are some facilitators who do not listen and pick instructions correctly”. According to another “such facilitators end up giving learners wrong instructions and therefore misleading learners”. Willingness to recommend Rabea to others During the course of the focus group discussions with the Rabea learners, and interviews with Rabea facilitators and education officials, it was evident that majority of the learners find the Rabea program interesting and like it as well. Because of the positive attitude towards the program and the fact that most evidence collected suggests that learners benefit from Rabea, it is not surprising that most of those interviewed agreed that they would recommend Rabea to others who have not previously heard of or exposed to the program. It was also evident from the discussions and interviews that some of the program participants had already taken the initiative to tell others about the program as indicated by a female learner during an interview for her profile: 30 “I am encouraging other women to join the centre so they can learn not only how to sew clothes but also to read, write and speak English. I always tell other women in our church that it is not too late for them to learn to read and write. I tell them learning new skills will make their lives easier”. [Interview for a personal profile, female 47, Rabea learner, Lire Payam, KajoKeji] Rabea facilitators are also willing to recommend the program to others. Some, as one facilitator indicated during an interview willing to inform their peers about the program because of its technical advantages over the traditional teaching approaches and media. “Yes, I will recommend the Rabea Program to teachers like me. Most teachers do not motivate learners and others do not use teaching aids. But if one attends the Rabea Program, he/she will learn how to motivate learners and use teaching aids. Most teachers also do not arouse the interest of learners. In the ordinary class teachers do not do role playing. Rabea teaches role playing that is why I will recommend it to some friends”. [Male Rabea facilitator, KajoKeji County] Key learning point There is evidence from the findings presented in this section to show that Rabea could be a popular program in the areas where this evaluation was conducted. The section gives evidence on what learners and other program participants like about the Rabea program. Most of those interviewed indicated that they like the program because of the learners-centered approach employed during class instruction by the facilitators, and use of teaching techniques which make learning practical, fun and enjoyable. It is possible that this positive attitude towards the program could be the reason for learners and others interviewed to say that they are willing to recommend the program to those individuals who are not currently exposed to it. To sustain the positive attitude towards the Rabea program, SSIRI should ensure that the current quality of implementation is not compromised in any way. 3.5 Awareness of Rabea B program among MoE and Education NGO partners Although SSIRI has designated staff to monitor the implementation of IRI programs in each of the states in Southern Sudan, it ensures that AES and other education officials at the state Ministry of Education are at the forefront of the delivery of these programs. This is aimed at fostering local program ownership and sustainability. It is therefore important for these partners to have adequate knowledge of the programs implemented and it is assumed that SSIRI staff put in substantial efforts to extensively education the education officials about the interactive radio programs. To assess the progress of SSIRI on this front, fourteen (14) individuals in this stakeholder category were interviewed, and during the in-depth interviews they were asked to explain their office’s strategic relation with the SSIRI program, and what they do to advance the objectives of Rabea. Those interviewed included eight (8) AES inspectors, three (3) AES Directors, one (1) Rabea facilitator, and two (2) NGO project managers. 31 The education officials- one of the major stakeholders in the sustainability of the Rabea program have knowledge of the program’s objectives and expected effects. All the fourteen (14) individuals interviewed indicated that they had heard of the Rabea B and other IRI programs such as the Learning Village, and were aware of at least one center in their area using the Rabea B program. When asked what they thought the objectives of Rabea were, most of them articulated general expectations and outcomes of any other non-formal education program. Although these responses were contextually appropriate, five (5) of those interviewed provided a much more explicit explanation of the focus of Rabea B program as indicated in the selected interview excerpts below: “I think Rabea or in fact IRI is to improve educational quality within the existing structures. Rabea improves spoken English of those who attend formal or informal education and even drop outs. They learn vocabularies which they do not know. The learners get Mathematics skills and the civic education topics are very important and relevant to our current situation”. [In-depth interview with Director AES, Torit County, Eastern Equatoria State] “The objective of this program is to raise the standard of English in the state to uncountable classes of people under trees, at homes, in the offices, at the adult learning centers”. [In-depth interview with Director AES, Wau County, Western Bahr el Ghazal State] “The primary objective of Rabea is raising the standard of English and Mathematics of the adult learners. This is a quick way of learning for millions of adult in Southern Sudan who did not get the chance to go to the school”. [In-depth interview with ALP facilitator Kosti Adult Education Center, Wau County, Western Bahr el Ghazal State] 3.6 Participation of MoE and NGO partners All the fourteen (14) individual who participated in the key informant interviews indicated that they were playing at least one role in the implementation of Rabea. Two (2) of the individuals interviewed said that they were trained by SSIRI as Rabea facilitators and actively teach adults at ALP centers in their areas using Rabea. Other roles mentioned during the interviews include: participating in State and County SSIRI or Rabea program meetings; distributing Rabea materials to the Rabea centers; identification and selecting teachers for training, co-facilitating of SSIRI training workshops; supervising SSIRI programs; mobilization of adult learners to attend Rabea or other IRI lessons; and ensuring that there is coordination between EDC/SSIRI staff and the state education officials during the implementation of IRI programs. The Director of AES in Eastern Equatoria told the evaluation team that the relevant education officials in her state work closely with the EDC/SSIRI staff, and in that regard take on a number of tasks such as training of the facilitators for the program, orientation of County and Payam level education officials on the program and supervision of the program. 32 “Right from 2006 up to now we take part in training the facilitators for the program, we take part in training the teachers for Learning Village also we also train together with the EDC staff the education officials at state level, county and payam levels. We orient the education officials at state county and even payam levels about the importance of the RABEA Program to our people. Also some of our ministry officials take part in supervising the program. I have not personally been doing the supervision but our inspectors do.” [In-depth interview with Director AES, Torit County, Eastern Equatoria State] A similar report was provided by an AES Payam Education Supervisor in Kajo Keji County as indicated in the two complimentary excerpts below. The Payam Education Official specifically adds monitoring of radio broadcast of IRI programs to the array of roles played by the education ministry partners. “I play an active role in the implementation of Rabea. I have participated in the implementation of this program since it was reintroduced in KajoKeji County in 2008. As an education supervisor, I often oversee and support the implementation of the Rabea and other radio programs used in the learning centers. I monitor the implementation of Rabea and in case of any problem like when the radio does not broadcast the program on a particular day I often report this to my superiors and the SSIRI outreach coordinator. Sometimes when there are trainings I do co-facilitate them with Awori Festo, the SSIRI outreach Coordinator. My role is interesting because I get to participate in training and in monitoring the program”. [In-depth interview with Payam Education Supervisor in KajoKeji County] Others said that they mobilize the members of their communities to attend or listen in to Rabea B lessons as indicated in an in-depth interview excerpt below: “My role in the implementation of RABEA is especially in meetings with local communities about education issues. I do sensitize and encourage them to attend the lessons and also to listen to the programs even when they are not in class. Secondly, when we meet with facilitators and learners I do sensitize them on the importance of the program and encourage them to participate in it. Generally speaking, my role is to sensitize the community on the importance of the program”. [Key informant interview with Payam Education Supervisor, Lire Payam, Kajo Keji County] The Director of AES in Western Bahr el Ghazal confirmed that the education officials among other roles mobilize the adults to attend Rabea programs. “The role I play is that of supervision of the program. We also give encouragement to people to enroll in this educative program so that they miss this golden opportunity of improving their English language skills. I have participated in this program since its beginning by coordinating the education officials and the EDC staff based here in Wau. Some of the education officials take part in facilitating the training workshops. They are not doing very well in facilitation but with time they will become better.” [In-depth interview with Director AES, Wau County, Western Bahr el Ghazal State] 33 Others closely follow-up on the implementation of SSIRI programs including Rabea B: “I have been given the responsibility of implementing SSIRI program. I have to monitor all the schools. I have seven and under the seven schools I have 5 trained facilitators and the rest are not trained. Another role I have to ensure that they are doing the real exercise in the field to enable the learners learn more and for effective learning to take place. I play a role of monitoring the program to see whether or not it is being implemented and also to know the teachers who have been trained. I like my role because whenever I do follow up of activities in the field it encourages and motivates the facilitators and learners to do more. When there is no supervision you find they become reluctant”. [Key informant interview with Payam Education Supervisor, Lire Payam, Kajo Keji County] Although not expressed by many, one aspect that might be of concern is that some education officials seem not to take monitoring or supervision of Rabea program seriously. This opinion was expressed by one County Education Inspector during an in-depth interview when asked to specify what he thought had been barriers to the Rabea program;”…... Some education officials [in this area] do not take monitoring of the implementation of Rabea lessons seriously”. Key learning point: Active involvement of the education officials in the implementation of Rabea program will go a long way to ensure that SSIRI’s sustainability plan is on course. The education officials should take on greater responsibility for implementation and expansion of the program. This does not seem to pose any challenges since the evidence from the interviews with the education officials indicates that they like the program, appreciate their role, are willing to participate in the implementation of Rabea, and are aware of the implication of their participation. 3.7 Issues of concern regarding the implementation of Rabea During the interviews and focus group discussion, the participants raised issues which they felt are affecting the successful implementation of Rabea in some areas, and therefore need to be brought to the attention of the program’s management. The issues emphasized most during the interviews in order of frequency of mention by the participants are:  Poor quality of radio signal - commonly mentioned during interviews in Kajo Keji, Yei and Wau County;  Downsizing and transfer of teachers (including Rabea facilitators);  Irregular attendance of classes by learners. 34 Poor quality of radio signal Although the methods of data collection employed do not enable the evaluation to draw an accurate conclusion on this issue, it is worth noting that the findings presented here are only indicative of an existing problem which needs to be highlighted. This issue was mainly raised during interviews with Rabea facilitators and education officials in Kajo Keji, Yei and Wau County, and during a focus group discussion with learners in Wau. This issue did not seem to be a key concern in the other evaluation areas of Juba and Torit as indicated in the following selected interview excerpts. Apart from the quality of the radio signal, coverage of the county by the radio broadcast is still inadequate as indicated by an education official in Yei County during an in-depth interview. “… FM radio station on which the broadcast is done does not cover most of the areas in this County were the ALP Centers are. For this reason we need pre recorded programs to be used for teaching learners in areas where there is no radio coverage. Limited radio coverage is a challenge in the implementation process of the program here in Yei”. [In-depth interview with Education Official, Yei County] “The major challenge and barrier I see in the implementation of the program in this area is that the signal of the radio is poor. And this is common in most Payams here. The example here is the schools in Tore Payam where the reception of Spirit FM is very poor”. [In-depth interview with Education Official, Yei County] Many Rabea Centers in Kajo Keji County receive no or poor FM radio broadcast signal; “The challenge I can see is about the radio reception. At first we had Grace FM which could not reach other areas. But now that are using Voice of Kajo Keji, there are other schools that cannot receive the radio broadcasts”. [Personal profile interview with Rabea facilitator, Kajo Keji County] In Wau County, Western Bahr el Ghazal, change of radio station in the recent past has not been helpful in resolving the poor quality of radio signal; “One of the challenges I have seen in the implementation of Rabea is the radio station, which is sometimes not effective. We realize that sometimes the station’s signal is not clear in some centers. Secondly, radios have technical problems and sometimes the radio fails to work. Sometimes at the station there is a problem and the learners cannot hear the lesson even if they tune to the station on that day”. [Personal profile interview with Rabea facilitator, Wau County] “Yes in the beginning the program broadcast was very good and successfully. But since it changed from 101 Fm to Wau FM there are a lot of challenges. The signal is not clear and it is disappointing the learners and the facilitators especially in Wau”. [Key informant interview, Education Official, Wau County] 35 Downsizing or transfer of teachers In a bid to improve the quality of teaching in Southern Sudan the Ministry of Education embarked on a restructuring program, which has seen many teachers especially those who do not meet the stipulated minimum qualification relieved of their jobs. This, as most of the key informants (education officials and NGO partners) reported during the evaluation, has and is still affecting the implementation of Rabea. It is evident from the interviews that a substantial proportion of the teachers trained by SSIRI to teach Rabea classes have lost their jobs, been transferred to other areas, or have been absorbed into the formal basic education system. This always creates capacity gaps in the implementation of Rabea as indicated in the following interview excerpts. “Another challenge is that during the downsizing of teachers by the ministry, some teachers decide to take away the radio because they are angry about being downsized. We have examples in Yei of some schools where the downsized teachers went with the radios and are now living in Uganda”. [Key informant interview, Education Official, Yei County] “Also another challenge is when transfers and downsizing of teachers are carried out. These exercises normally affect the implementation of the program especially when the trained person is transferred and no one is left at the center. [Key informant interview, Education Official, Wau County] “We train the teachers and sometimes the teachers are transferred to centers where there is no Rabea Program. Another problem is that some Rabea facilitators do go away with the radios when they are transferred to some other places. They assume the ownership of the radios not knowing that these radios are supposed to be left at the center”. [In-depth interview with AES Director, Torit County] Irregular attendance of classes by learners For learners to make meaningful learning gains, they need to regularly attend Rabea lessons. They particularly need not miss out on any lessons because the lessons are designed to foster incremental learning. However, as indicated by most of those interviewed in the counties of Juba, Torit, Kajo Keji and Wau, irregular participation of learners in the ALP and Rabea lessons is likely to affect the quality of adult learning in these areas. This issue of irregular attendance to lessons was also experienced by the evaluation team when they failed to identify a substantial number of individuals at seven randomly selected ALP Centers who had completed at least 10 or more Rabea lessons following a radio broadcast run of about 20 lessons. Apart from eventualities such as the death of a community member, most of those interviewed indicated that the major cause of absenteeism is the planting and harvest seasons respectively. 36 “Some of the challenges we face are that sometimes during harvest and during sowing seasons these learners don’t attend to their classes. And during community gatherings such as funerals and birth ceremonies and other activities within the communities the learners don’t attend to their classes because as adults and part of the community they have to attend these gatherings”. [In-depth interview with Rabea facilitator, Torit County] “Sometimes social problems interfere with activities of adult learning centers, including RABEA lessons. For example if somebody dies people do not come to class or listen to the radio. Sometimes the dead person may be a member of the RABEA class or a relative of some of the learners”. [In￾depth interview with Rabea facilitator, Kajo Keji County] Other issues of concern which came up during the evaluation: 1 Three of the individuals interviewed including a State Director for AES said that through their activity monitoring experience they have noticed that the pause time allotted during the lessons is too short. “I realize that the time interval given for learners and the facilitator to perform a certain task is limited compared to the instructions given. Our people like examples especially in Math and English. People need to see and do things practically and this requires that the radio provides more time for learners to participate”. [Key informant interview, Education Official, Torit County] “The time pause for facilitators and learners to respond seems to be too short. The pause is not quite enough for the teacher to carry out instructions. Practically the learners do not see any chart or display. The guides are only for the facilitators. There should be picture books with a sentence or two about a picture”. [Key informant interview, Education Official, Juba County] 2 Individuals who attend Rabea lessons and complete level 4 of ALP are uncertain of their education future thereafter. “The learners have also been posing questions to us whenever we come together. The question is; “What will their fate be in future regarding what they will do after living the ALP centers. They feel that there should be vocational skills imparted on them so that whenever they leave this level they can go and do other things for themselves using the same skills they would have gained from the Rabea class”. [Key informant interview, Education Official, Torit County] 3.8 Factors contributing to success of the Rabea program Through the evaluation findings we see patterns of actual and reported outcomes which suggest that the objectives set out for the Rabea program are being attained. For example, through the results of the achievement tests we see that individuals who attend more lessons perform significantly better than those who attend fewer lessons. This particular outcome suggests that the Rabea program could be having an effect on the performance of learners in English language and Mathematics. This finding coupled with the self reports on the benefits of Rabea provided by individuals attending or facilitating or monitoring the Rabea program are a reflection of the intended outcomes of the program. 37 It is important to examine how the program produces the expected outcomes. For example, what has SSIRI put in place to ensure that the expected outcomes of Rabea program are attained? What are the existing supportive contextual factors? A synthesis and analysis of the interviews indicates four (4) factors responsible for the seemingly successful implementation of the program. a) The time at which the program is broadcast on radio is convenient for adult learners: It was evident from the interviews that the first element responsible for the good response which the Rabea program has received from its intended audience timing of the radio lessons broadcast. One of the education officials in affirmation of this observation said, “The time of the radio lessons broadcast which is in the afternoon is very suitable for the adult learners as most of them are able to attend after working in their fields”. b) The civic education topics or content covered: Another issue which the evaluation participants felt was a facilitating factor in the implementation of the Rabea program is the content and specific topics covered in the radio lessons. As mentioned elsewhere in this report, most of those interviewed agreed that the topics especially those in the civic education segments of the lessons are relevant, cover issues which every Southern Sudanese appreciates, and improves people’s awareness of contemporary issues affecting them; c) The interactive nature of the program and use of radio as a teaching aid: The format of interactive radio instruction followed during the Rabea program also seems to make learning fun according to the education officials and Rabea facilitators, as one of them commented; “I think the interactive nature of the radio lessons is another element that has facilitated the program. This is because the program encourages learners to participate equally in class. The broadcasts attracts learners ad motivates them to be present in class; d) Training offered to the facilitators: There were suggestions among the education officials that the Rabea facilitators do their job well. This they said is indicate that they quality of training provided by SSIRI is good. Because of their performance, they are able to capture the attention of learners and strengthen the interest of learners in the Rabea lessons at the ALP centers. 38 3 CONCLUSIONS This report is a synthesis of findings for an evaluation of the performance of SSIRI’s program called Radio-based Education for All (Rabea). The design which was adopted for this type of evaluation has limitations. For example, because of lack of a control group we are not able to draw reliable conclusions about the actual impact of Rabea on the learners’ English language, Mathematics skills, and knowledge of local civic education issues. All information put together in this report is based on self-reported experiences by the beneficiaries and other program participants. However, the use of mixed-methods approach offers a platform for us to understand the story behind the performance of learners in their English language and Mathematics achievement tests. Based on the findings drawn through this data collection and analysis framework, we are able to draw reasonable conclusions about the performance of Rabea related to its relevance, effectiveness, and sustainability. Does the program make sense within the context of Southern Sudan? As mentioned in the introductory section of this report, the education sector in Southern Sudan was adversely affected by over two decades of civil war consequently, two generations in the South lack basic literacy and numeracy skills. Because of this, illiteracy rates are high and are estimated at 85 percent. The relevance of Rabea in such a context is highlighted in its objectives and design as well as the results attained to-date. The program is specifically based on the basic education curriculum of Southern Sudan, draws on local examples, and is designed to improve the English language and Mathematics skills of people in Southern Sudan. This evaluation gathered a trail of self-reported experiences indicating that Rabea is helping individuals in Southern Sudan to speak, read and write in English. There were also self-reports suggesting that the program is developing the learners’ mathematics skills and use of mathematics operations in daily life activities. The programs reaches the most difficult to reach populations through the FM radio broadcasts. Even without an instructor, over 100,000 individuals consistently listen to Rabea on radio at their homes. Learners can also access this program through a formal class setting at the nearest ALP centers, and be able to learn how to write, speak and read English, and learn about pertinent civic education issues. In 2010, SSIRI registered 9,885 adult learners attending Rabea in 139 ALP centers in Southern Sudan. However, for the program to successfully contribute to the education needs of the disadvantaged population it will rely heavily on improved geographical coverage through the burgeoning FM radio sector. Relevance is compromised when the most disadvantaged populations have less access to the benefits of the program. The program will also rely on the support of the Directorate of Alternative Education System towards ensuring that the ALP centers are duly functional- the necessary learning infrastructure exists and teachers are paid. 39 To what extent have the objectives of the Rabea program been achieved? SSIRI has developed 120 lessons for Rabea B. The broadcast of these lessons started in October 2007 on six FM stations. The number of broadcasting radio stations has now grown to 15 radio stations with a geographical coverage of 30 counties in Southern Sudan. Over 3,000 Rabea B facilitators have been trained to facilitate the radio lessons in 139 ALP centers. Each of the Rabea B facilitators trained received a functional freeplay radio or a set of audio digital devices and a teacher’s guide as instructional accessories. SSIRI continues to train new facilitators and extend the use of Rabea in new ALP centers in new counties throughout the 10 states of Southern Sudan. SSIRI has also trained Education Officials at different levels of government enabling them to be able to supervise, monitor, and support the implementation of Rabea and other IRI related activities such as teacher training and mentorship. Through this evaluation, we see success stories and evidence suggesting that Rabea B has been helpful in improving people’s English language and Mathematics skills. The evaluation team also obtained evidence suggesting that the training provided by SSIRI on how to use IRI has improved the skills of the Rabea facilitators. The self-reports provided by the facilitators suggest that the facilitators have for example improved the way they teach, interact with and involve the adult learners during Rabea lessons. Additionally, there is evidence suggesting that the learners find the civic education segments of the program interesting, and have learned more about pertinent civic issues such as the CPA (Comprehensive Peace Agreement) and DDR (Demobilization Disarmament and Rehabilitation) programs in Southern Sudan. However, a more rigorous assessment could be helpful in providing a more accurate picture of the impact of Rabea on the learners’ English language and Mathematics skills. Sustainability of the Rabea Perhaps the most important indication of sustainability highlighted in this assessment is that the Ministry of Education personnel are actively involved in the implementation of Rabea program. They train and supervise Rabea facilitators, and participate in the quarterly review of the programs and development of work plans. This is one of SSIRI strategies of working towards local Sudanese ownership of the program. The evaluation findings show a high level of enthusiastic commitment from the Education officials. Other indicators of the program’s sustainability highlighted in this report are: the positive attitude among learners and education officials towards the Rabea program, willingness of the learners and Rabea facilitators to tell others about the program, building the capacity of Southern Sudanese to manage the program through training of teachers and Education Officials, and provision of radio and teaching guides. 40 4 RECOMMENDATIONS The evaluation team sought input from all stakeholders (individuals who participated in the interviews and focus group discussions) on how to strengthen the implementation of the Rabea program. There were a number of varied suggestions, but in this report we present those proposals which are closely linked to the most pertinent issues of concern in the program’s implementation. Three suggestions therefore stand out: (a) address the issue of poor radio signal; (b) train more teachers to cater for the gaps created as a result of teacher transfer and downsizing; and (c) Ensure better storage of radios by the facilitators and schools. Address the issue of poor radio signals Weak radio signal came on top of the list of barriers facing the implementation Rabea. It was clear from the discussion and interviews that the learners in some areas where the evaluation was carried out such as Kajo Keji, Yei, Wau and Jur River, miss complete Rabea lessons or parts of a lesson because of poor FM radio broadcast signals. Success of IRI programs rely a lot on efficient audio broadcast technology. Certainly, for SSIRI to realize or sustain the sort of results presented in this report, the program will work towards ensuring that radio signal for the FM station broadcasting the lessons is good. Areas which have poor reception of an FM signal could also opt to use digital devices as one of the education officials interviewed in Kajo Keji indicated “Weak radio signal and low volume is an issue to most centers here in this Payam. To minimize this problem of radio signal, we can make use of recorded programs on CDs, and the radio volumes be improved- let something be done to improve and make radio volumes louder”. Rabea facilitator training It has already been mentioned in another section of this report that the government’s policy on teacher employment (teacher transfer and downsizing) is affecting the implementation of the program in the areas where this evaluation was carried out. It is obvious that because of what is happening, it is increasingly becoming difficult to attain a critical mass of teachers to facilitate the Rabea classes or even ensure that the Rabea program runs undisrupted. In view of this assessment, there were further suggestions that SSIRI conducts regular teacher capacity assessments and trainings to ensure that trained IRI teachers are available, and learners do not miss Rabea lessons. Storage of teaching materials including the radios During the interviews and focus group discussions there were concerns related to the safety of the radio used by the facilitators. Most of the key informant interviews reveal the need to ensure that the ALP centers take good care of the radios, and return them once they are transferred to another location. Therefore there were suggestions that each centre should have a place to store safely the equipment and all the other teaching aids so that they are not lost. 41 Appendix 2: Focus Group Discussion (FGD) Guide 1. Description of structure of discussion (by moderator) 2. What do you like?  What is your favorite subject/topic/character? 3. What don’t you like about the Rabea? 4. What program topics or subjects do you find relevant to your life? Why? 5. Please give me examples of knowledge and skills which you have learned from Rabea 6. What do you think of the facilitators? (Likes, dislike?) Why? 7. Why did you join the ALP center and start attending Rabea lessons? 8. Do you listen to the program outside this group? 9. Do you ever talk about topics discussed in Rabea outside this group? With whom? Please give examples of the discussions you have had outside this group 10. Please tell me how to improve this program 42 Appendix 3: Key Informant Interview Guide 1. Overall, what do you see as the primary objectives of Rabea Program? 2. Do you play any role in the implementation of the Rabea Program? What is your role? What do find interesting about this role? 3. What do you like about the Rabea for Beginners’ radio series? What do you like about the way the facilitators conduct the Rabea lessons? 4. What do you dislike about the way the Rabea facilitators conduct the Rabea lessons? 5. How has people’s exposure/attendance to Rabea learning series changed their Mathematics and English language skills? Moderator probe: What do people do differently in their daily life after attending Rabea lessons? Anything else? 6. What civic education issues do people discuss in the community? Interviewer probe: Would you say that this discussion is related to what people learn during the Rabea lessons? Moderator probe: do people talk about [mention civic education issues in Rabea program] 7. Overall, how effective or successful do you think the radio program has been at meeting its objectives? (Interviewer: Please check only one) (a) Very successful (program has met most of the objectives, most of the time)…. (b) Somewhat successful (program has met some of the objectives, some of the time… (c) Not successful (program has failed to meet its objectives)… Please explain or give example(s) of why you gave this answer 8. What do you think are the elements or characteristics of the program that have facilitated the program? What do you think are the elements or characteristics of the program that have been barriers to the program? 9. What do you think can be done to address any barriers or limitations you have described? 43 Appendix 4: Personal Profile Interview Guide 1. Please tell me your: a. occupation, marital status, number of children (if any), b. own education status, education status of grown up children (if any), whether or not you reside in a rural or urban area, – c. Type of housing structure that you live in, number of living rooms, and source of food. 2. How often do you attend Rabea lessons at the ALP center in terms of days per week, or weeks per month? 3. Why did you decide to attend lessons at the ALP Center? 4. What new things have you learned in the recent past when you attended a radio (Rabea) class? Please tell me what you like or dislike most about the radio classes 5. Please share with me a typical experience of any radio lesson which you will always remember. 6. What aspects of your life changed as a result of attending the radio (Rabea) lessons? What aspects did not change? Why do you think this happened to you? 7. I would like to ask you once again: Imagine that you did not attend the interactive radio instruction program: compare your general knowledge and skills in English language and Mathematics now with what the situation would have been if you did not participate in the program. 8. If you were given five (5) minutes to talk about the good things that you have acquired as a result of attending radio classes (RABEA), what important issues would you ensure are captured before you are told: ‘time is over’? 9. What don’t you like about the radio program? How can we make the program more interesting to people like you? 10. Would you recommend this RABEA program to a friend or close relative? Why? 44 Appendix 5: List of Radio Stations Name Frequency Town Counties covered Programs aired 1. Bakhita Radio 91.0 FM Juba Juba RABEA B2, RABEA Advanced 2. Junubna FM 100 FM Juba Juba, Terekeka LV P1, P2, P3 and P4 3. SRS FM 98.6 FM Juba Juba, Terekeka, Yei, Torit, and Lainya All LV Programs and RABEA B1 4. Voice of Kajokeji 92.0 FM Kajokeji Kajokeji All LV Programs and RABEA B1, B2 and Advanced 5. Grace FM 95.1 FM Kajokeji Kajokeji, Nimule RABEA B1, B2, and Advanced 6. Spirit FM 99.9 FM Yei Yei, Lainya and Morobo counties LV Programs and RABEA B1, B2 and Advanced 7. Voice of Eastern Equatoria 97.5 FM Torit Torit and Magwi counties All LV Programs and RABEA B1, B2 and Advanced 8. Magwi FM Magwi Magwi LV P1, P2 and RABEA B1 9. Voice of Love (Sout al Mohaba)FM 93.6 FM Malakal Malakal, Fashoda and Kadok counties LV P1, P2 and RABEA Advanced 10. Good News FM 89.0 FM Rumbek Rumbek Central, East and West Counties LV P1 and P2 RABEA B1, B2 and Advanced 11. Radio Jonglei 95.9 FM Bor Bor, Panyagor, Nyirol, part of Juba County RABEA Advanced 12. Bentiu FM Bentiu Rubkona LV P1, P2 and RABEA Advanced 13. Wau FM Wau Wau, Jur River All LV Programs and RABEA B1, B2 and Advanced 14. Yambio FM 90.0 FM Yambio Yambio, Nzara, Tombura, Mundri West and Maridi counties All LV Programs and RABEA B1, B2 and Advanced 15. Voice Of The Community 88.0 FM Kauda Haiban, Delling All LV Programs and RABEA B1, B2 and Advanced 16. Naath FM 88.0 FM Leer Leer County RABEA Advanced 17. Radio Nhomlau Aweil Aweil Central, East and West RABEA B1, B2 and Advanced 18. Kurmuk Kurmuk Kurmuk LV P1, P2, P3, P4, RABEA B1, B2 and Advanced 45 Appendix 6: Personal Profile 1- Saidia Helen Radio lessons help me teach topics that are not in textbooks, says facilitator A class of nearly 20 adults is seated in class, listening attentively to the teacher standing in front of them. The teacher tunes a radio and writes something on the black board. Then the RABEA broadcast comes on air. And so the lesson starts and the female teacher takes her learners through it. During the lesson the teacher engages the learners and they actively participate up to the end. Saidia Helen is the Rabea facilitator at Logili ALP centre in Kangapo II Payam, KajoKeji County of Central Equatoria state and have a chat with her. Saidia is happy that she is teaching these radio lessons and she is especially happy that she is teaching this group of adult learners. Since she started teaching the radio lessons, the 29 year old mother of three says her skills have improved tremendously. “My skills have been improved. I have learnt that you have to prepare before conducting any lesson.” She adds, “The radio lessons have improved the way I teach adults because the radio makes learners respond quickly. This helps the adults to stay focused on the lesson as they are not as quick at learning as young children are.” In addition, she says the radio teacher guides the classroom teacher. “My work has been simplified. I am just asked to do something and I do it. I don’t have to plan my own lessons like I do for the non radio lessons.” She says that in addition the radio lessons encourage the learners to listen attentively, which also facilitates the work of the classroom teacher.” She adds that, “It is also easy because it reduces the workload of the facilitator as the radio teacher has already arranged everything.” Although she is a qualified teacher with a grade three certificate from Uganda, Saidia attributes the improvement in her skills to IRI training that she attended in February 2010. Saidia says she likes teaching the radio lessons because ─ according to her ─ “the learners are normally active during the radio lessons rather than in the normal lessons.” In the course of teaching RABEA─ which she does three times a week─ Saidia has learnt that even without a radio, especially in case of any interruption on the radio broadcast, an IRI teacher can still use the teacher’s or facilitator’s guide to conduct a radio lesson. As an English language teacher, she likes the RABEA segment on letter combination and thinks she will never forget about it. “For I like the combination of the letters CL and CR and the sounds that these letter combinations make,” she explains. She says she also likes the segment on words with the same sound but with different meanings such as ‘there’ and ‘their’, ‘hill’ and ‘heal’, ‘where’ and ‘were’, ‘peace’ and ‘piece’. Looking back at the way she taught before being trained on IRI implementation, Saidia says she has now improved greatly, especially in the way she organizes and prepares for lessons. If asked to speak about the advantages of teaching using radio lessons she would sight the fact that the radio encourages learners to listen to the teacher facilitating the lesson. “Secondly I do not have to take long in preparing for the lesson because the radio teacher guides me on what to do.” She thinks that the radio lessons carry information that is not accessible from other sources. “The radio lessons have civic education and this is something that you cannot find in text books in the library. For example in health science there is topic concerning small pox.” The only reason she does not like teaching radio lessons, Saidia says is “because some of the learners sometimes forget what has been aired on the radio. Some learners forget what they have been taught. They think it is a mere radio broadcast and do not take the radio lessons seriously.” Secondly it is time consuming and if the teacher does not follow the broadcast as it is on air, there will be difficulty in trying to explain to the learners what is being said on the radio by the radio teacher. 46 Appendix 7: Personal Profile 2- Grace Basa, 45, KajoKeji County Radio lessons have taught me how to speak in public As a tailor she has gained additional skills to enable her do her work efficiently. Her writing and reading skills have improved. In addition, her newly acquired knowledge of numeracy enables her to do correct measurements on clothe before she does any sewing. Grace Basa trained as a tailor and has been training other young women who hope to use the new skills to generate some income and improve their lives and those of their children. The 45 year old mother of five is one of several adult learners attending the Adult Learning Program at Bibi Centre, in Lire Payam of KajoKeji County in Central Equatoria state. The centre teaches radio lessons, known as RABEA or radio-based education for all. She decided to enroll for classes at the centre in order to read, write and speak English. She thought this would improve her skill and make her a better tailor. By joining the centre and participating in the radio classes, Grace hoped to improve her reading and writing skills so she could intern impart knowledge to the young women she is training to become tailors. Today two years since she first entered the adult class and heard the radio lessons, Grace is happy that other than improving her writing and reading skills, her tailoring skills have also improved. “I think that my participation in the radio lessons has improved the way I do my work as a tailor. I can now take more accurate measurements on clothes before sewing. Previously I used to estimate sizes. I would just look at a customer and estimate the size of their waist or shoulder and sometimes I would ask people to help me take the measurements.” The mother whose first child graduated from university in 2006 and the youngest is in S1 attributes her current ability to express herself in English to the RABEA classes she attends at the adult learning centre. “Today I don’t have to rely on other people to translate speeches made in English. Whenever I used to go for a function where people are speaking in English, I would wait for someone to translate for me. But now I can hear it with my own ears.” She also thinks that the radio lessons have improved her speaking skills unlike in the past when she used to fear to speak in public. In some of the English language segments on RABEA there the radio characters engage in debates. Grace thinks this has been a very effective to help learners sharpen their speaking skills. “The debates have benefited us as we emulate the way the debaters speak,” says Grace. “With the radio lessons we have heard the way other women express themselves. For me this listening to others has helped me improve my speaking skills. The other thing is also that my pronunciation has improved,” Grace stresses. Grace is encouraging other women to join the centre so they can learn not only how to sew clothes but also to read, write and speak English. “I always tell other women in our church that it is not too late for them to learn to read and write. I tell them learning new skills will make their lives easier,” Grace explains. She says her favorite lesson in the RABEA series is about women’s rights in the community and in their homes. “I think that this lesson has taught some of us women at this centre to learn what we had not known about pour rights for all these years.” Grace plans to enroll in secondary school if she passes the primary leaving examinations next year. For now she is working on improving her English language skills and has begun reading story books that she borrows from a library at a neighboring primary school.