SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION JULY 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Stuart Leigh and Charles Tesar, PhD., Management Systems International. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION A subsidiary of Coffey International, Ltd. Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contracted under DFD-I-00-05-00251-00, Task Order #2 USAID/Sudan SUPPORT Project Rahaf Complex Soba, Khartoum, Sudan SSIRI Implementer: EDC SSIRI Evaluation: Management Systems International (MSI) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION i CONTENTS TABLE OF FIGURES AND/OR TABLES ............................................................................................III PROJECT SUMMARY............................................................................................................................IV ACRONYMS.............................................................................................................................................. V EXECUTIVE SUMMARY ......................................................................................................................VI I. INTRODUCTION.......................................................................................................................... 11 II. BACKGROUND – DEVELOPMENT OF THE PROBLEM & USAID’S RESPONSE .................................................................................................................................... 12 Southern Sudan ............................................................................................................................ 12 Primary education in Southern Sudan.......................................................................................... 12 Problem Stated ............................................................................................................................. 14 History of USAID Intervention in Southern Sudan ..................................................................... 15 SSIRI Program Description.......................................................................................................... 16 III. PURPOSE OF THE EVALUATION ........................................................................................... 18 IV. RESEARCH DESIGN AND METHODOLOGY........................................................................ 19 V. ANALYSIS AND FINDINGS........................................................................................................ 20 Process Findings .......................................................................................................................... 21 Outcome and Results Findings..................................................................................................... 26 VI. CONCLUSIONS............................................................................................................................. 31 Achievements:.............................................................................................................................. 31 Shortfalls: 32 Further Conclusions: .................................................................................................................... 34 VII. RECOMMENDATIONS ............................................................................................................... 34 VIII. LESSONS LEARNED ................................................................................................................... 41 IX. ANNEXES....................................................................................................................................... 43 Annex 1: Work Plan SSIRI Evaluation Activity MSI (Stuart Leigh and Charles Tesar) May 10, 2008 .............................................................................................................. 43 Annex 2 Usaid Evaluation Program Edc/Ssiri: Itinerary And Activities................................... 55 Annex 3: Southern Sudan Interactive Radio Instruction (SSIRI) Grade 1 Evaluation .............. 59 Annex 4: Summary Of SSIRI School Observations With Radio Reception Notes .................... 80 Annex 5: SSIRI Deliverables and Proposed Activities in Contract or Modification and Status of Each.............................................................................................................. 81 Annex 6: Shortwave Radio Reception Test Results.................................................................... 87 Annex 7: EDC/SSIRI Responses to Mid-Term Evaluation ........................................................ 88 Annex 8: Department Of Alternative Education Systems........................................................... 95 Annex 9: Map of Miraya-FM Radio Stations Projected Broadcast Coverage ............................ 96 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION ii Annex 10: People Interviewed or Met During SSIRI Evaluation (and partial transcription of handwritten meeting notes)................................................................ 97 Annex 11: SSIRI 2nd Response to Evaluation-“One Teacher At a Time”-June7 (32 p. with five EDC Annexes) ........................................................................................... 113 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION iii TABLE OF FIGURES AND/OR TABLES Table Name/Content Table 1 (Annex 3) Summary of SSIRI Broadcast Observations / Radio Reception Notes Table 2 (Annex 4) SSIRI Deliverables and Proposed Activities in Contract or Modification and Status of Each Table 3 (page 14) SSIRI Staff Composition Table 4 (Annex 5) Radio Reception Tests SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION iv PROJECT SUMMARY Project Name: Southern Sudan Interactive Radio Instruction (SSIRI) Project Implementing Partner: EDC Mechanism: Cooperative Agreement CTO: Lucy Kithome Start Date: June 22, 2004 Planned End Date: June 21, 2009 Total SSIRI Est. Cost: $15.85 million, including: -DCOF: $2.86 million (a part of the $15.85 million that targets expansion of The Learning Village in The Three Areas). -TERBIA/PS101: $7.62 million (a part of the $15.85 million that aims at broadcasting English language educational programming to populations in Southern Sudan and the Three Areas). Geographic Focus: Southern Sudan with special focus on the three urban centers, plus the Three Areas. The Southern Sudan Interactive Radio Instruction project was initially funded in 2004. It was designed to address USAID’s then Strategic Objective (SO) 6: “Improved Equitable Access to Quality Education,” which formed part of the USAID Sudan Interim Strategic Plan (ISP). The ISP goal was “Foundation established for a just and durable peace with broad participation of the Sudanese people.” Under the Fragile State Strategy, the USAID Southern Sudan Mission later shifted its strategic objectives and SSIRI then worked to address Strategic Objectives No. 9 “Avert and Resolve Conflict”, and No. 10 “Promote Stability, Recovery and Democratic Reform”. Currently it is focused on “Investing in People”, “Supporting the Comprehensive Peace Agreement (CPA)” and bringing peace dividends to the citizens of Southern Sudan through improved educational opportunities. The project lifespan is 5 years - June 22, 2004 - June 21, 2009. It is being implemented by the Education Development Center under Contract # 623-A-00-04-00054-00 (Effective Date 06-22-04 with a funding level of $5,000,000) and under Modification 4 with a funding level of $9,969,303. The total funding level is $15,469,303 ($5,000,000 through the original contract plus a subsequent budgeting of $500,000 through Modification 1 on July 11, 2005, plus further budgeting of $9,969,303 through Modification 04 on August 7, 2006, plus further budgeting of $383,478 through Modification 09 on March 13, 2008.) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION v ACRONYMS AES Alternative Education System (of the MoEST) ALP Accelerated Learning Program (of the AES) CCRI Cush Community Relief International CDC Curriculum Development Center CEC County Education Center CPA Comprehensive Peace Agreement GOS Government of Sudan GoSS Government of Southern Sudan IRI Interactive Radio Instruction LV Learning Village (SSIRI Interactive Radio Programs and Teacher’s Guides) MSI Management Systems International MDTF Multi-Donor Trust Fund MoEST Ministry of Education, Science and Technology PS101 Professional Studies 101 P1…P4 Primary 1, 2, 3, or 4 SMoE State Ministry of Education SRS Sudan Radio Service SSIRI Southern Sudan Interactive Radio Instruction Project TERBIA Teaching English through Radio-Based Instruction for All TTI Teacher Training Institute UNICEF United Nations Children’s Fund USAID United States Agency for International Development VSAT Very Small Aperture Terminals (used for data broadcasts) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION vi EXECUTIVE SUMMARY A two-person team was deployed by Management Systems International (MSI) during May 2008 to conduct the mid-term evaluation of the Southern Sudan Interactive Radio Instruction (SSIRI) project. The project is the extension of an initiative launched by the Education Development Center (EDC) in 2004. Significant modifications were made in June 2006 (MOD No. 4) and March 2008 (MOD No. 9). The project is due to terminate in June 2009. Project Summary: The goal of the SSIRI project is to improve basic education for children in formal primary schools and for out-of-school youth and adults. The project has focused on the development and implementation of radio programming for use in classrooms and complementary support to enable teachers to be more effective in meeting the learning needs of students. SSIRI was originally targeted for six states. However, the 2006 Modification called for the program to be expanded to all 10 states, plus the “Three Areas.” EDC has chosen five broad implementation strategies: 1. Production of radio broadcast learning programs and materials, offered to the Southern Sudanese public in two ongoing radio series (a) the Learning Village, which is intended for formally enrolled students in grades 1-4; and (b) TERBIA, which targets out-of-school learners; 2. Installation of a radio programming delivery system that provides programming to radio stations, training to formal school teachers and Alternative Learning System facilitators to incorporate SSIRI programs, and radios and support materials to key listeners; 3. In-service teacher training programs (radio-based) to raise the quality of all primary school teachers in Southern Sudan; 4. Incorporation of Internet and video technologies for teacher training as well as low-cost teaching technologies and resources as a complement radio instruction, or as an alternative to radio programming outside of broadcast coverage areas; and, 5. Strengthening of the Government of Southern Sudan’s (GoSS) recently created Ministry of Education, Science and Technology (MoEST) Evaluation Methodology: Upon arrival in Juba, the evaluation team conducted an in-depth desk review, met with USAID and EDC project personnel, interviewed MoEST personnel, members of the donor community and their associated program executives; interviewed administrators in five states; and conducted site visits to 16 schools and adult learning centers in five states. A formal work plan was submitted to USAID and EDC before commencing field work (Annex 1). For the most part, the work proceeded as planned; however, the evaluation team (which was accompanied by the SSIRI project Chief of Party, the USAID Mission’s Team Leader for Education, the USAID Cognizant Technical Officer (CTO), and a MoEST Senior Inspector) was prevented from traveling to the “Three Areas” because of security prohibitions dictated the Regional Security Officer. Findings: Production of radio programs for the Learning Village and TERBIA series is proceeding in consonance with projected work plans. All grades 1-3 Learning Village programs are complete and work is well underway on grade 4. Advanced and beginners TERBIA programs have been completed and are on-air; an intermediate TERBIA program is proceeding according to schedule. Agreements have been reached with the principal radio broadcasting radio stations in Southern Sudan and programs are regularly aired. Station managers offered high marks for both Learning Village and TERBIA, although competition is fierce for air time. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION vii While the numbers of trained teachers and their pupils are known, listenership or school and teacher use information for SSIRI broadcasts has not been gathered in any valid or systematic fashion. Consequently, project administrators must rely on staff reports that are of suspect validity and accuracy. These data are forwarded on to USAID without validation. Internal evaluation of grade 1 learning gains has been completed and results are encouraging, especially for English language development. Visits by the evaluation team to schools, however, was not able to yield valid findings on the impact of the Learning Village on classrooms, since observations were limited to 10 schools during broadcast hours. Nevertheless, evaluators found that only seven of 33 possible SSIRI-trained and supported teachers were using radios in their daily instructional program. A number of factors may help to explain this low usage: some radios have not been distributed; schools’ operations were quite disorganized at the time of the evaluation, since they were closed until early May for a national census gathering; SSIRI broadcasts for the term were only begun by Southern Sudan’s major radio station, Miraya FM, after the MSI team arrived; and, many teachers trained for radio programs have been transferred to other schools. In-service teacher training programs (radio-based) were to have been launched by 2007 but are not yet initiated, despite the universal need for teacher upgrading. Low-cost teaching technologies (MP3 players) are being tested in TERBIA classes as an alternative to radio, and more sophisticated ones (Internet and video) have been launched at the Arapi and Maridi teacher training institutes. A third internet/video production center is near completion in Malakal. MoEST strengthening is proceeding as planned. The Ministry has incorporated SSIRI programming at the central, state, and county level through the inspectorate structures of the Directorate of Alternative Education Systems (AES). MoEST personnel are actively consulting in program decisions and have benefited from EDC training and material support. Personnel of the MoEST are most appreciative of USAID and EDC support. Conclusions: Radio program production should meet project goals, but the quality of the audio programming has yet to undergo extensive scrutiny and validation. Although the evaluation team noted overall positive responses from listeners and teachers, this was not sufficient to render any judgments as to effectiveness or impact. Based on best practices, many programs should be edited to include longer pauses to allow for smoother classroom use. Print materials are of high quality. Radio program delivery: Radio instruction is a challenging format under any conditions, and is rendered especially complicated by the Southern Sudanese context. Under the most optimistic scenario, FM radio coverage, as projected through the end of the year, can be expected to extend to less than half the area of the country. Even in urban areas weak signals, moderate FM signal amplification by the radios distributed by SSIRI, too few radios, and large classes tend to impede use in classrooms. Shortwave radio is very problematic for regular SSIRI learner groups (Annex 5). While EDC has succeeded in resolving the issue of lack of power and battery expense by using wind-up and solar charged units, the aural effectiveness of single Freeplay radios is questionable in classrooms with more than 50 students, much less in classrooms with 180 or 200 as we witnessed. Many students simply can’t hear the programs. Multiple radios per class is an advised remediation strategy. Also important is the fact that radios and, in the case of the Learning Village series, SSIRI teacher’s guides—the foundations of the SSIRI primary courses—have not been distributed to some teachers who were trained in their use. Many teachers with radios have been transferred, or have quit the profession SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION viii and taken their units with them. A number of teachers trained and equipped with radios have quit using them due to low English skills or other frustrations. Broken units, non-functioning aerials, inaudible sound, and broadcast hours that coincide with school recess are but a few of the reasons cited for non-use. The evaluators were concerned that such feedback information has not been adequately integrated into SSIRI implementation. To a degree SSIRI’s technical problems are a case of “bleeding edge” adoption. Inexpensive solar or wind-up digital audio players of desired specifications may be hard to find. Interactive Radio Instruction (IRI) learners should be exposed to clear strong sound. There are currently no such audio signals reaching SSIRI learners’ ears in many if not most SSIRI classrooms and learning spaces. EDC has done some research on digital audio playback hardware options and provided some information on this research through its quarterly reports, but more needs to be done. To date, experimentation or piloting of MP3 technologies for various class sizes has not been of sufficient breadth nor documented sufficiently to guide purchasing on a large scale. However, we see this as an area of great promise for effective SSIRI implementation. EDC continues to grow SSIRI support staff, many of whom are posted in the countryside, but most lack sufficient training to offer effective technical assistance to teachers or to manage field operations. In many aspects of the SSIRI delivery system, there is an absence of sufficient feedback, systematic analysis, or ongoing evaluation that is critical to the success of such a challenging program. The teacher training program--Professional Studies for Teachers (PS101) – has been produced but has not been launched, despite the pedagogical shortcomings of a corps of some 25,000 teachers, 80 to 90 percent of whom are without formal training or facility in the English language. Instructors face enormous difficulties in dealing with classes regularly exceeding 100 or even 200 children, the great majority of whom are non-English speakers. Although other donors and NGOs are attempting to implement in￾service training throughout Southern Sudan, with insufficient curricula or resources, EDC has not yet made a systematic sustained effort to engage in this vital effort. MoEST strengthening is proceeding with SSIRI assistance, but it is faced with multiple challenges. The ministry is staffed with new and inexperienced personnel and is attempting to devolve operations to the States. It is doing so in the face of continuing budget reductions (current funding is US$ 96 million, as compared to US$111 million in 2007 and US$136 million in 2006)1 . This situation raises concerns about sustainability or scaling-up of SSIRI after the life of the project. Given that the GoSS itself is an inchoate entity and the MoEST is its extension, we commend EDC for measuring investments in the MoEST with prudence. Recommendations: All recommendations are designed to promote quality implementation in all SSIRI counties currently, and if the project ultimately goes to scale. This does not necessarily mean less geographic coverage but a need for focus and concentration. The team recommends implementation in demonstration schools, based on a high level of community involvement. This polestar idea should guide the project for the next 12 months. 1. Assuring clear, strong sound in all SSIRI learning environments and easy-to-use programs Teachers should begin to use multiple Lifeline radios in their classes. They should be informed of best radio frequencies for their locations, which may involve new FM or medium wave (AM) stations. As 1 Budget figures provided by MoEST Director of Alternative Education Systems, Kuol Atem, in an interview. These figures do not necessarily establish that these annual amounts were, in fact, appropriated or spent. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION ix soon as practical, the audio programs should be systematically analyzed and edited so that the pauses for response are of a length that allow typical teachers and children to comfortably complete the directed task. Further study should be done on “low cost technologies” (MP3 players) and players should be deployed for many sites, especially for TERBIA classes. 2. Working in communities EDC should adopt more rigorous approaches to site selection, ensuring that the site can either receive quality radio signals or that it has one or more MP3 players. Participation should also be predicated on community interest. This may be expressed in secondment of staff or counterpart designation and/or the willingness to work with SSIRI to create one or more model schools. This will assure appreciation of and faith in the technology by the County, and will enable other schools to learn from their example. EDC should continue to focus on relations with State and County education administrators. No County should receive an Outreach Coordinator before an Outreach Advisor is assigned to the State. Before a second SSIRI staffer is placed at any site, a secondment to the project should be received from the MoEST office in question or a counterpart person should be officially designated to work in parallel with the SSIRI person(s). EDC should aim to secure the collaboration, understanding and support of MoEST’s Basic Education Directorate (at all levels). It is advisable that the MOU between the MoEST and the State MoEs being developed by EDC include explicit approvals for Learning Village usage in primary schools by both the State Directors of AES and of Basic Education. The approval of the central MoEST Directorate of Curriculum should also be sought to support needed timetabling. There is more that SSIRI could be doing with other English language training groups, particularly those supported by Windle Trust. Windle appears to be a natural ally that can extend the benefits of the project to as many as 3,200 teachers. We urge further collaboration. 3. Monitoring and evaluation systems The evaluation team and project administrators are faced with a dearth of information relevant to the quality of program implementation. Although continuance of the SSIRI project would be predicated, in part, on the 2007 EDC in-house study of Learning Village impact in grade 1, (Annex 1), the team’s cursory analysis of the research concluded that it may suffer from significant instrumentation and procedural faults. Given these limitations, the study should not be accepted as reliable or valid without further scrutiny. Similarly, lack of quality feedback on the deployment of SSIRI-trained teachers and their use of radios must call into question the reliability of EDC reports and databases, and their accuracy in meeting USAID performance indicators. It is imperative that EDC launch intensive, objective and well-designed assessments of: effective radio transmission and listenership, incidence of radio use in the classroom, contributions of radio programming to learning, and the effectiveness of radio compared with other technologies. The team frequently heard complaints from teachers about the pace of radio instruction, audibility issues, and lack of radios. But, other than EDC having on order thousands of radios, there is little evidence to indicate that such feedback has been incorporated into revised programs or the delivery infrastructure. We recommend that EDC outreach staff reports be verified, scrutinized for accuracy, reviewed frequently, and responsive follow-up actions rapidly initiated. Similarly, teachers’ radio-use logs should be gathered SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION x and analyzed. In short, if the SSIRI program is to be continued, scaled or sustained, EDC and USAID management must be informed with rigorous and current assessments and feedback from target groups. SSIRI and County Education Office staff lack skills in instructional technology and media-based learning, as well as in follow-up support and monitoring. Even some senior SSIRI staff appear to have little understanding of effective instructional methodologies. As a result, training programs for teachers and trainers may be of marginal quality and impact. We recommend immediate international technical assistance to craft and implement innovative staff training and training of trainers, which are so vital to the success and sustainability of programming. There should also be more frequent and broader field monitoring by EDC, SSIRI and USAID. Although security and distance barriers plague travel to the field, the project would benefit from more informed decisions that can only be formulated through hands-on and direct observations. 4. Investing in teacher training, especially in-service training. The core of SSIRI’s PS101 teacher training program was produced during the SBEP effort and it is still to be applied. Small investments of time and resources would validate the program to provide support to teacher training programs, some of which are now in implementation and without content or learning resources support. We note particularly the Windle Trust’s in-service English language training project. We urge EDC to join with initiatives supported by the Multi-donors Trust Fund (MDTF) to launch the 12- part SSIRI teacher training program (PST) during the next project semester. A high priority should be to complete the next 24 programs in the PST series to support immediate teacher training. It will be a major contribution if SSIRI can provide the inputs to get the first year of the CDC’s recently designed in-service program off the ground. School-based/focused SSIRI training strategies should be designed so that all grade 1-4 teachers in schools selected for radio intervention can be trained, and sufficient radios supplied to meet school needs. This will ensure continuity of radio use if teachers are sick, transferred to other sites, or leave the school. Additional school-based strategies should be explored to assist teachers in improving their English, perhaps through simple English Resource Corners enabled through MP3 technology. Re: Continuation The SSIRI program has significant potential. With sufficient remediation of current implementation problems and attainment of greater project effectiveness, we believe the benefits to learners throughout Southern Sudan will be significant. If such is accomplished in the final year of the project, or if assessments show it to be attainable the following year or two, continuation will be warranted. If an additional year or two is required to attain such effectiveness, a short-term extension through a further Modification may be warranted. A longer term-project extension may call for another approach. The operation was launched through a central USAID mission buy-in facility (DOT-EDU) and has been replenished via a series of Cooperative Agreement (CA) modifications. If USAID is contemplating a long-term extension of the operation (e.g., 5-years), it should carefully weigh the merits of a competitive procurement alongside those of guaranteed management continuity. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 11 I. INTRODUCTION Southern Sudan is one of the lowest-ranked nations on Earth by any index of human development. In the field of education, the area could arguably be considered a candidate as the poorest. The UNDP classifies Southern Sudan in the very lowest decile of literacy, school access, gross enrollment and primary school completion. Its burgeoning population is composed of a myriad of language and ethnic groups that have little in common. The current Government of South Sudan (GoSS) language policy is English instruction for all grades over P3. But, the great majority of teachers use local Arabic in instruction, especially in the more northerly areas. Additionally, many returnee teachers from the north are “Arabic-patterned” teachers. Five decades of civil war, inter-tribal conflict, and internal and external displacement have sapped coherence from the educational system and ongoing conflict in the “Three Areas” that border Khartoum￾controlled northern Sudan remains a constant. As this mid-term evaluation began – in early May, 2008 – hostilities resumed anew in the Abyei region of the Three Areas. Reports are that many have been killed and injured and it is estimated that as many as 90,000 persons have fled the region as refugees. As a result of this and other rising tensions, security restrictions are increasingly curtailing operations throughout the whole of Southern Sudan. Unfortunately, the current security situation is more the norm than an exception, since independence from the UK in the 1960s. Major upheavals in the 1960s and again in the 1990s led to the creation of the Sudan Peoples Liberation Army (SPLA) and other military factions and outright civil war in the south. By 2000, the Khartoum Government had lost virtually all control of the countryside and maintained tentative presence only in major “garrison towns.” That period saw an unprecedented exodus of refugees to neighboring countries and the largest internally displaced population in the world. This resulted in the creation of a quasi-Southern Sudanese government by the Sudan People’s Liberation Movement in exile in Kenya. There are various estimates of the number of people dislocated, but of the 8-11 million inhabitants living in Southern Sudan, at least 95 percent were adversely affected or dislocated during the violence. Transhumance of this magnitude has transformed the demographic ecology, with the continued relocation of refugees still weakening Southern Sudan’s social fabric, more than three years after the Comprehensive Peace Agreement (CPA) was signed. Throughout this period of upheaval, the United States Government has maintained a neutral position between the Khartoum Government and the people of Southern Sudan. The overriding concern has been for the physical and social welfare of Sudan’s people, irrespective of political alliance or affiliation. The concern has been largely reflected in effecting peace. Even today, some eight years after launching support programs, the USAID/Sudan, Office of Southern Sudan Programs Mission’s principal objective is to foster and preserve peace between the newly formed Government of Southern Sudan (GoSS) and the Government of National Unity (GoNU), based in Khartoum. Immediate goals are to support the CPA as a measure for securing peace; continuation of direct services to the Sudanese people; and with respect to education, strengthening the GoSS Ministry of Education, Science and Technology (MoEST) to sustain innovative educational programs. Because of ongoing support for the Southern Sudanese population and various iterations of their government, relations between the United States and the GoSS remain good and productive. Faced with some of the most formidable, unique and often dangerous challenges, USAID has been partner to the SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 12 establishment and maintenance of GoSS and an especially critical contributor to the emergence and effectiveness of MoEST. Nevertheless, many deeply-rooted and systemic problems remain. Since the signing of the CPA, thousands of refugees returned to settle in former villages with little more than basic household items. They return mostly from refugee and displaced persons camps where the most essential human services were minimal. Depending on their country of exile, many Sudanese were treated as pariahs or ignored and left to fend for themselves. The majority of returnees have lived for the better part of a generation dependent on donor subsistence, and many have developed a culture of dependence that borders on addiction to handouts. They join those who remained in Sudan, or were similarly encamped near Khartoum, and share similar circumstances. Despite, these shortcomings, Sudanese are a resilient people and share the drive and motivation that mobilizes so many of the world’s poor: hope for a better life for their children. In this environment, any positive intervention for child welfare and human dignity, irrespective of its impact or scale, can have a significant impact on social advancement. To quote the USAID Education Team Leader, “even the mere fact that children are attending school, regardless of the quality of education, is a quantum step in human progress for the people of Southern Sudan.”2 It is with this modest, but potentially transformative, goal in mind that USAID has supported the Southern Sudan Interactive Radio Instruction (SSIRI) project since 2004. This report describes a mid-term, formative evaluation of the progress of that operation. II. BACKGROUND – DEVELOPMENT OF THE PROBLEM & USAID’S RESPONSE Southern Sudan USAID operations in Southern Sudan are tempered by its tenuous status as a potential member in the family of nation states. Born of the Comprehensive Peace Agreement of 2005, the area has sustained social, economic and military upheavals for over two generations. USAID must operate on the premise that support is not necessarily extended to the Government of Southern Sudan, but to the people of Sudan. The agency’s priorities are to preserve peace, engage citizens in support of the CPA and to advance human welfare in the region. Although the present Government of Southern Sudan was largely crafted in exile in Kenya, the period immediately following execution of the CPA has seen a massive nation building process. The government may be in a nascent stage of development, but there has been marked and substantive progress in establishing order, basic human services, and rule of law. Primary education in Southern Sudan According to an April 2008 survey conducted by the MoEST, there are estimated to be between 1.7 and 2.1 million students formally enrolled in primary school, grades 1-8. They are served by approximately 3,100 schools and taught by 24,000-27,000 teachers. It is estimated that 90 percent of teachers are competent in Sudanese Arabic, while about 20-30 percent have rudimentary English skills. English and local languages prevail as the core language of instruction only in states near the Ugandan and Kenyan borders, with Arabic dominating classrooms in the Three Areas and the northern and central states of Southern Sudan. 2 Inez Andrews. May 2008 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 13 Primary education is supported and delivered principally by the Government of Southern Sudan (there are few private or NGO-run schools) which receives funding through a joint revenue sharing agreement with the GoNU. This accord was memorialized in the Comprehensive Peace Agreement of 2005. Despite rapid growth of the school-age population and pressures on all educational resources, the national budget available for primary education has been in a state of decline since the CPA went into effect. In 2006, the MoEST had a budget US$136 million, which as seen reductions to US$111 million in 2007 and $96 million in the 2008 school year. The operating budget has declined by about 30 percent, while enrollment, because of increased stability and refugee returns, has likely doubled. The current budget provides for about US$45 per student per year (as compared with about US$9,500 in the US state of Missouri). There are no accurate data, but it is reckoned that about 95 percent of the budget is spent on salaries, with the remainder allocated to administrative costs. Teacher salaries, while also difficult to measure accurately, ostensibly range from a base of US$85 per month for beginning teachers up to US$400 per month for senior teachers. The MoEST also indicated that some 6,100 teachers involved in after-school adult learning programs, such as the SSIRI TERBIA Activity, are to be paid an additional US$235 per month. Were this salary scale a reality, this would place Southern Sudanese teachers amongst the highest paid in sub-Saharan Africa. However, based on conversations with teachers, their pay often lags for up to several months, unwarranted deductions are made by administrators, and the official salary scale is rarely honored. The primary education system is overseen by a decentralized administration in which the MoEST and the Ministry of Finance supply funds to the 10 State Governments. Funds are in turn allocated to County Education Directors who distribute salaries directly to teachers. State and County governments are involved with hiring teachers and monitoring and inspecting schools. Their involvement with schools however is minimal, as they have no means of transport and even less training as instructional administrators. Although personnel in these posts are new and without many resources, most are former teachers and have been involved with a smattering of professional development programs. Since funds are dedicated almost exclusively to salaries, there is no budget for capital investment or for the purchase of texts and learning materials. As a result, schools have not benefited from rehabilitation or expansion in many years. Classes, principally for lower grades that often enroll over 200 students, are often taught under trees or in open, lean-to type structures, where they are exposed to heat, rain and numerous distractions. As students abandon schools in early years, higher grades, characterized with 20- 30 students, benefit from classrooms. In all instances, however, there is a dearth of desks and seats, with pupils relying on stones, fallen logs, or just dirt. Facilities have no electricity, running water, or toilets, even though school enrollments reach near 2,000 pupils in some urban centers. Few schools (only one in 15 observed) have feeding programs, so children are expected to participate in a 6-period learning program from 8:30 AM until 2:05 PM without food. Most teachers use lessons drawn from small workbooks that guide the curricula. These are printed in English, so many Arabic-speaking teachers experience difficulty in their use. Each class has a chalkboard, on which a teacher typically writes a few sentences before retiring to a seat outside the classroom, where they await individual pupils to show them their work. Other than these two learning media (teacher workbook and chalkboard) there are no learning resources to assist learning. As with many developing nations, teacher absenteeism is high and schools are often closed for unofficial reasons. In April 2008, schools were interrupted for nearly a month so that teachers could help conduct the first national census. With malaria and parasititis endemic to Southern Sudan, teachers and students often miss prolonged periods of class with illness. There is little tradition of parental involvement in Southern Sudan schools and in many schools children are required to pay a fee to attend. These funds are SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 14 ostensibly used to ramp up pay for “volunteer” teachers, who often comprise up to a third of the teacher force but receive lesser pay than “government” teachers. Volunteer teachers represent only one group of poorly trained teachers. It is estimated that 90 percent of all teachers lack fundamental pedagogical skills. The majority comes to schools with little more than primary schooling, and perhaps less than 10 percent have had post-secondary training in either of the two teacher training institutes (which together graduate some 150 teachers annually). Although there are a number of well-trained teachers (mostly trained in normal schools and universities in Uganda and Kenya) who have applied for appointments, State education budget shortfalls have severely limited hiring new qualified teachers. Central Equatoria State, for example, the fastest growing urban district in Southern Sudan, has not hired a new teacher in three years. There are few reliable estimates of children who do not enroll; however, a CCRI survey in Jonglei concluded that “only 30,000 out of an estimated 100,000 school age children are enrolled in school in the eastern State.” The survey also found that adult literacy is below one percent and the worst affected are women who have a literacy level of below 0.05%. “There are an estimated 500 teachers teaching in 80 makeshift schools (under trees and Tukuls). About 90% of those teachers are untrained volunteers.” As to primary school dropout rates, simple arithmetical calculations support an estimate that only about one in four pupils reach grade 8. Without a substantive increase in the education budget to meet current needs and prepare for a burgeoning school-age population, the quality of education will inexorably continue to decline, school abandonment will accelerate, and more children will choose not to enroll. Systemic shortcomings notwithstanding, the Southern Sudanese primary education is in an inchoate state of development. Schools, however limited in space, resources and amenities are in operation. Teachers, despite their limited capacities and high absenteeism do hold classes, and receive sufficient pay (however irregularly) to remain in the profession. GoSS, State and County education administrators can be found at work most times of the day, and there is some semblance of a structured learning program in schools. There is substantial if guarded cause for hope. Problem Stated Southern Sudan poses a number of unique challenges to conventional human services programming. After nearly 40 years of intermittent and savage conflict, most of the area has achieved a degree of stability that fosters the return of refugees, many of whom have been away from Sudan for a generation. They have arrived with internally displaced persons from camps near Khartoum to deserted villages, fallow fields, a near-devastated physical infrastructure, limited or no human services, and deeply￾ingrained fears for personal security. Families and traditional social units have been dispersed throughout the world and most have survived the last 15 years as dependents of food programs and a smattering of educational services offered by host governments. The cycle of transhumance and expulsion from one nation to the other, each with internal problems spawning internecine conflict, has left many Sudanese with indelible fears. Few of those who stayed or fled have had exposure to life skills or values needed for self sustenance. As a consequence, Southern Sudanese are only just now beginning to develop an ethos of work, self reliance, or trust in higher authority. The problem is further complicated by living in an area that has no status as a nation-state. The Government of Southern Sudan does exist for certain, but there are no assurances that it will endure beyond the 2011 plebiscite. Where formerly there were defined ethnic groupings, today there are a mosaic of peoples and languages. After long association with the Khartoum Government, most older Southern Sudanese speak Arabic and SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 15 very limited English. But, some youth and refugees converse in English and GoSS has dictated that English is to be used as the language of instruction. A large percentage of rural Sudanese speak neither Arabic nor English and must rely one of some 54 local languages. Illiteracy rates are among the highest of any region on Earth. Limited communications and travel pose formidable obstacles to any social or economic transactions. And the situation continues to deteriorate. As security is perceived to improve, more refugees arrive, overwhelming the fabric of human services. Health and educational services are taxed beyond their already limited capacity. While families are anxious to educate their children they have no resources to devote to anything but survival. Although the GoSS invests some six percent of the annual budget in the education sector this fails to meet even minimal programming standards. Humanitarian aid for Southern Sudan faces the contradiction of extending relief on the one hand and trying to build internal capacities on the other. How much energy and resources are to be spent on each fuel a continuous debate. The dilemma is sharpened by a flagging interest in the international donor community, which has reduced earlier commitments of some US$150 million to education to less than US$50 million. And trumping all solutions is the constant threat of a renewal of conflict, which has already been reignited in the Three Areas, triggering another round of dislocation, suffering and death. History of USAID Intervention in Southern Sudan USAID-Sudan launched support for the people of Southern Sudan with the Sudan Basic Education Program in 2002. The purpose of the program was to support the development of an education system where one none existed. The Cooperative Agreement was managed by CARE-USA as the prime contractor with American Institutes of Research and the University of Massachusetts School of International Education. The program began as support at the local and county level and later developed into support of a newly-formed Secretariat of Education for Southern Sudan, to construct schools, and to establish a minimal education infrastructure. The goal of this project was to increase access and quality educational services directly to the people of Southern Sudan in the promotion of peace. As Southern Sudan became a more consolidated and formal political entity and peace appeared on the horizon, USAID chose to expand programming into the south through radio transmission of English language basic education and civics programming for adult and out-of-school audiences, as well as for in￾school audiences of primary school children and their teachers. In 2004 USAID/Sudan selected a media￾based approach to delivering vital basic education. The USAID Mission chose a global education facility—DOT-EDU—as the delivery mechanism and chose EDC the executing contractor. EDC brought substantial programming experience in radio and distance education to the USAID education portfolio. The project was driven by USAID/Sudan’s strategic objectives to deliver direct relief services to Southern Sudanese people and to provide “Improved Equitable Access to Quality Education” (SO 6). Absent a formal government in the area, and with ongoing civil and military conflict, it was decided that EDC would collaborate with a nascent ministry of education based in Nairobi. The goal early on was to develop and launch radio programming directly to schools, and to help set the stage to establish and support a more formal ministry when Southern Sudan gained a degree of autonomy and stability. During the initial days of the project, EDC established a media production center in Nairobi, Kenya to design and broadcast two major radio initiatives: the Learning Village, which broadcasts lessons set in a virtual Sudanese village, and TERBIA, which broadcasts radio programs aimed at improving English and civic education for adults and out-of-school youth. USAID/EDC programming has rapidly devolved operations to Southern Sudan. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 16 With the signing of the CPA in 2005 between the SPLM and the Government of National Unite (GoNU), there was a basis for ongoing peace and development in the country. USAID noted that there are high expectations that peace would bring programs to improve education, health, water, livelihood opportunities and other basic services – the “Peace Dividend.” In reality, however, the Government of South Sudan (GoSS) has lacked the capacity to respond to these needs. The fear was that if the GOSS failed to meet rising demands and expectations, the risk of conflict would increase. In this volatile situation, USAID’s strategic objectives were modified a number of times. Under the Fragile State Strategy USAID’s strategic objectives relevant to SSIRI became “Averting and Resolving Conflict” (SO 9) and “Promoting Stability, Recovery and Democratic Reform” (SO 10). Currently USAID and EDC-SSIRI are focused on “Investing in People”, “Supporting the Comprehensive Peace Agreement (CPA)” and bringing peace dividends to the citizens of Southern Sudan through improved educational opportunities. USAID now seeks to assist the Ministry of Education, Science and Technology to develop policy, programs and procedures in all areas of basic education. USAID chose to implement direct educational services to Southern Sudanese schools and children through the medium of radio, which provides an opportunity to impact on the largest number of beneficiaries, rapidly and with the least cost. USAID selected EDC to create radio programming and complementary instructional support services, implement the programs and explore alternative learning technologies. The SSIRI project is major element of USAID education investments in Southern Sudan. It includes supporting the use of educational technologies through orientation and familiarization activities, training, developing improved policy structures, and creating and implementing appropriate applications, especially in teacher training. SSIRI Program Description The SSIRI operation is based on the premise that radio is the most efficient and effective means for extending good education to populations that are dispersed across wide and remote areas and in need of qualified teachers. Instructional programs are intended to provide knowledge and skills needed for individual and national development, as well as support USAID’s goal of fostering peace and acceptance of the CPA. EDC implements five broad and interrelated programs: 1. Provision of direct instruction to primary school teachers and children via radio: the Learning Villages series; 2. English language and civic education to out-of-school youth and adults by radio: the TERBIA program; 3. Professional Studies for Teachers through face to face training, outreach and radio support; 4. Institutional strengthening to the GoSS MoEST (especially teacher training institutes) and its executing entities: State, County and Payam educational agencies; 5. A final and enabling strategy is to explore, test, pilot and validate other alternate and low cost technologies that might augment or replace radio where that option is not available or of quality. The Learning Village comprises interactive radio programming for grades 1-4, and is based on the Southern Sudanese primary school syllabus. Lessons (in English) are designed to support classroom instruction in English, math, local language literacy and life skills. The 100 lessons for each grade are designed and sequenced to be broadcast daily, for 20 weeks a year, in half hour segments. Actual programs are 28 minutes long. Lessons are designed to be engaging, interactive and incorporate pupil-centered instructional practices. They are designed and created by Southern Sudanese scriptwriters and producers whose target audiences SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 17 include the nearly 80 percent of teachers with limited English skills. Not only are the lessons intended to structure and support English learning by students, they also help Arabic-only teachers acquire English skills themselves, in their fulfillment of requirements to implement the national curriculum in English, which is the policy of GoSS MoEST. Learning Village programs are extended to the classroom by training selected teachers in the use and care of radios, providing them with a comprehensive teacher’s guide, and offering follow-up support by outreach staff to ensure the radios are in use. During the last year, project administrators have taken steps to integrate SSIRI training and support into the routine of County Education Officers, who are expected to coordinate the program after the life of project, as well as support IRI inspection staff at the MoEST level Teaching English through Radio-based Instruction for All (TERBIA) is a radio-based program produced by EDC-SSIRI to deliver English, civics, life-skills and nation-building education to out-of￾school youth and adults. Although the program is framed as an English instruction initiative, an important collateral objective is to foster understanding of the CPA. Topics of the TERBIA programs change regularly and include themes such as disarmament, demobilization, landmine awareness, women’s rights and other programming in support of civil society and good citizenship. TERBIA targets audiences with four levels of English skills through the broadcast of 60 lessons per level (low beginner, high beginner, intermediate, and advanced), for a total of 240 half hour programs. TERBIA B (for beginners) Levels 1 and 2 target Sudanese with little or no English, and are designed for listeners who meet in groups with teacher/facilitators. TERBIA B1 began broadcasting in late 2007 and TERBIA B2, was launched in May 2008. TERBIA-A (advanced) is intended for those who already have some competencies in English. A focus of the advanced broadcasts is to expand conversational English as well as stimulate learning and dialogue in civic education, independently or in small groups. The program began broadcasting in early 2007. TERBIA is transmitted in some FM reception areas and nationally by shortwave in the late afternoon (4 PM), or by shortwave in the morning (9:30 AM). While it is aimed at the general listening public, lessons are also intended to supplement instruction for adults and out-of-school youth enrolled in programs of the Alternative Education System (AES), such as the Accelerated Learning Program (ALP) where students can receive eight years of primary education in four years. TERBIA is used as an adjunct to this curriculum in an extension period of the same learner group meetings. SSIRI has trained teachers to organize and lead formal TERBIA classes, often in primary schools after their regular teaching hours. The MoEST has signaled intent to provide supplemental incentive pay to such teachers, who would weave TERBIA Broadcasts into the literacy and accelerated learning programs designed by MoEST. The need to address English learning among those with modest skills led to a decision to create TERBIA￾Intermediate. These programs are under development and broadcasting is projected to begin in late 2008. Professional Studies for Teachers (PST) is designed as a radio-based, distance learning course to improve instructional practices of Southern Sudanese teachers. Since the preponderance of primary school teachers lack even the most basic training in subject skills, pedagogy and instructional methods, PST is also intended to help them earn certification. The evaluation team notes later in this report that there are approximately 25,000 teachers in Southern Sudan, yet only 19,000 actual teachers are listed on the payroll. The remainder (and many others on the payroll) are local volunteers likely lacking even the most fundamental teaching skills. The PST course currently consists of 12 programs/modules that focus on effective classroom management, soon to be supplemented by additional audio programs/modules to further support in- SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 18 service teacher training. According to EDC’s Cooperative Agreement implementation plan, the PST was to launch by 2007 and be integrated into broader and more extensive in-service professional development initiatives supported by MoEST and the Multi-Donors Trust Fund (MDTF). While a four-year curriculum has been completed, implementation is currently stalled. Institutional Strengthening of the MoEST and local education offices is an expectation implicit in the recent Modifications (No. 4, No. 9) of EDC’s Cooperative Agreement with USAID. This is to take several forms: installation of an interactive radio inspectorate and capacity-building in the central GoSS￾MoEST so that SSIRI services will be sustained after the life of the project. Since the GoSS MoEST has devolved most school operations to the States, and they in turn to Counties, EDC’s development strategy is to upgrade the quality of educational management that is nearest to stakeholders and clients: schools and students. Thus, EDC intends to focus on developing local capacities at the inspectorates of the County Education Offices and the Payams (local governments). Strengthening of MoEST institutions has also included building the capacities of the three pre-service teacher training institutes (TTIs). In this instance, EDC has chosen to strengthen the TTIs as vehicles for accomplishing the goal of implementing alternative and low cost learning technologies. USAID has had a longstanding commitment to creating computer centers to develop innovative teaching resources through distance, e-based and video instruction. Alternative learning technologies. USAID and EDC have recognized the limitations of radio technology in reaching Southern Sudanese student and teacher populations. Shortwave signals fluctuate widely over distance and time of day, making them difficult to hear. FM broadcasts are of more uniform quality but the range of effective broadcasts from each transmitter is greatly limited. Similarly the timing of radio broadcasts presents a problem. TERBIA classes, for instance, tend to form and operate for adults after regular school hours and outside broadcast schedules. A fixed schedule for all listeners is simply not possible. To meet the needs of current and potential teachers, EDC and MoEST have opted to embrace innovations and broader technologies than that afforded by radio. In addition to investments in IT at teacher training institutes, EDC has proposed to expand into other audio options, such as MP3 players, which offer portable and unscheduled training (via a USB flash-memory stick, SD card, etc.). This will allow teachers to access programs when they want, as opposed to depending on radio broadcast schedules. EDC has also noted that large class sizes militate against the effectiveness of single Freeplay, wind-up (crank-style) radios, which have a limited range of audibility. The current Cooperative Agreement calls for the piloting and validation of MP3 players plus small speakers for use in classes of appropriate (small) size. Similarly, CD/FM/USB-MP3 players “boom boxes” are to be tested for viability and utility in large classes. EDC’s work plan also called for the installation of VSAT facilities in a number of TTIs by mid￾2008, the testing and validation of alternate “low-cost” learning devices by early 2008, and expansion of the chosen device(s) by late 2008. III. PURPOSE OF THE EVALUATION As the SSIRI project moves into its final year, USAID and EDC must make critical decisions for program adjustment and revision to accomplish immediate project goals before the termination of the project in June 2009. In addition, the programming horizon is rapidly approaching for the design of new and continuing USAID educational operations. Although EDC and USAID have maintained constructive and positive work relations, and EDC has largely met expectations, it is appropriate to assess the SSIRI effort from an objective and independent viewpoint. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 19 It is important to note the scope and limitations of the mid-term evaluation. A brief, three-week mission is hardly sufficient to conduct an in-depth appraisal of nearly five years of work, especially given the complex, changing and unstable environment in which the SSIRI project has been executed. Although a valid assessment of the impact of SSIRI interventions would include independently measuring its effect on student learners, the evaluation had to focus instead on intermediate effects of the operation: teachers’ behaviors and perspectives; SSIRI products and presence in classrooms; review of studies based on primary data research; information from many project reports and documents; and attitudes as measured though informal discussions. Nevertheless, the evaluation yielded sufficient primary and secondary data that, combined with direct observation, allows us to craft a composite assessment of the performance of EDC, the value and impact of project outcomes, and to identify strengths, limitations, emerging trends, and remediation strategies. The MSI team is confident that findings, conclusions and recommendations are sufficiently substantiated to relatively high and acceptable degree of confidence. IV. RESEARCH DESIGN AND METHODOLOGY Upon arrival, the MSI team was joined by a Sudanese national to serve as translator and assist in field support. After an initial debriefing with the USAID CTO and the EDC SSIRI Project Director, a comprehensive work plan was submitted to USAID; that document is attached as Annex 1. Limited by time and resources, it was decided to draw information through three media: (see list Annex 1) 1) Desk review of many project documents, including Cooperative Agreements, Modifications, progress reports, work logs and thematic studies, etc. 2) Interviews with key stakeholders, including GoSS MoEST personnel, SSIRI project staff, various USAID staff, other USAID education sector contractors, teachers, administrators, radio station managers, outreach personnel, project-related and non-project related NGOs working in Southern Sudan and ad hoc informants; and, 3) Primary data gathering was accomplished by school and training site visits. Given that the broadcast horizon is limited to several hours in the morning for Learning Village (for in￾school pupils) and a few hours in the afternoon for out-of-school learners using TERBIA, it was decided that no more than three (two for Learning Village and one for TERBIA) schools per day could be visited by the team. The team agreed to select from schools in five states where the project has program presence and in the Three Areas—the northern zone of transition—which were added to the SSIRI service area by the 2006 Modification of the 2004 Cooperative Agreement. The team agreed to be accompanied to all meetings by two representatives of USAID, one from the MoEST, and one from EDC, on the provision they act as observers and not become active participants in observations or interviews. Based primarily in Juba, the MSI and aggregate team’s travel schedule (Annex 1) included land travel in and around Juba in Central Equatoria State and to project sites in Torit and Arapi in Eastern Equatoria State. In addition, the field plan called for air travel to Yambio and Maridi in Western Equatoria State, Wau in Northern Bahr al Ghazal State, Malakal in Upper Nile State, and two of the Three Areas. These areas are much too inaccessible to visit by land. The team also agreed to present an overview of the study’s results prior to departure. The evaluation mission encountered a number of barriers to the proposed work plan. The desk review and interview process proceeded according to expectations. The team found USAID and EDC documents graciously provided on call. We wish to express our gratitude to USAID and the SSIRI management for SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 20 their full and candid disclosures. We had broad access to MoEST officials both at the GoSS, State and County levels. It proved very challenging to conduct observations of schools, which was considered the only means to gauge the impact of SSIRI. The goal of the study was to visit schools with no or little forewarning, to observe the level and quality of radio usage in actual classes, and to conduct follow-up discussions with teachers on the impact of the SSIRI program on student learning. We found that Southern Sudan primary schools had originally opened their doors for the new academic year in April, but the term was interrupted—so that teachers might work in conducting a national census—until early May. Consequently, we found a number of schools that were still in the process of re-organizing for the school year. Due to the census interruption, radio stations were in various stages of airing broadcasts of the Learning Village and TERBIA, so there were a number of instances of broadcast delays, rescheduling or teacher confusion over program schedules. For most of the areas selected for study, however, the team was able to find schools open and visit them during broadcast times. The outbreak of open warfare in one of the Three Areas to be studied led to the cancellation of travel, at the direction of USAID security offices in Khartoum. USAID’s Regional Security Officer issued a high￾security alert/travel ban that prevented the evaluation team from visiting the far north and any of the Three Areas. Thus the analysis of project activities in the Three Areas, which were targeted by the 2006 Agreement Modification No. 4, relied only on project documents describing activities and on conversations with SSIRI staff. Consequently, findings for other States visited and included in this report cannot be extended to these transition areas. There were sufficient data collected from the five States visited to render conclusions about the 10 States of Southern Sudan. By extrapolation, salient findings and recommendations might also be generalized to the Three Areas, although the MSI team is not comfortable making such assumptions. The three transition areas lie mainly within the sphere of the government in Khartoum, where it is reported that teachers who deliver instruction in English instead of Arabic are purposefully denied pay, and where most schools are likely beyond the current areas of FM radio coverage for SSIRI programs. Despite the inability to observe SSIRI operations in the northern transition areas, the team was able to visit 15 schools and adult learning sites in five States with SSIRI-trained teachers who were expected to be providing SSIRI instruction to learners using radios or MP3 players supplied by EDC (Annex 3: Table 1: Summary of SSIRI School Observations with Radio Reception Notes). The team was also able to visit the nation’s three teacher training institutes (TTIs) where internet, video and other technologies are being supported by EDC. The team interviewed key stakeholders in MoEST, upper level education Ministers and officials in five States, nine County Education Directors, approximately 85 teachers and school directors, and key representatives of the international donor community and their executing NGOs in the field. Although disappointed that we were unable to assess SSIRI activities in the Three Areas, and only a limited number of schools in routine operation, the team is confident that the methodology met expectations for reliability and validity. V. ANALYSIS AND FINDINGS The Scope of Work for this evaluation called for response to ten major questions. For the sake of flow and readability we have grouped findings under multiple questions. A few questions call for conclusions and will be referenced in that section. Major questions: 1. Does the current project respond to the GoSS/MoEST’s desired directions for Southern Sudan? SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 21 2. What is the nature and quality of the relationships between SSIRI and its local partners, communities, other USAID cooperating agencies, other NGOs and donor partners? 3. Are the technical areas and current approach appropriate for a USAID/Sudan’s follow-on investment? How should any future USAID investments be implemented/refocused? 4. Assess the overall impact of the SSIRI Program to date: To what extent is the program having an impact on access to primary education and English language literacy in Southern Sudan? 5. Assess program performance and progress towards achieving program results in all the key program areas as measured against targets established in the cooperative agreement, annual implementation plans and the performance monitoring plans. Are the results commensurate to the USAID investment in the program? 6. Provide a brief description of the program outcomes, deliverables, and products (IRI Programs, broadcasts, teachers’ guides, teaching and learning materials, etc). To the extent possible assess the quality of the deliverables. 7. Describe and evaluate the program accomplishments. 8. What impact has the program had on development of technology- based education in Southern Sudan? 9. Assess the quality and performance of EDC in managing program implementation. 10. What strategies has the program adopted in order to bridge the gender gap in education in Southern Sudan? As the project now enters its fifth and final year, many of its deliverables and targets have been met while others seriously lag (Annex 4, Table 2: SSIRI Deliverables and Proposed Activities in Contract or Modification and Status of Each). We find that the project has recently focused on establishing a presence in many more counties and creating the administrative conditions to manage the project over a much larger area. The project is now nearly doubling in staff size by recruiting 29 new staff to compliment the 30 now employed. How it manages the growth in the size of staff and its implementation in the next 12 months is an important issue given observed shortcomings in performance in Learning Village schools and TERBIA learning spaces. Process Findings Re: Questions 1 and 2: “Does the current project respond to the GoSS/MoEST’s desired directions for Southern Sudan?” and “What is the nature and quality of the relationships between SSIRI and its local partners, communities, other USAID cooperating agencies, other NGOs and donor partners?” MoEST Directions: In interviews with two state-level Ministers of Education and with three state Directors General and a number of central MoEST officials including the Director of Alternative Education, we heard no negative comments about EDC or the SSIRI project; constructive criticisms. Some we spoke with were not well-informed about the project, but to the degree they understood it, they saw it building the English capacities of teachers, pupils and out-of-school learners. This is a central plank of MoEST policy. EDC has a high level of credibility, in part because it assisted the Secretariat of Education before there ever was the GoSS or the MoEST. “The Secretariat could not implement such a program so EDC did it and we’re grateful.”3 The strongest criticism of the project that we heard was that EDC needed to accelerate turning over management of the project to the GoSS. 3 Kuol Atem, MoEST Director of Alternative Learning Systems, interview May 19. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 22 Quality of EDC Relationships: EDC is working to establish further formal relations between the SSIRI project, GoSS MoEST and its State offices through a three-way Memorandum of Understanding.4 Further institutional formalization of the project is taking place through a three-way MOU between MoEST, Miraya Radio, and EDC; a Partnership Agreement with Mercy Corps for outreach of TERBIA in the Civil Society Resource Centres (CSRC) in Agok, Abyei, Kurmuk, Malualkon, Kauda, Yei, Lainya; a Letter of Understanding with Bakhita Radio; an MOU with AMURT for services in Aweil East County with possible extension in Aweil South, Aweil Central and Aweil North; and EDC has fostered an agreement (now signed) between the Ministries of Education and Information in Eastern Equatoria State for broadcasting SSIRI programs over the impressive new radio station in Eastern Equatoria (Torit). Facilitating Work of the State and County Education Offices: SSIRI coordinates its activities through the office of AES at the central MoEST. SSIRI has two Senior Outreach Advisors who are each responsible for five states and it plans to hire one more. It is now placing Outreach Advisors at each State Education Office and Outreach Coordinators at County Education Offices. At the time the evaluation began SSIRI planned to be in 25 counties in 2008-2009. SSIRI sometimes trains state and country counterpart staff. In February 2008 it ran a workshop for state and county level staff to promote the understanding that SSIRI is part of the MoEST. SSIRI-MoEST relationships are cooperative. Identification of schools and learner groups is done by the state and county offices. While it is an intention of their cooperation, SSIRI Outreach Coordinators are not always being provided with a clear counterpart from the state or county MoEST office, or from the central MoEST. Some counties have SSIRI staff counterparts who are deeply knowledgeable about and active in the project (e.g. Wau - Jur River County); while others either have no designated project counterpart or whoever is identified by the organization as being closest to the project knows little about it. Effective site monitoring and support – essential for SSIRI implementation – is clearly being hampered by the absence of effective transportation. SSIRI has purchased 4 vehicles for central staff. USAID has approved purchase of 7 more vehicles for staff based at state education offices. SSIRI has also purchased 25 motorcycles or quad bikes for its county-based staff and is planning on purchasing nine more. The GoSS highly appreciates that EDC has hired nearly all Sudanese staff. In South Sudan only three are not Sudanese (Table 3 below). Through their speech or radio script word choices, however, the Sudanese actors and writers in Nairobi sometimes reveal to listeners that they have been living out of the country for many years. TABLE 1: SSIRI STAFF COMPOSITION SSIRI Location Number of Employees Sudanese Nationals Nairobi based 25 12 Sudanese Sudan based 42 39 Sudanese (all but 3) Total 67 51 Sudanese Other EDC Relationships: In addition to the relationships indicated by the MOUs cited above, EDC has a significant relationship with the National Democracy Institute (NDI). This group has bought wind-up and solar radios and provided them to SSIRI and distributed some of them for the SSIRI project in the Three Areas. In 2006-2007 it provided 1,357 Freeplay Lifeline radios to the project and all have been distributed (with some losses). In 2008 it has delivered another 2,500 to SSIRI’s storage in Juba, and another 2,500 are on the way. 4 Draft MOU now under GoSS review “Between The South Sudan Ministry of Education, Science and Technology (Ministry), State Ministry of Education (State) and Education Development Center”, provided by SSIRI. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 23 EDC is working cooperatively with other education programs, both those funded by USAID, such as Creative Associates HEAR program, and those funded by others, such as Windle Trust. Windle trust has won the confidence of SMoEs (e.g., Wau, Malakal). In Wau 400 teachers are participating in Windle’s intensive English program (3 hours a day, 5 days a week). In Malakal nearly 300 teachers are being served. Windle has tried integrating TERBIA into their programming in some places and they are open (especially in Malakal) to doing more. There is significant geographical overlap between SSIRI and HEAR, with emphasis on service in the Three Areas. A SSIRI Outreach Advisor and an Outreach Coordinator spent two weeks in July 2007 in Abyei working with HEAR staff on several tasks, including: orienting HEAR education and health staff on IRI methodology and The Learning Village program; reviewing and helping to finalize the HEAR 5- day training program; taking the lead in training the first cohort of Abyei teachers; and assisting in the development of a project master plan and detailed one-year implementation plan. (The team was not able to speak with staff from HEAR.) The team did speak with AED’s TAP program COP and Deputy COP. Relationships between EDC and AED are collegial. EDC has relationships with a number of others as indicated by the MOUs. In addition to those listed, EDC has relations with CCRI in Nyirrol County for TERBIA B groups, ROOF in Kurmuk for TERBIA B groups, and Internews (radio) for possible future radio broadcasts. EDC Nairobi – MoEST Maridi Relationship: SSIRI began with an emphasis on materials production. Production efficiencies called for a Nairobi based operation, which is where EDC’s Sudan Radio Service was and is housed. The original plan to soon create production capacity in Maridi and move SSIRI production activities there was delayed but is again being pursued by EDC.5 Architectural drawings for a Maridi recording and audio production studio have been completed and are being reviewed (we do not know by whom). Consultants are scheduled to arrive in Maridi in June 2008 for capacity building activities at the TTI and the Curriculum Development Center. First production there will likely be additional PS101scripts. Other Relationships: With regard to a role for the diaspora in the project, EDC and USAID staff both speculated that it may be due to a greater number of returnees that recent candidates for jobs seem to be of a higher standard of education and work experience than in former years. This could augur well for the hiring that SSIRI is now doing. With regard to the private sector, a few examples of foreign private companies providing assistance to SSIRI exist (e.g., Videomaker magazine’s producer). Otherwise, we met virtually no one from the private sector operating in Southern Sudan and so we were unable to assess potential alliances with this sector. Re: Questions 3 and 4: “Are the technical areas and current approach appropriate for USAID/Sudan’s follow-on investment? How should any future USAID investments be implemented/refocused?” and “Assess the overall impact of the SSIRI Program to date: To what extent is the program having an impact on access to primary education and English language literacy in Southern Sudan?” When the technical factors of audio delivery are properly handled, there is clearly great potential for raising the quality of instruction throughout a very wide geographical area. Currently, in too many SSIRI learning spaces audio delivery and use is not up to minimal standards. A primary touchstone of interactive radio instruction is that the learners should be “bathed in sound”. Otherwise one is dealing with an alternate instructional design – such as solo audio coaching for teachers via quiet audio in the classroom. 5 Letter to Edward Kokole, Acting Director, MoEST Quality Promotion & Innovation, fr. Tom Tilson, May 8, 2008. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 24 This is not what SSIRI or any IRI program is designed to do. If the audio source cannot be heard by most of the learners, the choice of hardware and/or audio delivery channel needs to be re-thought. The project began with shortwave (SW) because that was all that there was available at the time. Sudan Radio Service (SRS) works well and is the second most listened to radio station on any band in Southern Sudan. This is because it conforms to the norms of SW broadcast – generally news programming for individualized listening or listening in very small groups. SW is not generally adequate for schools broadcasting, which is almost always a daytime activity when SW reception is subject to degradation. Based on our tests with multiple radios (both Freeplay Lifeline and Freeplay Ranger) we established that the more widely available SSIRI SW signals, as received and amplified on SSIRI’s Lifeline radios, were not adequate for classroom instruction on most days that we listened. This condition leads to unpredictability about the lesson that inevitably frustrates teachers and discourages them from sticking with the programs. Thankfully, much clearer FM signals are now heard widely in and around major cities and towns and SSIRI is promoting FM to Learning Village and TERBIA users as a first choice. The team found an unexplained anomaly of the Lifeline radio where the Miraya FM signal is quieter on the standard frequency (101 FM) than on an unexplained phantom frequency (around 95 FM). This only happens on the Lifeline radio. SSIRI staff and Learning Village teachers had no idea that this was the case. Teachers in more than one observed school – like the P1 class teacher in Yabongo Girls Basic School in Yambio (pictured above) -- who were tuning to Miraya FM were bending over to put their ears near the Lifeline’s speaker, then rushing to fulfill the directives from the program, then bending again to try to hear the next section of the program while students were still responding to the last instruction. Sonically it was inadequate and it put the teacher in an uncomfortable position. In such classes, which may have as many as 180-200 pupils, those in the back have no chance to hear the program. School heads did not seem to know that two radios can be positioned at both ends of a classroom to assist audibility for those at the rear. This is not a sustainable condition. There is clearly a need to rapidly adjust the technology configurations being offered, eliminating insufficient ones and introducing new ones. In some areas this may mean using an audio source other than a radio. . EDC’s commitment to and integration of wind-up radios is visionary and appropriate. One Lifeline radio with a strong FM or AM signal is sufficient for a small class, especially indoors; and two of them in a larger room may also be acceptable. However, during the period of the evaluation, Miraya 101 FM signals in Wau, and Malakal were not adequate, regardless of how many Lifeline radios there were in the room. EDC has been in touch with a number of other radio stations and plans to hire some of them to extend radio distribution via FM and possibly AM (medium wave or MW). In particular, the MW station in Malakal is willing to provide airtime for a fee (around $1200 per month for both Learning Village in the morning and TERBIA in the afternoons). This station is said to have a much wider footprint than the local Miraya FM transmitter. In only one of seven radio observations was the radio signal fully adequate (at a TERBIA B1 class via Bakhita FM at St. Juveline School in Juba). At Yabongo in Yambio, with a second radio in the classroom Photo by Stuart Leigh, Yambio, Southern Sudan, May 20, 2008 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 25 and after retuning both radios to the “phantom” Miraya FM station at 95 on the dial, the audio was only barely adequate. According to the Station Manager of Miraya-FM, due to heavy urbanization, when their network is completed by the end of 2008 the Miraya-FM signal could reach 80% of the population. This statement is questionable.6 Even if the statement proves accurate, the strength of their signals in relation to Lifeline radios in classrooms would need to be assessed. Aware of issues with unreliability of various radio signals in various parts of the country, EDC has been investigating options in what it calls “low cost technologies” – by which it means .mp3 players with solar chargers and rechargeable batteries, or an integrated solar digital audio player that uses a proprietary codec (Megavoice brand). The cost factors related to audio programming for schools and to audio delivery options have changed markedly over the last 20 years. This opens an exciting range of possibilities that were not present in the earlier days of IRI. While audio quality was the most severe and intractable problem observed, there are other significant user issues impeding project success that are not related to program audibility and broadcast quality. IRI training also plays a role, as do teachers’ English capacities, and the absence of effective monitoring and problem solving by SSIRI outreach staff or participation by MoE counterparts. IRI Training and Technologies for Teachers: As part of capacity building SSIRI has trained approximately 875 teachers and 141 TERBIA facilitators, with focus on the use of IRI programs. SSIRI’s training model for teachers is a simple three-day approach, recently expanded to a five-day model. Insufficient audio reception confounded evaluators’ attempts to analyze teachers’ IRI classroom skills, meaning that the team could not judge the adequacy of SSIRI training to enable teachers to successfully implement the project with learners (especially Learning Village). The team did observe successful TERBIA B teaching in three sites, though two of these were aided by SSIRI observers with MP3 playback assistance. SSIRI has installed VSAT equipment at two sites and brought back on-line a third VSAT installed under the predecessor SBEP project. This has established Internet connectivity at the TTIs in Arapi and Maridi and a school in Juba. Video equipment and training has also been provided to the two TTIs. Each has begun to produce video programs. These inputs are greatly appreciated by TTI tutors. They note that the Internet will keep their lessons current and more vital. SSIRI is also in the process of equipping an educational computer center in Malakal with VSAT and 6 computers and 2 printers. This equipment is destined for the TTI there, should it ever again have a staff and return to educating students. Otherwise, it should be available under MoEST management for community and educator use. Monitoring and Evaluation: EDC-SSIRI staff provided to the MSI team for review numerous examples of Outreach Advisor (OA) and Outreach Coordinator (OC) bi-weekly and other monitoring reports. The quality and format of these reports varies greatly. Some are highly detailed. Most provide a useful summary of the staff person’s activities. Less frequently they provide information on the performance of 6 There is some uncertainty about such projected audience figures. In a study that Miraya itself commissioned called “Media Access and Use in Southern Sudan” by Graham Mytton, Audience Research Training and Consultancy, such a statement about urbanization and Miraya’s potential to serve 80% of Southern Sudanese is confused by such a statements as this: that Mirayas listenership is “59% rural and 41% urban …. Most Miraya FM listeners live in the rural areas. But Miraya has greater reach among urban people….” (% of urban Miraya listeners among all urban survey interviewees) “…The reason for this apparent paradox? There are more people in rural areas.” Independent demographic data is surely available and can inform this issue. Perhaps it can be part of EDC’s future survey work. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 26 the SSIRI teachers and facilitators. These reports may include separate sections on observations of LV and TERBIA classes, identifying user issues, complaints from teachers and reasons for their non-use of programs, technical issues, etc. In certain reports from the OA’s there may also be found performance reviews of the OC’s. While it is encouraging to see the more detailed observationally-based reports, we were told that they are not being received with the regularity that SSIRI managers would prefer. Absence of transport has been a factor. The future planned provisioning of motorcycles (we saw many at the SSIRI compound ready for distribution) should greatly help address these monitoring deficits. In discussions with teachers who have received radios and used them in the classroom, and in some of the OA and OC written reports to SSIRI management, there were numerous complaints that the sound was insufficient, the pace of learning was too fast to permit small group and individual interaction, and there were many technical problems receiving signals ranging from ineffective batteries to defective antennae to mix-ups in programming. Yet there appeared to be no systematic method for gathering such criticism, or acting upon those observations. New hiring of M&E personnel in Juba and new planning for M&E staff at the State level indicate that EDC is now addressing the need for more detailed and useful observational data. Concerns were raised over the adequacy, validity, and quality of field reports generated by EDC outreach staff and the way that data are treated in reporting to USAID. For example, delays in delivery (from another USAID partner, NDI) and distributing radios have had a deleterious effect on the project, since the core strategy revolves around interactive instruction delivered through this medium. An indeterminate, but not insignificant, number of teachers who have participated in SSIRI training have not received radios; and according to the evaluation team’s field observations, about one-third of those who received them are not using the radios at all, for reasons cited above. Nevertheless, raw data collected from schools lists all trained teachers and their students as beneficiaries of the SSIRI project, while many students sometimes are not receiving the benefits of radio instruction. Field data representing that teaching and learning is occurring is passed along to and accepted as valid by EDC monitoring staff, and submitted to USAID as evidence to performance. The difficulty of communications (universal and timely email, Thuraya, etc.) between Juba and SSIRI’s far flung project sites has posed challenges to the project in developing and providing quality control of monitoring systems. So, too, has the absence to date of sufficient vehicles at the State and County levels. The monitoring/evaluation opportunity afforded EDC by this MSI Mid-Term Evaluation itself has produced a searching response from the EDC-SSIRI management team. The EDC-SSIRI-COP has sent the MSI evaluators two documents indicating actions to be taken to address identified issues in the short and medium term (see Annex 6: “Responses to Mid-term Evaluation”, and Annex 10: “SSIRI 2nd Response to Evaluation-OneTeacherAtaTime-June7”). The USAID Education Team Leader has commended EDC for its openness and adaptability. EDC’s current suggested implementation adjustments, if followed, substantiate that they are determined to “get it right” and to effectively address newly identified problems – “One Teacher at a Time.” Outcome and Results Findings Re: Question 5: “Assess program performance and progress towards achieving program results in all the key program areas as measured against targets established in the cooperative agreement, annual implementation plans and the performance monitoring plans. Are the results commensurate to the USAID investment in the program?” SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 27 Annex 4, Table 2: SSIRI Deliverables and Proposed Activities in Contract or Modification and Status of Each offers findings of “results”. “Targets” here are stated as activities or service-to-people numbers. Measurements of project “outcomes” with regard to learning gains in English, science and local language literacy, or listeners’ understanding of civic concepts, etc. are beyond the scope of this evaluation. Discussion of EDC’s own measurement studies is confined to one study of learning gains among Learning Village P1 pupils (Annex 1). Learning Village Gains: SSIRI conducted a pre-test/post-test evaluation of P1 Learning Village pupils. The test results are encouraging, especially with regard to the English component where radio students outperformed control group students significantly (Annex 1). Gains in mathematics and local language literacy were less impressive. Certain features of the test administration and the wider observed context of SSIRI implementation caution us against generalizing too much from the results of this test. Pupil Numbers: The project has reported offering media-assisted instruction in classrooms with about 42,000 children. It was not possible for the MSI team to verify that the teachers trained by SSIRI ever taught those children, the quality of the instruction that those children receive, or the numbers of pupils given instruction of reasonable quality for a significant period of time (e.g. how many heard 1-10 audio lessons, how many heard 60-80 lessons, etc.). TERBIA Gains: From interviews with TERBIA teachers and with administrators, it appears that people enrolled in TERBIA B classes are pleased with their progress in English, as are their teachers. 62%, or 13 of 21 people, about half of them market shopkeepers, interviewed on the streets of Wau had listened to TERBIA. This is not a disciplined random sample but it shows some presence of the programming in the minds of citizens. One such interviewee indicated learning something about landmines and the CPA from TERBIA programming. The Director of Administration and Finance for SMoE-Wau was enthusiastic about TERBIA programs, saying they should be extended to all of his SMoE areas. TTI Gains: Qualitatively, TTI tutors report enthusiastically on the professional value of receiving basic computer skills training; and of being able to use the Internet for their lesson preparation, remaining current in world affairs, listening to the radio (BBC, CNN, etc.), communicating by Skype, and sharing of ideas and outputs with the Arapi TTI. They appreciate having been introduced to video production and have begun first attempts at producing programs. At Arapi we viewed “Types of Soil”, a credible first production effort by a tutor there. It was a demonstration of the relative water retention capacities of clay, loam, etc. Professional Studies for Teachers: SSIRI has two deliverables related to its teacher training program (PST). One of these consists of 12 audio programs supported by a 37-page Training Guide for SSIRI Outreach Staff and a 111-page Student’s Manual. The course is called PS101: Classroom Management and Administration. The course materials are complete but no application of them has begun. The MSI team was informed that the LV and TERBIA radio instructional material development had taken project priority. The course was originally to be implemented in association with the Maridi Curriculum Development Center (CDC), which has just finished designing a four-year in-service program. The MoEST has projected that the in-service course will be delivered through County Education Centers (still largely unbuilt). Delivery of the course is currently not occurring. EDC plans to integrate the first 12 PS101 audio programs/modules in the first of the four years of this course. SSIRI’s second PST deliverable calls for it to augment the first 12 programs with 24 more. We were not informed which of the four years of the in-service course will be supported by these 24 additional programs. The Multi￾donor Trust Fund (MDTF) is currently negotiating the selection of a contractor to carry out the training; SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 28 however, no one could project a likely date for contract signing or scaling up in-service training activities. This EDC activity has been impacted by this delay. SSIRI Integration in the MoEST: SSIRI has been incorporated in to the organizational / staffing plans of the MoEST at the central, state and county levels. This is currently only nominal in many places, but under the aegis of the AES department, the MoEST has hired or is hiring inspectors and senior inspectors responsible for SSIRI. There is a senior inspector for SSIRI at the central MoEST level. Secondment of SMoE staff to SSIRI in Wau was offered by their Director of Administration and Finance. The organizational chart of the MoEST AES Directorate exemplifies this integration (Annex 7: Department of Alternative Education Systems) Re: Questions 6, 7, 8: “Provide a brief description of the program outcomes, deliverables, and products (IRI Programs, broadcasts, teachers’ guides, teaching and learning materials, etc). To the extent possible assess the quality of the deliverables.” and “Describe and evaluate the program accomplishments.” and “What impact has the program had on development of technology based education in Southern Sudan?” SSIRI deliverables fall into four categories, which are discussed further below: 1. Learning materials (audio and printed teacher’s guides): • Learning Village audio and print: SSIRI has delivered over 75% of the Learning Village deliverables (auditioned and read by evaluators). • TERBIA B (Beginners) audio and print – EDC has finished 75 of the 120 planned programs in this series and the Guide Book for the first 60 programs (TERBIA B1) (auditioned and read by evaluators). • TERBIA A (Advanced) – audio only (auditioned by evaluators). • PS101 – EDC produced two new audio programs to act as bookends to the 10 previously produced teacher training programs produced by the SBEP project, together with a learner guide. These are part of the larger planned Professional Studies for Teachers (PST) series. A well designed guidebook for them was provided to us for inspection as a digital file. (We asked to be given PS101 programs to listen to, but we were not provided with them so we can say nothing about their style or substance.) The programs are generally of good quality and are useful educationally. From the standpoint of production values the TERBIA A programs are the most accomplished. They build on the SRS “Let’s Talk” series and offer both English learning opportunities (for listeners with more advanced English language skills), while presenting relevant social information, such as the implications of the CPA and citizens’ participatory roles in the country’s future. TERBIA B is similarly well produced, though purposely at a much lower level of English language. The Learning Village programs present a coherent model course for grades 1-4. The series’ audio programs are of adequate production quality, though they reveal certain careless production values (bridge music edits that are truncated, absence of fades in musical treatments, unintelligible voices at times, songs that are too fast for most primary English learners or whose lyric choices lead to hurried or unnatural phrasing). Most seriously, the lack of pauses of sufficient length for translation and interaction can be problematic for teachers, especially where their English skills are very low. Many pauses are shorter than the time the observed teachers needed to complete the task. The printed and wire bound teachers’ guides are of high quality. They are simple, easy to read, well related to the audio, and durable. They include removable SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 29 perforated paper manipulatives (money cards, rulers, etc.) to support further learning activities and enrich the classroom environment. The format of the Learning Village programs poses problems for schools in their timetabling. Treating three different subjects for about 10 minutes each creates an issue for some schools in how the day’s periods are allotted and how recess can be scheduled to accommodate all Learning Village classes. Other schools find no difficulty, for example, taking steps to let the P3 classes have an early recess break so that they are later free to tune in and study when the program for P3 is on the air. TERBIA B (for beginners) introduces itself as “the program that teaches students to read, write and speak English. Enjoy learning English with us in the TERBIA market, the education market.” At less than 30 minutes, it fits well into afternoon study periods. This is especially true of students enrolled in the Accelerated Learning Program of the MoEST. AES policy is that learner groups should try to meet for 2- 3 hours. This is a comfortable amount of time in which to do a TERBIA B audio class and also fulfill other aspects of the Accelerated Learning Program syllabus. The level of the TERBIA B programs is well suited to the English language development needs of many enrolled in the ALP program, though an SSIRI staffer wondered if the math level might be too low for some adults in TERBIA B groups. TERBIA A is introduced as “a program to teach English and civic education.” It seems to be successful in part because it knowingly takes advantage of the conventions of open broadcast. It has a self-selecting audience which speaks English at a higher level than TERBIA B listeners, and it has an interesting sound. It uses dramatic techniques to simulate real conversations about important issues. It is clearly reinforcing and introducing ideas of democracy and civic participation. 2. SSIRI training for teachers and educational administrative and technical personnel: The team observed parts of one Learning Village training in two centers. While this was not enough to determine the quality of the five-day training design, on the two partial days of observation the MSI team noted a lack of participant-based methods, with focus on the instructor and chalk board. Training of trainers (ToT) for both SSIRI staff and some MoEST counterparts has been done, but it was not possible to assess its quality or adequacy. Given the apparent lack of understanding of SSIRI personnel about principles of student-centered instruction, there remain substantial challenges to be overcome in effectively training SSIRI Outreach and MoEST counterpart staff. Repeatedly, the team noted that teachers had very poor English skills and that new strategies were needed to uplift their standards. For example, the SSIRI Outreach Coordinator in Malakal said that last month he trained 196 teachers introducing them to LV and TERBIA. He assessed their English and found that 50 (25%) were “OK in English”. The rest are in need of English language training and support. SSIRI is addressing this need by offering exposure to simple English in the LV and TERBIA B programs. At the same time, LV programs pose a comprehension load to the teachers that reduces the possible audience for the programs to perhaps 50 percent of the nations’ teachers (by EDC estimates) or possibly less. Such a foreign language load can lead to teachers ceasing use of SSIRI programming. We would urge EDC to pursue the commendable goal it has already proposed of developing an “English for Arabic Patterned Teachers” programming strategy. 3. Technologies / Equipment The provision of Internet connectivity at the TTIs is clearly a very useful contribution to improving the relevance and currency of teacher education. Teachers now feel that they can better stay up to date in their work and in their lives as informed citizens. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 30 The Freeplay Lifeline radios may be sufficient for classes of up to 40-50 children if the radio signals reaching them are strong. Both FM and shortwave can produce output levels on the Lifeline that are of sufficient volume for such use, but this is not what was observed in the field. The shortwave signals are usually of very low volume and/or are clouded by static at the times of day when they are needed. This year’s TERBIA B1 and TERBIA B2 SW frequencies are so close to one another in the 15 meter band that a teacher would not be able to tell which one s/he is tuning to. In one test (in Malakal) we were only able to find one of these two SSIRI TERBIA B SW frequencies at TERBIA broadcast time. While Miraya FM provides a much better option in many parts of the country, at its home frequency (101 MHz) on the Lifeline radio it proved insufficient in volume. For example, as noted earlier, in Yambio, Miraya FM signals were weak. To compensate, a project worker assisted the teachers (atypically) by placing a second radio at the rear of the long rectangular classrooms we observed (49 pupils in P1, 68 in P2). This assistance would almost certainly not have been offered had we not been visiting to evaluate that day. By contrast, we observed a TERBIA B class held indoors in Juba (St. Juveline School) with a single Lifeline radio tuned to Bakhita FM that was strong and clear. There was one area that reported radio failure and/or weakening of radios’ output levels that led us to guess at a 5-10% failure/damage rate over a year or so. The smaller Freeplay ranger radios are virtually useless for interactive radio classes or for more than a few listeners. While there are issues with the Freeplay radios in this broadcasting and utilization environment, EDC’s commitment to and introduction of battery free or rechargeable battery systems has helped make such new technologies commonplace. We see this as being of great importance. 4. Institutional capacity building To fulfill its mandate, SSIRI not only has had to build its own substantial staff and organizational structure but also seek ways to strengthen MoEST so that, eventually, government can take over and sustain the project. SSIRI’s has a current staff of 67 people with an expanding network of offices around the country (Table 3, SSIRI Staff Composition, page 14). SSIRI has run numerous trainings and workshops for internal and MoEST staff, e.g., CDC staff being trained in scriptwriting, TTI staff being trained in basic computer skills, Internet and video); and sometimes SSIRI has run special workshops designed primarily for State MoEST employees (e.g. as in Yei, February 2008). Re: Question 10: “What strategies has the program adopted in order to bridge the gender gap in education in Southern Sudan?” SSIRI has paid significant attention to gender issues. From Primary 1-3, SSIRI used a female teacher to encourage and to project women as smart, independent and able teachers. The use of two teachers in LV P4, a male and female, promotes gender balance; as does the overall radio format, which features girls and boys equally. LV P1&2 stories depict girls and boys as equal players at school with equal opportunities. There is also a story of a female pupil (Nyadak) playing football, and male pupil (Deng) fetching water from the river. A female character, Nako, sometimes initiates an intelligent discussion point during the program, which is followed up as an after-broadcast activity. Currently, stories in the P4 LV programs talk about balanced “roles” at home (males helping with household chores and females participating in discussions). EDC staff also report that they have gender-balanced planning and production teams, as well. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 31 VI. CONCLUSIONS EDC is on schedule with some but not all of its deliverables (see Annex 4, Table 2). In this transformational period in Southern Sudan, late fulfillment of some deliverables may be reasonably attributed to external circumstances, such as MoEST policies or the absence of them, MoEST projects that intersect with EDC projects (such as the teacher professional development component), available counterpart personnel, security situations and travel restrictions, and environmental conditions. Failures by the MoEST to smoothly manage certain of its own responsibilities have likely also affected timely and effective project implementation, (e.g., absent or late payment of TERBIA facilitator incentives and teacher salaries). That being said, we conclude that SSIRI management teams have not fully absorbed certain lessons of earlier IRI implementations in other challenging environments or paid sufficient attention to the details of project field implementation. This has resulted in weak or non-existent IRI performance in the majority of school sites visited. For example, audio (sound) itself is a major problem. In some States insufficient numbers of radios and teacher’s guides were made available for the teachers trained by the project. Insecurity on major roads has been a hindrance to road transport in parts of Southern Sudan; indeed, motorized travel is only now being made possible for project staff stationed outside of Juba. And while we realize that the sample of schools was largely urban, and thus had very large class sizes, and that this sample may not reflect the situation in some smaller classes in rural or suburban areas, we think our observations are indicative of serious common issues. We conclude that the long-term health of this important and worthy project is at risk if identified shortcomings are not attended to rapidly. We also believe that these problems can be successfully remedied this year with proper attention. USAID has asked us to indicate Achievements and Shortfalls as part of our conclusions. Achievements: Developing a Project of Significant Potential Impact: Working in a complicated transitional environment, EDC has established a project that could significantly improve the English and numeracy skills of many thousands of people in Southern Sudan. The project has high credibility and may soon have a presence in many counties from which much service may be given. Integrating SSIRI in the MoEST: EDC has developed relationships of mutual trust and respect with the MoEST at central and state levels and is now creating relationships of similar tone with the counties. The presence of SSIRI officers in the MoEST organogram (Annex 7) is significant. The actualization of effective personnel in these roles remains to be validated. Providing Quality Learning Materials: The Learning Village and TERBIA Audio and print materials are for the most part well-structured. There are flaws in the production and/or design of programs, most notably in the duration of pauses for pupil responses, but with careful identification of problems and timely revision, the programs should play an important role for a number of years in regularizing effective teaching and improving educational outcomes in Southern Sudan. Applying New Technologies: EDC has wisely adopted economical energy solutions that free users from the expense of batteries. The application of single Freeplay Lifeline radios is well suited to small indoor classes (40-50 pupils); and multiple radios can be successfully added for larger classes. EDC is also wisely researching solar MP3 solutions, which should provide an important alternative to radio at a reasonable cost where radio is impractical. The project has also introduced TTIs to VSAT/Internet and video so tutors now have greater potential to develop their own professional skills to better serve both SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 32 pre-service and in-service teachers. All of these efforts, together with the planned ICT Summit and study tours, are advancing technology-based education in Southern Sudan. Improving the current state of IRI utilization in classrooms, however, is a precondition for maintaining the confidence of the MoEST in this particular educational technology application. The “low-cost technology” solution (MP3 players or other digital playback systems) should play an important role in providing quality audio where radio is insufficient. Investing prudently as MoEST budgets are reduced: MoEST strengthening is faced with multiple challenges. The ministry is staffed with new and un-experienced personnel and is attempting to devolve operations to states, in the face of continuing budget reductions (current funding is US$ 96 million, as compared to US$111 million in 2007 and US$136 in 20067 ). This situation raises cause for concern about sustainability or scaling of SSIRI after the life of the project. Given that the GoSS itself is an inchoate entity and the MoEST is its extension, we commend EDC for measuring investments in the MoEST with prudence. Staffing: EDC has created a project staff of 67 people, including 51 based in Sudan who, when provided with further training so that they themselves may effectively train others, and when stationed wisely with counterparts from the MoEST, should be able to create exemplary model SSIRI schools and begin to replicate those successful models more widely. This staffing is positioning the SSIRI team to be ready to go to scale as soon as 1) staff training and skill levels are raised and 2) implementation quality justify such a leap forward. Shortfalls: Radios and radio program delivery: Radio instruction is a challenging format under any conditions, and especially complicated by the Southern Sudanese context. Under the best of conditions, FM radio coverage, as projected through the end of the year, can be expected to extend to less than half the area of the country. Even in urban areas weak signals, moderate signal amplification by the radios distributed by SSIRI, too few radios, and large classes tend to impede use in classrooms. While the Sudan Radio Service broadcasts by shortwave and is the third most listened to radio service in the country (after Miraya FM and the BBC World Service)8 , shortwave radio is very problematic for SSIRI learner groups. Signals are weak and unpredictable. While EDC has succeeded in resolving the issue of lack of power and battery expense by using wind-up and solar charged units, the aural effectiveness of a single Freeplay radio is questionable in classrooms with more than 50 students, much less in classrooms with 180 or 200 as we witnessed. Students simply can’t hear the programs. Using multiple radios in each class is a possible successful remediation strategy, though schools have not been trained to do this and typically do not do so. The absolute number of radios given per school is less the problem than the way care and responsibility for them is managed at the school level. School heads do not seem to have priority over teachers in managing the disposition of radios. Training policies and SSIRI-school agreements need to include the directive that radios are the collective asset of the school, and that even if single radios are entrusted to individual teachers for overnight care, in order to fill each teacher’s classroom with sound, they cannot keep another teacher from more than one of the school’s radios. 7 Budget figures provided by MoEST Director of Alternative Education Systems, Kuol Atem in an interview. These figures do not necessarily establish that these annual amounts were in fact appropriated or spent. 8 “Media Access and Use in Southern Sudan”, survey and report by Graham Mytton, Audience Research Training and Consultancy, October 2007. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 33 Also important is the fact that radios and, in the case of the Learning Village series, SSIRI teacher’s guides—the foundations of the SSIRI primary courses—have not been distributed to some teachers in certain areas who were trained in their use. Many teachers with radios have been transferred to other schools possibly outside broadcast coverage areas, or have quit the profession and taken their units with them. A number of teachers trained and equipped with radios have quit using them due to low English skills or other frustrations. Broken units, non-functioning aerials, inaudible sound, and broadcast hours that coincide with school recess are but a few reasons cited for non-use. We are concerned that such feedback information has not been adequately integrated into SSIRI implementation and remediation strategies. In-service Teacher Training: The teacher training program (PST) has not been launched, despite the pedagogical shortcomings of a corps of over 21,000 teachers, 80-90 percent of whom are without formal pedagogical training, or facility in the English language. Instructors face enormous difficulties in dealing with classes regularly exceeding 100 or even 200 children, the very great majority of whom are non￾English speakers. Although other donors are attempting to implement in-service training throughout Southern Sudan, and with insufficient curricula or resources, EDC has not yet made a systematic sustained effort to engage in this vital effort. One of the impediments to progress is that EDC has chosen to wait for other partners (MDTF and its in-service training contractor, and the MoEST) to initiate their parts of the national in-service training plan. While this may be a choice made after careful analysis in the belief that waiting will offer a better chance of the PST program being implemented, sustained and used by the MoEST, if this deliverable is to be attained, immediate engagement is needed. Technology: There is much promise in the sorts of technology EDC is piloting but so far the specific hardware choices have not been uniformly successful. SSIRI MP3 players with a single AAA battery do not last for the full TERBIA class period. The speakers used to amplify the MP3 players have an output level that is adequate only for small classes. To a degree this is a case of “bleeding edge” adoption. Wind-up and solar radios and digital audio players and portable speakers powered by solar-rechargeable batteries are not easily found in many varieties and power output ranges. We do not know if the market for these devices is yet sufficiently competitive or segmented to have produced models with sufficient output power for larger classes. It remains the case that no amount of improvement in monitoring or training will create an effective IRI learning experience if the sound of the program is not loud and clear. In many, if not most SSIRI classrooms and learning spaces, a clean, strong audio signal is not reaching pupils’ ears. EDC has done some research in hardware options and provided some information on this research through its quarterly reports but more needs to be done. To date, experimentation or piloting of MP3 technologies for various class sizes has not been of sufficient breadth nor documented sufficiently to guide purchasing on a large scale. The relative virtues and shortcomings of various available hardware options in actual performance are not yet sufficiently understood. We think that more research and budgeting more per unit will produce more options and effective solutions. Program Design: The pace of the programs, which is related to both the lengths of the interactive pauses for teacher and pupil response and the speed with which the treatment of one objective transitions to another, together with the English language comprehension load for the typical SSIRI primary teacher, result in an SSIRI program experience for many teachers that is not easy to manage – even if they can hear it well. Not enough attention has been paid to interaction pacing for these users and how that promotes or impedes effective classroom behaviors and outcomes. At 28 minutes the programs are probably a few minutes too long since successive programs are broadcast each ½ hour and more than two minutes is needed to transfer radios from one room to another and for the teacher to wind it up and tune it again. This will be even more the case if schools begin to use more than a single radio in each classroom, as the team recommends. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 34 Staff Training and Monitoring: EDC is continuing to employ program support staff, many of whom are posted in the countryside, but most lack sufficient training to offer technical assistance to teachers. In virtually all aspects of the SSIRI delivery system, there is an absence of feedback, systematic analysis, or ongoing evaluation that is so critical to the success of such a challenging program. Transportation: Part of the problem with SSIRI field implementation is that it is only now, in the final year of the project, that field staff are being provided with motorcycles so that they can make rounds of schools. This delay has kept them from encountering the realities in many of their schools. Numbers of Beneficiaries: In the original project proposal and subsequent modifications, the projections of the number of people to benefit from SSIRI are far above the figures attained thus far. It is unlikely that EDC will approach many of these targets. Further Conclusions: While it is not EDC’s responsibility to elevate the English skills of teachers by any other means than those cited in its Agreement with USAID, this critical deficit has been noted repeatedly and is leading to a reduction in the possible audience for Learning Village (EDC estimates that only 50% of all P1-P4 teachers may be able to use the programs due to the level of English their skills.) This is a condition that leads to many teachers who are given initial training to drop out of the program. This multi-faceted problem calls for innovative strategies on how teachers can be led to work on their English on their own through school-based and community-based strategies. EDC has laudable ideas in this area – one being a new course in “English for Arabic Patterned Teachers”. This idea is not yet fully developed but it deserves further attention. Finally, wherever there is a SSIRI MP3 player (and it should be possible to put one in every school), the learning environment for teachers could be further enriched, and learning opportunities for teachers greatly increased by providing “anytime access” to additional audio and print designed to enhance teachers’ English language skills. VII. RECOMMENDATIONS Based on international experience, we know that effective implementation of SSIRI will be possible given enabling certain conditions, including but not limited to: 1) clean, strong audio reaching the ears of the learners and teachers (a multipart problem); 2) radio formats that consistently demonstrate writers’ understanding of how long it really takes for BOTH teacher and students to thoughtfully respond to each radio cue; and songs that are composed with an understanding about how musical rhythm, tempo and lyrical choices can reflect and reinforce natural native English phrasing; 3) teachers and head teachers with basic understanding of IRI requirements; 4) teachers who are able to understand basic instructions given to them by the radio; 5) capable field staff able to introduce and support users until they can self-manage IRI; 6) a supportive well-informed co-participating MoEST; and 7) good project communications internally and, when needed, to users. Due to EDC and USAID accompanying the evaluators on the evaluation tour, their staff were able to observe the same facts on the ground, and they have access to the same information as the evaluators. Thus they developed their own views of project conditions and have likely come to similar conclusions as have the evaluators. The team shared its insights with them during and after field visits and they were SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 35 made aware of the identified and likely recommendations. Consequently, in their “Responses to Mid-term Evaluation” (Annex 6), they identified a number of Issues and Proposed Actions. It may be useful for the reader to refer to that document in considering our recommendations, as some of EDC’s proposed actions closely respond to our suggestions. A substantial and well-considered investment has been made in the SSIRI project. Much depends on the project’s success. Further investment in the approach will be warranted if it can be shown to be working well in more than a very few sites. We recommend that there be ongoing internal assessments of the project and possibly a further independent assessment undertaken in another six to nine months to see whether the identified problems have been largely addressed and overcome. We offer the following specific recommendations to assist EDC in bringing SSIRI’s many local implementations to a new standard of performance. Focusing in 2008-2009 on Quality Implementations of Learning Village and TERBIA Emphasis in the next 12 months of the project should be on creating high quality user experiences in Learning Village and TERBIA classes. This is a multi-part problem that can be broken down to optimizing broadcast and digital playback strategies, getting the balance right between the two audio delivery options and their applications for various locations and learner groups; deploying sufficiently robust radio and digital playback equipment, providing adequate staff and teacher training, effectively monitoring, and continuing to seek ways to impact the English language capacity of Sudanese teachers. Our recommendations are in four main areas: 1. Assuring clear, strong sound in all SSIRI learning environments Audio Technologies (Radio Reception, MP3 and the Audio Experience): Further study should be done rapidly on “low-cost” audio devices and compatible and sufficient solar chargers and rechargeable batteries. EDC must find one or maybe two solid systems that it can buy in large quantities and get out to the field with assurance that they will serve the intended learner groups. NB: There is no virtue in spending a few dollars or even a few scores of dollars less on a system if it is not delivering to the listeners’ ears the essential audio. Teachers should be informed about the phantom Miraya station at 95 MHz that for some still unexplained reason has a more powerful output than the nominal 101 MHz that teachers have been instructed to use. Freeplay should also be queried about why their Lifeline receiver is picking up this louder phantom station. Also on the agenda for that conversation should be whether they could produce a unit with higher sound output. We commend EDC on pursuing with Freeplay the possibility of including an MP3 player in a wind-up radio unit. Such a wind-up Freeplay MP3-equipped unit with stronger output level would be a very useful tool. We tentatively suggest that one designed to receive an SD card (for recorded programs) might be less susceptible to breakage than one with USB port for a memory stick. Teachers and head teachers must be informed that all classes (whenever possible) should have two Lifeline radios operating at the same time and that these radios are school property to be safeguarded nightly by some locally effective agreement; and that no teacher should be so possessive that a radio in his/her care cannot be taken by the next Learning Village teacher who needs it. With sequential broadcasts and a single stream at each grade level an entire school would only need two radios. With two streams four radios, and so on. A back-up radio to temporarily replace lost or broken ones should always be available at the County office. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 36 MP3 for TERBIA B Groups: TERBIA B groups should be transferred over time to MP3 players. Shortwave is problematic and often unreliable in the afternoon. Afternoon airtime on FM is not going to be easily available for 30 minute programs. Airtime is also very costly (SW costs are currently about $160,000 per year), and airtime must be purchased on recurrent basis. Such costs must be factored comparatively when considering the extra costs of distributing robust digital playback equipment and recorded media. No longer are the materials and labor costs associated with recordable audio media a tremendous cost item, as was the case with audio cassettes. Refreshing of both TERBIA and Learning Village programs, if required, can be done with a rotating stock of digital media. The project can distribute newly recorded SD cards or memory sticks, and as they do so retrieving those originally distributed to be refreshed themselves with new programs. Moving to recorded media will also answer the complaint of TERBIA B facilitators that their afternoon classes often start late due to learners’ late arrival and thus they cannot tune in on time to the radio programs. Study of Economics of IRI by Radio and MP3 (or other codec) for Formal and Informal Settings: One of the long-held verities of Interactive Radio is that using radios is the most cost-effective way to provide quality educational opportunity to large numbers of learners in developing countries. EDC or some other entity should closely examine this assumption given the changing technological landscape. EDC is already committed to making TERBIA Available by MP3 to some of its TERBIA groups. Might Learning Village follow? At the time that the “cost-effectiveness verity” emerged, the personal computer hardly existed. No study of comparative cost-effectiveness of IRI vs. digital playback devices has yet been done. While such a study may be beyond the scope of EDC’s current Agreement, this would be an interesting contribution to the field and should be considered for a future year. There are many reasons from the educational user’s point of view why recorded audio on disc or memory stick or SD card or some other digital device is superior to radio; especially now that power may be generated by winding a crank or by a solar device.9 With such energy sources the argument that spinning a disc drains batteries faster than a radio is not so persuasive. A robust MP3 solution would free teachers from the need to get a class settled and tuned in by a certain minute on the clock. From the standpoint of pedagogy, having the chance to repeat some part of the program can be very useful. Reception issues would be a thing of the past. Program administrators would not have to compete and cajole for precious air-time that broadcasters are often loath to surrender for their own good reasons, and are sure to charge annual fees to provide. Marketing and P/R: Rename “Low-Cost Technologies”: In the realm of marketing and public relations, we believe that the term “low-cost device” is appropriate to discussions between contractors and funders but ineffective and misleading between the project and the public. The term is unspecific and does not help educate the public about this important new educational technology. We suggest norming around the terms “Digital Audio Players” or “Digital Audio (Playback) Technologies”. This will be far more informative, more attractive, and educative in itself. This is a more consumer-focused branding strategy that is more likely to interest and excite users than the current one, which may merely suggest to them that no one is spending much money on this intervention. Medium Wave (MW) Radio Stations: As soon as SSIRI needs to broadcast outside the Malakal town area and the range of strong Miraya signals, it should contract with the medium wave (AM) station in Malakal. The station there is highly interested. This is a government-owned station that we understand (and that EDC should verify) reaches far beyond the range of Miraya-FM in Upper Nile State. This 9 Even audio CDs could play a part in certain applications, such as teacher education. Audio CDs offer some superior pedagogical options to simple MP3 files (either on disc or flash memory) as they are controlled by simple MP3 players in that one can fast forward or review through the contents of a CD audio program. With a simple MP3 player, hitting the advance or review button jumps to the next program. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 37 would be a good home for Learning Village and for TERBIA A and for TERBIA-B as long as it distributed by radio. While we did not visit the sister MW station in Wau, we would venture the same recommendation for the MW station in Wau should conditions and willingness be similar there. Audio Programs Get the Pauses Right: The key radio scriptwriters should be brought to Southern Sudan to see a number of their programs at work in Southern Sudan schools. This should help them to internalize the time really needed for teachers and pupils to complete their directives. We understand the scripts are evaluated summatively (FE’d) in Kenya but we wonder what this process is like and if it is being done by one or two teachers who are by now so experienced with the programs that they have atypical expert capacities, or atypical students, or take certain process shortcuts. Consider Longer Teacher Led Activity Sections: Writers should also consider injecting longer teacher led activities of perhaps 1 -2 minutes in length that can at least occasionally, if not daily, allow teachers to work with a larger number of children in series; to practice communicative language teaching methods in a more intensive way that will be easier to generalize to further English teaching after the program is over; and to establish more relaxed and naturalistic communication in the class. Examine all Songs for Sing-ability: Those songs that are too fast or too unwieldy for young non-English speakers to voice easily and naturally should be rewritten. Some of the songs are very nice as they are, including the Learning Village signature tune. 2. Working in Communities Model Schools: We recommend that SSIRI adopt a model school approach in every county where it works, making it a priority for each local SSIRI staff and their counterpart(s) to create at least one demonstration site where SSIRI is working very well. Depending on the circumstances, coordinators may extend the same level of attention to more than one school. We would leave it to SSIRI to decide what the limit should be on the maximum number of schools that any coordinator should attend to in this next year. Some already have 30 schools. We do not counsel ceasing SSIRI support to any of the schools that wish to continue, but rather judiciously budgeting staff time so that some maintenance support is given to all, but more attention be focused on a single chosen school so at least one site under each staffer’s management becomes a proper model to the community and to other schools of what should be happening in SSIRI classes. Other benefits should flow from this focused commitment. For example, parent teacher association activities to promote community resourcing may logically follow from discussions initiated at meetings about SSIRI and its requirements. We can imagine school groups deciding to create needed latrines as such SSIRI generated conversations of concern develop. In any new county that SSIRI enters this year, the SSIRI / MoEST county team should reflect on how it can best tighten focus on an initially small set of schools within a day’s round trip travel and within range of a good audio signal (or equipped with a sufficient MP3 player). Once things are working well in all these schools, staff energy may be similarly extended to more schools. This chosen model school could in some areas represent a de facto “cluster school”. Choosing Project Sites Carefully: SSIRI should make expansion into new counties conditional on demonstrated interest by the local community and the County Education Office. This may be expressed in seconding of staff or counterpart designation and/or the willingness to work with SSIRI to create one or more model schools. The local counterpart should be available most of the time for SSIRI activities, SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 38 however if this person has many other required duties, such a requirement may be difficult to apply. In existing SSIRI counties where there are already many schools, this requirement should be more firmly observed since more work will be required of the SSIRI staff and counterpart. Keeping “clustering” in mind, selection of model school sites should be done based on 1) the accessibility of that school to the SSIRI staff person and counterpart, and 2) the centrality of that site to a larger group of schools. Continue to Focus on Relations with the States and Counties: Because the counties receive their directives, budgets and resources through the states, the states must be made efficient before putting major resources into bringing the counties to similar levels of efficiency (or possibly in effective tandem with the counties). No county should receive an Outreach Coordinator before an Outreach Advisor is assigned to the state. Counterpart Identification, Incorporation and Training: EDC’s “Responses to Mid-term Evaluation” of May 29 (Annex 6) indicate that they may now post multiple people to the same county offices. Whether this will be socially comfortable at each local MoEST office will need to be carefully thought through in advance on a case-by-case basis. We urge careful high level consultations before placing multiple EDC staffers in a single location. We recommend that in any event, and certainly before a second SSIRI staffer is placed at any site, a secondment to the project be received from the MoEST office in question; or at least a counterpart person within that office be officially designated to work in parallel with the SSIRI person(s). While sustainability is conditioned by many factors, perhaps at the top of the list is Sudanese ownership of the ideas and (eventually) the ways and means. If multiple people are posted to a single office it should be clear that only one of them – and if the project is successful, maybe only the MoEST employee – will remain after a year or two. It will likely be hard to shift more highly paid SSIRI staffers to lower government pay scales. Making the counterpart personnel already employed at the state and county levels full partners - now - with expectations and responsibilities that are not trivial will be key to the long range viability of this project. Training in school support and SSIRI-related reporting processes, as well as regular travel facilitation for local MoEST staff, should be delivered with this in mind. SSIRI and County Education Office staff lack skills in instructional technology and media-based learning, as well as in follow-up support and monitoring. Even some senior SSIRI staff appear to have little understanding of effective instructional methodologies. As a result, training programs for teachers and trainers may be of marginal quality and impact. We recommend immediate international technical assistance to craft and implement innovative staff training and training of trainers, which are so vital to the success and sustainability of programming. Relationship of the Directorates of Basic Education and AES: EDC should aim to secure the collaboration, understanding and support of MoEST’s Basic Education Directorate (at all levels). It is advisable that the MOU between the MoEST and the State MoEs being developed by EDC include explicit approvals for Learning Village usage in primary schools by both the State Directors of AES and Basic Education. One could understand a Director of Basic Education questioning the way a radio program that spans three subject areas in a half hour fits into the approved timetable scheme. It may not be self-evident to all that a 30-minute Learning Village program that covers three subjects, each in brief, should be given one of the six full teaching periods each day. Directors may have in mind allotting single subjects to single periods. They may well say, as we were told by one teacher that an Acting Director of Basic Education had advised his school, that Learning Village does not follow the primary English syllabus. Whether or not this is the case, in some areas (especially where SSIRI is not universal within a county or state) there may be another dominant way of teaching and an alternate dominant mode of school inspection of Basic Education classes. There needs to be clear proactive communication from the Basic Education Directorate that the Learning Village/SSIRI approach is approved and that timetables can SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 39 and should be adjusted to accommodate it. It is also advisable that the MOU between the MoEST and the State MoEs being developed by EDC include explicit approvals for Learning Village usage in primary schools by the State Directors of AES and Basic Education, and by the Directorate of Curriculum, (as we understand that they manage timetable directives and requirements). Promotional and Communication Strategies: Informational and promotional outreach should be done by SSIRI to the MoEST at all levels, and by the MoEST to its offices at all levels, to its head teachers, and to its teachers so that they understand that the SSIRI programs are designed to model activities in short sections that teachers may expand upon after the broadcast. Other popularization approaches maybe tried. For example, we recommend the immediate launch of the SSIRI’s proposed 50-minute education magazine program that the MoEST and Miraya-FM have already agreed to produce under SSIRI funding. We think this outreach strategy has immense potential to martial community support for schools, from latrine building to establishment or invigoration of parent-teacher associations. Partnering with NGOs for pre- and in-service teacher training: There is much more that SSIRI could be doing with Windle Trust and other NGOs. In particular, the potential multi-state partnership with Windle warrants more attention on the part of EDC. They appear to be a natural ally that can extend the benefits of the project to as many as 3200 teachers. As we were told by the Windle coordinator in Malakal, “There are 10 senior tutors at the national level and all could be given a (TERBIA) training by SSIRI.” As Windle informed us that they “do not deal with pedagogy”, their trainees could also be an accessible well-managed group of beneficiaries for SSIRI’s PD101 audio/print course. Creative collaborations should be pursued at the central (Juba) and local levels. An outcome of such collaboration could be teachers becoming more able to cope with LV and TERBIA teaching. 3. Monitoring and Evaluation Systems The evaluation team and project administrators are faced with a dearth of information relevant to the quality of program implementation. Although continuance of the SSIRI project would be predicated, in part, on an EDC in-house study of Learning Village impact in grade 1 (carried out in 2007 - Annex 1), the team’s cursory analysis of the research concluded that it likely suffers from significant instrumentation and procedural faults. Given these limitations, the study should not be accepted as reliable or valid without further scrutiny. Similarly, lack of quality feedback on the deployment of SSIRI-trained teachers and their use of radios must call into question the reliability of EDC reports and databases, and their accuracy in meeting USAID performance indicators. It is imperative that EDC launch intensive, objective and well-designed impact assessments of effective radio transmission and listenership; incidence of radio use in the classroom, contributions of radio programming to learning, and comparisons of radio with other technologies. The team frequently heard complaints from teachers about the pace of radio instruction, audibility issues, and lack of radios. There is little evidence to indicate that such feedback has been incorporated into revised programs or the delivery infrastructure. We recommend that EDC outreach staff reports be regularized, scrutinized for accuracy, reviewed frequently, and responsive follow-up actions rapidly initiated. Similarly, teachers’ radio-use logs should be gathered and analyzed. In short, if the SSIRI program is to be continued, scaled or sustained, EDC and USAID administrators must be informed with rigorous and current assessments and feedback from target groups. There should also be more frequent and broader field monitoring by EDC, SSIRI and USAID. Although security and distance barriers plague travel to the field, the project would benefit from more informed decisions that can only be formulated through hands-on and direct observations. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 40 4. Investing in teacher training, especially in-service training PS101: The core of SSIRI’s PS101 teacher training program was produced during the SBEP effort and has yet to be implemented. Small investments of time and resources would validate the program to provide support to teacher training programs, some of which are now in implementation without content or learning resources support. We note particularly NGOs supporting in-service English language training. We would urge EDC to join with initiatives supported by the Multi-donors Trust Fund (MDTF) to launch the SSIRI teacher training program (PST) during the next project semester. A high priority should be to complete the next 24 programs to support immediate teacher training. It will be a major contribution if SSIRI can provide the inputs needed to get the first year of the CDC’s recently designed in-service program off the ground. EDC may also consider piloting PS101 in some other manner before the MDTF in-service contractor becomes active in implementing the recently completed CDC design. Re-examine Strategies to Enrich IRI Training for New Teachers: SSIRI’s training model for teachers has been a simple three-day one. It is now being extended to five days to try to address teachers’ need for greater confidence in responding on cue in front of their pupils to English directives from the radio. While the extra two days will surely be well spent and valuable, we do not believe it will substantially affect teachers’ basic English competencies. For that purpose, other longer term English language support strategies are needed. SSIRI’s proposed English for Arabic Patterned Teachers course is one. Another is SSIRI’s establishment and teachers’ use of English Resource Corners in the schools (as outlined below). With regard to the five-day IRI training itself, this training needs to be revised to provide teachers with more hands-on examples of working with the programs. The training should also go beyond standard IRI topics, such as care and use of the radio, and the specific training to use LV, and provide broader instructional skills to this largely untrained core of participants. We highly recommend a training module on Communicative Language Teaching principles and practical methods, both as found in the existing programs and as teachers should be applying after the radio-assisted teaching period. Additional School-based Support for Teachers: If possible, each qualifying school should have an MP3 player. The criteria for such provisioning are for EDC and the MoEST to decide. Such equipment, appropriate programs, and possibly related print materials will enable a tremendous range of options to improve teachers’ English and teaching skills. EDC has already contemplated an MP3 player per teacher enrolled in the upcoming in-service program. We would suggest a more economical approach: identify at least a pair of teachers from a single school to participate together in the in-service course and provide the MP3 player to their school for them to use as a team, together eventually with other teachers at the school. This equipment would then do double duty as 1) an enabler of the in-service course in that school and 2) the cornerstone of the school’s English Resource Corner (see below). The English Resource Corner itself would have the capacity to support teachers’ English skills development and the further school based training of new and replacement teachers taking on SSIRI classes. For the latter purpose we propose School-based SSIRI Training, as described below. School-based SSIRI Training: Conceiving of this program at scale in a few years, one can see that trained SSIRI teachers are sure to go on maternity leave or be transferred. It is not practical to offer the standard three-day training every time a new person comes into the picture. There should be a School￾based SSIRI Training Guide designed to be used by any SSIRI-trained teachers or head teacher in that school, or by any teacher there with obvious competence in managing Learning Village classes, or by the Outreach Coordinator of his/her MoEST counterpart, if available. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 41 The school-based team then trains the new teacher(s) during a series of short afternoon or weekend meetings.10 Quality training assistance, follow-up, and coaching can be done as possible by inspectors or project staff. But, the assumption is that there should never be a long delay in having a replacement teacher take over the SSIRI duties of a former teacher. The pupils must come first. Model schools can promote School-based Training by example and even act as informal exemplary “cluster centers.” English Resource Corners for Teachers’ Self-paced English Language Development: A range of school-based strategies should be explored both for the training of new teachers who need to take over SSIRI classes, as above, and for ongoing, self-paced school-based English language skills development for teachers.11 We recommend setting up English Resource Corners in SSIRI-supported schools. It does not take much to create one. Schools need only to designate a space with a sign and a small shelf to indicate it as an official learning space in the school. It can also be temporarily in a community space or even a teacher’s home if the school conditions are not yet conducive. In the English Resource Corner should be one or more copies of the Learning Village teacher’s guides, an MP3 player and digital media with all SSIRI programs, and possibly other English language teaching and teacher development audio and eventually print. This may be a project most appropriately implemented in all SSIRI schools in a later project year, though piloting the concept should start this year. 5. Re: Continuation The SSIRI program has significant potential. With sufficient remediation of current implementation problems and attainment of greater project effectiveness, we believe the benefits to learners throughout Southern Sudan will be significant. If such is accomplished in the final year of the project, or if assessments show it to be attainable the following year or two, continuation will be warranted. If an additional year or two is required to attain such effectiveness, a short-term extension through a further Modification may be warranted. A longer-term project extension may call for another approach. To date, no phase of the SSIRI program has been tendered for competitive procurement. The operation was launched through a central USAID mission buy-in facility (DOT-EDU) and has been replenished for seven years, with nine Cooperative Agreement (CA) modifications. If USAID is contemplating a long-term extension of the operation (e.g., 5-years), it should carefully weigh the merits of an open and competitive procurement against the merits of management continuity. VIII. LESSONS LEARNED Monitoring and evaluation systems should be designed as an integral element in project design, planning, and execution. Optimally, the M&E process should be overseen by independent and rigorous observers who gather data on a quarterly basis, especially for initiatives with little or limited validation. Without timely and quality feedback from the field, critical revisions in project management must rest on unsubstantiated and often erroneous premises. Performance metrics and Cooperative Agreement deliverables should specify quality criteria as well as gross number indicators. For example, a target of “2,000 teachers trained” does not measure the impact of that training on learning, or whether it results in new or adapted professional behaviors. 10 The model IELTLM “School-based Training Guide” is worthy of study. It is a complete design for replicating at the school level the project’s 15-hour training course to enable primary teachers to begin using audio in their classes. 11 We also suggest study of an output of USAID’s Improving English Language Teaching and Learning in Mindanao Project, “Real World English Professional Enrichment Course: A Certificate Program for Teachers.” SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 42 It may be inadvisable to rely on another NGO to procure a commodity that is a key element of the program, as happened with radio delivery through NDI. Capacity-building, especially for project staff should not be treated as a one-time-only orientation. Cultural perspectives and differences notwithstanding, local staff, however well-educated, require systematic strengthening in competencies. Project execution teams must guard against accepting assumptions that may prevail in one nation or area when extending to a new environment, especially one with a long history of dislocation and instability. These circumstances make it even more important to pilot, experiment and test operational assumptions, and not simply accept them on the basis of past merit. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 43 IX. ANNEXES Annex 1: Work Plan SSIRI Evaluation Activity MSI (Stuart Leigh and Charles Tesar) May 10, 2008 BACKGROUND The mid-term evaluation of the Southern Sudan Interactive Radio Instruction project will be conducted from 8 May to 31 May, 2008. The process and product assessment process will actively engage stakeholders USAID-Sudan, the Southern Sudan Ministry of Education, Science and Technology (MoEST) and the contractor, Educational Development Center (EDC) in a rigorous analysis of the impact, benefits, processes and outcomes of the SSIRI activity from its inception to 2008. Led by two MSI evaluators, Stuart Leigh and Charles Tesar, the team comprises representatives from the above-mentioned organizations, who will travel to project sites throughout Southern Sudan. Members have agreed to employ an open and consultative format; however, the ultimate responsibility for gathering and interpreting information rests with the MSI evaluators. PURPOSE The principal deliverables of the consultancy will be an oral debriefing (on 30 May) and written report (the first draft of which will be submitted to USAID-Sudan on or before 31 May). The evaluation will provide answers to a set of questions posed by USAID Sudan, as well as assess the execution of the SSIRI project from four critical perspectives: 1. Performance of EDC-SSIRI with regard to the outcomes and deliverables stated in the original and modified Cooperative Agreement. This will entail measurement of performance in meeting project goals, objectives, reporting requirements, deliverables and targets. 2. EDC’s response to challenges of the Southern Sudan operational context. In this instance, the team is concerned about the Contractor’s adaptation to rapidly evolving conditions. Given the responsibility of EDC to ensure that USAID resources have been used to best and highest utility, how have lessons learned and broadened knowledge of the South Sudan situation been integrated into programming. 3. The Contractor’s recognition of best practices in education, newly-emerging content delivery technologies and best international practices will also come under scrutiny. Has EDC incorporated into programming the most efficient, effective and reliable methods appropriate to the Southern Sudan context? And finally 4. The Contractor and SSIRI project’s contribution to the accomplishment of USAID-Sudan’s evolving development strategy in Southern Sudan. Is the operation compatible with the USAID Mission’s present and continually developing needs? And in particular, is SSIRI considered of sufficient importance by the USAID Education Team and the Ministry of Education, Science and Technology to be continued in its present or a restructured format in the USAID portfolio. METHODOLOGY The evaluation process will incorporate the following methodologies: SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 44 1. Desk review of all SSIRI project documents, including the Cooperative Agreement, work plans, internal evaluations, annual and quarterly reports and consultant reports; 2. In-depth debriefings, face-to-face debriefings with EDC project management and USAID Education team members (CTO) charged with SSIRI project oversight; 3. Discussions with a wide range of MoEST officials, including: those GOSS functionaries responsible for integrating SSIRI assets into educational operations; State Directors, County officials, TTI personnel, school headmasters, and more importantly, as many teachers as possible; 4. Consultations/interviews with other USAID Mission personnel, including the Mission Director and Health and Education team leaders, possibly the Democracy and Governance team leader, and USAID contractors involved with education and associated activities; 5. Field visits to radio broadcasting organizations; 6. Site visits to at least eight (8) States in Southern Sudan, including several states in the Transition Areas—to the extent security limitations permit; 7. Site visits to the major teacher training institutes (TTIs) to guage the impact of the SSIRI on pre￾service and in-service teacher professional development; 8. Classroom observations of the impact of radio instruction on students, and follow on discussions with students and teachers; 9. Observations of TERBIA working groups; 10. Examination and review of a representative sample of radio and video instructional units from each series together with any integrated print materials and descriptive documents related to the planning of the instructional programs; and, 11. Site visits to inspect alternative technologies and teacher training programs. The MSI evaluators have been provided with a set of questions that will form the basis for the evaluation activity and report. These are listed as 10 major questions (A-J) with subsidiary questions. For each subsidiary question relevant primary and secondary data types and their sources will be designated. These will form the basis for the findings, which will form the basis for the conclusions; which in turn will underpin the recommendations. A draft Table of Evaluation Questions, Data Types and Sources is included below as Table 1. OUTCOMES and DELIVERABLES The MSI evaluation is expected to document overall performance of EDC in the execution of the SSIRI activity. This will include the consideration of primary and secondary research (with special focus on the findings of the MSI team) that describes of the overall impact of the project since its inception, including modifications to its aim, scope and dimensions. The Final Report will provide documentation, findings, conclusions and recommendations specific to the: 1) past and overall performance of the EDC team in implementing the operation; SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 45 2) Need for corrections or revisions of activities of the current operation during the remaining life of project; and 3) Continuation, termination or revision of the operation in the face of emerging needs in Southern Sudan. The Final Report will be submitted in a format consistent with MSI’s agreed-upon format with USAID Sudan. A draft table of contents is attached as Annex I. The work, site visit and travel schedule of the MSI team has been coordinated with USAID, MoEST and EDC. A time/task activity chart is presented as Annex II. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 46 Table 1: Evaluation Questions, Data Types and Sources (Acronym: KII = Key Informant interviews) Question A: Does the current project respond to the GoSS/MoEST’s desired directions for Southern Sudan? Data type Source 1. Does the current project respond to the MoEST’s desired directions for Southern Sudan? Interviews / MoEST docs esp. MoEST (Central) and USAID 2. How does SSIRI work with the MoEST (central, state and county)? MOU’s & MOA’s. / KIIs EDC / MOEST (Central, State, County) 3. How can this relationship be further strengthened? KIIs EDC, USAID, MoEST 4. Assess the role of Sudanese MoEST staff in this program. EDC workplans, KIIs EDC / MoEST/SMOE 5. To what extent is the project working towards local Sudanese ownership of the Program? EDC workplans, hiring records, KII EDC quarterly reports, org chart, SSIRI, MoEST 6. Describe and assess what measures are being taken to develop Sudanese capacities. EDC workplans, sample training schedules and records and evaluations from trainings, KIIs EDC/ MoEST/SOME 7. Describe and assess what measures are being taken to ensure sustainability beyond the program. EDC workplans, KIIs EDC / MoEST 8. How does the program complement other MoEST and education services in the country? MoEST org chart, KIIs MOE KIIs and docs, GEE, HEAR, and TAP Question B: What is the nature and quality of the relationships between SSIRI and its local partners, communities, other USAID cooperating agencies, other NGOs and donor partners? 1. How does SSIRI work with its partners? Is there a better way to strengthen SSIRI’s support to its partners? KIIs, documentation of articulation of programs (MOAs, MOUs) EDC, USAID, Other partners 2. How does SSIRI work with other USAID CAs, NGOs and donors? How can this relationship be further strengthened? KIIs with AED, Winrock, Creative Associates AED, CA, Winrock, AMURT, MDTF Question C : Are the technical areas and current approach appropriate for a USAID/Sudan’s follow-on investment? How should any future USAID investments be implemented/refocused? SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 47 1. What approaches/models should be expanded in the follow-on project? What are the strengths and innovative activities being undertaken that should be continued, scaled-up and emphasized? All Aggregate 2. How can activities focus to support the current strategy? Strengthen the CPA? USAID strategy statement / KIIs USAID / MoEST / SMOE 3. What is the best approach for working in this transitional period? MoEST development plans to preview convergence needs, Hiring records / Org charts / Budgets EDC / MOE / USAID 4. How can USAID’s education development investments work better with other education stakeholder investments? How can future programming work synergistically to transition from relief to development? KIIs MoEST / USAID / EDC / MDTF 5. Are the SSIRI interventions adequate for improving access to quality services on a large scale? (See Question C6) Demographics – number of children, MoEST / USAID / EDC 6. Is there a role for utilizing the private sector for implementation and service delivery? Is there a role for Sudanese from the Diaspora? KIIs, List of private commercial entities of scale and their areas of operation. SWAN / MoEST ministers Program related questions: Question D: Assess the overall impact of the SSIRI Program to date: To what extent is the program having an impact on access to primary education and English language literacy in Southern Sudan? 1. To what extent has the program succeeded in providing quality education to Sudanese school age children KIIs, Pre-test post-test results, test forms, sampling plan, evaluation write-up MoEST staff at all levels (princ. Teachers, etc.), parents, EDC, 2. How far has the program succeeded in developing English literacy to out of school youths and adults? KIIs with Terbia program trainers and participants, others / test results MoEST staff at all levels (esp in Alternative Ed. System) / EDC 3. How far has the program succeeded in developing the capacity of the MoEST to manage radio-based education programs at both the national and state/county levels KIIs, EDC quarterly reports, MoEST EDC / SMOE / MoEST/ SMOE SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 48 4. To what extent is the program having impact on teaching and learning in Sudan? As in 1 and 2 immediately above, aided by historic data on earlier state As in 1 and 2 immediately above, USAID earlier studies describing 5. Describe the extent to which the program has enhanced the people’s understanding of civic education concepts including governance, peace building and development. EDC evaluations, KIIs at school level, audio programs and print (no Terbia tests yet) EDC, schools, SMOE, Terbia A group from last year, Terbia B facilitator 6. Is the SSIRI approach, specific interventions and geographic coverage adequate for the provision of quality education to large number of Southern Sudanese? (Redundant w/ Question C5) EDC workplans, progress reports, radio transmission maps, KIIs EDC / MoEST / USAID . Miraya radio, Internews, SW and FM radio reception direct observations Question E: Assess program performance and progress towards achieving program results in all the key program areas as measured against targets established in the cooperative agreement, annual implementation plans and the performance monitoring plans. Are the results commensurate to the USAID investment in the program? 1. According to the Cooperative Agreement; annual work plans and PMP is the program meeting its objectives? Cooperative Agreements / Workplans, PMP (M&E plan); and Quarterly reports) EDC / USAID via MSI 2. What are the challenges, successes and lessons learned? KIIs, Quarterly reports MoEST all levels, EDC 3. What can be done to increase program reach? KIIs, MoEST all levels, EDC 4. Describe and evaluate the areas where the program has fallen short of achieving results. KIIs, EDC summary MoEST all levels, EDC, USAID Question F: Provide a brief description of the program outcomes, deliverables, and products (IRI Programs, broadcasts, teachers’ guides, teaching and learning materials, etc). To the extent possible assess the quality of the deliverables. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 49 1. Describe the quality of the IRI programs, broadcasts, teacher’s guides and other teaching and learning materials. List of all print, audio, video produced; List of technology items distributed; Sample of audio programs (#1, 40. 80, 100 from each LV series); Print materials for teachers, Broadcast schedules and periods in which they have taken place by program type and level, Reception maps and reports, KIIs Radio Miraya, EDC, primary observations in classes, examination of equipment types used Question G: Describe and evaluate the program accomplishments. 1. What are the notable successes? KIIs, EDC quarterly reports, other USAID program grantees reports AED, CA, Winrock 2. What has accounted for the success of the program? Enrolment data if available or estimates from MoEST staff, KIIs MoEST school level reports 3. Identify particular strategies, activities or programs that are effective and describe why they have worked. All Question H: What impact has the program had on development of technology based education in Southern Sudan? 1. How has the program enhanced technology based learning? What are the lessons learned; what else can be done to bridge the technological gap in the education sector in Sudan? Pre – post intervention reports of available technologies (written or KIIs), observations MoEST / EDC / SMOE Question I: Assess the quality and performance of EDC in managing the implementation of the program. 1. Assess EDC’s technical and management capacities and performance. KIIs MoEST, USAID, EDC 2. Assess EDC’s communications and relations with USAID and the MoEST. KIIs MoEST, USAID, EDC 3. Assess EDC’s compliance with the grant agreement terms and conditions. Cooperative Agreements / Workplans, PMP; Quarterly reports EDC / USAID via MSI SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 50 Question J: What strategies has the program adopted in order to bridge the gender gap in education in Southern Sudan? 1. Describe ways through which the program addresses issues of gender balance and inclusiveness. What else can be done? GEE plans, EDC Cooperative agreement, workplan and reports; participant enrollment information; training records, KIIs EDC, USAID, Winrock Data / Documents on hand or needed Status From MSI-Washington Delivered Computer file Annual Plan Gantt Chart 2007-08.xls Delivered Computer file Annual Work Plan 2007-08 Final March 5 08.doc Delivered Computer file Bi-monthly April 1-15 08.doc Delivered Computer file Biweekly Feb 08.doc Delivered Computer file March 1-15 08.doc Delivered Computer file March 16-31 08.doc Delivered Computer file SSIRI Q14 Oct-Dec 07 FinalFeb 7.doc Delivered Computer file From MSI-Sudan 623_A_00_04_054_EDC.DOC Delivered Computer file EDC 4th qt 07 report.doc Delivered Computer file EDC Coop Agree.DOC Delivered Computer file EDC LWA.doc Delivered Computer file EDC Revised Program Descrip Mod 4.DOC Delivered Computer file EDC-SSIRI eval recs.doc Delivered Computer file Education Partner Meeting DRAFT AGENDA021108.doc Delivered Computer file GEE Program Description Jan 08 No Graphics.doc Delivered Computer file Inez's comments on SOW.doc Delivered Computer file Paid Media Campaigns Dubai April 2006.ppt Delivered Computer file Version 3 0 TOR GEE Longitudinal Study 6 Feb 2008.doc Delivered Computer file From EDC - Sudan Kokole update SSIRI activities.doc Delivered Computer file SSIRI PMP revised Mar 20 08.doc Delivered Computer file sSIRI Q7 report - FINAL 28april06 lk.doc Delivered Computer file SSIRI Q8 report - FINAL.doc Delivered Computer file SSIRI Q9 report - final.doc Delivered Computer file SSIRI Q10 report - FINAL 30 Janaury 2007.doc Delivered Computer file SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 51 SSIRI Q11 report - Jan-Mar 07 Draft April 22.doc Delivered Computer file SSIRI Q12 Report April-June 2007 - Aug 6.doc Delivered Computer file SSIRI Q13 Report July-Sept - Oct 29 FINAL.doc Delivered Computer file Memorandum of Understanding between SSMoEST (Ministry), State Ministry of Education (State), and EDC Delivered Paper Map of Southern Sudan Delivered Paper Suggested List of Additional Data - (some of this information will be found in review of quarterly reports) As suggested by Annual Work Plan July 2007-September 2008 or 4th Quarter Report for 2007 Pre-post test student performance report Delivered Map of SSIRI areas of operations Delivered List of participating schools and programs List of all MOAs and MOUs completed or MOUs themselves Delivered List of all MOAs and MOUs in draft form annotated for status (early draft, negotiating, signed) Delivered MOU between MoEST and EDC (unsigned) Delivered Map showing all radio stations and transmitters broadcasting SSIRI programs Delivered Map showing all VSAT installations being used Delivered List or summary of produced programs instructional materials (Learning Village, Terbia, PST) LV,T Delivered List of all schools by Central, State, County Enrolment figures for SSIRI students (Learning Village) ( figures for each year 2006, 2007, 2008) Delivered Enrolment figures for SSIRI students (Terbia in school) ( figures for each year 2006, 2007, 2008) Delivered Enrolment figures for SSIRI students (Terbia out-of-school) ( figures for each year 2006, 2007, 2008) Delivered Enrolment or utilization figures for PST participants ( figures for each year 2005. 2006, 2007, 2008) Do not yet exist Lists of trained teachers Delivered Summary of low cost technology items acquired and status of implementation UNICEF statistics on schools and students and teachers from any year pre-2004, or pre 2003 Delivered Organizational chart for EDC SSIRI Delivered Organizational chart for MoEST at as many levels as possible AES delivered Any draft audience survey mechanisms for radio audiences (SRS or Internews or Miraya Radio) Terbia for Beginners Facilitators Guide SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 52 Terbia for Intermediate planning documents and sample scripts Do not yet exist List of all radio stations and their locations and frequencies for broadcasting SSIRI programs PST (PS101) Facilitator’s Guide Delivered PST implementation plan 2007 plan PST Briefing paper with notes on implementation for PS101 Delivered Planning paper on production of one additional series of 12 lessons Planning documents on weekly radio series on education that may be planned with MoEST and Miraya Delivered Any MOUs/ MOA or similar official agreements with Miraya Delivered MOU with Bakhita Radio Delivered Examples of low cost technologies Produced Documentation of progress on solar or handcrank devices for rechargeable batteries Eg. shown Plan for national ICT conference Clarification of monitoring and collecting reports from users of SSIRI programming Vehicle procurement orders / lists Delivered Suggested by 2007 Quarter 4 Report: Terbia for Beginners summative evaluation test design or progress report Do not yet exist Technical Working Group report for AES (scheduled for November 2007) Delivered List of schools participating in Terbia listening groups List of six states and 20% of the SS counties projected to receive additional help in 2008 Delivered Summary report (if exists) on capacity building SSIRI planning workshop - Rumbek Oct 29-Nov 2, 2007 Any report given to Radio Miraya on impact of IRI Delivered Any video programs used in training or outreach. (including Amaury Blondet’s piece) Delivered – not readable Reprogramming budget submitted to USAID Not delivered Suggested by February report: Outreach Handbook Delivered Suggested by March 31 report: M & E plans developed for Terbia for Beginners and audience survey plan for Terbia Advanced Do not yet exist USAID Strategy statement Delivered SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 53 Letter to Edward Kokole, Acting Director General, Quality Promotion and Innovation (MoEST) from T Tilson Delivered SSIRI Monitoring and Evaluation Plan Delivered Map of Southern Sudan with Evaluation sites highlighted Delivered Below are document names as provided by EDC. Some are redundant with the generic names above Outreach Coordinator Reporting andPlanning_Tools_Nov2007 Delivered TerbiaBfacilitatornames_submittedAESDepart_Dec2007 Delivered TERBIAB_facilitatortraining_report_Oct2007 Delivered 12-15-07 USAID Sudan Education Technologies_press release Delivered LCT Final Cost matrix Delivered LCT Final Devices Delivered LCT Orientation Workshop Report-Juba_Feb08_Mwamba Delivered Report on the LCT.......summary'08- Juba Delivered Rumbekplanningmeetingreport_Nov2007 Delivered Rumbek staff workshop agenda_April 2007 Delivered Agenda--27thJuly2007Staff Workshop Delivered EDC - SSIRI Project Summary on Three Areas Oct 07_1 Delivered EDC-SSIRI PMP Approval Delivered SSIRI PMP revised Mar 20 08 Delivered SSIRI scale up concept v6 edit 12 13 07 Delivered SSIRI-Terbia-PS101ProjectDescription--14thNovember2006 Delivered Terbia-PS101ProposalFINALJune6,2006 Delivered VSAT installations_proposal Delivered DRAFT_Prof Studies implementation plan_Aug 2007 Delivered PS101_OA briefing document_28_03_07 version Delivered PS101_OA Training Guide_August 2007 Delivered PST_strategy_Sept2007 Delivered TheStudent_sManual[2]_June version (developed under SBEP) Delivered P1-FieldTestingReportJune2007 Delivered SSIRI_educationreport[1]_CCRI_Watt Delivered IRITEACHERSTRAINING(1)_CCRI_Watt Delivered SSIRIandMercyCorpMOUpartnership_April2008_FINAL Delivered Miraya MOU_2008 Delivered SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 54 Draftagreement_EES state radio station_April2008 Delivered AmurtIRIMOUMay20 Delivered 6090 MoEST MOU 00 3-28-08 Delivered CDCinvitation_scriptwritingworkshop_March Delivered Directorate of Gender Equity and Social Change_survey Delivered ERDF_SSIRIPresentation_Dec2007(1) Delivered FAST TRACK_phase 3 training_SSIRI training guide Delivered FastTrackTraining,Maridi_report on IRI session Delivered MoEST_EDC_YeiworkshopReport_Feb2008 Delivered SSIRI Training Guide_MoEST_March2008 Delivered Yei Workshop Certs Delivered FM Radio Broadcast Stations Delivered SSIRI report Miraya Radio-FInal Jan 17 Delivered Voice of Kauda - radio transmission range Delivered AES Org Structure Delivered AES_implementation_guide_doc_1__SSIRI edits Delivered AES_TWGmeetingminutes_Nov2007 Delivered SSIRI program locations_staffing_May08 Delivered SSstatesregionscountyapr05_map Delivered From Janet: Delivered TeachersTrained Delivered From Lucy Delivered GEEProgramPresentationFeb08 Delivered TAPADVISOR Delivered TAPpresfor21Feb0820.2.08 Delivered Terbia Estimated Targets of Participants Delivered Counties and Payams Delivered SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 55 Annex 2 Usaid Evaluation Program Edc/Ssiri: Itinerary And Activities MAY, 2008 Date Time Activity Contact person Team Members Remarks Wednesday 7 TBD MSI Consultants and other team members arrive in Juba Inez/ Lucy IA MK, LK, TT, 2MSI, GD, MoEST12 Thursday 8 Friday 9 Two days • Orientation in Juba • Team Review together Work Plans and Statement of Work • Interview with USAID Inez & Lucy IA MK, LK, TT, 2MSI, GD, MoEST Saturday 10 All day • Interview Partners￾AED, Winrock International and if possible Creative Associates International Inez & Lucy MK, LK, TT, 2MSI, GD, MoEST Monday 12 All day • Interview Central MOEST, CE SMOE and partners Inez /Lucy IA MK, LK, TT, 2MSI, GD, MoEST Tuesday 13 9:00 11:00 14:00 16:00 • Visit the Learning Village (Primary 1 and 2 if possible) (Group 1) • Visit English Literacy lesson in secondary school in the morning (Group 2) • Visit EDC Offices • Visit English Literacy class that uses low-cost technologies in the afternoon Rejoice Manasseh Kidenybinyi@yahoo.co.uk John Onek +88216433 6175 onekakila@yahoo.com MK, LK, TT, 2MSI, GD, MoEST Wednesday; 14, 2008 8:00 12:00 14:00 • Depart for Arapi, (about 4 hrs) • Arrive Arapi • Visit the Computer and Media Centre • Observe the use of the Internet for Sarah Kiden +8821643341966 sskiden@yahoo.com MK, LK, TT, 2MSI, GD, MoEST 12 The full names of team members: IA- Inez Andrews, MK- Mitch Kirby, LK-Lucy Kithome, TT- Thomas Tilson, GD-Garang Chut Deng and 2MSI Consultants TBD SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 56 Date Time Activity Contact person Team Members Remarks 15:30 strengthening educational programs and discuss with staff • Depart for Nimule (no accommodation at Arapi)-45 Minutes drive • Spend the night at Nimule Thursday 15 8:00 Depart for Arapi TTI Sarah Kiden, IT Specialist +8821643341966 sskiden@yahoo.com MK, LK, TT, 2MSI, GD, MoEST Thursday 15 8:45 9:00 12:00 13.30 16:30 • Arrive Arapi • Observe video production in progress • Hold discussions with staff • Depart for Torit (3 hrs) • Arrive at Torit • Spent Night at Torit Sarah Kiden IT Specialist +8821643341966 sskiden@yahoo.com MK, LK, TT, 2MSI, GD, MoEST Friday 16 9:00 11:00 1:30 • Visit Torit schools for Learning Village • Visit State Ministry officials • Drive back to Juba • Spent weekend in Juba Ohisa Reuban, OC +8821667900009, 0477142941 Ohisa_reuben@yahoo.com MK, LK, TT, 2MSI, GD, MoEST Saturday 17 • Draft report MK, LK, TT, 2MSI, GD, MoEST Monday 19 9:00 11:00 14:00 • Further interviews with Central MOEST officials • Further Interview with CE SMoES & partners • Planning meeting on the second leg Inez/Lucy MK, IN, LK, TT, 2MSI, GD, MoEST Tues 20 Charter a Plane SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 57 Date Time Activity Contact person Team Members Remarks Tues 20 6:30 9:00 9:30 11:00 14:00 • Fly early to Yambio • Arrive at Yambio • Visit Learning Village class in morning • Visit State Ministry • Visit English Literacy group in late afternoon • Spent night at Yambio Robert Singira, OA 0477156804 Singira_robert@yahoo.co.uk IA, LK, TT, 2MSI, GD, MoEST Wed 21 7:30 10:00 11:00 12:00 14:00 15:00 • Fly early to Maridi • Arrive in Maridi • Visit Maridi TTI • Observe use of Internet for teacher/tutor capacity development • Hold session with staff on use of video • Fly to Wau • Spent Night at Wau Robert Wuda, OC +8821643341969 wudarobert@yahoo.co.uk IA, LK, TT, 2MSI, GD, MoEST Thurs 22 9:00 11:00 14:00 • Visit Learning Village • Visit English Literacy classes • Visit SMoE office • Spent night at Wau Bullen Nginzo, OA +8821643333760 bnginzo@yahoo.com Martina Lino, OC +8821643341967, cpsta@edc-ssiri.org IA, LK, TT, 2MSI, GD, MoEST Friday 23 6:30 9:00 11:00 14:00 16:00 • Fly early to Malakal • Visit L. Village • Visit SMoE staff • Visit the TTI￾Internet facilities • Visit English Literacy program • Spent night at Malakal Acuil Arop, OC +8821643332477 aarop@edc-ssiri.org IA, LK, TT, 2MSI, GD, MoEST Participant observation Sat 24 9:00 • Fly to Kauda (Three Areas) • Spent night in Kauda Angelo Guido, OA +882165026 2502 Tambua2003@yahoo.com IA, LK, TT, 2MSI, GD, MoEST Drafting report Sun 25 9:00 14:00 • Visit State Officials • Observe English Literacy on low cost technologies • Spend night in Kauda Angelo Guido, OA +882165026 2502 Tambua2003@yahoo.com IA, LK, TT, 2MSI, GD, MoEST This is a working day in this region SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 58 Date Time Activity Contact person Team Members Remarks Monday 26 7:00 9:00 • Fly to Agok • Get ground transportation to Abyei (45 minutes drive) • Visit State officials • Spend the night at Abyei Chirilo Cho, OC +8821643341905 cchol@edc-ssiri-org IA, LK, TT, 2MSI, GD, MoEST Tuesday 27 8:00 9:00 12:00 • Back to Agok • Visit a Learning Village school in the morning • Fly back to Juba Chirilo Cho, OC +8821643341905 cchol@edc-ssiri-org IA, LK, TT, 2MSI, GD, MoEST Wednesday, 28-29 • Wrap up and draft report writing Inez/Lucy IA, LK, TT, 2MSI, GD, MoEST Friday 30 9:00- 16:00 • Debriefing at USAID Compound Inez/ Lucy IA, LK, TT, 2MSI, GD, MoEST= Saturday 31 Consultants leave for US to finalize the final report SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 59 Annex 3: Southern Sudan Interactive Radio Instruction (SSIRI) Grade 1 Evaluation May, 2008 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 60 TABLE OF CONTENTS LIST OF ACRONYMS ............................................................................................................................. II EXECUTIVE SUMMARY ......................................................................................................................III 1.0 BACKGROUND............................................................................................................................... 5 1.1 Description SSIRI and the Southern Sudan context...................................................... 5 1.2 IRI enrolment and participation .................................................................................... 5 2.0 EVALUATION METHODOLOGY............................................................................................... 7 2.1 Purpose of the Evaluation ............................................................................................. 7 2.2 Pretest sample ............................................................................................................... 7 2.3 Posttest sample.............................................................................................................. 8 2.4 Test Development ......................................................................................................... 8 Test planning 9 Test Construction 9 2.5 Data Collection, entry and analysis............................................................................. 10 Training of Test Administrators ..........................................................................10 3.0 FINDINGS ...................................................................................................................................... 11 3.1 Grade 1 achievement in English, Mathematics and Local Language literacy ............ 11 3.2 Use of IRI and contextual factors that promote IRI effectiveness .............................. 15 4.0 DISCUSSION AND RECOMMENDATIONS ............................................................................ 17 4.1 Achievement basic skills in English, Local language Literacy and Mathematics ................................................................................................................ 18 4.2 Recommendations....................................................................................................... 20 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 61 LIST OF ACRONYMS AMURT Amanda Marga Universal Relief Team EDC Education Development Center, Inc. IDPs Internally Displaced Persons IRI Interactive Radio Instruction M&E Monitoring and Evaluation MoEST Southern Sudan Ministry of Education, Science and Technology NGO non-governmental organization NRRDO Nuba Relief, Rehabilitation and Development Organization NSCSE New Sudan Centre for Statistics and Evaluation PST Professional Studies for Teachers REACH Radio Education for Adults and Children SBEP Sudan Basic Education Programme SSIRI Southern Sudan Interactive Radio Instruction TERBIA Teaching English through Radio-Based Instruction for All TERBIA A Terbia Advanced Level TERBIA B Terbia Basic Level TTI Teacher Training Institute USAID United States Agency for International Development USAID/SFO USAID/Sudan Field Office SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 62 EXECUTIVE SUMMARY Southern Sudan Radio Instruction Program (SSIRI) is a radio-based learning intervention to increase access to quality education opportunities and to improve teaching at the lower primary level in Southern Sudan. The P1 IRI programs were first broadcast in 2005. Programs address Southern Sudan curriculum goals for English, Local Language Literacy, and Mathematics. In addition to P1, programs have been introduced in P2 through P4 classes. This report presents findings of the first evaluation of learning achievement at the P1 level. The evaluation answers questions on whether learners that are exposed to IRI programs achieve basic skills in literacy and numeracy as expected for the Primary 1, the extent to which teachers use the IRI methodology as stipulated in the programs, as well as the contextual factors that enhance the effectiveness of IRI. To answer evaluation questions, data was collected from a sample of 49 schools with 738 learners selected to represent 376 IRI participating schools. For learning assessment, a curriculum-based achievement tests was administered two times, a pretest in April and a posttest in November 2007. Findings summarized below respond to specific evaluation questions and provide insights into the characteristics of the teachers and learners in governments schools in Southern Sudan. . IRI participating schools in Southern Sudan enrolled 42,045 learners, 62.4 percent boys and 37.6 girls for all levels of primary in 2007. With 53.8 percent of P1 learners being older than 8 years, learners were generally older than expected for the P1 grade. Most teachers (88.2 percent) in IRI classes have a formal teaching qualification; most of them (60.0 percent) have also been trained in IRI. The first question of the evaluation was whether learners that are exposed to IRI programs achieve basic literacy and numeracy skills stipulated in the curriculum for the Primary 1 level. Significant gains were registered in all three subtests, even though learners performed poorly in Local Language Literacy. Learners registered a gain of 27.8 percent in English, from a mean pretest of 15.4 percent to a mean posttest of 43.2 percent. Learners gained 12.7 percent in local language literacy, from the mean pretest of 15.3 to 27.7 percent in the posttest, while the gain in mathematics was 21.8 percent (from 35.1 percent to 56.8 percent). The results showed, also, that learners in IRI schools performed better in all three subtests than learners in non-IRI control schools. The mean difference between IRI and control learners was 14.1 percent for English, 4.3 percent Local Language Literacy and 4.3 percent for Mathematics. There were benefits in learning for learners who attended school and were present during IRI lessons, and for learners whose teachers were trained in IRI performed better than those whose teachers did not receive IRI training. While several suggestions for improvement were made throughout the report, recommendations and follow￾up action for project improvement are as follows: 1. There is need to set performance standards or targets for each grade level. This will be useful in providing guidance on what the critical learning behaviors and outcomes for each grade level ought to be. Besides reporting progress based on curriculum standards (and not mean scores),pperformance standards will be useful for benchmarking the Southern Sudan education system against global standards in the education sector, as well as for tracking performance in the different regions of the country. 2. MoEST needs to clarify what ‘local language literacy’ is in the context of Southern Sudan and why it is important to learn a local language if it is not synonymous with teaching children in their ‘mother tongue.’, SSIRI could then come up with improved strategies for teaching local language literacy, as well as for MoEST to develop a comprehensive area/local language policy and tools for its implementation. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 63 3. Field monitors should systematically investigate why the programs are not being used. A good place to start would be to ensure that there is good reception in all areas, that broadcast times are published in all states, and that Outreach Coordinators assist teachers with timetabling issues where those arise. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 64 1.0 BACKGROUND 1.1 Description SSIRI and the Southern Sudan context Southern Sudan Interactive Radio Instruction (SSIRI) is a project that uses radio-based learning and other technologies to expand non-formal and alternative education opportunities for Southern Sudanese people. Its goal is to support efforts to improve access and quality in education in Southern Sudan and the three regions of Abyei, Blue Nile, and Southern Kordofan. With funding from USAID, SSIRI is jointly implemented by Southern Sudan Ministry of Education, Science and Technology (MoEST) and Education Development Center, Inc. (EDC). SSIRI is one of several USAID funded programs that were designed to help reconstruct an education sector which was crippled by many years of civil war. According to estimates by the New Sudan Centre for Statistics and Evaluation (NSCSE) and UNICEF, Southern Sudan had a gross enrolment ratio of 25.3 percent at the primary school level in 2004, which is the lowest access to primary education in the world13. At 35.0 percent, Southern Sudan also had the lowest ratio of female to male enrolment. Other key findings of the NSCSE investigation indicated low survival rates to P5 (28 percent) and even lower rates at P7 (2 percent). Only 6 percent of teachers received at least one year of pre-service training, 45 percent with two weeks to three months of in-service teacher training, and 49 percent of teachers have no training at all. Only 7 percent of the country’s teaching force are women. The adult literacy rate is just 24 percent, and even lower for women at 12 percent. . Given the largely untrained teaching force, learners experience a wide range of pedagogies and teaching practices with such variations and inconsistencies presenting further obstacles to learning. In 2005 the Ministry of Education, Science, and Technology (MoEST) announced its aims to (a) enroll 200,000 additional pupils to reach a total of 700,000 pupils, and (b) to recruit 8,000 more teachers by the end of 2006. As ambitious as this goal may seem, it still left more than half of Southern Sudan’s 1,500,000 school-aged children without access to quality education. Some of the issues mentioned here are at the heart of SSIRI and its IRI, TERBIA and PST elements. The three components of SSIRI are Learning Village, a supplementary program to enhance teaching and learning of the primary school curriculum; Teaching English through Radio-Based Instruction for All (TERBIA)14 that engages youth and adult learners in Civic Education and English Language instruction, and Professional Studies for Teachers (PST), a twelve-week accelerated in-service teacher training course. Project objectives that correspond to these components include improving access to learning opportunities, enhancing the primary program to attain higher learning achievement gains, improving teaching skills of participating teachers, and improving the capacity of government officials in management of radio programs. 1.2 IRI enrolment and participation In the two years of implementation (2006 and 2007), SSIRI has achieved important milestones of developing P1 to P4 IRI programs, enlisting lower primary classes to participate in IRI, distributing radios and teachers guides to IRI participating schools, enlisting and contracting with radio stations to broadcast SSIRI programs, and training teachers in the IRI techniques. SSIRI lessons were being broadcast to 42,045 Primary 1 and 2 learners, 36,711 in 7 states of Southern Sudan, and 5,334 learners in the Three Areas (Blue Nile, Abyei, and Southern Kordofan) by the end of December, 2007. Table 1 presents learner enrolments by area/state and sex.. 13 Towards a baseline Best Estimates of Social Indicators in Southern Sudan, UNICEF and New Sudan Centre for Statistics and Evaluation in association with UNICEF, May 2004 14 Teaching English through Radio-Based Instruction for All SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 65 Table 1: Number of IRI learners in 2007, by state and sex State Female Male Total Southern Sudan Western Equatoria 4 914 3 276 8 190 Jonglei 14 429 7 995 22 424 W. Bahr el Ghazal 462 308 770 Upper Nile 651 434 1 085 Central Equatoria 315 210 525 Lakes 1 222 640 1 862 Sub-Total 23 106 13 605 36 711 Three Areas Blue Nile 986 778 1 764 Southen Kordofan 378 252 630 Abyei 1 764 1 176 2 940 Sub-Total 3 128 2 206 5 334 Total 26 234 15 811 42 045 Approximately 2 of 3 learners in IRI participating schools (62.4 percent) are boys, while girls make only 37.6 percent, a slight improvement over the 35 percent participation rate of girls cited earlier. SSIRI has also made progress in training P1 and P2 teachers in the IRI methodology, a necessary condition for effective and efficient use the IRI programs. Table 2 indicates that a total of 828 teachers have been trained in IRI. Only 14.9 percent of the trained teachers are women.. Teachers who were trained in IRI have also been provided with radios and teachers guides. Table 2: Number of teachers trained in IRI, by state and sex Region Male Female Total Southern Sudan 643 109 752 Three areas 62 14 76 Total 705 123 828 SSIRI has a number of project activities that include developing programs, airing the lessons, distributing radios and other materials, training of teachers, and monitoring implementation of the programs. All these are routinely monitored by the M&E section. The focus of this evaluation was to assess the impact of IRI. The evaluation set out to investigate the extent to which IRI was implemented in Southern Sudan and the Three Areas, and whether or not IRI participating teachers were maximizing the opportunity provided by the programs by using IRI as prescribed in their training. A sample of P1 pupils who had listened to the IRI programs (IRI learners) and those who were not using radio programs (control learners) were selected and subjected to learning achievement test in Mathematics, English, and Local language Literacy to investigate if IRI was an effective tool for learning. This report presents results of the evaluation, an interpretation of the findings, and recommendations for further action by the MOEST and/or SSIRI. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 66 2.0 EVALUATION METHODOLOGY 2.1 Purpose of the Evaluation The purpose of this evaluation is to investigate whether P1 learners exposed to IRI programs achieved basic numeracy, basic English language comprehension skills, and functional literacy in a local language as stipulated in the curriculum. The evaluation further investigated whether teachers are using the IRI programs as expected in terms of their interaction with the radio lesson, their interaction with learners, and using IRI materials such as teachers’ guides. Factors that enhance the effectiveness of IRI as attendance of IRI lessons, and the learning environment were also investigated. To that end, the evaluation focused on the following questions: 1. Do learners that are exposed to IRI programs achieve basic skills in literacy and numeracy as expected at the Primary 1? 2. To what extent do teachers use the IRI methodology as stipulated in the programs? 3. What are the contextual factors (learner, teacher, or learning environment) that enhance the effectiveness of IRI? 2.2 Pretest sample The population of IRI learners for which data was received was 42,045 learners attending 328 schools. The design of the evaluation study was pretest posttest with control groups. A multi-stage purposive sampling strategy was used in both the pretest and posttest. Factors that were considered in the selection of evaluation sites included: 1. Whether IRI was implemented in the area, in terms of radio reception and actual tuning into radio broadcasts by teachers 2. Donor’s emphasis on the Three Areas, as well as urban areas, 3. Practical and logistical considerations such as accessibility of sites, flight availability, availability of test administrators, financial resources, availability of accommodations and transport. The Learning Village is currently being implemented in 17 locations, 5 locations where implementation is done through partnership, and 12 locations where SSIRI project staff oversee implementation. The program reaches 376 schools and at least 376 P1 classes. Some schools have more than one P1 stream. Four (4) of 10 states in Southern Sudan wereselected. Two (2) of the Three Areas were also selected, as well as 1 of the 3 major towns that met the criteria above. This gave adequate representation of all the areas where the program is currently being implemented. The towns/areas of Maridi, Panyagor, Malakal, Kauda, Aweil East were selected as pretest sites. In Aweil East IRI was implementation through AMURT, a local NGO. These locations were stratified along urban/rural, the Three Areas, and the larger Southern Sudan. For the urban areas, Malakal was selected; Kauda represented the Three Areas; and Maridi, Aweil and Panyagor represented the greater Southern Sudan. The table below shows the classes targeted for the assessment. . The following table shows the sample calculation for classrooms used for testing P1 literacy, English, and numeracy. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 67 Table 3: Pretest sample from IRI participating schools, by state/location Location Total P1 classes IRI classes Control classes Total classes Total Learners Maridi 31 6 1 7 101 Panyagor 63 12 3 15 174 Malakal 20 4 1 5 78 Aweil 53 10 2 12 181 Kauda 18 3 1 4 30 Abyei 12 2 1 3 45 Total 197 38 8 46 609 In each location there are IRI schools where implementation is carried by SSIRI staff, and other schools where implementation is supported by partner NGOs. Both types of schools were included in the sample. At the school level, a random sample of 15 learners was selected. As far as possible, the learners for the posttest who have attended at least 80 percent of the lessons. 2.3 Posttest sample To the extent possible, the posttest was administered in the same schools that participated in the pretest, with the intention also to select the pupils who participated in the pretest. Two locations were substituted during the post test, Malakal and Kauda as shown in Table 4. Table 4: Posttest sample of schools, by state Location IRI classes Control classes Total classes Total Learners Maridi 6 1 7 99 Panyagor 12 3 15 227 Malakal 4 1 5 73 Aweil 10 2 12 184 Abyei 2 1 3 50 Juba 5 2 7 105 Total 39 10 49 738 First, Malakal was replaced with Juba. Malakal schools were closed during the postest due to flooding and an outbreak of meningitis. Juba was a good location in that IRI broadcasts through FM radio that makes the signal stronger and clearer than most areas. The second location that was substituted was Kauda in Nuba Mountains. Schools in this area started their broadcast much later than other schools due to the late start of their school year. An equal number of schools that were pretested in Kauda were redistributed to Juba and Maridi. 2.4 Test Development This section describes the rationale for developing a Local Language Literacy, Mathematics and English Language achievement tests. The tests developed werea curriculum-based mastery tests. Stages of the test development process included test planning, item writing, and pilot testing. The test administration procedure is also described briefly. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 68 Test planning The test development process commenced by a content analysis for Primary 1, 2 and 3, performed by the Education Advisor and Test Development Specialist. Instructional objectives from the Mathematics, Local Language and English syllabi were analyzed with the intent to distinguish between developmental and terminal objectives. Three test plans were developed. In the absence of grade-level reading lists that usually indicate the reading levels of learners, the teachers’ guide was particularly useful in that it specifies new English language and local language vocabulary and the numeracy skills that are presented in each lesson. The syllabus and teachers guides also guided the test development process in terms of the cognitive skills that children have to master at this formative stage of being introduced to formal learning. Test Construction The purpose of the test was to assess and evaluate if learners have mastered basic literacy skills in a local language as well as basic numeracy skills, and whether they could understand simple communication in English at the end of the Primary 1 syllabus. The guiding principles during test development was that assessment procedures should match the intentions of each learning target, hence, the behaviors elicited from learners included recalling certain facts, as well as performing certain tasks. For instance, the intention of the learning targets on language during the early stages of learning is that learners should comprehend language and begin to produce simple language. Their comprehension of language in the lessons is demonstrated by the acting out in response to simple instructions, hence, the assessment of language skills comprised mainly of requesting them to perform actions when given simple instructions. One form of the test was constructed for each of the three learning areas for Primary 1. Where possible, a set of parallel items was presented from which the test administrator would select the item to present to the learner. Table 5 presents a summary of the skills assessed and the weighting of each skill area. Table 5: Skill areas and corresponding test items for Primary 1 Test, 2007 Skill Area Intended Learning Target Tasks Points Recalling names 1, 2 4 Simple comprehension of language English 7 2 Language Production of language (speaking) 3,4,5,6 11 Production of language (speaking) 5 2 Production of language (reading) 1, 3, 6 9 Local Language Literacy Production of language (writing ) 2, 4, 7 8 Counting and writing numbers 1, 2, 5, 6 11 Mathematics Number operations 3, 4, 7, 8 8 Naming and drawing shapes 9, 10 4 The English and Local Language tests each had 7 tasks, while mathematics was comprised of 10 tasks. All tests were developed in English, but only the English subtest was to be administered to the learner in English. Test administrators were required to translate the Local Language Literacy and Mathematics subtests from English to the area local language, presumed also to be the medium of instruction for schools in the area. rial Testing SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 69 Items were pilot tested by test administrators in two centers in Rumbek. Trial testing assessed whether the questions elicited the intended behavior/skills, and whether the correct difficulty levels in terms of content and language were maintained. The amount of time it took to administer the test was important in that children at the P1 age have a short attentions span, and whether the proposed administration procedure was reasonable. After trial testing, a debriefing session was convened to receive additional feedback on how the test functioned. Interactions between learners, test administrators, and the test were noted for interpretation and for improving the test. Explanatory notes for each test form and a quick reference guide for the test administrators were developed. Trial testing provided feedback on the reasonableness and appropriateness of the test for testing literacy and numeracy skills at P1, and whether the learners were able to handle the format of the test. 2.5 Data Collection, entry and analysis Training of Test Administrators Training of test administrators was conducted in Rumbek by the SSIRI technical team, using a test administration booklet. Test administrators were briefed on the purpose of test, how the test was developed, the behaviors that each item intended to elicit, and how it was to be scored. Test administrators practiced administering the test in pairs, and then went out to the schools to conduct trial tests. Live Testing The pretest was administered close to the beginning of the school year in April 2007 over a period of 15 working days, while the posttest was conducted in November 2007 for another period of 15 working days. There were 5 teams of test administrators, each consisting of 3 people. Each of the team members was assigned a specific responsibility in the test administration. Data Entry Data entry commenced soon after the testing. Two data entry assistants entered the data into MS-access. Data was transported into MS Excel and finally into SPSS for analysis and developing result tables. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 70 3.0 FINDINGS The design of the evaluation study was pretest posttest with control groups. The original sample of learners that was selected from seven locations, namely, Maridi, Panyagor, Aweil Abyei, Kauda and Malakal. Pretests were administered at these locations at the beginning of the year. However, Kauda was later dropped from the sample because the school year starts much later than in other areas and, thus, they would have covered fewer than half the broadcasts at the time of posttesting. Malakal schools were closed during the posttest due to flooding and an outbreak of meningitis. Juba was chosen as a replacement for the two locations. Juba represents an urban town and has an FM station with IRI broadcasts that has a clear signal. A total of 57 schools participated in both pretest and posttest. Forty-six (46) schools participated in the pretest. Sixteen (16) of the schools were dropped at the posttest and replaced by 11 schools. Hence the posttest sample comprised of 41 schools, 31 being IRI and 10 being control schools. 3.1 Grade 1 achievement in English, Mathematics and Local Language literacy Performance on the pretest and posttest The first question of the evaluation was whether learners that are exposed to IRI programs achieve basic skills in literacy and numeracy as expected at the Primary 1 level. The mean pretest for IRI learners was 15.4 percent in English, while the posttest score was 43.2 percent. Learners registered a gain of 27.8 percent. Learners gained 12.7 percent in local language literacy, from the mean pretest of 15.3 to 27.7 percent in the posttest. In mathematics the pretest score was 35.1 percent, while the posttest scores was 56.8 percent as shown in Table 6. Table 6: Pretest and posttest means for all IRI learners, by subtest Subtest N Maximu m Score Mean Mean Percent Mean Gain (%) English Pretest 415 17.0 2.6 15.4 English Posttest 415 17.0 7.4 43.2 27.8 Local Lang Pretest 418 19.0 2.9 15.2 Local Lang Posttest 418 19.0 5.3 27.9 12.7 Mathematics Pretest 419 23.0 8.1 35.1 Mathematics Posttest 419 23.0 13.1 56.8 21.8 The pretest scores were higher in mathematics than in than in English or Local Language Literacy. This is typical of performance at the Grade 1 level, mainly because children acquire more numeracy skills from non structured and non deliberate learning than they do with literacy skills. A paired-samples comparison indicates that differences in the pretest and posttest are significant,15 an indication that after a year of schooling children did benefit from the learning activities including IRI. IRI methodology deliberately sets out to make learning interactive and interesting, and to motivate all children to attend and participate. Comparison of IRI learners and a non-IRI control group Some schools in the control sample were replaced, which means that the pretest and posttest control samples are not equivalent. However, this analysis treated the control samples as a random group of learners and 15 English Pretest/Posttest: t =20.41, p =.00; Local Language Literacy Pretest/Posttest: t =11.60, p =.00; Mathematics Pretest/Posttest: t=14.85, p = .00 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 71 compared the posttest scores with the pretest scores. Control learners performed significantly better than IRI learners in all three pretests as shown in Table 7. Table 7: Pretest means, by type of learner and subtest Learners Subtest N Maximum Possible Mean Mean Percent Mean Diff (%) English IRI Learners 535 17.0 2.6 15.4 Prettest Control Learners 60 17.0 4.6 26.8 -11.4 Local Lang IRI Learners 546 19.0 2.9 15.2 Prettest Control Learners 60 19.0 5.8 30.4 -15.2 Mathematics IRI Learners 539 23.0 8.1 35.1 Prettest Control Learners 60 23.0 11.5 50.1 -15.0 The deficit was 11.4 percent for English, and as high as 15 percent for Local languages and Mathematics. A sample of learners, most of which took the pretest, participated in the posttest at the end of the school year. The posttest sample comprised of 41 schools, 31 schools that participated in IRI, and 10 control schools. Table 8 below compares performance of IRI learners on the posttest with that of control learners. Table 8: Posttest means, by type of learner and subtest Learners Subtest N Maximum Possible Mean Mean Percent Mean Diff (%) English IRI Learners 585 17.0 7.4 43.2 Posttest Control Learners 103 17.0 4.9 29.1 14.1 Local Lang IRI Learners 585 19.0 5.3 27.9 Posttest Control Learners 104 19.0 4.5 23.6 4.3 Mathematics IRI Learners 586 23.0 13.1 56.8 Posttest Control Learners 103 23.0 12.1 52.5 4.3 In English IRI learners had a posttest mean of 43.2, while the learners in the control group posted a mean of 29.1 percent. The mean posttest for IRI learners was 27.9 percent in Local Language Literacy, compared to a posttest score of 23.6 percent for control learners. In mathematics the IRI posttest score was 56.8 percent compared to a posttest score of 52.5 percent for control learners. The difference between IRI and control learners was wider in English, where IRI learners seem to have benefited the most. An independent samples t-test comparison of means between IRI and control learners yielded significant differences16, an indication of the ‘value-added’ by IRI. There were no significant differences between boys and girls in the pretest. Boys and girls posted similar performance on the posttest as shown in Table 9. 16 English Posttest: t =6.37, p =.00; Local Language Literacy Posttest: t =3.56, p =.00; Mathematics Posttest: t=2.49, p .01 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 72 Table 9: Posttest mean scores by type of learners and sex Type of Learner Sex of Learner Subtest N Maximu m possible Mean Mean Percent English Posttest 329 17.0 7.4 43.6 Male Local Literacy Posttest 329 19.0 5.5 28.8 Mathematics Posttest 330 23.0 13.5 58.9 English Posttest 256 17.0 7.3 42.8 Female Local Literacy Posttest 256 19.0 5.1 26.7 IRI Learner s Mathematics Posttest 256 23.0 12.5 54.2 English Posttest 46 17.0 4.9 28.8 Male Local Literacy Posttest 47 19.0 4.2 21.9 Mathematics Posttest 56 23.0 12.4 53.7 English Posttest 57 17.0 5.0 29.3 Female Local Literacy Posttest 57 19.0 4.8 25.0 Control Learner s Mathematics Posttest 56 23.0 12.4 53.7 However, there was a significant difference between boys and girls participating in IRI in the mathematics posttest scores. Contrary to expectation and previous IRI results elsewhere, IRI implementation put boys at an advantage over girls. Mean comparison by age of learner A number of learners that participated were quite young, younger than the school going age of 7 years old. For IRI learners results generally showed an increase in performance as age increased in two of the subtests with the oldest age group performing best. However, the 7-9 age group performed better than the older children in Mathematics as indicated in Table 10 below, even though the difference is not significant. Table 10: Posttest mean scores for IRI learners, by age category Age categories Subtest N Maximum possible Mean Mean percent English 59 17.0 7.1 41.8 6 years Local Language Literacy 59 19.0 4.2 22.1 Mathematics 59 23.0 11.9 51.7 English 447 17.0 7.1 41.8 7 -9 years Local Language Literacy 448 19.0 5.1 26.8 Mathematics 447 23.0 13.3 57.8 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 73 10 and English 182 17.0 8.0 47.1 above Local Language Literacy 182 19.0 6.0 31.6 Mathematics 183 23.0 12.8 55.7 The mean differences for English and Local Language Literacy were significant17. A similar analysis for the control group returned non-significant means for both English and Mathematics. The data seems to suggest that age has to be taken into consideration for future enrolment policy and practice in schools. Mean comparison by number of IRI lessons attended Of the 713 IRI learners who participated in the posttest, 125 did not have data on the attendance variable. Thirty nine (39) learners (6.6 percent) posted high attendance, which means that they attended 80 of 100 lessons or more, while 139 learners (23.6 percent) posted medium attendance (60-79 lessons). 410 learners (69.7 percent) attended 59 lessons or fewer. Table 11: Posttest mean scores for IRI learners, by lessons attended Attendance Subtest N Maximum possible Mean Mean percent High English 39 17.00 8.1 47.5 80-100 lessons Local language Literacy 39 19.00 5.3 28.1 Mathematics 39 23.00 14.3 62.0 Medium English 137 17.00 8.6 50.6 60-79 lessons Local language Literacy 137 19.00 5.7 30.1 Mathematics 137 23.00 13.8 59.9 Low English 440 17.00 6.9 40.4 0-59 lessons Local language Literacy 440 19.00 5.2 27.2 Mathematics 440 23.00 12.7 55.3 The group that posted the lowest participation in IRI performed significantly lower in all 3 subtests as indicated in Table 11. With medium attendees performing better in English and Local Language Literacy, and high attendees performing better in Mathematics, the results suggested that a minimum of 60 lessons is what learners needed to do well in all three subtests. 3.2 Use of IRI and contextual factors that promote IRI effectiveness Means comparison by IRI training The majority of the teachers in the sample (80.4 percent) had completed primary or secondary schools schooling, but did not have a formal teaching qualification, hence, initial IRI training prepared them to better receive and utlilize the radio programs. 73.7 percent of P1 teachers in IRI participating schools received face-to-face training on IRI techniques. Performance for all learners was disaggregated by whether or not their teachers had received IRI training. Table 12 indicates that learners in classes where 17 English: F = 3.2, p = 04;; Local Language Literacy: F=8.5, p = .00, Mathematics: F = 2.3., p =.10) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 74 teachers were trained in IRI performed significantly better in English, and were at par with control learners in a third subtest. The mean differences were significant in English and mathematics. Table 12: Posttest means for all learners, by training of teachers in IRI IRI training Subtest N Maximum Mean Mean Percent Trained English 541 17.00 7.3 42.8 Local Language Literacy 541 19.00 5.1 27.1 Mathematics 542 23.00 13.2 57.5 Not English 147 17.00 5.9 35.0 trained Local Literacy 148 19.00 5.3 27.9 Mathematics 147 23.00 11.9 51.6 Broadcast Lessons missed About two-third of the learners (63.5 percent) were in classes that missed eleven or more days of broadcasts due to radio problems according to teacher self-reports, while 36.5 percent were in classes that missed only up to ten days of broadcasts. Missing more that 11 days of broadcasts affected learner performance significantly for English and Mathematics as shown in Table 13. Table 13: Posttest mean scores by broadcasts missed due to radio related problems Lesson missed during the year N Maximum Mean Mean Percent English 209 17.0 8.5 49.9 0-10 lessons Local Language Literacy 209 19.0 5.7 29.8 Mathematics 210 23.0 14.1 61.2 English 166 17.0 6.8 40.1 11-20 lessons Local Language Literacy 166 19.0 5.9 31.0 Mathematics 166 23.0 12.9 56.2 English 196 17.0 6.6 38.9 21 and over lessons Local Language Literacy 196 19.0 4.5 23.7 Mathematics 196 23.0 12.2 53.2 Availability of Teacher’s Guides Performance was disaggregated by whether or not teachers had a teacher’s guide for only part of the year or for the full year as indicated in Table 14. The results seem to suggest that the availability of the teacher’s guide did not benefit the learners as was expected, but this may be due to the fact that teachers were only asked if they had the guide, and not if they used it. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 75 Table 14: Posttest mean scores by availability of teacher’s guide Avaliability of teacher’s guide N Maximum Mean Mean Percent English 178 17.0 7.7 45.3 No guide Local Language Literacy 178 19.0 4.8 25.2 Mathematics 178 23.0 13.1 56.8 English 177 17.0 6.1 36.1 Had guide since Term 1 Local Language Literacy 177 19.0 4.7 24.6 Mathematics 178 23.0 11.4 49.7 English 214 17.0 8.0 46.9 Had guide since Term 2 Local Language Literacy 214 19.0 6.4 33.4 Mathematics 214 23.0 14.3 62.2 Medium of Instruction Teachers were asked about the medium of instruction used in their classrooms. Dinka was used as the medium of instruction for 49.2 percent of the learners, Juba Arabic was used for 28.0 percent of the learners, while 21.1 percent of learners were reported to be in classes that used English as the medium of instruction. Table 15 indicates a definite benefit in English Language performance for children who used English as the medium of instruction. Other than that, learners who were taught in Dinka posted average performance in English and very good performance in Mathematics. Learners who were taught in Juba Arabic posted the weakest performance in each of the three subtests. Table 15: Posttest mean scores by medium of instruction Medium of instruction Subtest N Maximum Mean Mean Percent English 330 17 8.1 47.4 Dinka Local Language Literacy 330 19 5.9 31.0 Mathematics 330 23 13.9 60.3 English 195 17 5.8 34.1 Juba Arabic Local Language Literacy 195 19 4.3 22.6 Mathematics 195 23 11.9 51.9 English 60 17 8.5 50.2 English Local Language Literacy 60 19 5.4 28.2 Mathematics 61 23 12.4 53.7 Location (state) The data was also disaggregated by locality, and reported for the five states that were involved in assessment of learning. Table 15 reflects the posttest means for IRI learners by state. With a composite mean score of 53.5 for all three subtests Abyei state posted the highest performance while Western SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 76 Equatoria performed lowest (a composite mean score of 32.3). Learners posted the strongest performance in Mathematics and weakest in Local Language Literacy in all 5 states. Table 16: Posttest mean scores for IRI learners, by state State Subtest N Maximum Possible Mean Mean percent Composit e Mean English 46 17.0 9.3 54.9 Abyei Local language Literacy 46 19.0 8.1 42.8 53.5 Mathematics 46 23.0 14.4 62.7 Nothern Bahr English 150 17.0 7.7 45.1 el Ghazal Local language Literacy 150 19.0 6.6 34.9 46.2 Mathematics 150 23.0 13.4 58.5 Western English 133 17.0 5.3 30.9 Equatoria Local language Literacy 133 19.0 3.8 20.2 32.3 Mathematics 134 23.0 10.5 45.8 English 163 17.0 8.2 48.5 Jonglei Local language Literacy 163 19.0 4.3 22.8 43.8 Mathematics 163 23.0 13.8 60.2 Central English 93 17.0 7.3 42.9 Equatoria Local language Literacy 93 19.0 5.5 29.2 44.5 Mathematics 93 23.0 14.1 61.4 While the overall percentage for low category learners was 69.7 percent, the proportion of low category learners in Western Equatoria was 79.9 percent, which means that they missed considerably more IRI lessons. Participation in IRI lessons was lowest in Central Equatoria (85.9 percent in the low category), even though learners in the state performed better in Mathematics. 4.0 DISCUSSION AND RECOMMENDATIONS The focus of the evaluation was to assess the impact of IRI on learning at Primary 1 in Southern Sudan. The evaluation answers questions on the extent to which teachers use the IRI methodology as stipulated in the programs, whether learners that are exposed to IRI programs achieve basic skills in literacy and numeracy as expected at the Primary 1, and whether there are contextual factors (learner, teacher, or learning environment) that enhance the effectiveness of IRI. Performance of IRI and control learners was also compared. This section discusses findings and makes recommendations that have implications for strengthening implementation of SSIRI. 4.1 Achievement basic skills in English, Local language Literacy and Mathematics The results of the evaluation indicate that IRI had a significant impact on learning. IRI learners posted significant gains between the pretest and posttest in all three subtests. But if the mean score is considered as a percent of the curriculum content that was mastered, performance in Local Language Literacy was poor (27.9 percent), somewhat better in Mathematics (43.2 percent), and even better in English (56. 8 percent). IRI learners registered the highest gains between the pretest and posttest in English. This was due mostly to the SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 77 fact that IRI programs are produced and broadcast in English, even though there are Local language Literacy and Mathematics segments in each program. The Local Language Literacy and Mathematics segments are directly communicated to the class in English, and teachers are asked to translate into the local language. Hence, there is more exposure to the use of English language. Conversely, there was no exposure or modeling of local language use from the radio teacher, perhaps accounting for the poor performance on the local language literacy test. Reasons postulated for poor performance in the local language test include the fact that the Local Language Literacy test was also developed in English, with test administrators being directed to read each item and translate it into the area local language, presumably the medium of instruction for the learners. Reports from the field indicate that the strategy did not work well; there were inadequacies in communicating the tasks to the learners in cases where test administrators were not sufficiently proficient in the area local language. Second, the curriculum stipulates the teaching of ‘mother tongue’. Mother tongue in has been operationalized IRI programs to mean an ‘area local language’ or language of the catchment area.. But whatever the case, most schools do not teach in mother tongue or in the local languages beyond what IRI programs prescribe. There are no local language learning materials in most schools. This means that children do not get to see the written text, let alone acquire the necessary reading or writing skills in a local language. Where schools do make an effort to teach literacy in the local language, learners in many classes speak different languages, which means that what is presumed to be first language for all students may, in fact, be a second language for many of the students. Neither IRI nor the conventional teaching programs prepare teachers adequately for local language teaching, or to handle such complexities. A further investigation (see Table 15) revealed that 20 percent of the learners use English as a medium of instruction, which suggest that there is no exposure to a Local Language for this group of learners. While SSIRI’s role is to make sure that IRI is implemented to the fullest for learners to derive maximum benefit , SSIRI can also assist in influencing MoEST towards putting in place or clarifying certain policy frameworks. First, there is need to set performance standards on learning targets. This will be useful in providing guidance on what the critical learning behaviors and outcomes for each grade level ought to be, as well as reporting progress based on curriculum standards (and not mean scores). Secondly, there is need to clarify what ‘local language literacy’ is in the context of Southern Sudan and why it is important to learn a local language if it is not synonymous with teaching children in their ‘mother tongue.’ This would help SSIRI to come up with strategies for teaching local language. To the extent that children in the earliest grades learn better in a language they are familiar with, the relevant authorities in MoEST should be assisted to develop a comprehensive area/local language policy and tools for its implementation. Comparison of IRI and control groups In comparing IRI and control learners, control learners performed significantly better in all pretests, a result that is difficult to explain since the project aimed at a random sample in both groups. The results indicate that the control sample had an advantage over the IRI sample at the beginning of the school year. Conversely, IRI learners performed significantly better than non-IRI control learners in all three posttests. IRI methodology deliberately sets out to make learning interactive and interesting. Using song, play and a variety of actors, learners are required to listen attentively and respond actively to different activities several times during the broadcast. Implicit in each lesson are teacher training strategies designed to motivate the learners and increase their chances of success for both teachers and learners. For example, the radio actively engages teachers in organizing in a certain manner; by calling children to the front, asking children to work in pairs or small groups, work as a whole class, and to explore the learning environment. In addition, the lessons make a point of reviewing and reinforcing skills and concepts taught in earlier lessons and suggesting activities for before and after the broadcast. Teacher practices are also reinforced, and there is a likelihood of a transfer of these instructional techniques in other classroom settings. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 78 Comparison by sex and age for IRI and control groups Performance between boys and girls was similar in all pretest scores, and in all but one posttest scores. Boys performed significantly better than girls in the mathematics posttest. The IRI methodology has been shown to be successful in providing equal opportunity to boys and girls, and being an ‘equalizer’ in learning achievement in other places. SSIRI should examine their programming, teacher training, and actual classroom practices to ensure that girls are not being unduly disadvantaged in mathematics. Additional strategies should be found to make mathematics learning more accessible to girls so that they can succeed in a school culture where girls and women are a minority. One of the possible actions would be to increase role models for girls at the lower primary grades by bringing in more women teachers into the schools. According to our sample, 40 percent of P 1 learners were older. This is common in most developing countries generally, and typical for post-conflict situations in particular. Children have to reach a certain critical age to better cope with learning of concepts. Even though the correlation between age and performance was not perfect, the results indicated that from 7 years old children performed better; they were developmentally ready to engage in learning activities. SSIRI can avail this information to MoEST to assist in determining school readiness and the official school-going age. Contextual factors that enhance IRI effectiveness A number of factors are necessary for effective learning using IRI. A well functioning radio and clear signal are necessary. Also, teachers have to be trained in the IRI approach and they have to use the radio consistently. Both teachers and children have to participate as directed by the radio teacher and teachers and have to follow through with activities prescribed for the period before and after the broadcast. These and related issues were the subject of the questionnaire administered to teachers in IRI schools sampled for assessment. Learner performance was then disaggregated by factors such as training of teachers in IRI and whether teachers were using a teacher’s guide. Mean differences for learners whose teachers were trained in IRI were significantly better in English and Mathematics. There was no difference in Local Language Literacy in mean scores for learners whose teachers were trained in IRI (27.9 percent) and for learners whose teachers were not trained in IRI (27.1 percent). This finding further confirms the observations above on the need to revisit, not only the testing, but the policy on the teaching of Local Language Literacy. A more pressing issue for SSIRI is that all segments of IRI programming are supposed to demonstrate value added to of targeted learning areas, thus, adequacy of Local Language Literacy segments of IRI programs should be investigated systematically. Locality was another contextual factor that yielded differences in performance. Of the five states that were sampled, learners from the Abyei state performed best in all three subtests (with a composite score of 53.5), while those from Western Equatoria posted the weakest with a composite score in thethree subtests 32.3. Contrary to the findings, Western Equatoria was expected to perform better since the region is at a relative advantage in terms of being situated in a more reachable location, has good radio reception, and schools in the state continued to run even when there were interruptions in other places. As the state that hosts the SSIRI office within Southern Sudan, Central Equatoria also performed below expectations. From self reports of teachers, an aggregate of 69.7 percent in all five states fell in the low attendance category (which means that they were exposed to 59 lessons or fewer). We found from self-reports of teachers that at 80 percent, Western Equatoria had the second highest proportion of learners in the low IRI attendance category, a factor that may laregely explains low performance in the state. Low participation rates in any instructional initiative undermine its effectiveness. At the very least, there is a serious implementation threat for SSIRI if teachers report that they are not using the programs. Field monitors should systematically investigate why the programs are not being used. A good place to start would be to ensure that there is good SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 79 reception in all areas, that broadcast time schedule is published in all states, and that Outreach Coordinators assist teachers with timetabling issues where those arise. In addition, the SSIRI M&E section should insure that an agreed number of field monitoring reports are received from monitors and verified every month 4.2 Recommendations The discussion above posted a number of suggestions for improvement; some of these are recommended for follow-up action by SSIRI, while those with implications for policy-making can be investigated further and followed up with MoEST. 1. There is need to set performance standards or targets for each grade level. This will be useful in providing guidance on what the critical learning behaviors and outcomes for each grade level ought to be, as well as reporting progress based on curriculum standards (and not mean scores). Performance standards are also useful for benchmarking the Southern Sudan education system against global standards in the education sector, as well as tracking performance in the different regions of the country. 2. MoEST needs to clarify what ‘local language literacy’ is in the context of Southern Sudan, why it is important to learn a local language if it is not synonymous with teaching children in their ‘mother tongue.’ SSIRI could then strengthen its strategies for teaching local language literacy and the MoEST could further develop a comprehensive area/local language policy and tools for its implementation 3. Field monitors should systematically investigate why the programs are not being used. A good place to start would be to ensure that there is good reception in all areas, that broadcast times are published in all states, and that Outreach Coordinators assist teachers with timetabling issues where those arise. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 80 Annex 4: Summary Of SSIRI School Observations With Radio Reception Notes Learning Village School Location Class/es observed Teacher prepared “Lifeline” Radio signal / Notes 1 Libya Basic Juba P2 Aided Used .mp3 – no Miraya today, children largely uninvolved 2 St. Theresa Torit P1 Aided Used .mp3 – no Miraya today, class of 180 unsuccessful 3 Yabongo Girls Basic Yambio P1, P2 Yes Miraya FM – barely adequate, LV observed partially successful 4 Mangbomob Basic School Yambio None n/a Late unannounced visit 5 Maridi 1 (Charles only) Maridi None n/a Late unannounced visit 6 Hai Salam Basic School Wau P1 Yes Used Miraya – inadequate signal, LV observed not at all successful 7 Police Boys Basic School Malakal P2, P3 Yes Miraya FM - inadequate signal, LV observed not successful 8 Police Girls Basic School (only Charles visited) Malakal None n/a Late unannounced visit 9 St. Theresa Basic Juba None No L.V. timed unannounced visit, LV not being done 10 Bukuk “A” 1 Basic Juba None No L.V. timed unannounced visit, LV not being done 11 Juba 1 Girls Basic Juba None No L.V. timed unannounced visit, LV not being done Terbia Site Visited 1 Juba 1 Girls Basic Juba Terbia B Yes Used .mp3, batteries failed 2 Nabaguu Primary School Yambio Terbia B Yes Used .mp3, no radio today, lesson was successful 3 Police Boys Basic Malakal Terbia B No Failed, SW signals inadequate 4 St. Juveline School Juba Terbia B Yes Bakhita FM adequate and lesson was successful With each school possibly having 3 teachers doing Learning Village there were a total of 33 possible observations we could have made had all been active during our visits. As it happened, we saw all or part of seven Learning Village classes. 7/33 = 21%. Another way of looking at this could be at the level of the school. We saw only two schools that appeared to possibly be ready to do the program on their own via radio whether or not we would have been there (Hai Salam and Police Boys). Two schools out of 11 possible - 2/11 = 18%. Because there was no FM radio signal the week we visited Libya and St Theresa (Torit) school one might wish to eliminate those from such an analysis resulting in two schools out of 9 - 2/9 = 22%. In any event, Learning Village utilization during this period was somewhere around 20%. TERBIA usage by radio during this period was on the order of 25%. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 81 Annex 5: SSIRI Deliverables and Proposed Activities in Contract or Modification and Status of Each Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status 623-A-00- 04-00054-00 end: 6/15/08 Identify, select and train approximately 18 southern Sudanese to write, produce and implement IRI programs, and set up user systems in counties and communities. Writers trained adequately, implementation (outreach and training) staff trained. Total Sudanese staff including scriptwriters in Nairobi and staff in Sudan is 51. Develop and pilot test Grade 1 radio programs in order to design and test a model of program development with the SOE and SBEP partners’ curriculum and teacher training staff, and test instructional strategies in selected communities in southern Sudan and with different kinds of users (children in and out of school, youth and adults); Completed 2005 Set up the necessary temporary infrastructure in Nairobi while creating the permanent infrastructure in Maridi (housing and utilities, production and communication systems, studios, writing space) to allow program development to proceed there in 2005; Nairobi production facility set up but no physical progress yet made in Maridi. EDC reports they completed design for studio and house renovation in Maridi in 2005 but did not begin construction because USAID asked them to focus on “3 Areas”. EDC expected that the above process would take about one year to eighteen months, and the end of which they expected to have: ~1/1/2006 developed 150 30-minute Grade One programs in Nairobi and Maridi; 100 Gr 1 programs completed. Lessons reduced to 100 per grade in consultation w/ USAID so series could be used in ALP classes developed by SBEP. ~1/1/2006 identified an appropriate channel to transmit them nationally; SW initially identified and used, FM added later ~1/1/2006 identified and provided initial training to a cadre of six writers, two radio producers, and four presenters in Nairobi and Maridi; Nairobi staff identified, training provided, two CDC-Maridi staff trained in scriptwriting. Currently EDC has 10 scriptwriters, twp producers, two digital editors, and about six actors. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 82 Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status ~1/1/2006 built or renovated facilities so that they are ready for writing and production operations to take off in Maridi in 2005; Writing operations possibly ready, production not. Equip. given: generator, VSAT, computers, video. Architectural drawings for studio being reviewed. ~1/1/2006 established a limited presence in selected communities (Three Areas?); Assuming this refers to the Three Areas, EDC has established a limited presence in each of the three. Early 2005 established a good working relationship with the SOE, county commissioners, SBEP partners, and some selected communities so that implementation in schools and communities can move ahead in 2005; Done by 5/08. Reports from USAID and MoE staff that EDC-MoE relationships are good and improved in period from April 2007 to present. conducted a feasibility study of other technologies and moved towards implementing them. Discussions and tables of various devices appear in quarterly reports, subsequently VSAT, computers, wind-up radios, some .mp3’s, video were deployed After 42 months (~June 22, 2004 – December 21, 2007), EDC expected to have: 1/1/2008 developed 150 programs for each of Grades 2 and 3, with accompanying guides for the teachers; 100 Gr 2 programs with Teach Guides distributed 100 Gr 3 programs with Teach Guides distributed 1/1/2008 set up user groups (i.e. schools and out-of-school settings) that have wind-up radios, blackboards, supplies of consumable materials such as chalk, guides and manuals for teachers, and opportunities for training and assistance; EDC reports 845 classrooms using Learn. Village as of 1/1/08, 959 teachers reported trained in 2007 4th Qtr. Report, (885 listed in training log & later report from EDC); 972 radios distributed & traced, 10% of the 1357 from NDI (135) cannot be traced 1/1/2008 broadcast three years of programs, that will have enabled one cohort of learners to have covered three grades, three cohorts to have begun their education; Three years SSIRI Learning Village (L.V.) b’casts on air 1/1/2008 laid the foundation for more grades of IRI to be developed; Done. Grade 4 L.V. in progress (as of the 15th of May 2008, 15 Grade 4 programs recorded) 1/1/2008 provided daily radio-delivered teacher training to perhaps 500 teachers, perhaps many more, to complement (at minimal cost to the SOE) the face-to-face training; Two PS101 programs produced to complement 10 SBEP programs; participant’s workbook and training materials developed. Training not begun, hiring T/A to extend audio series. 1/1/2008 provided information about learning outcomes among a sample of users; Grade 1 learning gains study completed. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 83 Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status 1/1/2008 created a cadre of southern Sudanese instructional materials writers who can contribute to meeting other educational priorities, and a fully equipped center in Maridi in which to develop them; Cadre of Sudanese materials writers created in Nairobi, Maridi production center not created, though drawings for the studio have been prepared. 1/1/2008 installed a network of perhaps eight VSATs and planned perhaps as many solar￾powered FM radio stations in southern Sudan; VSAT being supported at Arapi, reactivated at Maridi, and new computer centers with VSAT installed at Malakal TTL/Ministry and Juba Day Secondary School. Others planned for Panliet, Aramweer, and Panyagor TTIs. 1/1/2008 responded to the changing political and economic environment in southern Sudan with suggestions for further technology support to education and development, or the expansion of currently operating systems. MoEST demand has been key in EDC decision to expand services for both L.V. & Terbia and to provide “low-cost” digital players, plus more sophisticated technologies for TTIs. Has conducted R&D to determine most appropriate technologies, which is still ongoing. Subsequently EDC sought support from the Displaced Children and Orphans Fund (DCOF) to implement and expand access to basic education (Grades 1-4) through interactive radio instruction in the Three Areas. With DCOF support through Modification 04 ( to No. 623-A-00-04-00054- 00) further targets were outlined for the period from mid-2006 to mid-2008, now extended to June 21, 2009. Mod. 04 Nearly 30,000 students in the Three Areas provided with quality instruction (SSIRI) Evaluator travel to 3 Areas not possible, EDC reports show 490 Terbia B, 1409 Terbia A listeners in groups, 5334 Learning Village pupils in 3 Areas 976 SSIRI teachers in the Three Areas trained, monitored, supervised, and supported in child-centered pedagogical practices and the use of SSIRI programs A total of 109 Learning Village teachers and Terbia Facilitators trained in the Three Areas (Qtr. Report 15) 12,000 educators (for children, youth, and adults) in the Three Areas receiving radios for use with their classes NDI distributed 32,000 radios throughout the Three Areas. It is the understanding of EDC that these radios include those that would have been handed over to EDC for distribution. Locations and types of radios or recipients not clarified. 12,000 educators in the Three Areas trained on the care and use of radios. NDI may have trained on behalf of SSIRI, participants and locations unknown, SSIRI assisted NDI in at least one training (EDC quarterly report) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 84 Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status 200 interactive radio programs (100 each for Primary Grades 3 and 4) produced 100 for Gr 3 completed, 15 for Grade 4 completed as of 5/15/08. Series to be completed in late 2008. Over three years EDC would develop and implement Terbia programs as follows: 06/07 06/08 Provide Terbia A advanced English and literacy instruction to Southern Sudanese with moderate proficiency in English by producing approximately 120 lessons, 20 weeks for Year 1; and 20 weeks for Year 2 at 3 times per week. 60 lessons of 120 produced. ( Note: Quality of provision by SW broadcast uncertain - both for Terbia and LV. EDC using FM stations, both national Miraya network and local stations.) 12/06-05/07 Broadcasting Year 1 Terbia A programs for 20 weeks 60 lessons broadcast for 20 weeks beginning Feb 12, ‘07 (EDC report) 07/07-11/07 Broadcasting Year 2 Terbia A programs for 20 weeks, with repeat from Jan – May 2008. 60 lessons rebroadcast for 20 weeks beginning August ‘07 (EDC report) 07/08-05/09 Broadcasting Year 3 Terbia B programs for 20 weeks, with repeat from Jan – May 2009. Future 06/07 06/08 Provide Terbia B basic English, literacy and numeracy instruction to Southern Sudanese with little or no knowledge of these subjects by producing approximately 120 lessons, 20 weeks for Year 1 and 20 weeks for Year 2 at 3 times per week 60 lessons of 120 produced (Terbia B1). (Program discs provided to evaluators.) 15 of 60 Terbia B2 lessons readied in Nairobi for broadcasting - scheduled to begin June ’08. 12/06 – 05/07 Broadcasting Year 1 Terbia B programs for 20 weeks 60 B1 lessons broadcast for 20 weeks beginning Oct. ‘07 (EDC report) 07/07- 11/07 Broadcasting Year 2 Terbia B programs for 20 weeks with repeat from Jan – May 2008. 60 B1 lessons rebroadcast from 5/07 (EDC report) 07/08- 05/09 Broadcasting Year 3 Terbia B programs for 20 weeks with repeat from Jan – May 2009. Future 06/09 Develop the capacity of the MoEST to manage radio-based education programs at both the national and state/county levels. Some evidence of SSIRI management capacity at the state and county level, but not yet sufficient. 06/09 Develop an understanding of civic education concepts including governance, peace building and development. Extensive content in Terbia A on civic education topics. No evidence on listenership or impact. EDC’s proposal estimated targets for numbers of recipients/participants to be affected by the Terbia Programs, and they also provided estimates of the quantities of materials to be delivered. Year 3 1. Terbia A – 225,000 occasional listeners Unknown (2006-2007) 2. Terbia A – 22,500 regular listeners – assessed by annual survey Unknown SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 85 Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status 3. Terbia B – 15,300 in listening groups: 5400 S. Sudan, 7200 in 3 Areas, 2700 in 3 urban areas Key Indicator - unknown for this time period for both Terbia A&B. See next year cumulative figure. Terbia B - Facilitators Trained: 180 in S. Sudan, 240 in 3 Areas, 90 in 3 urban areas = 510 Unknown for this time period Terbia B - Teachers Guides Distributed = 510 as above Unknown Terbia B - Student Exercise Bks and Pencils: 5400 in S. Sudan, 7200 in 3 Areas, 2700 in 3 urban areas None Year 4 1. Terbia A – 225,000 occasional listeners Unknown, EDC to conduct listener survey in 2008 (2007-2008) 2. Terbia A – 22,500 regular listeners assessed by annual survey 8186 total Listeners in Groups - 6777 Southern Sudan, 1409 in Three Areas–(Q14 Report). Survey to be conducted - mid-2008 to estimate listenership 3. Terbia B – 18,000 in listening groups: 5400 S. Sudan, 9900 in 3 Areas, 2700 in 3 urban areas Key Indicator - 1288 Listeners in Groups in Southern Sudan, 490 in the Three Areas (Q14 Rpt) Terbia B - Facilitators Trained: 180 in S. Sudan, 330 in 3 Areas, 90 in 3 urban areas = 600 114 facilitators trained for Terbia B in S. Sudan and 31 in the Three Areas. Terbia B - Teachers Guide = 600 as above Unknown Terbia B - Exercise Bks & Pencils: 5400 in S. Sudan, 9900 -3 Areas, 2700-3 urban areas None Year 5 1. Terbia A – 225,000 occasional listeners Future (2008-2009) 2. Terbia A – 22,500 regular listeners assessed by annual survey Future 3. Terbia B – 20,700 in listening groups: 5400 S. Sudan, 12,600 in 3 Areas, 2700 in 3 urban areas Key Indicator - Future; and where Terbia A groups exist these would be counted Terbia B - Facilitators Trained: 180 in S. Sudan, 420 in 3 Areas, 90 in 3 urban areas = 690 Future Terbia B - Teachers Guide = 690 as above Future Terbia B - Exercise Bks & Pencils: 5400 in S. Sudan, 12,600-3 Areas, 2700-3 urban areas Future PS 101 Teacher Professional Studies – one year effort 160 In-Service SSIRI teachers trained for 12 days Not begun 160 In-service non-SSIRI teachers trained for 12 days Not begun 52 Pre-Service Teachers trained for 12 days Not begun 8 TTI Tutors trained for 2 days Not begun SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 86 Contract # or Due Date Deliverable / Proposed Activity in Contract or Modification Status 10 TTI tutors or County Education Officers trained for 2 days Not begun 8 SSIRI Outreach Coordinators trained for 2 days Not begun 5 NGO Tutors trained for 2 days Not begun Exploring New Technologies and More Extensive Use of Radio (via Modification No. 9) Weekly 50-minute evening program on education with Miraya and MoEST MoEST committee established and plans developed for initiating program 24 radio modules (programs) designed with MoEST to support in-service teacher training Consultant hired to begin June 10 mid-2008 Conduct ICT Summit with the MoEST Events organizer being selected Conduct at least two seminars for senior MoEST officials on educ. technologies Not begun Conduct study tours to two countries that have relevant experience in use of learning tech. Not begun Develop plan for sustainability and to ensure full integration of SSIRI at all MoEST levels SSIRI is fully integrate into the Ministry at GoSS and state levels. Feb 08 workshop with most states and 3 Areas on integrating SSIRI into states/areas Introduce VSATs at Aramweer TTI, another TTI, one CEC VSAT installed at Malakal SMoE and TTI Develop MP3 system to support teacher training in at least 1 TTI and 3 learning centers MP3 systems identified and being trialed Develop at least 6 illustrative video segments for teacher training 1 staff at Arapi and 3 at Maridi produce 1st videos Assess a variety of options for alternative energy sources to power SSIRI learning devices Ongoing Develop, pilot test, and implement on or more MP3-based option for recharging batteries 3 solar panels and 1 cranking charger being piloted SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 87 Annex 6: Shortwave Radio Reception Test Results Date Site Frequency Time Program Weather Ranger (R) or Lifeline (L) Radio 5/23 Police Boys Basic - Malakal 15485 kHz 15390 kHz 4:00 Terbia B1 Terbia B2 Clear Only one frequency was found (R) Quality – quiet, static, not useful (R) 5/23 Police Boys Basic - Malakal 12070 4:00 Terbia A Clear Stronger than Terbia B, static (R) 5/26 EDC - Juba 11905 kHz 9:30 P2-LV Overcast Lots of static, hard to tune (L) 5/26 New York Hotel - Juba 12070 kHz 4:00 Terbia A Clear Can’t find station (L) 5/26 New York Hotel - Juba 15390 kHz 4:00 Terbia B2 Clear Loud with static – usable (L) 5/26 New York Hotel - Juba 15485 kHz 4:00 Terbia B1 Clear Softer with static – not usable (L) 5/27 New York Hotel - Juba 15215kHz 9:00 P1-LV Clear Audible signal, static, too soft for class (L) 5/27 New York Hotel - Juba ~14200 kHz 9:00 P1-LV Clear “Phantom frequency”?, lower static, stronger but still too low for a class (L) 5/27 New York Hotel - Juba 11905 kHz 9:30 P2-LV Clear Far too soft, static (L) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 88 Annex 7: EDC/SSIRI Responses to Mid-Term Evaluation 29 May 2008 To: Lucy Kithome, USAID Inez Andrews, USAID From: Tom Tilson, Chief of Party Re: Responses to Mid-term Evaluation I am writing to present some initial ideas in response to some of the findings of the SSIRI Min-Term Evaluation Team. These comments are based on observations of classes and discussions during the last two weeks, plus an informal debriefing by the Team on May 22 in Wau. The Evaluation Team noted the challenging environment of Southern Sudan for implementing a program including the following: • The Ministry of Education, Science and Technology and the state level structure are only three years old • Weak human capacity at all levels • Poorly trained teachers and a majority of teachers with weak English skills • Only recently have teachers started to be paid and, even then, payments are often late and are below the correct amount, all of which impacts on motivation. • Difficult infrastructure making travel by road and communications difficult, plus a long rainy season that isolates much of the country • Very large classes in urban schools In addition, even many of EDC staff do not have the level of experience required for the responsibilities that they hold, and there has not been insufficient training of the staff. Some of the positive comments by the Team include the following: • Strong support for SSIRI from the Ministry of Education, Science and Technology, Department of Alternative Education Systems • Good support developed at the state and county levels in the states • Most of the programs completed • Field staff are mobilized Education Development Center, Inc. Southern Sudan Interactive Radio Instruction (SSIRI) Project UMEME Plaza (Off Ngong Road on Old Naivasha Road) Œ P. O. Box 25010 00603 Œ Nairobi Œ Kenya Œ Telephone: 3870906/3873088/3872269 Fax: 254-20-3876520 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 89 • Training is ongoing Although there are some notable achievements to date, there are a number of concerns and challenges that need to be addressed, most importantly, the need for well functioning IRI lessons in classrooms – both for the Learning Village and Terbia. A key finding of the Evaluation Team is that classes are not functioning well – technology problems including reception on Miraya FM, battery issues with digital devices, and the ability of radios and other technologies to serve large classes; plus inadequately trained teachers. While these are symptoms, the underlying causes relate to insufficiently trained EDC staff; inadequate support, monitoring and accountability of outreach staff; and insufficient monitoring of classes by senior management. All of this is exacerbated by the post-conflict environment that is Southern Sudan. The response to the findings requires a restructuring of parts of the project. The following ideas form a comprehensive approach to rectifying the problems identified: Issue Proposed Action Focus on quality in a few schools and do not undertake the level of scale up anticipated for 2008 – Demonstrate that the system works There is need to focus on a few schools to demonstrate successful implementation of the IRI system • In 2008 EDC will not work in all states and counties. EDC will develop criteria for the selection of the counties. • Within each county, EDC staff will focus on a small number of schools (from 1-5) and establish a “model school” in each location • Only scale up modestly or just hold to current levels in seven states; plus the Three Areas (only in Agok in Abyei) • Postpone implementation in Lakes, Warrap and Unity Provide appropriate staffing, training, monitoring and accountability There is need for additional senior level staff • Hire Senior Training Advisor (contract expected to be signed by early June) who will have responsibility for: o Developing and implementing capacity building program for EDC field staff o Developing/refining training materials, monitoring and reporting forms, etc. (in collaboration with M&E Specialist) o Designing a supervision support program for IRI teachers o Setting expectations for monitoring, teacher support, and reporting • Hire Deputy Chief of Party for Operations. EDC currently has a shortlist of candidates. This person will have overall responsibility for administration, procurement, finance, logistics, and HR both in Nairobi and in Southern Sudan. Not only will this person help to ensure a well functioning and responsive administrative and financial system, but by relieving the Chief of Party from significant work in these areas, the COP will be free to spend more time in Southern Sudan on the technical implementation of the project. There is need to provide a sufficient number of outreach staff • The responsibilities for training, monitoring, and supporting Learning Village teachers and Terbia facilitators require more intensive action. Thus, the staff must be augmented and all staff trained. • At most counties where EDC is present there is need to increase Outreach Coordinators from one to three and, in a few cases, even four. New staff will be hired and/or some of the currently deployed OCs will SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 90 Issue Proposed Action be reassigned to consolidate efforts in fewer counties. Hopefully, in some locations, a strong county officer could replace one of the OCs. o One OC to focus on just a few schools to ensure quality implementation; classes are to be visited at least twice a week and teachers will receive ongoing mentoring. Will develop at least one “model school” per site. o One OC to monitor and support other participating schools in the county through occasional visits to most of the schools, collecting good data on implementation, and organizing periodic refresher courses for the teachers. o One OC to support Terbia programs o And, where PS101 is implemented, possibly one additional OC to support this program • In order to continue efforts to fully integrate the SSIRI programs within the SMoE, all staff will coordinate with and seek to fully involve county and payam staff in activities. EDC plans to provide bicycles (payams) and motorbikes (counties) to facilitate school visits. An important goal is to develop ownership of the program at all levels – states, counties, payam sand schools. Capacity building for state, county, and payam officials is a priority. • The State level M&E Assistants will have an important responsibility to help ensure that the monitoring and training activities are ongoing as planned and that the required data are collected, analyzed, and forwarded to senior level staff. • The Outreach Advisor for each state will have overall responsibility of implementing the new structure with supervision from the Senior OA and additional support from the Senior Training Advisor and Deputy Chief of Party for Implementation. They will be held accountable for the activities and reporting of the OCs. The OA will also closely work with state officials, especially the Director for Alternative Education Systems, on developing state level plans and implementation. • Once it is demonstrated that the IRI program is working work well in a particular location and that the county and payam officials are capable of assuming increased responsibilities for supporting SSIRI, the EDC staff will be reduced accordingly and/or redeployed for expanding SSIRI activities in other locations. • There is need to provide some motivation and recognition of good teachers: o Teachers who gain strong competence in teaching IRI lessons could be selected as part-time mentors for teachers in other classes and schools. o SMoE officials could seek ways to motivate teachers, even through occasional visits to schools to recognize the good work of teachers. o EDC could initiate a “teacher of the month” program. • EDC is hiring a Recruitment Specialist to be based in Juba to help ensure ongoing recruitment of staff for new positions and for anticipated replacement of some staff; the person will also provide initial orientation and will support on the job training. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 91 Issue Proposed Action EDC staff – capacity building and accountability There is need for more ongoing training, support, monitoring, and accountability of field staff at all levels • With the recent hiring of approximately 25 new staff, EDC will hold a training workshop in July for all staff. • Under the direction of the new Senior Training Advisor, new training materials will be developed that will improve the training capabilities of our field staff. The new materials and approaches will include better use of media. • The outreach staff at each level will be given clear guidance on their roles and responsibilities including details on training, monitoring, classroom observation, troubleshooting technology problems, coaching teachers, collecting data, and biweekly reporting. The biweekly reports will include important summary information for the Biweekly Report to USAID as well as detailed information on activities and findings in the schools. • The expectations regarding visits, support for teachers, and reporting will be emphasized as part of their contracts, and continued employment will be dependent on their progress. • EDC will provide ongoing support and training to its field staff including quarterly meetings at the state level. All state and county level workshops will include SMoE personnel. In addition, EDC is ready to offer a contract to two IT Specialists who will move from one location to another in order to provide training and support to EDC staff on computer use, how to work with EDC computer-based forms, and use of email. • In addition to the OCs, similar high expectations will be set for the state level OAs, and the Senior Outreach Advisors. • For OCs who do particularly well, we might create a Senior OC position with greater responsibilities and remuneration. Senior level monitoring There is need for senior level staff to provide ongoing monitoring down to the school level • Senior level staff including the Chief of Party, Deputy Chief of Party for Implementation, Senior Training Advisor, and the M&E Specialist will regularly visit field sites including observation of Learning Village and Terbia classes. At least one of these people will conduct such a visit every two weeks. • In September 2008, an EDC team including the Regional Director will visit several sites through a charter flight. USAID will be part of the team. • Such monitoring visits will be conducted quarterly. English ability of teachers A significant challenge is that a majority of teachers do not have sufficient English skills to use effectively the SSIRI programs • EDC has expanded the length of initial training workshops from 2 days to 4-5 days so that teachers can have more practice using the IRI programs, especially the role of translating instructions from English into the local language • EDC will test out the use of teacher's guides that include instructions in Arabic as well as English • EDC is working with the scriptwriters to reduce the information burden on teachers, especially instructions that must be translated – reduce the SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 92 Issue Proposed Action content, break up instructions into small pieces, simplify the language, etc. • EDC plans to create a new English for Arabic Pattern Teachers course based on the Terbia programs – providing instructions in the Beginner series in Arabic rather than English and, thus, enabling individual listening by teachers, developing a guide/ workbook for teachers, and facilitating the formation of school-based listening groups for teachers, perhaps meeting at least once a week. • In the future, EDC will consider developing a new audio series for teachers, perhaps weekly, and both in Arabic and English, that would provide specific information on upcoming lessons, such as content, English usage and vocabulary, and suggested activities. • It might be worth considering for the current school year to give more attention to locations in Southern Sudan that have a stronger English base, such as the Equatorias. • Finally, EDC will seek to strengthen its ties with Windle Trust, starting in Malakal where they are teaching English to ten groups of 30 teachers each. Not only could Terbia be incorporated into their program, but this would give us an opportunity to introduce and train all of the 300 teachers. Maybe we could provide an incentive for a special Saturday morning program for the teachers on IRI. There is the potential of working with Windle Trust in all ten of their locations. • EDC will also incorporate a full training program on IRI for teachers as part of future Fast Track training programs. Radios and digital players The Freeplay Lifeline radios are not working well – volume too soft for large classes, especially on Miraya FM • There seems to be a problem with the FM tuner causing the Miraya FM signal to be broadcast from more than one frequency. EDC will inform teachers of this problem and request that they try tuning to Miraya at an alternative frequency – about 95. EDC has already contact Miraya Radio about this problem. • A radio for every 40-50 students; thus, a class with 80 teachers could have a second Lifeline radio in the back of the class. • EDC has just received a new supply of Lifeline radios, the first in about six months, and has already begun distribution. • EDC has revised its procedures to emphasize that the radios are issued to the schools, not individual teachers, although teachers will continue to be caretakers. • EDC will try to reduce reliance on shortwave frequencies through increased use of local FM stations and additional sets of digital players. EDC will also request that Merlin try to provide greater separation of frequencies between Terbia for Beginners -1 and Terbia for Beginners -2. • The shortwave broadcasts for the Learning Village are too late in the morning (even at 9AM) for optimal signal strength. EDC could explore the possibility of some schools beginning at 8AM in order to receive a better shortwave signal. The digital players are not working well • Although the MP3 player with small speakers provides excellent quality sound for up to a medium size class, at times the batteries are not holding an adequate charge. EDC will continue its assessment of the current and new devices. In the meantime, EDC will issue larger solar panels (5 watt SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 93 Issue Proposed Action instead of 1.5 watt) to help ensure batteries will receive a sufficient charge. • EDC will soon deploy 100 of the JWIN radio/CD players with rechargeable C cells and 5 watt solar panels; EDC will closely monitor these systems in the field. • EDC will also examine successful, and more expensive, use of digital players in other EDC projects in neighboring counties for possible application in Southern Sudan, especially for Terbia classes. • EDC will videotape successful IRI classes for training and dissemination activities Improvement of the lessons There is need to review and revise the initial series of lessons • The pace of some of the lessons seems too fast and some pauses too short. In the short-term, EDC will review and extend the pauses as appropriate. • In the longer term, the lessons should be reviewed in terms of sequencing, child centeredness, interactivity, and opportunities to give teachers an occasional longer pause to work with children on specific activities. • Also, in the longer term, EDC will consult with others regarding refocusing the Learning Village lessons to emphasize English and, perhaps, mathematics, but not local language literacy. Based on experience to date including the summative evaluation of grade 1, plus the most important needs in the classroom, it may be appropriate to focus on English language instruction and mathematics. • EDC will continue to ensure that there are no major errors in the lessons. For example, the Evaluation Team viewed Terbia for Beginners, Part 1, Lesson 12, and some errors were noted. EDC subsequently reviewed in detail this lesson with all scriptwriters and producers. There was a lively discussion on ways in which the lesson could be improved, but it also turned out that the obvious errors noted in the field were NOT present. The field staff had a pre-broadcast version of the lesson and the errors noted had already been corrected in the version that was broadcast. EDC will update the audio files of the field staff in June. The scriptwriters and producers agreed to hold a similar extensive review of lessons with all staff on a regular basis. • EDC will hire a few additional scriptwriters so that we always have two in the field conducting formative evaluation. Implementation of Professional Studies for Teachers – PS 101 There is need to implement PS 101 • EDC has prepared all of the audio lessons and training materials for PS 101. It will now proceed with implementation activities. • This series is designed to be implemented through the County Education Centers as part of the MoEST in-service teacher education program. However, such centers and the implementation of the in-service program are only beginning, probably no more than one or two counties. EDC will work with the MoEST on the identification of at least one CEC for implementation of PS 101. • EDC will also include at this center, or perhaps through another county office, IRI teachers. This trial will not only be a benefit to the teachers, but will also provide important feedback on the effectiveness of the SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 94 Issue Proposed Action program, and lessons learned for improving the lessons as well as for the development of new series. • EDC will initiate an implementation strategy for PS 101 by incorporating it in the refresher courses planned for IRI teachers EDC will hire a senior staff member to direct this activity (the previous person is now the Deputy Chief of Party) plus county level staff. • EDC will explore the possibility of implementing the program through collaboration with Windle Trust in Malakal. Technologies for strengthening teacher education at TTIs There is need to focus on technology interventions in just two TTIs • Give highest priority to effective use of the Internet for: o Developing communications among staff from different institutions and with colleagues elsewhere o Identifying websites with materials or courses for strengthening capacity of staff and for improving instructional materials o Developing a portal for TTI staff • Improve video production. Based on just one workshop, the staff at both TTIs have already developed the basic skills of video production. They now need guidance on how video can most appropriately be used to strengthen their courses. • EDC is providing a full-time IT person at each institution to support the technologies and to provide training to staff (position offered for Malakal). • Beginning in June, EDC will provide a senior international consultant for two months to strengthen the Internet and video capacity in both Maridi and Arapi. • Establish clear TOR and a MoU with the SMoE in Malakal on the use of the new computer center there. Since the visit of the Team to Malakal, the Minister has confirmed the following re. the TTI: 1. recruitment of tutors to begin immediately; 2. the enrollment process to begin; and 3. a contractor has been secured and a down payment will be provided for renovation to start immediately. Also, the Ministry will provide furniture, security and administration for the current location of the computer center. Priority for use will be given to IRI teachers, the few TTI staff who have been identified - Principal and some tutors, and AES staff. Finally, the Minister will designate a person from the Ministry to work with EDC to help ensure the success of this effort. • Postpone implementation of computer centers in other locations until a good model is developed at Maridi and Arapi • Begin initial planning for developing programs for print-based and local materials that could be incorporated into these resources centers SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 95 Annex 8: Department Of Alternative Education Systems Director Alternative Education Systems Coordinator SPLA & Other Security Organs Education Deputy Director Alternative Education S ystems Senior Inspector Intensive English Language Course Senior Inspector Basic Adult Literacy Programme Senior Inspector Accelerated Learning programme Senior Inspector Community Girls Primary schools Senior Inspector Interactive Radio Instruction Senior Inspector Pastoralist Education Camera Officer Sound Systems Officer Head Technician Secretary Alternative Education Systems Senior Inspector Agro-Forestry Education SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 96 Annex 9: Map of Miraya-FM Radio Stations Projected Broadcast Coverage SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 97 Annex 10: People Interviewed or Met During SSIRI Evaluation (and partial transcription of handwritten meeting notes) Date Person Notes 1 8-May Lucy Kasyoka Kithome Management Specialist USAID 254-20-862-2000 – Nairobi 0477-196984 LKithome@usaid.gov Met at first USAID meeting 2 8-May Thomas Tilson COP – SSIRI 254-20-3873088/ 3872860 ttilson@edc.org ttilson@yahoo.com 3 8-May Amuda James Scopas Outreach Coordinator, Kajo Keji Thuraya 88216 6790 0951 Gemtel – 256 (0) 477 193225 Met on first visit to EDC office 4 8-May Emanuel Wala (USAID) 0477-170447 Met at first USAID meeting 5 8-May Ruben Mayen Senior Inspector for Community Girls Schools MoEST-AES Met at first USAID meeting, traveled together to Torit 6 9-May Patrick Fleuret Director, USAID Mission to Sudan USAID program in Sudan is the 2nd largest , S Sudan is the largest in Africa, well run and results oriented, other donors are Jaica and EU, WB SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 98 7 10-May Cecilia Sierra Salcido, CMS Bakhita Radio Directress Sudan Catholic Radio Network Juba Archdiocese 0477 127081 bakhitaradio@yahoo.co.uk 8 10-May Jose Da Silva Vieira Radio Bakhita (news director?) Met with Sister Cecilia 9 10-May Helene Carlson USAID Met at USAID 10 10-May Mary Hopps Economic Growth Formerly Human Capacity - USAID Could not keep Saturday May 10 appointment 11 10-May Martin ?? USAID Health officer 12 12-May Maree Melican 256 (0) 477116990 mmelican@edc-ssiri.org SSIRI – EDC Deputy Chief of Party 13 12-May Huub Gales Station Manager Miraya-FM 256(0)477125652 912178366 galesh@un.org 14 12-May Jino Meri Program Officer / Monitoring and Evaluation AED-TAP Program jmeri@aed.org 0477 109547 Stopped in to see COP and met briefly at MoEST SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 99 15 12-May Dawne M. Deppe COP - GEE Program Winrock International 256 (0) 477 155-514 249 129 357 101 ddeppe@winrock.org 16 12-May John Dabi MoEST - Consultant Brother of Hakim in planning dept. / Works with Shadrack Shawl (sp.?) 17 13-May Ferid Barmabu Ladu Head Teacher Libya Basic School 705 students, 599 are boys, English teacher does the radio program 18 13-May Rev. John Wani Baya Assistant Head Libya Basic School 19 13-May Teresa Simon Juma Grade 2 Teacher Libya Basic School Bakhita was fuzzy on 91 FM this day 20 13-May John Onek 88216 433 6175 onekakila@yahoo.com SSIRI training for teachers 21 13-May Arsenio Beda Terbia B Facilitator Juba 1 Girls Basic School (and adult education school) Used .mp3 player - noted battery runs down in both player and speaker 22 13-May John Ohide Camillo Grade 2 Teacher St Teresa School – Torit Observed his open air class 23 13-May Ohisa Reuban SSIRI Outreach Advisor – Torit 88216 67900009 256 477 142941 Ohisa_reuban@yahoo.com Met at training and at observation 24 13-May George Omoli County Field Education Officer for AES Torit Inspector for Intensive English, Accelerated Learning Program and Community Girls Schools SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 100 25 13-May Mr. Brown Inspector for SSIRI SMoE - Torit Not available to meet 26 13-May Alex Locor Nartisio Director General Ministry of Information – Torit 0477-171056 Met in his office and toured radio station together 27 13-May Onek Emmanuel Rozers Teacher, AIC Primary School Torit Interviewed at the training site in green shirt. He is coordinated by Omoli 28 13-May Barnaba Lokutai Profiro County Education Director - Torit 29 15-May Francis O. Abalu Director Directorate of Radio and Broadcasting – Torit 30 15-May H.E. Francis Ben Ataba Minister of Education Science and Technology Eastern Equatoria State – Torit 88216 43093673 francisataba@yahoo.com 31 16-May Patrick Were Tutor (English) Arapi TTI 32 16-May Grace Asiendzo Tutor (Agriculture) Arapi TTI Made the demonstration video we viewed 33 16-May Sylvester Alao Accounts and Finance Arapi TTI 34 16-May James Betcu Logistics, Arapi TTI SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 101 35 16-May Robert Andruga Tutor (Curriculum Studies, science, sociology) Arapi TTI 36 19-May Odur Nelson Deputy Director of AES, MoEST 37 19-May H.E. Stephen Lokuron Lemi Minister of Education for Central Equatoria State 249-811-820538 256 477 103759 New in position since late last year. Not heard of SSIRI or knows very much about it. What is lacking in general is communication. It is hard to run schools this way. “We need help in transformation from Arabic patterned teachers” We have a few NGOs including Unicef and Windle Trust. Of 4000 teachers 50% are volunteers. There are impediments at the County level. With inservice training “we are proudly doing Windle. “The role of the County is only for inspectors, if called upon the State could do the training. Better to call upon us to do training.” 38 19-May Isidoro Asok Buse Acting Director General for some, Central Equatoria State 0121841210 Sat in on the meeting with the Minister 39 19-May Satimon El-Hag Deputy Director of AES – CES Center Mohammad Katir Interviewed in Arabic only Trainings for Terbia are set by EDC with collaboration of GoSS. His office is involved to ID teachers and locations. There is no EDC person attached to his office yet but they found a need for a SSIRI inspector and they appointed Samuel Lasio. He’s paid by the state. He will help train and monitor. Does Satimon train? No only coordinates and assists since they have no resources. Problems: 1) GoSS not paying the facilitators; 2) told at training they would get radios and they did not. They got some in 2006, some not functioning. ! for each venter was given. There are 31 centers in Juba that are interested to do Terbia but have no radios or supplies. He has no idea how many radios are in service or functioning. He knows the EDC OA for the area – Rejoice. She has asked for a room there and said she’ll come with a table and chair next week. Not all counties have OCs. He proposed some names but EDC sometimes does not want to hire the person. (Samuel). Satimon had an introduction to low cost technologies and training in Admin and Thuraya (10 days).He needs transport to do job. Reports from County inspectors go straight to EDC. He thinks the users of Terbia are very positive – sometimes they get neighbors to come because they fell left out. Users suggest it be on CD since they are often not there on time for bcast. Thuraya batteries are consumed soon. He was unclear who should get notices about problems with radios – his office or EDC? He has no knowledge of the Learning Village programs. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 102 40 19-May John Lujang Wani Dep. Director Teachers Education and Development Juba- Sudan 249-126012630 jlujangwani@yahoo.com He has been in his post for two months or so (Leu). Teacher education project is going to Save the Children at a cost of some 5 million of MDTF funds. Fasttrack program was for teachers who could go back to their classrooms and program was to act as a “gateway to inservice” In certain areas it worked OK. In others there was no job for the trainees to go back to. There will be changes in the new program with monitoring classroom visits. SSIRI will come in to reinforce the program. He got a call from Tom Tilson this morning about it. The program will target 6000 teachers working through the state and county. MoEST will work with Save managers and hire tutors “from here”. They will be used as instructors in the country centers and as Senior iInspectors at the county level. The training of Arabic patterned teachers is very important. This effort must come under the Directorate of Quality Promotion and Innovation. Only about 20% of the 19000 (or 21000) teachers are OK in English. 41 19-May Betty Poni Senior Inspector for SSIRI She has not yet been trained by EDC. She knows a little about it. She has observed 1 Terbia class. Asked about mp3 she had a little understanding of it. “A gadget for playing audio”. The State inspectors for SSIRI report back to EDC not to Betty so she is out of the loop. She does visit EDC offices. She took the workshop on mp3. She would agree on the need for joint trainings and she needs to get a grasp of the new computer applications (Excel), Transport is a problem. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 103 42 19-May Kuol Atem Bol Director of AES bkuolatem@yahoo.com atemkuol@moest.gov.sd 256 477 107065 249 121 249027 The Secretariat could not implement so EDC did it and we’re grateful. “Now is the time for the MoEST to take hold again. Maybe the technical part can still be with EDC”. He helped organized an orientation and induction workshop in Yei in Feb. 08 attended by GoSS, Directors of States, Senior Inspectors of SSIRI. This was a move to start to move SSIRI within his department. They are developing courses and materials. They’ve appointed inspectors for SSIRI and Senior Inspectors for the State. There should be one AES inspector for each county. English competence is a criteria for them. And 1 Supervisor for the Payams. Operations money is not there. Suffering from salaries, capital and operations funds is what we want from EDC. Transport for inspectors, air tickets for MoEST staff to visit states. Next year his department will be promoted to a full Directorate with 7 units (add Agro-Forestry and Pastoralist Education). In implementation the primary development of content should be at the National Curriculum Center in Maridi. Learning Village is really supporting the teachers and the children. It is building capacity. It is becoming so attractive to the children. The SW did not come clearly. The former COP (Lisa Kaplan) got lots of messages like – “Why don’t you increase the capacity so it can be heard?” . For 2008 we’ll pay theteachers. For next year we’ll pay the outreach coordinators. The budget is affecting everything. Is there a problem with LV being under Primary and Terbia being under AES? “No. Kokole and Atem will work as one.” Terbia B should do well in Arabic patterned schools. Terbia A should be helpful for Windle students – though the “Windle books are from elsewhere” (Kenya). MDTF paid $110K for development of adult literacy materials in Maridi. The printing to be done in Kenya. Problems: Going to scale. LV in AM pushes other subjects later. Expansion may be at the expense of other subjects. Terbia is Go! Should be on both mp3 and radio. FM is clearer than SW. Advise EDC to move from Nairobi to South Sudan. Then senior inspectors can take over. The best thing by EDC is that they have hired Sudanese. They should accelerate the process of handing back to the Sudanese. To sustain we need computers at the State level. This program can die if sustainability is not regarded. Give us the tools and training to gain capacity at the Center, the State, and the County. The first priority should be local language literacy and English. 43 19-May Elizabeth Leu COP AED – TAP Program 256 (0) 477 110966 eleu@aed.org Florence in the MOE works with Winrock. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 104 44 19-May Grace Akukwe, PhD Senior Technical Advisor Technical Assistance Program AED 882 16212 97501 256 477 152332 249 126 121108 gakukwe@aed.org 45 19-May Getahun Gebru Senior Operations Officer Human Development, AFTH3 World Bank Sub-Office Ministries Complex Kololo Road, Juba 256 477 105 667 Dama Line 5345-3016 ggebru@worldbank.org 19-May Shadrack Shawl Director of Planning, MoEST Recommended meeting that did not take place 46 May 20 Yambio Simon Gordino Bangafu Deputy Head Teacher Yabongo Primary School doing IRI demonstration for us 47 20-May Edward Kasran EDC-SSIRI Senior Outreach Advisor (for 5 states) 48 20-May Izikia Kelliopa, Teacher, Yambio Grade 2 - observed SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 105 49 20-May Nama Bullen Director General MOE – Western Equatoria State Yambio 256 4 7718 7948 Thuraya 8821655582418 namabullen@yahoo.com They are following a self-contained model. First build the capacity of the teachers. Then give them printed materials. They need English programs for teachers. She has found a group that was not formally organized using Terbia B. 50 20-May Jasama Andrew, Nabaguu Primary School Terbia teacher / facilitator – observed Taught with computer aid since the broadcast was not on this day. NOTES 51 21-May Friday Okamona Acting Director, Curriculum Development Center Maridi 52 21-May Margaret Ayite Technical Advisor AES – MoEST 53 21-May Isaya Wario Noah Principal TTI Maridi 54 21-May Christopher Loboyong Dean, TI- Maridi 21-May Alex Wowoyo Deputy Principal, TTI- Maridi 55 21-May Amia John Friday ICT Assistant EDC-SSIRI based in Maridi Feb 22-23, 2008 set up TTI computers, Feb 23 had video training for 3 days, then added 9 more days in April, 8 tutors were involved; followed by special training for 3 other tutors on video; Training in basic computer skills from March 3-6 for 6 tutors with a review from March 31 to April 7. Internet training for 8 (all male) tutors with 5 more days in May. He rund the help desk. He was trained in Nairobi in VSAT maintenance and installation. Challenge there is power. Need generator to be wired and finished. Also need 1-2 more hard drives for the video computer. The wireless router also needs a UPC (why not attach it to another with a splitter? 56 21-May Kotoba Peter Bey Tutor, internet trainee Benefits: 1) communication is the main one. 2) News updates 3) search for educational materials, 4) Listen to the radio on line (BBC, CNN, Miraya) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 106 57 21-May Kodani Bosco Roman Tutor – Maridi and Video Group Leader “I speak as a movie producer” We can use it for 1) evaluation of school practices, comparative uses between TTIs 2) showing how to use on-line resources 3) showing pedagogy and methods. “Earlier trained teachers were conservative. We need to be dynamic and current” They need a wind sock and external mic. These cameras they have do not have external mic inputs. They showed films on 1) Using the Environment in taching, 2) Personal Hygeine. Problems in production but a start was made. 58 21-May Robert Wuda, EDC OC Yes, he said, last year’s LV schools know about this year’s schedule. 59 21-May County Education Director Maridi He has 47 primary schools, 356 teachers, 8671 pupils in P1-P8. There is an inspector for AES – Bangama (not ther today). In some schools there are radios. And teachers are trained. He has heard of Terbia and LV though he has never listened to it. He knows the EDC OC Robert Wuda who works directly with the Director. The Senior Inspector for AES at the County is James Daniel. Each of his 5 payams has 1 supervisor. Bamgama gets around on a bicycle. He says he is overworked. They used to have the Intensive English course. Not now. Inservice programs are planned but not financed. They will do training in the long vacation. He understands the design. They will be tested. Problems: 1) Teachers not trained 2) Not enough learning materials 3) Learning environment not good 4) Water at school 5)Getting the untrained teachers upgraded by the process Nama Bullen described does not work since only 87 of the 356 teachers showed up to be tested (exposing themselves to possible firing!) They are now doing evening classes for Arabic patterned teachers (1/4 of them). The CEC may be ready in November. They may start the inservicing on April 1 with SSIRI assisting. The job description for AES Inspector comes from GoSS. 60 May 22 Wau Ismail Mohammed Dahia Head Teacher Hai Salam Basic School – observed school Interviewed in Arabic Issue is teacher can hear the English but they do not know how to translate it quickly into the local language. Class size of Grade 2 over 180. Some over 200. Managing is beyond the control of the teachers. They let ½ the class leave the room and play. English is so important but how? If the children hear LV it is very attractive. But how. There is a radio issue and a teacher issue. He suggests more Inservice training on English for teachers. He knows about Terbia. What about teachers using it? They are all doing Windle in the afternoon. 5 days a week from 2-5. He says Windle will take 15 of his teachers to Uganda for advanced training. There are 15 centers that teach Terbia in the afternoon- (Who said this?) and SW is not clear in the AM but clear in the PM. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 107 61 May 22 Matthew Elemen Musa Director of Planning & Budget Problem of incentive salaries for ALP program. It would be helpful to extend it to all areas. More training for teachers is needed, Terbia is being used widely by citizens. Last year there were 10 learning groups but they stopped. They need more radios to go to other towns. They started in Wau. Equipment for villages needed. Ruben Mayen of MoEST told him that the AES MoEST had sent the top up money for AES incentives for 2007 and the first 3 months of 2008. Musa said where is it. We have not seen it. Sara said that there was a feeling in the schools that the LV teachers should get an incentive like the faciliatators. They need to be informed by the SMoE that this is not the way it works. Musa suggested a 1-day workshop for trainers to enlighten them. He is having his teachers to Windle (100 this year) 300 were started last year. They offer leveled classes. SSIRI could do joint classes with them. 62 22-May Elias Enrico Wandu Director of AES Wants Learning Village in all 35 schools in Wau 63 22-May Radyap Sidrea Director of Secondary Schools Wau SMoE 64 22-May Ahmed Hassan Director General / Head of AES? 65 22-May Andrea Henry Akuar County Education Officer Facilitator for Terbia For some another group he knows of months did Terbia but they stopped because of salary. He continued. Terbia is done together with ALP. The radio started at 10AM on FM. He would hope to put it on Wau radio for local use (up to 24 miles from town). 4:30 is a better time for it on FM. 60 teachers were trained by SSIRI (as facilitators or P1-3 teachers?). But there were radios for only 20 of them. 2 went to Wau and 18 went to Jor River County. Terbia groups are only in Wau town. Terbia B has no problems, They want to expand it. It needs motivation in incentives for facilitators. Learning Village is not working well. The start time of the series this year was not done. The Guides are not enough in quantity even for those trained (33) Only 18 of them got radios. They wish to serve all. 66 22-May Donato Ugali Director of Basic Schools Wau The school day has 7 periods of 40 minutes each. The is no problem in timetabling. This needs only local arrangement. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 108 67 22-May Grace Akukwe was present She said Development Partners need to involve the MoE in planning and executing activities. MOE has no funds, they rely on the partners, this creates a learning opportunity. Note that no MOE staff went along on today’s school observation. Recommendation to get equipment especially for the departments working with SSIRI, computers so that the State and counties can have common system. 68 22-May John Peter Gonyo Ali Director of Administration and Finance SMoE W.B.Gs.Wau Suggest EDC establish own offices separate from the MOE since the EDC staff have to keep shifting around and there is no space for them. They promised a container but it never happens. SSIRI is a good project. There is a military garrison 210 miles from here that is using Terbia regularly. The MoE (through him) would be happy to second someone to the project and pay them to do SSIRI. The person he had in mind he could not say the name immediately but it was something like (Karia Dominick). I mentioned this to Tom Tilson. 69 May 23 Malakal Observed teacher Police Boys Basic School Observation Notes: Grade 2 (very quiet radio). Class held outside. I got the impression that there was less English being learned in this class today than if there was no radio. Interesting orthography with Arabic R to L and English L to R on the same blackboard. Grade 3 listens inside. They did not expect our visit to their room. The children all knew the song. The sing-along song in the Grade 3 program was much slower than the one in Grade 2 and much more singable. Grade 3 kids count from 25 to 60 by 5’s. The radio ran down in mid lesson. 70 23-May Joseph Mitugo Informally assisting AES From Kenya 249 918 427 488 kihorob@hotmail.com 71 23-May William Dak Head Teacher At school where we observed We used LV P1, P2, P3 last year although the voice was low we use it because there is a program there. We put the radio in the middle of the room. The program is important because it’s in English and if you compel the students they will learn English. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 109 72 23-May William Gwang Deng Director General, MoE Upper Nile State The radios came. Some activities going well. Others need more work to be done. Orthography needs to be supported. It goes in one ear and out the next for many. “Some treat it like regular news”. He has listened to Terbia 1 time. The AES Director went out today (name: Ouyap ______). The MW radio is state owned and it covers all of the Upper Nile. There are no reported complaints about LV. With so many as 200 in a class he is moving to a shift system next week. ½ the teachers in the AM and half in the PM. (How will this help unless all teachers teacher upper and lower levels). They are now doing the new curriculum for P1-4. Grades 5-8 are still the old system. They are trying for the Intensive English program. They just finished training 130 and Windle will take over this group starting 5/26 and run until October 2008 2-5 pM 5 or 6 days a week. More help is needed to the AES system. AES teachers get an additional 185/mo to augment the 175 they get regularly. The state budget is down from 200K/month to 140K/month. No money for services, buildings, etc. There is internet at the MOE. Many NGOs subperform and there is need for due diligence over performance. He was asked about the MOE counterpart to the SSIRI person. Ruben Mayen clarified that it is for the GoSS AES to clarify the SSIRI person. 73 23-May TTI Principal - Malakal His school is closed. Buildings are being used by the MOE but there is no staff. The Islamic University had the building. They were asked to leave in 2006. The University and the MOE are now using parts of it. 74 23-May Julius Onen Center Manager & Senior Tutor Windle Trust, Malakal Windle is not using Terbia now. Terbia trained 45 facilitators and Windle was involved. Windle’s audience is especially primary teachers. GoSS wanted Windle to focus only on ALP and English for beginners but it is not enough. They do not deal with pedagogy. There is an opening here to use the PS101 as well as the Terbia and maybe LV with Windle teachers. LV can be a sister program emphasizing speaking and listening and pronunciation. Take the IRI then bring in live interaction in class. Repeat and write. They tried Terbia in Juba in 2 Windle classes. They want to integrate it not only in class talk but maybe through Radio Bakhita. He would like to see more about the differences between LV and Terbia to integrate them both in his Windle usage. There are 3 levels of instruction in Windle. There are 10 senior tutors at the national level and all could be given a training. The national head (Acting Director) is a fellow in Juba named James__________(?). They are in Bor, Wau, Raja, Mangala, Tarateka, Tori, Malakal, Juba, and one or two other sites. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 110 75 23-May Engineer from Nairobi Visit to TT Resource Center - equipment sited off the campus of the TTI. A VSAT was being installed and it will provide a link for the MOE. The equipment will eventually move to the TTI when and if that gets up and going. The building was very congenial for future collective use – much like the Maridi TTI IT center can be. The Center will have 6 computers, 2 printers, all networked. Furniture to be provided by the MoE. A full time IT specialist will be assigned by SSIRI to the location. We aksed for uplink and downlink speeds but did not receive the information. 76 23-May Terbia B Facilitator They are not using the radio now because the program is not on the radio. We informed him that it was on this afternoon and would he teach a lesson. Yes. The facilitator said that no radio had been given to him. He tried but the SW signal was barely adequate. The Terbia B1 and B2 were next to one another on the dial in the 15 meter band so there was no way to tell which program one was receiving. Only one of them was audible. The other Terbia B could not be found. Terbia A in the 12 meter band was strong and much clearer. 77 23-May Osama Mahmoud Director of Radio Radio Malakal They may soon have a 1 KW FM station at 98 FM. 78 23-May Luka Benjamin Head of Engineering Radio Malakal The station started in 1992. They were upgraded in 1994. They have help from GOAL in the Netherlands. Now they have a Harris 5 KW Medium Wave (AM) transmitter. They have radio drama (Amak Tabani – about a tailor who collects money and does not do the work, etc). They also have children’s stories on Monday evening. The station can be heard far away in Unity, Jonglei, all of Upper Nile, even in Ethiopia (Gambella). They broadcast from 7-8:30 AM and from 5-8 PM. Their programming is education and health from UNICEF, MOH, GOAL. It is one of three stations owned by the government of GoSS – Juba, Malakal, and Wau. They get news read to them over the phone from Juba which they revoice. Unicef did a survey of their audience. They broadcast in 6 African languages (each evening? Sam e 10 minutes text in an hour.) They have no mp3 disc playback though they can take the output of a computer into their mixer. The tower is near the TTI. It is a 30 meter mast. Capacity building is needed for staff. Achuil has taken steps to get SSIRI on their air. 79 23-May Achuil Arop SSIRI Outreach Coordinator Aarop@edc.ssiri.org 88216-43332477 He gave an orientation on LV and Terbia to Windle. The divided into 10 groups. Terbia B was considered good for them. IN the meeting with Windle’s Julius he said that he sat with the Minister and suggested that they take the top 100 of the 300 teachers and have the best teach the lower classes. The others should be sent to Windle. Last month he trained 196 teachers introducing them to LV and Terbia. He assessed their English and found that 50 were OK in English. They should be posted to the right places. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 111 80 23-May Alexandra Sicotte-Levesque Senior Radio Producer / Trainer BBC World Service Trust 249-183-481-851 249-91-457-47-34 (mobile) Alexandra.sicotte￾levesque@bbcwst.net 81 23-May Sonia Whitehead Senior Researcher - Sudan Sonia.whitehead@bbc.co.uk 82 21-May Kutiyote James Francis Coordinator for Teacher Education – CDC Maridi 83 26-May John Tanza Mabusu Radio Service Manager Sudan Radio Service EDC - Nairobi 0720-293-302 84 26-May Agnes Andi Teacher – Grade 1 St. Theresa Basic School Juba She has the guide at her school, Was ill the day we saw her and so not teaching. Last year did 69 LV lessons. She knew that this year’s start date was May 19. She has not yet taught because she was ill last week. She preferred to answer questions in another language. She has 195 students in her class all on the floor. They all knew and enjoyed singing the Learning Village them with me. Her method was to establish management first with a story and a song, then to tell them to prepare to hear something new from the radio. All is well with using it and the pacing is right unless they begin to get unruly and then she may lose her place in the lesson if she is dealing with them. Some children start out fine but then fall out of the lesson. She has learned a generalizable skill from the radio – specifically to revise the content of the lesson. She uses this even in other subjects. There are three radios in the school. A Care representative was selling English primers at the school. We wondered if they are supporting the use of the books in the classes and if that may be competing for mind share with teachers. 85 26-May Teacher – Grade 2 Does not use the LV because of timetabling, and she may just prefer her own teaching. Sara thought she might just be slow to pick up on it since she is an older teacher and set in her ways. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 112 86 26-May Francis Lokoses Loku Teacher – Gr 1, Gr 3 last year) Bukuk “A” 1 Basic School Juba This school had two radios but one teacher was transferred from Grade 1 to the office and took the radio with him to the office. It is not used now. The Grade 2 and 3 teachers were not present today due to a funeral. The Grade 2 and 3 students were not meeting at 10:50. The Grade 1 teacher had taught about 30 Grade 3 lessons last year. This year he had taught only one of the first 5 lessons aired last week – and only the math section since the timetabling was not easy to work out. He said that he is the English teacher only for that grade and that when the other teacher is done with other subjects only then can he teach and the radio may not be timed right at that time. The principal however wants the school to norm the timetable around the LV schedule. We could not check this with the school head since he left before we could talk to him. He felt that the speed of the lessons is OK. The teacher’s guide was locked in a cupboard that he could not gain access to at the time we were there. He had had a training – 1 week first and then a second week to supplement it. 87 26-May Michael Lokiyek Arkangelo Teacher and Adult Ed Facilitator Juba Michael was trained last September. He taught Learning Village and Terbia last year. He teaches at a school with a Basic School in the AM with 1230 students. Today they did not do LV because of funeral and sickness for two teachers. The only remaining trained teacher was needed elsewhere at radio time. The class was not really organized on time and I think it might not have happened if we had not been there. It was the first time he taught Terbia by radio this year. He had trouble tuning the radio from an interference zone. His timing was individualistic. He spoke over the radio much of the time but when he was not teaching his own way he was attending to it and keeping the class of 43 very active. The studentsw ranged from 10 to 40 years old. They were taught prepositions. The bag is under the table, etc. He treated the class like an immersion lesson not translating into the mother tongue each time. The regularizing function of the radio was evident. There were higher order thinking issues offered such as “Ask the students why it is important to hear the news from various sources.” With regard to student learning with and without radio, he has been teaching such courses for 5 years and “There is a big difference. The students are interested to learn. They can express themselves in English.” Math may be too easy fro some of them. However I noticed that one student could not write the numbers 1-5. The other trained teacher there is named Emmanuel. Michael teacher P8 and P6 English and Agriculture at P4 plus evenings. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 113 Annex 11: SSIRI 2nd Response to Evaluation-“One Teacher At a Time”-June7 (32 p. with five EDC Annexes) June 7, 2008 To: EDC/SSIRI Staff From: Tom Tilson, Chief of Party Re: Mid-term Evaluation and new implementation strategies “One teacher at a time” This is a very important document! My apologies because this document is long and detailed, but there is much information to be conveyed immediately. Please read this document carefully and begin working with others on implementing the tasks right away. Each field staff member from SOA to OC is to acknowledge receipt of this document no later than 10 June to his or her supervisor. • Each OA is responsible for reporting receipt of this document by every OC he or she supervises, and submit the report as follows: o W. Equatoria, C. Equatoria, E. Equatoria, Jonglei, and Upper Nile to Edward (then to Maree) o W. BeG to Bullen (then to Maree) o The Three Areas to Maree • Janet Mola is responsible for reporting on the M&A Assistants to Tom. • Athanas Mwamba is responsible for reporting on all IT staff to Tom. Acknowledging receipt of this document is an important first step regarding our ability to communicate with each other. However, the most important activity is to review carefully this document. Using the reporting guidelines above, Edward, Bullen, Maree and Tom are to review this document in detail with the SOAs, OAs, M&E Specialist, and ICT Coordinator. These people, in turn, are to review this document in detail with the OCs, M&E Assistants, and IT staff no later than 15 June. This will mean setting up face-to-face meetings with every staff person or, if that is not possible, lengthy phone calls. Education Development Center, Inc. Southern Sudan Interactive Radio Instruction (SSIRI) Project UMEME Plaza (Off Ngong Road on Old Naivasha Road) Œ P. O. Box 25010 00603 Œ Nairobi Œ Kenya Œ Telephone: 3870906/3873088/3872269 Fax: 254-20-3876520 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 114 The SOAs, M&E Specialist, ICT Coordinator, and Deputy Chief of Party will need to confirm by 15 June that this document has been reviewed in detail by every field staff member. Deadlines There are several deadlines included in this memo. They are listed below for your reference. See the text later in this document for additional information for each the tasks listed in the table below. June 10 1. Acknowledgement of receipt of this document. 2. Send to Juba a list of needed radios for all previously trained IRI teachers June 15 1. Confirm that his document has been reviewed in detail with all field staff. 2. Complete communications plan for each outreach staff. 3. Submit a plan for activities in June 4. OCs to have begun DAILY visits to chosen schools and learning groups 5. Inform all IRI teachers of new broadcasting schedule. (See Annex II for the broadcasting schedule.) 6. Arrange for distribution of radios to all previously trained IRI teachers as part of refresher course. 7. Communications Specialist to submit updated plan on activities to promote IRI programs. 8. Begin working on an expansion of PS101 with the Curriculum Development Center 9. IRI Advisors to establish plan for reviewing the length of pauses and for doing FE in Southern Sudan June 30 1. Submit State and County Plans for 2008 2. Complete MoU for each state 3. Every outreach staff to submit July activity plans to supervisor with copies to Deputy Chief of Party and Chief of Party (tentatively plan on EDC training the week of 14 July) 4. Complete implementation plan for PS101 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 115 Mid‐Term Evaluation I just completed participating in a three-week evaluation of the SSIRI project. A Mid-Term Evaluation is standard for all USAID projects. Although this review was later than normal in the life of a project given that our contract only has one more year, it is not too late for us to make some important corrections18. This memo focuses on the results of the recent Mid-Term Evaluation of SSIRI and some of the key implications for our work in the days and months ahead. Summary Report of the Mid-Term Evaluation The Team visited Juba, Torit, Arapi, Yambio, Maridi, Wau and Malakal. The trip to the Three Areas was cancelled. In Arapi and Maridi, the Team only looked at the technology initiatives at the TTIs. In other locations, the Team observed Learning Village and Terbia classes and held discussions with teachers, head teachers, education officials, and representatives from other USAID education projects and the World Bank. I went into the evaluation thinking that the project was in pretty good shape, but came out of the experience recognizing that we have some major problems. I will begin with an outline of some findings and recommendations of the Evaluation Team. I should receive the full report of the Evaluation Team later in June: Summary of Evaluation Team Report Overall Findings 9 Radio program production progressing according to plan 9 Listener Guides are of high quality, well integrated 9 Major problems with radio and digital technologies 9 Fewer than 1 in 7 Learning Village trained teachers using radios Comments on the IRI Programs 9 Program design is solid / good, clear objectives 9 Regularizes learning 9 Some good interactivity – potential for more 9 Teachers should be able to learn some generalizable skills useful in other classes 9 Some questions: Some pause lengths seemed too short; the pace often seemed too fast 9 Speed of some songs (do they reinforce natural English?) 9 Level of English in the programs demands too much in relation typical teacher’s capacity Major issues (A) radios and digital technologies 9 Broken radios (~5% in one year) 9 Observed radio experience for learners - not good 18 The team consisted of two external evaluators ( Stuart Leigh, Team Leader and a highly experienced IRI Specialist) and Charles Tesar; two USAID staff (Lucy Kithome and Inez Andrews);a representative of the AES Department/MoEST (Reuben Mayen); and a Sudanese communications specialist (Sarah Mursal). I served as a resource person. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 116 9 SW signals: problems with strength and clarity / times of day (Terbia) 9 Miraya FM signal too soft in some locations (problem with simultaneous broadcast of Miraya FM on a different frequency) 9 Radio volume too low for large classes 9 MP3 players with solar panels – batteries failing; even the JWIN not loud enough for large class (B) Teachers not using programs 9 Teachers dropping out - frustrated by sound issues or by their insufficient English skills to participate Recommendations 9 More study of digital technologies 9 Teachers need to be instructed in the use of radios: e.g., tuning shortwave, using 2 or more Lifeline radios in larges classes, tuning in Miraya FM on alternate frequency (around 95 on the dial), etc. 9 Audio programs need to be systematically analyzed and edited 9 Create at least 1 Model School per SSIRI county 9 Fully involve the states since their influence flows to counties 9 Train EDC staff and counterparts at all levels 9 Ensure full involvement of Basic Education Directorate as well as AES (because Learning Village is part of basic education) 9 Work with other ESL/EFL trainers, e.g. Windle Trust, starting in Malakal 9 Assess radio transmission and listenership, incidence of radio use in classrooms, radio programs’ effect on learning, comparisons of radio with other technologies 9 Focus on Feedback: Outreach staff reporting should be regularized and reviewed frequently, and responsive follow-up rapidly initiated 9 Broader more frequent field monitoring including senior staff (EDC/SSIRI – USAID) 9 Implement complete PS 101 soon and develop the new programs In short, the Mid-Term Evaluation was a wake-up call. Yes, we have much to be proud of – we’re enthusiastically supported by the Ministry of Education, Science and Technology at all levels, we have good radio programs, great teacher's guides, we’re on schedule in terms of production, and we have good partners such as Miraya FM and Bakhita Radio. But our radio programs still need improvement and, most importantly, we fell short in the one place where it matters the most – in the classroom. Almost all classes that the Team observed had problems – the radios didn’t work right, the batteries in the digital devices ran down, the volume of many devices was too soft, and many teachers had difficulty carrying out their roles. And when the Team members talked to teachers about their experience with SSIRI last year, they found out that most of the teachers had stopped using the programs after a few weeks. We fell short (including myself) in not visiting classes regularly, in not reporting the problems that we saw, or when problems were reported in not understanding the depth of the problem and taking action. We didn’t train our field staff adequately or hold them accountable. We didn’t give adequate training to classroom teachers and facilitators or provide follow-up support. Implications for SSIRI of the Mid-Term Evaluation We now have a great challenge and opportunity. It’s our last chance to demonstrate to USAID, to ourselves and, most importantly, to the people of Southern Sudan that SSIRI can make an important SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 117 difference to the lives of children, youth, adults and teachers – that SSIRI can enrich the lives of others by providing good quality educational programs. Fortunately, USAID and the Ministry remain very supportive. We are all committed to making sure that SSIRI works well, but we have to change some key elements of our strategy. We have to begin implementing our new strategies IMMEDIATELY, as there is another USAID team that will visit several of our sites in just over two weeks. See my other memo on the upcoming USAID Education Audit. The following sections describe findings and actions to be taken. The topics are in rough order of priority. Implementation at the classroom level 1. The highest immediate priority is for each OA to review in detail this document with each OC that he or she supervises and to start working on new the plans and activities accordingly. This review is to be completed with a confirmation sent by 15 June. 2. Our overall top priority, which is to start immediately, is to ensure quality learning experiences in Learning Village and Terbia classes. • An OC in every county where we are working is to focus on ensuring excellent IRI lessons in just a small number of classrooms, starting with “one teacher at a time.” • We will establish at least one Model School in each county, beginning with a few model teachers. (A Model School may have just 2 or 3 teachers who are exceptionally good, not all teachers.) 3. We will reduce our plans for expansion this year. These plans were based on the assumption that we were having a good experience with IRI lessons in the classes and learning groups, but this is not the case. • We will expand activities only modestly in six states (W. Equatoria, C. Equatoria, E. Equatoria, Jonglei, W. Bahr el Ghazal, N. Bahr el Ghazal); maintain activities in one state (Upper Nile), and expand in the Three Areas (Abyei is doubtful at the moment). We will postpone activities in Lakes, Unity, and Warrap States. • We will consolidate staff in some states by shifting OCs from some new counties to more central counties. SOAs and OAs should consider selecting counties, in part, on the interest and willingness of the officials to become fully engaged in SSIRI. Also, ease of access should also be a consideration. 4. We will develop an improved training program with better materials for our own staff so that each person has the necessary skills and tools to carry out the work. • With the help of a new Senior Training Advisor and a consultant, we will revise, improve and expand our training materials starting in June. • We will make greater use of multimedia materials including video. • There will be a training workshop for all EDC outreach staff and, possibly, Ministry counterparts, in mid July. There will be follow-on training in the states. 5. We will improve the training program and materials that our outreach staff need for training teachers, facilitators, and Ministry officials. 6. We will work to ensure that all of our outreach staff report regularly – initially, there will be weekly reports on daily visits to schools as well as the usual bimonthly reports. Each staff member will be held accountable for the activities and timely reports. This will be a condition of continued employment. 7. For OCs who do especially well, we may create a new position called Senior OC with some additional responsibilities. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 118 8. The Outreach Advisor for each state will have overall responsibility for implementing our new approach. There will be supervision and support from the Senior OA, the Deputy Chief of Party, the M&E Specialist, and the Senior Training Advisor (to be hired). 9. In order to continue efforts to integrate fully the SSIRI programs within the SMoE, all EDC staff will coordinate with and seek to fully involve state, county and payam staff in activities. Technologies Radios and digital technologies There were major problems with the radios and digital technologies in the classrooms observed by the Evaluation Team. The following is a table indicating the problem and action to be taken: Problem Action to be Taken The Team found that many teachers who had received IRI training last year, especially in Wau and Malakal, did not receive radios because our supply of radios had run out. Work with the Juba office to obtain the necessary radios and plan for distribution by 15 June. I would suggest that the radios be distributed as part of a refresher training to be held as soon as possible. Timetabling in the schools was a significant problem because either a grade 2 or grade 3 class is missing the Learning Village because of recess OCs to work with the Head Teacher to try to arrange a staggered recess period so that no class would miss the IRI lesson Listening to programs on shortwave was problematic. Part of the problem is that the programs are broadcast during the daytime when shortwave signals are the weakest 1. Diagnose the shortwave reception during class time in the mornings and afternoons. Report your findings to the OA, SOA, and Deputy Chief of Party. Seek alternatives including a request for digital devices if the shortwave reception is not good and FM or medium wave is not an option. 2. Train teachers on tuning in shortwave stations and on using the antennae 3. Provide further separation in the frequencies for Terbia for Beginners -1 and Terbia for Beginners -2 (done effective June 9). See Annex II for broadcast time and frequencies. 4. Additional FM stations will be used for broadcasting the Terbia lessons. 5. EDC will explore the possibility of using medium wave stations in Malakal and Wau for Terbia and, perhaps even for the Learning Village. 6. Expand the use of digital media players Volume of the Lifeline radio too soft 1. In larger classes, provide a second (or third) Lifeline radio at the sides or back of the classes. Use one Freeplay radio in a class for every 40 students. 2. For Miraya FM, if the sound is too soft on 101, try tuning in on 95 on the dial. Volume on JWIN radios too soft when using an MP3 player 1. EDC will check on sound levels of programs recorded on the Nextar. 2. EDC will test the use of flash drives and SD cards in the JWIN radios 3. EDC will identify larger boom boxes Batteries on the digital devices go dead, especially 1. EDC will look for an alternative to the Nextar MP3 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 119 Problem Action to be Taken with the Nextar MP3 player player 2. EDC will provide a larger solar panel for charging the AA batteries in the small speakers Digital systems not yet of sufficient quality 1. Try out iPod system connected to a Freeplay radio being used by EDC in neighboring countries 2. EDC will continue to assess alternative devices, some of which have already been identified 3. EDC will institute more rigorous M&E of the devices in the field, especially with support from the Technologies Specialist Change the term “low-cost technologies” We will start using the terms “digital devices” or “digital media players” Teacher training technologies 1. The observations by the Mid-Term Evaluation team regarding the visit to the TTIs include the following: a. There is a good start regarding the use of the Internet, but more work needs to be done. b. The team appreciated the fact that some staff at both institutions have already mastered the basic techniques of video production, but much work is needed on designing appropriate videos that will enhance the classes at the TTIs. 2. EDC will now focus activities on Arapi and Maridi TTIs plus the Curriculum Development Center on using the Internet and video production (TTI only). We will also provide basic support to the new computer/VSAT center in Malakal. 3. We will postpone the development of additional centers until our model is better developed at Maridi and Arapi. Improved monitoring and reporting A major shortcoming of our activities to date has been the poor monitoring and reporting process. Perhaps the single most important action to be taken in order to improve the implementation of SSIRI programs is to enact a monitoring and reporting process that works well. Each outreach staff member will be held strictly accountable for monitoring and reporting activities. • In order to implement the strategy outlined in this document, we must be able to communicate readily among all staff, but especially between the OCs and OAs. The first activity to test this premise is for all outreach staff to report on the receipt of this document by June 10. • We can only continue to support staff who are able to establish and maintain regular communication – at least every other day – either though email or by phone (Thuraya or mobile phone). That is, we will be forced to close down activities and/or replace staff in counties where we are not able to maintain regular communication. • EDC will provide computers and Thuraya phones to all staff including solar panels where needed, and even an Internet connecting device for isolated locations. • We can make greater use of SMS messages for regular and quick updates or queries (Every initiated SMS needs a reply so that the sender knows that the message has been received.) • A communications plan and agreement (communications channels, time, problems to be addressed, etc.) is to be worked out with each outreach staff - Deputy Chief of Party with SOAs, SOAs with OAs, OAs with OC, M&E Specialist with M&E Assistant, and ICT Coordinator w IT staff. It is the responsibility of each staff member with a Thuraya to ensure that the batteries remain charged and that he or she has the phone turned on at least during certain specified times of the day. (Janet Mola can assist in designing a format for the communications plan.) This plan is to be completed by 15 June. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 120 • See Annex IV for a list of current staff, phone numbers, and email addressed. (Please send me corrections or updates to this list. • Each OA and SOA needs to prepare a log of communications with OAs and OCs respectively, and to submit the log at the end of each month. (Janet Mola can assist in preparing a log sheet. The M&E Specialist and M&E Assistants can monitor these logs.) • In order to assist outreach staff to improve their computer skills, EDC is hiring two IT Officers who will move from one location to another to provide training and support on computer skills, the use of email, and how to work with EDC computer-based forms. Lack of computer skills will not be an excuse for failing to provide reports on time, as telephone is an option. • Direct all computer-related queries to Athanas Mwamba. • Each OC is to visit an IRI class almost every day, provide support to the teacher/facilitator, and report on the findings and support activities. For the next few months, the OC will provide a WEEKLY detailed report to the OA and M&E Assistant on activities – name and date of schools and listening groups visited, any new data, problems identified, assistance given, training activities carried out, meetings with others, etc. • The M&E Assistant in each of the six states, plus additional coverage for the other locations, is to coordinate with the OA in developing a monitoring plan for each OC and to ensure that the progress is reported weekly. The M&E Assistant with support from the OA is to send the weekly report to the OA, who is to forward the reports to the SOA, M&E Specialist, and Deputy Chief of Party. • Senior level staff including the Chief of Party, Deputy Chief of Party, Senior Training Advisor, and the M&E Specialist will regularly visit field sites including observation of Learning Village and Terbia classes. At least one of these people will conduct such a visit every two weeks. • In September 2008, an EDC team including the Regional Director plus USAID representation will visit several sites using a charter flight in order to visit classes and talk to teachers and officials. Such monitoring visits will be conducted quarterly. Planning • The Communications Specialist needs to expand a range of activities to advertize the IRI programs including occasional special launching events. The major purpose is to increase the listenership of Terbia programs, but also to inform teachers and communities about the Learning Village. An update of such a plan should be submitted by 15 June. • One of the findings of the Evaluation Team was that IRI teachers from last year were not aware of that IRI had started again this year. Please inform all teachers of the new broadcast schedule in Annex II by 15 June. o Note that P4 will begin broadcasting on June 16 and that ALL the Learning Village and Terbia programs will be repeated starting on that date. This will give OCs time to continue training. o It would be best if the County Education Officer, perhaps together with the OC, issued a written memo to all IRI teachers about the broadcast schedule. The OC is encouraged to draft this document for the County Education Officer incorporating the broadcast schedule in Annex II. • Each OA and OC is to have a state and county level plan respectively by 30 June. Some components of the plan are to include: o The Deputy Chief of Party with assistance from the SOAs is to provide an outline for sample state and county plans by 15 June. • A MoU is to be drafted among MoEST, EDC, and the State by 30 June. See Annex V for a draft MoU. The details for EDC and each state have to be modified to meet the circumstances in each state. • All outreach staff are to submit monthly plans to their supervisors by the end of the previous month, e.g., the plan for July is to be submitted no later than 30 June. In preparing the plans, assume that there will be an EDC training workshop during the week July 14. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 121 • Any individual who is having difficulty completing these plans and reports, is to consult with his or her supervisor well before each deadline. EDC will do everything possible to give assistance, but, in the end, it is the responsibility of each person to submit all reports on time.. • The submission of all the above plans (state/county annual plan, monthly plan, and MoU), as with the required reports (weekly and bimonthly), is a condition for continued employment with EDC. Improving our programs The Mid-Term Evaluation Team appreciated the quality of our radio programs, and they especially liked our teacher's guides. However, they also had the following comments for improving our programs: • Provide more opportunities to actively involve the learners. One of the evaluators was critical that the lessons were not child centered. • Review to ensure up-to-date English language instruction pedagogy • Review the pace of the lessons to ensure that they are not too fast • Review the pauses to ensure that they are not too short • Provide occasional longer pauses so that teachers can work with children on specific activities • Simplify the language used for instruction to make it as easy as possible for teachers to do the translations • Provide more songs and ensure that the pace and content are appropriate for reinforcing the teaching of English or other objectives • Strengthen formative evaluation through broader and more systematic review as lessons are being developed and introduce more FE in Southern Sudan, perhaps by increasing the number of scriptwriters so that 1-2 can rotate in Southern Sudan in order to carry out FE. • In the future, consider replacing the local language literacy component with additional, improved English language instruction Some of these suggestions can still be applied to the remaining lessons being developed, but for already completed lessons, probably the only thing possible without a major effort would be to adjust the pauses. The following actions can be taken by the IRI Advisors and scriptwriters by 15 June: • Establish a plan for reviewing pauses • Begin implementing a plan for conducting FE in Southern Sudan Professional Studies for Teachers The Evaluation Team emphasized our shortcoming in not implementing PS101. Not only is this a deliverable under our contract with USAID, but it has important content for teacher development. We have postponed implementation for several reasons including the fact that the MoEST has not made progress on implementing the in-service teacher education program through County Education Centers. In addition, we have given priority to implementing the Learning Village and Terbia. EDC will do the following: • For PS101 develop an implementation plan by 30 June including: o At least one SSIRI county where we can involve IRI teachers. One model is to engage teachers who may be attending IRI refresher courses. o At least one county (to be determined) that is implementing the in-service program through a CEC • By the week of 15 June, begin working with the CDC in Maridi on the development of additional audio series, probably continuing with Level 1 of the in-service program SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 122 Additional support for teachers in English Many teachers would like to use the Learning Village programs, but are struggling because of their weak English skills. EDC plans to do the following in order to help these teachers: • EDC has expanded the length of initial training workshops from 2 days to 4-5 days so that teachers can have more practice using the IRI programs, especially for translating instructions from English into the local language • EDC will consult with MoEST regarding testing teacher's guides that include instructions in Arabic as well as English. • EDC is working with the scriptwriters to reduce the information burden on teachers, especially instructions that must be translated – reduce the length of instructions, break up instructions into small pieces, simplify the language, etc. • EDC plans to create a new English for Arabic Pattern Teachers course based on the Terbia programs • In the future, EDC will consider developing a new audio series for teachers, perhaps weekly, and both in Arabic and English, that would provide specific information on upcoming lessons, such as content, English usage and vocabulary, and suggested activities. Some specific activities to be initiated by 30 June include the following: • Establish a plan for translating some of the teacher’s guides into Arabic and for assessing the effectiveness of using bilingual guides • Establish a plan for initiating the new English for Arabic Patterns Teachers based on the Terbia programs Administrative Issues Staffing In order to carry out our activities with sufficient support, EDC will hire the following new staff: 1. Deputy Chief of Party for Operations (Nairobi) – This person is to provide management of administration, finance, logistics, procurement, and HR and will help to ensure that the SSIRI activities proceed as effectively as possible. This person will have responsibility for SSIRI activities in both Kenya and Southern Sudan. Candidates are being interviewed. 2. Administration and HR officer (Juba) - Candidates are being interviewed. 3. Financial Assistant (Juba) – Candidates are being interviewed. 4. Short-term Logistics Assistant (Juba) – To be advertized. 5. Senior Training Advisor (Juba) - This person will to oversee all new training activities and materials development (contract being offered) 6. Training Consultant (Juba – two months) – The consultant will assist in the development of new training materials (Phoebe McKinney is to begin on 9 June). 7. Senior Teacher Training Advisor (Maridi) – This individual will oversee the development of new teacher training programs including the expansion of PS101 and a new English for Arabic Pattern Teachers based on Terbia. (Candidates being reviewed) 8. Teacher Training Consultant (Maridi). This person will work with the CDC to begin the design for expanding PS101. (Nick Boke is to arrive on June 10.) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 123 9. Learning Technologies consultant (Maridi and Arapi - two months) – This person will focus on training and developing materials on the use of the Internet for curriculum development and staff improvement plus training on video production. (Richard Cavagnol is to arrive in Nairobi June 10.) 10. Recruitment Specialist (Juba) – This person will recruit staff for new positions and for replacing departing staff as may be needed. (Position being advertized) 11. Teacher Training Specialist (Juba) – This person will be responsible for implementing PS101. (Position is yet to be advertized) In addition to the new staff positions listed above, we are also considering adding more OCs to certain counties. If at all possible, one of these positions should be covered by a county education staff member. The responsibilities would be as follows: • One OC is to focus on just a few schools to ensure quality implementation. The OC is to visit classes at least twice a week and provide support and mentoring to teachers. This OC is to develop at least one Model School per site. • One OC is to monitor and support other participating schools in the county through occasional visits to most of the schools, collecting good data on implementation, and organizing periodic refresher courses for the teachers. • One OC is to support the Terbia programs • And, where PS101 is implemented, possibly one additional OC to support this program Transportation The success of our initiatives will depend in part on transportation, both for our staff and for at least some staff at the GoSS, state, county, and payam levels. Thus, EDC will procure the following: 1. One vehicle is being procured for each OA (Procurement has been initiated) 2. One vehicle is being procured for assignment to the MoEST/AES Department 3. If approved by USAID, one motorbike will be procuring for each country and state (area) where we have staff. (Waiting approval from USAID) 4. One bicycle for each payam where we are working (OCs need to move ahead on this procurement immediately) SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 124 (EDC) Annex I – Some characteristics of Model Class 1. A good radio with loud volume (We found in several locations that the Miraya FM signal was much stronger around 95 on the dial instead of the official frequency of 101) 2. The radio is sufficiently charged for the lesson 3. There is a second Lifeline radio in the back of large classes – one radio for each 40 students 4. The teacher has the radio clearly tuned in before the time of the lesson and places it in a good location in the classroom 5. If a Terbia group is using a digital device, ensure that it is working well, that the batteries are well charged, and that you have the correct lesson ready on the device. Have spare sets of fully charged batteries. 6. The teacher has read the teacher's guides prior to the broadcast and is prepared with all required materials and with information written on the blackboard as may be needed 7. The class is ready and attentive at the beginning of the lesson 8. The lesson begins right on time - not late 9. The teacher does an excellent job translating the instructions, organizing the children as requested by the radio teachers, writing on the blackboard as needed, and fully engaging the children. The teacher is clear about what instructions get translated into the local language and which are to be given in English. 10. The children are attentive and participating actively 11. The teacher follows up well with post-broadcast activities based on the teacher’s guide. 12. The teacher uses the SSIRI program regularly - every day, or three times a week for the Terbia class 13. The teacher fill in the proper SSIRI form after the lesson. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 125 (EDC) Annex II – Time and frequencies of broadcasts Broadcasts begin 12 May 2008. The whole series will start again on 16 June. Radio Service Schedule Program Category Frequency Shortwave Mo-We-Fr 9:30 a.m. Mo-We-Fr 4:00 p.m. Terbia for beginners 1 (Repeat) 15660 KHz 15760 KHz Mo-We-Fr 9.30a.m. Mo-We-Fr 4.00 p.m. Terbia for Beginners 2 (Repeat) 17660 KHz 15390 KHz Mo-We-Fr 9.30 a.m. Mo-We-Fr 4.00 p.m. Terbia Advanced (Repeat) 15530 KHz 12070 KHz Bakhita FM Radio (Additional FM stations to be added) Mo-We-Fr 4:30 p.m. Tue-Thu-Sat 4.30 p.m. Mo-We-Fr 10.30 a.m. Tue-Thu 8.30 p.m. Terbia for beginners 1 Terbia for beginners 2 Terbia Advanced Terbia Advanced (Repeat) 91 FM Voice of Kauda (Additional Internews stations to be added) Th-Fri-Sat 5.30 p.m. Tentative FM Shortwave-The Learning Village Mon-Fri 9.00 a.m. Primary 1 15215 kHz Mon-Fri 9.30 a.m. Primary 2 11905 kHz Mon- Fri 9.00 a.m. Primary 3 15750 kHz Mon-Fri 9.00 a.m. Primary 4 (Beginning 16 June) 15760 kHz Miraya FM Mon-Fri 10.00 a.m. Primary 1 FM 101(*) Mon-Fri 10.30 a.m. Primary 2 FM 101 Mon-Fri 11.00 a.m. Primary 3 FM 101 Mon-Fri 11.30 a.m. Primary 4 (Beginning 16 June) FM 101 (*) In some locations, try tuning to 95 for better reception. 1 June 2008 SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 126 (EDC) Annex III – List of Telephone Numbers and Email Addresses No. Full Name Title Duty Station Phone Number E-mail 1 Achuil Arop Outreach Coordinator Malakal 8821643332477 aarop@edc-ssiri.org 2 Ale Peter Micheal Outreach coordinator Yei 0477-122315 apeter@edc-ssiri.org/ ale_lomichael@yahoo.com 3 Amuda James Outreach Coordinator KajoKeji 882166790 0951 0477- 103225 amuda_james@yahoo.co.uk / jamuda@edc-ssiri.org 4 Angela Wangechi Receptionist-Nairobi Nairobi 0722-759776 awangechi@sudanradio.org 5 Angelo Guido Outreach Advisor Kauda 882165026-2502 tambua2003@yahoo.com / aguido@edc-ssiri.org 6 Angelo Ochan Abariamoi M & E Assistant Torit 0477-215900 abariamoi2000@yahoo.co.uk 7 Athanas Mwamba ICT Programs Coordinator Maridi 8821667900954, 0722- 826562 0477-153505 amwamba@edc-ssiri.org 8 Awori Emmanuel Festo Outreach Coordinator KajoKeji +256-782713199 aworiemmy@yahoo.com 9 Beatrice Anusu Custodian-Nairobi Nairobi 0734-342074 banusu@edc-ssiri.org 10 Bullen Nginzo Senior Outreach Advisor Wau 8821643341906 0733- 788281 0477-153502 bnginzo@yahoo.com / bmurangi@edc-ssiri.org 11 Charlton Doki Communication Specialist Nairobi cdoki@edc-ssiri.org 12 Chirillo Chol Outreach Coordinator Agok 8821643341905 cchol@edc-ssiri.org 13 Cho Bol Outreach Coordinator Panyagor 8821643341968 726821229 cbol@edc-ssiri.org 14 Cisella Alex Script Writer Nairobi 0722-719852 calex@edc-ssiri.org SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 127 No. Full Name Title Duty Station Phone Number E-mail 15 Daniel Kenyi Dima IT Officer Malakal dkenyi2003@yahoo.co.uk] 16 David Duku Script Writer Nairobi 0723-944780 dduku@edc-ssiri.org 17 Donald Thige Digital Editor Nairobi 0722-997400 dthige@edc-ssiri.org 18 Edward Kasran Senior Outreach Advisor Juba 8821643339903 0722- 072438 0477-153508 ekasran@yahoo.com/ ekasran@edc-ssiri.org 19 Esther Ndung'u Senior Admin & Human Resource Officer Nairobi 0722-203313 endungu@edc-ssiri.org 20 Evans Sebi Script Writer Nairobi 0724-079590 esebi@edc-ssiri.org 21 Evans Igu Tibi M & E Assistant Juba igutibi@yahoo.com 22 Francis Mwangi Digital Editor Nairobi 0722-409203 fmwangi@edc-ssiri.org 23 Grace Wambui Admin Assistant Nairobi 8821643331043 0733- 282592/ 0722-861838 gwambui@edc-ssiri.org 24 Irene Njeri Custodian Nairobi 0721-237735 injeri@edc-ssiri.org 25 Jane Namadi Production Advisor Nairobi 0721-555982 jnamadi@edc-ssiri.org 26 Janet Mola M&E Specialist Juba 882164334-1907 0733 716209 / 0720-216385, 0477-204057 jmola@edc-ssiri.org 27 Jesca Wude Script Writer Nairobi 0735-455306 Jwude@edc-ssiri.org 28 John Friday Amia IT Assistant Maridi 882166790 0935 jamia@edc-ssiri.org / amiaj109rf@yahoo.com 29 John Mark Opoka Outreach Coordinator Magwi 882166790 0008 0477- okolongoopoka@yahoo.com SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 128 No. Full Name Title Duty Station Phone Number E-mail 161256 / jopoka@edc-ssiri,org 30 Joseph Njama Ndirangu Project Driver Nairobi 0722-332762 jndirangu@edc-ssiri.org 31 Josephine Osike Script Writer Nairobi 0727-435931 josike@edc-ssiri.org 32 Josh Agukoh IT Coordinator Nairobi 0720-589838 jagukoh@edc-ssiri.org 33 Joyce Geri Terbia Script Writer Nairobi 0722-679010 jgeri@edc-ssiri.org 34 Khalil Said Morsal Outreach Coordinator Kauda 8821643336174 kuwa_khalil@yahoo.co.uk /kmorsal@edc-ssiri.org 35 Kpiboroano Joseph V. Outreach Coordinator Yambio 0477-198601 36 Ladu Mikaya Project Driver Juba 8821650262504 37 Maree Melican Deputy Chief of Party Juba 8821643331658 0721- 831561 / 0477-116990 mmelican@edc-ssiri.org 38 Martina Costa Outreach Coordinator Wau 8821643341967 mcosta@edc-ssiri.org 39 Mary Oyella Goretti M&E Assistant Juba 8821650262507, 0477- 153509 moyella@edc-ssiri.org 40 Mercy Karuri Senior Finance & Operations Administrator Nairobi 0733-747969 mkaruri@edc-ssiri.org 41 Mercy Kolok Scriptwriter Nairobi mkolok@edc-ssiri.org 42 Moses White Outreach Coordinator Mundri 882166790 0927 +249- 29436529 mwhite@edc-ssiri.org 43 Moses Ayiga IT Officer Juba 0477-133238 mosesayiga@yahoo.com 44 Mukwasa Marjan M & E Assistant Yambio +249-9129322246, 0477- mukwasamarjan@yahoo.co SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 129 No. Full Name Title Duty Station Phone Number E-mail 110305 m 45 Nicholas Lugalia General Clerk/Custodian Nairobi 0721-170625 nlugalia@edc-ssiri.org 46 Nicodemus Bior Script Writer Nairobi 0720-473870 nbior@edc-ssiri.org 47 Obang Okumbul Outreach Coordinator Pochalla 8821643341904 ookumbul@edc-ssiri.org 48 Onek John Orach Outreach Coordinator Juba 8821643336175, 047- 114189 onekakila@yahoo.com/oorac h@edc-ssiri.org 49 Ochan Richard Bongo M & E Assistant Jonglei ochanbongo@yahoo.co.uk 50 Phillip Yorka Jaraba Finance Officer Juba 882165026-2506 , + 249 912874466 , 0477-153506 pjaraba@edc-ssiri.org 51 Rejoice Binyi Outreach Advisor Juba 882166790-0929, 0 477- 121582, +249-121920429 kidenbinyi@yahoo.co.uk/ rbinyi@edc-ssiri.org 52 Remijio Joseph Jamba Outreach Advisor Bor +249-919440781 adropiako@yahoo.com 53 Reuben Ohisa Outreach Advisor Torit 882166790-0009 256 , 0477-142941 ohisa_reuben@yahoo.com / rohisa@edc-ssiri.org 54 Richard Aluma Administrative and Logistics Officer Juba 882165026-2505, + 249 913437984 , +249129083653, 0477- 153500 raluma@edc-ssiri.org 55 Robert Singira Outreach Advisor Yambio 8821667900933, 0477- 156804 singira_robert@yahoo.co.uk / rsingira@edc-ssiri.org SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 130 No. Full Name Title Duty Station Phone Number E-mail 56 Robert Wuda Outreach Coordinator Maridi 8821643341969 wudarobert@yahoo.co.uk / rwuda@edc-ssiri.org 57 Serah Kiden IT Officer Arapi 882164334-1966 skiden@edc-ssiri.org / skiden@yahoo.com 58 Simaya Kenyi Kodi M & E Assistant WBEG 59 Sheila Mweha Finance Coordinator Nairobi 0721-556240 smweha@edc-ssiri.org 60 Solomon Chike Outreach Coordinator Kurmuk 8821643341373, + 249- 924140639 solochike@yahoo.com / schike@edc-ssiri.org 61 Tom Tilson Chief of Party Nairobi 8821643336173, 0477103936, 0733-440036 ttilson@edc.org 62 Tiondi Partrick Uga Outreach Coordinator Magwi/Nim ule +256-714266428 0477-215900 tiondipartick@yahoo.com 63 Tito Achire Outreach Coordinator Ikotos lapatokok2000@yahoo.com 64 Victor Vasquez IRI Advisor Nairobi 0722-209622 vvasques@edc.org 65 Victor Tombe Lako Outreach Advisor￾Secondary Schools Juba +249-918275563 vlako@edc-ssiri.org, vtombe@hotmail.com 66 Wairimu Gachie Production assistant Nairobi 0722-825011 wgachie@edc-ssiri.org 67 Wani Stephen Scriptwriter Nairobi swani@edc-ssiri.org 68 Wesley Kibet Senior Project Accountant Nairobi 0721-206070 wkibet@edc-ssiri.org SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 131 (EDC) ANNEX IV - EDC-SSIRI ORGANISATIONAL CHART- SUDAN BASED STAFF EDUCATION DEVELOPMENT CENTER INC Deputy Chief of Party Maree T Melican ICT Programs Coordinator Athanas Mwamba Senior Outreach Advisor WES. CES, EES Jonglei, Upper Nile Edward Kasran Outreach Advisor Three Areas (Abyei, S. Kordofan, S. Blue Nile) Angelo Guido Z. Chief of Party Thomas Tilson Senior Outreach Advisor NBeG, WBeG Bullen Nginzo M & E Specialist Janet Mola Okoko Admin & HR Officer VACANT Outreach Advisor (Secondary Schs) Victor Tombe Finance Officer Phillip Yorka OA- W. Equatoria Robert Singira OC’s Robert W- Maridi Moses W – Mundri Joseph V -Yambio OA- C. Equatoria Rejoice Binyi M. OC’s John Onek-Juba Amuda S-Kajokeji Ale Peter M. - Yei OA- E. Equatoria Ohisa Reuben L. OC’s Opoka John-Magwi Tito Achire-Ikotos Tiondi Patrick - Magwi OA-Jonglei Remijio Joseph OC- Upper Nile Achuil Arop OC’s Chol Bol-Panyagor Obang O- Pochalla OC’s Martina Lino C.-Wau VACANT- Jur River VACANT -Raga OC’s VACANT- Aweil VACANT - Nyamllel OC’s Khalil Said Morsal-Kauda Chike Solomon-Kurmuk Chirillo Chol- Abyei M & E Assistants Mary Oyella-Juba Mukwasa M-Yambio Simaya Kenyi- WBeG Evans Igu Tibi-Juba Ochang R. B- Jonglei Angelo O.A- Torit VACANT-Three Areas VACANT-NBeG IT Officers Serah Kiden-Arapi Moses Ayiga-Juba Daniel D.-Malakal Beneth S.-Maridi IT Assistants Amia John F-Maridi Finance Assistant VACANT Logistics Officer Richard Aluma Project Drivers Ladu Mikaya SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 132 (EDC) Annex V – Draft Memorandum of Understanding Memorandum of Understanding Between The South Sudan Ministry of Education, Science and Technology (Ministry), State Ministry of Education (State) and Education Development Center, Inc. (EDC) I. Introduction Education Development Center, Inc. (“EDC”) has been awarded Award No. 623-A-00-04-00054-00 from the U.S. Agency for International Development (“USAID”) entitled Support to Education in Southern Sudan (“Prime Award” or "SSIRI"). The aim of this project is to improve the performance of children in literacy, English and numeracy in Southern Sudan. This Memorandum of Understanding ("MoU") is drawn up between EDC and the South Sudan Ministry of Education, Science and Technology (“Ministry” or “MoEST”) and the State Ministry of Education (“State”); there is no relationship between the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education and USAID under this MoU. II. Purpose of the Memorandum of Understanding The purpose of the MoU is to define clearly the roles of EDC and the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education. This MoU verifies the intention of all parties to collaborate with the ultimate aim of improving the performance of children, youth and adults in literacy, English and numeracy in South Sudan. Each party’s relationship to the other under this MoU is that of an independent contractor. Nothing in this MoU is intended to constitute a partnership, agency, employer, employee or joint venture relationship between the parties. Neither party may incur any debts or make any commitments for the other. III. Period of Performance of the Memorandum of Understanding The MoU describes specific areas of cooperation as they relate to EDC and the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education. This MoU will take effect on April 1, 2008, and will end on June 30, 2009. IV. MoU and Project Management The EDC Program Director is Kent Noel or his designee, Thomas Tilson, Chief of Party. The Program Director is duly authorized to act on behalf of EDC in all technical matters pertaining to the performance of the Program, except where specifically indicated otherwise in this MoU. The EDC Agreement Officer for this MoU is Robert Rotner, Senior Vice President and Treasurer or his designee, Kathryn Sarken, Assistant Director, Office of Sponsored Programs. The EDC Agreement Officer is ultimately responsible for all matters related to the administration of this MoU on behalf of EDC. The terms of this MoU cannot be changed without the approval of the EDC Agreement Officer. The South Sudan Ministry of Education, Science and Technology and the State Ministry of Education must direct all communications with USAID regarding this Agreement through EDC’s Agreement Office. V. Program Areas of Cooperation SSIRI is an integral program of the Ministry of Education, Science and Technology at the central, state, county and payam levels. The specific programs include the following inter-related radio-based programs: • The Learning Village for primary schools SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 133 • Teaching English through Radio-Based Instruction for All (Terbia) • Professional Studies for Teachers The Learning Village programs are implemented throughout South Sudan in primary schools. Terbia programs are implemented in secondary schools, AES learning centers, and other listening groups. The Professional Studies for Teachers program is part of the MoEST in-service teacher training program and is usually implemented through County Education Centers. VI. Roles and Responsibilities The parties agree that EDC will, in order to implement the SSIRI Project, require contributions of time, Ministry and State resources, collaboration and cooperation; therefore, EDC, the State and the Ministry agree as follows: 1. The South Sudan Ministry of Education, Science and Technology through the Department of Alternative Education Systems agrees to provide the following in support of SSIRI: • Provide leadership to the States on the development of strategies and plans for implementing SSIRI radio programs • Support the States on the development of an annual plan for SSIRI • Sponsor workshops with State officials on SSIRI activities • Provide support as may be necessary through the Curriculum Development Center in Maridi for the development of SSIRI programs • Provide access and support to national secondary schools, teacher training institutes, and County Education Centers as may be necessary for implementing SSIRI programs • Collaborate with EDC in providing data and statistics on schools and alternative education centers • Provide EDC outreach staff with information on school and alternative learning center calendar/s • Provide office space for 1-2 senior EDC staff in the MoEST compound • Pay salaries to AES/SSIRI teachers/facilitators within each State in accordance with MoEST policy • Identify partners to assist with program implementation • Provide ongoing monitoring of SSIRI activities at the State level The Ministry accepts full and sole responsibility for the payment of all costs associated with carrying out the above tasks. 2. The State Ministry of Education agrees to provide assistance with the implementation the SSIRI programs as follows: For the Learning Village • Provide information on the location of all schools and Community Girls’ Schools in the state • Report on the number of teachers by grade level in the state For Terbia Report on the potential number of Terbia groups in the following centers: • ALP • IELC • Adult Literacy SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 134 • Other groups such as women, youth, church, etc. For Professional Studies for Teachers • Identify County Education Centers (CECs) implementing the in-service teacher education program In order to support the implementation of the SSIRI programs, the State agrees to contribute and carry out the following: • Develop a State Annual Plan for SSIRI activities • Identify and arrange for appropriate State education officials (Alternative Education Systems, primary and secondary, and teacher education ) to participate in SSIRI orientation, planning, and Training of Trainers workshops • Provide TOT trainers to co-facilitate with EDC outreach staff in the training of facilitators for Terbia and Head Teachers and teachers for the Learning Village • Provide regular, ongoing monitoring of SSIRI activities in each county/payam to ensure effective implementation • Provide a report in writing to EDC on a quarterly basis with information regarding the progress of the radio programs including updated enrollment figures, successes, challenges or difficulties that they have experienced, and data on any recruitment of more schools or Terbia groups • Prepare and submit reports to EDC/SSIRI on all SSIRI activities including training workshops, monitoring reports, etc. • Provide leadership to the county education officers on the development of plans for implementing SSIRI programs. The County plans should include: • The names and locations of participating schools, listening groups, and groups of teachers for Professional Studies for Teachers • Training activities • Process for distributing radios and guides • Monitoring process • Collaborate with EDC in providing school and alternative education center data and statistics • Provide EDC outreach staff with information on school and alternative learning center calendar/s • Identify appropriate County education officials (primary, secondary, AES inspectors/field education officers, and CEC trainers) to participate in SSIRI orientation, planning, and training of trainers workshops • In collaboration with EDC outreach staff, plan and facilitate workshops to train facilitators for Terbia, Head Teachers and teachers for the Learning Village, and CEC officials and local instructions for Professional Studies for Teachers • In collaboration with EDC Outreach staff and payam officials, County officers will conduct regular, ongoing monitoring of SSIRI activities in each county to ensure effective implementation • Provide a space for EDC outreach staff in the State education office and, if possible, in the county offices as well The State accepts full and sole responsibility for the payment of all costs associated with carrying out the above tasks. 3. Education Development Center, Inc. agrees to provide technical assistance, equipment and materials. Specifically EDC agrees to: SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 135 • Provide the following personnel per State: Outreach Advisor at the State capital level (some States may share an Advisor), Outreach Coordinator(s) at the county level, and M&E Assistant at the State level (some States may share an M&E Assistant). • Provide radios: Radios for each participating school equal to the number of grade 1 and grade 3 streams, a radio for each AES learning group, and radios or an MP3 device for teachers in the Professional Studies for Teachers program • In some circumstance, EDC may provide alternative technologies in the place of radios such as a MP3 player with speakers • Provide teacher's and facilitator’s guides: One guide for each teacher or facilitator. • Orientate GoSS, State and County education officials to SSIRI • Work with education officials to identify schools and P1, 2, 3 and 4 teachers • Work with education officials to identify Terbia listening groups and facilitators • Work with education officials to identify teachers for the Professional Studies for Teachers program • Train education officials, Head Teachers and P1 to 4 teachers on the Learning Village • Provide implementation guidelines for each of the Terbia programs • Provide Learning Village and Terbia monitoring forms • Provide ongoing technical support for CEO and Learning Village schools and teachers • Provide ongoing technical support for LEO (Locality Education Officer) and Terbia facilitators. • In collaboration with State and county officials, EDC will conduct TOT training on SSIRI radio programs • EDC will provide for the broadcast of SSIRI programs on shortwave and, to a more limited extent, on FM stations • EDC may also provide on occasion promotional materials such as a brochure, flyer or banner • Provide some financial assistance for conducting workshops, possibly including supplies, tea breaks, meals, and travel, accommodations and per diem for teachers having to travel from another location. Such payments will be consistent with SMoE regulations, guidelines, and practice as well as with EDC and USAID regulations. • Provide a bicycle for each participating payam. The payams agree to use the bicycles for travel required to support the implementation of SSIRI activities. VII. Payment No payment will be made by EDC to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education for the work performed by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education under this MoU. No payment will be made by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education to EDC for the work performed by EDC under this MoU. VIII. Ownership of Work Product Work Product shall mean all data, reports, curricula, specifications, outlines, drafts, software, videotapes and any other materials or deliverables, in any medium, which are prepared and/or developed by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education in the performance of its obligations under this MoU. Subject to the terms of this MoU, all intellectual property rights in the Work Product prepared and/or developed by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education hereunder (the “Work Product”) are the sole and exclusive property of the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education. Notwithstanding the foregoing, in order to permit EDC full and effective use of the Work Product as contemplated by this MoU, the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education hereby grants EDC a non-exclusive, irrevocable, worldwide, fully paid-up, royalty-free SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 136 license (the “EDC License”) to the Work Product, with the right to sublicense, make, use, reproduce, distribute, display, perform, transmit, improve, and create derivative works based upon such Work Product, in any manner and in any medium by any means now known or hereinafter invented, for purposes of the Program and in other ways consistent with EDC’s educational mission including development of commercial applications of the Work Product. EDC will include in the Work Product that are so used an acknowledgement of the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education as the developer and owner of copyright of the Work Product, using substantially the same form of acknowledgement as is included by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education in the Work Product as delivered to EDC. If the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education intends to use or incorporate any work owned by third parties, images of people or entities into the Work Product, the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education must first, (a) identify the work to be included in writing to EDC, (b) identify in writing any limitations on use of the work and (c) obtain all rights in such work necessary for EDC to exercise all rights to the Work Product arising from this MoU. IX. Amendments The MoU may only be amended by formal written modification. Amendments may be initiated by EDC to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education or by the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education to the EDC Agreement Officer. Amendments shall not be binding on either party until the amendment has been signed by the authorized representatives of both parties. X. Termination and Suspension 1.EDC may terminate the MoU at any time, in whole and in part, upon written notice to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education, whenever it is determined that the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education has materially failed to comply with the terms and conditions of the MoU and has failed to cure said breach within thirty (30) days of notification. 2.The South Sudan Ministry of Education, Science and Technology and the State Ministry of Education may terminate the MoU at any time, in whole and in part, upon written notice to EDC, whenever it is determined that EDC has materially failed to comply with the terms and conditions of the MoU and has failed to cure said breach within thirty (30) days of notification. 3.This MoU may be terminated at any time, in whole or in part, by mutual written consent of the parties. Both parties shall agree upon termination conditions, including the effective date, and, in the case of partial terminations, the portion of the MoU to be terminated. 4.If at any time USAID determines that continuation of all or part of the funding for the Prime Award should be suspended or terminated, then EDC may, following notice to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education, suspend or terminate the MoU in whole or in part. If the Program is suspended and the situation causing the suspension continues for sixty (60) days or more, then EDC may terminate this MoU in whole or in part upon written notice to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education. XI. Executive Order 13224 The South Sudan Ministry of Education, Science and Technology and the State Ministry of Education shall comply with Executive Order 13224 and U.S. laws that prohibit transactions with, and the provision of SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 137 resources and support to, individuals or organizations associated with terrorism. It is the legal responsibility of the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education to ensure compliance with these Executive Orders and laws. XII. Limitation It is expressly understood that neither EDC nor USAID has any obligation to provide any support to the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education in addition to that specified in this agreement. It is further understood that the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education does not have any obligation to provide any support to EDC in addition to that specified in this agreement. XIII. Indemnification The South Sudan Ministry of Education, Science and Technology and the State Ministry of Education agrees to defend, indemnify, and hold harmless EDC and USAID, its trustees, directors, officers, employees and agents, and their respective successors, heirs and assigns from and against any liability, damage, loss, cost, fee or expense (including reasonable fees of attorneys and other testifying or consulting professionals, and expenses of litigation) incurred by or imposed upon EDC or any one or more of them in connection with any allegations, claims, suits, actions, demands, whether threatened or pending, or judgments arising from or relating to the performance of this MoU, to the extent such liability, damage, loss, or expense is caused by or results solely from the negligent or intentional acts or omissions of the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education, its directors, officers, employees or agents. EDC agrees to defend, indemnify, and hold harmless the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education, its trustees, directors, officers, employees and agents, and their respective successors, heirs and assigns (the “South Sudan Ministry of Education, Science and Technology Indemnitees” and the “State Ministry of Education Indemnitees”) from and against any liability, damage, loss, cost, fee or expense (including reasonable fees of attorneys and other testifying or consulting professionals, and expenses of litigation) incurred by or imposed upon the South Sudan Ministry of Education, Science and Technology Indemnitees and the State Ministry of Education Indemnitees or any one or more of them in connection with any allegations, claims, suits, actions, demands, whether threatened or pending, or judgments arising from or relating to the performance of this MoU, to the extent such liability, damage, loss, or expense (i) is solely caused by or results solely from the negligent or intentional acts or omissions of EDC, its trustees, directors, officers, employees or agents, or (ii) arises from the breach of any other warranty or representation made by EDC. XIV. Liability Neither EDC nor USAID assume liability for any third party claims for damages arising out of this MoU. XV. Disputes Any dispute or disagreement which cannot be resolved by EDC and the South Sudan Ministry of Education, Science and Technology and the State Ministry of Education and any controversy, claim, or dispute otherwise arising out of or in connection with this MoU or breach thereof, or the Program, shall be resolved by arbitration. XVI. Authority; Binding Effect By his or her signature below, each signatory hereto represents and warrants that he or she is duly authorized to enter this MoU on behalf of the party he or she purports to represent such that, upon execution and delivery, this MoU shall be a binding obligation of such party. SOUTHERN SUDAN INTERACTIVE RADIO INSTRUCTION (SSIRI) PROGRAM MID-TERM EVALUATION 138 This MoU constitutes the entire agreement among the Ministry of Education, Science and Technology, the State Ministry of Education, and EDC with respect to the subject matter hereof and supersedes all prior agreements and understandings, whether oral or written, between us relating to the subject matter. The work and services required herein shall not be transferred or assigned by the South Sudan Ministry of Education, Science and Technology or the State Ministry of Education without the prior written consent of EDC. In witness thereof the South Sudan Ministry of Education, Technology and Science, State Ministry of Education and Education Development Center, Inc. The South Sudan Ministry of Education, Technology and Science Name: Title: Date: The South Sudan State Ministry of Education Name: Title: Date: Education Development Center, Inc. Name: Robert Rotner Title: Senior Vice President and Treasurer Date: