1 BASIC EDUCATION STRATEGIC OBJECTIVE (BESO) II – BASIC EDUCATION PROGRAM FINAL EVALUATION JANUARY 2007 This publication was produced for review by the United States Agency for International Development. It was prepared by Abebe Brehance, Samuel Taddesse, Gebeyehu Woldaregay, Joyce Wolf and Management Systems International (MSI). Management Systems International 600 Water Street, SW Washington, DC 20024 BASIC EDUCATION STRATEGIC OBJECTIVE (BESO) II – Basic Education Program FINAL EVALUATION Contract under No. GEW-I-00-02-00021-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. BESO II – FINAL EVALUATION i CONTENTS ACRONYMS....................................................................................................... H IIH EXECUTIVE SUMMARY .............................................................................. H III H I. INTRODUCTION............................................................................................ H 1H A. Background.......................................................................................... H 1H B. Purpose of Evaluation.......................................................................... H 2H C. Methodology........................................................................................ H 2H II. FINDINGS AND CONCLUSIONS............................................................... H 3H A. Pre-Service Teacher Education............................................................ H 3H B. In-Service Teacher Education............................................................ H 10H 3. recommendations............................................................................... H 18H HC. Supplementary Media and Reading Materials................................... 19H D. Strengthening Leadership H and Planning and Management Capacity ........................................................................................ 24H HE. Support for Female Teachers ............................................................. 35H HIII. AED PROJECT MANAGEMENT ........................................................... 41H HA. BESO II-BEP Program Management ................................................ 41H HB. Monitoring, Evaluation, Reporting and Analysis .............................. 43H HIV. HOST GOVERNMENT SATISFACTION .............................................. 44H HV. LESSONS LEARNED.................................................................................. 47H HA. Major Constraints on Project Implementation................................... 47H HB. Impact on Project Implementation and Project Response ................. 48H HC. Sustainability of Project Achievements............................................. 48H E. Linkages between BESO-BEP and the Community H Government Partnership Program................................................. 49H ANNEXES HAnnex A: List of Individuals Interviewed.......................................................... 51H HAnnex B: Data Collection Form......................................................................... 55H HAnnex C: Schools Visited .................................................................................. 70H HAnnex D: Scope of Work ................................................................................... 72H HAnnex E: Project Interventions .......................................................................... 75H BESO II – FINAL EVALUATION ii ACRONYMS AED Academy for Educational Development AIR American Institutes of Research BESO Basic Education Strategic Objective BEP Basic Education Program CGPP Community Government Partnership Program COP Chief of Party CRC Cluster Resource Center EMA Educational Media Agency EMIS Education Management Information System ESDP Education Sector Development Program GFDRE Government of the Federal Republic of Ethiopia ICDR Institute for Curriculum Development & Research IRC Instructional Resource Center IRI Interactive Radio Instruction ISP Integrated Strategic Plan KETB Kebeles Education and Training Boards MERA Monitoring, Evaluation, Reporting & Analysis MMIS Materials Management Information System MOE Ministry of Education NLA National Learning Assessment NOE National Organization for Examinations PC Pedagogical Center PMIS Personnel Management Information System PMP Performance Monitoring Plan REB Regional Education Bureau PTA Parent Teacher Association SDU Staff Development Unit SNNPR Southern Nations, Nationalities, and Peoples Region SO Strategic Objective TALLULAR Teaching and Learning Using Locally Available Materials TDA Tigray Development Association TEI Teacher Training College TESO Teacher Education System Overhaul TOT Training of Trainers TTI Teacher Training Institute TWG Technical Working Group USAID United States Agency for International Development WEO Woreda Eduction Office BESO II – FINAL EVALUATION iii EXECUTIVE SUMMARY The purpose of this evaluation is to create a comprehensive review of the Basic Education Strategic Objective (BESO) II – Basic Education Project (BEP) funded by USAID and implemented by the Academy for Educational Development (AED) and the American Institutes of Research (AIR). Its goal is to determine the successes of BESO-BEP interventions, examine their impacts in a sub-set of five regions, identify and analyze implementation issues which have occurred, and assess the effectiveness of AED project management. The findings and recommendations of the evaluation will be used by UAID/Ethiopia and the Government of Federal Democratic Republic of Ethiopia to determine future strategies for the enhancement of quality and equity in education. PRE-SERVICE TEACHER EDUCATION The primary purpose of this component of BESO-BEP is to enhance the quantity and quality of primary education professionals and teacher graduates and to contribute to the application of active learning, continuous assessment, large class management, and action research in primary schools. The project has met or exceeded several of the associated targets and deliverables for the component of the project, but it has also failed to meet others. The majority of BESO-BEP pre-service activities has targeted Teacher Education Institutes (TEIs) where faculty and administrative staff have received training in active learning, continuous assessment, large class management, and action research. In addition, instructors have received training in how to train other trainers in order to prepare them for the role they are planned to play in providing in-service teacher training in primary schools linked to the TEI. Additional training in a wide range of skills has also been provided to selected administrative staff and faculty in an attempt to strengthen capacity at TEIs. An obstacle to achieving an improved capacity is presented by the high turn over in personnel and the extremely heavy workloads of most instructors. Both of these constraints are beyond the scope of what AED can influence, but they further define the context in which the AED interventions were expected to have an impact. Based on visits to six TEIs, utilization of the libraries, computer centers, and pedagogical centers located at TEIs, which have been revitalized and supported by BESO-BEP, is high, but there are maintenance and staffing problems in several of the centers at these TEIs. One of the difficulties of the BESO-BEP program is that it can provide resources and funding, but it cannot dictate how they will be used. Part of sustainability is the ownership which grows out of making the decisions about resources and funding, but those decisions can have an impact on the utilization of the resources which have been provided. BESO-BEP has established and funded Cluster Coordination Units at all TEIs to accelerate the spread of active learning, continuous assessment, large classroom management, and action research to clusters of 20 to 100 primary schools in the areas of the six TEIs which were visited. The links to these schools consist of resources supplied to the center schools in the clusters, use of the schools for student-teachers’ practicum experiences, establishment of model classrooms in some of the schools, and in-service training of principals, key teachers, and supervisors by instructors from the TEI. The cluster coordinators are working to develop strategies to involve TEI instructors in in-service training as some form of incentive is necessary for the sustainability of the training. The sustainability of the links between TEIs and primary schools that have been created or intensified by BESO-BEP funding are important to the sustainability of all in-service teacher support. If each TEI can provide the same training and support to a selection of primary school clusters in its area that BESO-BEP has to its selection of clusters, then the model established by BESO-BEP can be expanded to reach the remainder of the primary schools of Ethiopia. BESO II – FINAL EVALUATION iv Recommendations: • Given the heavy workload of most of the instructors who were interviewed, some incentive needs to be provided if these instructors are expected to take on the additional tasks of in-service teacher training and support for female students. Within the Ethiopian context, it is unrealistic to expect the TEIs to offer financial incentives for these activities. The most realistic use of incentives for these purposes that the evaluation team encountered were found at the Debre Berhan TEI, where instructors received credits for their in-service efforts. These credits could then be used to support the instructors’ career advancement. However, as in the case of supplying adequate staff or scheduling courses to utilize resources supplied by BESO-BEP, the project has no power to force TEI management to adopt approaches such as this. • The need for support in life skills for female students has intensified since BESO-BEP began because of the cancellation of hostels for students at TEIs, yet the materials supplied by BESO￾BEP and the gender offices at TEIs facilitated by BESO-BEP appear to be underutilized due to lack of staff time. Again, some form of affordable incentive is needed. • In order to consolidate the gains of past BESO-BEP investment in preparing and strengthening the active learning, continuous assessment, large classroom management and action research skills of cycle 1 teachers, support needs to be extended to the private sector Teacher Training Institutes. IN-SERVICE TEACHER EDUCATION This component of BESO-BEP is designed to improve the skills of teachers to apply active learning student-centered, continuous student assessment, large class management, and action research methodologies, supply education materials and equipment to schools, and facilitate the exchange of experiences and collective approaches to solving educational problems among school principals and teachers. The model for in-service teacher training in the new curriculum that has been implemented in the BESO￾BEP schools is built upon a cluster structure of schools, in which BESO-BEP resources remain in the Center school for all of the cluster schools to use and principals and key teachers are expected to pass on the instruction and materials they have received from BESO-BEP to the other teachers in the cluster. Teachers are trained through a cascade pattern in active learning, continuous assessment, management of large classes, and active research. In all of the schools visited, the resources supplied by BESO-BEP have been maintained by the cluster schools, some amount of in-service training has occurred within the cluster, and self-instruction materials have been utilized by the teachers to varying degrees. Principals and teachers report impressive changes in their attitudes, behaviors, and skills. As in any cascade system, there is some question about the loss of content, but, even if not fully implemented, there appears to be a better understanding of the key principles and the use of them that has reached most of the classrooms in the BESO-BEP supported schools. Teachers report that active learning approaches have increased the self-confidence, team work and competencies of primary education students. A major obstacle to maintaining and/or expanding this model of in-service teacher training lies in the problem created by frequent turnover in personnel. The loss of a trained person within the cascade context means more than just the loss of that individual; it creates a break in the chain of training. In four schools in which the key teacher had left, all teacher training and study of the self-instructional kits had cceased. This is not the responsibility of the project, but it is the reality within which the project has had to operate. To compensate for the turnover in personnel, BESO-BEP has had to continuously train new BESO II – FINAL EVALUATION v persons in the same subjects rather than being free to devote more effort to follow-up of the training received. The planned sustainability and expansion of this model of in-service teacher training depends upon the degree to which the TEIs and woredas can become the trainers for the education system. The linkages with primary school clusters, which have been established at most TEIs, have resulted in training and monitoring activities at all linked schools that were visited. Training by woreda personnel had taken place at two of the nine non-BESO schools that were visited. In addition to training, woredas and TEI personnel have been involved in follow-up activities to ensure the quality of the training and skills transferred through the cascade model; TEIs were doing so at linked schools, but woredas had focused most of their follow-up activities on BESO-BEP schools. The Community Government Partnership Program (CGPP) focuses on developing the relationship between communities and primary schools, which can inspire the community to provide increased support to schools. The evaluation team found that the schools in which both BESO-BEP and CGPP have operated have better community-school relationships and appear to be more highly energized in general than the sum of the two programs would suggest. This synergy may be the result of combining a top￾down training approach with a bottom-up empowerment approach. Recommendations: • Rather than BESO-BEP repeatedly training the same persons in each school, generally the principal and the key teachers, four teachers suggested that a better strategy would be to train a variety of teachers at the school, making each an expert on a different area of the curriculum. This would help to combat the turnover found in many of the schools, diversify the persons giving training at the school level, and prevent investing most of the success of the program in a few individuals who may or may not be up to the task. • Because current follow up is insufficient to determine how great the problems of transfer in information through the cascade model are, research into the relative differences in understanding of active learning, continuous assessment, large class management, and action research between Center and Satellite schools could be conducted for a better understanding of this problem. • Certification for teachers on each self-instructional module upon which they have received in￾service training and discussed the related materials in a study group could increase commitment and provide a standard for assuring that teachers are familiar with active learning, continuous assessment, large class management, and action research. • Large class size often prevents adoption of active learning and continuous assessment methodologies. Some attempts are being made to reduce this problem: the Community Government Partnership Program and other community focused programs have lead to community involvement in building a sufficient number of classrooms at most schools and the MOE is currently conducting research into the possible underutilization of teachers at many schools. A creative search for other solutions needs to be conducted. SUPPLEMENTARY MEDIA AND READING MATERIALS It is clear that there is a strong need for supplemental media and reading materials in Ethiopian schools. In all of the schools visited, appreciation for the BESO-BEP materials was strongly stated and the presence of more reading materials was one of the major differences between BESO-BEP and non-BESO schools. While the evaluation team found variation among schools in which media and reading materials were used and how assessable they were to teachers, at least some of the materials were being used at BESO II – FINAL EVALUATION vi every school. Where the materials were stored and the degree to which teachers had been informed about their presence also influenced utilization. Primarily, use of the materials supplied by BESO-BEP tends to depend upon how relevant they are to the needs of the teachers. For example, if the materials match the subject that a teacher teaches, then they became important enough to read and use. If English is being taught, then participatory radio approaches make that instruction easier and more enjoyable for both teachers and students. The supplementary materials alone cannot create mastery of the skills and knowledge presented but, when the materials are linked to in-service training on the same topics, and especially when combined with continuing study of the materials by teacher sub-groups, then the materials appear to be effective in increasing teacher skills in the areas being addressed. Getting the supplementary materials into the hands of the teachers who can best make use of them can be an obstacle. In order to expand this program to additional schools, more than just publication will have to be considered because distribution, linkage to training programs, and storage can all produce barriers to utilization. Woreda education officials would be the logical choice for distribution and, although the experience with woreda distribution of BESO-BEP materials has not always been good, almost half of the woredas that the evaluation team had data about did distribute BESO-BEP reading materials to schools in their areas. Recommendations: • The Center of Excellence publishing facility at the Debre Berhan TEI offers a successful model for how the supply of reading materials could be expanded through local production. The creation of one center for publication in each region would not only increase the amount of material, but also increase interest in local authorship as it has at the Debre Behan TEI. • Guidance in who should receive each type of reading material and where it should be stored after reaching the schools might increase utilization. STRENGTHENING LEADERSHIP, PLANNING, AND MANAGEMENT CAPACITY Over the last few years, decentralization of the education system in Ethiopia has deepened. Responsibilities and resources for managing basic education have devolved from the regional state government to woredas and kebeles. Such deepening of education system decentralization requires massive capacity building for effective planning and management of the education system at all levels. The BESO-BEP project has launched various activities that are aimed at strengthening and increasing (a) leadership skills, (b) education planning and management capacities; and (c) school leadership and supervision. Project contract deliverables are focused on the provision of key management tools and the skills and capacity to use them. Although progress on contract deliverables for this component has been made, the project has missed most of its output targets. BESO-BEP has introduced new planning and management tools for education personnel management, education resources management, and education planning. It has trained cadres of educational officers, planners and leaders at all levels to effectively manage the education system. However, high turnover throughout the education system, especially among woredas heads and deputies, has meant that there are still many key positions filled by persons without training. High turnover in regional and woreda offices has limited the effectiveness of BESO-BEP planning and management capacity development efforts. The high personnel turnover has prevented the creation of a critical mass of trained information technology and educational officers to maintain, operate, and use the computerized management information systems. BESO II – FINAL EVALUATION vii Access to education data remains a major problem even when computer systems have been supplied and persons trained on the database. In many cases, the evaluation team found the person with the training to retrieve the requested data was not present or had moved to another position; if the person in charge is absent, the system stops functioning until his or her return. Leadership training at the woreda and school levels has improved the supervision and management of schools and increased the participation of community members. School principals report that they have used their training constantly in their preparation of school plans, that it has helped them to identify and prioritize school problems, and that they have changed their management style to become more participatory. Improvement in school leadership and supervision has meant improved lesson planning and increased application of active learning and student-centered teaching. It has also meant increased use of continuous assessment and remediation to increase student competencies. Training of community leaders has led to increased support for the school and an increased understanding of active learning. Recommendations: • In the remaining life of the project, more attention is needed to fill the gap created by the high education personnel turnover as well as the increase in the number of woredas in each region. • More central and regional information technology professionals should be trained in the technical features and management skills of the planning and management tools developed by BESO-BEP. The possibility of opening up the training of information technology to the private sector should be explored as a means to expand the pool of trained information technology professionals in the country. • The project should continue to train a critical mass of trainers of trainers for woreda and school training programs. In addition, more regional experts should be trained in how to manage school and woreda programs. SUPPORT FOR FEMALE TEACHERS The purpose of this component of BESO-BEP is to enhance the capabilities of female teachers and promote their participation in educational leadership. Historically Ethiopian culture has assigned to women a subordinate role, which has frequently led to over-burdened and difficult lives, little educational achievement and a relative lack of self-confidence. BESO-BEP has produced modules on gender issues which have an impact in an educational context such as how to promote gender sensitive classrooms, enhance female teachers participation in school and community activities, and a self management guide for female primary school teachers. The project has also provided community training in awareness of negative cultural practices such as rape, female abductions and early marriage. In addition, leadership training has been offered to female teachers who are perceived as being ready and eligible to be promoted to leadership positions. The leadership training BESO-BEP offered in support of female teachers is well conceived and appears to be fully appreciated by those who have participated. There is a reccurring suggestion from both female teachers and woreda officers that substantially more female teachers should receive female leadership training, both to increase the number of female teachers who are willing to take on leadership roles and to further integrate the concepts into the education system. However, at the present levels of training, and without some motivation for sharing information within schools and/or clusters, substantial changes in attitude and behavior cannot be expected. BESO II – FINAL EVALUATION viii The gender modules supplied by BESO-BEP are valued by those teachers who have access to them, but often do not reach the people who could make the most use of them, such as female student councilors and leaders of Girls’ Clubs. Without being integrated into school activities, such as organized study groups that focus on the modules or discussion of the materials as part of Girls’ Club presentations, the modules often end up sitting on library shelves. 3. RECOMMENDATIONS Recommendations: • The current leadership training focuses on the most successful of the female teachers in a school, generally 2nd cycle teachers, and has as its goal preparing them for career advancement. The development of course materials on gender issues for 1st cycle teachers, where most female teachers are found, would equip them with information both for their own career advancement and to increase understanding that they can pass on to their students, male and female. This material could be distributed to the new private teacher training facilities in the hopes that it would be introduced into the curriculum. • One limitation of the approaches implemented by BESO-BEP is that primarily women were offered training focused on the problems of female achievement and modules on gender issues. Because men are in charge of most school institutions, women cannot create deep change without the assistance of men. The various types of training offered to male regional, woreda, principal, and kebele personnel should contain some of the same material offered to female teachers. Male teachers also need to know how to create a gender sensitive classroom. • Any future supports for female student-teachers at TEIs will have to address a major change which occurred when the Ethiopian government abolished dormitory living at TEIs. Female student now, more than ever, need assistance in developing skills for the management of their lives, which BESO-BEP has developed materials to support. Making students more aware of the existence of these materials and increasing the amount of counseling available for female students would help to offset the difficulties created by living on a small financial allotment in a new and unknown community. AED MANAGEMENT According to the Chief of Party, Technical Working Group Chairperson and the USAID CTO, BESO II faced serious management problems in the initial base years. Interviews by the evaluation team have reported that things began to change when Dr. Ernest O’Neil was brought in as Chief of Party in August 2004. BESO-BEP is now adequately staffed and budgeted to execute contract deliverables. The evaluation team found that the program has adopted appropriate management and operational procedures, that its staff worked well together, and the relationships between the project personnel and the MOE, regional bureaus, woredas, and TEIs are generally strong and collegial. AED’s Monitoring, Evaluation, Research and Analysis (MERA) system is functioning well and has delivered all its contract deliverables. MERA has enhanced the policy research capabilities of education personnel in the MOE and REBs. BESO II – FINAL EVALUATION ix LESSONS LEARNED Impact of Constraints on Project Implementation and Project Response The only major change between BESO I and BESO-BEP is that, while BESO I focused on two regions and the Federal Ministry of Education, BESO-BEP covers all regions and the Federal Ministry of Education. At the same time, the funding for educational reforms was dropped from US$ 80 million in BESO I to US$30 million in BESO-BEP.BESO-BEP has dealt with the extremely large geographic area and number of schools and educational offices and limited resources through two strategies. The first involved only attempting to directly supply interventions to a limited number of schools, creating centers in which multiple resources interacted to create models, rather than attempting to spread its resources thinly over all schools. For example, in the five regions visited by the evaluation team the percentages of schools receiving direct support from BESO-BEP were low. Region Number of Schools No. of BRP Schools Percent of All Schools Supported by BEP Afar 205 28 13.7% Amhara 5061 335 6.6% Benishangul Gumuz 314 56 17.8% Oromia 6466 440 6.8% SNNPR 2866 193 6.7% The second strategy involved providing much of its support to the institutions which managed, supported, and trained this vast array of schools and teachers. This served both to strengthen the overall system and build in an increased level of sustainability. The political crisis in 2005, which created great turnover in staff, reduced the number of professionals who had been trained by BEP who were in the key positions to implement the project. This meant that that a massive amount of training of new education professionals Sustainability of Project Achievements The most sustainable achievements of BESO-BEP appear to be: • The implementation of the new curriculum on all levels of the education system. The MOE reports this as the most sustainable achievement of the project and AED staff report a growing awareness of the usefulness of the student-centered methodologies. • The success of the first and second National Learning Assessments have created a demand for this data and the expertise gained by the educational personnel who conducted the second Assessment will make this a sustainable interventions. • The demonstration of a model for the delivery of in-service training, resources, and supplementary materials to schools through the use of the cluster structure. However, this model will only be sustainable if incentives for TEI instructors and woredas education officers to continue training are offered. • Production of instructional materials through the approach found in the Debre Berhan Center of Excellence. The success of this publishing facility could be replicated at other TEIs in other regions. • The revitalization of pedagogical centers at TEIs and schools. The enthusiasm for and utilization of these centers was apparent to the evaluation team. BESO II – FINAL EVALUATION x In addition, according to AED BESO-BEP staff, there has recently been a gradual increase in awareness regarding the usefulness of the technology assisted management information systems. Project Impact on Educational Policy BESO-BEP has conducted research into policy change issues and produced a number of policy research reports. The project has printed and widely distributed the resulting research papers and discussed these reports at various forums, including a series of workshops. The BESO-BEP goal has been to inform policy decisions in the MOE, regions, and woredas. Although the project has attempted to provide information to guide policy decisions, it primarily sees its role as the implementation of policies created by the central and/or regional governments. Linkages between BESO-BEP and the Community Government Partnership Program The Community Government Partnership Program (CGPP) focuses on developing the relationship between communities and primary schools. The program helps communities understand what they can contribute to the educations of their children and helps schools understand how they depend on the community to help them with this task. Communities can monitor student attendance and behavior, contribute funds and labor for classrooms and latrines, and work with the school to create income generating projects. The CGPP also works to provide community education on social issues, such as early marriage and HIV/AIDs. The evaluation team had the consistent impression that the schools in which both BESO-BEP and CGPP have operated are making greater progress than simply the sum of the two programs would suggest. This synergy may be the result of combining a top-down training approach with a bottom-up empowerment approach. World Learning, Save the Children USA, CARE, and the Tigray Development Association (TDA) all implement the CGPP in various regions. They all participate in the Technical Working Group quarterly meetings, where they present progress reports and plan joint activities. The bESO-BEP and CGPP have coordinated their activities through joint training activities, logistical support, and development of training manuals. Currently there is an attempt to increase the number of schools supported by both BESO-BEP and the CGPP to 80%. Given the evidence that the evaluation team witnessed of increased impact at schools where these two programs overlapped, this is a very positive direction. Project Design The design of BESO-BEP is extremely broad in the geographic and educational spread defined as the range of the project. The time allowed within which to address this large population of teachers and number of institutions has been extremely short. These factors have made the project unwieldy to implement and made it difficult to achieve sustainable impacts. The design has, however, allowed for experimentation in various ways of achieving the changes desired. This has led to the development of several successful models, the management and implementation of which can now be transferred to the education system. Some of the most promising interventions are: • The two Centers of Excellence that were visited demonstrated very successful models for increasing the capacity of the educational system to produce teaching materials and tools. The difficulty in expanding this approach lies in the inability to directly influence the TEI staffing and curriculum decisions necessary for the support of the Centers. • The model of a cluster structure and cascade in-service teacher training, supported by self￾instructional materials has demonstrated an effective way to support improved teaching in a large number of schools. The first step toward expanding the use of that model to all Ethiopian schools BESO II – FINAL EVALUATION xi has been taken by the regions through the creation of school clusters associated with TEIs. The second step has been started by BESO-BEP through the creation of cluster coordination centers at TEIs. The TEIs need to complete that step by finding incentives for their instructors who can provide in-service training and monitoring at those clusters linked to the TEI. The third step would be to duplicate the publishing Center of Excellence at other TEIs in all regions in order to take over preparation of materials from BESO-BEP and increase the production of self￾instruction kits. • There is a great need for supplementary reading materials in Ethiopian schools. The subjects about which to produce materials that were selected by BESO-BEP are appropriate and needed, with the possible exception of the HIV/AIDs materials due to the increasing number of other organizations supplying materials on this issue. While there is a strong need for these materials, they often are not utilized because they do not reach the specific teachers who will use them. Some method needs to be found for increasing awareness of the existence of these materials and making sure that the materials are directed to the right person. • Capacity building for KETBs has been very successful in increasing support for schools in ways where communities can contribute, such as monitoring the attendance and behavior of students and/or increasing resources for the school. The evaluation team found that the schools in which both BESO-BEP and CGPP have operated have better community-school relationships and appear to be more highly energized in general than the sum of the two programs would suggest. Both of these impacts indicate the importance of including communities in educational interventions and making them partners with school personnel in the task of educating their children. • In all five regions visited by the evaluation team, the team was informed that affirmative action for the hiring and promotion of female teachers and the increased admission of female students at TEIs were driven by BESO-BEP gender awareness activities. The heightened awareness of gender issues in schools, TEIs, and educational management is a more significant impact than any of the specific interventions in this component. That impact could be further expanded and deepened by not focusing gender training and materials only on females, but by covering the same issues in all pre- and in-service teacher training for both men and women. BESO II – FINAL EVALUATION 1 I. INTRODUCTION A. BACKGROUND The Basic Education System Overhaul (BESO I) was a seven year (1994-2001) cooperative effort for the Ethiopian Government and USAID to improve the quality and gender equity of primary education in Ethiopia. The Academy for Educational Development (AED) was one of the implementers of that project, which worked in partnership with regional and lower level government structure personnel in two regions, Tigray and Southern Nation Nationalities and Peoples Regions (SNNPR). Its interventions focused on: teacher development through pre-service and in-service training; effectiveness of the content and quality of materials; interactive radio interaction (IRI); planning; policy; decentralized management; and more efficient school financing. The overall strategy for USAID/Ethiopia’s Basic Education Strategic Objective (BESO II) design was similar to that of BESO I. However, during BESO II support was expanded from two regions to nine regions and two city administrations. AED, which had been one of the contractors implementing BESO I, was selected to implement the five year contract for BESO II in 2002. This was the year in which Ethiopia was hit by a severe drought and famine, which led to the creation of a new overall strategy for the Mission. The USAID/Ethiopia Strategic Plan was revised in 2004, restating its involvement in education from a Strategic Objective focused on Quality and Equity in Primary Education System Enhanced to one focused on Human Capacity and Social Resiliency Increased. The Intermediate Results from the former Strategy Plan were not significantly different from the new Strategic Objectives, so AED continued to work on basically the same interventions and activities. Specifically, since 2004, AED has been responsible for the sub-objectives of: 1. Planning, management, monitoring and evaluation for delivery of primary education services strengthened. 2. Quality of primary education improved and, as the sub categories of that objective: 3. Quality teaching force improved. 4. Application of student-centered, active–learning methods. 5. Supplementary reading materials development and distribution strengthened. Within the new strategy, which changed the name of the project from BESO II to Basic Education Program (BEP), the components of the AED contract remained the same and did not result in significant variation in the BESO II activities. The project design is extremely broad in the geographic and educational spread defined as the range of the project. The time allowed within which to address this large education population and number of institutions and to achieve the objectives of the project has been extremely short. In addition, the budget for the program has been limited. These constraints have influenced what BESO-BEP could and could not do, as will be addressed in the evaluation report within the discussions of the various components and in the lessons learned section. However, these constraints also presented opportunities for the creative development of realistic approaches, which could be sustained within the current context of the country, with or without the support of BESO-BEP. During the first two years of the project, progress in the implementation of BESO II was slow. The Mid-term evaluation carried out in 2004 found the project design extremely broad and the time allowed to implement it extremely short, plus a slow start to the project and a number of major management issues. The second two years of the project, which had an end date of August 2006, have made rapid progress in spite of political unrest and high educational turn over. This evaluation covers the progress made from August 2002 to the present. BESO II – FINAL EVALUATION 2 B. PURPOSE OF EVALUATION The purpose of this evaluation is to examine the implementation of the Basic Education Program (BEP) by AED. The findings will be used to determine the effectiveness of BESO-BEP interventions, examine their impacts in a sub-set of five regions, identify and analyze implementation issues which have occurred, and assess the effectiveness of the project management, including the impact of management on the implementation of the project. The components of BESO-BEP which will be discussed in this evaluation directly address these objectives. • Pre-Service Teacher Education is designed to improve the quality of the teaching force and the application of student-centered, active-learning methods. • In-Service Teacher Education is designed to improve the quality of the teaching force and improve the application of student-centered, active-learning methods. • Supplementary Media and Reading Materials is designed to improve the application of student￾centered, active-learning methods and strengthen the development and distribution of supplementary reading materials. • Strengthening Leadership and Planning and Management Capacity is designed to improve planning, management, monitoring and evaluation for delivery of primary education on all levels of the system. • Support for Female Teachers and Student Teachers is designed to enhance the self-confidence and capabilities of female teachers and teacher trainees and create a deeper understanding of gender issues. Specifically, the evaluation has been asked to address the following issues: • The project’s overall results to date, including both the targets met and missed and the impact of the project on the targeted beneficiaries. • The effectiveness of the management of AED in achieving the USAID Education Program priorities. • The relationship between the activity of the project and USAID’s Basic Education Program Results Framework. • The degree to which BESO-BEP has responded to perceived needs of its beneficiaries. • Lessons learned. • The relationship between the design of the project and the evaluation findings. The findings and recommendations of the evaluation with be used by UAID/Ethiopia and the Government of Federal Democratic Republic Ethiopia to determine future strategies for the enhancement of quality and equity in education. C. METHODOLOGY The evaluation team conducted three types of research between November 27, 2006 and December 5, 2007: • AED documents that supplied information about targets, implementation plans, and achievements and an analysis of how cost effective the interventions have been. The evaluation team relied totally on AED’s assessment of their achievements as it would have been difficult to verify the information in the time allotted. • Key informant interviews with central Ministry of Education department heads, Regional Education Bureaus personnel from the five regions examined in the evaluation, 16 woreda education officers, 6 Teacher Training Colleges (TEIs), and AED staff in Addis Ababa and the five regions. (See Annex A for list of names) • Structured questionnaires (data collection form included as Annex B) were used to obtain information from directors, teachers, students, kelebe and PTA heads, and pedagogical center coordinators (See Annex A for list of names) at 37 randomly selected schools in the five regions. (See Annex C for full list of schools and woredas). Due to time constraints, school selection was not fully random, as it was influenced by geographic proximity to the TEIs and Regional Bureaus being visited. BESO II – FINAL EVALUATION 3 Chart 1: Schools Visited by Evaluation Team School Type Afar Amhara Benishangul Gumuz Gumuz Oromia SNNPR Total Centers 1 5 2 4 1 13 Satellites 3 4 1 3 2 13 Non-BEP 2 1 2 4* 9 TEI Link 2 1 3 Urban 2 6 2 7 3 20 Rural 2 5 2 2 5 17 Large 2 5 1 7 3 19 Small 2 6 3 2 5 18 * The team working in SNNPR was accidentally taken to a number of WLI/CGPP rather than BESO-BEP schools. Key information from the data collection forms was entered into a quantitative data analysis program (Systat) in order to translate some of the qualitative interview data into quantitative tables. Due to the limited number of samples, 37 schools, 14 woredas, and 6 TEIs, no attempt was made to create statistically significant conclusions. Instead, simple frequencies were used to statistically portray the findings from the field research. II. FINDINGS AND CONCLUSIONS A. PRE-SERVICE TEACHER EDUCATION 1. FINDINGS a. AED Contractual Obligations The primary purpose of this component of BESO-BEP is to enhance the quantity and quality of primary education professionals and teacher graduates and to contribute to the application of active learning, continuous assessment, large class management, and action research in primary schools. In all cases, project deliverables and targets presented in Table 1 are taken from the project contract scope of work, which, according to the project CTO and COP, has not changed contractually since its initial signing. A complete list of BESO-BED interventions in this component are found in. Annex E, Table 6 and were adopted from review of AED’s annual reports. Actual results were derived from AED’s annual reports for base years and performance result reports provided by the project component technical heads. As Table 1 indicates, the project has met most of the associated targets and deliverables. The project has met or exceeded several of the associated targets and deliverables for the component of the project, but it has also failed to meet others. BESO II – FINAL EVALUATION 4 TABLE 1: AED CONTRACTUAL OBLIGATIONS Base Yr. One Base Yr Two Option Yr one Option Yr 2 Option Yr 3 S/N Contract Deliverables Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date 1 Establishment of a viable teacher strengthening system that receives continued support from the pre￾service training institutions Reach 18,000 teachers Target exceeded 25,050 (female= 9270) primary school teachers reached Reach 35,000 teachers Target not met 17,683 (female = 5,589) primary school teachers reached Reach 1000 teachers in 5 TEIs Target exceeded Involving 10 TEIs 10,569 teachers were reached Reach 2000 teachers in 10 TEIS Target exceeded 12,038 (female= 5,007) reached in 22 TEIs Reach 3800 teachers in 19 TEIS Not due Overall, the project has exceeded planned output levels Through option year 2 the planned output was 56,000 teachers.; cumulatively the project has reached more than 75,000 teachers. 2 Number of TEIs that demonstrated a commitment to increase female graduates (disaggregated by certificate and diploma cources) through specific policies introduction of new mechanisms of support or other concrete verfiable actions 15 TEIS 10 TEIs in certificate program - Strengthen Amhara and Oromiya regional network -Establish the 3 networks -Organize national workshop on sensitizatio n for policy makers, Ed. Officials and other partners For female students -Amhara and Oromiya regional networks strengthened -The three networks for the five regional TEIs established - A national sensitization workshop was conducted for higher officials of the ministry education, Regional education bureaus and partners 10 TEIs in the certificate program increased the enrollment of female students and no information from the TEIs in diploma program. Targets partially met Based on the regional policy a 30% quota in favor of female enrollment in the TEI has been implemented eg. in Arssi zone - Sensitization workshop conducted. Target met Observation No tutorial classes for girls in the TEIs observed as the time of teams visit. BESO II – FINAL EVALUATION 5 b. Training of Instructors and Staff The most direct, sustainable, and pervasive way in which BESO-BEP has improved the quality of primary education is through accelerating the implementation of the new Ethiopian curriculum. The two foci of that effort have been support for pre-service and in-service training of teachers. BESO-BEP pre-service activities have primarily targeted Teacher Education Institutes (TEIs) where faculty and administrative staff have received training in active learning, continuous assessment, large class management, and action research. In addition, instructors have received training in how to train other trainers in order to prepare them for the role they are intended to play in providing in-service teacher training in primary schools linked to the TEIs. In all of the TEIs visited, active learning and continuous assessment has permeated the curriculum of the institutes and was in use in the courses taught and in the in-service training provided to schools. Active research projects were being conducted by faculty members in spite of their busy schedules. Among the six TEIs visited by the evaluation team, a wide range of other training for faculty and staff was reported. For example, training for selected staff had occurred in the United States and in African countries, school registrars had been trained to manage student records, instructors had been trained in computer use, database management, and internet use, and coordinators of multi-media language labs had been trained. Among the six TEIs visited by the evaluation team, a major barrier to increasing the capacity of the personnel at the TEIs has been the high turn over in positions. The other problem that limited the success of some interventions was the extremely heavy workloads of instructors, which limited their commitment. Both of these constraints are beyond the scope of what AED can influence, but they further define the context in which the AED interventions were expected to have an impact. AED has, in schools, woredas, and regional offices as well as in TEIs, responded by training more and more individuals to fill the positions that emptied and were filled again. It remains the responsibility of the TEIs to provide appropriate staff and to manage their workloads. The transformation of Teacher Training Institutes into Teacher Training Colleges and the delegation of the preparation of primary school cycle 1 (grades 1-4) teachers to the private sector will have an impact on the investment BESO-BEP has made in the development of cycle 1 teachers over the past several years. The current focus in the TEIs is to bring active learning and continuous assessment approaches into the cycle 2 (grades 5-8) curriculum. Because it is cycle 1 teachers who will in the future most need training in active learning, continuous assessment, and management of large classes, the lack of project involvement in the private training of cycle 1 teachers could erode the achievements which have been made. c. Strengthening Resource Centers at Teacher Training Colleges Many of the resources supplied to TEIs by BESO-BEP were focused on strengthening libraries, computer centers, and pedagogical centers, plus the creation of cluster coordination centers and centers of excellence at the Debre Berhan, Asela, and Jijiga TEIs. All of the six TEIs visited for this evaluation had received resources such as books, furniture, computers, printers, copy machines, training, and funding for the revitalization or creation of these centers. Among the six TEIs visited, variations were found in the degree of utilization, maintenance, and staffing of these centers. Libraries. In all six TEI that were visited, the libraries were in heavy use; in three, almost every chair in the library was filled with faculty and students during the evaluation team’s visit. Librarians and library staff were interviewed at all six TEIs and they reported strong appreciation for the training, furniture, and books they had received from BESO-BEP. The libraries were orderly, clean, and created an appropriate atmosphere for study. Most of the libraries were well staffed and were open for regular, scheduled hours each day, although the library at the Awasa TEI was closed when the team visited because the only librarian on duty had gone for coffee. At the Asela TEI, eight librarians, one of whom had been trained by BESO-BEP, were scheduled to work in shifts of four and two were on duty when the evaluation team visited; the Debre Berhan TEIs library was run by a complete staff, which included head librarian, who had been trained by BESO-BEP, a technical assistant, two circulation workers, an acquisition clerk, and a binding worker. In the TEI libraries at Jimma, BESO II – FINAL EVALUATION 6 Gonder, and Debre Berhan, all of the books have been catalogued; in the TEI libraries at Asela, Awasa, and Hossana the majority of the books have not been catalogued. Computer Centers. Interviews with the coordinators of the computer centers at the six TEIs visited indicated that they had received adequate training and that they were committed to maintaining well-run centers. But the coordinators also reported a number of technical and maintenance problems. For example, the coordinator of the computer center at the Asela TEI reported that most of the computers in the center are obsolete or not operational and there is no staff member trained in hardware maintenance. Power fluctuations and internet service for only one computer at a time limit the usefulness of the center. This is troubling because, according to the agreement between AED and the TEIs, the TEI is required to provide an on-going budget and supplies to insure the operation and maintenance of the equipment supplied by BESO-BEP. Utilization of the various centers depends in many cases on how the centers have been incorporated into the TEI’s instructional schedule, although, as mentioned before, AED has no way to influence TEI course choices. Courses in computer use were offered at all six institutes and were so popular that generally two or three students had to work together at a computer. But, when there was no class meeting in the computer center, several of the coordinators reported that the computers often sat unused. On the other hand, the computer centers at Debre Berhan were full of students and faculty at the time of the team visit. Without practicing what was learned in class, it is doubtful that the students had more than a very shallow understanding of computer use. Once they begin their careers as teachers, there will, in most cases, be no computers in the schools where they work and whatever skills they learned at the TEIs will fade. The coordinator of the computer center at Debre Berhan agreed with this scenario, but believed that some familiarity with computer use is still better than being a complete stranger to the technology. Pedagogical Centers. Utilization of centers also depends upon the presence of staff to keep the centers open and manage activities within them. At the Awasa TEI, the pedagogical center, which had been revitalized and equipped with BESO-BEP assistance, has stopped functioning. The former coordinator of the center reported that it would actually take a minimum of five people to staff the center continuously. He had just been promoted to instructor and is no longer able to run the center as it is not within his job description. At the Asela TEI, the former coordinator of the Center of Excellence, which is designed to support Teaching and Learning Using Locally Available Materials (TALLULAR) activities, had left his position to become an instructor because the center remained outside the TEI staff structure and he could not receive the normal benefits of a staff member or any promotions while in the center position. His training in Malawi had been supported by BESO-BEP and his continuing commitment to TALLULAR methodologies was demonstrated by the book he had just finished writing on the subject. The course structure created by the TEI can also have an impact of the utilization of the pedagogical centers. For example, no course was offered at the Asela TEI in the production of TALLULAR in spite of the impressive displays and skills found in the Center for Excellence. Students could sign up for a workshop of the same type provided for pedagogical center coordinators from nearby schools, but most were unwilling to add to their study load without receiving credit. All six of the pedagogical centers visited were well stocked with creative TALLULAR tools and the space and the displays of the centers were well maintained. An interesting contrast exists between the two Centers of Excellence at the Asela and Debre Berhan TEIs. Both were established by BESO-BEP, which supplied equipment, training, supplies, and budget for operation. Both offer notable displays of their products But the publishing facility at Debre Berhan is busy, constantly expanding its activities, and linking itself to other TEI programs, while the TALLULAR center at the Asela TEI is underutilized and staffed with dissatisfied coordinators. The major differences lie in the degree to which one TEI has incorporated the Center into its structure, while the other has forced the Center to remain an unattached project supported by donor funding. There could be – and this is speculation on the part of the evaluation team based on far too few examples (but might be worth investigating) – a status issue influencing the amount of ownership which the TEIs have shown toward the pedagogical centers, where staffing, course offerings, and incorporation into the TEI structure have all surfaced as issues in the TEIs visited. Libraries lie at the heart of all academic institutions, are the storehouse of written information associated with education, and tend to be a symbol of institutional pride. BESO II – FINAL EVALUATION 7 Computer centers represent new technologies, generating excitement and a sign of movement of the TEIs into a more modern era. But pedagogical centers were described by some instructors as a rejuvenation of an older technology, one which is based upon an admission of a lack of educational resources. Commitment to pedagogical canters at TEIs could be weakened by a perceived lack of status, in spite of the fact that, as the evaluation team’s visits to schools found, school enthusiasm for pedagogical centers was extremely high. d. Linkage between Teacher Training Colleges and Schools BESO-BEP has established and funded Cluster Coordination Units at all TEIs to accelerate the spread of active learning, continuous assessment, large classroom management, and action research to a set of primary schools in the area of the TEI. All schools in four of the regions the evaluation team visited have been organized into clusters with a center and satellite schools similar to the BESO-BEP clusters. The Afar region has not created clusters, but the education staff at the Dubti woredas had organized the schools of their woredas into clusters. The six TEI visited by the evaluation team had each established links with between 22 and 60 schools, linkages managed through the Cluster Coordination Unit. The links to these school consisted of resources, such as typewriters and duplication machines, supplied to the center schools in the clusters, use of these schools for student-teacher’s practicum experiences, establishment of model classrooms in some of the schools, and in-service training of principals, key teachers and supervisors by instructors from the TEI. The cluster coordinators are working to develop strategies to involve TEI instructors in in-service training in active learning, continuous assessment, large classroom management and action research at these schools. In Debre Berhan, the cluster coordinator manages records of in-service training provided by instructors and the number of hours spent conducting in-service training are translated into points, which can be used to advance the career of the instructor. The TEI instructors were also asked to conduct observations of teachers in linkage schools, which, along with reports of students who had spent one year teaching in those schools as their practicum, suggest that active learning is being used effectively in these schools. Nine of the schools visited (out of 28 schools visited in the three regions with TEIs), including four schools linked to TEIs, had received training from TEI instructors. In addition, the pedagogical centers at all of the TEIs visited, including the Center of Excellence at Asela, were involved in training principals, key teachers and supervisors from near-by and linked schools in how to prepare and use teaching aids. e. Support for Female Students at TEIs BESO-BEP has provided seed money for the establishment and development of networks among TEI female instructors. The project has developed and disseminated six modules and four courses on gender issues. The project has also promoted and supported affirmative action and quotas for female student enrollment in TEIs and paid for special tutorial and remedial classes for female students, designed to help close the gap between male and female student academic performance. It is too early to judge if this assistance has closed the gap; however, it has increased the number of female teacher trainees in all TEIs. In addition, the project has promoted special counseling to female students based on the modules provided. The female student teacher supports developed by BESO-BEP were handed over to the TEIs to manage. Consequently, great variations can be seen. In both Debre Berhan and Gonder there are a range of extremely active and supportive female associations that provide counseling and guidance to female students. In both of these TEIs, gender topics are addressed in civics courses taught by female instructors who are familiar with the gender modules produced by BESO-BEP and have received female leadership training. Support for female students in the other four TEIs was weaker, in part due to a smaller percentage of female instructors. There seems to be a need for a critical mass of instructors who have received female leadership training to be present before the activity level of female student support Chart 2: Percent of Instructors who are Female in the Six TEIs Visited by the Evaluation Team Debre Birhan Gonder Awasa Hossana Jimma Asela 12% 13% 9% 8% 4% 8% BESO II – FINAL EVALUATION 8 at the TEI becomes noticeably strong. For example, due to the low number of female instructors at Asela, 4 out of 50 instructors, male instructors taught what they described as courses on “gender,” but these instructors had had no training in that subject and were unaware of the BESO-BEP modules on gender issues. Although remedial teaching and tutoring have been incorporated into instructor’s job descriptions at TEIs, in most institutions little remedial work has happened. In some of the TEIs visited, female students have received tutorial assistance sponsored by BESO-BEP. Due to the use of continuous assessment in courses, instructors can recognize who needs remedial assistance before serious problems develop, an indication of how BESO￾BEP interventions can support one another. BESO-BEP paid for tutorials and remedial classes for female student-teachers between 2002 and 2004 and instructors cooperated by offering remedial courses for female students. Last year, with BESO-BEP’s financial support no longer present, almost no instructors have been willing to offer tutorials. Instructors claim to be overwhelmed by their workload. BESO-BEP has attempted to promote the use of the modules it has produced on gender issues at TEIs by creating study-groups in various female associations and producing a study-group guide for the modules. However, there is relatively little evidence of the use of the modules at the TEIs. For example, none of the third year female students who were interviewed indicated any awareness of these materials. Appointed gender focus instructors generally described having too little time to engage in activities to support female students by discussion of the topics in these modules, although they also reported that there was a clear need for such support and that they appreciated the modules A new constraint to female student success at TEIs was recently introduced when hostel living arrangement at the TEIs were discontinued. The stipends given to students to support themselves in the communities around the institutes are inadequate and have led to greater stress on students, especially female students. The personal development skills materials created by BESO-BEP could be of great assistance to help in coping with these problems, but no evidence of their systematic use in assisting female students were found in the TEIs visited. f. Sustainability Support by BESO-BEP to TEIs has been totally integrated into the government’s own efforts to spread understanding and use of the new curriculum. This has primarily involved strengthen instruction in active learning, continuous assessment, management of large classes, and action research among instructors and students at TEIs. Sustainability was planned from the beginning because the interventions were designed to implement policies what had already been created by the government. The libraries, computer centers and pedagogical centers revitalized and equipped by BESO-BEP are sustainable to varying degrees, depending upon the TEIs’ willingness to provide the supports required, especially in the areas of maintenance and staff, to insure the accessibility of centers for faculty and student use. For example, the structural problem in the integration of the Asela Center of Excellence into the TEI will have to be solved before it can become sustainable. The sustainability of the links to primary schools that have been created or intensified by BESO-BEP funding are important not only to this component of BESO-BEP involvement, but to the sustainability of all in-service teacher support. This is a strategy for sustainability built into the support for TEIs from the beginning. However, this system will not be sustainable unless incentives are offered to instructors to encourage their participation. The TEIs themselves are concerned about sustainability. For example, the Gonder TEI has established a sustainability committee to research and recommend how the BESO-BEP activities can be continued. But the committees have yet to produce any recommendations. BESO II – FINAL EVALUATION 9 2. CONCLUSIONS The majority of BESO-BEP pre-service activities has targeted Teacher Education Institutes (TEIs) where faculty and administrative staff have received training in active learning, continuous assessment, large class management, and action research. In addition, instructors have received training in how to train other trainers in order to prepare them for the role they are planned to play in providing in-service teacher training in primary schools linked to the TEI. Additional training in a wide range of skills has also been provided to selected administrative staff and faculty in an attempt to strengthen capacity at TEIs. An obstacle to achieving an improved capacity is presented by the high turn over in personnel and the extremely heavy workloads of most instructors. Both of these constraints are beyond the scope of what AED can influence, but they further define the context in which the AED interventions were expected to have an impact. Based on visits to six TEIs, utilization of the libraries, computer centers, and pedagogical centers located at TEIs, which have been revitalized and supported by BESO-BEP, is high, but there are maintenance and staffing problems in several of the centers at these TEIs. One of the difficulties of the BESO-BEP program is that it can provide resources and funding, but it cannot dictate how they will be used. Part of sustainability is the ownership which grows out of making the decisions about resources and funding, but those decisions can have an impact on the utilization of the resources which have been provided. BESO-BEP has established and funded Cluster Coordination Units at all TEIs to accelerate the spread of active learning, continuous assessment, large class management, and action research to clusters of 20 to 100 primary schools in the areas of the six TEIs which were visited. The links to these schools consist of resources supplied to the center schools in the clusters, use of the schools for student-teachers’ practicum experiences, establishment of model classrooms in some of the schools, and in-service training of principals, key teachers, and supervisors by instructors from the TEI. The cluster coordinators are working to develop strategies to involve TEI instructors in in-service training as some form of incentive is necessary for the sustainability of the training. The sustainability of the links between TEIs and primary schools that have been created or intensified by BESO-BEP funding are important to the sustainability of all in-service teacher support. If each TEI can provide the same training and support to a selection of primary school clusters in its area that BESO￾BEP has to its selection of clusters, then the model established by BESO-BEP can be expanded to reach the remainder of the primary schools of Ethiopia. 3. RECOMMENDATIONS • Given the heavy workload of most of the instructors who were interviewed, some incentive needs to be provided if these instructors are expected to take on the additional tasks of in-service teacher training and support for female students. Within the Ethiopian context, it is unrealistic to expect the TEIs to offer financial incentives for these activities. The most realistic use of incentives for these purposes that the evaluation team encountered were found at the Debre Berhan TEI, where instructors received credits for their in-service efforts. These credits could then be used to support the instructors’ career advancement. However, as in the case of supplying adequate staff or scheduling courses to utilize resources supplied by BESO-BEP, the project has no power to force TEI management to adopt approaches such as this. • The need for support in life skills for female students has intensified since BESO-BEP began because of the cancellation of hostels for students at TEIs, yet the materials supplied by BESO-BEP and the gender offices at TEIs facilitated by BESO-BEP appear to be underutilized due to lack of staff time. Again, some form of affordable incentive is needed. • In order to consolidate the gains of past BESO-BEP investment in preparing and strengthening the active learning, continuous assessment, large classroom management and action research skills of cycle 1 teachers, support needs to be extended to the private sector Teacher Training Institutes. BESO II – FINAL EVALUATION 10 B. IN-SERVICE TEACHER EDUCATION 1. FINDINGS a. AED Contractual Obligations This component of BESO-BEP is designed to improve the skills of teachers to apply active learning student￾centered, continuous student assessment, large class management, and action research methodologies, supply education materials and equipment to schools, and facilitate the exchange of experiences and collective approaches to solving educational problems among school principals and teachers. A complete list of the interventions associated with this component is included in Annex E, Table 7. These interventions were compiled from AED’s annual reports. As indicated in Table 2 below, the project’s performance on this component is generally very positive. The data were derived from AED’s contracted scope of work (for planned outputs), annual reports (for base years) and program result reports (for option years) provided by the component leaders. The MOE had requested that USAID train 21,400 primary school first cycle teachers in order to upgrade them to the primary school second cycle level. Consequently, USAID asked AED to reallocate budget for the training. Birr 3.7 million was taken out of the in-service budget and given to the Educational Media Agency (EMA) to develop and produce teacher distance education self-instructional course material in collaboration with eight colleges of teacher education. The course was designed to be completed by a teacher over two years. After a long delay the modules were produced under the guidance of EMA. 16,411 teachers have completed the distance learning. b. Use of the Cluster Structure for Training One of the most successful and sustainable aspects of BESO-BEP is the synergy created by the use of the cluster structure of schools, cascade in-service teacher training, and self-instructional teaching materials. All interviews indicated great enthusiasm among teachers for the cluster structure, which they believed offered them support and contacts with other teachers. Of course, the degree to which the schools of a cluster actually interact varies due to distance, transportation, and the dedication of principals and key teachers. But almost all of the BESO-BEP schools visited reported that they now met as a cluster to discuss problems, exchange experiences, coordinate training schedules, and design exams in common. Teachers claimed that the cluster structure, combined with training and study group activities, had renewed their interest in teaching. A dozen reported that they now prepare common examinations for some subjects and grades. In the Holeta Town cluster in Oromia, the staff from all of the schools had agreed to a common approach to group exercises, oral questions, monthly tests, classroom projects and fieldtrips. A teacher at Gundo Beret in Amhara reported that the cluster teachers now felt “brotherly and sisterly.” The table below indicates the number of schools where teachers reported planned, educational interactions among teachers in their schools and/or clusters. Chart 3: Educational Teacher Interactions within Schools Center Schools Satellite Schools non-BESO Schools N Study Groups of Teachers meet to discuss training and/or materials 6 7 1 14 Monitoring of classrooms by principals, department heads, and peers. 5 3 2 10 BESO II – FINAL EVALUATION 11 TABLE 2: PERFORMANCE SUMMARY - IN-SERVICE Base Yr. One Base Yr Two Option Yr one Option Yr 2 Option Yr 3 S/N Contract Deliverables Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date 1 Number of primary teachers completing at least one self instruction kit reported by region /zone and by gender Materials being developed Materials still being developed 1st cycle kits for 12,000 teachers Nothing reported 12,000 1st and 2nd cycle teachers complete a package of self instructiona l kits Target exceeded 15,966 teachers (6906 females) were trained in rounds 12,000 first and second cycle teachers complete a package of self instructional kits Target exceeded 36,583 teachers 11,768 female were trained in rounds Not Due The project has exceeded its target 2 Number of woreda with cluster Resource centers equipped with adequate training resources budget and staff to support school cluster activities Nothing planned Nothing planned Equip 5 woreda cluster resource center Target met 5 woreda cluster resource centers equipped Equip 10 woreda cluster resource centers Target exceeded 11 woreda cluster resource center equipped Equip 15 wordas cluster resource center Not due The project has exceeded its target 3. Number of multi￾session self instructional its contently print materials, audio cassettes) for in￾service continuing education for primary teachers developed and field tested (reported by topic or them of each kit, and languages of Develop 2 kits for 1st cycle primary Target Met 1st cycle kits developed Develop 2 kits for 2nd cycle primary Target Met 2nd cycle kits developed Developed monitoring guide collect data and revise 1st cycle kits Target Met Developed monitoring guide collected data and reviewed 1st cycle kits Develop monitoring guide collect data and review 2nd cycle kits Target Met Developed Monitorin g guide collected data and review of 2nd cycle kits Not due BESO II – FINAL EVALUATION 12 Base Yr. One Base Yr Two Option Yr one Option Yr 2 Option Yr 3 S/N Contract Deliverables Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date instruction, and handed over to REBs. 4. Number of multi￾session self￾instructional kits content, print materials, audio cassettes for in￾service, continuing education for primary teachers produced and disseminated Produce 25,000 1st cycle kits Target met 25, 000 1st cycle kits developed and distributed Produce 15, 0002nd cycle kits and 20,000 1st cycle kits Target exceeded 45,000 1st cycle kits printed and distributed, and 11,303 cycle 1 components printed distributed Print 10,000 copies of 1st cycle kits and distribute Target not met Cycle 2 kits printed in 10,000 copies and distributed Nothing planned Nothing reported Print 10,000 copies of 2nd cycle kits an distribut-ed Not due BESO II – FINAL EVALUATION 13 The resource centers established in the Center schools by BESO-BEP offered both a duplicating machine and typewriter for the use of all principals and teachers in each cluster. These resources were in place in all of the Center schools visited. The duplicating machines at the Center schools, whether electric for schools with electricity or a crank model, were in frequent use. The ability to duplicate materials was so valued by the schools that were visited that no machines that were not usable were observed, and the maintenance was supplied by the schools, often by a fee contributed by each school in the cluster. In two Oromia schools, the fee was 5 birr at Horsis and 2 birr at Dera for each teacher in the cluster. Generally Center schools created a schedule for use of duplicating machines in order to avoid congestion during the period when exams were being duplicated. When the evaluation team visited one Satellite school in Amhara, the principal of the small Satellite school asked for a ride with the team to the Center school in order to make use of the duplicating machine. The typewriters were used less frequently, primarily due to a lack of typing skills among the teachers and a lack of budget to hire a typist. In a number of schools they sat accumulating dust. One principal in Benishangul Gumuz did all of the typing himself. The typewriters and duplicating machines were generally located in the room adjacent to the principal’s office, although other locations for the resources center were found in some schools. Teachers reported that the other materials supplied by BESO-BEP, such as stationary, were almost always divided up among the schools in the cluster, although other resouces, such as microscopes, generally resided at the Center school. BESO-BEP has also established pedagogical centers in the Satellite schools of each cluster. There is great variation in the activities of these pedagogical centers. Only one or two appeared to be infrequently used. Most were found to be an active part of the school, a positive resource for the teachers, and were staffed by trained coordinators who took great pride in what their centers had to offer. Training for the coordinators of the pedagogical centers at schools has primarily been supplied by TEIs, especially by the Center of Excellence at the Asela TEI. In some schools, such as one in Oromia, the pedagogical center had built furniture for the entire school. In others, the large collection of teaching tools could be seen to be in frequent use due to systems created for teachers to check out specific items. In most of the schools, teachers, students, and coordinators went out of their way to show the evaluation team the creative tools they had produced. Several teachers reported that they had been given center coordination duties on top of heavy teaching loads, which meant that the centers were only open for limited hours and the evaluation team found the centers closed about a third of the time. c. Cascade Model for Training In all of the schools visited, principals and key teachers from the Center school had received direct training from BESO-BEP and many of the Satellite schools also reported BESO-BEP training for principals and key teachers. While there was some variation in the list of topics they reported having had training in, the list always included all or most of the basic four: active learning, continuous assessment, large class management, and active research. Some Center schools appeared to have received a great deal of training, such as a large urban school in Amhara which reported that its principal, key teacher and school supervisor had received a total of 62 days of training by BESO-BEP. The same school personnel were generally trained repeatedly on a variety of topics, an approach which can result in strong, able leadership but also increases the negative impact of high turnover or can lead to little impact in the case of poorly motivated leaders. Turnover in trained personnel can result in a break in the chain of training. In four BESO-BEP schools visited during the evaluation, the loss of the key teacher had resulted in a complete cessation of all teacher training for the school. In addition, not all school leaders interviewed appeared to have used their training to create change in their institutions, which can create a bottleneck in the spread of information to the teachers who would benefit. While variations were found in the patterns of how these trained personnel shared what they had learned with the teachers in their own school and the principals, key teachers, and other teachers in the Satellite schools, some staff training had occurred in all schools visited. BESO II – FINAL EVALUATION 14 The most serious question about cascade training is about the amount of content lost over the multiple layers of training. The direct training, which principals and key teachers received from BESO-BEP, involved many days of concentrated instruction. When these directly trained members of the staff returned to their schools, the training which they offered to either teachers in their own school or the staff from Satellite schools generally was reported as occupying on average one day per month during the school year. A number of teachers said that they did not feel that they had received an adequate amount of training to master the subjects. Without adequate follow-up to observe how much teachers in Center schools and Satellite schools have learned, it is difficult to estimate how much of the initial training is lost, but the differences in the number of days of training would suggest a steadily more shallow version is transmitted. BESO-BEP lacks a consistent follow-up system to ensure the quality of the training and skills that are transferred. Welenchitia in Oromia reported that classroom supervision and follow up had been provided by, in addition to school-based supervision, the AED regional office and the Boset woreda, while most of the schools visited reported little or no follow-up visits. At all levels and in all regions visited by the evaluation team, interviewees indicated their concern over the lack of follow-up. d. Self-Instructional Kits for Teachers Self-instructional kits for active learning, continuous assessment, and large classroom management were found in 75% of the BESO-BEP school visited for the evaluation. Principals and teachers said that they were extremely useful and had had a real impact on the skills, attitudes, and behavior of the teachers. In most of the schools, the kits appeared to be used and principals and teachers reported that they were. Chart 4: Presence, Use, and Storage of Self-Instructional Kits Used Were kits are stored Type of School Kits at the school Reported Evidence Principal Dept Library Pedogog Center 12 8 8 6 3 3 2 Satellite 9 5 4 2 1 3 3 non-BESO 2 2 2 In many of the schools there was evidence of use: charts for when specific topics in the kits were to be discussed, study groups who met weekly to work with each module, or sign out records indicated when the kits were taken out of the library or pedagogical center. But there were also schools where teachers said the materials were great, but that they did not use them because they did not have time. Some teachers described a lack of a “reading habit” and said that training was a more effective way to learn than the use of reading materials. The most consistently reported use of the kits was when they were used in conjunction with in￾service teaching. In one school in Oromia, study groups that discussed the kits each week had stopped meeting this year when the key teacher moved to another school and no in-service training was taking place. In three other schools, one in Amhara and two others in Oromia, the same pattern was found: the key teacher had left, no training had taken place this year, and the new revised self- instructional kits were still in their wrappers. In most of the schools sign-out systems had generally been created for teachers to borrow these materials. Where the kits were plentiful, teachers were allowed to check them out for longer, up to an entire year in one small school, Dubti 4 EST, in Afar where there was one kit for each teacher. There were some indications that the kits stored in the principal’s or department head’s office were more consistently used than those kept in the library or pedagogical center. The value associated with the kits was demonstrated by Barro Primary, a non-BESO-BEP school in Benishangul Gumuz where the principal had borrowed the kits from a BESO-BEP school to share with his teachers. In a BESO-BEP school in Oromia, the kits had been duplicated to be used by more the teachers. In Afar, teachers described thoroughly reading the kits and supplying feedback for improvements to AED. BESO II – FINAL EVALUATION 15 e. Impacts of Training and Materials To varying degrees, a better understanding of and attempts to use active learning, continuous assessment, management of large classes, and active research are now found in all levels of the education system. Changes in attitudes and behaviors among the teachers were reported in all of the schools visited. Both principals and teachers felt that the training they were offered by BESO-BEP focused on their greatest needs. Many felt that they had not understood the new curriculum and had had no idea how to implement it prior to the training. One teacher in a Dubti school in Afar, commented that, in addition, the in-service training had “brought a close relationship among teachers because of the uniformity of training.” The impacts of active learning techniques were most frequently mentioned. When specifically asked, principals and teachers would admit that there had been initial resistance to using active learning approaches among teachers, who found that active learning required more planning time, and the community, who feared that no learning was taking place without lectures. But the evaluation team now encountered no teachers who reported that they did not like active learning approaches and did not think that they made school more interesting both for themselves and for their students. As one principal in Oromia said, “Teachers were addicted to lecturing, but when they saw the impact (of active learning), they became enthusiastic.” Changes in teaching behavior were also reported by principals and teachers in almost all schools visited. Although all teachers claimed to use active learning approaches in their classrooms, the only evidence of the use of this approach to teaching could be found in the rearrangement of student seating into groups rather than rows. Principals and teachers reported increased self-confidence, communication skills, and completion of projects among students being taught by an active learning approach. The teachers interviewed at Dubti Awash Sheleko in Afar mentioned an especially wide range of important changes: closer relationships between teachers and students, a closer relationship among teachers due to the uniformity of training, the ability to identify the students who needed special support, more active and participatory students, and self- motivated students. In the Kegnazmaach Haysema school in Afar, the principal described a student revolt when the key teacher was transferred due to their enthusiasm for his new teaching style. Principals credit continuous assessment by teachers in identifying student weaknesses and reducing repetition. Teachers report that they have attempted to use continuous assessment in their classrooms, but about half said that they had encountered difficulties due to large class size. Among the BESO-BEP schools that were visited, the largest class size that was reported was 75 students, but that number was listed six times. Among the non￾BESO-BEP schools visited by the evaluation team, Abamote primary in Amhara had 90 students in one class and Baro primary in Benishangul Gumuz reported 120 students in one class. One result of the difficulty of applying continuous assessment in the large classes of the first cycle (grades 1-4) has been the promotion of a much higher percentage of students due to teacher uncertainty about how to assess them. Large class size appears to be the result of lack of budget more often than an actual shortage of trained teachers or classrooms. SCOPE and other community focused programs have lead to community involvement in building a sufficient number of classrooms at most schools. In some communities an adequate number of teachers were not available; in others the supple had outpaced the demand. Last year, at the Basona Werana woreda in Amhara, over 1300 applicants applied for 45 positions. There is some suggestions that teachers may be underutilized at many schools when their assignments may not include responsibility for a large enough number of class periods. The MOE is currently conducting research about this issue. BESO-BEP has addressed this problem with training in how to manage large classes. The basic technique involves grouping students into smaller groups for the purpose of implementing active learning and continuous assessment methodologies effectively. Students can be grouped in various ways. For example, students can be grouped into smaller homogeneous groups based on their learning needs; they can also be grouped by ability levels, common interests or by friendship for sharing reading materials. It also involves developing activities for group learning and group work as well as organizing group remediation support, e.g., peer tutoring, cross￾age tutoring, home tutoring and the use of community volunteers for tutoring. In many schools, the evaluation team found teachers engaging in creative, interesting, and useful research due to the training they had received by BESO-BEP. In addition to training in active research methodology, BESO II – FINAL EVALUATION 16 BESO-BEP supplied 1000 birr to cover supplies, transportation, and other expenses associated with research and training conducted to solve school problems. By conducting their own research, teachers learned how to jointly decide what problems the school faced, what questions to ask to find out more about those problems, how to gather information, and how to translate that information into solutions. In Oromia, one school conducted research on “Why girls are absent from school?” and introduced changes to help solve the problem; in Amhara, a school conducted research on “Why does the school have a discipline problem?” and has introduced solutions based on its findings. Other schools have conducted active research to solve problems such as “Why students have trouble reading English,” “Why students drop out?” and “Why female students do not participate as male students do?” f. Contrast to non-BESO-BEP schools The cluster model of school organization is now found in all the regions except Afar visited during the evaluation, but the degree to which it has been implemented varies from region to region. But the cluster system of organization provides the structure for the adaptation of cascade in-service training similar to what is now offered by BESO-BEP. Because the evaluation team was visiting six TEIs in three of the regions, they tended to visit schools in the same geographic area of the region, plus they also made an effort to visit schools linked to TEIs. Of the nine non-BESO schools visited in those three regions, five reported receiving training from a TEI or woreda. The table below indicates the amount of training and monitoring by TEIs and woredas that the evaluation team recorded during their visits to schools. The schools linked to TEIs all had received training from those institutions, an indirect impact of BESO-BEP support for TEIs and their linkages to schools. While the woredas were conducting some training and monitoring at non-BESO schools, they were carrying out many more monitoring activities at BESO-BEP schools than those not part of the project. Chart 5: Training and Monitoring by TEIs and Woredas in Amhara, Oromia and SNNPR Training by woredas Training by TEI Monitoring by woredas Monitoring by TEI BESO-BEP school 4 4 5 3 non-BESO-BEP school 3 2 1 2 non-BESO-BEP school linked to TEI 0 2 0 2 None of the center schools of non-BESO-BEP clusters that were visited had as many resources as those supplied by BESO-BEP, but most had a typewriter. All of the non-BESO schools that were visited by the evaluation team knew of the program and wished that they could be part of it. Two of these schools in SNNPR claimed that their students outperformed BESO-BEP students on the eighth grade National Examination, although they offered no evidence to support this. g. Linkage to the Community Government Partnership Program (CGPP) The Community Government Partnership Program (CGPP) focuses on developing the relationship between communities and primary schools, which can inspire the community to provide material support to schools through activities such as building classrooms and latrines. CGPP also assists in educating the community in social issues which have an impact on the school, such as gender issues. The evaluation team had the consistent impression that the schools in which both BESO-BEP and CGPP have operated are making greater progress than simply the sum of the two programs would suggest. This synergy may be the result of combining a top-down training approach with a bottom-up empowerment approach. Two Oromia schools, one, Welenchiti, urban with many resources, and one, Horsis, rural and extremely poor, provide good examples of this synergy. In addition to both being BESO-BEP and CGPP schools, both had received training from the Asela TEI, so the actually represent nodes where many of the USAID activities converge. In both schools, principals and teachers were busily trying to improve their teaching, conduct research to solve school problems, monitor one another, and meet in study groups to magnify the impact of BESO II – FINAL EVALUATION 17 training and instructional materials. They were also working with the community to conduct surveys of students in the community, provide resources for the school, build classrooms, and address student problems. Student drop out at Horsis had, which had been extremely high, declined 74% in the last two years. The walls of offices and classrooms were filled with charts illustrating the successes and plans for the schools. By creating a scoring system for community involvement, the impact of CGPP on BESO and non-BESO schools can be examined. Based on the interviews conducted by the evaluation team, schools were given a score of zero when no community involvement was mentioned, one point if the school reported that the community had assisted in building classrooms, another point if the community had become active in addressing student behavior, attendance, and/or other problems, and a third point if the community had supplied other resources to the school, such as income generating projects. (This is, of course, a very rough measure as those being interviewed at the school may not have accurately represented the actual involvement.) The following table indicates the average scores of schools involved with BESO, CGPP, or both. Chart 6: Average Community-School Involvement Scores Past or Present involvement with CGPP No Past or Present involvement with CGPP Score N Score N BESO-BEP school 1.91 5 .38 16 Non- BESO-BEP school 1.30 3 .17 6 h. Sustainability Ownership is the key to the sustainability of the in-service component of BESO-BEP. All levels of the education community are aware of the program and appear to feel that they need the interventions provided by BESO-BEP, which should create a motivation to sustain the approach to in-service teacher education that has been piloted. The cluster structure is in place in most of the regions and can be utilized by future in-service teacher support plans. The obstacles, other that funding, which is always mentioned, are creating and implementing realistic training programs for TEIs and woredas. Linkage to TEIs for continued training were built into the BESO-BEP program from the beginning. Linkage to the woreda for continued training was also established during the implementation of BESO-BEP. The activity of the woredas visited during this evaluation in providing training was extremely varied. Some have taken on impressive additional training activities, others appeared to be conducting no training. . BESO-BEP lacks a systematic follow up process to insure the quality of the training and skills which are being transferred. Without checking to see what is actually being implemented in the classroom, the entire cascade approach can disintegrate. Need for and appreciation of the duplication machines has made them a sustainable contribution to the clusters of schools, which have consistently taken on responsibility for their maintenance. The pedagogical centers rejuvenated by BESO-BEP are utilized by both teachers and students and have become an integrated part of almost all of the schools visited’ 2. CONCLUSIONS The model for in-service teacher training in the new curriculum that has been implemented in the BESO-BEP schools is built upon a cluster structure of schools, in which BESO-BEP resources remain in the Center school for all of the cluster schools to use and principals and key teachers are expected to pass on the instruction and materials they have received from BESO-BEP to the other teachers in the cluster. Teachers are trained through a cascade pattern in active learning, continuous assessment, management of large classes, and active research. In all of the schools visited, the resources supplied by BESO-BEP have been maintained by the cluster schools, some amount of in-service training has occurred within the cluster, and self-instruction materials have been utilized by the teachers to varying degrees. BESO II – FINAL EVALUATION 18 Principals and teachers report impressive changes in their attitudes, behaviors, and skills. As in any cascade system, there is some question about the loss of content, but, even if not fully implemented, there appears to be a better understanding of the key principles and the use of them that has reached most of the classrooms in the BESO-BEP supported schools. Teachers report that active learning approaches have increased the self￾confidence, team work and competencies of primary education students. A major obstacle to maintaining and/or expanding this model of in-service teacher training lies in the problem created by frequent turnover in personnel. The loss of a trained person within the cascade context means more than just the loss of that individual; it creates a break in the chain of training. In four schools in which the key teacher had left, all teacher training and study of the self-instructional kits had cceased. This is not the responsibility of the project, but it is the reality within which the project has had to operate. To compensate for the turnover in personnel, BESO-BEP has had to continuously train new persons in the same subjects rather than being free to devote more effort to follow-up of the training received. The planned sustainability and expansion of this model of in-service teacher training depends upon the degree to which the TEIs and woredas can become the trainers for the education system. The linkages with primary school clusters, which have been established at most TEIs, have resulted in training and monitoring activities at all linked schools that were visited. Training by woreda personnel had taken place at two of the nine non￾BESO schools that were visited. In addition to training, woredas and TEI personnel have been involved in follow-up activities to ensure the quality of the training and skills transferred through the cascade model; TEIs were doing so at linked schools, but woredas had focused most of their follow-up activities on BESO-BEP schools. The Community Government Partnership Program (CGPP) focuses on developing the relationship between communities and primary schools, which can inspire the community to provide increased support to schools. The evaluation team found that the schools in which both BESO-BEP and CGPP have operated have better community-school relationships and appear to be more highly energized in general than the sum of the two programs would suggest. This synergy may be the result of combining a top-down training approach with a bottom-up empowerment approach. 3. RECOMMENDATIONS • Rather than BESO-BEP repeatedly training the same persons in each school, generally the principal and the key teachers, four teachers suggested that a better strategy would be to train a variety of teachers at the school, making each an expert on a different area of the curriculum. This would help to combat the turnover found in many of the schools, diversify the persons giving training at the school level, and prevent investing most of the success of the program in a few individuals who may or may not be up to the task. • Because current follow up is insufficient to determine how great the problems of transfer in information through the cascade model are, research into the relative differences in understanding of active learning, continuous assessment, large class management, and action research between Center and Satellite schools could be conducted for a better understanding of this problem. • Certification for teachers on each self-instructional module upon which they have received in-service training and discussed the related materials in a study group could increase commitment and provide a standard for assuring that teachers are familiar with active learning, continuous assessment, large class management, and action research. • Large class size often prevents adoption of active learning and continuous assessment methodologies. Some attempts are being made to reduce this problem: the Community Government Partnership Program and other community focused programs have lead to community involvement in building a sufficient number of classrooms at most schools and the MOE is currently conducting research into BESO II – FINAL EVALUATION 19 the possible underutilization of teachers at many schools. A creative search for other solutions needs to be conducted. C. SUPPLEMENTARY MEDIA AND READING MATERIALS 1. FINDINGS a. AED Contractual Obligations This component of BESO-BEP is designed to support student-centered active learning methods by providing teachers with additional teaching materials and resources, which will improve the quality of instruction in primary schools and TEIs. A complete list of all supplementary media and reading interventions compiled from AED-BESO annual reports are included in Annex E, Table 8. As Table 3 below indicates, the project has successfully met contract deliverable targets. The contract deliverable target are obtained from the project contract SOW and the actuals are derived from AED-BESO annual reports (for base years) and program results reports (for option years). b. Production of Instructional Materials BESO-BEP has developed, published and disseminated nationwide self-instructional kits for teachers, socially relevant materials, and interactive radio instruction. In general, the production of the instructional materials followed these steps: • Curriculum analysis is conducted to identify gaps between the curriculum and available teaching and instruction materials. • The MOE and REB staff are invited to a workshop to review the curriculum analysis findings regarding the gaps and are then asked to prioritize what additional instructional material must be developed and produced. • Authors are selected and they prepare a draft. Most material has been developed by international experts in collaboration with local experts from the MOE, universities and the private sector. • The draft material is published and distributed to regions and for MOE to review and comments. • The draft is piloted in selected primary schools. • The draft is revised on the basis of the feedback received from MOE, REBs, and primary schools. • The revised materials are published and distributed to schools. • A guide on how to use the material is prepared and workshops conducted, during which the guide is distributed. • Develop an monitoring and evaluation procedure for tracking and evaluating the effectiveness of the instructional material. • The instructional material is integrated into the Teacher Development Program. c. Self-Instructional Kits for Teachers Self-instructional kits for active learning, continuous assessment, and large classroom management were found in 75% of the BESO-BEP schools visited for this evaluation. Principals and teachers said that they were extremely useful and had had a real impact on the skills, attitudes, and behavior of the teachers. In most of the schools, the kits appeared to be used and principals and teachers reported that they were. BESO II – FINAL EVALUATION 20 TABLE 3: PERFORMANCE SUMMARY - SUPPLEMENTARY MATERIALS Base Yr. One Base Yr Two Option Yr one Option Yr 2 Option Yr 3 S/N Contract Deliverables Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date 1. Existing curriculum Reviewed and revised based on school feed back on key socially relevant topical (reported by topic, grade and language of instruction) Review of G 1-4 curr. Completion of regional curr assessment in cohort target regions Target met Review of G1-4 curriculum completed & revised regional curr assessment completed in school 1 tar gate Review of G5-8 curr, Regional curr. Assessment completed in 2 cottager regions Target met Review of G5-8 and regional curr. Completed in cohort regions Nothing planned Develop monitoring guide collect data and review cycle 1&2 materials Target met Monitoring guide developed, data collected and reviewed cycle 1&2 materials 2. Curriculum on identified socially relevant topics integrated and proto type materials media produced (reported by topic grade level and language of instruction) Develop modules on socially relevant topics integrated washroom curr for G1-4 for cohort 1 regions Target met Developed the modules for G1-4 for cohort 1 regions Develop modules on society relevant topic to be integrated in to the classroom curriculum or cohort 2 regions 3 smart Arts Award per cohort 1 regions Target met Develop and integrated the modules into the curr for G 5-8 cohort 1 region and G 1-4 for cohort 2 regions 3 smart arts awards per cohort 1 regions Modules on socially relevant topics for G 5-8 to be integrated in to the classroom curriculum for cohort 2 regions & distribute 10,000 copies Target met Modules for G 5-8 integrate into the curr for cohort 2 regions & distributed 10,000 copies Revise modules on social relevant topical for 1-4 & print item in 10,000 copies Target met Revised modules and printed them in 10,000 copies Revise modules on socially relevant topics for G 5-8 & print them in 10,000 copies Not due BESO II – FINAL EVALUATION 21 Chart 7: Presence, Use, and Storage of Self-Instructional Kits Used Where kits are stored Type of School Kits at the school Reported Evidence Principal Dept Library Pedogog Center 12 8 8 6 3 3 2 Satellite 9 5 4 2 1 3 3 non-BESO 2 2 2 In many of the schools there was evidence of use: charts for when specific topics in the kits were to be discussed, study groups who met weekly to work with each module, and/or sign out records indicated when the kits were taken out of the library or pedagogical center. But there were also schools where teachers said the materials were great, but that they did not use them because they did not have time. Some teachers described a lack of a “reading habit” and said that training was a more effective way to learn than the use of reading materials. The most consistently reported use of the kits was when they were used in conjunction with in￾service teacher training. In one school in Oromia, study groups that discussed the kits each week had stopped meeting this year when the key teacher moved to another school and no in-service training was taking place. In three other schools, one in Amhara and two others in Oromia, the same pattern was found: the key teacher had left, no training had taken place this year, and the new revised self- instructional kits were still in their wrappers. In most of the schools sign-out systems had generally been created for teachers to borrow these materials. Where the kits were plentiful, teachers were allowed to check them out for longer, up to an entire year in one small school, Dubti 4 EST, in Afar where there was one kit for each teacher. There were some indications that the kits stored in the principal’s or department head’s office were more consistently used than those kept in the library or pedagogical center. The value associated with the kits was demonstrated by Barro Primary, a non-BESO-BEP school in Benishangul Gumuz where the principal had borrowed the kits from a BESO-BEP school to share with his teachers. In a BESO-BEP school in Oromia, the kits had been duplicated to be used by more of the teachers. In Afar, teachers described thoroughly reading the kits and supplying feedback for improvements to AED. d. Socially Relevant Reading Materials The socially relevant supplementary materials were less frequently used than the self-instructional kits. This is partly due to the more specialized nature of these materials; all teachers could benefit from the kits, but only teachers involved in teaching the subjects addressed in these materials found them useful. The principal at Dera, Oromia pointed out that only the teachers in grades 5-8 used these materials, while the self-instructional kits are used by all of the teachers. Generally, if the materials fit the needs of the teachers, then they are used. If the evaluation team happened to interview a teacher responsible for lessons in civics, then about 60% of the time that teacher was aware of the BESO-BEP booklet on that subject and had used it for lesson plans and course content. But many teachers teaching those subjects did not know of the booklet, including an instructor who taught civics at Debre Berhan TEI. The same is true for the materials on General Science and Environmental Education. If the teacher is aware of the BESO-BEP material on that subject, then they do use it. The evaluation team found that about half of the teachers in charge of discussions about HIV/AIDs with Girls’ Clubs who they interviewed were aware of the BESO-BEP materials or had used them. However, there are many materials on HIV/AID now available at schools and teachers cannot generally identify which are those produced by BESO-BEP. In two schools where separate libraries had been created for female students, the HIV/AIDs materials as well as the BESO-BEP modules on gender issues were found there. BESO II – FINAL EVALUATION 22 Chart 8: Presence, Use, and Storage of Socially Relevant Reading Materials Used Where materials are stored Type of School Materials at the school Reported Evidence Principal Dept Library Pedogog Center 7 7 2 1 1 4 1 Satellite 3 3 1 1 1 As the chart above indicates, the socially relevant materials are more likely to have been placed in the school library than were the self-instructional kits, probably due to the perception of them as general reference materials. Unfortunately, among the schools visited for the evaluation, school libraries were found locked and the person with the key missing more than half of the time, which suggests less availability of these materials. Generally the only evidence of use was whether the booklets physically looked used. Generally the material covered in these booklets is seen as useful, of the correct level of complexity for primary school, and easy to understand. However, some of the materials may not be equally appropriate for all regions. For example, two of the teachers interviewed at Haysema primary in the Afar region pointed out that the Environmental Education booklet did not fit their environmental issues. e. Interactive Radio A model for how to create interactive and participatory radio programs was created by BESO-BEP and integrated into the already existent MOE radio instruction system. The major contribution of the BESO-BEP intervention has been the production of a model for a participatory methodology that can be used with instruction. This model has been adopted by the MOE units responsible for producing other radio programs. The BESO-BEP approach to support for media learning has transferred ownership to the MOE and, consequently, built in sustainability. The primary subject of the interactive radio programs produced by BESO-BEP has been English language, a subject which can benefit from radio presentation more than almost any other. Interactive English programs have been delivered to radio stations and teacher’s radio guides were distributed through regional offices. In the schools visited by the evaluation team, English language interactive radio classes were taking place according to several patterns. In two schools, students from a number of classes were collected to jointly listen to and participate in the lesson. It was clear from watching such classes that the students were enjoying themselves. In about a third of the other schools, radios stored in the principal’s office were frequently being borrowed and returned during interviews with principals as teachers used them individually with their classes. In about half of these schools, cassettes were used rather than depending on the radio broadcast. In the remaining schools, teachers did not report using interactive radio lessons, generally saying that this was due to poor broadcast reception and the lack of cassettes. In those schools where interactive radio was being used for English instruction, teachers reported great appreciation for the programs. They said that the use of interactive radio made the task of lesson planning easier and that their students enjoyed the interactive radio classes more than English classes without radio. One teacher admitted that she felt her students received a better lesson than she could offer due to her insecurity about her own English abilities. f. Distribution of Materials The supplemental reading materials produce by BESO-BEP were most frequently reported as having come to schools through principals and key teachers returning from BESO-BEP training. Variations in what principals and key teachers choose to bring back could account for some of the variation in what materials are actually BESO II – FINAL EVALUATION 23 found at the schools. While most trained principals and key teachers appeared to have made a solid effort to make these materials available for the teachers in their cluster, there were schools in which the evaluation team could not find the materials, especially the socially relevant booklets. Turn over in principals and key teachers could have contributed to the absence of these materials. On the other hand, the advantage of having principals and key teachers bring the materials to the schools after their training is not only logistical, because it helps to insure that the materials become linked to the in-service training trained personnel will be supplying. As the chart below indicates, in some cases, the woreda has distributed some of the BESO-BEP supplementary materials to schools, in other cases the materials all came from BESO-BEP directly, and in some cases from both. The principal of a school in Afar reported that she had contacted the woreda when she felt that her school needed more self-instructional kits and they had supplied them. The principal of one school in Oromia had gone by car to the woreda office to bring supplementary materials back to his school. In Amhare, the Basona Werana woreda received many self-instruction kits from BESO-BEP without instructions for distribution. They delivered one set to each school in the entire woreda. Chart 9: How BESO-BEP Supplementary Material were Provided to Schools *Region Woreda Number of schools reporting supplied by BESO-BEP Number of schools reporting supplied by woreda Number of schools reporting supplied by both Afar Dubti 1 3 1 Awash Fontale 1 Amhara Basona Werana 2 2 1 Oromia Dodota Sire 2 Boset 2 Holeta Town 2 2 2 Total 10 6 4 *This information is not available for all 26 BESO-BEP schools in all regions and woredas. g. Contrast to non-BESO-BEP Schools Perhaps the most significant and consistent difference between BESO-BEP schools and other schools lies in the amount of resources, such as these supplementary materials, that can be found in BESO-BEP schools. This difference was easily observable during the visits of the evaluation team because most schools other than BESO-BEP have an extreme lack of reading or media materials. In interviews conducted at non-BESO schools, teachers most often mentioned the reading materials supplied by BESO-BEP as what they would have most appreciated if they had been a project school. h. Sustainability As indicated in the outline of steps in the development of the BESO-BEP supplemental reading and media materials, MOE and REB staff are involved in the selection of what is to be addressed throughout the process. Interviews with MOE and REB personnel indicated a strong sense of ownership of these products. The involvement of these levels of the educational system from the beginning not only helped to focus the materials on what were perceived as the greatest needs, but also should help to establish sustainability of the materials through the sense of ownership this has been generated. Desire for materials such as the ones produced by BESO-BEP and an extreme need for such materials, could make this component sustainable. Perhaps the Debre Berhan Center for Excellence, or additional centers on the same model, could take on the publication of the materials. However, the logistics of distribution to schools and storage at the schools can create other barriers. A detailed plan for how the materials should be stored and circulated within schools and clusters could assist schools in making the materials more available and utilized. A systematic, consistent pattern of materials distribution from the woreda could reinforce links between schools and woreda and assure that all schools receive an adequate amount of materials. While the BESO II – FINAL EVALUATION 24 history of distribution of BESO-BEP materials from woreda offices has not been good and real barriers to distribution exist due to the long distances to many schools and the unavailability of vehicles for covering those distances, a number of woredas were distributing the project supplementary materials to the schools, sometimes both BESO-BEP and non-BESO, within their area. 2. CONCLUSIONS It is clear that there is a strong need for supplemental media and reading materials in Ethiopian schools. In all of the schools visited, appreciation for the BESO-BEP materials was strongly stated and the presence of more reading materials was one of the major differences between BESO-BEP and non-BESO schools. While the evaluation team found variation among schools in which media and reading materials were used and how assessable they were to teachers, at least some of the materials were being used at every school. Where the materials were stored and the degree to which teachers had been informed about their presence also influenced utilization. Primarily, use of the materials supplied by BESO-BEP tends to depend upon how relevant they are to the needs of the teachers. For example, if the materials match the subject that a teacher teaches, then they became important enough to read and use. If English is being taught, then participatory radio approaches make that instruction easier and more enjoyable for both teachers and students. The supplementary materials alone cannot create mastery of the skills and knowledge presented but, when the materials are linked to in-service training on the same topics, and especially when combined with continuing study of the materials by teacher sub-groups, then the materials appear to be effective in increasing teacher skills in the areas being addressed. Getting the supplementary materials into the hands of the teachers who can best make use of them can be an obstacle. In order to expand this program to additional schools, more than just publication will have to be considered because distribution, linkage to training programs, and storage can all produce barriers to utilization. Woreda education officials would be the logical choice for distribution and, although the experience with woreda distribution of BESO-BEP materials has not always been good, almost half of the woredas that the evaluation team had data about did distribute BESO-BEP reading materials to schools in their areas. 3. RECOMMENDATIONS • The Center of Excellence publishing facility at the Debre Berhan TEI offers a successful model for how the supply of reading materials could be expanded through local production. The creation of one center for publication in each region would not only increase the amount of material, but also increase interest in local authorship as it has at the Debre Behan TEI. • Guidance in who should receive each type of reading material and where it should be stored after reaching the schools might increase utilization. D. STRENGTHENING LEADERSHIP AND PLANNING AND MANAGEMENT CAPACITY 1. FINDINGS a. AED Contractual Obligations Over the last few years, decentralization of the education system in Ethiopia has deepened. Responsibilities and resources for managing basic education have devolved from the regional state government to woredas and kebeles. Such deepening of education system decentralization requires massive capacity building for effective planning and management of the education system at all levels. The BESO-BEP project has launched various activities that are aimed at strengthening and increasing (a) leadership skills, (b) education planning and management capacities; and (c) school leadership and supervision. BESO II – FINAL EVALUATION 25 As shown in Table 9 in Annex D, project contract deliverables are focused on the provision of key management tools and the skills and capacity to use them. These management tools include: • Computerized personnel management information system for effective education human resource planning and management • Computerized instructional materials procurement and distribution systems or materials management information systems to efficiently manage the distribution and logistics of education resources • Promoting the monitoring and evaluation of the performance and quality of the education system by conducting periodic assessments • Education management information system and projection model to facilitate the use of education information for planning, budgeting and decision-making • Leadership and school supervision training to KETB members and school principals to enhance the supportive role of school principals and promote and foster team-based school management. Progress on contract deliverables is summarized in Table 4 below. Although progress was made on contract deliverables, the project has missed most of its output targets. The reasons for missed target are, in each case, included in the table. b. Computerized Information Management Systems BESO-BEP has committed a lot of resources to promote the use of computerized information management systems for planning and budgeting and effective management of the educational system. For example, the development and implementation of the Personnel Management Information System (PMIS) at the central Ministry of Education (MOE) institutions, regional and woreda levels, which computerizes employee’s files in the conventional archives is designed to increase capacity for effective planning and management of the education system human resources. As summarized in Table 2, there has been progress in the development and implementation of the software at the central MOE institutions and Regional Education Bureaus and selected woredas. Progress on the implementation of the Material Management Information System at the national, regional and woreda aimed at improving the management of the procurement and distribution of educational materials is hampered by the continuing decentralization process and relocation of stores and warehouses. The MMIS which was expected to be functional in eight regions by the end of Option Year One is yet to achieve that goal. According to information gleaned from the evaluation teams’ interviews and observations the implementation and use of the Educational Management Information System and the Projection Model has been slow due to modification being made to the projection model and high staff turnover. BESO II – FINAL EVALUATION 26 TABLE 4: PERFORMANCE SUMMARY- LEADERSHIP AND CAPACITY BUILDING Contract Base Yr 1 Base Yr 2 Option Yr 1 Option Yr 2 Option Yr 3 Deliverable Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date 1. Number of regions that (a) have developed; (b) are maintaining and using computerized personnel system for at lease basic personnel functions (hiring, assignment, salary & benefits, incremental increases PMIS with min. of 13 functions installed at 11 regional HQs. Prelim. Database set-up started in all the above locations Target not met: Design and development of the application software and the development of 15 data collection formats was completed for the MOE, central institutions and for eight regions Prelim. Database set￾up completed in all the locations started in Year 1. Procedures for maint. established. Maint. all locations in progress with evidence of PMIS utilization Target not met. The PMIS database is completed and handed over only to 3 regions. Also, the PMIS for MOE central institutions was completed Complete database in remaining 3 regions Computerized PMIS established in 5% of woredas in the regions and functioning Target not met. However, Implementation of PMIS database in Oromia, Somali and SNNP completed; out of 30 woredas the implementation of computerized PMIS is completed only in 10 woredas Computerized PMIS established in 5% of woredas in the regions and functioning Target not met. 85% completed; delay in the procurement of computers and related accessories, longer time required for reconciling data collection, delay in the training core group Not available PMIS has been installed at the central MOE institutions, all regions and at handful of selected woredas. However, the evaluation has found no evidence that the PMIS is being used as personnel and human resources management tool. 2. Number of regions that (a) have developed; (b) are implementing improved instructional materials procurement and distribution systems on a MMIS installed in 11 regional HQs. Prelim. Database set-up started in all locations. Study of procurement process completed. Design of a Target not met Implementation of the new/modified procurement process completed in 11 regions. New /modified distribution system provided to 11 regions for implementation Target not met. However, systems analysis and design wa s completed and standard data collection formats developed and were Study completed and recommendations for strengthening the procurement and distribution process provided Targets not met; the planned activity was canceled In many regions, the MMIS database, which sensitive to store location, is under revision. In SNNPR the system is up and running and being BESO II – FINAL EVALUATION 27 Contract Base Yr 1 Base Yr 2 Option Yr 1 Option Yr 2 Option Yr 3 Deliverable Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date routine basis commodities distribution system for the regions completed tested, updated. 3. Number of regions that are undertaking improved learning assessment and other monitoring of educational quality at the classroom and school levels 2 regions undertaking learning assessment Target met. The First National Learning Assessment was completed. 5 regions undertaking learning assessment 1 national assessment Target met. The Second National Learning Assessment was completed 1 national assessment undertaken BESO-BEP has completed several policy study papers. 4. Number of regions utilizing education information more efficiently for planning, budgeting, monitoring and evaluation All 11 regions provided 1 annual planning tool and a decision support tool Target not met. Concept of IPBCS, the decentralized planning mode, was introduced in 8 regions All 11 regions provided with 1 MQS and 2 additional decision support tools. All regions using a long range perspective plan model for developing UBPE strategies Target not met. However, long-range education planning projection model with base-year data was installed in computer used by the woredas. - All regions utilize planning models - Assessment of woreda capacity training program undertaken - woreda capacity building modules reviewed and updated - REB woreda capacity building trainers’ capacity strengthened Target not met. However, orientation on how to use the planning model was given to Oromia, Somali, SNNP, Afar, Benshangul, Dire Dawa and Harari REB professionals - WCB assessment was completed for 62 woredas Capacity of education officers of 25% of the woredas in the regions strengthened Target not met. 85% completed; some delay in the translation of training material Capacity of education officers of 25% of the woredas in the regions strengthened EMIS and the Project Model installation has been slow. However, regions visited have indicated that they now are able to develop pedagogically￾oriented mid￾term education plans. BESO II – FINAL EVALUATION 28 Contract Base Yr 1 Base Yr 2 Option Yr 1 Option Yr 2 Option Yr 3 Deliverable Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date - revie and updating of WCB training modules was pushed to Option Year Two - The training of REB WCB trainers was also pushed to Option Year Two 5. Number of schools and Kebele Education and Training Board (KETB) with increased capacity in supervision, planning, etc. No target Nothing to report No target Nothing to report - assess needs of KETB and school leaders - training material developed - training conducted for 300 members of KETB and school leaders Completed – Target met: -training needs assessment of KETB in sample woredas was completed - 5 KETB training modules were reviewed and updated - Training was given to 465 KETB members nationwide Training conducted for 400 members of KETB and school leaders Target exceeded; actual trained = 584 Training conducted for 500 members of KETB and school leaders Data not available The Woreda Capacity Building training and the KETB school leadership training have been rated as very useful and effective by persons interviewed by the evaluation team 6 Number of school headmasters, regional zonal and ) 1,500 headmasters and Regional Zonal & Woreda . Target exceeded 1,545 headmasters and 634 Train 3,500 headmasters and 800 Regional, Zonal and Target exceeded 4,437 (female = 95) Train 1,200 headmasters and 600 woreda offices . Target exceeded 2,336 headmaster (252 females) Train 1,200 headmaster and 600 woreda offices Target exceeded 4,816 headmasters (396 . Train 1,200 head maxes and 600 woreda offices Not available BESO II – FINAL EVALUATION 29 Contract Base Yr 1 Base Yr 2 Option Yr 1 Option Yr 2 Option Yr 3 Deliverable Planned Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to Date zonal and woreda education support personnel trained in constructive teacher support methods (reported by region /zone, by type of personnel trained and by gender offices will be trained and 634 regional, zonal and woreda education officers trained Zonal and woreda offices 95) headmasters and 756 (female = 17) education officers trained (252 females) and 1,103 educational offices (116 females) were fromed. offices (396 females) and 2,658 education offices (157 females) were framed offices BESO II – FINAL EVALUATION 30 Chart 10: Evaluation Teams’ Interview Findings and Observations Education Management Information Systems (EMIS) Personnel Management Information Systems (PMIS) Material Management Information Systems (MMIS) MOE The system is up and running. The projection model has been modified and installed. The software has been installed and personnel data has been entered. According to MOE department heads the system has been installed and functioning. Regional Education Bureaus In SNNP the projection model has been loaded but is not yet operational due to needed modifications. In Afar, Benishangul Gumuz and Amhara regional state education bureaus the projection model has yet to be installed and tested. The software has been installed at all the regional offices the evaluation team visited. Regions are still in the process of entering employee records into the database. The evaluation team was not been able to obtain management reports generated from the database. The Amhara REB indicated that the management reports are developed manually. Benishangul Gumuz and SNNPR were still in the process of validating the data encoded into the database as a result of hugh staff turn overs. In SNNPR the MMIS was being updated from hard copies of Model 19. According to the person responsible for operating and updating the database he has not yet been asked to generate any type of management report. In Amhara the MMIS software has been installed but is being modified as a result of further decentralization of woredas. In Benishangul Gumuz, the evaluation team was told that installation of the MMIS is still in the planning stage. Woredas Education Offices The EMIS and Projection Model are still in the planning stage. Most of the woredas visited by the Evaluation Team indicated that they developed their 5-yr education plans, using handheld calculators, during the workshop on strategic planning. PMIS has been installed in selected woredas, e.g., Awassa City Administration, Bahir Dar City Administration and Gonder City Administration are in the process of encoding the backlogged employee data into the database. In Amhara region, Bahir Dar City Administration the use fo computerized information management is about a year old. MMIS is expected to b einstalled targeted woredas. c. Planning and Management Capacity Building Training Based on an assessment of training and technical support needs of the educational planning departments of the Ministry of Education and the Regional Education Bureaus a series of planning and management capacity building training were provided in 2006. Consequently, a series of three seminars and four workshops were conducted from January through October 2006. Three seminars were conducted between January and March 2006. Seminar 1 dealt with the role of a planning officer in decentralized context. Seminar 2 dealt with strategies to improve access, quality and equity of primary education. Seminar 3 was focused on leadership and management. Also three workshops were delivered between April and October 2006. Workshop 1 was directed at using data for developing a strategic plan; workshop 2 dealt with ensuring successful implementation of regional plans, and workshop 3 was directed at designing and using an indicator system for monitoring education. In seminar 1 and 2, staff from MOE planning department, REBs bureau heads, deputy bureau heads and heads of the planning department and planning experts participated. In seminar 3, REBs bureau heads and deputy bureau heads participated. In workshops 1, 2 and 3, MOE planning department staff, REBs planning department heads and staff participated. According to persons interviewed by the evaluation team, except for Seminar 3, Seminar 1 and 2 and all the workshops were very well organized and the delivery was effective. In SNNPR, the head of the planning department indicated that workshop 1 and 2 were relevant and useful for him. Similar comments were received from Amhara and Benishangul-Gumuz regions. Seminar 3 received a relatively low rating. According to interviewees, although biographical excerpts leaders such as Nelson Mandela, Gandi, etc was provided, the participants did not receive the copy of workshop presentations to fellow the workshop. They also indicated that the delivery of the workshop material lacked the dynamizium of the previous seminars. .The workshop BESO II – FINAL EVALUATION 31 material was later complied and delivered to the participants. In fact, the AED office in Addis Ababa did not have a copy of the workshop material for the evaluation team. The evaluation team was told that follow-up by BESO-BEP to ensure that the systems and the training provided are being put to use is lacking. Without adequate follow-up, frustration builds and users of the management tools give up or may resort back to older methods. Furthermore, in all the regions visited there is no backup system or procedure. That is, if the person in charge is absent, the system stops functioning until his/her return. For example, in the SNNP and Amhara regional education bureaus there was just one IT professional per bureau to maintain the computers and the local area networks. d. Woreda Capacity Building Training Woreda Capacity Building Program is a training program designed to strengthen woreda education and capacity building offices around Ethiopia to contribute towards effective provision and expansion of quality primary education and achieve the goal of Universal Primary Education (UPE) by 2015. The primary objective of the program is enabling woreda education offices to produce a 5-year and annual educational plan. To enable this, BESO-BEP produced a workbook that contained 16 modules organized in three parts: planning and managing a decentralized education system, preparing woreda education plan, and woreda educational plan implementation. The material was translated in four local languages and distributed to all regions. In 2004, 57 trainers were trained from all regions and 3,119 woreda education and capacity building officers and experts from 600 woredas around the country were trained. Between 2004 and 2006 a total of 3,200 woredas education officers were trained. In 2006 additional modules – ‘Enhancing Girls’ Education’ and ‘Mobilization of Resources to reach UPE by 2015’ were added. In the second round of WCB training a total of 589 male and 27 female woreda education officers were trained on all the 17 modules. An impact assessment conducted by USAID/BESO II using a sample of 62 woredas indicated that almost 67% of the woredas had prepared an adequate educational plan documents. In some regions such as Amhara and Oromia the proportion of woredas with adequate plan document was high (between 83.3% and 86.4% respectively). In Benishangul Gumuz-Gumuz, Harari and SNNP, about 50% of the woredas had produced adequate education plans. The assessment result for Gambella (0% adequate) and Tigrai (25 % adequate) is very low. According to training participants the Evaluation Team interviewed, interactive and participatory training approach gave life to the training and gave direct hands-on experience to each trainee. They indicated that their education plan was developed during the training workshop and later refined with assistance from REB planning experts. However, many of the trained woreda education and capacity building officers had left their positions. In addition new woredas were created requiring additional trained education officers. The critical mass of trained personnel was not achieved and there are shortages of trained personnel in all the regions the evaluation team visited (SNNPR, Amhara, Benishangul Gumuz, Afar and Oromia). Interviewees, that included woreda education office heads, education planning officers, woreda supervisors, indicated that the WCB training has improved their planning, and data collection, processing and analysis skills, the selection and use of education indicators as well as improving their education leadership and supervision knowledge and practices. They in particular indicated that the projection techniques, identification and use of education indicators and preparation of education plans they learned from BESO-BEP were very useful. They also indicated that they now understand what educational leadership and supervision involves and how it should be applied. e. School Leadership Training School principals in all of the schools visited by the evaluation team reported great satisfaction with the leadership training they had received from BESO-BEP. They reported that they used the documents from that training constantly in their preparation of school plan. When comparing the strategic plans they have produced since receiving training to those created before training, principals described current plans as more specific, activities were now presented as a sequence of actions, and all plans were accompanied by financial BESO II – FINAL EVALUATION 32 budgets. The instruction in planning skills was reported to have been used by almost all principals who had been trained by BESO-BEP. Principals also reported that the training they had received helped them to identify and prioritize school problems and to base their planning on the actual conditions of their environment. In addition, many principals claimed to have changed their management style due to the training they had received. The difference most commonly mentioned was a shift to a more participatory approach in which the principal invited the entire staff of the school to participate in decision making and planning activities. Kebele Education and Training Boards (KETBs) are responsible for the management and expansion of education at the local level. For each KETB, the members trained include the chairperson of the KETB, who is also chairperson of the kebele, the board secretary who is the principal of the local school and other four members who could be teachers from the local school or members of the local Parent Teacher Association (PTA). The major goal is to enhance the use of active learning and create a suitable environment for teachers and school principals. KETB members and school principals were trained on the following topics. • Community participation and mobilization • School resources mobilization and effective utilization • Education planning and implementation at kebele level • Education leadership and administration at kebele level • Planning, monitoring and performance appraisal at school level A cascading training approach was used to have a multiplier effect. BESO-BEP provided training-of- trainers training to selected regional and woreda education officers who in turn trained selected cluster center schools KETBs. As reorted in the BESO-BEP Woreda Capacity Building final report a total of 628 KETB members have received training. The breakdown by region is summarized below. It should be noted, however, that the training has not reached all regions, woredas and localities Chart 11: Number of KETB Members Trained (Rounds 1, 2, & 3) Region Female Male Total Addis Ababa 9 36 45 Amhara 43 97 140 Beneshangul 3 44 47 Oromia 18 156 174 SNNP 13 103 116 Tigray 20 86 106 Total 106 522 628 SOURCE: Woreda Capacity Building Final Report. The data on Afar was not reported in the Final Report. In the Afar region, where no training of KETB members had taken place, Dubiti woredas staff strongly suggested that training for community leaders be introduced as they had in some instances become obstacles to the introduction of active learning in schools. In the Awash woredas, staff members also urged the induction of KETB training, reporting that communities did not understand how to support the schools. In an area of Amhara where KETB training had not taken place, woredas officers also pointed out the need for this training, explaining that KETBs could do many things which their lack of understanding and knowledge now prevents. Some of the impacts of principal, KETB and PTA training are summarized below. BESO II – FINAL EVALUATION 33 Annual School Plans Budgeting Mobilize Local Community Resources Income Generating Projects Large schools All large schools visited have annual school plan posted in the principal’s office. The evaluation team did not get or see the schools annual budget. Local communities have built classrooms and donated money for furniture None reported Small schools More than 50% of schools visited have annual plans The evaluation team did not get or see the schools annual budget. Local communities have built classrooms and latrines for school staff and students Rural small schools have agricultural projects in collaboration with the local farmers. Urban schools All large schools visited have annual school plan posted in the principal’s office. The evaluation team did not get or see the schools annual budget. Local communities have built classrooms and donated money for furniture None reported Rural schools More than 50% of schools visited have annual plans. The evaluation team did not get or see the schools annual budget. Local communities have built classrooms using locally available material and also built housing for teachers. Agricultural projects in collaboration with the local farmers. f. Progress in Educational Capacity Building The primary purpose of this BESO-BEP activity is to improve the capacity of the education professionals to prepare their longer term, mid term and annual plans. It is also to facilitate the effective and efficient management of human resources and educational materials and assets and further the use of active learning and student-centered teaching. As shown in Table 4, the project has made progress in enhancing the primary education sector planning and management capacities. Regions, targeted woredas and schools are now able to prepare mid-term and annual plans and budgets. The necessary tools and training has been delivered by the project. However, the use of these tools and training needs to be deepened by providing adequate staffing and resources, which is outside the purview of the project. For example, in most of the cases on which the evaluation team attempted to gather data at the TEI, regional education bureau or woreda education office the individual who knew how to generate the requested data from the computerized management information system was not present. In some cases the person who new the formula for calculating, e.g., enrollment rate or primary school completion rate at the regional level (SNNPR), was not present. In the Amhara regional education bureau it took more than two days to manually calculate the requested data although the EMIS and Projection Model databases were installed and supposedly functioning The evaluation team was also told that there is very high personnel turnover. Many of the persons the teams interviewed were in their current position for less than a year. High turnover in regional and woreda offices has limited the effectiveness of BESO-BEP planning and management capacity development efforts. The high personnel turnover has prevented the creation of critical mass of trained IT and educational officers to maintain, operate and use the computerized management information systems. The evaluation team was also told that the frequent restructuring of woredas and decentralization of zones has slowed down the development and use the computerized management information systems; and diluted the number of trained personnel. In addition, the shifting of warehouse and store locations has slowed down the finalization of the MMIS system in several regions including Afar, Oromia, and Amhara. Fewer turnovers are found among school principals than in regional and woreda offices, but it still introduces a limiting factor. For example, according to the school principals interviewed, in cases where the school principal and the KETB chairpersons were trained, they worked together collaboratively. On the other hand, untrained KETB chairpersons were difficult to work with. In addition, according to key teachers interviewed, BESO II – FINAL EVALUATION 34 schools headed by trained school principals functioned harmoniously and as a team. The principal made sure he shared his knowledge and training with the teachers, facilitated the formation of teachers study teams to review instructional kits and supplementary materials, managed in a more participatory manner, and improve teacher lesson planning and teaching. Morale of teachers was also high. g. Sustainability Region Education Bureau heads, woreda education office heads, and school principals have all indicated that they have formed a sustainability committee to find ways and means for sustaining the results and initiatives begun by BESO-BEP. In most cases, these committees were formed more than a month ago. However, to date, they have not produced any recommendations. There is a critical shortage of information technology professionals to maintain the local area networks and computer assets in all of the regions. In all the regions visited, the evaluation team observed that the regional and woreda human resources departments were understaffed. The head of the human resources department was generally responsible for data entry and maintenance of the PMIS in addition to his personnel principal responsibilities. Regional office, woredas, TEIs and schools storekeepers lacked training. Receipt of commodities and withdrawal of items from the store were usually managed by the same individual and, in general, the storekeepers are reluctant to release the commodities to users. This hampers the effective use of the MMIS tools. 2. CONCLUSIONS BESO-BEP has introduced new planning and management tools for education personnel management, education resources management, and education planning. It has trained cadres of educational officers, planners and leaders at all levels to effectively manage the education system. However, high turnover throughout the education system, especially among woredas heads and deputies, has meant that there are still many key positions filled by persons without training. High turnover in regional and woreda offices has limited the effectiveness of BESO-BEP planning and management capacity development efforts. The high personnel turnover has prevented the creation of a critical mass of trained information technology and educational officers to maintain, operate, and use the computerized management information systems. Access to education data remains a major problem even when computer systems have been supplied and persons trained on the database. In many cases, the evaluation team found the person with the training to retrieve the requested data was not present or had moved to another position; if the person in charge is absent, the system stops functioning until his or her return. Leadership training at the woreda and school levels has improved the supervision and management of schools and increased the participation of community members. School principals report that they have used their training constantly in their preparation of school plans, that it has helped them to identify and prioritize school problems, and that they have changed their management style to become more participatory. Improvement in school leadership and supervision has meant improved lesson planning and increased application of active learning and student-centered teaching. It has also meant increased use of continuous assessment and remediation to increase student competencies. Training of community leaders has led to increased support for the school and an increased understanding of active learning. BESO II – FINAL EVALUATION 35 E. SUPPORT FOR FEMALE TEACHERS 1. FINDINGS a. AED Contractual Obligations The purpose of this component of BESO-BEP is to enhance the capabilities of female teachers and promote their participation in educational leadership. Historically Ethiopian culture has assigned to women a subordinate role, which has frequently led to over-burdened and difficult lives, little educational achievement, and a relative lack of self-confidence. BESO-BEP has produced modules on gender issues which have an impact in an educational context such as how to promote gender sensitive classrooms, enhance female teachers participation in school and community activities, and a self management guide for female primary school teachers. The project has provided community training in awareness of negative cultural practices such as rape, female abductions, and early marriage. In addition, leadership training has been offered to female teachers who are perceived as being ready and eligible to be promoted to leadership positions. A full list of all of the interventions and deliverables associated with support for female teachers, which has been derived from AED-BESO annual reports, is included as Table 10 in Annex E. As Table 5 below indicates, the project has met most of its targets, yet an insignificant number of female teachers have been hired for leadership positions. AED reports some large increases in the percent of female leaders in four regions during option years 1 and 2. The information about targets planned and achieved is compiled from the project contract SOW (targets) and AED-BESO annual reports (actual for base years) and program results report (actual for option years). STATISTICAL DATA ON FEMALE LEADERS AND % OF INCREASE BY REGIONS - OPTION YEARS I AND II Region 1997 1998 % Increase Remarks Addis Ababa 65 65 0 Afar 2 6 200% Extremely small N Amhara 433 400 decrease Benshngul Gumuz 18 18 0 2 promoted to the House of representative Dire Dawa 5 5 0 Gambella * * * *No data available due to security reasons Harari 15 22 46.7% Oromia 223 ** ** **No data 2nd year Somali 4 4 0 SNNRR 116 201 73.3% Tigray 30 69 130% Total ***688 790 15% ***cannot include Oromia data for one year without the other BESO II – FINAL EVALUATION 36 TABLE5: PERFORMANCE SUMMARY - FEMALE TEACHER AND STUDENT-TEACHER SUPPORT Base Yr 1 Base Yr 2 Option year 1 Option year 2 Option year 3 S.N Contractual deliverables Planne d Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to date 1 Number of TEIs that demonstrated a commitment to increase female graduates (disaggregated by certificate and diploma cources) through specific policies introduction of new mechanisms of support or other concrete verfiable actions 15 TEIS 10 TEIs in certificate program -Strengthen Amhara and Oromia regional network -Establish the 3 networks -Organize national workshop on sensitization for policy makers, Ed. Officials and other partners For female students -Amhara and Oromia regional networks strengthened -The three networks for the five regional TEIs established - A national sensitization workshop was conducted for higher officials of the MOE, Regional education bureaus and partners 10 TEIs in the certificate program increased the enrollment of female students and no information from the TEIs in diploma program. Targets partially met Based on the regional policy a 30% quota in favor of female enrollment in the TEI has been implemented eg. in Arssi zone - Sensitization workshop conducted. Target met Observation No tutorial classes for female students in the TEIs were occurring at the time of team visits. 2. Percentage increase of female teachers in primary school leadership positions as directors and deputy directors Develop and produce materials and modules Target met : 6 courses and 4 modules develop￾ed Leadership training for 550 female teachers from all regions. 9 set of Posters production and distribution Target not met: 452 female teachers trained Target met: Produced and distributed to 300 CRCs, 22 TEIs 11 regional ed. Bureaus, and Addis Ababa abd Dire Dawa administrations and partners. 6 courses and 4 modules developed and produced and distributed for use in the TEIs and cluster schools. Target met. 82% of the training target met which is high The target for poster production and distribution has been met. Data assessment shows improvement in the increase of female leaders during Option I and II years. BESO II – FINAL EVALUATION 37 Base Yr 1 Base Yr 2 Option year 1 Option year 2 Option year 3 S.N Contractual deliverables Planne d Actual Planned Actual Planned Actual Planned Actual Planned Actual Progress to date Observation Insignificant number of female teachers promoted to come to leadership positions. 3 Instructional and /or motivational short courses, training modules (Audio, Video, Print) targeted at women teachers developed produced and delivered (reported by topic type of media and language) 4 Short courses 2 Trainin g modules Target met 6 Teach ers modul es develo ped and being transla ted into 4 langua ges 8 short courses 4 training modules Target not met 6 Teacher module s develop ed and translat ed into 4 languag es Develop 4,000 copies each of 6 short courses & 4 training modals Target exceeded 34,689 copies of 6 short course & 4 training modules produced Project has achieved its deliver￾ables BESO II – FINAL EVALUATION 38 b. Develop Materials on Gender Issues BESO-BEP has produced modules on gender issues that are especially important in a school context. The modules provide useful information and frequently include plans for conducting training through which to pass on the information found in the materials. All of the modules are impressive in spite of a tendency to reduce the effectiveness of the modules by pointing to the gender problems in a few cultures as examples, rather than asking readers to describe and discuss limitations and problems created by gender beliefs in their own experiences and cultures. In all of the schools where the modules were found, there appeared to be a genuine appreciation for them and, in some schools, Girls’ Clubs or other female oriented organizations had made use of the modules. For example, at one school in Oromia the modules were being read aloud to female students by female teachers. However, there appeared to be a problem in getting the booklets into the hands of the persons who could most use them. Although these modules were sometimes kept in a room used by the Girls’ Club or female councilors, as with the other supplementary materials supplied by BESO-BEP, these modules were also often found in libraries where they were not easily assessable. The agendas of topics covered in Girls’ Clubs presentations at schools generally dealt primarily with traditional gender topics, such as early marriage, harassment, and abduction, rather than the more subtle and school based issues addressed in the modules. Fifteen radio programs, the themes of which were extracted from the ten modules and teacher leadership courses, were produced and sent to regional radio stations for broadcast. In addition, these programs were produced as cassettes and sent to 289 Center schools and 22 TEIs. The goal of these broadcasts and cassettes was to make the information in the modules and training courses more accessible to teachers and students. However, among the schools visited by the evaluation team, none of the teachers or principals, when asked about BESO-BEP supplementary materials or support for female teachers, mentioned these materials as having been used at their school. c. Leadership Training for Female Teachers AED reports that 2382 female teachers have received leadership training over the last three years in eleven regions. Chart 13: Female Teachers Trained in Leadership Skills by Region Region 2004 2005 2006 Addis Ababa 272 44 48 Afar 57 28 15 Amhara 103 272 88 Benishangul Gumuz 40 19 Dire Dawa 20 12 Gambella 24 15 Harari 20 13 Oromia 59 424 118 Somali 40 12 SNNPR 117 166 127 Tigray 180 49 TOTALS 608 1258 516 Of those trained, 77% are from Addis Ababa, Amhara, Oromia, and SNNPR. The leadership training for female teachers offered by BESO-BEP appears to have been valued by all who have participated. One-fourth of woreda officers, school principals, and teachers who were asked for recommendations about this component of BESO-BEP suggested providing leadership training for many more female teachers. Over 75% of the woreda officers who were interviewed suggested an increase in the number of female teachers receiving this training, in part because of the difficulties they have encountered as they BESO II – FINAL EVALUATION 39 attempt to hire and/or promote female teachers according to the government’s current affirmative action policies. Many woreda officers reported having offered leadership roles to female teachers who had turned down this opportunity for career advancement. The reasons given by the female teachers generally included a desire to avoid more responsibility, both because of their already overburdened schedules due to home responsibilities and because of a lack of confidence. In addition, many changes in position would have involved moving from one area to another, which is more difficult for women than for men. As a result, an insignificant number of women teachers have been promoted to leadership positions in primary schools. Woreda officials feel that the leadership training could assist female teachers in developing more self confidence and, therefore, make them more likely to accept leadership roles being offered to them. In spite of the number of female teachers AED reports as having received training, the data collected by the evaluation team found on average less than one female teacher in each BESO-BEP cluster who had received this training, and very little training appears to have occurred this year. Of the 37 schools visited, only eight reported that one or more female teacher had received leadership training. In many of the schools where a female teacher had participated in this training, she, or other teachers, reported some attempts to pass on what she had learned to other female teachers in her school and/or cluster. However, no systematic arrangements were made for sharing her experience. In four of the eight schools where a female teacher had been given leadership training, that teacher had moved away from the school to take a new position Given that the criteria for receiving female leadership training is readiness to move into a leadership position, frequent turn over among the female teachers who have received training should have been expected. The training fulfills its purpose of training female teachers for leadership positions, but an opportunity for spreading the content of that training to other teachers has not been missed. Chart 14: Female Teachers Receiving Leadership Training in the 37 Schools Visited by the Evaluation Team Schools in which: Afar Amhara Beni￾shangul Oromia SNNPR Number of female teachers who had received training 2 3 .0 7 2 Trained female teachers passed on training 1 2 .0 4 .2 Trained female teacher left the school 0 1 0 3 0 The criteria for how the teachers are selected for this training are not transparent to the female teachers in the schools. This can lead to suspicions of favoritism when the teacher trained is neither offered a leadership position nor has played a major role in support of female students at her school. The content of the female leadership training is appropriate for the desired result. However, much of the content consisted primarily of raising awareness of the problems to be overcome without offering many specific strategies for how to overcome them. One example of a specific strategy that was said to have come directly from BESO-BEP female leadership training was found in two Amhara schools in the Debre Berhan area. This strategy for helping female students remain in school involved the creation of a “modesty” room for girls, offering privacy, sanitary napkins, and underpants for girls when their periods began during school time. Female teachers and sales by female students had supplied the funds for the rooms and the girls at the schools had supplied the labor to construct what were rapidly evolving into girls’ clubhouses. d. Contrast to non-BESO-BEP Schools Other than the presence of a few female teachers who have received leadership training and the presence of modules on gender issues related to education, there appears to be little difference between BESO-BEP and non-BESO-BEP schools in their support for female teachers and students. Girls’ Clubs, teacher counseling of female students, and female teacher associations were found in all schools visited by the evaluation team. BESO II – FINAL EVALUATION 40 e. Sustainability In all of the schools visited, optimism about changing gender roles and excitement about what women could and were doing was found. In Afar, all of the female teachers in the Dubiti area had created their own association, which meets regularly to discuss problems and raise money for poor female students. In Benishangul Gumuz, the Bambasi woreda sponsors a women’s empowerment committee, which works with female teachers and students. Everywhere there is recognition of gender issues and a will to create change. This mood can help to sustain change and BESO-BEP gender related interventions are appreciated, have been appropriated, and are seen as supporting the changes which are occurring. But the interventions are also perceived as too small to have much impact. Female teachers who have had BESO-BEP leadership training frequently suggested that the program be expanded to more teachers, both to overcome a lack of confidence, which often prevents female teachers from accepting leadership roles, and to inject into the system concepts and strategies which will be passed on to female students from their teachers, creating a deep and lasting change in the next generation. A system for continuously training female teachers, either through the woredas or the TEIs, would not only sustain the gains already made, but could expand them to more female teachers and the female students who they influence. As in the case of the supplementary reading materials, the modules on gender issues for female teachers can provide a very useful source of information if they reach the right hands. An effort to direct the distribution these materials to the female teachers who act as advisors for female students and lead Girls’ Clubs, where they are needed and will, therefore, be used, would do a great deal to integrate these materials into the system and increase their utilization. One school in Oromia is currently working with an NGO to see if further duplication of these modules might be possible. The sustainability of supports provided by BESO-BEP to TEIs are questionable. Control of the resources and supports provided by BESO-BEP has been handed over to the TEIs to utilize, a strategy which should produce ownership and, consequently, sustainability. However, instructors at all of the TEIs visited by the evaluation team reported that they currently have too little time to utilized these interventions and none of the four the third year female students who were interviewed reported that they had received any of the supports developed by BESO-BEP other than two who had received tutoring. On the other hand, the various new institutional bodies, such as women’s associations, gender offices, and guidance counselors could have a long-term impact. 2. CONCLUSIONS The leadership training BESO-BEP offered in support of female teachers is well conceived and appears to be fully appreciated by those who have participated. There is a reccurring suggestion from both female teachers and woreda officers that substantially more female teachers should receive female leadership training, both to increase the number of female teachers who are willing to take on leadership roles and to further integrate the concepts into the education system. However, at the present levels of training, and without some motivation for sharing information within schools and/or clusters, substantial changes in attitude and behavior cannot be expected. The gender modules supplied by BESO-BEP are valued by those teachers who have access to them, but often do not reach the people who could make the most use of them, such as female student councilors and leaders of Girls’ Clubs. Without being integrated into school activities, such as organized study groups that focus on the modules or discussion of the materials as part of Girls’ Club presentations, the modules often end up sitting on library shelves. 3. RECOMMENDATIONS • The current leadership training focuses on the most successful of the female teachers in a school, generally 2nd cycle teachers, and has as its goal preparing them for career advancement. The development of course materials on gender issues for 1st cycle teachers, where most female teachers are found, would equip them with information both for their own career advancement and to increase BESO II – FINAL EVALUATION 41 understanding that they can pass on to their students, male and female. This material could be distributed to the new private teacher training facilities in the hopes that it would be introduced into the curriculum. • One limitation of the approaches implemented by BESO-BEP is that primarily women were offered training focused on the problems of female achievement and modules on gender issues. Because men are in charge of most school institutions, women cannot create deep change without the assistance of men. The various types of training offered to male regional, woreda, principal, and kebele personnel should contain some of the same material offered to female teachers. Male teachers also need to know how to create a gender sensitive classroom. • Any future supports for female student-teachers at TEIs will have to address a major change which occurred when the Ethiopian government abolished dormitory living at TEIs. Female student now, more than ever, need assistance in developing skills for the management of their lives, which BESO￾BEP has developed materials to support. Making students more aware of the existence of these materials and increasing the amount of counseling available for female students would help to offset the difficulties created by living on a small financial allotment in a new and unknown community. III. AED PROJECT MANAGEMENT A. BESO II-BEP PROGRAM MANAGEMENT 1. FINDINGS BESO-BEP has 76 employees (46 at project headquarters and 30 in the field) led by a senior management team that includes the Chief of Party (COP), Deputy COP for Operations, and Deputy COP for Technical, with Six Component Team Leaders, who work on technical matters with the regions and report to the Deputy COP for Technical. Sectional Heads including the Chief Financial Officer, Chief Administrative Officer and the IT Administrator report to the Deputy Chief of Party for Operations. The program has placed planning and management officers, in-service officers, finance personnel and drivers in selected regions. There are also field team leader as the focal person who works with the REBs and assists with annual and quarterly plans. According to the program USAID CTO and the AED project staff, this management structure has been sufficient for producing contract deliverables. The evaluation team was provided BESO II-BEP operational manuals and job description for each employee. Review of the job description manual provided by the COP indicates that each employee has clearly delineated responsibilities. The management structure and the evaluations team’s observations indicate that the project staff functions as a team. According to the deputy COP for operations and some of the staff members interviewed, the administration and finance personnel support the project implementation and program staff. As it develops its annual implementation plan, AED reviews the plan with REBs to make sure the plan reflects the regions’ priorities. Similarly, once AED develops its annual progress report (i.e., annual report), the report is reviewed with the regions to obtain their concurrence and endorsement on the outputs and results. Once this is done, both the annual progress report and the implementation plan are discussed at the Technical Working Group (TWG) meeting. The TWG recommendations are recorded and incorporated into the annual implementation plan. The TWG monitors progress of the BESO-BEP project, both activities under the AED Contractual Agreement as well as the Government’s progress in meeting conditions precedent and covenants. It also approves recommendations for international consultants and advisors. The TWG consists of members of the MOE, the Ministry of Finance and Economic Development, and USAID, which make up the core group. The technical group includes the core group plus the REB heads and their deputies. The TWG meets approximately every BESO II – FINAL EVALUATION 42 quarter. Through this process all the beneficiaries are informed about AED’s activities and about what is expected of the beneficiaries in order to benefit from USAID’s assistance In addition, there is an annual MERA workshop for the project staff. At the workshop results of the past year are reviewed and plans for improvement are developed to be incorporated and monitored into the following year’s implementation plan. Through these processes, all AED staff are informed about the planned activities and past performance, as well as what needs to be done to improve AED’s performance. According to the Chief of Party, the Technical Working Group Chairperson and the USAID CTO, the program had faced serious management problems in the initial base years. For example, reports were late and there was internal team discordance. TWG also pressed for improvements in the management of the program. To resolve these problems, AED home office senior managers came to Addis Ababato make changes that would improve the program. Interviews by the evaluation team have reported that things began to change when Dr. Ernest O’Neil was brought in as COP in August 2004. The COP was able to inspire team work and reports began to flow in a timely manner. BESO-BEP is now adequately staffed and budgeted to execute contract deliverables. The program has adopted appropriate management and operational procedures to satisfy its contract deliverables. The evaluation team has also observed that the field teams’ relationships with REB education officers and planners are very good. In Amhara, Benishangul Gumuz, and SNNP the team leader of the field staff participate in REBs staff and technical meetings. According to the REB head and deputy heads, planning department and education officers the team has interviewed, the BESO-BEP planning and management officers as well as the in-services officers have earned their respect. AED field staff have consistently been responsive the needs of the REBs. AED field staff members who have been interviewed by the evaluation team have indicated that their requests to the project head office in Addis Ababa have been responded to promptly. They are given adequate support to execute their responsibilities and to fulfill requests from the REBs. Comparisons of AED-BESO annual budget and actual expenditures consistently show that the project has under spent. However, the explanations for the under spending are: • Problems and delays in procuring commodities and equipment, for example, for installing PMIS in targeted woredas. 30 woredas were supposed to have PMISs installed during Option Year 2; however, that did not occur as a result of delays in procuring the computer hardware. Instead, the budget has been re-budgeted for Option Year 3 and the installation of the PMIS will occur this year. • A reduced number of workshop and seminar participants is another reason for under spending. Many times, a smaller number of trainees came to the workshop than planned because of staff turnover and/or reassignment. • Reduced number of training events. Although several rounds of training are budgeted for in the implementation plan, fewer training events are held for one reason or another. For example, there may be conflicts with government planned meetings or the REB feels the staff have had enough training. • Failure of implementing partners, e.g., the MOE, in implementing component activities for lack of adequate capacity. TESO, ELIP, Civic Education and Material Development were activities implemented by the MOE. However, the MOE did not perform these activities on time. At the start of BESO II, some regions had expressed to USAID a desire for direct grants to them rather than to contractors such as AED on the grounds that the money could be used more cost-effectively. To effectuate this, USAID tasked AED to assess the financial management capacities of the regions so that they can be certified (i.e., financial certification) to manage donor funds. However, this task was interrupted because USAID had another program, Double Entry System (DES) that was assisting government institutions to transition from single entry to a double entry system. . The TWG and USAID have, from time to time, asked AED to reallocate funds to priority activities as expressed by the regions. BESO II – FINAL EVALUATION 43 It is very difficult to asses the cost-effectiveness of the project interventions. However, the per unit cost for supplementary material development, printing and distribution seems to be reasonable. It is less than 20 Birr, or US $2.30, per booklet. The evaluation team’s discussions with AED staff have indicated that most of the BESO-BEP activities have been cost effective. The budget for project activities are determined at the quarterly and annual implementation planning meetings. 2. CONCLUSIONS BESO-BEP is now adequately staffed and budgeted to execute contract deliverables. The program has adopted appropriate management and operational procedures to satisfy its contract deliverables. The relationship between the project personnel and the MOE, REBs and woredas and TEIs is strong and collegial. B. MONITORING, EVALUATION, REPORTING AND ANALYSIS 1. FINDINGS The main objective of the Monitoring, Evaluation, Reporting, and Analysis (MERA) component is to identify, generate, analyze, report and disseminate information concerning the actual progress of project interventions, the impact of those activities over time and the improvement of quality and equity in the educational system. The following are the specific objectives of the component. • report project performance and impact • complete the assessment of the Ethiopian Education and training or carry out surveys, topic specific analysis, or case studies that provide lessons from which innovations can be learned in the education system • conduct internal MERA workshop to report findings from PMP assessment results, reflect on results and refine measurement tools Table 11, which is found in Annex E, provides a complete list of MERA interventions and deliverables compiled from AED-BESO annual reports. Table 12, also in Annex E, provides a complete summary of MERA targets and performance complied from the AED contract SOW (targets) and annual reports (for base year actual) and program results report (actual for option years). MERA has reviewed project results and refined performance indicators in collaboration with the MOE, REBs, TEIs, and project staff. It has developed and submitted to USAID a Performance Monitoring Plan (PMP) for assessing and reporting project progress and performance. The research sub-component in MERA is carried out in collaboration with the Education Sector and Planning Department of the Ministry of Education. The measurement of the overall quality of primary education is jointly carried out by AED and NOE. Research results are disseminated to the beneficiaries and the public using published reports, national conferences, seminars, and mass media. This MERA exercise seems to have developed and enhanced local policy research capacity. MERA has trained personnel in the educational system when various assessment programs are carried out. The research reports reviewed by the evaluation team are of high quality. The evaluation team bases this judgment on the review of the following documents: • Successful Schools in Ethiopia • The Cost of Educational Wastage in Ethiopia • Ethiopian Schools National Learning Assessment (ESNLA) Grade 4 • ESNLA Grade 8 • Action Research BESO II – FINAL EVALUATION 44 • Assessment of the effectiveness of the pre-service and in-service linkage program (results to be available in Option Year Three) MERA has developed various data collection instruments and also developed a data quality assessment methodology that is consistent with USAID directives. It is based on observation and field validation of data. According to the USAID CTO, BEP’s PMP reports are submitted in a timely manner. 2. CONCLUSIONS AED’s Monitoring, Evaluation, Reporting and Analysis (MERA) system is functioning well and has delivered all its contract deliverables. MERA has enhanced the policy research capabilities of education personnel in MOE and REBs. IV. HOST GOVERNMENT SATISFACTION Ministry of Education The Ministry of Education (MOE) regards the BESO-BEP project as having permeated all of its plans and activities. All BESO-BEP activities are planned jointly with the government and no activity is implemented without MOE. BESO-BEP sees its role as implementing and accelerating the strategies of the MOE. This level of integration with MOE policies, educational planning and management, curriculum, and support for the education system has led a sense of ownership over BESO-BEP activities. The deep interrelationship between BESO-BEP activities and government plans, plus the very real resources which BESO-BEP have contributed to central and regional education offices, have created a great deal of satisfaction with the BESO-BEP. The interventions mentioned by the MOE as the most important on their level and the most sustainable contributions of BESO-BEP are the assistance supplied in the implementation and expansion of the new curriculum and the first and second National Learning Assessments. The government of Ethiopia had proposed a new student-centered curriculum, but TEI instructors and teachers had felt that they did not understand how to implement these curriculum changes. BESO-BEP, with its consistent focus on active learning, continuous assessment, large classroom management, and active research, has gone a very long way in a short amount of time to implement these practices on all levels of the education system. The National Learning Assessments carried out with BESO-BEP support were the first of their kind in the history of Ethiopian education and allowed the country to undertake a national learning assessment at grades 4 and 8. In addition, the MOE has found the BESO-BEP research on policy a new approach because it focuses on evaluating current policies, which offers substancial assistance to their planning and management responsibilities. The MOE, through the TWG, is currently attempting to identify which BESO-BEP interventions could be taken over as programs by the REBs and which will continue to need input from outside sources. They are attempting to be specific in their recommendations and expect to complete this analysis by the end of January 2007. Regional Bureaus of Education In general, all of the Regional Education Bureaus felt the same deep involvement in BESO-BEP because of joint planning, which has resulted in satisfaction with the project and a sense of ownership in its interventions. In addition, the resources which BESO-BEP has contributed to regional education offices have created a great deal of satisfaction with the BESO-BEP. The chart below summarized some of the specific comments and suggestions offered by the five regional bureaus visited by the evaluation team. BESO II – FINAL EVALUATION 45 Chart 15: Regional Bureau Opinions about BESO-BEP Region Comments and Suggestions Afar The acting head of the Regional Bureau who met with the evaluation team has had only limited involvement with the project. He highlighted the impacts that the program has had on in-service teacher training, based on reports he had seen from the schools, and the assistance given to the Bureau in producing better strategic plans. He sees the greatest problem to be overcome as the difficulties they have encountered in collecting data from schools due to the great distances involved and the lack of communication. For example, only five or six woredas have telephone lines. He sees he greatest obstacle to the sustainability of the BESOBEP achievements as the lack of TEIs in Afar, which means that there is currently a lack of instructors who can carry out the in-service training provided by BESO-BEP. Consequently, he feels that the support most needed in Afar is for the new TEI which is planned for the region. Amhara The personnel interviewed at the Amhara REB stressed the importance of the region’s role in determining its own needs and having the assistance geared to these needs. The rapid needs assessment conducted by BESO-BEP and then revised by regional officers has been very helpful in this process. In addition, the region felt that the quarterly BESO-BEP reviews of progress has been an excellent management tool. In addition, the REB felt that the training for their personnel has been interactive and very effective. The greatest concern of the bureau was that the BESO￾BEP interventions had reached only a very small subset of the primary schools in the region and that a very great deal of work remains in disseminating training and resources to schools throughout the region. The REB has established a Sustainability Committee to begin looking at ways to sustain the achievements of BESO-BEP. Benishangul Gumuz The Benishangul Gumuz REB also rated the training they had received as very effective. Of the BESO-BEP interventions, they found the instructional kits and supplementary materials especially useful. The BESO-BEP gender training has influenced the Region’s policies about female teachers and students, with a focus on increased female teacher recruitment. The organization of school clusters has assisted in managing schools, but lack of transportation and the remoteness of many schools has made follow-up on the BESO-BEP interventions very weak. Oromia The Deputy Head of the Oromia REB gave BESO-BEP credit for influencing how teachers are trained in this region. He found the translations of BESO-BEP instructional material into local language especially helpful. Regional policy decisions have been influenced by BESO-BEP, as, for example, the creation of a policy proposing a 5% increase in female principals in the entire region. All six TEIs in Oromia have been structurally linked to sets of primary schools in preparation for TEIs’ expansion of their in-service training activities to schools that have not been reached by BESO-BEP. He felt that the Center of Excellence at the Asela TEI offered a very positive contribution, which should be adopted at all TEIs in the region. He agreed that the structural problem concerning the Center at the Asela TEI presented problems, but believed that they would soon be addressed. The most stressed suggestion that was made involved the need for more training in how to use data, saying that the emphesis has been on the use of computer systems to gather, organize and store data without sufficient attention to how to use it for management decisions in a decentralized system. He also drew on recent examples he had seen in Japan and Kenya to stress how important Continuous Professional Development is and proposed the need for a policy requiring teachers to receive more training on a regular basis, perhaps requiring recertification. SNNRP SNNRP Bureau personnel finds the new teaching and learning methodologies being implemented through in-service BSO-BEP training very effective in increasing the knowledge and skills of teachers and principals. They are concerned by differences in school leadership and management between BESO-BEP and non-BESO schools. BESO II – FINAL EVALUATION 46 Woredas All woredas personnel who were interviewed by the evaluation team expressed appreciation for the training they had received from BESO-BEP. Turn over has been quite intense on this level of the education system, which meant that there was great variation in the length of time and depth of experience with BESO-BEP among those interviewed. Some of their comments and suggestions are listed in the following chart. Chart 16: Woreda Opinions about BESO-BEP Woreda Comments and Suggestions Dubti The Examinations Expert and the Education & Supervision Expert have worked to expand the BESO-BEP interventions to other primary schools in their woredas area by creating cluster structure, but had found that they lacked budget to make use of this structure. Repetition and drop out rates have decreased in the BESO-BEP schools. Distances among schools and lack of transportation has created serious difficulties in follow-up and data collection, but they have made a strong effort to observe teachers in their classrooms. They have had a difficult time recruiting female principals and recommend that the leadership training for female teachers be extended to all female teachers. Awash Fontale The Head of the Woredas was new to the position and not very familiar with BESO-BEP interventions. The Education Supervisor reported that he had learned from the BESO-BEP training how to evaluate how well teachers are doing, especially the use of the evaluation forms which support better observations and feedback for the teachers. He had adequately understood how to make things such as lesson plans or how large classes could be managed before this training. He feels that there is a need more encouragement for female students. For example, they have found that feeding programs bring more female students into the primary schools. The long distances between the woreda office and the schools creates a serious barrier as there is only access to a car on average one day per month. Communicate by cell phones. Bahir Dar Gonder Zuria A sustainability committee has been established to brainstorm and find cost-effective ways to sustain the BESO-BEP achievement and program activities. Basona Werana The acting Head of the woredas reports that Continuous Professional Development (CPD) is a major focus of their activities. Teacher training takes place at the woredas once a month; the trainers are from both the woreda and Debre Berhan TEI. His recommendations are to scale up the coverage, expanding the program to more schools and to supple more educational materials. He also believes that sharing of experience among woredas would be useful. The Primary Education Expert reported a focus on CPD. He received self-instructional kits from BESO-BEP without instructions about where to distribute them so, because there was an adequate number, he distributed them to all schools in the woredas. He reports that the TEI is a great resource in this area and has taken on a great deal of in-service training. Different instructors go to the clusters to provide training on different topics, which allows the teachers to experience a variety of instruction. Abssosa The concept of school clustering has helped the woreda to optimize the use of resources, training, disseminate new methodologies. The student-centered approaches have motivated and changed the attitudes of both teachers and students. For example, student dropout, absence, and tardiness have all dramatically declined. Bambasi The woredas gathers teachers for the BESO-BEP clusters once a month on a Friday for training and discussion at the woreda. The evaluation team observed one meeting during which teachers developed and presented how they plan to implement active learning and continuous assessment methodologies using BESO-BEP supplementary materials on Science. Dodota Sire The acting Head of the woredas recommends that the TEI instructors go to the schools and train the teachers rather than depending on training of principals and key teachers reaching the other teachers. The woreda has created a training plan for continued in-service activities and has a plan for further strengthening of the pedagogical centers at the schools. The woreda Training Expert BESO II – FINAL EVALUATION 47 made a similar suggestion, proposing that the teachers from each cluster all come to the woredas to be trained by TEI instructors rather than using a cascade approach. Boset The education office Supervisor reports that he supervises the BESO-BEP Center schools three times and the Satellite schools two times per semester and has found that the active learning approaches have been implemented in all of these schools. At first the teachers had resisted this approach, but now all accept although they still have difficulties with continuous assessment. The linkages provided by the cluster structure have allowed for effective research into issues such as the cause of dropout and assisted in creating solutions. Holeta Town The Head of the woredas education office also suggested that it would be better if instructors directly trained all of the teachers rather than depending on the training that would be provided by principals and key teachers. He felt that far too much information was lost over the layers of the BESO-BEP training structure. He reported that BESO-BEP training for the woredas staff had been very effective and that the cluster structure was important because it allowed teachers to share experiences and discuss educational issues. Gomma According to the woreda Supervisor, the Jimma TEI has provided training for woredas personnel. Student-centered approaches have improved student self-confidence, communication skills, and creative thinking. The woreda has established a Sustainability Committee to determine how to sustain the BESO-BEP activities. Awassa Shebedino According to the Head of the woredas, there is a noticeable difference between BESO-BEP and non-BESO schools in terms of female student enrollment. Community participation is also higher. Misha Primary Schools School level personnel also report satisfaction with BESO-BEP interventions. In most cases they see the supports which have been offered as those which they needed. Generally they express a need for more of the various interventions and/or more follow up associated with their implementation. The intervention which is discussed in the most enthusiastic terms on the school level is the support given to student-centered, active-learning methods. Teachers and principals often described profound changes in teacher and student attitudes and behavior as a result of the introduction of these approaches. In addition, the various reading materials provided by BESO-BEP and the rejuvenated pedagogical centers were almost always mentioned as major contributions to the schools. The value placed on the material resources supplied by BESO-BEP, duplicating machines, typewriters, and other resources and supplies, was apparent in their distribution and maintained. V. LESSONS LEARNED A. MAJOR CONSTRAINTS ON PROJECT IMPLEMENTATION The only major change between BESO I and BESO-BEP is that, while BESO I focused on two regions and the Federal Ministry of Education, BESO-BEP covers all regions and the Federal Ministry of Education. At the same time, the funding for educational reforms was dropped from US$ 80 million in BESO I to US$30 million in BESO-BEP. The aftermath of the May 2005 general election resulted in a very high turnover of staff in all the regions as well as in the FMOE. For example, many deans and school principles were dismissed or transferred elsewhere. This created an environment in which implementation of the interventions became increasingly difficult. BESO II – FINAL EVALUATION 48 The transformation of TTIs into TTC and the delegation of the preparation of primary school cycle 1 teacher by the private sector present a serious problem of loss of the impact of BESO investment in the development of cycle 1 teachers over the past several years. The current focus in the TTCs is to bring stream-based (or subject-based) active learning into the cycle 2 (grades 5-8) curriculum. B. IMPACT ON PROJECT IMPLEMENTATION AND PROJECT RESPONSE BESO-BEP has dealt with the extremely large geographic area and number of schools and educational offices and limited resources through two strategies. The first involves only attempting to directly supply interventions to a limited number of schools, creating centers in which multiple resources interacted to create models, rather than attempting to spread its resources thinly over all schools. For example, in the five regions visited by the evaluation team the percentages of schools receiving direct support from BESO-BEP were low. Chart 17: Percent of All Schools Supported by BEP Region Number of Schools No. of BESO-BEP Schools Percent of All Schools Supported by BESO-BEP Afar 205 28 13.7% Amhara 5061 335 6.6% Benishangul Gumuz 314 56 17.8% Oromia 6466 440 6.8% SNNPR 2866 193 6.7% The second strategy involved providing much of its support to the institutions which managed, supported, and trained this vast array of schools and teachers. This served both to strengthen the overall system and build in an increased level of sustainability. The political crisis in 2005, which created great turnover in staff, reduced the number of professionals who had been trained by BEP who were in the key positions to implement the project. This meant that that a massive amount of training of new education professionals had to take place. C. SUSTAINABILITY OF PROJECT ACHIEVEMENTS The most sustainable achievements of BESO-BEP appear to be: • The implementation of the new curriculum on all levels of the education system. The MOE reports this as the most sustainable achievement of the project and AED staff report a growing awareness of the usefulness of the student-centered methodologies. • The success of the first and second National Learning Assessments have created a demand for this data and the expertise gained by the educational personnel who conducted the second Assessment will make this a sustainable interventions. • The demonstration of a model for the delivery of in-service training, resources, and supplementary materials to schools through the use of the cluster structure. However, this model will only be sustainable if incentives for TEI instructors and woredas education officers to continue training are offered. • Production of instructional materials through the approach found in the Debre Berhan Center of Excellence. The success of this publishing facility could be replicated at other TEIs in other regions. • The revitalization of pedagogical centers at TEIs and schools. The enthusiasm for and utilization of these centers was apparent to the evaluation team. In addition, according to AED BESO-BEP staff, there has recently been a gradual increase in awareness regarding the usefulness of the technology assisted management information systems. BESO II – FINAL EVALUATION 49 D. PROJECT IMPACT ON EDUCATIONAL POLICY BESO-BEP has conducted research into policy change issues and produced a number of policy research reports. The project has printed and widely distributed the resulting research papers and discussed these reports at various forums, including a series of workshops. The BESO-BEP goal has been to inform policy decisions in the MOE, regions, and woredas. Although the project has attempted to provide information to guide policy decisions, it primarily sees its role as the implementation of policies created by the central and/or regional governments. As were mentioned in the previous section on Host Satisfaction, REBs often commented on the specific influences of BESO-BEP on their policies. E. LINKAGES BETWEEN BESO-BEP AND THE COMMUNITY GOVERNMENT PARTNERSHIP PROGRAM The Community Government Partnership Program (CGPP) focuses on developing the relationship between communities and primary schools. The program helps communities understand what they can contribute to the educations of their children and helps schools understand how they depend on the community to help them with this task. Communities can monitor student attendance and behavior, contribute funds and labor for classrooms and latrines, and work with the school to create income generating projects. The CGPP also works to provide community education on social issues, such as early marriage and HIV/AIDs. The evaluation team had the consistent impression that the schools in which both BESO-BEP and CGPP have operated are making greater progress than simply the sum of the two programs would suggest. This synergy may be the result of combining a top-down training approach with a bottom-up empowerment approach. World Learning, Save the Children USA, CARE, and the Tigray Development Association (TDA) all implement the CGPP in various regions. They all participate in the Technical Working Group quarterly meetings, where they present progress reports and plan joint activities. The bESO-BEP and CGPP have coordinated their activities through joint training activities, logistical support, and development of training manuals. Currently there is an attempt to increase the number of schools supported by both BESO￾BEP and the CGPP to 80%. Given the evidence that the evaluation team witnessed of increased impact at schools where these two program overlapped, this is a very positive direction. F. PROJECT DESIGN The design of BESO-BEP is extremely broad in the geographic and educational spread defined as the range of the project. The time allowed within which to address this large population of teachers and number of institutions has been extremely short. These factors have made the project unwieldy to implement and made it difficult to achieve sustainable impacts. The design has, however, allowed for experimentation in various ways of achieving the changes desired. This has led to the development of several successful models, the management and implementation of which can now be transferred to the education system. Some of the most promising interventions are: • The two Centers of Excellence that were visited demonstrated very successful models for increasing the capacity of the educational system to produce teaching materials and tools. The difficulty in expanding this approach lies in the inability to directly influence the TEI staffing and curriculum decisions necessary for the support of the Centers. • The model of a cluster structure and cascade in-service teacher training, supported by self-instructional materials has demonstrated an effective way to support improved teaching in a large number of schools. The first step toward expanding the use of that model to all Ethiopian schools has been taken by the regions through the creation of school clusters associated with TEIs. The second step has been started by BESO-BEP through the creation of cluster coordination centers at TEIs. The TEIs need to complete that step by finding incentives for their instructors who can provide in-service training and BESO II – FINAL EVALUATION 50 monitoring at those clusters linked to the TEI. The third step would be to duplicate the publishing Center of Excellence at other TEIs in all regions in order to take over preparation of materials from BESO-BEP and increase the production of self-instruction kits. • There is a great need for supplementary reading materials in Ethiopian schools. The subjects about which to produce materials that were selected by BESO-BEP are appropriate and needed, with the possible exception of the HIV/AIDs materials due to the increasing number of other organizations supplying materials on this issue. While there is a strong need for these materials, they often are not utilized because they do not reach the specific teachers who will use them. Some method needs to be found for increasing awareness of the existence of these materials and making sure that the materials are directed to the right person. • Capacity building for KETBs has been very successful in increasing support for schools in ways where communities can contribute, such as monitoring the attendance and behavior of students and/or increasing resources for the school. The evaluation team found that the schools in which both BESO￾BEP and CGPP have operated have better community-school relationships and appear to be more highly energized in general than the sum of the two programs would suggest. Both of these impacts indicate the importance of including communities in educational interventions and making them partners with school personnel in the task of educating their children. • In all five regions visited by the evaluation team, the team was informed that affirmative action for the hiring and promotion of female teachers and the increased admission of female students at TEIs were driven by BESO-BEP gender awareness activities. The heightened awareness of gender issues in schools, TEIs, and educational management is a more significant impact than any of the specific interventions in this component. That impact could be further expanded and deepened by not focusing gender training and materials only on females, but by covering the same issues in all pre- and in￾service teacher training for both men and women. BESO II – FINAL EVALUATION 51 ANNEX A: LIST OF INDIVIDUALS INTERVIEWED Location Institution Person(s) Interviewed Position in Institution Addis Ababa USAID Aberra Mokonnen Tesfaye Kelemeworke Chief Primary Education, CTO AED-BESO-BEP, CTO AED Project HQ Johnson Odharo Mulatu Keffelew Teshome Nekatibeb Bahru Shikur Markos Tadesse Alemnesh H/Mariam Desalegn Garsamo Demissie G/Mariam Semegnew Kassaye Tedela Kebede Kassau Ali Abraham Hagos Woubjeg Gebre Kirstos Baraki Zeselassie Daba Hundi Tesfau Mohammed COP Deputy COP, Operations Acting Deputy COP, Technical Pre-service Teacher Education In-service Teacher Education Female Teacher Support Instructional Material Develop PMIS Database Trainer Planning & Management Computer Programmer PMIS Planning & Management In-service teacher Education te Woreda Capacity Building Chief Financial officer Materials Development IRI Script Writer & Team Lea Ministry Office Education TWG Members Tizazu Asare Zewdu G/Kidan Ejeta Negari Brehanu H/Mariam Yeshtilla Mulat Head, Planning & Management, TWG Secretary Expert, Quality Assurance & Examination Agency Expert, Programme Production, EMA Head, Curriculum Department Head, Education Program & Teacher Education Regional Education Bureau Seid Mohammed Acting Head/Eduction Support Dept Head AED Regional Office Adane Abebaw Teacher Education/Team Leader Afar AED Regional Office Ahmed Erbrahm Planning & Management Dubti Woreda Kedir Nega Hassen Muhie Examinations Expert Education & Supervision Dubti Awash Sheleko Ahmed Hassen Amsale Abdurahman Yesuf Abegaz Principal Key Teacher Teacher Dubti 2 Ersha Seid Debrib Abdu Jamma Principal Teacher Dubti Dubti Elementary Semegne Abeje (F) Principal Awash Fentale Woreda Hussen Hader Shifera Mammo Woreda Head Education Supervisor Awash Fentale Keganazmatch Haysema Demetas Tadissa Tagay Depebe Mohanned Edris Principal Acting Deputy Teacher SNNPR AED Regional Office Begashaw Aberra Henock Mesfin Planning & Management Officer In-service Officer Awassa Regional Education Bu Tonja Toma Head, Planning Dept. BESO II – FINAL EVALUATION 52 Location Institution Person(s) Interviewed Position in Institution Labena Tona Head, Education Program & Human Resources Dept. Mohammed Ahmed Head, Supervision Section & Acting Head Shewaye PMIS Manager Berhanu H/Michael MMIS Manager Sisay Shiferraw Network Administrator Awassa City Admin. Mitiku Weracho Head, Supervision Team Ethiopia Tikdem Primary School Ayalew Assefa Female 1 Female 2 Female 3 Principal Vice Principal Teacher Teacher Haiq Primary School Seifu Bekele Nigest Testo (F) Principal Vice Principal Awassa TTC Paulos Rike Bizaat Terefe Asst. Academic Dean Instructor and IRC coordinator Awassa Gebadar Primary School Kosit Sima Adane W/Senbet Principal Shebedino Woreda Tadesse Chachamo Head, WEO Konsore Hano Primary Leliso Legato Principal Shebedino Nure Dulecha Primary School Tamerat Bakato Regassa Lankamo Key Teacher Teacher Hadiya Zone Education Dawit Demisse Head, ZED World Learning Gugsa Tadesse Hadiya Zone Hossana TTC Tadesse Lema Zekyos Bule Academic Vice Dean Admin. Dean Misha WEO Mattewos Mekeso Head, Admin. Services Wasgebeta Primary Degu Yimam Principal Guna Bonochora Mulugeta Moloro Principal Misha Woreda Abushura Primary Demisse Abiyu Key Teacher AED Field Office Made Michael Endeshaw Amare Bizuneh Team leader & In-service Officer Planning & Mgmt Officer Amhara Regional Education Bureau (REB) Fanta Moges Teodros Shewareget Setu Aynalem Yayerad Belay Nesanet Tilahun (F) Deputy Head, REB Head, Human Resources Dept. Head, Planning & Information Head, Personnel Admin. Administrative Secretary Bahir Dar City Admin. Education Dept. Abiy Hailu Demirew Teferi Kasse Abate Head, Records Vice Head Education Dept. Inspection Team Coordinator Ewket Fana Primary Deboch Degefu Principal Marriemia Bet Primary Yezi Alem Getachew (F) Principal Dil Chibo Primary School Zerifu Takele Minale Aychew Alene Zegeye Principal Librarian Pedagogical Center Coordinator Bahir Dar Sertse Dingel Abebe Negash Workinesh Damitew (F) Belaynesh Getu (F) Almaz G/Yohannes (F) Principal Librarian Girls’ Club Librarian Pedagogical Center Coordinato Gonder Beles Walka Primary Mihret Yimer Principal Infranz School Tamrat Ferede Getnet Demisse Vice Principal Cluster Supervisor BESO II – FINAL EVALUATION 53 Location Institution Person(s) Interviewed Position in Institution Gonder TTC Senaye Zegeye Alemayehu Tsegaye Melishu Tizazu (F) Mulugeta Mamo Mulugeta Habte Shewaneseh Fisseha Yesuf Yeshaw Vice Dean, Development & Re Head, Cluster Coordination Un IRC Coordinator IRC Expert Chief Librarian Head, Staff Development Unit Academic Vice Dean Kebele 16 Primary School Yimer Alemu Mohammed Salih Vice Principal Teacher Gonder City Administration Tigabu Manamino Zewude Belay Program & Training Primary Education Expert Atse Behaffa Primary School Asrat Kassa Mulugeta Berihun Principal Librarian Basona Werana Woreda Ayalew Belete Amisalu Ayahiluh Acting Head Primary Ed Expert Gundoberet Primary Wasse Ali Birtkan Kebed (F) Tslehay Tonma Principal Vice Principal Pedagogical center coordinator Mush Primary Genanew Dessie Herut Bekele (F) Principal Teacher Abamote Primary Kurabachew Dresse Principal Basona Werana Debre Berhan TEI Seid Mohammed Aby Yimer Getu Tefena Tiruyie Bro (F) Gebrehiwot Niegusse Simon Wlagearehu Muluneh Bezahie Berhane Bizuneh (F) Kiros G/Michael Yimegnushal Woldenesekol Acting Dean Acting Research & Developm Staff Development coordinator Civics Instructor Center of Excellence coordinat Head Librarian Computer Center coordinator Cluster Center coordinator Cluster Center Assistant 3rd year Student Benishangul Gumuz AED Field Office Assefa Jano Beyene G/Selassie Team leader/Planning & Mana In-service Officer Hoha Kutir 4 Primary School Abera Kejela Daniel Nigatu Principal Key Teacher Regional Education Bureau Habtamu Hika Atinkut Yalew Kessete Admas Kedro Deni Head, REB Head, Planning & Programs Chief Expert, Teacher Ed Head, Personnel Dept. Assosa WEO Shuma Ayana Kassa Kemo Rep. & Head, WEO Head, Planning Dept. Assosa Selam Bir Primary School Seboka Abdissa Amana Desesa Principal Pedagogical center Coordinat Bambassi WEO Solomon Hailu Dereje T/Mariam Acting Head Planning & Program Expert Kashamando Primary School Wosenu Beshah Tatek Aga Principal Key Teacher Bambasi Baro Primary School Tesfaye Amare Tigabu G/Selassie Principal WEO Cluster Supervisor Oromia Regional Education Bureau Ato Teshane Deputy Education Head AED Regional Office Rago Biwen Teacher Education Officer Dodota Sire Dodota Sire Woreda Wakums Fita Elias Gemedu Acting Head Training Expert BESO II – FINAL EVALUATION 54 Location Institution Person(s) Interviewed Position in Institution Dera Primary Kuri Simbiru Brinesh Belay (F) Gepathu Laegssa Principal Teacher Teacher Horsis primary Misikir Mulugint Habtamu Abaaire Principal Key Teacher Boset Woreda Kebede Atehemma Muktar Haji Head of Ed Supervisor Wilenchity Primary Tesfaye Bore Amaru Asseta (F) Wubit Fekadu (F) Principal Teacher Teacher Boset Dongene Tiyo Primary Belay Lemmi Tesfaye Argaw Principal Teacher Holeta Town Woreda Malatu Dugo Woreda Head Genet 44 Primary Tolessa Baba Belaynesh Sahlu (F) Vice Principal Teacher Adolesa 19/67 Primary Chalchissa Tolla Tsehai Bleay (F) Vice Principal Teacher Holeta Town Asela TEI Abdella Gerbi Abebe Megersa Gadisa Bedada Meseret Tesso (F) Kemal Gegechu Mohamid Ibrahan Haji Edao Aregaeh Magersa (F) Jeylan Hussein Vice Dean Instructor/AED coordinator Instructor 3rd year Student Center of Excellence coordinat Former Center of Excell Coor Center of Excellence technicia Librarian Computer Center coordinator Jimma Jimma Zonal Education Galib Habatemam Head Jimma TTC Zenab Abawoli Seifu Abagrdi Habti Dubossa Asnaketch Demissie (F) Meseret Gadissa (F) Asiba Keder (F) Dean Academic Vice Dean Chief Librarian Instructor Student Student Gomma WEO Tesfaye Dinka Takele H/Gabriel Getachew Denu Head, Finance Head, Planning & Statistics WEO Cluster Supervisor Agaro 2 Primary School Takele Gebre Seboka Dufera Mekdes Mulat (F) Principal Agaro City Schools Supervisor Librarian Ras Desta Damtew Ahmed Adafita Abera Abebe Yeshi Hailu (F) Principal Librarian Special Education Teacher Gomma Bulbulo Primary School Teshome Arega Setina Abazmab (F) Mamuye Brehanu Principal Vice Principal Pedagogical Center Coordinator BESO II – FINAL EVALUATION 55 ANNEX B: DATA COLLECTION FORM Date: Name of institution: Position of interviewee: Number of years in position: Name of interviewee: Sex of interviewee: BESO II – FINAL EVALUATION 56 BESO/BEP Final Evaluation Pre-Service Teacher Training Are the teachers with pre-service training that you select now different from those in the past? More teachers with 3year degrees certificate? Better trained? More female teachers? Has BEP played any role in the criteria used for trainee selection? On a scale of 1 to 3 please rate effectiveness of BESO/BEP’s pre-service teacher training in improving pedagogical skills of teachers. 3 Very Effective 2 Effective 1 Not Effective Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP’s pre-service teacher training? BESO II – FINAL EVALUATION 57 Pre-Service Teacher Training Discuss, rate, and observe the following BESO/BEP interventions at TEIs: Skill development for instructors – SPSS, web design, materials production, teaching methods. Computer stations - instructional resource center, Library – reference materials, Multi-media lab, student record management system, etc Are these resources easily available for use by facility and students? How many books have been provided by BESO? How relevant are they? How well catalogued? How well do the library facilities create an appropriate environment for study? How are these resources maintained? How effective has BESO/BEP been in disseminating new methods of teaching (active and student-centered); better lesson planning; student assessment; school resources organization and management in the linked primary schools? BESO II – FINAL EVALUATION 58 Pre-Service Teacher Training How receive? How use? What impact? Was training provided on how to use the library? Computer stations? Skill development materials? How has BESO/BEP support been useful in changing your teaching methodology to active and student-centered? Please provide concrete examples. How frequently do you use Library, Computer Stations, and Skill Development Centers provided by BESO/BEP? Could you give examples where the resources have been useful and their impact in the classrooms? In your opinion, how useful has BESO/BEP support been in strengthening active and student-centered learning-teaching in your school(s)? Please give concrete example on the changes? In your opinion have these resources and supports improved your pedagogic skills? Please give concrete examples: BESO II – FINAL EVALUATION 59 In-Service Teacher Training How has your institution been involved in in-service teacher training? On a scale of 1 to 3 please rate effectiveness of BESO/BEP’s in-service teacher training in improving pedagogical skills of your teachers. 3 Very Effective 2 Effective 1 Not Effective What have been your experiences with active learning? Was there resistance to this type of teaching? Why? What plans have been made to support continued in-service training of teachers? Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP’s approach to in-service teacher training? BESO II – FINAL EVALUATION 60 In-Service Teacher Training How receive? How use? What impact? How was in-service teacher training offered for teachers in your school? What was the role of the cluster in this? Can we see the plan for when training was passed on from directors and key teachers to other teachers? How have you used the in-service training that you received? Please provide specific examples If a 1-4 teacher, have you used active learning in your classroom? How? Has the in-service training been adequate to provide teachers with mastery of the techniques? In your opinion, how useful has BESO/BEP support been in strengthening active and student-centered learning-teaching in your school(s)? Please give concrete example on the changes? Has there been any follow-up to evaluate how well teachers are able to use this training? BESO II – FINAL EVALUATION 61 Have you received supplementary materials from BESO/BEP? If yes, which areas: - Self-instructional kit for classroom management - Self-instr kit for continuous assessment - Self-instr kit for active learning - Civic education - HIV/AIDs - Environmental education - General science What has been your institution’s role in developing and evaluating these materials? On a scale of 1 to 3, how well do the supplementary materials provided by BESO/BEP match the actual needs of your school(s)? 3 Very well 2 Well 1 Not at all Do you have a plan to sustain the development and use of these materials? Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP’s supplementary materials? BESO II – FINAL EVALUATION 62 Supplementary Materials How receive? How use? What impact? How were the BESO/BEP supplementary materials provided to you? Was training/orientation provided to you on how to use each type of supplementary material? How useful are these materials provided by BESO/BEP? How do you use these supplementary materials? Please give examples. How have these supplementary materials enriched the curriculum and improved pedagogic skills? Examples? In your opinion, are these materials sufficient to produce mastery of the subjects? Have they resulted in changed behaviors and practices of teachers and students? Has there been any follow-up to evaluate how well teachers are able to use these materials? BESO II – FINAL EVALUATION 63 BESO/BEP Final Evaluation Leadership and Capacity Building Training Did you receive leadership and/or capacity building training? If yes, capacity building in what areas? - Personnel Management Information Systems - Material Management Information Systems - Project Planning - Model & Decision Support Systems - Computerized Monitoring Systems On a scale of 1 to 3, how well does the training provided by BESO/BEP match your needs. 3 Very well 2 Well 1 Not at all On a scale of 1 to 3, rate your satisfaction with the quality the training provided by BESO/BEP. 3 Very satisfied 2 Satisfied 1 Not satisfied Do you have a sustainability plan to maintain and update leadership and capacity building training? Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP’s leadership and/or capacity building training? BESO II – FINAL EVALUATION 64 Leadership and/or Capacity Building Training How receive? How use? What impact? How were the BESO/BEP leadership or capacity training provided to you? Was the training interactive and participatory or a series of lectures? What was the format: workshop, seminar, or classroom setting? How have you used the leadership or capacity building training that you received? Please provide specific examples. Do you ave a strategic plan to guide development of your institution? If yes, have was the BESO/BEP training used in developing this strategic plan? Please explain how. In your opinion, how useful has BESO/BEP support been in strengthening active and student-centered learning-teaching in your school(s)? Please give concrete example on the changes? Has there been any follow-up to evaluate how well teachers are able to use this training? BESO II – FINAL EVALUATION 65 BESO/BEP Final Evaluation Female Teacher Support System How has BESO/BEP helped support female teachers and students: How effective was leadership training for women who are ready for leadership positions? What support systems have been developed for female students at both TEIs and schools to prevent dropout? What linkages to women’s/girls’ associations exist at your institution? Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP’s support for female teachers and students? BESO II – FINAL EVALUATION 66 Women Teacher Support System How receive? How use? What impact? How many female teachers in your woreda or school have received training? Before they were in leadership positions are after they were already in those postitions?/ Where, for how long, and with which instructors was leadership training for female teachers conducted? If you received leadership training for female teachers, how have you used that training? For students at TEIs, how have the supports for female students (affirmative action, personal development skills training, and remedial courses)? Do you belong to a women’s association? How does it support you? How do you help to support female students? Girls’ clubs? Has BEP leadership training increased the number of female principals, deputies, and department heads? What prevents more women from taking these positions? BESO II – FINAL EVALUATION 67 BESO/BEP Final Evaluation Linkages Describe what has the relationships between Cluster Center schools and Satellite schools have been. How do they interact? How effective has this linkages created or supported by BESO/BEP been? 3 Very Effective 2 Effective 1 Not Effective Discuss, rate and observe BESO/BEP resources provided for both Center and Satellite schools: What plans have been made to maintain these resources? Do you have any recommendation or suggestion for improving the effectiveness of BESO/BEP clusters? BESO II – FINAL EVALUATION 68 Linkages How receive? How use? What impact? Was training in how to use the resources at the Center school provided? What activities do the teachers in the cluster schools engage in together? How does the cluster arrangement assist in￾service teacher training? How frequently do you use the resource center? Are there scheduling problems? What do you specifically use it for? Give examples. What support has the relationship between Center and Satellite schools provided? Please provide concrete examples Collect the following data if possible. BESO II – FINAL EVALUATION 69 Note how data stored (electronic or hand written), who provided, how available, etc. 2002 2003 2004 2005 Female student enrollment Male student enrollment Female student completion Ale student completion Female teachers Male teachers How was this data supplied? BESO II – FINAL EVALUATION 70 ANNEX C: SCHOOLS VISITED AED-BEP School Non- AED￾BEP School BESO TEI Linkage School CGPP School Urban Rural Afar 1 Dubti Dubti Awash Sheleko √ √ 3 Dubti 2 Ersha √ √ √ Dubti Elementary √ √ Awash Fontale Keganazmatch Haysema √ √ SNNP Awassa Awassa Ethiopia Tikedem √ √ Gebeya Dar √ √ Haiq √ √ Sidama Shebedino Nure Dulecha v √ √ Konsore Hano v √ √ Hadiya Misha Was Gabeta √ V √ Guna Beno √ √ Abushura √ √ Amhara Bahir Dar Bahir Dar Ewket Fana √ √ Mariemia Bet √ √ Dil Chibo √ √ Sertse Dingil √ √ N. Gonder Gonder Zuria Beles Walka √ √ √ Infranz √ √ √ Atse Bekaffa √ √ BESO II – FINAL EVALUATION 71 AED-BEP School Non- AED￾BEP School BESO TEI Linkage School CGPP School Urban Rural Kebele 16 √ √ √ N. Shewa Basona Werana Gundo Beret √ v √ Mush V √ √ Abamote V v Benishangul Gumuz Assosa Assosa Hoha Kuter 4 √ √ √ Selamber √ √ Baro √ √ Bambasi Keshmando √ √ √ Oromia Arsi Dodota Sire Dera Yenegew √ √ Horsis √ √ E. Shewa Boset Welenchiti √ v √ Dongere Tiyo √ √ W. Shewa Holeta Genet 44 √ √ Adolesa 1968 √ √ Hamle 19/67 Primary v √ Jimma Gomma Agaro No. 2 √ √ √ Bulbulo √ √ √ Ras Desta √ √ √ BESO II – FINAL EVALUATION 72 ANNEX D: SCOPE OF WORK The evaluation team shall conduct a comprehensive final evaluation of Basic Education Program. The evaluation shall: a) examine the results and effectiveness of the project in relation to the objectives of the contract and to the USAID/ Basic Education Program Results Framework; b) evaluate performance against the goals delineated in the contract and annual work plans; c) examine the effectiveness of the project management; d) identify and analyze implementation issues, challenges, implementation barriers and their causes; e) identify lessons learned including innovations; and f) gauge Ministry of Education’s (MOE), regional state education bureaus’ (RSEBs), city administration education bureaus’, woreda education offices’, teacher education institutions’ (TEIs) and primary schools’ involvement and satisfaction with the progress of the project; Specifically, the evaluation team shall address the following issues: A. Project Results and Input: The evaluation will examine the project’s overall results to￾date, including: a) A systematic review of established targets at all levels under the BEP in the contract and implementation plans. If performance has exceeded or fallen short of targets, the reasons (positive or negative) shall be identified. b) The impact of the project on the targeted beneficiaries. For individual project activities, the evaluation shall answer the following questions: (1) Have the activities achieve their intended results? (2) Did the planning and execution of the project incorporate strategies for sustainability from the beginning? (3) How effective have the interventions been in addressing the problems? (4) How sustainable are the results of the program? (5) How cost-effective have the interventions been? (6) Have the trainings provided under the project been sufficient and used to transfer the relevant skills and knowledge appropriate to beneficiaries? (7) What is the opinion of the school directors, teachers, woreda and regional education officials about the trainings and relevance to their needs? (8) Do woreda education and regional officials feel a sense of ownership of the project? (9) What is the level of utilization of the books, equipment and other resources provided to the institutions and if not utilized, what were major constraints impeding their utilization? (10) How have BEP activities and outputs been integrated, where appropriate, into initiatives of the Government at TEIs, cluster centers, woredas, regional and national levels in terms of the host country Government policy and planning? 11) Are project activities planned and coordinated with other development partners? How? BESO II – FINAL EVALUATION 73 12) Has BEP implemented the recommendations that have been forwarded by the Basic Education Program Technical Working Group (BESO-II TWG)? If not, what are the reasons? 13) Were there any specific policy reforms that this program has contributed to? B. Project Management: The evaluation team will examine the effectiveness of the management of AED in achieving the USAID Education Program priorities. The evaluation should answer, at minimum, the following questions: • Is the contractor, AED, in Ethiopia adequately organized and structured to enable it to meet the objectives of the project? • Are there clear and appropriate delineations of the responsibilities of each staff member from the lowest to the top management level, as well as delegation of responsibilities? Do auxiliary offices (administrative, finance, etc.) give the necessary support to the project implementation and program staff? What has been the effect on project implementation and the implication on the program’s anticipated results? • How effective has AED’s management been in building team capacity and ability to wok towards a common objective? • Has the change in the management structure and leadership contributed to the achievement of the project goals? • Has AED’s management, organization and use of its resources (human, financial, material and time) been able to tackle project priorities in a timely and efficient manner? • Has AED’s M&E system established for the program been adequate and efficient for tracking inputs, ensuring that supporting processes are put into place in a timely fashion, and measuring the quality of resulting outputs and changes? C. Relationship to Mission Plans: Based on contractor’s project documentation and the USAID/Ethiopia Integrated Strategic Plan, the evaluation report shall define in summary the relationship between the activity of the project and USAID’s Basic Education Program Results Framework. The report shall also establish to what degree the Basic Education Program Results Framework has been supported by the contractor. The evaluation will examine the logical link between results within the program and to that of USAID/Ethiopia’s Integrated Strategic Program. Is the development hypothesis of the program realistic? Do the assumptions considered for the realization of objectives at all levels hold true? D. Host government Satisfaction: the evaluation team will determine the degree to which BEP has responded to perceived needs of its beneficiaries: teachers, education managers, government partners at the school clusters, woredas, regional and national levels. Which activities are perceived as most important? Which ones are least? Why? E. Lessons Learned: The evaluation team will draw out lessons learned. Specifically, i. What have been the constraints that impact project implementation (policy environment, operational, institutional [within AED, USAID and/or host country partner institutions]) and what has been the impact on project implementation? ii. Have the constraints been rectified? How? iii. How sustainable are the achievements gained? BESO II – FINAL EVALUATION 74 iv. How does project-generated technical information and/or formal research impact educational policy reform? v. Have linkages between the Community Government Partnership Program (CGPP) and BEP been fully optimized? F. Program Design: the evaluation team will analyze the evaluation findings, including the budget levels in relation to the project objectives and expected results, the reasonableness of the targets in relation to the implementation time-frame, the contractor’s organizational structure, implementation environment, etc. EVALUATION METHODS Data Collection: Both quantitative and qualitative methods will be required to collect data from a number of sources, including field visits, the contractor, different departments in the MOE, RSEBs and WEOs officials, PTAs/KETBs, primary schools, TEIs, etc. Techniques/instruments to be used to capture data may include structured questionnaires, direct project activity observations, interviews and other rapid appraisal methods. Once in the field, the evaluation team is expected to interview MOE, RSEBs, city administration education bureaus, woreda education officers, TEIs, KETBs/PTAs, school directors/administrators, teachers/instructors, students, the contractor’s staff in Addis Ababa as well as in regional offices. The BEP Performance Monitoring Plan (PMP) will be a key source of information. The evaluation team will use a combination of purposive and random sampling methods based on sampling framework that includes factors affecting the nature and implementation of the BEP. Data Analysis The evaluation will employ tables and graphs whenever necessary to show its findings. Results reported in averages (χ) needs to be supported with variances (σ 2 ) and number of observations. Results in percentage should be accompanied with the number of observation. BESO II – FINAL EVALUATION 75 ANNEX E: PROJECT INTERVENTIONS TABLE 6: PROJECT INTERVENTIONS: PRE-SERVICE TEACHER EDUCATION S.N Key Intervention Contractual Deliverables 1 Installation of CDS/WINISIS software and initiation of automation of library usage. 2 Formation and improved participation of library clubs to widely use and support the libraries 3 Increase use of the internet for better access to virtual libraries and materials 1.a. Number of TEIs that have improved the quantity and quality and use of library materials 1 Wider production and use of teaching and learning using locally available resources (TALULAR) for instructional support 2 Establishment of Cluster Coordinating Units in all TEIs and allocation of Birr 156000 to support their linkage functions 3 Wide spread use of TALULAR in 433 pre-service-in-service linkage primary school training of 10569 in￾service teachers in these schools 1.b. Number of TEIs that have increased the use of instructional resource centers/or equivalent 1. Purchase of reference books, equipments and materials with government allocated budget 2 Allocation of full time additional librarians 3 Follow up training to all instructors on computer skills and SPSS, CDS/WINISIS 4. Constructing and/or Expanding additional buildings and reading rooms 1.c. Number of TEIs that have increased funds in their annual budge to develop their library and /or center 1. Putting up the management of the computer centers, instructional resource centers, cluster coordinating units, multi-media laboratories and centers of excellence 2 Assigning additional full time personnel to the computer centers, instructional resource centers, cluster coordinating units, and centers of excellence 3. Constructing /or expansion of library and center of excellence buildings 2. Number of TEIs that have demonstrated commitment of human, materials or financial resources to support BESO II key activities 1 Establishing furnished cluster coordinating units/committees and assigning full time personnel to the units/committees 2 Putting the organization and management of the cluster coordinating units in the official organograms of the TEIs 3 Assigning regular government budget to the unit activities 3. Number of TEIs that have established functioning cluster coordinating units or Task forces/ committees to coordinate activities (New) BESO II – FINAL EVALUATION 76 S.N Key Intervention Contractual Deliverables 4 The cluster coordinating units supporting primary school teachers 1 Look at component, Women teacher and female student support for key activities 4. Number of TEIs that demonstrated a commitment to increase female graduates, disaggregated by certificate and diploma courses through specific policies, introduction of new mechanisms of support or other concrete and verifiable actions 1. Establishing units /task forces/ committees 2 Participating in the national continuous assessment workshop 3 Design action plan to implement continuous assessment in their respective TEIs and school area cluster /satellite linkage schools 4 Implementing the designed action plan 5. Number of TEIs that have developed and using exemplary continuous assessment model in their courses and have extended the same to their school area cluster/or satellite schools (New) 1 Curriculum development of Alternative Teachers education for pastoral population 2 Revise the curriculum and use it to train facilitators 3 Conduct training on radio program to students 4 Implement radio broadcast programs 5 Furnish and equip the center 6 Prepare instructional kits to be distributed to the graduates 6. Development of center of excellence for Jigjiga Teacher Training Institute (TTI) /New 1. Establish equipped and furnished center 2 Train instructors in and outside of the country 3 Establish linkage with school area clusters to produce teaching aids 7. Development of center of excellence for Assella TTI/ Teacher Training college (New) 1 Produce TEI curriculum supplementary materials in all subjects 2 Produce primary school supplementary materials for school area cluster schools 8. Development of center of excellence for Debrebrehan Vocational and Teacher Training college (New) BESO II – FINAL EVALUATION 77 TABLE 7: PROJECT INTERVENTIONS: IN-SERVICE TEACHER EDUCATION S/N Key Interventions Contract Deliverables 1 - A monitoring guide on organizing for motive evaluation and instruments for teacher’s feedback was prepared - Data collectors from regions were trained to monitor the formative evaluation process - Kits were revised based on teacher’s feed back and local experts in English and in four languages, namely, Afan Oromo, Amharic, Somali, and Tigringa a) Number of multi- session self instructional kits content print materials audio cassettes for in-service continuing education for primary teachers developed and field tested (reported by topic or them of each kit and language of instruction) and handed over to REBS - First cycle and second cycle instructional kits were printed in Amharic, Afan Oromo, Somali ;and Tigringa and distributed to primary schools b) Number of multi session self instructional kits (content, print materials, Audio cassettes) for in service, continuing education for primary teachers produced, piloted and disseminated 2 - Teachers were trained in rounds on there use of self- In fractional kits, professional hand book and on Active learning, continuous Assessment Managing large class size, Action research Gender issues and curriculum Integration Number of primary teachers completing at least one self instructional kit (reported by region/21 me and by gender) 3 - School headmasters, educational officers were given training on different topics - The personnel mentioned above were also provided with training with teachers on topics such as instructional leadership, in-built supervision, - Topics covered from the professional hand book were school based professional development training curriculum integration and continuous assessment Number of school headmasters, regional zonal and woreda educational support personnel trained in constructive teacher support methods (reported by region/Zone, by type of personnel trained and by gender) 4 - Teacher Training institutes 433 primary schools and teachers were involved in establishment of a viable teacher strengthening system Establishment of a viable teacher strengthening system at receives continues support from the preservice training institutes 5 - Established Woreda Resource centers one in SNNP, Beninshangul-Gumuz and Somali and two in Oromia region - Equipped the woreda Resource centers with adequate training resources Number of Woreda with cluster Resource centers equipped with adequate training resources, budget and staff to support school cluster activities 6 - Compilation of data of female leadership not yet done at zonal and worda level percentage increase of female teachers in primary school leadership positions as directors and deputy directors BESO II – FINAL EVALUATION 78 TABLE 8 PROJECT INTERVENTIONS: SUPPLEMENTING MEDIA DEVELOPMENT TRAINING S/N Key Interventions Contract Deliverables A - Produced grades 1-4 IRI English programs and transmitted for lessons per week nation wide from the regional radio station - Teachers guides were printed and given out to schools - Developed and implemented a short training program for teachers in 12,000 primary schools in the country - Conducted IRI methodology workshops - Regions developed strategy and budget for the training program - IRI Methodology was integrated into teacher training - A national utilization assessment was conducted Number of primary schools in priority woredas using interactive radio in active learning made in grades 1-4 (as reported by region subject and by language of instruction ) B - Self-instructional teachers kits with modules, manuals posters and manipulations were developed field tested and produced. Copies were printed and distributed to primary schools in the country - School level component kits containing pasteurized lessons manuals, pocket boards and by books for English were produced and distributed to primary school in the country - Instructional materials comprising of 6 modules for individual teachers of grades 5-8 were developed and distributed. - Teachers were trained at central and cluster/ School level venues in the use self instructional teasers’ kits steady groups were established at school level to discuss further on self instructional kits Instructional and /or motivational short courses training modules Audio, video, Print targeted at women teachers developed, produced, delivered (reported by topic, type of media and language) 3 - Draft materials on civics and Ethical education, environmental education and HIV/ AIDS were produced in Afan Oromo, Amharic, Somali and Tigregna fied-tested and revise accordingly - Monitoring guides for cycles 1x2 supplementary materials were developed - Data collectors from 9 regions were trained to monitor the formative evaluation process Existing curriculum reviewed, and revised based on school feedback on key socially relevant topics (reported by topic, grade and language of Instruction) 4 - Modules on socially relevant topics for grades 5-8 were integrated into the classroom curriculum in four languages Afan Oromo, Amahric Somali and Tigrigna - Modules on HIV/AIDS manuals for grades 5-8 teachers, Civic and Ethical education for grades 5 to 6 Teachers Activity Grade, were printed - Printing of revised Environmental Education Activity Book in 10,500 copies and HIV/AIDS grade 4 student book in 54,355 copies in progress Curriculum on identified socially relevant topic integrated and prototype materials media produced (reported by topic grade level and language of instruction) BESO II – FINAL EVALUATION 79 TABLE 9: PROJECT INTERVENTIONS: LEADERSHIP AND PLANNING AND MANAGEMENT CAPACITY BUILDING Key Interventions Contract Deliverables 1. Conduct a system wide study on the efficient management of the education sector human resource 2. Design and development of computerized personnel management information system (PMIS) 3. Development of data collection forms 4. Orientation workshop for senior education management officers (or core group) on the function and use of PMIS 5. Basic computer skills training 6. Develop personnel status forms 7. Develop operation manuals 8. Assists with data collection and data entry 9. Facilitate validation of personnel data files 10. Conduct an intensive training on how to use and management the PMIS software for regional planning and management officers and IT employees from the Regions 11. Conduct training for data encoders and supervision of data entry 12. Conduct advanced network administration training 13. Procure and distribute computers to all Regional Education Bureaus 14. Provide additional training for on-line users on how to use the PMIS software 15. Assess design and implement a local area network. Provide user’s raining for all connected users. 16. Conduct advanced training on network administration for selected REB staff. 17. Strengthen regional-level PMIS and expand it to selected Woredas; 18. Procure and supply computers, printers, UPS for the selected Woredas 19. Finalize installation and use of the software 1. Number of regions that (a) have developed; (b) are maintaining and using computerized personnel system for at lease basic personnel functions (hiring, assignment, salary & benefits, incremental increases 1. Conduct system wide study on more efficient distribution and logistics of educational material at the regional and woreda levels 2. Design & develop the computerized materials management information system (MMIS) 3. Develop uniform asset coding structure for both fixed and non-fixed assets and materials 4. Develop data collection forms 5. Establish(baseline) initial inventory levels 6. Translate the program into Visual Ge’ez 7. Provide training on basic computer skills 8. Provide technical assistance and trainings to MOE and REBs to enable them fully utilize MMIS software 9. Train the key personnel in REBs to orient them about the main features of the MMIS, including data processing and provide data entry support. 10. Provide technical assistance in setting up the database and assist with data entry and oversee the data preparation to keep standards 11. Conduct training in appropriate report generation and their use in the day-to-day management 12. Finalize installation and use of the software 2. Number of regions that (a) have developed; (b) are implementing improved instructional materials procurement and distribution systems on a routine basis 1. Develop policy research agenda in consultation with central MOE and REBs 2. Prioritize the policy research agenda 3. Write terms of reference for the selected research topics 4. Recruit, hire and train policy researchers 5. Draft research document and distribute for review and comment 3. Number of regions that are undertaking improved learning assessment and other monitoring of educational quality at the classroom and school levels BESO II – FINAL EVALUATION 80 6. Finalize research document 7. Complete the publication and distribution of the research document 8. Conduct workshop to review key findings and recommendations of the research document 1. Develop and implement an integrated planning and budget process at the regional level 2. Strengthen the capacity of the Planning and Projects Department officers at the federal and regional level by: - developing training material - conducting successive seminars on planning, monitoring, strategies to improve access quality and leadership - conducting successive workshops on the use of indicators for monitoring the system and reforming school supervision 3. Upgrade and redesign the projection model from the three models (IPBCS, what if and the world bank model) that are in use in different regions and the center 4. Provide training on the features of the new projection model and provide technical assistance on the installation of the model both at the center and the Regional bureaus 5. Conduct Round 1 and 2 training of Woreda Education Officers on capacity building modules in various subjects ranging from introduction to the Educational System, to managing a Decentralized Education System, etc. 6. Foster data-based decision-making at all levels 4. Number of regions utilizing education information more efficiently for planning, budgeting, monitoring and evaluation 1. Assess training needs of Woreda Education officers, Kebele Education and Training Board members and school principals 2. Develop training modules/materials to enhance the capacities and skills of WEOs, KETB members and school principals to effectively manage the education/school system in their localities 3. Draft and translate training modules 4. Conduct Training of Trainers 5. Conduct training on Woreda Capacity Building modules 6. Conduct rounds of training for Kebele Education Training Board members and school personnel in various topics ranging from Community Participation at School Level to Educational Planning and Implementation at Kebele Level, etc. 5. Number of schools and Kebele Education and Training Board (KETB) with increased capacity in supervision, planning, etc. TABLE 10: PROGRAM INTERVENTIONS: FEMALE TEACHER AND STUDENT TEACHER SUPPORT S.N Key Intervention Contractual Deliverables 1 Strengthen Amhara and Oromiya regional networks to establish linkages within and between regions to build the capacity of gender focal points 2 Establish three networks for Harar and Jigjiga, SNNPR and Addis Ababa, and Tigay 3 Organize a national workshop on sensitization to increase female teachers Number of TEIs that demonstrated a commitment to increase female graduates(disaggregated by certificate and diploma courses) through specific policies introduction of new mechanisms of support or other concrete verifiable actions 1 Develop and produce ten materials (6 courses and 4 modules) based on the needs assessment conducted in 2003 addressing personal and professional needs of female teachers and students (focusing on materials on personal needs and professional needs) 2 Give training on leadership for female teachers from all regions of Ethiopia Percentage increase of female teachers in primary school leadership positions as directors and deputy directors BESO II – FINAL EVALUATION 81 3 Produce and distribute posters that promote women leadership TABLE 11: AED MANAGEMENT INTERVENTIONS & DELIVERABLES Key Interventions Contract Deliverables Baseline and recommended targets established/refined for all BESO-II Performance Monitoring Plan (PMP) IT indicators during the first 60 days after the contract effective date, so that they can be reporting in the FY 2004 R$ prepared by USAID in February 2002. On-going program-related monitoring and evaluation (M&E) of impact providing USAID required information for activities for all PMP indicators no later than November 1 of each year for the period ending September 30 (i.e., previous school year), beginning in 2002 and continuing through each year of the contract Annual surveys, topic-specific analysis, case studies, or other research that document lessons learned from new and innovative approaches in primary education quality and equity generated, distributed by the contractor (disaggregated by topic) Number of regions that are undertaking improved learning assessment and other monitoring of educational quality at the classroom levels (reported by region/zone, with reference of language of instruction). Completion of the analysis, report writing, publication and dissemination activities of the Second national Learning Assessment. BESO II – FINAL EVALUATION 82 TABLE 12: PROJECT INTERVENTIONS: MERA Key Interventions Contract Deliverables Develop M&E plan with refined indicators and targets Develop data collection instruments Test data collection instruments and implement Collect indicator data and analyze Compile indicator data Conduct internal review of indicator data Prepare PMP Report and submit to USAID Baseline and recommended targets established/refined for all BESO-II Performance Monitoring Plan (PMP) IT indicators during the first 60 days after the contract effective date, so that they can be reporting in the FY 2004 R4 prepared by USAID in February 2002. Develop M&E impact assessment tool Recruit field M&E specialist Train M&E specialist Conduct evaluation exercise Review findings with relevant partners and beneficiaries On-going program-related monitoring and evaluation (M&E) of impact providing USAID required information for activities for all PMP indicators no later than November 1 of each year for the period ending September 30 (i.e., previous school year), beginning in 2002 and continuing through each year of the contract Facilitate the development of a research agenda in collaboration with the MOE and REBs Prioritize research topics Draft terms of reference with the TWG and finalize Recruit and hire researchers and writers Edit and publish research paper Publish and disseminate research paper Conduct a workshop to review key findings Annual surveys, topic-specific analysis, case studies, or other research that document lessons learned from new and innovative approaches in primary education quality and equity generated, distributed by the contractor (disaggregated by topic) Conduct preparatory activities for national learning assessments Assess the curriculum of regions for drafting data collection instruments Conduct a conference for establishing a national advisory committee Prepare and review data development instruments with the national advisory committee Draft terms of reference Recruit researchers and writers Conduct the assessment Number of regions that are undertaking improved learning assessment and other monitoring of educational quality at the classroom levels (reported by region/zone, with reference of language of instruction). Completion of the analysis, report writing, publication and dissemination activities of the Second national Learning Assessment.