Mid-term Evaluation of Basic Education Strategic Objective (BESO) II Project Funded by USAID/Ethiopia Implemented by the Academy for Educational Development in consortium with American Institutes of Research April 28, 2004 Submitted under GEM Delivery Order 811, Contract No. FAO-I-00-99-00010-00 by Birhanu Dibaba, Teacher Education Specialist Melanie Sanders-Smith, Evaluation Specialist & Team Leader Wossenu Yimam, Educational Planning and Management Specialist through International Science and Technology Institute, Inc. (ISTI) 1820 N. Fort Myer Drive, Suite 600 Arlington, VA 22209 USA ii EXECUTIVE SUMMARY The purpose of this evaluation is to conduct a comprehensive review of the Basic Education Strategic Objective (BESO) II project funded by USAID and being implemented by the Academy for Educational Development (AED) and the American Institutes of Research (AIR). As a formative evaluation, the findings and recommendations provide USAID and the Government of the Federal Democratic Republic of Ethiopia (GFDRE) with information and analysis that will help them make decisions related to the design and implementation of the project. AED’s contract was signed in July 2002 and has two base years and three option years. This evaluation covers the first eighteen months of the base period. With a focus on quality and equity in primary education, BESO-II supports USAID’s strategic plan and is consistent with the GFDRE’s Education Sector Development Program (ESDP). The major emphasis of BESO-II is on capacity building and the project activities address related needs in pre-service and in-service teacher training, supplementary media development and training, integration of socially relevant topics into the curriculum, women teacher support systems, personnel and materials management, and monitoring and evaluation systems. Achievement of Targets – Despite the slow project start-up, AED has made good progress toward achieving quantitative target outputs specified in their contract. With approximately six months remaining (25% of Phase I), many of the target outputs have already been met or exceeded and AED expects that most of the others will be completed by the end of Phase I. There are only a few Year 2 targets that may not be fully achieved. At the March 2004 Technical Working Group (TWG) meeting, the chairman acknowledged that significant progress has been made in project implementation, but noted that AED must maintain a rapid pace if all targets are to be achieved before the end of the base period. AED reports the following: • Pre-service: Target achieved with 19 Teacher Education Institutes (TEIs) having improved the quantity and quantity of library materials, and the resources and operation of their Instructional Resource Centers (IRCs); AED is also developing minimum standards for TEI libraries and IRCs. • In-service: Exceeded target (35,000) with 37,539 teachers reached through face￾to-face, training of trainers (more than 80%) or the self-study professional development handbook (less than 20%); additional teachers will be reached soon when the self-instruction kits (translated into five languages) are distributed to 45,000 grades 1-4 teachers and 15,000 grades 5-8 teachers; 4,437 school directors/vice-heads (target is 3,500) have been trained in constructive teacher support methods; 756 (target: 800) education officers have been trained. • Supplementary Media Development: AED estimates that 4,000 schools (target: 1,500) will be using interactive radio instruction (IRI) for English in grades 1-2, which is broadcast nationally, as soon as the current program for training teachers is completed; targets for grade 3 will be reached in 2004 and for grade 4 once the GFDRE completes the development of the curriculum. iii • Socially Relevant Curriculum Integration: AED has reviewed grades 1-4 curriculum; is in process of reviewing grades 5-8 curriculum and assessing regional curriculum; developed and field-tested modules on socially relevant topics for grades 1-4; materials being developed on socially relevant topics for grades 5-8. AED expects to complete these activities and reach the target by the end of the base contract period. • Women Teacher Support Systems: This component of the project had a slow start with delayed hiring, but AED has made some progress in developing linkages between women’s associations and women teachers (Amhara region), establishing linkage activities, and developing short courses and modules targeted at women teachers. AED still has much to do to accomplish the target outputs for this component (linkages in 4 regions and final production of the short courses modules). • Planning and Management Information Systems: Although a lot of progress has been made in this area, the targets will not be 100% reached in all regions by the end of the base period. (1) The Personnel Management Information System (PMIS) software has been developed and customized to reach region-specific needs, translations are nearly complete, manuals have been developed and translated, training has been conducted, and data collection and reconciliation is underway. Full implementation of PMIS is expected in all but three the regions (Gambella, Oromia, SNNP) by August 2004. (2) The Materials Management Information System (MMIS) software has been developed and translated, a user’s manual has been developed and is being translated, training conducted, and inventory and data entry is underway. AED expects that MMIS will be fully operational in most regions by August 2004. (3) AED has installed a long-range planning projection model and three decision support systems in the regions, facilitating their development of annual, short-term, and long-term plans. (4) A computerized monitoring system has been developed and integrated into the planning projection model. (5) AED has conducted workshops in all regions on integrating annual planning with the budgeting process. (5) A course workbook has been developed for the Woreda Capacity Building program and the training is scheduled to take place in April 2004. • Monitoring, Evaluation, Reporting, and Analysis (MERA): AED developed a Performance Monitoring Plan for the project, but submission of the draft and final plans did not meet USAID’s timeliness and quality standards. AED is conducting several studies that are at various stages of design and implementation. The National Learning Assessment is on target. In the remaining months, AED will need to move quickly to complete all the tasks required to meet the target outputs, complete implementation of the eleven regional and one central/nations Implementation Plans, provide follow-up support to ensure that the inputs to date are fully operational, and work to ensure that the desired impact is achieved and measured. Considerable work remains to be done and AED will need to be diligent and remain focused to complete it all. AED appears to have picked up their pace and should meet their contractual obligations in most areas. iv Satisfaction and Impact: Overall, beneficiaries report a high degree of satisfaction with the design of BESO-II, and the quality and relevancy of implementation activities. They are especially pleased with the technical assistance, training, and materials they have received through BESO-II. They are, however, highly unsatisfied with the timeliness and transparency of project implementation, particularly issues related to regional implementation plans, financial matters, and project procurements, including vehicles, equipment, and materials. Satisfaction was highest with beneficiaries at the central level and lowest at the regional level, with TEIs and cluster schools somewhere in between. Beneficiaries provided anecdotal indications of impact at their institutions. In particular, the teachers at the TEIs and primary schools are applying active learning methodologies in their classrooms, using continuous assessment strategies, providing extra support to female students, and benefiting from newly established forums for sharing experiences. The evaluation team witnessed some of these changes during site visits. Beneficiaries claim that these changes have resulted in increased enrollment of girls, lower dropout rates, and lower repetition rates. AED will collect data to verify these claims as they implement the PMP over the coming months. Constraints: Project implementation is constrained by a number of factors external and internal to AED. The external constraints include: the devolution process in the primary education system, a new curriculum for primary school, the use of a large number of local languages in the primary schools, understaffing of the MOE at all levels, frequent transfers within the MOE resulting in a high turnover rate, challenges inherent in the clustering system, lack of MOE incentives and budgets for in-service training, drought conditions, and political unrest. Constraints related to USAID include: the contracting mistake made in initially authorizing AED to issue grants to TEIs (this authority was later rescinded) and the need to identify another sub-disbursement mechanism resulting in a 10-month delay in funding their implementation plans; challenges in getting approvals for non-expendable property; delays in getting the source-origin waiver for project vehicles; and issues associated with the Valued Added Tax (VAT) requirements. And within AED, the constraints include: slow recruitment of staff, a number of issues related to field staff, consultant selection, lack of accountability for deliverables, challenges in focusing on high priority tasks, organizational structure and other issues between the operations team and technical teams, and financial and administrative management. Lessons Learned and Recommendations: USAID tasked the evaluation team with exploring lessons learned and making recommendations in three key areas: project design, implementation, and AED management. • Project Design: The project design is extremely broad in geographic reach and the time allowed to achieve the objectives in each region is short. This has made it unwieldy to implement, difficult to focus and to achieve sustainable impact. Recommendation A.1: USAID should add no new tasks to AED’s base-period contract and AED should focus on completing current implementation plans by the end of the base contract period. v Recommendation A.2: An early deliverable of BESO-II, Phase II should be a sustainability plan for all activities implemented in Phase I. Recommendation A.3: USAID should limit BESO-II, Phase II activities to the current scope of work. While it is necessary to be flexible and responsive to changing MOE needs and USAID priorities, it is also important to remain focused and provide the follow-up support necessary for maximizing the impact of base￾period investments. Possible adjustments to the current scope of work that could maximize impact include: A.3.1 – Revisit the geographical coverage, especially in light of USAID’s new strategic plan and emphasis on food insecure areas. A.3.2 – Focus resources on the most committed TEIs. A.3.3 – Expand the number of TEIs receiving support for Centers of Excellence, but remain focused on teaching aids and teaching materials. A.3.4 – Provide follow-up support and materials to the first cohort of graduates from the Higher Diploma program. A.3.5 – Conduct on-going rounds of leadership training to keep pace with the continuous turnover of personnel. A.3.6 – Provide additional resources to the Phase I “impact” cluster schools, rather than adding new impact clusters each year, to deepen impact. A.3.7 – Support creative approaches to dealing with the issues associated with clustering, i.e., distance materials, school-based training, or expert-team approach. At this point, it appears that there is more money remaining in AED’s contract than they will likely spend by August 2004. However, it is not clear that there are sufficient resources currently in the contract for Phase II, should USAID chose to extend AED’s contract through the option period. Recommendation A.4: AED should conduct a thorough analysis of the activities they expect to complete before August 2004 and their associated costs, and calculate the amount that will remain unspent. USAID, in consultation with MOE, should then determine whether it is best to (a) make adjustments in the budget to increase funding for on-going activities that could make the best use of additional funds, or (b) carry-over any remaining funds from Phase I to Phase II or to a BESO-III. Recommendation A.5: To ensure there is adequate funding for Phase II, USAID and MOE should carefully consider their expectations for the future and the implementing partner should conduct a careful analysis of the corresponding administrative requirements, including the type and number of staff. vi There will continue to be a challenge having the desired impact on the in-service teacher training activities without sufficient financial support from the MOE. Recommendation A.6: USAID should seek a commitment from the MOE to provide funding for per diem and transportation to teachers who participate in in￾service training. If this commitment is not made, USAID should reconsider its expectations for impact on the in-service training program. • Project Implementation and Management: The issues related to project implementation require discussion and action from USAID, MOE, and AED. First, the one that requires action from the MOE: Recommendation B.1: USAID should reach agreement with the MOE on the most significant constraints within the MOE’s control: clustering issues, understaffing, personnel transfers, and incentives for in-service training. Until these issues are adequately addressed, they will continue to hamper BESO-II implementation. If there are adequately addressed, USAID will need to address the implications related to project design and implementation. To respond to the primary concerns of beneficiaries: Recommendation B.2: AED should conduct a thorough assessment of its financial and procurement system and procedures to better understand the bottlenecks and implement solutions, especially the relationship between the technical and financial teams of AED. And to deal with a related internal AED concern: Recommendation B.3: AED should implement a strategy for achieving greater clarity and transparency in financial and procurement related matters. To address the personnel management issues of greatest concern to USAID and AED staff: Recommendation B.4: AED should give immediate and high priority attention to addressing management issues, including possible management changes that affect project implementation and the perceptions of the client: B.4.1 – Conduct a thorough review of the issues related to the field staff and implement solutions, including an overall review of the effectiveness all AED and local staff as suggested by the TWG. B.4.2 – Reach agreement with USAID on the number of type of consultants that can be used to expedite and complete Phase I activities, and commit to obtain USAID approval before executing consultant agreements. vii B.4.3 – Implement a system for making management and staff accountable for missing deadlines (that are within their control) and for drifting from the priorities of the project. B.4.4 – Conduct team-building activities to create greater understanding and cooperation between the field and central offices, and between the operations and technical teams. Implementation of these recommendations should address the key issues related to design and implementation. There are, of course, other improvements that could be made, but USAID, MOE, and AED should focus on these first. As the end of the base period approaches, USAID and MOE are left with two basic options: (1) end BESO-II, design BESO-III, and conduct an open competition among implementing contractors, or (2) continue BESO-II by exercising part or all of the option years in AED’s contract. The evaluation team recommends: Recommendation B.5: Exercising some portion of the option years in AED’s contract to allow sufficient time for: (a) AED to follow-up on Phase I activities, deepen impact, and plan for sustainability; and (b) USAID and MOE to redesign BESO in light of the Mission’s new strategic plan and lessons learned to date. In exercising the option, USAID should reach agreement with AED on how and when the implementation and management issues will be resolved. viii TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS................................................................. X I. INTRODUCTION ...........................................................................................1 A. Background ............................................................................................................... 1 B. Purpose of Evaluation .............................................................................................. 1 C. Methodology .............................................................................................................. 2 1. Key Informant Interviews....................................................................................... 2 2. Field Visits.............................................................................................................. 2 3. Document Review................................................................................................... 3 II. KEY FINDINGS .............................................................................................3 A. Achievement of Targets............................................................................................ 3 1. Pre-Service Teacher Training ................................................................................. 4 2. In-Service Teacher Training ................................................................................... 5 3. Supplementary Media Development and Training................................................. 5 4. Socially Relevant Curriculum Integration .............................................................. 6 5. Women Teacher Support Systems.......................................................................... 6 6. Planning and Management Information Systems ................................................... 7 7. Monitoring, Evaluating, Reporting, and Analysis (MERA)................................... 8 B. Satisfaction and Impact............................................................................................ 9 1. Central MOE........................................................................................................... 9 2. Regional Education Bureaus................................................................................. 11 3. Teacher Education Institutes................................................................................. 12 4. Cluster Schools ..................................................................................................... 16 C. Relationship to Mission Strategic Plan ................................................................. 18 1. Relationship of IRs to the SO ............................................................................... 19 2. Relationship of Project Components to IRs.......................................................... 19 3. AED’s Implementation of Project Components ................................................... 20 4. Measuring Outputs and Impact............................................................................. 21 D. Lessons Learned...................................................................................................... 26 1. Design and Resources........................................................................................... 26 2. Project Implementation......................................................................................... 29 3. AED Management ................................................................................................ 34 4. Linkages with CGPP............................................................................................. 38 III. RECOMMENDATIONS............................................................................39 ix A. Design and Resources ............................................................................................. 39 B. Project Implementation and Management........................................................... 41 IV. ANNEXES................................................................................................43 A. Evaluation Statement of Work .............................................................................. 43 B. List of Individuals Interviewed.............................................................................. 47 C. Selected Documents Reviewed............................................................................... 51 D. Table of Output Targets and Progress.................................................................. 52 E. Proposed Revisions to the Contract Output Targets........................................... 64 x ACRONYMS AND ABBREVIATIONS AED Academy for Educational Development AIR American Institutes of Research BESO-II Basic Education Strategic Objective CGPP Community Government Partnership Program CRC Cluster Resource Center ELIP English Language Improvement Program EMA Educational Media Agency ESDP Education Sector Development Program GFDRE Government of the Federal Republic of Ethiopia ICDR Institute for Curriculum Development & Research IP Implementation Plan IR Intermediate Result IRC Instructional Resource Center IRI Interactive Radio Instruction ISP Integrated Strategic Plan MERA Monitoring, Evaluation, Reporting & Analysis MMIS Materials Management Information System MOE Ministry of Education NLA National Learning Assessment NOE National Organization for Examinations PC Pedagogical Center PMIS Personnel Management Information System PMP Performance Monitoring Plan REB Regional Education Bureau SDU Staff Development Unit SNNP Southern Nations, Nationalities, and Peoples SO Strategic Objective TEI Teacher Education Institute TEMSD Teaching & Education Management Staff Development Department TESO Teacher Education System Overhaul TOT Training of Trainers TTC Teacher Training College TTI Teacher Training Institute TWG Technical Working Group USAID United States Agency for International Development VAT Value Added Tax WCB Woreda Capacity Building 1 I. INTRODUCTION A. Background Since 1994, USAID has supported Ethiopia in the effort to improve the quality and equity of education in an expanding primary education system. The Basic Education System Overhaul (BESO) I project was a 7-year (1994-2001) cooperative effort of the Ethiopian government and USAID in Tigray and the Southern Nations, Nationalities, and Peoples (SNNP) regions. The focus of BESO I through AED was to provide assistance in support of teacher development (pre-service and in-service); effectiveness of the content and quality of materials; interactive radio instruction (IRI); planning, policy, decentralized management; and more efficient school financing. The Basic Education Strategic Objective (BESO) II project began in 2002 and continues to support enhancement of quality and equity in primary education. Under BESO-II, USAID has extended its support to all the regions of Ethiopia for most project activities. The major emphasis of USAID’s assistance, besides the Community Government Partnership Program (CGPP), which is outside AED’s purview, is in pre-service and in￾service teacher training, supplementary media development and training, socially relevant curriculum development and training, women teacher support systems, personnel and materials management, and monitoring and evaluation systems. BESO-II is consistent with the objectives of the government of Ethiopia’s Education Sector Development Program (ESDP), with a clear objective of improving quality and equity at the primary level. It also contributes to three of USAID’s intermediate results in basic education: • IR1: Quality of professional education personnel enhanced • IR2: Teacher-learner support systems strengthened • IR4: Education planning and management strengthened The Academy for Educational Development (AED) was the implementing contractor for BESO-I and continues this role in BESO-II. American Institutes of Research (AIR) is serving as a subcontractor to AED. BESO-II is a 5-year contract, signed in July 2002, with two base years and three option years. This evaluation covers the first eighteen months of the base period. B. Purpose of Evaluation The purpose of this evaluation is to do a comprehensive review of the BESO-II project funded by USAID and being implemented by AED. As a formative evaluation, the findings and recommendations provide USAID and the Government of the Federal Democratic Republic of Ethiopia (GFDRE) with information and analysis that will help them make decisions related to the design and implementation of BESO-II. 2 USAID directed the evaluation team to look specifically at: • Project Inputs and Results • Ministry Satisfaction • Relationship to Mission Plans • Lessons Learned • Recommendations This report, therefore, addresses each of these areas, in roughly the same order presented in the evaluation scope of work. (See Annex A for the full Statement of Work.) C. Methodology The 3-person evaluation team used rapid appraisal techniques to collect data necessary for responding to the statement of work. These techniques included key informant interviews, field visits, and a review of major project documents. 1. Key Informant Interviews The team conducted interviews with personnel in the following organizations: • USAID – staff in the education office • AED – 5 expatriate staff and 13 central office staff selected through a criteria￾based sampling methodology • Ministry of Education (MOE) – heads (or their representatives) of the Planning and Projects Department, Teaching and Education Management Staff Development (TEMSD) Department (including TESO), Educational Media Agency (EMA), Institute for Curriculum Development and Research (ICDR), and National Organization for Examinations (NOE) • BESO Partners – Save the Children, World Learning International, and Tigray Development Association • TWG - A discussion with the TWG on the draft report of the evaluation, which resulted in further inputs to the evaluation See Annex B for a list of individuals interviewed. 2. Field Visits The team visited four regions selected by USAID: Amhara, Oromia, SNNP, and Somali. At these locations, we interviewed key informants, reviewed documents, verified procurements, and observed activities. Within each region the team visited: • Regional Education Bureau (REB) – One per region • Teacher Education Institutes (TEIs) – At least one Teacher Training Institute (TTI) or Teacher Training College (TTC) in each region; the sample included a range of TEIs and two Centers of Excellence • Woredas – One for each set of cluster schools visited 3 • Cluster Schools – At least two per region, with a mix of urban and rural schools, as well as impact and general schools; some cluster schools in the sample are also working with the BESO/CGPP • AED - Operations and technical staff stationed in each region See Annex B for a list of sites visited and individuals interviewed at those sites. 3. Document Review There are a number of key documents that were central to the evaluation study. Most of these documents were developed by USAID or AED. The team reviewed the USAID strategic plan, AED contract, implementation and performance monitoring plans, progress and status reports, and other project related documents. See Annex C for a full listing of documents reviewed. II. KEY FINDINGS A. Achievement of Targets Based on instructions from USAID, this section reports on AED’s achievements in relationship to the target outputs specified in their contract. AED provided current data for all project components (see Annex D) based on revised targets agreed upon with USAID in mid-2003. While Annex D provides complete detail on the targets, this section will provide a summary review of progress toward targets, prospects for achieving the targets by the end of Phase I, and some achievements that go beyond the targets. Implementation constraints are treated at a general level in the Lessons Learned section, so this section only addresses constraints specific to project components. There are two important points to consider when reviewing this section of the report: • The evaluation team relied exclusively on AED’s assessment of their achievements because (1) the output targets are strictly quantitative and (2) it would have been extremely difficult for the evaluation team to have verified all the numbers within a short time period. • The qualitative assessment of project outputs, which is based on feedback from a sample of beneficiaries and site visits, is covered in the next section of the report (Satisfaction and Impact). It will address issues related to the beneficiaries’ satisfaction with BESO-II design and implementation, application of training, utilization of procurements, evidence of impact, and sustainability. Despite the slow project start-up, delays, and missed deadlines, AED has made good progress toward achieving the target outputs, according to their reports. With approximately six months remaining (25% of Phase I), many of the target outputs have already been met or exceeded and AED expects that most of the others will be completed by the end of Phase I. There are only a few Year 2 targets that may not be fully achieved. At the March 2004 Technical Working Group (TWG) meeting, the chairman 4 acknowledged that significant progress has been made in project implementation especially after the reorganization of the AED and the recommendations of the TWG, but noted that AED must maintain a rapid pace if all targets are to be achieved before the end of the base period. 1. Pre-Service Teacher Training The targets for pre-service were reasonable and AED has had little trouble in achieving them. In fact, at this point all the target outputs for pre-service training have been met or exceeded, with one exception. AED reports that their achievements in pre-service are: • 19 Teacher Education Institutes (TEIs) that have improved the quantity and quality of library materials; AED also developed minimum standards for TEI libraries • 19 TEIs that have improved the resources and operation of an Instructional Resource Center (IRC); AED is also developing minimum standards for Pedagogical Centers (PCs)/IRCs • 12 TEIs (more than double the target) that have provided adequate funds (using their own funds) in their annual budget to maintain their library • 19 TEIs that demonstrate adequate commitment of human, materials, or financial resources to the activities The one target that has not been met is “the number of TEIs that increased the percentage of women graduates with GPA 2.75 and above.” AED has collected data for female students, but unfortunately they found no trends. USAID and AED have discussed this issue and agreed that GPA is not a reliable indicator for quality training because there is a lot of disparity among TEIs in their use of continuous assessment and calculations of GPAs. AED is facing a challenge in getting all the TEIs to complete implementation of their own plans that were funded through subcontracts from AED. Until each TEI fully implements its plan and has reported on its activities and procurement, AED cannot release the final payment to that TEI. While many have completed their plans, others have not. AED has recently conducted a risk assessment, identifying the TEIs that are showing the least progress in implementing their plans. AED will follow-up with these TEIs to help them complete their activities, providing professional assistance with conducting workshops, finalizing procurement, and preparing the required reports. Where these efforts are insufficient, AED will begin the process of recovering funds from the TEIs. One of the main problems TEIs face in completing their plans are that the teachers have heavy workloads—associated with normal class loads, extension classes, distance education, and the higher diploma program—and frequent turnover in TEI leadership and professional staff. Since the beginning of BESO-II, 10 of the 19 TEIs have changed deans, and one has changed twice. This has meant that AED has had to provide more support than originally planned. 5 2. In-Service Teacher Training AED is making progress in producing the self-instructional kits and has exceeded the training targets for school heads and officers in regional, zonal, and woreda offices. AED’s progress to date includes: • 2 self-instructional kits for first cycle teachers have been developed, translated, and field tested for grades 1-2 and grades 3-4 • 2 self-instructional kits for second cycle teachers have been developed and are in the editing stages; targeted for translation and field-testing by the end of Phase I • 50% (22,500) of the total number of kits for first cycle teachers have been produced; the remainder (22,500) will be produced by the end of March 2004 • 163 teachers from 5 regions have completed at least one module of the kits through field testing; after kits have been distributed AED will collect more data on the number of teachers completing at least one module • 4,437 school directors (including vice-heads who have been delegated responsibility for academic affairs) have been trained in constructive teacher support methods (target: 3,500); 756 education officers (target: 800) have been trained; the targets will be exceeded by the end of Phase I • 37,539 teachers (target is 35,000) have been reached, over 80% in face-to-face training and the rest through distance education; the coverage among satellite schools would be greater if the teachers did not have to travel so far, if they received per diem during training, and if there were some external incentives It is also important to note that AED’s cluster training in the satellite schools is just one round of training, which is below the capacity plan envisaged by AED. One of the things AED plans to focus on in the remaining months of Phase I is to provide the materials and equipment requested by impact Cluster Resource Centers (CRCs) and ensure that training has the desired impact at the classroom level. The multiplier training has not been as extensive as planned, due to great distances between cluster centers and satellite schools, the lack of funds to cover transportation and per diem, and absence of any other incentive for teachers to attend the training. To address this constraint, AED (a) has developed a Self-Study Professional Handbook that teachers can use without the need to travel and (b) is proposing to work with REBs and woreda education officers to form “training zones,” which is a grouping of woredas. Under this new approach, “staff development fellows” would be nominated from each school, with the expectation that they would train the other teachers in their school. The overall number of satellite teachers required to travel would be reduced (compared to the current CRC-satellite school approach), and BESO-II would cover the per diem costs. School directors would take greater responsibility for assuring school-level training, contributing to sustainability. 3. Supplementary Media Development and Training AED has exceeded the target for reaching grades 1-2 through interactive radio instruction: 6 • AED estimates that 4,000 schools (target: 1,500) will be using interactive radio instruction (IRI) for English in grades 1-2, which is broadcast nationally, as soon as the current program for training teachers is completed • Delays in ICDR’s development of the curriculum for grades 3-4 has delayed AED’s efforts; vacancies at EMA prevent AED from replacing three IRI staff 4. Socially Relevant Curriculum Integration Good progress is being made in the area of socially relevant curriculum integration and it appears that AED will achieve the targets. The status of the targets is: • Reviewed grades 1-4 curriculum (ICDR syllabi); completed regional curriculum assessment in cohort 1 target regions • Review is in progress for grades 5-8 curriculum (ICDR syllabi) and regional curriculum assessment in cohort 2 target regions; plan to be completed by the end of March 2004 • Developed and field-tested modules on socially relevant topics for grades 1-4 for cohort 1 and 2 regions • Materials being developed on socially relevant topics grades 5-8 cohort 1 regions; plan to be completed by the end of March 2004 AED has developed, translated, and field-tested the following supplementary materials: • Student booklet entitled Let’s Talk about HIV/AIDS for grade 4 students • Teacher’s Activity Guide for Civic and Ethical Education Using Student￾Centered Active Learning Methods: Some Suggestions for grade 1 to 4 teachers • Environmental Education Activity Book for grade 1-4 teachers 5. Women Teacher Support Systems AED had a late start in this area due to delays in hiring their Women Support Officer. It has also taken considerable time to prepare drafts and then have them reviewed by many stakeholders and subsequently revised. However, they have progressed in developing linkages between women’s associations and women teachers, establishing linkage activities, and developing short courses and modules targeted at women teachers. So far, AED has progress in the following areas: • 1 regional linkage established (target is 4) within the Amhara region between the REB, Women’s Affairs and TEIs gender offices; TEIs have organized a competition among the best performing female students in the TEIs and the best three are to be rewarded • A linkage between the Oromia and Somali regions has been created and they have identified common problems of female trainees and have devised strategies to solve the problems with the membership of the linkages with teachers’ associations • Completed Gender Issues Awareness Workshops and one national activity on networking and study skills • 6 short courses targeted at women teachers are in the development stage; expect to be completed by March and delivered by July 2004 7 • 4 modules targeted at women teachers are in the development stage; many stakeholders have reviewed the materials, including the beneficiaries, and AED/Washington is doing the final review. When endorsed by USAID, some will be translated into 2 local languages and then will be delivered in print form and through radio broadcast • Training on leadership skills was given to 59 female teachers in the Oromia region 6. Planning and Management Information Systems When assessing achievement of targets for this component, it would appear to be lagging if one only looked at the final output target. However, it is useful to look at the steps being taken toward each target: • Personnel Management Information System (PMIS): PMIS software modified to meet region-specific needs and installed in all regions. Translated into Amharic; translation into Afan Oromifa and Somali in progress. Thirteen data collection formats prepared and printed in all but 2 regions. Data collection training workshop held in 7 regions. Database management personnel trained in 7 regions. Managers of PMIS trained in 5 regions. Operations manual and user manual completed. Operations manual translated into Amharic; translation into Somali and Afan Oromifa in progress. Data collection and reconciliation is underway in 7 regions. Full implementation of the PMIS is expected in all regions by the end of the base contract period except in three regions: Gambella, Oromia, and SNNPR. • Materials Management Information System (MMIS): MMIS software has been adapted and installed in most regions. Initial orientation given in all regions. The coding structure for inventory control has been developed, which is applicable for all regions. Inventory is completed in 4 regions and is under way in 5 regions. Entry of inventory control data is completed or in progress in 5 regions. User training has been implemented in four regions. The interface of the software has been translated into Amharic and Oromifa languages. User’s manual has been developed. Translation in progress into Amharic and Oromifa. AED expects that the system will be operational in most regions by the end of Phase I. • Using education information for planning and decision making: AED has installed a long-range planning projection model and three decision support systems in all but one region (Dire Dawa because of hardware issues). Up to five experts from all regions have been given a weeklong training session in the use of these systems. In addition, the regional planning officers have conducted periodic, on￾going short training in their respective regions. The regions have capability to use the planning projection model for annual, short-term, and long-range plan development. Oromia, Somali, Benshangul Gumuz, and Gambella regions have developed five-year plans using the projection model. AED anticipates that all the regions will be using the computerized system for developing long-range plans by the end of Phase I. • Integrating monitoring & evaluation into the planning process: A computerized monitoring system at the regional level has been developed and integrated into the planning projection model. 8 • Integrating activity/program planning with budget: AED has conducted at least one workshop in all the regions on the methodology and process of the decentralized planning and budgeting system. In six regions AED has conducted at least two workshops on this topic. AED has developed a computerized model for estimating capital budget and for recurrent budget. The planning team has also been tasked with developing the new Woreda Capacity Building (WCB) program, added through a contract modification. This program will cover over 600 woredas in Ethiopia through a 10-day training program in approximately 35 locations all over the country. Approximately 3,000 woreda personnel will be trained through this program by the end of June 2004. Every woreda will use its own education system and population databases for the development of the woreda plans. A course workbook for the training program has been developed and validated through a national workshop with participation from all regions. The workbook has been translated into three local languages: Amharic, Afan Oromifa, and Somali. Translation into Tigrigna is in progress. Development of woreda levels educations system databases and school age population projections for 26 years has also been completed. AED’s efforts in implementing WCB have not been as smooth as USAID would like. After signing a subcontract agreement for developing training modules, the MOE put it on hold for nearly a month while they identified their own writers. USAID was also involved in writing modules and was disappointed in some of AED’s efforts to write modules and incorporate feedback. The WCB has placed heavy demands on the technical and managerial time of AED’s regional and central staff. Though it received authorization from USAID to hire additional staff or consultants as needed, AED did not identify a specific level of effort needed for this activity, and consequently has stretched its staff, resulting in delays with WCB and compromises to other planning activities. 7. Monitoring, Evaluating, Reporting, and Analysis (MERA) This component of the project got off to a very slow start, with long delays in submission of the Performance Monitoring Plan (PMP). A lot of activities are now underway and, if all the activities are completed, AED will exceed the targets in this area. AED has made the following progress to date: • PMP submitted in December 2003 (due in October 2002). • Annual dissemination workshop planned for April 2004 to review performance indicators and targets for 2004 at education system, IR and Sub-IR levels; create awareness on the responsibilities of data collection analysis and reporting; review progress in monitoring and evaluation of AED activities, especially those in the PMP. • Three studies in planning stage (target: 2): (1) study on successful schools in Ethiopia and how wastage (dropout and repetition) is being minimized (yet to be designed), (2) the economic costs of wastage (designed, but behind schedule), and (3) educational governance study (TOR developed and consultants are being 9 hired). At the March 2004 Technical Working Group, however, MOE counterparts noted that the first two studies had been delayed significantly, despite repeated requests for AED to move them forward. • 2 nd National Learning Assessment (NLA), which is part of the PMP, is in progress, with data collection taking place in April 2004. B. Satisfaction and Impact This section of the report will focus on beneficiaries’ satisfaction with BESO-II and the impact it has had on their institutions. The information for this section is based on interviews and site visits with a sample of the beneficiaries, which are grouped into four categories: central MOE, Regional Education Bureaus (REBs), Teacher Education Institutes (TEIs), and cluster schools. Overall, beneficiaries report a high degree of satisfaction with the design of BESO-II, and the quality and relevancy of implementation activities. They are especially pleased with the technical assistance, training, and materials they have received through BESO-II, and report that these are having an impact on their institutions. They are, however, highly unsatisfied with the timeliness and transparency of project implementation, particularly issues related to delays in completing implementation plans, financial matters, and project procurements, including vehicles, equipment, and materials. Satisfaction was highest with beneficiaries at the central level and lowest at the regional level, with TEIs and cluster school somewhere in between. For each beneficiary, the components they are involved with are deemed the most important; no beneficiary could identify a project component or activity they considered unimportant. 1. Central MOE The evaluation team talked to representatives of five departments and organizations collaborating with BESO-II on national and central programs: Planning and Projects Department, Teaching and Education Management Staff Development (TEMSD) Department (including TESO), Educational Media Agency (EMA), Institute for Curriculum Development and Research (ICDR), and National Organization for Examinations (NOE). They all reported a high level of satisfaction and are generally pleased with the progress that has been made so far. Beneficiaries at this level believe the project design is very responsive to their needs and is critical to filling in gaps that the MOE cannot fill with its own resources. They are pleased with the training, the technical assistance, and financial support they have received from BESO-II. There is no component that they would recommend being eliminated in subsequent USAID efforts. In particular, they mentioned: Intermediate Results Comments on Specific Activities IR1 Activities In-service and pre-service activities are progressing very well. Women’s support systems are now starting to move, but it is too soon to make a judgment about the activities. Assistance with distance education has been timely and 10 Intermediate Results Comments on Specific Activities helpful. Funds provided for the higher diploma and ELIP programs are valued. BESO-II’s full support of TESO has been appreciated. IR2 Activities Highly satisfied with Interactive Radio Instruction (IRI). BESO-II’s assistance—material and technical assistance— has been immense. Schools like IRI and want more, so more BESO-II assistance may be requested. Development of socially relevant curriculum seems to be progressing well and the materials are very helpful. All the financial assistance is appreciated. There is a smooth and positive working relationship with AED. IR4 Activities Woreda Capacity Building is progressing well. Lots of other good things started, but AED’s imbalanced division of labor within this IR affects timeliness. Would have appreciated being consulted more regarding work with education management information system activities. PMIS is going well and promises to be very useful. Looking forward to full implementation of MMIS. Have benefited greatly from assistance with educational planning assistance. MERA Activities Especially pleased with assistance on the National Learning Assessment; AED has given valuable input to this very important activity. A number of studies have been initiated but there is little time left in contract to complete them. Beneficiaries expressed confidence that most of these activities are sustainable because of the strong commitment of the MOE. Also, the demand for the activities is estimated to be sufficiently high that institutions will find a way to continue them, with or without BESO-II funding. Nevertheless, there should be a phase out plan to ensure steady progress toward sustainability. The major impediment to sustainability, at this point, is the constant turnover of MOE management and staff throughout the country. This makes it difficult to sustain capacity within the various institutions when personnel leave and new, untrained personnel take their place. There is also a lack of commitment on the MOE and REBs side to allocate resources, both human and budget for programs and innovations that have been started with BESO-II. This group of beneficiaries describes AED as responsive, helpful, and friendly and reported very favorable working relationships with AED management and staff. They are also highly pleased with the quality of AED’s work. They believe, however, that there have been too many delays with some of the activities and with financial and procurement actions. The beneficiaries acknowledge that some of the delays are due to constraints outside AED’s control, but wish AED could be more proactive in dealing with these constraints. Things improved as a result of AED’s re-organization, but project management could benefit from further refinement, particularly in regards to relations between AED’s technical and operations teams, which at this moment has not been conducive for an efficient flow of BESO-II resources. 11 From their perspective, AED is open to recommendations from the Technical Working Group (TWG) and takes their feedback well. AED is positive and agreeable. The only issue is that they do not always implement the recommendations with the same diligence and follow-through that the TWG would like. 2. Regional Education Bureaus The evaluation team interviewed representatives from the Regional Education Bureaus (REBs) in four regions and received their feedback from others at the March 2004 TWG meeting. The findings reveal a very low level of satisfaction among REBs. The satisfaction levels vary from region to region and among BESO-II components. The most significant variables related to REB satisfaction are: • The timing, amount, and quality of AED attention to the REB - Those regions where AED was quickest to hire field staff, collaborated with the REB on recruitment and selection of staff, those with the most staff, and those with the best qualified staff are happiest with AED’s assistance. • The amount and quality of communications with the REBs - REBs mentioned that AED often has direct communication with the TEIs, for example, and bypasses the REB. There is also some displeasure with the perceived imbalance of attention given to regions by AED. When REBs feel left out of communications and decisions it affects their commitment to, and ownership of, the activities. In the end, this will have an impact on sustainability since so much depends on REBs. • The timing of the project activities – The REBs where AED has progressed most with PMIS, MMIS, and planning activities are most pleased. The reverse is also true: those that have experienced the most delays and received the fewest inputs are the least satisfied. They have few issues with the quality of the activities. In addition to these issues, REBs remain unhappy about: • The timing of AED’s release of funds to TEIs. • Delays with project procurements—including vehicles, fax machines, photocopiers—and procurement of equipment that cannot be serviced locally because of unavailability of spare parts. One REB reported receiving used furniture, which it found unacceptable. • The lack of transparency regarding finance and procurement matters. • The lack of authority given to field offices to make decisions about project activities and to manage operation-related funds. • The lack of a focal point and team leadership in the field offices. They acknowledge that there are many challenges, but believe it is AED’s responsibility to deal with the external and internal constraints they face, rather than using them as excuses for delays. Nevertheless, they are hopeful that, once all the inputs are complete, BESO-II will have made a great contribution to the efficiency and effectiveness of their efforts to improve 12 the education system. They are anxious to enjoy the full benefits of BESO-II and look forward to completion of the information systems, educational planning, and studies. Specific comments from REBs in the study sample include: Intermediate Results Comments on Specific Activities IR1 Activities Highly satisfied with pre-service component and it has some of the most operational activities. Support to TEIs is very valuable, including Centers of Excellence. Assistance with distance education is very important and effective. In￾service is greatly boosted by the active learning methodology, which gives greater substance to the training. IR2 Activities REBs had no comments on these activities. IR4 Activities Greatly appreciate assistance with preparing annual and 3- year strategic plans and look forward to help with long-term planning. Need more training in the information system and AED needs to ensure compatibility of planning models with existing databases. MMIS is a very valuable management tool. PMIS is responsive to REB needs and should help greatly in managing personnel. More training needed on all information systems. Materials produced for WCB are good and REBs look forward to participating in the training. MERA Activities Looking forward to completion of the organizational structure study. Some of the challenges to full and effective implementation at the REB level include: frequent turnover in REB leadership; REB manpower shortage; shortage of BESO-II manpower (too few staff with too many responsibilities); insufficient communication and lack of responsiveness to REBs regarding personnel recruitment and project activities; and delays in BESO-II procurement of materials, computers, and vehicles. Looking toward the sustainability of BESO-supported activities, much relies on the commitment and participation of REBs and some additional support from BESO-II. REBs will need to assign the necessary manpower and budgets for: keeping data current for the information systems; maintaining computer hardware; retaining personnel who are capable of managing the software and troubleshooting problems; and providing the necessary technical and financial support to woredas, TEIs, and schools. The REBs are positive about the quality of AED’s work, commenting that it is very practical, useful, and customized. In most cases, there are smooth working relationships. They were also complementary about AED’s openness to the TWG’s recommendations and commitment to responding to them. Despite implementation and management problems, they would like to see BESO-II continue and believe that it is important to see through all the activities that have been started. 3. Teacher Education Institutes The evaluation team interviewed staff from Teacher Education Institutes (TEIs)—which includes Teacher Training Institutes (TTIs) and Teacher Training Colleges (TTCs)— 13 within the study sample regions. They are appreciative of the technical assistance, training, and financial support given by BESO-II and believe it has greatly strengthened the capacity and performance of their institutions. Of course, some of the TEIs were more satisfied than others. The focus of their dissatisfaction is on the funding process. First, some do not understand how the funding limits were determined, and believe that the amounts were too low (especially when compared to levels received under BESO-I) and appeared unfair because some TEIs received more than others. Second, TEIs are unhappy about the amount of time it took between initial discussions about their implementation plans and when they received the first transfer of funds. In some cases, TEIs had to borrow money from other sources until the BESO-II funds arrived. They have various understandings of why there were delays, which suggests that there was inadequate communication between TEIs and AED on this issue. However, it may also reflect the capacity issues within the TEIs. Third, a few feel constrained by what they understand to be a rigid budget, without much flexibility among budget line items. Lastly, they are displeased with the delays in receiving the final transfer of funds. While some acknowledge that it is due to their inability to complete the implementation of their plans (which is required before the final payment), others are confused and frustrated by the reporting requirements tied to the final payment. Despite delays in receiving funding, at this point all TEIs have received the bulk of their funds (some have received all their funds) and are in various stages of implementing their plans. The activities and procurements in their implementation plans, plus BESO-II technical support, are contributing achievement of IR1 by: • Strengthening use of active-learning methods • Increasing library resources • Strengthening Pedagogical Centers (PCs)/Instructional Resource Centers (IRCs) • Increasing the number of, and support for, female students and instructors • Strengthening Staff Development Units (SDUs) The TEIs report that they have been relatively successful in applying what they have learned through the various training programs. However, they recommend providing additional training to broaden and deepen the capacity of the TEIs. There appears to be fewer benefits from the equipment and materials provided through BESO-II due to lack of access or training. The evaluation team found that some equipment (computers, copiers, laminators, etc.) was not accessible because it was still in the shipping container, locked in storage rooms, or heavily protected. In other cases, the equipment was accessible, but not fully utilized because the staff was not sufficiently trained to use it or it needed repair. (Please note that the agreement between AED and the TEI requires that the institution “provide on-going budget or supplies, operation and maintenance of equipment…”) The TEIs also reported some promising signs that BESO-II will achieve its desired impact. Examples of the impact from the study sample include: 14 • Use of active-learning methods and continuous assessment methods – The TEIs surveyed are teaching active-learning methods. The Assela TTI, for example, has also begun standardizing the methodology by producing model lesson plans in science, math, and language using the principle of active learning. Several of the TEI have also conducted continuous assessment workshops and most are using continuous assessments in their classes. • Improved library resources – TEIs in the study sample have attended training in library management and some participated in a study tour. They have purchased such things as books, computers, copiers, laminator, circulation desks, card catalogs, fans, security grills for the window and doors. The librarian in Jijiga TTI was pleased to report that, as a result of BESO-II, he has improved: his ability to select books (now preparing request for more books that are relevant to TTI); the borrowing system; the cataloging system (starting to computerize); and student use of the library (he provides orientation to student group-leaders on how to use the library and then they orient their groups). • Strengthened media centers – The Assela TTI has filmed teachers from local schools who demonstrate the active-learning methodology (and some who do not) and will use the video for instructional purposes at the TTI. The Media center in Jijiga TTI has acquired videos from Uganda on activity learning and a video on the impact of early marriage on education. The SDU head attended a training program at EMA on how to produce radio programs. He is now developing radio scripts on environmental science for four grade levels (28 programs each). He also conducted training at the TTI on how to use the media to educate, especially in the areas of HIV/AIDS and the environment. • Upgraded computer centers – Most TEIs have purchased computer equipment and subscriptions to the Internet. Some of the centers are not fully utilized, but the computer center in Gondar is one that is accessible and properly utilized by instructors. Instructors commented on and appreciated the availability of computers, Internet, equipment, and the training they received. • Improved Instructional Resource Centers (IRCs) - While many TEIs are making efforts to improve their IRCs, the TTI in Assela is one that is making the greatest investment through the extra funds they are receiving as a Center of Excellence. The staff is very enthusiastic about this and was eager to provide a tour of the building that is being remodeled to accommodate the expanded IRC. It will serve as a resource for other TEIs and primary schools in the surrounding area. The IRC in Hossana was pleased to report on the training on teaching and learning using locally available materials (TALULAR), which provided many ideas for enhancing the IRC. The Hossana IRC has extended itself to the local schools by providing instruction support materials, training IRC coordinators, and sharing experiences. 15 • Increased support for female students and instructors – This area has shown strong progress in several TEIs. The TEIs are organizing girls’ clubs, offering tutorial classes, and counseling to female students. Very recently, the Hossana TTC organized a girls’ club that now has its own office. The club has organized a series of meetings with female students of the college, with about 35-65 students attending. It also collaborates on a tutorial program, which the evaluation team observed while on campus. They have also: lobbied to extend the library hours to accommodate the needs of the girls, requested a session with the college management to address the female students and answer their questions, and started planning a workshop so that role models can share their experiences. The Awassa TTI has recently formed a girls’ focal unit, which has organized a 1-day workshop on gender stereotypes and HIV/AIDS. They plan to conduct research on the disparity in the participation of female students in different departments, and organize tutorial sessions for female students. The evaluation team witnessed a meeting of the girls’ club at Assela TTI, which has organized tutorials twice a week for female students who are lagging academically. It has been so successful that the male students are now requesting tutorials. Efforts at the TEIs to increase the number of, and support to, female instructors is less impressive. • Improved capacity of Staff Development Units (SDUs) – Several TEIs are showing improvements in their SDUs. In Gondar, for example, the SDU has developed an action plan and has also organized a 1-day workshop to get feedback on the curriculum materials they developed for on grades 7-8. The Hossana TTC’s SDU has organized workshops on: curricular materials (syllabus and module) preparation, esthetic and physical education, professional science one, professional science four, and female assertiveness. They also started offering tutorial sessions for female trainees and computer training for instructors and staff. The SDU at Awassa TTC has developed materials for the TESO program, organized two workshops on the TESO program, and organized an educational tour for staff members to share experience with Adama TTC on writing modules. • Strengthened relationships with primary schools – TEIs are also improving their relationships with local primary schools and will expand the relationship as they implement the practicum requirement. In Gondar, the IRC serves as a resource center for the nearby primary schools. Teachers go to the center to share experiences or get teaching materials. The Adama TTC provided training on action research for local primary teachers. In Hossana, the TTC has strengthened their linkage with the nearby cluster schools by: inviting them to attend a workshop on active learning and continuous assessment, and preparing instructional support materials and teaching aids. Transportation is a major constraint in strengthening the relationship between TEIs and primary schools, and will become a more serious constraint with full implementation of the practicum requirement. 16 Without a doubt, BESO-II has contributed to the breadth and depth of the TEIs’ capacities. The TEIs also believe that BESO-II has had helped improve students’ performance and decrease the dropout rate, particularly for female students. At this point, it is not clear how sustainable these efforts are. Clearly, there is a strong commitment in most TEIs to support the objectives of BESO and they can see how these activities will improve quality and equity. However, the heavy workloads of instructors and the frequent turnover of management (at various levels) will affect the sustainability of these capacity building efforts. 4. Cluster Schools The evaluation team visited cluster schools in four regions. The sample included (a) cluster resource centers (CRC) and satellite schools, (b) urban and rural schools, and (c) impact and general clusters. The team interviewed administrators and teachers, toured the campuses, and briefly observed classrooms and other activities. The amount of BESO-II inputs to any given school is small. One reason is AED had to work with a large number of clusters to reach the target number of teachers. Another reason is that AED was assigned one set of cluster schools for the first year and another set for the second year, limiting the time and effort that could go to each school. Nevertheless, beneficiaries within the CRCs are generally pleased with the design of BESO-II and AED’s implementation of it. It is providing them with much needed assistance that they would not likely get otherwise. CRC beneficiaries are also quite pleased with the quality of BESO-II training and assistance. Their only complaint is the amount of time they have been waiting for equipment and materials. Some of them have been waiting for months and they do not know why. (The reasons for the delays are addressed below, in the section on Lessons Learned.) Where equipment and materials have arrived, there are occasional issues with utilization and access. Some of the equipment is not fully utilized because of budget constraints within the schools. Duplicating machines, for example, require stencils, toner, and paper and schools often do not have a budget for these, so the machines are often idle. Access is also a problem. Due to government property management regulations, those who sign for receipt of the materials and equipment are held responsible. This places pressure on these individuals to secure and protect property, which often results in extremely controlled, and therefore limited, access. Despite the relatively small amount of inputs, there was very impressive impact at the school level. It is rare to find much impact after only a few days of training, but there are positive signs that the Integrated School Leadership and In-service Teacher Development training of trainers (TOT) activity has had impact. The impact was strongest, not surprisingly, among the “impact” cluster schools, especially the Cluster Resource Center (CRC). The satellite schools in the impact clusters have experienced less impact, and the general (non-impact) clusters schools have experienced the least impact. 17 Schools reported the following changes resulting from the training sessions and materials they have received from BESO-II: • Use of the active learning methodology – Teachers have increased their use of active-learning/child-centered methodologies. This is particularly impressive given the large size of the classes and the risk of losing control when using these methods. The evaluation team witnessed a small classroom with several dozen students sitting in groups at tables working together on math. The class was well managed and participation was high. Under such circumstances, this kind of control is only possible with strong classroom management skills, which the teachers have also learned through the BESO-II training. In addition, teacher￾student relationships have improved. Instead of fear and suspicion, trust has developed. Students are more likely to ask questions and they are also receiving higher grades on their examinations. Teachers are enjoying their successes and are requesting more training on methodology. • Improved school leadership – Teachers report that head teachers have improved their supervisory skills as a result of the training. Head teachers and facilitators are providing better support to teachers in the use of active learning methodologies and continuous assessment. (Note that the TWG believes there is a need to focus more on school leadership.) • Cluster collaboration – The training modules on cluster management have also been effective. At a rural impact center, the evaluation team was greeted by the cluster directors committee, which is composed of head teachers from each school in the cluster. They meet regularly to discuss the needs of their schools and opportunities for collaboration. Teachers are also collaborating more with teachers in their school and other schools in the cluster. Several schools reported that the teachers in the cluster meet monthly for a study group. They appreciate the structured forum that provides an opportunity to share experience and work together on ways to address common problems. Teachers also collaborate with their colleagues on developing uniform lesson plans and examinations, as well as action plans. The collaboration has resulted in increased efficiency and effectiveness. • Support to female students - Through the use of continuous assessment strategies, teachers and administrators report that they are better able to monitor the performance of students. This has been particularly helpful in responding to the needs of female students. Most cluster centers in the study now offer tutorial classes, not just for girls who are struggling academically, but for all girls. One of the rural schools in this study has organized a girls advisory committee with female representatives from the student body, the staff, and the community. They discuss and seek solutions for the problems that female students face. Other schools have also organized girls’ clubs. One of the main objectives of the clubs 18 is to work with the community to help increase awareness of the importance of girls’ education. The evaluation team observed some multiplier training from a recent TOT on gender issues. Female teachers from the satellite schools gathered at the CRC for a day of training. It was a highly participatory and lively training session. The team also witnessed a class that was participating in an interactive radio instruction (IRI) program. The team was unintentionally distracting so it was difficult to be “unobtrusive observers” and see normal activity. None of the schools in the sample were in the pilot-test groups for the instructional kits. However, teachers’ feedback from AED’s pilot testing indicates a high level of satisfaction and interest in the kits. School leaders believe that these efforts have had a positive impact on female enrollment, as well as the repetition and dropout rates for all students. AED will provide objective and verifiable data on this in the next few months. The impact would be even greater if the constraints could be effectively addressed. One of the main constraints is the distance between schools in the cluster. This makes it difficult for teachers in the satellite schools to take advantage of the materials and equipment in the CRCs. Transportation problems also make it less likely that teachers will participate in cluster study groups or attend training at the CRC. And because no funds are available for transportation or per diem, there is no external incentive for teachers to travel for training, study groups, or using the CRC. The CRCs also face constraints in conducting the multiplier training for the satellite schools. Most lack a room big enough for conducting training for large groups of teachers. They also lack the resources to duplicate training materials. Schools might also be able to increase girls’ enrollment if they had the resources to provide adequate facilities for girls and fences that would better ensure their safety. Sustainability of these efforts will largely depend on near-term follow-up support from BESO-II, resources and technical support from woredas and REBs, and modifications to the MOE’s in-service policies to require teachers to participate in in-service and provide them with the necessary incentives and resources to do so. C. Relationship to Mission Strategic Plan The purpose of this section is to “establish to what degree the BESO-II Results Framework has been supported by the AED/BESO-II project.” A complete treatment of this topic requires addressing four basic issues: (1) the logical relationship between the BESO-II intermediate results (IRs) and the basic education strategic objective (SO); (2) the relationship of project components to the IRs; (3) whether AED is successfully implementing each of the project components; and (4) the extent to which successful implementation and impact will be captured through reporting on the contract’s target 19 outputs and/or AED’s performance monitoring plan (PMP). Each of these issues is addressed below. 1. Relationship of IRs to the SO According to the Integrated Strategic Plan (ISP) for FY 2001-2006, USAID/Ethiopia’s basic education SO is: quality and equity in primary education system enhanced. The SO is supported by four IRs and several sub-IRs: IRI: Quality of professional education personnel enhanced 1.1 Use of child-centered/active learning methods in pre-service teacher training institutions enhanced 1.2 Child-centered/active learning methods in in-service teacher training reinforced 1.3 Personal and professional support systems for women teachers strengthened IR2: Teacher-learner support systems strengthened 2.1 Teachers using relevant supplementary media and materials to support active learning 2.2 Socially relevant topics (e.g., HIV/AIDS, civics, environment) integrated into the curriculum (IR3: not related to the AED contract) IR4: Education Management Strengthened 4.1 More efficient systems developed by regional bureaus for i) personnel management, and ii) distribution and logistics of educational materials 4.2 Better utilization of Educational Management Information System (EMIS) at all levels 4.3 Improved planning, monitoring and evaluation of student assessment capacity at all levels 4.4 Ministry of Finance reforms implemented by Regional Offices of Finance and Regional Education Bureaus (dropped in contract modification no. 2) There is a clear and logical relationship between each sub-IR and the corresponding IR. Likewise, there is a strong relationship between the IRs and the SO. Thus, if each set of sub-IRs is achieved, then the related IR will be achieved, and if the IRs are all achieved, USAID will have contributed to enhancement of Ethiopia’s quality and equity in Ethiopia’s primary education system. The logic of the results framework is sound. 2. Relationship of Project Components to IRs When developing the ISP, it was USAID’s intention to “build on the lessons learned in BESO-I to achieve more focused impact in the future BESO-II plan period.”* Clearly, the ISP and the BESO-II project were designed to be compatible, and thus we find a good * USAID/Ethiopia ISP 2001-2006, p. 49. 20 match between the IRs and the project components, both from the original contract and contract modification, as outlined below. Intermediate Result Related Project Components IR1: Quality of professional education personnel enhanced -Pre-service teacher training -In-service teacher training -School leadership training -Women teacher support systems -Distance education for second cycle teachers -English language improvement program -Support to Teacher Education System Overhaul (TESO) program IR2: Teacher-learner support systems strengthened -Pre-service teacher training -In-service teacher training -School leadership training -Supplementary media development & training -Socially relevant curriculum integration -Teacher training in IRI program -Review of curriculum -Civic education materials and training IR4: Education planning and management strengthened -School leadership training -Personnel & materials management, planning, monitoring, evaluation, & information systems -Woreda capacity building There are sufficient project components to support achievement of the IRs. Therefore, if the project components are successfully implemented, the IRs will be achieved. 3. AED’s Implementation of Project Components The next important issue is the extent to which AED is making progress toward completion of the target outputs and achieving the desired impact, as expressed in the IRs and sub-IRs. AED’s implementation of project activities got off to a slow start, but progress is accelerating at a steady pace and AED expects to complete most of the outputs by the end of the base period. Completion of targets is treated more thoroughly in an earlier section of the report (Achievement of Targets). The evaluation team explored the issue of impact through interviews with beneficiaries at all levels of the MOE, which is reported more comprehensively in the previous section on impact. While all of our findings are anecdotal and based on a small sample, they are impressive and promising. Especially striking was the breadth and depth of impact, reported by head teachers, at the school and classroom levels that resulted from a single TOT. Schools have started girls’ clubs and have provided tutorial sessions for girls. Their ability to respond to the academic needs of the girls (and boys) has been enhanced through their use of continuous assessment strategies. Teachers have appreciated the opportunity to share experiences with other teachers within the cluster through study groups, initiated as a result of the BESO-II TOT. As teachers have applied the active learning methodology, they have witnessed improved performance of the students, as well as decreased dropout and repetition rates. Schools attribute the improvements to 21 BESO-II. Similar impact can be found at the TEIs. The impact will likely increase when AED completes delivery of equipment and materials to the “impact clusters” and provides the necessary follow-up support. In the spring, AED will collect quantitative data—based on indicators in the PMP—that will better describe the impact of BESO-II activities. If AED completes the target outputs and achieves the desired impact, one can safely assume that BESO-II implementation supports the Result Framework. 4. Measuring Outputs and Impact The final issue is whether USAID and AED can adequately document their success through reporting on IRs, contract output targets, and PMP indicators. To address this, it is helpful to study the relationships between the ISP, AED contract, and PMP, which are outlined on the table below. As is evidenced by all the blank cells on the table, there are many gaps in the linkages among the Results Framework (first two columns) and the AED contract (third and fourth columns) and the PMP (last column). These sorts of discrepancies should not cause great alarm if one accepts the assumptions that (a) indicators and activities evolve over time to be responsive to changing needs of the host government and realities of implementation, and (b) when USAID approved the PMP in late 2003, they endorsed it as a current description of what USAID expects AED to implement and measure. Likewise, there are updates to the target outputs that were mutually agreed to after extensive discussions between USAID and AED on this issue and in July 2003. AED documented the suggested modifications in a letter to USAID. AED and the education office of USAID expected that these changes would be incorporated in the second contract modification, but this did not happen due to constraints with USAID’s contracts office. (See Annex E for the recommended modifications to the output targets.) USAID should also consider these issues in their review of the recently approved new ISP. The IRs and indicators should be closely tied to the realities of project implementation and performance monitoring so that: (a) the right achievements are being monitored, allowing USAID and the contractor to get appropriate credit for the achievements in the project; and (b) USAID and the contractor are not collecting performance data on two different sets of indicators. 23 Relationship of Strategic Framework to BESO Contract SO - Quality and equity in primary education system enhanced. Indicators: (1) Gross enrollment rates, grades 1-4 nationally, by gender; (2) Reduced dropout rate, grades 1-4 nationally, by gender. Intermediate Results (Strategic Plan) Indicators (Strategic Plan) Project Components/Sub￾IRs (Contract) Target Outputs (original contract; does not include outputs added through contract modification) Indicators (AED’s PMP) Ind. 1.1 % active learning methodology used in TTC/TTIs nationally Pre-service training (1.1) IR1.1: % TEI teacher trainers participating in at least 2 AED￾sponsored active learning programs/year & are applying skills in their courses Ind. 1.2: % TTI & TTC graduates scoring above 2.75 GPA (out of 4), by gender # TEIs that increased percentage of women graduates with GPA 2.75 and above… # TEIs that have: improved quality & quantity of library materials; improved the resources & operation of an instructional resource center; provided adequate funds in annual budget to maintain library and/or center # TEIs that demonstrate adequate commitment of human, materials, or financial resources to the activities Ind. 1.3 % teachers in selected zones who use child-centered/active learning methodology regularly In-service training (1.2) IR1: % teaching time using active learning methods in AED cluster schools IR1.2: % primary school teachers participating in at least 2 AED-sponsored active￾learning workshops/year & are using skills in classroom IR1: Quality of professional education personnel enhanced # multi-session self-instructional kits for in-service (developed, tested, produced) IR2.1: % primary teachers who integrated active learning 24 Intermediate Results (Strategic Plan) Indicators (Strategic Plan) Project Components/Sub￾IRs (Contract) Target Outputs (original contract; does not include outputs added through contract modification) Indicators (AED’s PMP) # primary teachers completing at least one self￾instructional kit methods as a result of using self-instructional kits Women teacher support systems (1.3) -Linkages established & nurtured between regional women’s associations & women teachers… -Instructional and/or motivational short courses, & training modules targeted at women teachers… IR1.3: (1) % female TEI students who completed at least one course/module to enhance professional competence, (2) % TEI female teachers who completed at least 1 course/module developed to enhance… (3) personal & professional support system for women teachers enhanced School leadership training (1.2, 1.3) # school headmasters, regional, zonal, & woreda education support personnel trained in constructive teacher support methods IR1.3: % impact cluster school directors organizing at least training sessions that focus on strengthening leadership in schools Establishment of a viable teacher strengthening system that received continued support from pre￾service training system Ind. 2.1: % schools in selected zones that are using interactive radio in an active-learning mode in grades 1-4 Supplementary media development and training (2.1) # primary schools that are using IRI in active learning mode in grades 1-4 IR2: improved achievement in math & English in schools using IRI # clusters in priority regions that are operational…. IR2: Teacher-learner support systems strengthened Socially relevant curriculum integration (2.2) Existing curriculum reviewed & revised based on school feedback on key socially relevant topics IR2: % AED cluster schools regularly using active learning modules in HIV/AIDS & civics; IR2.2: % AED cluster school teachers who integrated modules on HIV/AIDS, civic ed, & environmental ed 25 Intermediate Results (Strategic Plan) Indicators (Strategic Plan) Project Components/Sub￾IRs (Contract) Target Outputs (original contract; does not include outputs added through contract modification) Indicators (AED’s PMP) Curriculum on identified socially relevant topics integrated & prototype materials/media produced Ind. 4.1 # of regions utilizing improved personnel management systems Personnel management information system (4.1) #regions that have developed & are maintaining computerized personnel systems… IR4.1: # regions successfully implementing improved computerized PMIS Ind. 4.2 # of regions utilizing new Ministry of Finance reformed budgeting and accounting systems Capacity building of the MOE and REBs to manage direct financial support (4.4) IR4.4: # REBs assessed and appropriate follow-up action implemented Materials management information system (4.1) #regions that have & are using improved instructional materials procurement & distribution systems… Educational management information system (4.2) #regions utilizing education information more efficiently for planning & decision making IR4.2: # REBs using EMIS applications Planning, monitoring, evaluation, and student assessment at all levels (4.3) #regions that are undertaking improved learning assessment & other monitoring of educational quality at the classroom and school levels #regions integrating monitoring & evaluation into their planning… #regions successfully integrating planning of activities and programs with budget or educational finance IR4:# REBs using improved systems for policy/planning, mngt, & monitoring & evaluation IR4.3: # REBs with improved capacity for developing long￾range plans and annual education plans IR4: Systems for managing personnel, instructional materials, budgeting, and monitoring and evaluation strengthened and used # of policy studies disseminated 26 D. Lessons Learned This section of the report explores lessons learned in three key areas: design and resources, project implementation, and AED management. Within each of these areas, we cover the most critical issues and constraints, the impact of the constraints, and what has been done to address the challenges. 1. Design and Resources a) Support of USAID and GFDRE’s Objectives As discussed earlier in the report, the project design supports the achievement of USAID’s strategic objectives and intermediate results in basic education. Furthermore, USAID’s strategic plan and BESO-II design are fully supportive of the GFDRE’s Education Sector Development Program (ESDP), which was established in 1997. The strategy and design were a collaborative effort between USAID and the BESO Technical Working Group (TWG) and other key partners in Ethiopia. Our findings reveal strong support at all levels of the MOE for the design of BESO-II. There is nothing in the design that is not supported by the MOE. There are, of course, more things that could be done to increase the likelihood of achieving USAID and MOE objectives, but these would require additional resources and an expanded scope of work. b) Scope of Work BESO-I achievements are considered significant by USAID, MOE, and AED.* In fact, when designing the ISP and BESO-II, USAID built on the “experience and lessons learned in BESO-I.” However, the design of BESO-II may have gone too far in expanding USAID’s efforts in basic education in terms of geographic reach and allowed insufficient time to achieve the objectives. Geographic Reach – In BESO-I, AED focused their efforts in two regions (Tigray and SNNP) and, to a lesser extent, at the national level. In that situation, AED could concentrate resources and provide a greater depth of technical assistance, thus facilitating achievement of impact. However, in BESO-II, AED has been tasked with working in all nine regions and two city administrations for most activities, and with the MOE on central and national activities. This has made the project a bit unwieldy. The wide geographic reach has been challenging in terms of: • Planning time - It took considerable time for AED to work with each of the departments at the central MOE, as well as each of the 11 Regional Education Bureaus (REB) and 19 teacher education institutes (TEIs). Customization of plans for each of the beneficiaries took many meetings and follow-up discussions. In the case of the nine new regions, this also required time to orient the new partners to BESO-II and to build interest and commitment. AED’s delay in staffing up compounded this challenge. * USAID/Ethiopia ISP, 2001-2006. For more information, the ISP refers to the BESO Bulletin edited by Aberra Makonnen of USAID/Ethiopia, BESO Revisted by Cameron S. Bonner, and the BESO-I mid-term evaluation. 27 • Travel time – Because of the great distances between regions, and partners within each region, much time has been spent traveling around the country. Ethiopia is a large country and a lot of time can be consumed by travel. Again, this challenge may have been mitigated if AED had staffed their field offices earlier. • Languages – Ethiopia is a country of many languages, and so the greater the geographic spread of the project, the more languages there are to deal with. This requires a lot of time and effort to translate and produce materials. Initially, AED planned to produce materials in two languages, but the instructional kits, for example, are being produced in five languages and the Woreda Capacity Building materials are being produced in three languages. Although it was not in the original scope of work, AED thought it was important to translate the Personnel Management Information System (PMIS) into three major languages. An additional complication in translating the PMIS is that there were software limitations at the initial stages. • Impact – With resources spread so thinly across the entire country, it is challenging to achieve and measure impact in any given region or institution. This is especially true with the huge number of clusters assigned to AED (more than 700 for the two base years combined). This means that even when activities are successfully implemented, it is not easy to find measurable impact. To concentrate resources a little more, BESO-II is focusing its efforts on four “focus regions,” three Centers of Excellence at TEIs, and 40 impact clusters. • Short Implementation Period – Two years is a relatively short period for most development projects. However, it seemed that BESO-II was, in many ways, a continuation of BESO-I and therefore considered part of a 9-year development effort, rather than just a two-year effort. There was also a related assumption on USAID’s part that there would be a smooth transition from BESO-I to BESO-II. That was a logical assumption, but AED did not sufficiently plan for the transition and did not hire staff soon enough. They have had to work hard to make up for the time they lost in the first year and they will have to continue a fast pace to achieve the targets. AED expects that they will almost completely achieve the targets. So, in the end, it may appear that two years was sufficient—but only sufficient to deliver all the inputs. Two years is not enough time to implement all the project activities in all the regions and institutions AND then provide the required follow￾up to ensure impact and work toward sustainability. c) Resources For BESO-I, with a focus on two regions, there was over $80 million in USAID funding ($49.3m in project assistance and $31m in non-project assistance). USAID estimated that BESO-II activities, with expanded geographic coverage and additional activities, would require funding of only $30 million. The funding-to-coverage ratio was further reduced when AED proposed a bid of around $25 million. This was later increased to $30 million through a contract modification. The increased scope of work, without an increase in budget, essentially required that AED “achieve more with less.” Despite this paradox, the expectations of USAID and the beneficiaries have remained high. 28 AED is having difficultly meeting the challenge of using all the contract funds and currently there is a large pipeline. As of December 2003, there was approximately $13 million in the pipeline, which means that roughly 57% of the contract amount has been expended over 70% of the contract period. Of the contract funds, overall program expenditures are only slightly behind where they should be at this point. However, some program areas are spending at a faster rate than others, as indicated in the table below: Intermediate Result Percentage of budget expended as of 12/03 IR1: Teacher Enhancement 66% IR2: Learning Materials 93% IR4: Planning & Management 45% Total 65% A closer examination of expenditures reveals which project components are lagging the most: Project Component Percentage of budget expended as of 12/03 Pre-service Teacher Training 73% In-service Teacher Training/ School Leadership 64% Supplementary Media (IR) 110% Socially Relevant Curriculum 28% Women Teachers’ Support System 29% Planning & Management 51% MERA 7% Overall 65% To use all the remaining contract funds by the end of the base period, AED would need to double its monthly burn rate to around $1 million. The burn rate is accelerating in the final months of the base period, but it is unlikely that AED will be able to use all the available funds. The next concern is whether there is sufficient funding to implement Phase II with the current budget. AED’s budget for the 2-year base period is $19 million (including the $4.8m added through a contract modification). This is an average of $9.5 million per year. However, the budget for the three option years is $11.3 million, which averages out to $3.8 million per year. The lower budget assumes Ethiopianization of all positions and a gradual phase down. To ensure there is adequate funding, USAID must carefully consider their expectations for Phase II and the corresponding staffing requirements, both in terms of type of staff and numbers of staff. 29 2. Project Implementation Many of the important lessons learned so far in Phase I of BESO-II relate to project implementation and fall into the following categories: (a) deliverables, (b) funding TEIs, (c) project procurements, and (d) external constraints. a) Deliverables The BESO-II design is ambitious and the timeframe is short, making it difficult, even under the best of circumstances, to achieve everything on time. The problem with the tight schedule is compounded by the numerous external constraints (discussed later) and the slow project start-up. In mid-2003, AED and USAID agreed to make some adjustments (which were to have been documented in the second contract modification) based on a mutual understanding of what was realistic under current conditions. However, two key deadlines for deliverables were missed, much to USAID’s dismay: the Implementation Plan and the Performance Monitoring Plan. Implementation Plan – The Implementation Plan (IP) is not listed in the contract as a required report. It is safe to assume, though, that the requirement for an IP replaces the “Annual Plan.” However, it is not logical to use the due date for the Annual Plan as the due date for the IP. (The contract states that the Annual Plan is due “by July 7 for the next contract/Ethiopian fiscal year.” On the same page of the contact, it states that the plan is due “each June.”) So, it is unclear when the IP was required. Regardless of a specific deadline, it took far longer to submit a “final” IP than USAID thought reasonable. AED presented its first draft of the IP in November 2002, three months after the contract was signed. For some projects, three months would be more than enough time to prepare a draft implementation plan, but given the complexities of the BESO-II environment, it is impressive than it was done in the first three months. (As discussed elsewhere in the report, it may have been done sooner had AED been able to recruit, hire, and train the qualified staff immediately after the contract was signed.) AED faced the following challenges in preparing the IP: • Budget expectations – The November 2002 draft IP was developed by pulling together all the various requests by beneficiaries. It immediately became obvious that the beneficiaries had assumed a higher level of resources (based on BESO-I levels per region) and had submitted unrealistic plans. • Number of beneficiaries – The second draft of the IP that was submitted in February 2003 was based on extensive discussions with a large number of beneficiaries. It took considerable time for AED to work with each of the departments at the central MOE, as well as each of the 11 Regional Education Bureaus (REB) and 19 teacher education institutes (TEIs) to develop customized implementation plans. Because there were so many beneficiaries new to BESO-II, it took extra time to orient them and build commitment. The “final” version of the IP was present in July 2003 to the TWG and was accepted. 30 Performance Monitoring Plan – According to the contract, the Performance Monitoring Plan (PMP) was due 60 days after the effective date of the contract, which would have been in October 2002. The final version was submitted to USAID on December 1, 2003, 14 months after the due date. While it is perhaps unrealistic to have expected a final PMP to have been submitted in October 2002—just after project start-up and during implementation planning—an additional 14 months is excessive. USAID’s expectations were based on the fact that an 11-month “bridge” from December 31, 2001 to November 15, 2002 had been provided to AED so that it had a continuous on-the￾ground presence. Given this unusual support, it was expected that AED would have minimal “start-up” requirements. Given AED’s experience in-country, both it and USAID assumed that implementation planning would proceed quickly and smoothly. In August 2002, USAID organized a workshop with BESO-II partners on performance monitoring. It was important that AED provide USAID with the needed information soon after the workshop so that USAID could prepare its own PMP. Despite several reminders by USAID, AED did not present a draft PMP for eight months. AED then hired a consultant, who did little more than copy USAID’s PMP. This was, of course, unacceptable. Also disappointing to USAID was that it appeared that AED was not giving due attention to the PMP, despite the fact that it was a top priority for USAID. In May 2003, AED submitted another draft PMP. This draft was produced through involvement of key field staff. USAID considered this an improved version and gave AED some feedback, which they incorporated. At this point, they still lacked baseline data but it was late in the school year and so AED had to wait until the fall to collect baseline data. USAID was not completely satisfied with the baseline collected, but accepted the PMP, which was submitted as final in December 2003. b) Funding TEIs The TEIs are generally satisfied with BESO-II, like the design, and appreciate the assistance. The planning process took quite awhile and then the plans had to be approved by the TEI and USAID. However, the main issue TEIs have is with the delays in signing their subcontracts and releasing the funds. Because of the delays, some TEIs have had to borrow money from other sources until the BESO-II funds arrived. There are a variety of explanations among the TEIs and the AED field staff as to the reasons for the delays—none of them entirely accurate and none of them show any understanding of the constraints AED has faced in disbursing funds to the TEIs. Some believe that it is because of an overly bureaucratic financial system within AED’s Addis office, some think it is because AED’s headquarters is slow to wire funds, some were told the “paperwork was lost.” The fact that there are such different understandings indicates poor communication or a lack of transparency, and unfortunately has resulted in a loss of trust. The central AED office in Addis gave some explanations for the delays: • As discussed earlier, it took a long time to complete the planning process for all 19 TEIs. 31 • It took USAID nearly ten months to clarify what mechanism could be used for disbursing funds to the TEIs. As in BESO-I, AED was originally given authority to give grants to the TEIs. However, USAID’s contracting office later determined that this was a mistake, rescinded the sub-granting authority, and then instructed AED enter into subcontract agreements with the TEIs. • While waiting for USAID’s decision, AED worked with all 19 TEIs to price their plans in a manner that would be conducive to fixed-price subcontracting, which apparently was not an easy process. Because AED was working on the pricing while waiting for USAID’s decision, they were ready to quickly sign the subcontracts and release the initial payment to the TEIs. • There were some minor delays that AED faced internally in getting wire transfers from the US because they were not prepared for the rapid spike in cash requirements. Final payments for the original subcontracts have also taken longer than desirable because of several factors. The subcontract requires that the TEIs report on workshops, study tours, and procurements as the condition for final payment. This apparently is a big shift for some TEIs and they have difficulty with the concept of fixed-price contracting and still put more emphasis on producing receipts (which are not required) rather than reporting on deliverables. This has been a struggle. It was also been a challenge for some TEIs to complete all their activities in a timely way because of competing demands on their time. The teachers already have heavy workloads, many have been pulled away for TESO activities, and others are busy with the conversion from TTIs to TTCs. The frequent turnover in TEI management has also made it difficult to quickly implement their plans. c) Project Procurements Cluster schools are also quite pleased with the design of BESO-II and have benefited from the project inputs, especially the training of trainers (TOT). The main issue the schools have is that AED has been slow to deliver the equipment and materials. Some say that they have been waiting for months. At this point, the status of procurement is, according to AED: 60% procured and delivered 21% procured but not yet delivered 12% procurement in process 7% procurement request pending AED acknowledges that there have been delays in procurement, but is constrained by several factors. • The timeliness and quality of the requests from the schools has been poor and the specifications are insufficient to procure. Unfortunately, AED’s field staff is not familiar enough with the equipment and materials in the market and therefore not in a good position to improve the specifications. Therefore, getting the specifications “procurement ready” has taken longer than expected. 32 • Due to limitations with electricity, many schools need manual equipment (typewriters and duplicators), which are not in the local market. This means that the equipment must be imported, which lengthens the waiting time. • In early 2003, AED also had to deal with a new Value Added Tax (VAT) requirement that affected local procurements. USAID informed AED that VAT was an unallowable expense and provided a letter to be given to vendors certifying that AED is exempt from VAT. The letter has not be accepted by vendors, so AED is forced with the choice of either paying the VAT expenses on local purchases, or procuring from the United States. The option of paying VAT means that, at least for the meantime, AED must absorb the costs. The importing option results in extra time due to shipping and customs clearance processes. d) External Constraints Within the Ministry of Education, there appears to be a strong commitment to BESO-II at all levels. Officials from the central level to the school level express great interest in BESO-II activities and are supportive of the objectives of the project. Nevertheless, there are some significant constraints within the MOE that impact on AED’s implementation of BESO-II activities. These constraints include: • Decentralization – The devolution of authority for primary education is a continuing process, which provides on-going challenges to implementing all aspects of BESO-II. Because REBs have been busy with decentralization-related issues, it has been difficult for AED to get their attention. Through this process, assignments of education officers and school directors have changed and these changes mean that AED has new people to work with. The clusters have also undergone changes, with several revised lists provided to AED. • New Curriculum – During the development period, TEIs were busy with TESO activities and it was difficult to get them to focus on BESO-II activities. With the curriculum not ready until September 2003, some related BESO-II activities were delayed. TESO’s delays in preparing teaching materials in turn caused delays with BESO-II’s reproduction of materials. Similarly, a recent change in the English curriculum by ICDR, for example, required AED and EMA to reorganize and edit the scripts and audio programs (over 100 programs for each grade) for the Interactive Radio Instruction (IRI) lessons for grades 1 and 2. • Language – The GFDRE’s commitment to the rights of student to learn in their own languages adds many challenges to producing materials and developing information system in a country where so many languages are used. This adds considerable effort, time, and costs to many project activities. • Understaffing at all levels of the Ministry – The chronic understaffing means that MOE personnel are stretched so thin that they cannot always devote sufficient time and attention to BESO-II activities. This causes some delays in planning and implementation. For example, there are not enough people assigned at the REBs to serve as inventory managers, so it is a big challenge to collect and maintain data in the Materials Management Inventory System (MMIS). • Frequent staff turnover - The MOE has a practice of frequently transferring personnel at the central and regional levels. Each time BESO-II has a new person to work with, time is lost building new relationships and creating ownership and 33 orienting the new person to BESO-II. It also means that BESO-II has lost a trained person and must either work with an untrained person or provide additional training; either situation affects implementation. This situation impacts all BESO-II activities. • REB’s clustering – USAID and AED have limited influence over the schools that are assigned to each cluster and the clusters that are designated as BESO-II clusters. This means that: (a) AED must work with clusters that are far apart, increasing the amount of travel time; (b) the clusters also often have schools that are far from each other, especially in the rural areas, thus making it difficult to have the desired multiplier effect on the in-service training; and (c) there is not a perfect correspondence between the clusters assigned to AED/BESO and those assigned to the BESO/CGPP partners, limiting the amount of collaboration among BESO partners. Likewise, AED is at the mercy of the REB’s timing in assigning cluster schools. Therefore, AED had to wait for up to three months in some regions for assignment of their cluster schools for Year 2 in-service activities. • Lack of incentives for in-service training – It has been difficult to achieve the desired targets for in-service training because of the lack of funds for per diem and transportation for teachers to attend in-service training. While BESO-II provides funds for teachers attending the TOT sessions, it does not fund the multiplier training to be delivered to the satellite-school teachers. AED has tried to address this constraint by providing certificates to help motivate teachers, is working with the MOE to give in-service credits to teachers, and provide TOT to all schools—including satellite schools—so that they can provide the multiplier training without requiring teachers to travel for multiplier training. • Late start of TEI academic year – This academic year, some TEIs started as late as November, due in part to the new curriculum. This meant that AED had to reschedule pre-service and in-service activities. There are also several constraints outside of the MOE’s control, but which also affect project implementation: • Drought – Ethiopia suffers from severe drought. The most serious periods have resulted in school closures in some areas. • Political unrest – There has also been significant fighting that has created a difficult working environment in Gambella. For the past several months, AED has had to curtail some project activities. • USAID – AED has been constrained by a number of factors related to USAID: the mistake made in authorizing AED to issue grants to TEIs resulting in a 10- month delay in funding their implementation plans, challenges in getting approvals for non-expendable property, delays in getting the source-origin waiver for project vehicles, and issues associated with the Valued Added Tax (VAT) requirements. 34 3. AED Management a) Personnel Management AED has managed to assemble a highly qualified and committed team of professional and support staff. With few exceptions, the partners and beneficiaries have been satisfied with the caliber of AED staff. There are, however, a few personnel related issues that reflect on AED’s management of personnel. Recruitment - USAID has expressed concern about AED’s delay in recruiting Ethiopian staff, which took between eight and nine months to complete. The delays resulted in a slow start-up, especially in developing the Implementation Plan. One AED staff member assumed responsibility for the bulk of the planning, which was a labor￾intensive effort and may have been completed faster with more staff. USAID believes recruitment should have been a much quicker process given AED’s continued presence in Ethiopia between BESO-I and BESO-II and USAID’s efforts to get AED to focus on this issue. USAID notes that preparations for local recruitment could have been done during the bridge period, with final hiring being done upon signature of the award. Yet, even for the local positions that had been identified by AED, it did not recruit until over nine months into the project. AED realizes that earlier attention to hiring would have changed the nature of the start-up period. It would have meant that implementation planning responsibilities could have been distributed over a greater number of people, reaching more regions and institutions in a shorter period of time. However, the evaluation team could not determine whether the planning period would have been completed any sooner if local staff were on board earlier because of so many other variables in the planning process. One important personnel-specific constraint is the unusually significant role of the MOE in reviewing job descriptions, recruitment advertisements, and short-lists of applications, as well as consultations with MOE staff at several levels during the process. This clearly adds time to recruitment and hiring. As of six months ago, AED had filled most available positions. USAID has acknowledged that the implementation pace has improved as a result of having a full complement of staff on board, but is concerned that it will be difficult to make up for early delays. Field staff - There is great deal of dissatisfaction with issues related to field staff: the numbers, delays in hiring, collaboration with the REB in the recruitment and hiring process, staff’s terms of reference, supervision, delegations of authority, resources, and relationships with the REB. Initially, the staffing levels for each regional office were far below the levels in BESO-I. In May 2003, it became apparent to USAID and AED that the staffing levels in the field were too low. USAID approved an increase in the numbers and AED immediately began the recruitment process. While the number of field staff has increased significantly over 35 the past year, there are still some regions without any coverage and some that could benefit from more coverage. Once the recruitment process began in full force, AED faced challenges in fully staffing their field offices. AED has had difficulty in finding people to work in regions with harsh conditions, such as Afar and Gambella; getting approval from REBs for candidates; and getting REBs to release their staff for AED contract positions. An internal constraint that AED has is that their salaries are relatively low, making it difficult to attract qualified people. Several REBs are also unhappy about what they perceive as lack of transparency and insufficient collaboration in the recruitment and hiring process. They report that either they were left out of the process completely or AED was not responsive to their input. AED reports that they tried to actively involve the REBs in the hiring process for their regional officers by asking the REBs if they had nominees or would like to advertise the positions locally. They also sent the REBs the short list of candidates for their review and selections and usually asked if they would like to interview the candidates. In all cases, AED asked the REBs for their final approval before they hired a person. Given the radically different perspectives on this issue, it is clear that the REB-AED relationships could benefit from improved communications on personnel issues. Since staff has been hired, there have been some issues with AED’s terms of reference for the staff, the authority granted to them, and their supervision. There is some concern that not all their scopes of work are clear and some may not have sufficient work to do. Staff is further constrained by a perceived lack of authority to independently make decisions related to technical or operational issues. Their understanding is that they must consult with AED’s central office on all technical issues and that they have been delegated insufficient authority to manage their financial transactions. This is further complicated by the fact that each of the field staff members report to a different person in AED’s central office. This not only affects internal relations and efficiency, but also results in lack of a focal point for REB-AED communications. AED is addressing this last issue by assigning team leaders for each of the field teams. Field activities have also been hampered by not having fully functional offices. Some lack a fax machine, copier, and Internet connection. And, at this point, none have vehicles (they have been purchased, but are awaiting final release by the GFDRE). Some REBs have also been slow to make space available for AED staff. In some regions, the relationship between AED and the REB has not satisfied the expectations of the REB. Collaboration on recruitment and hiring, as mentioned above, is a sore point for some, others feel that AED bypasses them too often in communications with partners in the region, and some feel that they are not well informed of on-going activities. The challenge with addressing some of these issues is that there is frequent turnover in the REB. So for example, if AED collaborated or communicated on a particular issue with the REB, the institutional memory of that is lost when personnel are transferred. REBs are also understaffed and the existing staff has heavily workloads, 36 making it difficult to get their attention. These challenges must be addressed, with adjustments made by both parties, to strengthen the partnership. Consultants – USAID has expressed concern for the process AED uses to recruit and select consultants. It appears that AED had identified some potential candidates from their own networks without an open competition. Furthermore, AED has had consultants begin work prior to USAID approval of the candidates. USAID has informed AED that they should openly recruit for consultancies and that USAID approval is required for each candidate. If they do not obtain USAID approval, USAID will not approve related payments. There has also been, on AED’s part, difficulty in getting the number and caliber of consultants needed in certain areas. There is a preference on the MOE’s part to contain the number of consultants (and subcontractors). This has placed a heavier load on full￾time staff and forced them to work outside their areas of expertise. USAID has expressed concern that some of the full-time expatriate staff do not seem to have the expertise to handle their assignments and that consultant staff was brought in to redress technical weaknesses among AED full-time staff. There also appears to be a strong preference for local consultants. Expatriates and Ethiopians on AED’s staff feel constrained by the limited pool of local consultants, especially in highly specialized areas. As implementation intensifies in the final months, AED may need extra support to complete all the activities. USAID and AED need to agree on the number of consultants that will be needed and move quickly to identify, approve, and mobilize them. Accountability for Deliverables – Various parties expressed concern that supervision of senior management and staff is too loose and thus affects AED’s ability to keep pace with the Implementation Plan and related deliverables. The planning team in particular appears to have a problem with delegating tasks, which results in some underutilization of staff. This practice has affected the timeliness of the deliverables. It appears that AED has made insufficient effort to change this practice. Setting Priorities – USAID is concerned that too much time is spent on tasks of relatively little importance, including some tasks not in the contract. USAID is especially concerned because time spent on these activities means less time on high priority tasks, such as completing the Performance Monitoring Plan in a timely manner. Not following USAID’s guidance on setting priorities has been considered “unresponsive” to the client. Workload distribution – There was some mention of disparities in the amount of work each AED team has. Some believe there is an imbalanced workload between staff in field offices and the central office, and among various parts of the central office. However, little mention was made of this issue in interviews with AED staff and so it was not pursued further. b) Organizational Structure After the TWG meeting in April 2003, AED and USAID recognized that unless something was changed, the project would fall further and further behind. So, in May, 37 AED sent a 2-person team to Addis to assess the management and implementation issues. As a result of that visit, several key changes were made in the organizational structure of the project: • Creation of management team – A management team was formed that consists of the Chief of Party, Deputy Chief of Party, and Teacher Development Team Coordinator. This team meets formally on a weekly basis and informally several times a day to deal with technical and operational issues. • Grouping of IR1 and IR2 components – Pre-service, in-service, school leadership, instructional materials, IT support for TEIs, and women teachers’ support were combined to form the Teacher Development Team. • Realigning IR4 components – The Chief of Party was assigned to coordinate all IR4 components, including planning and management, and MERA activities. • Reassigning monitoring responsibilities – The DCOP was given the responsibility for monitoring and reporting on implementation of activities and helping ensure that AED focuses on project outcomes. Within several months of this change, USAID and TWG recognized the improvement and publicly praised AED for the progress. AED staff—both expatriates and nationals— agrees that the change has been positive. Specifically, staff and management mentioned that the change has helped balance the workload, given technical direction in some areas where it was weak, added clarity on team responsibility and accountability, increased efficiency, improved the lines of communication, and that there are fewer funding and procurement delays. One area of the management structure that could use some attention is the perceived rigid line between the operations team and the two technical teams. Some people on technical teams feel that they are not sufficiently empowered to be fully accountable for their activities, as they are at the mercy of the operations team. The operations team, on the other hand, feels that the technical teams do not fully appreciate the competing demands on their time and the need to follow the regulations. Bringing the teams together in some forum might break down the barriers and help both sides strengthen the sense that they are all part of one team and need mutual support. Gaining a deeper understanding of the responsibilities and challenges of the other team would also be helpful. c) Financial and Administrative Management There are three key issues related to AED’s financial and administrative management: the pipeline, the slowness, and the perceived lack of transparency. The financial issue of greatest concern to USAID is the large amount of money in the pipeline. As of December 2003, only about $6 million of the $19 million in the contract had been spent. USAID is concerned that because the money is not start flowing faster, there is a large pipeline which leads to reduced allocations in basic education for the Mission. The slow spending rate is, of course, related to factors already discussed elsewhere in the report: slow project start-up, delays in releasing funds to TEIs, delays in procuring equipment and materials for cluster schools, and various other delays in implementing project activities. 38 From the beneficiaries’ perspective, the greatest concern is the speed at which funds and procurements arrive to the beneficiaries. The sentiment of the beneficiaries can be summed up by a comment made in one of the regions visited, “It’s easy to get technical support from BESO, but not money or equipment.” Given that this is a widespread feeling among beneficiaries, and has caused some discontent, AED should conduct a thorough review of their systems and procedures to determine where improvements could be made. In particular, it appears that the Deputy Chief of Party has been given too much authority in relationship to the Chief of Party. From AED’s point of view, this division of responsibility (Chief of Party focusing on providing technical support and having overall responsibility for the projects, and the Deputy Chief of Party having full administrative and financial responsibilities) is consistent with what they proposed and is appropriate to the highly complex nature of the project. An issue internal to AED is that there is a great deal of confusion and possible misunderstanding regarding financial, administrative, and procurement matters. There is a wide range of views among central and field staff regarding reasons for delays in releasing funds, procurement constraints, delegation of authority, segregation of duty, and the need for timely and accurate budgets and spending projections. From AED’s management perspective, they have been as open as possible in sharing their financial information and, in fact, the amount of financial information they share on this project far exceeds what is normal on their projects in other countries. This suggests the need for AED to work on correcting the misperceptions. 4. Linkages with CGPP USAID’s efforts to strengthen quality and equity in primary education include the Community Government Partnership Program (CGPP). This program supports the following IR and sub-IRs in USAID’s Results Framework for basic education: IR3: Community-government partnerships in education increased 3.1 Regional, zonal, and woreda offices’ support to increased community involvement in school management strengthened 3.2 Parents and community leaders more engaged in school management 3.3 Access to and survival in education (especially girls) in disadvantaged circumstances increased through flexible schooling While AED is responsible for implementing activities under IR1, IR2, and IR4, the responsibility for IR3 has been given to Save the Children USA, Tigray Development Association (TDA), and World Learning Ethiopia (WLE). CGPP covers all regions, with each organization responsible for some portion of the regions. To a limited extent, AED and CGPP partners are working with the same schools and are providing a compatible, but different, set of inputs. It is USAID’s desire that the four organizations collaborate to achieve greater impact in the target schools. The four partners all attend the TWG meetings and are aware, at least on a broad level, of the activities of the other partners. They have also met several times to discuss ways to improve their collaboration. At their meeting in January 2004, they initiated a mutual 39 agreement on principles and procedures for collaboration. They met again in February to further refine the agreement. The purpose of the agreement is to: (1) strengthen implementation success through cooperation and sharing resources, and (2) facilitate cooperation and collaboration among all the partners, with the MOE, and with USAID. The draft agreement includes nine areas of collaboration. At this point, the agreement is not fully operational. There is a lot of good will among the partners and they are all committed to contributing to a common goal. There are not, however, a lot of significant or substantive areas for collaboration. For example, there are GCPP schools that AED does not work with, and vice-a-versa. The schools were assigned by REBs, whose intention was to spread the benefits of USAID’s assistance rather than concentrate it in fewer schools. In retrospect, more effort should have been made to negotiate the assignments with the REBs. Similarly, there is not a lot of overlap in the content areas of their work: AED’s work focuses on teachers, while the CGPP partners’ work focuses on the community. While the opportunities for collaboration have been constrained by external factors, there are some opportunities that have not been fully optimized. For example, all the partners work on gender issues, but there have been a few instances where AED and Save have duplicated—rather than coordinated—their efforts to establish girls’ advisory committees at the school level. Likewise, there should have been more coordination in the development of the Woreda Capacity Building (WCB) module on community participation. The CGPP partners have done considerable training with woredas and could have added value to the content of the module. At a minimum, they could have helped minimize duplication of effort. Although the modules have been finalized at this point, there is still an opportunity for AED to coordinate with CGPP partners on the WCB program by recruiting trainees and promoting the program. If all partners commit themselves to full implementation of the collaboration agreement, the problems with duplication and lack of coordination should be resolved. III. RECOMMENDATIONS A. Design and Resources With the lessons learned regarding the overly ambitious design of BESO-II, the evaluation team offers the following recommendations: Recommendation A.1: USAID should add no new tasks to AED’s base-period contract and AED should focus on completing current implementation plans by the end of the base contract period. Recommendation A.2: An early deliverable of BESO-II, Phase II should be a sustainability plan for all activities implemented in Phase I. 40 Recommendation A.3: USAID should limit BESO-II, Phase II activities to the current scope of work. While it is necessary to be flexible and responsive to changing MOE needs and USAID priorities, it is also important to remain focused and provide the follow-up support necessary for maximizing the impact of base￾period investments. Possible adjustments to the current scope of work that could maximize impact include: A.3.1 – Revisit the geographic coverage, especially in light of USAID’s new strategic plan and emphasis on food insecure areas. A.3.2 – Focus resources on the most committed TEIs. A.3.3 – Expand the number of TEIs receiving support for Centers of Excellence, but remain focused on teaching aids and teaching materials. A.3.4 – Provide follow-up support and materials to the first cohort of graduates from the Higher Diploma program. A.3.5 – Conduct on-going rounds of leadership training to keep pace with the continuous turnover of personnel. A.3.6 – Provide additional resources to the Phase I “impact” cluster schools, rather than adding new impact clusters each year, to deepen impact. A.3.7 – Support creative approaches to dealing with the issues associated with clustering, i.e., distance materials, school-based training, or expert-team approach. There are many needs in the basic education sector in Ethiopia beyond the current scope of work. However, USAID should not expand the type of activities it supports until Phase I activities are fully functional, all necessary follow-up support has been provided, and a sustainability plan is in place. It is important to recall the two of the lessons learned from BESO-I:* • A tighter strategic focus will mitigate fragmentation of efforts and assure that resources are deployed lead to desired results. • Due to wide variation in regional factors, successful pilot activities need to be undertaken in five to six regions (or about half) in order to position an activity for national replication and/or impact. At this point, it appears that there is more money remaining in AED’s contract than they will likely spend by August 2004. However, it is not clear that there are sufficient resources currently in the contract for Phase II, should USAID exercise any of the option years. Recommendation A.4: AED should conduct a thorough analysis of the activities they expect to complete before August 2004 and their associated costs, and calculate the amount that will remain unspent. USAID, in consultation with MOE, should then determine whether it is best to (a) make adjustments in the budget to * USAID/Ethiopia ISP 2001-2006, p. 52. 41 increase funding for on-going activities that could make the best use of additional funds, or (b) carry-over any remaining funds from Phase I to Phase II or to a BESO-III. Recommendation A.5: To ensure there is adequate funding for Phase II, BESO-II, USAID and MOE should carefully consider their expectations for the future and the implementing partner should conduct a careful analysis of the corresponding administrative requirements, including the type and number of staff. There will continue to be a challenge having the desired impact on the in-service teacher training activities without sufficient financial support from the MOE. Recommendation A.6: USAID should seek a commitment from the MOE to provide funding for per diem and transportation to teachers who participate in in￾service training. If this commitment is not made, USAID should reconsider its expectations for impact on the in-service training program. B. Project Implementation and Management The issues related to project implementation require discussion and action from USAID, MOE, and AED. First, the one that requires action from the MOE: Recommendation B.1: USAID should reach agreement with the MOE on the most significant constraints within the MOE’s control: clustering issues, understaffing, personnel transfers, and incentives for in-service training. Until these issues are adequately addressed, they will continue to hamper BESO-II implementation. If there are adequately addressed, USAID will need to address the implications related to project design and implementation. To address the primary concerns of beneficiaries: Recommendation B.2: AED should conduct a thorough assessment of its financial and procurement system and procedures to better understand the bottlenecks and implement solutions, especially the relationship between the technical and financial teams of AED. And to address a related internal AED concern: Recommendation B.3: AED should implement a strategy for achieving greater clarity and transparency in financial and procurement related matters. To address the personnel management issues of greatest concern to USAID and AED staff: Recommendation B.4: AED should give immediate and high priority attention to addressing management issues, including possible management changes that affect project implementation and the perceptions of the client: 42 B.4.1 – B.4.1 – Conduct a thorough review of the issues related to the field staff and implement solutions, including an overall review of the effectiveness all AED and local staff as suggested by the TWG. B.4.2 – Reach agreement with USAID on the number of type of consultants that can be used to expedite and complete Phase I activities, and commit to obtain USAID approval before executing consultant agreements. B.4.3 – Implement a system for making management and staff accountable for missing deadlines (that are within their control) and for drifting from the priorities of the project. B.4.4 – Conduct team-building activities to create greater understanding and cooperation between the field and central offices, and between the operations and technical teams. Implementation of these recommendations should address the key issues related to design and implementation. There are, of course, other improvements that could be made, but USAID, MOE, and AED should focus on these first. As the end of the base period approaches, USAID and MOE are left with two basic options: (1) end BESO-II, design BESO-III, and conduct an open competition among implementing contractors or (2) continue BESO-II by exercising part or all of the option years in AED’s contract. The evaluation team recommends: Recommendation B.5: Exercising some portion of the option years in AED’s contract to allow sufficient time for: (a) AED to follow-up on Phase I activities, deepen impact, and plan for sustainability and (b) USAID and MOE to redesign BESO in light of the Mission’s new strategic plan and lessons learned to date. In exercising the option, USAID should reach agreement with AED on how and when the implementation and management issues will be resolved. 43 IV. ANNEXES A. Evaluation Statement of Work USAID/ETHIOPIA EVALUATION OF AED/BESO PROJECT STATEMENT OF WORK I. Title Comprehensive evaluation of the Academy for Educational Development (AED) BESO II Project (663-C-00-02-00349-00) II. OBJECTIVE The objective of this Delivery Order is to provide a team to conduct a comprehensive evaluation of the AED/BESO II Project. Results of this evaluation will assist USAID/Ethiopia and the Government of the Federal Democratic Republic of Ethiopia (GFDRE) in determining the effectiveness of the AED/BESO II Project in target regions; measure its impact to date; identify and analyze any implementation problems, draw out lessons learned and make recommendations for change. The findings and recommendations of the evaluation will provide USAID/Ethiopia and the GFDRE with guidance on the way forward in the development of USAID/Ethiopia’s Basic Education Strategic Objective (BESO II) strategies to enhance quality and equity in education. III. BACKGROUND The first BESO was a seven-year (1994 - 2001) cooperative effort of the Ethiopian Government and USAID to improve the quality and gender equity of primary education in Ethiopia, working with the Ministry of Education (MOE) and in the Tigray and Southern Nations, Nationalities and Peoples’ (SNNPR) Regions. AED/BESO I focused assistance in support of teacher development (pre￾service and in-service teacher training); effectiveness of the content and quality of materials; interactive radio instruction (IRI); planning; policy; decentralized management; and more efficient school financing. The overall strategy for USAID/Ethiopia’s second generation, BESO II (2002 - 2007) is much the same as BESO I, to enhance quality and equity in primary education. In BESO II, USAID has expanded its support from two regions to all regions in the country for most activities. The Academy for Educational Development (AED), the BESO I institutional contractor, is also the contractor implementing key BESO II activities. Project support from AED includes technical assistance, training and commodities as well as pilot activities in support of enhanced quality of professional education personnel; strengthened teaching/learning support systems and strengthened educational management. 44 Specifically, the AED/BESO II project supports: a) teacher development in all the 20 Teacher Training Institutes (TTIs) and the Teacher Training Colleges (TTCs); b) a cluster-based in-service program will cover about half of the teaching force (50,000 teachers) at the primary level over the life of the activity; c) upgrading of second cycle (grades 5-8) primary teachers (21,400) through distance education over the life of the project; d) development of supplementary reading materials in social science subjects such as Civics, HIV/AIDS, Environment, etc. as well as Interactive Radio Instruction (IRI); and e) systems for women teachers and women teacher candidates. AED/BESO II project activities implemented at the central/national, regional and weredas (districts) also provide comprehensive capacity building in education planning, management, information systems, monitoring and evaluation. The project is fully consonant with the national Education Sector Development Program (ESDP), with a clear objective of improving quality and gender equity at the primary level. The AED/BESO II contract was signed in July 2002 and implementation activities began at that time. The contract is a five year including two base years and three option years. This evaluation will cover the progress made from August 2002 to the present (middle of base year 2). IV. STATEMENT OF WORK The Team shall conduct a comprehensive evaluation of the AED/BESO II project. The evaluation shall (a) examine the results and effectiveness of the project in relation to the objectives of the contract and in relation to the USAID BESO II Results Framework; (b) analyze implementation problems; (c) draw out lessons learned for improving the project design; (d) measure Ministry of Education (MOE), Regional Education Bureau (REB),Teacher Education Institutions (TEIs), and teachers’ satisfaction with project activities; and (e) make recommendations for change in project design and implementation modalities if necessary. Specifically, the Team shall address the following issues: A. Project Results and Input: The evaluation will examine the project’s overall results to date, including: i. A systematic review of established targets (e.g. Central and National as well as Regional under the established AED/BESO II Objectives and Implementation Plan). If performance is not on target, reasons (positive or negative) shall be identified. ii. The impact the project in total has had to date on the targeted beneficiaries. For individual project activities, the evaluation 45 shall answer questions such as “What are the achievements gained so far and how sustainable are they? What are the prospects for expanding interventions that are judged effective? How cost-effective have the interventions been? What research has been conducted and have the research findings had any impact on policy reform? Has the training provided under the project (knowledge, information and skills) been utilized? How? Are the books, equipment and other resources provided to institutions being fully utilized? How have AED/BESO II project activities and outputs been integrated, where appropriate, into initiatives of the Government at cluster, weredas, regional and national levels in terms of government policy and planning? Are project activities planned and coordinated with other development partners? How? Has AED/BESO II implemented the recommendations that have so far been forwarded by the Technical Working Group (TWG)? If not, what are the reasons?” If performance is not on target, reasons (positive or negative) shall be identified. B. Relationship to Mission Plans: The evaluation report shall establish to what degree the BESO II Results Framework has been supported by the AED/BESO II project. The report will also examine how well the AED/BESO II project has enhanced social resiliency, as defined by USAID/Ethiopia. The report will also examine how the Mission's education activities might advance social resilience most effectively, with particular reference to how it might work in greater synergy with the Mission's health activities C. Lessons Learned: The evaluation team will draw out lessons learned. Specifically, i. Has the contractor assembled the necessary resources, including personnel and organizational set-up in a timely manner, to ensure timely and efficient implementation of the project? ii. What are the constraints that impact project implementation (policy environment, operational, institutional [within AED, USAID and/or host country partner institutions]) and what has been the impact on project implementation? iii. Have the constraints been rectified? How? iv. How sustainable are the achievements gained? v. Are the project design and resources appropriate for reaching stated objectives? vi. How does project-generated technical information and/or formal research impact educational policy reform? 46 vii. Are linkages between the Community Government Partnership Program (CGPP) and AED/BESO II fully optimized? D. Ministry Satisfaction: the Team will determine the degree to which the project has responded to perceived needs of its beneficiaries: teachers, education managers, government partners at the cluster, weredas, regional and national levels. Which activities are perceived as most important? Which least? Why? A. Design Recommendations: the Team will analyze the evaluation findings, including the present budget levels in relation to the project objectives and expected results, the reasonableness of the targets in relation to time, contractor’s organizational structure, implementation environment, etc., and make recommendations for changes in the present project design and implementation plan. 47 B. List of Individuals Interviewed Location Institution Name Title Addis USAID Cheryl Kim Chief, HID/DG Aberra Makonnen Deputy Director/CTO Tesfaye K/Work M&E Specialist AED Thomas Tilson Chief of Party Deborah Hanley Deputy Chief of Party Johnson Odharo TDT Coordinator Joseph Bastian Planning & Management Advisor Kara Janigan Instructional Materials Advisor Bahiru Shikur Pre-service Training Officer Alemnesh H/Mariam Women Teacher Support Officer Jerusalem Mekonnen HR Specialist Endeshaw Shenkute Head Driver & Gen. Service Abraham Hagos In-service TDT Officer Mohammed Hussien Procurement Specialist Rahel Mekuria Database Assistant Asmelash Meressa Computer Programmer Woubejig G/Kirstos Senior Secretary Teshome Nekatibeb M&E Advisor Yemisrach G/Michael Senior Secretary Hiwot Tsegaye Senior Secretary Baraki Zesilase Chief Financial Officer Plan./MOE Sitotaw Yimam Head NOE/MOE Awash Gebru Head ICDR/ MOE Tizazu Asare Head EMA/MOE Demissew Bekele Head TEMSD/MOE Tibebu Zenebe Expert TESO/MOE Wubie Kassaye Coordinator Save the Children Sophie Makonnen Chief of Party TDA Tesfazghi Aberra Exec. Director World Learning Robert Gurevich Chief of Party Amhara Gonder CTE Zerihun Mekonnen Dean Senay Zegeye Vice Dean for R & D Seraye Esubalaw Academic Vice Dean Shewaneseh Fiseha Head SDU Sertse Abebe Librarian Misganaw Shibabaw Head IMU Susan Nicholson Instructor, VSO Greg Somerville Instructor, VS O Wosenyelesh Yadessa Female trainee Haimanot Mekonnen Female trainee Tigist Estifanos Female trainee Mengistu Abebe Instructor 48 Location Institution Name Title Gonder WEO Endale Getaneh Assistant Head Atse Bakafa CRC Tadege Alene Director Kebele “16” Elem. School Solomon G/Mariam Director Enfraz CRC Tamirat Ferede Deputy director REB Tilaye Getie Head Fanta Moges Deputy Head Theodros Shewarget Head, HRD Girma Zeleke Head, PMMS YayehYirad Belay Head, PAS Amelework Mamo Expert, Public Relations Habtamu Bizuneh Head, PPME AED/BESO II Amde Michael Endeshaw In-service Training Officer AED/BESO II Chalachew Lema Admin & Finance Officer WLE Worku Ambelu M & E Assistant WLE Mengistie Alemu SDA, WEO Merawi ICS Getachew Eshetu Director WEO, Mecha Getnet Nigate Training Expert Oromia REB Adamu Ayana Bureau Head Dereje Asfaw Deputy Bureau Head Teshome Lemma Planning Program Panel Head Etefa Merga Training Expert Abebe Jira BESO II Coordinator AED/BESO Tesfaye W/Michael Planning & Mngt. Officer Shenkute Mamo In-service Training Officer Amare Bereda In-service Training Officer Rago Birru Pre-service Training Officer Gemechu Dechasa Operations Officer Adama TTC Fetene Regassa Assistant Dean Desta Abera PC technician G/Yesus G/Giorgis Librarian Adama WEO Fekadu Moreda Head Gujo Zewdie Expert Assela TTI Boki Tola Principal Assela Pri. School Feleke Shiferaw Director Tigil Fire Pri. School Amantu Fayiso Director Bokoji Pri. School H/Michael Eshetu Director Ulule Kereyu Pri. School H/Mariam Gelana Director Tulu Negaso Pri. School Abebe G/Egziabher Director Somali REB Abdulfetah ShekAbdulahi Head Mustefa Mahumud Deputy Head Abdulahi Mohamed Training expert Jijiga TTI Ato Anes Principal 49 Location Institution Name Title Elias Omar SDU Head Abdulkadir Haji ATEPAP chairman Abeba Wondafrash Librarian Zonal Ed Office Zacaria M.Gudal Assistant Head Girma Beshahu Expert Chinaksan CRC Nimaan Shidee Director Hussien Girre CRC Girma Negash Facilitator Samuel Assefa Teacher Daniel Amenu Teacher Amare Yisma Teacher Ahmed Gurey Satellite Nahusenai Mekonnen Facilitator Nigussie Tefera Facilitator Sintayehu Daba Facilitator Mengistu Shewategegn Unit leader AED/BESO Weis Mohamed Regional Advisor SNNP WEO, Lemo Yakob Bezabih Head Tagesse Lambe Head, Education Programs Tadesse Chamisso Supervisor Ambicho CRC Demeke Haile Director Tesfaye Lenjore Teacher Hossana TEC Birhanu W/Giorgis Dean Solomon Lema Vice Aca.Dean Bizuayehu Begashaw Librarian Muluneh Guracha SDU Head Girma Worku PC/RC Head Masresha Dehinanew Comp. Lab Head Ms Libey V. English RC Head Tessema Kassaye Music teacher Gebeyehu Feleke Science Lab. Head Aster Samuel Girls’ Club Coordinator Tigist G/Amanuel Female student Minyilu Mengistu Female student Workinesh Mitiku Female student REB Birhanu Belayneh Deputy Bureau Head Tadesse Woldu Head, Curr. & Research Dept. Girma Bekele Assistant Head, Educ. Programs Galonde Waketa Head, EPPIE Section Yoseph Deboch Head, Personnel Section Shanka Biramo Head, Pool Service Tessema Abebe Head, Gen. Serv. and Procurement Sabba Philipos Project Storekeeper AED/BESO II Simegnew Kassaye Planning & Management Officer Awassa City Edu. Desk Zeleke Wondimu Supervisor 50 Location Institution Name Title WLE Office Birhanu Manalew Regional Director Dereje Bekele Program and Training Officer Awassa CTE Sahelu G/Wold Academic Vice Dean Abebe H/mariam Dean of Students Wondimagegn Tuji Head, DE Unit Emebet Kassa Librarian Awoke Biazen Assistant Head of Library Asnakech Mulugeta Asst. Librarian Deriba Workineh SDU Coordinator Girma Legesse Head, Computer Lab Negessu Kasse Member of Girls Focal Unit Bete Kihnet Cluster school Tibilet Tedla V/director Teshome Gebeto Unit Leader Desta Lodamo Department Head 51 C. Selected Documents Reviewed Academy for Educational Development (AED). January 2003, April 2003, November 2003, February 2004. BESO-II Quarterly Performance Report and Activity Status Report. Addis Ababa, Ethiopia. _____. July 2003. BESO-II Annual Report, September 2002-June 2003. Addis Ababa, Ethiopia. _____. July 2003. BESO-II Implementation Plan (revised final). Addis Ababa, Ethiopia. _____. July 2003. Letter to USAID with Proposed Revisions to the AED/BESO-II Scope of Work. Addis Ababa, Ethiopia. _____. September 25, 2003. Letter from Beverly Jones, AED Senior Vice President, to Cheryl Kim, USAID Chief of the Office of Human and Institutional Development/Democracy and Governance. Washington, DC. _____. December 2003. BESO-II Performance Monitoring Plan. Addis Ababa, Ethiopia. BESO-II Partners. January 12, 2004. Draft Agreement on Principles and Procedures for Collaboration for BESO-II Partners. Addis Ababa, Ethiopia. BESO-II Technical Working Group. April 2003, July 2003, and November 2003. Meeting Minutes. Addis Ababa, Ethiopia. USAID/Ethiopia. November 2000. Integrated Strategic Plan: FY 2001-2006. Addis Ababa, Ethiopia. USAID/Ethiopia, Contracts Office. August 2002. Contract 663-C-00-02-00349-00. Addis Ababa, Ethiopia. USAID/Ethiopia. August 5, 2003. Letter from Mary Lewellen, Mission Director, to Steven Mosley, President of AED. Addis Ababa. 52 D. Table of Output Targets and Progress Table 1: Pre-Service Teacher Training Output Targets No. Output Statement Phase I Target Modified (if different) Outputs to Date Comments 1. a) Number of TTCs and TTIs that have improved the quantity and quality of library materials 18 19 19 Achieved. Gone beyond by developing minimum standards for TEI libraries. The standards will be approved in April. b) Number of TTCs and TTIs that improved the resources and operation of an Instructional Resource Center (or equivalent); 18 19 Achieved. Gone beyond by developing minimum standards for TEI PCs & IRCs. The standards will be approved in April. c) Number of TTCs and TTIs that have provided adequate funds in their annual budget to maintain their library and/or center, 5 12 Achieved. Our studies on record show that 12 TEIs have increased their annual budget for libraries and PCs/IRCs. 2. Number of TTCs and TTIs that demonstrate adequate commitment of human, materials, or financial resources to the activities 5 19 All TEIs have contributed significantly to support project activities. Some have contributed as much as 20-30% budget to support AED/BESO activities, especially in curriculum development and training, procurement and renovation of facilities. 3. Number of TTIs/TTCs that are implementing plans to improve the percentage of women graduates successfully completing their training [modified statement] 15 10 Only baseline Showing 9 TTIs & 4 TTCs Our baseline studies comparing graduates of 2001 & 2002 show that only a moderate increase with fluctuations of GPAs from year to year. We have shared this information with AID and they recognize the difficulty in measuring this output. Other progress and achievements to date - Development of Centers of Excellence: Although not required in the contract, they will provide a slow but assured journey to producing quality teachers for primary schools. All three centers have taken off and should be fully functional by the end of the Phase I. Development of the software for MIS for TEIs is completed, pending testing. Overall constraints in achieving targets - The major constraint is slow performance of some of the TEIs--some are behind schedule, others have completed their activities but have not reported and so contractually are not done. The most common excuse is engagement in regional in-service, heavy teaching load, teaching evening and week-end extension classes, 2nd cycle distance education tutorials and management, and higher diploma program. They are also busy developing teaching materials for the new teacher education syllabi. We will keep on pressing as well as providing technical assistance to help out. 53 Table 2: In-Service Teacher Training Output Targets No. Output Statement Phase I Target Modification (if different) Outputs to Date Comments 1. a) Number of multi-session self￾instructional kits (content, print materials, audio cassettes) for in￾service, continuing education for primary teachers developed and field tested (reported by topic or theme of each kit, and language of instruction) Mod 1 Statement: a) Number of multi￾session self-instructional kits (content, print materials, audio cassettes) for in￾service, continuing education for primary teachers developed, translated and pilot kits produced. C1 & C2 - 1st cycle kits: 2 kits developed one for G1-2, and G3-4, to have a complete set for 1st cycle primary. C1 -2nd cycle kits: 2 kits developed, one for G5-6, and G7-8, to have a complete set for 2nd cycle primary. C1 & C2 - 1st cycle kits: 2 kits developed, translated and field￾tested, one for G1-2, and G3-4. C1 -2nd cycle kits: 2 kits developed, one for G5-6, and G7-8. Currently editing the first draft. C1 & C2 - 1st cycle kits: 2 kits developed, translated and field-tested, one for G1-2, and G3-4. C1 -2nd cycle kits: 2 kits developed, one for G5-6, and G7-8. Currently editing the first draft. Translation and field-testing will be completed before end of August. 54 Table 2: In-Service Teacher Training Output Targets No. Output Statement Phase I Target Modification (if different) Outputs to Date Comments b) Number of multi-session self￾instructional kits (content, print materials, audio cassettes) for in￾service, continuing education for primary teachers produced for widespread dissemination Mod 1 Statement: b) Number of multi￾session self-instructional kits (content, print materials, audio cassettes) for in￾service, continuing education for primary teachers Field tested, revised and produced for widespread dissemination C1 - 1st cycle kits produced total 25,000 C2 -1st cycle kits produced total 20,000 for G1-2. C1 - 2nd cycle kits: produced total 15,000 for G5-6 and G7-8 C1 & C2 – 50% of the total number of G1-2 and 3-4 kits (45,000) have been produced. C1 & C2 – remaining 50% of the total number of G1-2 and 3-4 kits (45,000) will be produced before the end of March. C1 - 2nd cycle kits: produced total 15,000 for G5-6 and G7-8 will be produced after field￾testing and before the end of August. 2. Number of primary teachers completing at least one self-instructional kit (reported by region/zone and by gender) C1 -1st cycle kits: by 12,000 (1-4) teachers 163 teachers from 15 schools in 5 regions have completed at least one module of the G1-2 and 3-4 kits through field testing Kits will be disseminated starting the end of March and we anticipate collecting usage data by the end of the school year. 3. Number of school headmasters, regional, zonal and woreda education support personnel trained in constructive teacher support methods (reported by region/zone, by type of personnel trained, and by gender). Program reaching: School heads: 3,500 Regional, zonal, woreda officers: 800 Currently the project has reached 4,437 school directors (including vice-heads) and 756 education officers in all regions. The project will reach its target as planned. The project also will reach more than the target before August 2004. 4. Establishment of a viable teacher strengthening system that receives continued support from the pre-service training system. Program reaching: Teachers: 35,000 The project has reached 37,539 teachers (beyond its target) It is expected that the project will reach more teachers and school directors through 656 cluster centers and 2287 satellite schools. Current effort is to ensure training has impact at classroom level. 55 Table 3: Supplementary Media Development and Training Contractor Outputs Targets No. Output Statement Phase I Target Modification (if different) Outputs to Date Comments 1. Number of primary schools in priority zones/regions that are using interactive radio in an active learning mode in grades 1-4 (reported by region, subject, and by language of instruction) Mod 1 statement: Revise IRI English programs to reflect recent ICDR English syllabi changes. 1,000 1,500 4,000 G1-2: IRI English programs are being broadcast nationally; we estimate that the programs have reached around 30% of the schools (4,000). G3-4: ICDR did not finish the curriculum development for grade 3 until June 2003, delaying AED’s efforts. AED is estimating that they will be completed by the fall. ICDR has not yet finished the curriculum development for grade 4. AED lost several months because we could not fill 3 vacancies until EMA filled their vacancies. All AED positions are now filled. 2. Number of "clusters" “ quality promoting”, similar mechanisms in priority zones/regions that are operational, as evidenced by a functioning teacher support resource center with an operation and maintenance plan and budget (reported by region/zone and by gender) Mod 1: Removed entirely. 56 Table 4: Socially Relevant Curriculum Integration & Training Contractor Output Targets No. Output Statement Phase I Target Modification (if different) Outputs to Date Comments 1. Existing curriculum reviewed, and revised based on school feedback on key socially relevant topics (reported by topic, grade, and language of instruction) Mod 1 Statement: Existing curriculum reviewed, and recommendations for revisions made based on school feedback on key socially relevant topics (reported by topic, grade, and language of instruction) Review of G1-4 curriculum (ICDR syllabi) completed & revised. Regional curriculum assessment completed in cohort 1 target regions. Review of G5-8 curriculum (ICDR syllabi) completed. Regional curriculum assessment completed in cohort 2 target regions. Review of G1-4 curriculum (ICDR syllabi) & recommendations made. Regional curriculum assessment completed in cohort 1 target regions. Review G5-8 curric. (ICDR syllabi) & recs made. Regional curric. Assessment completed in cohort 2 regions. Review of G1-4 curriculum (ICDR syllabi) completed. Regional curriculum assessment completed in cohort 1 target regions. Review of G5-8 curriculum (ICDR syllabi) in progress. Regional curriculum assessment in cohort 2 target regions in progress. Report of the review of the ICDR and regional curriculum for C1 regions for the scope and sequence of coverage for socially relevant topics in G1-4 completed. Report of the review of the ICDR and regional curriculum for C1 regions for the scope and sequence of coverage for socially relevant topics in G5-8 will be completed by the end of March. 2. Curriculum on identified socially relevant topics integrated and prototype materials/media produced (reported by topic, grade level, and language of instruction) Mod 1 Statement: Socially relevant topics identified and prototype modules developed and produced (reported by topic, grade level, and language of module) Modules on socially relevant topics to be integrated into the classroom curriculum developed, for G1-4 for cohort 1 and 2 regions. Modules on socially relevant topics to be integrated into the classroom curriculum developed, for G5-8 cohort 1 regions 3 Smart Arts Awards granted per cohort 1 region. Modules on socially relevant topics developed & translated for G1-4 for cohort 1 regions. Modules on socially relevant topics pilot-tested, revised, & distributed for G1-4 for cohorts 1 & 2 regions & for G5-8 in cohort 1 regions. 3 Smart Art Awards granted per cohort 1 regions. Modules on socially relevant topics integrated into the classroom curriculum developed and field- tested, for G1-4 for cohort 1 and 2 regions. Modules on socially relevant topics to be integrated into the classroom curriculum for G5-8 cohort 1 regions in development process. The following supplementary materials have been developed translated and field tested for G1-4: student booklet for grade 4 students entitled “Let’s talk about HIV/AIDS”, Grade 1 to 4 Teacher’s Activity Guide for Civic and Ethical Education using Student-Centered Active Learning Methods: Some suggestions, and Environmental Education Activity Book for grade 1-4 teachers. Materials for G5-8 in the process of being developed. Will be completed by end of March 57 Table 5: Women Teacher Support Systems Contractor Output Targets No. Output Statement Phase I Target Modification (if different) Outputs to Date Comments 1. Linkages established and nurtured between regional women's associations and women teachers on an intra- and inter-regional basis (reported by region/zone and type of linkage) Linkage activities within regions: 8 Linkage activities between regions: 2 Linkage activities within regions: 4 Linkage activities between regions: 2 Linkage established within region: 1 Linkage activity established between regions: 1 Late start. Linkage established between the REB, Women’s Affairs and TEIs gender offices. Amhara Region TEIs, REB & WOA have created linkage. They have organized competition among best performing female students in the TEIs and the best three from TTIs and TTCs are to be rewarded. Regarding linkage between regions, Oromia & Somali TEIs, REBs & WOAs created linkage & have identified common problems of female trainees. They’ve devised strategies to solve the problems with the membership of the linkages with teachers associations (female wing), NGOs, etc. Gender Issues Awareness workshops and academic support completed. One national activity among regions on networking and study skills. 2. Instructional and/or motivational short courses, training modules targeted at women teachers developed, produced, delivered (reported by topic, type of media, and language) Short courses developed, produced and delivered: 8 Training modules developed, produced and delivered: 4 Short courses produced and distributed: 12 Stand alone training modules produced and distributed: 6 6 short courses in development 4 modules in development The materials have been revised by many stakeholders including the beneficiaries. The final comments are being received by highly experienced professionals in Washington. When endorsed by USAID, some will be translated to 2 local languages, will be delivered in print form & thru radio broadcast. Delay in hiring Women Support Officer and problems with course/module development. The short courses and four modules will be completed by March and delivered by July 2004. 1. Other progress: Training on leadership skills given to 59 Oromia Region female teachers 2. Overall constraints: The material development took much time and the need to be reviewed by many stakeholders added to the delay. The interests of organization to join the linkages may not be easily won. 58 Table 6: Personnel and Instructional Materials Management, Planning, Monitoring, Evaluation & Information Systems and Student Assessment Output Targets No. Output Statement Phase I Target Outputs to Date Comments 1. Number of regions that a) have developed; b) are maintaining and using computerized personnel systems for at least basic personnel functions (hiring, assignment, salary & benefits, incremental increases (reported by region/zone) Prelim. database set-up completed in all the locations started Year 1. Procedures for maint. established. Maint. all locations in progress with evidence of PMIS utilization. PMIS software modified to meet region specific needs and installed in all regions. Translated into Amharic; translation into Afan Oromifa and Somali in progress. Thirteen data collection formats prepared and printed in all but 2 regions. Data collection training workshop held in 7 regions. Database management personnel trained in 7 regions. Managers of PMIS trained in 5 regions. Operations manual and user manual completed. Operations manual translated into Amharic; translation into Somali and Afan Oromifa in progress. Data collection and reconciliation is underway in 7 regions. The PMIS program is ready to be turned over by April to the following regions: Amhara, Dire Dawa, and Harari. We expect that we will be able to hand over the program to all the remaining regions except Oromia and Gambella and, possibly, SNNPR by August 2004. Complete PMIS system for the REB and all the institutions reporting directly to the REB could be turned over to the Oromia and SNNPR bureau before the end of August 2004. All woreda level data will also be collected by August, but full implementation will not take place until later in 2004. No activities could be undertaken in Gambella during the past several months because of the security situation prevailing in the region. Discussions with regional officials indicate that collection of data from schools & woredas will be difficult under the present circumstances. We have developed and implemented the PMIS system for the MOE, EMA, ICDR, and NOE. This is not in our contractual agreement. The systems for all these four institutions are ready, and personnel at different levels have been trained. The system is operational and we are ready to hand over to the system to the respective institutions. At the time of the proposal, we did not anticipate the translation of software, 13 formats, and the manual of operations into local languages. 59 No. Output Statement Phase I Target Outputs to Date Comments 2. Number of regions that a) have developed; b) are implementing improved instructional materials procurement and distribution systems on a routine basis (reported by region/zone, with reference to languages of instruction) Modified target: a-MMIS installed in 11 regional HQs. b-Preliminary database set-up started in all locations. c-Initiated a study on the procurement process & distributions systems. d￾Recommendati ons for strengthening procurement & distribution process will be provided to 11 regions for implementation . MMIS software has been adapted and installed in all regions. Initial orientation given in all regions. The coding structure for inventory control has been developed. This is applicable for all regions. Inventory is completed in 4 regions and is under way in 5 regions. Entry of inventory control data is completed or in progress in 5 regions. User training has been implemented in four regions. The interface of the software has been translated into Amharic and Oromifa languages. User’s manual has been developed. Translation in progress into Amharic and Oromifa. We will be able to hand over the MMIS to the following regions by April: Benshagul-Gumuz, Dire Dawa, and Harari. We expect that the system will be operational in all regions except Gambella by August 2004. We are developing and implementing MMIS for the MOE. This is outside our contractual obligation. MOE system will be ready to hand over within two months. In addition, at the time of the proposal we did not anticipate translation of the program and other documents into local languages. 3. Learning Assessment Moved to MERA component 4. Number of regions utilizing education information more efficiently for planning and decision making All 11 regions provided with 1 MQS and 2 additional decision support tools. We have installed long-range planning projection model and three decision support systems in all but one region (Dire Dawa). Up to five experts from all regions have been given a week-long training in the use of these systems. In All of the outputs mentioned here lay the foundation for undertaking the important activity of developing the perspective plan with the strategic objective of achieving UPE in all the regions. This planning process will take place from April-July. 60 No. Output Statement Phase I Target Outputs to Date Comments All regions using a long￾range perspec￾tive plan model for developing UPE strategies. addition, the regional planning officers have conducted periodic ongoing short training in their respective regions. The regions have capability to use the planning projection model for annual, short-term and long-range plan development. Oromia, Somali, Benshangul Gumuz, Gambella regions have developed five year plans using the projection model. We have helped five regions to develop short￾term plans. This is beyond our contractual obligations. In Tigray Region, the development of a Perspective Plan for UPE was completed during BESO-I and we have supported further refinement, detailed program development, and estimation of investment for primary education during BESO-II. We have started additional preparation for UPE plan development by giving short workshops on UPE definitions and strategies in three regions. We anticipate that all the regions will be using the computerized system for developing long-range plans by August 2004. In addition to the three decision support systems, we incorporated Minimum Quality Standards as an integral part of the planning projection model. In Dire Dawa, we were unable to install the systems because of lack of availability of computers with sufficient capacity. The computers will be made available soon. 5. Number of regions integrating Monitoring and Evaluation into their planning process and using it to follow progress 11 regions at HQ level. A computerized monitoring system at the regional level has been developed and integrated into the planning projection model. The model is capable of tracking the performance of the education system on key targets and outputs. M&E is integrated into our planning process for all regions. The model also allows the monitoring system to be incorporated into the woreda plans. In the absence of computers, the woredas can use the formats for developing monitoring reports. 6. Number of regions successfully integrating planning of activities Integrated annual planning and We have conducted at least one workshop in all the regions on the methodology and process of Development of preliminary budgets is an integral part of our planning activities in all regions. The decentralized planning and budgeting model 61 No. Output Statement Phase I Target Outputs to Date Comments and programs with budget or educational finance budgeting process implemented at regional level decentralized planning and budgeting system. In six regions we have conducted at least two workshops on this topic. We have developed a computerized model for estimating capital budget (investment) and for recurrent budget (investment). is a bottom-up model capable of building up the plans from woreda level to the regional level and then to the national level. Technical capability has been apply this model at every woreda in the country. This capability will be employed in the Woreda Capacity Building Program to develop woreda plans. Other progress and achievements - The IR4 team has been extensively involved in developing the new Woreda Capacity Building Program (WCB). This program will cover all 611 woredas in the country through an 8-day training program in approximately 35 locations all over the country. Approximately 3000 woreda personnel will be trained through this program by the end of June 2004. Every woreda will use its own education system and population databases for the development of the woreda plans. A course workbook for the training program has been developed and validated through a national workshop with participation from all regions. The workbook has been translated into three local languages: Amharic, Afan Oromifa, and Somali. Translation into Tigringa is in progress. Development of woreda levels educations system databases and school age population projections for 26 years have also been completed. The WCB Program will be launched nation-wide in April 2004 and will be completed by June 2004. Overall constraints in achieving targets - We have faced four major constraints: First, the impact of the new Woreda Capacity Building Program on our IR4 staff. This program has placed heavy demands on the technical and managerial time of our regional and central staff. With no additional staff, the program has coped with the demands so far; but some delays have been inevitable. Second, the Woreda Capacity Building Program has strained the organizational capacities of the regional education bureaus. The regional education bureaus have diverted many of their professionals directly involved in planning area to the development and implementation of the WCB Program. This makes it difficult for them to devote their attention and time to the regular IR4 activities. In addition the government’s ICT program has also placed demand on the planning and information system experts. Many of the people we have trained have been pulled out of the regional education bureaus to provide support to the Woreda ICT Program. In addition, the intensive WCB training program has occupied or will occupy most of the calendar from January to June 2004. This makes it difficult to find time to implement IR4 regular programs such as development of perspective plan for UPE. Third, high turn over of staff, especially at the senior management level, does not provide continuity to our programs. Planning process and dialogue on policies and techniques require understanding and commitment of senior management and experts without which it is difficult to achieve stability in planning. Fourth, the REBs have been restructured in several regions, again causing complications and delays in our efforts. 62 Table 7: Monitoring, Evaluating, Reporting, and Analysis (MERA) Output Targets No. Output Statement Phase I Target Outputs to Date Comments 1 Baseline and recommended targets established/refined for all BESO-II Performance Monitoring Plan (PMP) IR indicators during the first 60 days after the contract effective date, so that they can be reported in the FY 2004 R4 prepared by USAID in February 2002 Annual MERA planning workshop. M&E plan with refined indicators and targets to be reported for year 2005. All partners submitted their PMP USAID held a workshop on the PMP in September 2002 for all partners. In discussion with the Mission, AED is only responsible for its own PMP. 2 On-going program-related monitoring and evaluation (M&E) of impact providing R￾4 required information for activities for all PMP indicators no later than November 1 of each year for the period ending September 30 (i.e., the previous school year), beginning in 2002 and continuing through each year of the contract. PMP indicator report in Nov. Annual M&E dissemination conf. PMP completed Planned for April 2004 This workshop will review performance indicators and targets for 2004 at education system, IR and Sub￾IR levels; create awareness on the responsibilities of data collection analysis and reporting; review progress in monitoring and evaluation of AED activities, especially those in the PMP. 3 Annual surveys, topic-specific analyses, case studies, or other research that documents lessons learned from new and innovative approaches in primary education quality and equity generated, distributed by contractor (disaggregate by topic) 2 studies completed and findings distributed. Illustra￾tive topics: Impact the cluster prog. On teacher professional develop. Several studies are planned or initiated. 1. Economic costs of wastage resulting from inefficiency of school system (designed) 2. Repetition and causes (designed) 3. Active learning in TEIs and cluster primary schools (baseline data have been collected) 4. Educational governance study (TOR 1. Key study for MOE; focuses on economic impact of having an inefficient education system. 2. Aims to analyze causes of repetition and suggest interventions to reduce repetition. 3. Study to determine development of active learning teaching-learning practices in schools & TEIs for IR1 indicators 1 & 2 in PMP. 4. Focuses on impact of recent decentralization 63 No. Output Statement Phase I Target Outputs to Date Comments developed and consultants are being hired) 5. Methodology for and estimation of teacher demand and cost for primary teacher up to 15 years (study is near completion) on the roles & responsibilities of different levels in education & leadership 5. Study has developed estimates for teacher demand based on the educational data & AED computer models used in the regions. 4 Mod 1: New Target: Undertake a national learning assessment and other monitoring of educational quality at the classroom and school levels NEW TARGET Complete 1 National learning assessment including regional inputs, and provide results to regions for use in planning future regional assessments 2nd NLA is in progress. Data collection is going to take place in April 2004. The preparatory stage is under way. The National Advisory Committee and the Technical Working Group provide leadership to the whole activity. National Workshop has been conducted to create awareness of the Second NLA. Instruments have been prepared and pilot tested. They were also translated into different instructional media. With the help of an international consultant and a national workshop, the instruments were also validated. A sampling plan has been also prepared. Preparations are underway for printing instruments and training data collectors and students. 64 E. Proposed Revisions to the Contract Output Targets Shortfalls in Meeting Contractual Obligations and Suggestions for Meeting Output Targets or Contract Modification Revised September 22, 2003 Background The AED/BESO-II Project was contracted on the 12th of August 2002 and expected to begin operation from September 2002 through August 2003 for Year Base 1, and September 2003 through August 2004, for Year Base 2, thus ending Phase I. Due to the length of the planning process, the impact of beneficiary changes such as curriculum revisions on the contractual outputs, and the restructuring of activity budgets to more efficiently address intermediate results, AED is pleased to propose the following contract modifications. Please note that only Base Period output targets for which AED is proposing modifications are addressed – Base Period output targets for which AED is meeting or exceeding the targets have not been addressed herein. Neither have any output targets for the Option Period been addressed. Proposed Contract Adjustments Component 1. Pre-service OUTPUTS CURRENTLY STATED: Output 1(a) Target Year 1 requires that 10 TEIs, and 9 TEIs in Year 2, have improved the quantity and quality of library materials. Output 3 Target Year 1 that requires 10 TEIs have increased the percent of women graduates with GPA 2.75 and above. AED COMMENT: Since the Ministry of Education has introduced a new training curriculum to be implemented within Year 1 of the project, we decided to address all 19 institutions. As a result, we have funded all TEIs, conducted workshops and provided technical assistance to enable them to develop teaching materials and furnish their libraries and improve the instructional/pedagogical centers. However, since there was a delay in the USAID contractual and legal approval of the funding mechanism to the TEIs until late May 2003, procurement of library materials such as books and educational CDs from international suppliers which are long-lead procurements could not begin in time to receive these materials by the end of Year 1. In fact, they will not arrive until October 2003. 65 AED has collected GPA data for female students for two year cohorts. The disturbing finding is that there is NO pattern for female students, for example of the 10/12 TTIs only 5 showed consistent increase, including the BESO I TTIs. In the TTCs, the trend is the same. Furthermore, 3 TTCs are running combined programs for the first time this academic year. Our baseline is compounded by the new curriculum that is completely different – content, duration and training methods. As a result of the above factors, AED cannot meet the current Year 1 Target of 10 TEIs. We are working actively with all 19 TEIs through their subcontract activities to ensure that adequate mentoring and tutoring activities are being designed and implemented to directly address the female student academic performance. PROPOSED REVISION: Revise the language: Output 1(a) Target Year 1 to say: “None.” Output 1(a) Target Year 2 to say: “19.” Output 3 Target Year 1 to say: “Zero.” Output 3 Target Year 2 to say: “10.” Component 2. In-service Output Statements 1 and 2: Self-instructional Kit Development OUTPUTS CURRENTLY STATED: Output Statement 1 (a), requires that instructional kits be developed for Cohorts 1 and 2 regions and field-tested. Output Statement 1 (b) requires that varying numbers of copies of these kits be produced for widespread dissemination. Output 3 Years 1 and 2 Targets require 1,500 school heads/600 officers and 3,500 school heads/800 officers Output 4 Years 1 and 2 Targets require 18,000 teachers and 35,000 teachers. AED COMMENT: The initial stages of planning for Year 1 took into consideration the following realities. The Afar and Somali school years finish at the end of May. Field testing can not be done in the last month of school as teachers are testing their students. All kit materials, translated into local languages, would have had to be ready for field testing by mid-February or earlier in order to distribute the pilot kits and allow the teachers two months to use them. The Amhara and Oromia school years finish at the end of June. The above considerations also apply. It was for this reason that the field testing was initially planned to take place when the school open for the 2003/04 school year. 66 AED is proposing the following new schedule to address these timing issues. 1. Orientation should be provided to teachers in pilot schools in mid-September. 2. Field testing should take place from the beginning of October to mid-November 3. Production will begin as soon as possible after field testing is completed Regarding the numerical targets incorporated in Outputs 3 and 4 – AED believes these are intended to be cumulative numbers, not additive – for instance in Output 4 we will reach 18,000 teachers in Year 1 and an additional 17,000 teachers in Year 2, not an additional 35,000 teachers in Year 2 for a total 53,000 by the end of the Base Period. PROPOSED REVISION: Revise the language to say: Output Statement 1(a): “Number of multi-session self-instructional kits (content, print materials, audio cassettes) for in-service, continuing education for primary teachers developed, translated, and pilot kits produced.” Output Statement 1(b): “Number of multi-session self-instructional kits (content, print materials, audio cassettes) for in-service, continuing education for primary teachers field tested, revised, and produced for widespread dissemination.” Output 3 Target Year 1: “Program reaching School Heads: 1,500. Regional, zonal, woreda officers: 600.” Output 3 Target Year 2: “Program reaching School Heads: 2,000. Regional, zonal, woreda officers: 200.” Output 4 Target Year 1: “Program reaching Teachers: 18,000.” Output 4 Target Year 2: “Program reaching Teachers: 17,000.” Component 4. IRI. OUTPUTS CURRENTLY STATED: Output 1 Target Year 1 requires that 500 schools in priority zones/region are using interactive radio. Year 2 requires that 1,000 schools use interactive radio. Output Statement 2 – operational teacher support resource center activity. AED COMMENT: Output – Target Year 1 is not possible since the IRI English programs for grades 1 and 2 had to be completely revised by EMA resulting from the ICDR revisions of the English syllabi. Until these changes are completed, the lessons cannot be broadcast. Thus, the Year 1 output is not under the control of AED. 67 AED believes Output Statement 2 is mistakenly incorporated in this component since it really addresses the outputs and results of the In-service Component 2. PROPOSED REVISION: Revise the language to say: Output 1 Target in Year 1: “Revise IRI English programs to reflect recent ICDR English syllabi changes.” Output 1 Target in Year 2: “1,500.” Remove the Output 2 Statement since it is not relevant to this component. Component 5. Socially Relevant Curriculum: OUTPUTS CURRENTLY STATED: Output 1 Statement and Years 1 and 2 Targets require that AED review and revise the ICDR syllabi and curriculum. Output 2 Statement and Years 1 and 2 Targets require that AED develop modules on socially relevant topics and integrate them into the classroom curriculum. AED COMMENT: A requirement for us to revise ICDR controlled syllabi and curriculum is not within AED’s control. While we can review and recommend revisions to the syllabi and curriculum, we cannot ensure implementation of such recommendations. Likewise, AED cannot control the integration of the materials into the classroom curriculum. PROPOSED REVISION: Revise the language to say: Output Statement 1: “Existing curriculum reviewed and recommendations for revisions made based on school feedback on key socially relevant topics (reported by topic, grade, and language of instruction).” Output 1 Target Year 1: “Review G1-4 curriculum (ICDR syllabi) and recommendations made. Regional curr. assessment completed in Cohort 1 regions.” Output 1 Target Year 2: “Review G5-8 curriculum (ICDR syllabi) and recommendations made. Regional curr. assessment completed in Cohort 2 regions.” Output Statement 2: “Socially relevant topics identified and prototype modules developed and produced (reported by topic, grade level and language of module).” 68 Output 2 Target Year 1: “One module (book) each on three socially relevant topics developed and translated for G1-4 for Cohort 1 regions.” Output 2 Target Year 2: “One module (book) each on three socially relevant topics pilot tested and revised for G1-4 for Cohorts 1 and 2 regions and for G5-8 in Cohort 1 regions. 3 Smart Arts Awards granted per Cohort 1 regions.” New Output Statement 3: “Final materials produced and distributed to AED/BESO II Cluster Schools in Cohorts 1 and 2 regions.” New Output 3 Target Year 1: “None.” New Output 3 Target Year 2: ““One module (book) each on three socially relevant topics produced and disseminated for G1-4 for Cohorts 1 and 2 regions and for G5-8 in Cohort 1 regions.” Component 6. Women Support: OUTPUTS CURRENTLY STATED: Output 2 Statement suggests 3 types of media in the materials and requires that the type of media be reported. Output 2 Target Year 1 requires that 4 short courses and 2 training modules targeted at women teachers be developed, produced, and delivered. Year 2 requires that 8 short courses and 4 training modules be developed, produced, and delivered. AED COMMENT: We have recently engaged two women-owned local subcontractors to create 4 linkage activities within each of the four focus regions and link with 2 other regions. Additionally, the subcontractors will develop a total of 6 short courses with supportive materials, plus 4 stand alone training modules, however, the courses and modules will not be delivered by the subcontractors but by AED/BESO II staff. Since the development of the Year 1 materials will be completed at the end of Year 1, AED proposes that the delivery of the materials be moved into Year 2. The reasons are that: 1. We target women teachers in teacher education institutions and primary school female teachers and the schools are closed from July to mid-September. 2. Production and distribution of the course and module materials should be spread over a long period of time to allow for follow-up and maximum impact. Furthermore, distribution of print materials is within our control, however, the nature of radio and video production and program scheduling is very long-lead and quite expensive both in production and in utilization. Thus, we prefer to focus on print materials in the Base Period. PROPOSED REVISION: 69 Revise the language to say: Output 2 Statement: “Instructional and/or motivational short courses and training modules targeted at women teachers developed, produced, and delivered (reported by topic, type of media, and language).” Output 2 Target Year 1: “Short courses in development: 6. Stand alone training modules in development: 4.” Output 2 Target Year 2: “Short courses produced and distributed: 6. Stand alone training modules produced and distributed: 4.” Component 8. Planning and Management OUTPUTS CURRENTLY STATED: Output 2 Target Year 1 requires that a procurement process study be completed and a design of a commodities distribution system be completed. Target Year 2 requires that the new/modified systems be implemented in 11 regions. Output 3 Target Year 1 requires that 2 regions undertake regional learning assessments. Target Year 2 requires that 5 additional regions undertake learning assessments. One National Learning assessment is included. AED COMMENT: Output 2 – AED considers the procurement study and design of a commodities distribution system to be integrated. By the end of Year 1, the integrated study will be initiated and will be completed during Year 2. AED will then present the study and findings to the regions and will offer to provide technical support for implementation. However, AED cannot control the implementation of the modified procurement/commodities distribution system as this is the decision of the regions. Output 3 – AED recommends that the learning assessment be shifted to Component 9 – MERA. A national learning assessment will be conducted by NOE with AED/BESO support during 2003-04, and the final report will be submitted by August 2004. There will be regional representation on the National Advisory Committee and the Learning Assessment Technical Working Group. In this way, the regions will contribute to and learn from the design and implementation of the study and, thus, regional capacity in learning assessments will be strengthened. PROPOSED REVISION: Revise the language to say: Output 2 Target Year 1: “MMIS installed in 11 regional HQs. Prelim. Database set-up started in all locations. Initiate a study on the procurement process and commodities distribution systems.” 70 Output 2 - Year 2: “The recommendations for strengthening the procurement and distribution process will be provided to 11 regions for implementation.” Output 3 Statement: (Move the entire Output to MERA, Component 9.) Component 9. MERA OUTPUTS CURRENTLY STATED: Output 1 Target Year 1 requires a national consultative workshop. Output 3 requires 2 studies in each of the base years. AED COMMENT: Output 1 - During September 2002, USAID held the MERA consultative workshop and addressed the overall BESO II indicators for all partners of the program. Thus, AED believes the references to both the BESO II PMP and the first year consultative workshop requirement should be removed. Regarding MERA planning and reporting, AED assumes that these only refer to AED activities, with the exception of a few SO-level indicators. AED assumes that other USAID partners have their own MERA requirements. Thus, it may be appropriate to change the term in the output table from “MERA” to “AED/MERA” Output 2 - Regarding the annual report on AED indicators, AED understands that the first report for the year September 2002 – August 2003 is to be submitted in November 2003. Thus, the first reporting requirement is in Year 2, not Year 1. Output 3 - two studies were proposed in the Implementation Plan, one on nomadic and pastoral education and a second on dropouts with a focus on how to reduce the dropout rate; neither study has been approved by USAID/MOE. AED would like to propose that in Year 1 two studies are to be approved and initiated, PROPOSED REVISION: Revise the language to say: Output 1 Target Year 1: (Remove the phrase “Initial MERA consultative workshop”.) “AED/BESO II M&E Plan with refined indicators and targets for IR 4 to be reported in 2003.” Output 2 – Target Year 1 – (Delete target.) Output 3 Target Year 1: “2 Studies proposed, approved, and initiated.” Output 3 Target Year 2: “2 Studies completed and findings distributed.” New Output 4 Statement: “Undertake a national learning assessment and other monitoring of educational quality at the classroom and school levels.” 71 New Output 4 - Year 1: (No target.) New Output 4 - Year 2: “Complete 1 National Learning assessment including regional inputs, and provide results to regions for use in planning future regional assessments.”