USAID FROM THE AMERICAN PEOPLE FINAL PERFORMANCE EVALUATION OF THE IMPROVING QUALITY OF PRIMARY EDUCATION PROGRAM (IQPEP) FINAL REPORT DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS LIST OF TABLES AND FIGURES ............................................................................................. iv ACRONYMS.................................................................................................................................. v EXECUTIVE SUMMARY.............................................................................................................. 1 EVALUATION PURPOSE AND EVALUATION QUESTIONS .............................................. 2 Evaluation Purpose .................................................................................................................... 2 Evaluation Questions ................................................................................................................ 2 A. Approach, Inputs, and Results: ......................................................................................................................... 2 B. Program Management: ........................................................................................................................................ 3 C. Host Government Satisfaction: ........................................................................................................................ 3 D. Lessons Learned:................................................................................................................................................. 3 ACTIVITY BACKGROUND ........................................................................................................ 4 EVALUATION METHODS AND LIMITATIONS ..................................................................... 6 Evaluation Team ........................................................................................................................ 6 Evaluation Methods ................................................................................................................... 6 Limitations ................................................................................................................................. 7 FINDINGS AND CONCLUSIONS ............................................................................................. 7 A. Approach, Inputs, and Results: Findings ............................................................................. 7 Component 1: Strengthened Pre-service Teacher Education ....................................................................... 7 Component 2: Enhanced In-service Teacher Training .................................................................................. 10 Component 3: Decentralized Planning and Management ............................................................................. 16 Component 4: Improved Gender Equity and Participation .......................................................................... 18 Component 5: Monitoring, Evaluation, Research, and Analysis (MERA) .................................................. 20 A. Approach, Inputs, and Results: Conclusions .................................................................... 21 Pre-Service Training ............................................................................................................................................... 21 In-service Training .................................................................................................................................................. 22 Decentralized Planning and Management ......................................................................................................... 22 Gender ...................................................................................................................................................................... 23 Monitoring, Evaluation, Research, and Analysis (MERA)............................................................................... 24 B. Program Management: Findings ....................................................................................... 24 B. Program Management: Conclusions ................................................................................. 27 C. Host Government Satisfaction: Findings .......................................................................... 28 C. Host Government Satisfaction: Conclusions ................................................................... 29 RECOMMENDATIONS AND LESSONS LEARNED .............................................................. 29 ANNEXES ...................................................................................................................................... 1 ANNEX 1: FIELD RESEARCH SCHEDULE ................................................................................................................ 2 ANNEX 2: LIST OF PERSONS & ORGANIZATIONS INCLUDED IN THE STUDY ..................................... 3 ANNEX 3: RESEARCH METHODOLOGY MATRIX ............................................................................................ 12 ANNEX 4: DATA COLLECTION TOOLS .............................................................................................................. 13 ANNEX 5: IQPEP PMP OUTPUT INDICATORS, 2009-2014 CUMULATIVE ................................................ 48 ANNEX 6: IQPEP PMP RESULTS INDICATORS, 2009-2014 CUMULATIVE ................................................ 49 ANNEX 7: ADDITIONAL FIGURES FROM INTERVIEWS AND SURVEYS ................................................... 51 ANNEX 8: ANALYSIS MATRICES .............................................................................................................................. 73 ANNEX 9: BIBLIOGRAPHY ......................................................................................................................................... 75 ANNEX 10: STATEMENT OF WORK ...................................................................................................................... 79 iv LIST OF TABLES AND FIGURES Figure 1. IQPEP CAUSAL CHAIN................................................................................................................................. 5 Figure 2. SUMMARY OF LOCATIONS AND INFORMANTS .............................................................................. 6 Figure 3. COMPONENT 1 OUTPUTS: 2009 – 2014 ............................................................................................... 7 Figure 4. COMPONENT 1 RESULTS: 2009 – 2014 .................................................................................................. 8 Figure 5. COMPONENT 2 OUTPUTS: 2009 – 2014 ............................................................................................. 10 Figure 6. COMPONENT 2 RESULTS: 2009 – 2014 ................................................................................................ 10 Figure 7. IQPEP-PMP AND EGRA INTERVENTION SCHOOLS IN PERCENT ............................................ 12 Figure 8. IQPEP-PMP AND EGRA INTERVENTION SCHOOLS READING FLUENCY PATTERN ........ 12 Figure 9. GRADE 2 TRENDS IN MEAN SCORES .................................................................................................. 13 Figure 10. GRADE 3 TRENDS IN MEAN SCORES ................................................................................................ 13 Figure 11. ACTIVE LEARNING IN THE CLASSROOM ........................................................................................ 14 Figure 12. BARRIERS TO DOING ACTIVE LEARNING ...................................................................................... 14 Figure 13. USE OF ACTIVE LEARNING METHODS (% OF TIME) IN PRIMARY SCHOOLS.................... 15 Figure 14. CENTER COORDINATORS' PERCEPTION OF IQPEP IMPACT ON TEACHING ................ 16 Figure 15. COMPONENT 3 OUTPUTS: 2009 – 2014........................................................................................... 16 Figure 16. COMPONENT 4 RESULTS: 2009 – 2014 ............................................................................................. 18 Figure 17. COMPONENT 5 OUTPUTS: 2009 – 2014........................................................................................... 20 Figure 18. IQPEP STAFF SATISFACTION AS EMPLOYEES OF IQPEP ............................................................. 25 Figure 19. IQPEP STAFF RATING OF MANAGERIAL SUPPORT ..................................................................... 25 Figure 20. IQPEP PROVISION OF EDUCATIONAL SUPPORT TO STAFF ................................................... 26 Figure 21. IQPEP AND HUMAN RESOURCE NEEDS .......................................................................................... 26 Figure 22. IQPEP AND FINANCIAL NEEDS ........................................................................................................... 26 Figure 23. IQPEP AND MATERIAL NEEDS .............................................................................................................. 27 Figure 24. HOST GOVERNMENT SATISFACTION .............................................................................................. 28 Figure 25. PERCEPTIONS OF PROGRAM IMPACT .............................................................................................. 28 Figure 26. SUSTAINABILITY ........................................................................................................................................ 29 v ACRONYMS AA Addis Ababa ABL Activity-Based Education AED Academy for Educational Development AL Active Learning ALM Active Learning Method ANOVA Analysis of Variance AR Action Research Av Average BESO-I Basic Education System Overhaul BESO-II Basic Education Strategic Objective CA City Administration CAEB City Administration Education Bureau CDP Continuous Professional Development CoEs Center of Excellence CSPro Census and Survey Processing System CRC Cluster Resource Center CTE College of Teacher Education EFA Education for All EQUIP2 Educational Quality Improvement Program 2 EC Ethiopian Calendar EDDATA Education Data EGR Early Grade Reading EGRA Early Grade Reading Assessment EGRW Early Grade Reading and Writing ENLA Ethiopian National Learning Assessment EQUIP2 Education Quality Improvement Program 2 ESDP Education Sector Development Program ETP Education and Training Policy FCA Formative Continuous Assessment FGD Focus Group Discussion FHI360 Family Health International 360 GBV Gender-Based Violence GC Girls’ Club GEAC Girls’ Education Advisory Committee GEQIP General Education Quality Improvement Program GER Gross Enrollment Rate GPA Grade Point Average GTP Growth and Transformation Plan GU Gender Unit HDP Higher Education Diploma HQ Headquarters IL Instructional Leadership IQPEP Improving Quality of Primary Education Program IRC Instructional Resource Center IT Information Technology ITRC Information Technology Resource Center KETB Kebele Education and Training Board LCU Linkage Coordinating Unit LPS Linked Primary School vi M&E Monitoring and Evaluation MDG Millennium Development Goals MERA Monitoring, Evaluation, Research, and Analysis MIS Management Information System MOE Ministry of Education NA Not Available NER Net Enrollment Rate NGO Non-Governmental Organization NICHD National Institute of Child Health and Human Development NLA National Learning Assessment NRS National Regional State PACT Private Agencies Collaborating Together PASDEP Plan for Accelerated and Sustained Development to End Poverty PDS Professional Development Schools PIRS Performance Indicator Reference Sheet PMIS Personnel Management Information System PMP Performance Monitoring Plan PRSP Poverty Reduction Strategy Plan PS Primary School PSTA Parent Student Teacher Association PTA Parent Teacher Association RC Resource Center READ Reading for Ethiopia’s Achievement Developed REB Regional Education Bureau RH Reproductive Health RSEB Regional State Education Bureau RTI Research Triangle Institute ScHB Science Handbooks SCRC School Cluster Resource Center SDU Staff Development Units SEO Sub-city Education Office SIC School Improvement Committee SIK Self-Instruction Kits SIP School Improvement Program SMHB Subject Matter Handbooks SMC School Management Committee SNNP Southern Nations, Nationalities and Peoples SNNPR Southern Nations, Nationalities and Peoples Region SOW Scope of Work SPSS Statistical Package for Social Studies SRMIS Student Record Management Information System SS Secondary School STTA Short-term Technical Assistance TALULAR Teaching and Learning Using Locally Available Resources TOT Training of Trainers TSG Teacher Study Group TTI Teacher Training Institute TWG Technical Working Group UPE Universal Primary Education USAID United States Agency for International Development vii WCB Woreda Capacity Building WEO Woreda Education Office WCRC Woreda Cluster Resource Center WpM Words per Minute ZED Zonal Education Department 1 EXECUTIVE SUMMARY Background: Launched on August 4, 2009, and ending on August 4, 2014, the Improving Quality of Primary Education Program (IQPEP) was a five-year, country-wide program in Ethiopia. IQPEP directly supported 2,615 primary schools, 30 Colleges of Teacher Education (CTEs), 200 focus woredas (districts), and all regions and city administrations of Ethiopia to achieve two major goals: 1) improved reading proficiency in early grades and enhanced learning achievement of primary school students, and 2) improved planning, management, and monitoring of primary education. Three outputs were considered to be essential to achieve those goals: 1) strengthened pre-service teacher education; 2) enhanced in-service teacher professional development; and 3) improved decentralized educational planning and management. Improved gender equity was also an essential goal within the program, as was monitoring and evaluation of program results. Evaluation: The purpose of this performance evaluation was to gain an independent view of the program’s overall performance in order to help USAID’s Ethiopia Education Office, the Ministry of Education, and Regional State Education Bureaus make evidence-based decisions on future educational programming. This evaluation was conducted as a non-experimental, mixed methods, summative research design. Both quantitative and qualitative methods were used through semi-structured interviews, quantitative surveys, focus group discussions, classroom observations, and document and archival reviews. Locations were selected using purposive sampling of accessible IQPEP CTEs, woredas, and schools stratified by geography (urban/rural). Across seven regions of the country, 635 individual informants from all groups of stakeholders were interviewed. The following sections present the evaluators’ main findings. Pre-Service Training: The linkage school concept achieved its goal of linking the CTEs to the school setting. Linkage Coordinating units were crucial to the success of the program. Many CTE instructors became TOT (Training of Trainers) trainers for IQPEP and brought the training back to their CTE classrooms. While the student programs were well received, there is little likelihood of their being sustained. Few of the Information Research Technology Centers (ITRCs) are functioning adequately; they do not appear to be fulfilling their intended purpose. However, gender programs at the CTEs were working well in those institutions that had them. In-Service Training: IQPEP trained 80,347 teachers and 10,112 principals, as well as Woreda Education Offices (WEOs) and other educators. Despite the large number of training recipients, evaluators found little evidence of post-training follow-up; with high turnovers in all categories, sustainability is likely to be affected. Both the technical and human resource side of the Woreda Cluster Resource Centers (WCRCs) and School Cluster Resource Centers (SCRCs) were found to be not very functional, due to distance, communication difficulties, and poor or lacking leadership. Teachers and others trained by IQPEP were highly satisfied with the trainings and expressed a desire for them to continue. Reading Centers were an important addition to many schools, but had too few reading books for the size of schools. Teacher Study Groups (TSGs) were an important addition for peer-to-peer training and have good prospects of sustainability if school principals provide them with leadership. Active Learning and Continuous Assessment are helping to change the Ethiopian classrooms, if not yet to the extent found in more well equipped educational institutions. Decentralized Planning and Management: Personnel Management Information System (PMIS) at the woredas was a major success, due to its carefully planned, phased-in approach. Principals, WEOs, Regional State Education Bureaus (RSEBs), and City Administration Education Bureaus (CAEBs) received excellent training for their administrative roles and in instructional leadership. Gender: Pact/IQPEP was creative in its approach to challenging gender issues in schools and society, and was highly rated by people at all levels. Large numbers of females were trained to become principals 2 and some have already been appointed as a result. The Girls’ Education Advisory Committees (GEACs), Girls’ Clubs, Girls’ Rooms, and other activities at the CTEs and primary schools have made a real impact. Monitoring, Evaluation, Research and Analysis (MERA): MERA completed four excellent policy studies, assisted with two National Learning Assessments (NLAs), and produced 23 evaluations to monitor IQPEP. Setting high annual targets for its Performance Monitoring Plan (PMP) —many of which could not be reached—kept the IQPEP staff pushing hard to achieve their goals. Three Early Grade Reading Assessments (EGRAs) were administered, which helped make Early Grade Reading and Writing (EGRW) a national concern. Program Management: IQPEP was adequately organized and structured to meet or exceed most of its objectives. In addition to being well structured, the program was tightly managed, which is likely the reason such large numbers of teachers and others could be trained. At the regional level, program staff worked closely with the RSEBs. The IQPEP staff deserve to be commended for completing such an immense and complex project: they were clear about their responsibilities and performed them with real professionalism. Without a cost-benefit analysis the evaluators relied on staff perceptions that resources were used effectively and transparently, and that initiatives were implemented in a timely and efficient manner. EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this performance evaluation is to gain an independent view of the overall performance of the Improving Quality of Primary Education Program (IQPEP) in order to help USAID’s Ethiopia Education Office, the Ministry of Education, and the Regional State Education Bureaus make evidence￾based decisions on future educational programming. Specifically, this evaluation: 1. Identifies the major quantitative and qualitative results (input, output, and outcome) of the program in the last four years and compares those results with the deliverable targets to determine the success of the program; 2. Assesses the approach and methodology used to achieve the program objective and key results, in order to determine the effectiveness of the strategy employed; 3. Assesses the program’s management, organizational structure, staff composition, and relationships/partnerships with the Ministry of Education (MOE), Regional State Education Bureaus (RSEBs), City Administration Education Bureaus (CAEBs), Colleges of Teacher Education (CTEs), and schools in order to determine overall program management efficiency; 4. Explores the level of satisfaction on the part of the Ministry of Education, Regional State and City Administration Education Bureaus, CTEs, and schools; 5. Identifies and analyzes implementation barriers, issues, challenges, and their causes, and provides actionable recommendations; and 6. Identifies lessons learned. EVALUATION QUESTIONS The evaluation questions are drawn from the Scope of Work (SOW) and organized into four sections. A. Approach, Inputs, and Results: 1. Did the program meet established targets at all levels under the IQPEP contract, performance monitoring plan, and implementation plans? If performance has exceeded or fallen short of targets, what are the reasons (positive or negative)? 3 2. Did the activities achieve their intended results? 3. Were there over- or under-achievements? If so, what are the reasons for over- and under￾achievements? 4. Did the planning and execution of the program incorporate strategies for sustainability from the beginning? 5. How effective has the approach (strategy) been in addressing the problems? 6. Were inputs provided timely? 7. Have the trainings provided under the project been sufficient and used to transfer the relevant skills and appropriate knowledge to beneficiaries? 8. What is the level of utilization of equipment and other resources provided to the institutions and, if not utilized, what were major constraints impeding their utilization? 9. How have the initiatives undertaken by IQPEP woredas been similar or different from non￾IQPEP woredas? What has been the value added by IQPEP in the 200 woredas? Are there indicators of greater success? B. Program Management: 1. Is the contractor (previously AED and—as of the third year—FHI360) in Ethiopia adequately organized and structured to enable it to meet the objectives of the program? 2. Are there clear and appropriate delineations of the responsibilities of each staff member from the lowest to the top management level, as well as delegation of responsibilities? Do auxiliary offices (administrative, finance, etc.) give the necessary support to the program staff? What has been the effect on program implementation and the implication on anticipated results? 3. How effective has IQPEP management been in building team capacity and the ability to work towards a common objective? 4. Has IQPEP’s management, organization and use of its resources (human, financial, material, and time) been able to tackle program priorities in a timely and efficient manner? 5. Has IQPEP’s Monitoring Evaluation Research and Analysis (MERA) system been adequate and efficient for tracking inputs, ensuring that supporting processes are put into place in a timely fashion, and measuring the quality of resulting outputs and changes? C. Host Government Satisfaction: 1. To what degree has IQPEP responded to perceived needs of its beneficiaries: teachers, education managers, government partners at the school clusters, woredas, regional and national levels? Which activities are perceived as most important? Which ones are perceived as least important? Why? 2. Do regional and woreda-level education officials feel a sense of ownership of the project? 3. What is the opinion of the school directors, teachers, woreda and regional education officials about the trainings and relevance to their needs? 4. How have IQPEP activities and outputs been integrated, where appropriate, into initiatives of the Ethiopian government at CTEs, cluster centers, woredas, regional and national levels, in terms of government policy and planning? D. Lessons Learned: 1. What are the constraints that impact program implementation (policy environment, operational, institutional [i.e. within the program’s implementer, USAID, and/or host country partner institutions]) and what has been the impact on program results? 2. Have the constraints been rectified? How? How sustainable are the achievements gained? 4 3. How does project-generated technical information and/or formal research impact educational policy reform? ACTIVITY BACKGROUND Launched on August 4, 2009, and ending on August 4, 2014, IQPEP was a five-year country-wide program in Ethiopia working with the MOE, the RSEBs and CAEBs, the CTEs, the Woreda Education Offices (WEOs), schools, kebeles, and communities to build quality and equity within Ethiopia’s rapidly expanding primary education sector. IQPEP was a USAID program implemented during the first two years by the Academy for Educational Development (AED), which in 2011 merged with Family Health International (FHI) to form FHI360. Pact/Ethiopia was a subcontractor focused on the program’s work in gender equity and participation. IQPEP was a capacity-building program that focused on improving the planning and management of primary education and transforming the teaching-learning processes. The program continued some of the activities of previous USAID-funded primary education programs in Ethiopia addressing teacher development, planning and management, and gender equity: Basic Education System Overhaul (BESO I, implemented 1995–2002); Basic Education Strategic Objective (BESO II, implemented 2002–2005); Basic Education Program (BEP, implemented 2005–2007); and Educational Quality Improvement Program (EQUIP II, implemented 2008–2009). However, IQPEP also introduced new emphases within programs, most notably an emphasis on improving early primary grade students’ reading and writing proficiency. IQPEP has been closely aligned with new MOE initiatives, particularly the ministry’s General Education Quality Improvement Program (GEQIP)—then GEQIP II—the School Improvement Program (SIP), the Teacher Development Program (TDP), and the Management and Planning (MAP) programs within GEQIP and GEQIP II. IQPEP directly supported 2,615 primary schools, 30 CTEs, 200 focus woredas (districts), and all regions and city administrations of Ethiopia to achieve two major goals:  Improved reading proficiency in early grades and enhanced learning achievement of primary school students  Improved planning, management, and monitoring of primary education Three outputs were considered to be essential to achieve those goals: 1) strengthened pre-service teacher education; 2) enhanced in-service teacher professional development; and 3) improved decentralized educational planning and management. Improved gender equity was also an essential goal within the program, as was monitoring and evaluation of program results. IQPEP therefore has had five interrelated program components: Component 1: Strengthened Pre-service Teacher Education Component 2: Enhanced In-service Teacher Training Component 3: Decentralized Planning and Management Component 4: Improved Gender Equity and Participation Component 5: Monitoring, Evaluation, Research, and Analysis (MERA) The five program components were designed in an integrated manner to contribute synergistically to the achievement of IQPEP’s goals and objectives. Pre- and in-service teacher education were viewed within the program as points on a single continuum and, as such, key activities such as Early Grade Reading and Writing (EGRW) and the CTE-Linkage School Program were relevant to both pre-service and in-service teacher development. Likewise the decentralized planning and management component related equally to pre-service and in-service teacher development insofar as improving the education management system—the environment in which teachers and principals work—affects both teachers-in- training and teachers who are already deployed in schools, as well as their supervisors. Similarly, issues of gender, equity, and participation resonated throughout pre-service and in-service teacher education, as well as planning and management, and hence the inter-relatedness of that component with the first three program components was manifest. Finally, while IQ PEP's comprehensive PMP focused on the first four components of the program, MERA cut across the entire program. The aim was for each program compo nent to be seen as a sub-strategy contributing to the overall strategic vision that guided IQ PEP as a whole. The following chart was constructed by the evaluators to illustrat e the causal chain implicit in the IQ PEP program design. The links between inputs, outputs, outcomes, and impacts represent the assumptio ns behind t he program model, and the primary focus when testing the model's effectiveness. FIGURE I. IQPEP CAUSAL CHAIN COMPONENT I: Strengthened Pre-Service Teacher Education 2: Enhanced In￾Service Teacher Training 3: Decentralized Planning and Management 4: Improved Gender Equity and Participation S: Monitoring. Evaluation. Research, and Analysis (MERA) Training Linkages Resource Centers Publications Training Resource Centers Publications Training Resources Equipment Training Publications Peer-to-Peer Support Policy Studies National Leaming Assessments OUTPUTS Numbers of: functioning ITRCs; functioning LCUs in CTEs; teachers with face-to-face training related to instruction (Math, EGR. Instructional Kits, and ScHBs); principals and deputy principals with face-to-face training related to instruction and instructional leadership; CTE instructors with training. Numbers of: school teachers completing cluster-based training programs; principals and RSEBICAEB officers who completed face-to￾face training related to instruction and instructional leadership; WCRCs and SCRCs established; SCRC supervisors trained. Numbers of: principals. MOE officers, regional education/city administration officers, WEO officers, and KETB members trained in planning and management; number of PMIS packages established at WEOs. Numbers of: GEACs and GCs established; supplementary gender materials distributed; GBV, RH, HIV/AIDS, life skills. and study skills manuals translated and distributed; training for CTE Gender Units; training on educational leadership and management for female teachers; policy studies. Numbers of: policy studies conducted, written up, and disseminated; National Leaming Assessments supported such as EGRA. OUTCOMES New teaching and school management practices are adopted and institutionalized. New resources and expertise are accessed by teachers; new instructional practices and materials are adopted and implemented in the classroom. New planning and management practices are implemented across multiple levels of the education system; new equipment is used and maintained. New resources and support are accessed by girls, teachers, and administrators in CTEs and schools; new practices in teaching and management that address gender issues are implemented in CTEs and schools. New knowledge and information are disseminated and subsequently influence educational policy making and system reform. IMPACTS Result I Result 3 Result 4 Result S Result I Result 3 Result 4 Result S Result 2 Result 4 Result I Result 3 Result S Result 6 Result 2 Result I: Improved learning in primary schools and CTEs; Result 2: Improved planning, management, and monitoring of primary education at various levels of the education system; Result 3: Improved professional capability of teachers; Result 4: Strengthened capacity of CTEs and primary schools; Result 5: Improved retention in primary schools; Result 6: Successfully addressing gender issues. 5 EVALUATION METHODS AND LIMITATIONS EVALUATION TEAM AMEX International, Inc. fielded an evaluation team comprised of Team Leader, Richard Kraft; Evaluation Specialist, Andrew Epstein; Teacher Education Specialist, Mamo Mengesha; and Planning and Management Specialist, Tilaye Kassahun. Habtamu Tabor, Melaku Mengistu, and Lissan Gebrewold served as Evaluation Associates. EVALUATION METHODS This evaluation was conducted as a non-experimental, mixed methods, summative research design. Both quantitative and qualitative methods were used through semi-structured interviews, quantitative surveys, focus group discussions, classroom observations, and document and archival reviews. An inception plan and draft data collection tools were submitted to USAID Ethiopia on June 16, and subsequently revised and approved for use by the USAID Deputy Education Director on June 18. Data collection was conducted from June 18 to July 14 in Addis Ababa and six regions throughout Ethiopia including Oromia, Amhara, Tigray, Afar, SNNPR, and Beneshangul Gumuz. The field work schedule and list of participants are detailed in Annexes I and 2 respectively. Locations were selected using purposive sampling of accessible IQPEP CTEs, woredas, and schools stratified by geography (urban/rural). A total of 635 individual informants were interviewed, including: IQPEP HQ and regional staff; MOE, regional, and woreda Officials; CTE officials and faculty; primary school principals and school leaders; teachers; students; PT As; school boards; and kebeles. Informants were selected based on who was available. The following table summarizes the number of locations and informants from which data was collected for this evaluation: FIGURE 2. SUMMARY O F LOCATIO N S AND INFO RMANTS Locatio ns Number Regions 7 Woredas 8 CT Es 6 Schools 26 Stakeho lders Number National & Regional MOE Officials 40 IQPEP Staff 38 Woreda Education Officials 4 1 CTE Officials and Staff 36 School Principals and Head Teachers 73 Resource Center Coordinators 36 Teachers 182 Parents/Community Members 54 Students 135 Ten different data collection tools were developed for the purpose of this summative evaluation. Copies of each tool are presented in Annex 4: I. MOE/Regional Education Official Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from officials working for the Ethiopia Ministry of Education, regional state education bureaus, and city administration education bureaus. 2. Woreda Education Official Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from officials working for woreda education offices. 3. College of Teacher Education (CTE) Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from CTE administrators and instructors. 6 7 4. School Leader/Principal Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from school principals, head teachers, and deputy head teachers. 5. Cluster Resource Center/Pedagogical Center/Reading Center/Science Center Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from coordinators of reading centers and cluster resource centers. 6. IQPEP Staff Interview Protocol. This tool was designed as a semi-structured interview to collect both quantitative and qualitative data from IQPEP staff at the HQ and regional levels in all departments. 7. Teacher Survey. This tool was designed as a structured survey to collect quantitative data from teachers who work in IQPEP-supported schools. 8. Teacher Focus Group Discussion Protocol. This tool was designed to collect qualitative data from group discussions among teachers who attend IQPEP-supported schools. 9. Student Focus Group Discussion Protocol. This tool was designed to collect qualitative data from group discussions among students who attend IQPEP-supported schools. 10. Kebele/PTA Focus Group Discussion Protocol. This tool was designed to collect qualitative data from group discussions among parents and other community members who participate in kebeles or PTAs. Analysis methods included frequency distributions of semi-structured interviews and surveys; qualitative coding and content analysis of field notes and interview transcripts; triangulation of evaluator collected data with existing Management Information System (MIS) and assessment data. LIMITATIONS An evaluation team was contracted in early June 2014 and the team began its work on June 13. With classes already over and most schools into their final examinations, the team moved rapidly into the field on June 18 to test its various protocols for interviews and focus groups in schools in Addis Ababa. Conducting an educational performance review in the absence of classrooms to observe proved challenging. IQPEP regional staff helped the team locate teachers, principals, and children, thus allowing the evaluators to gather sufficient data; the team conducted 687 interviews with participants and stakeholders, either individually or in small groups. The team separated into two groups; each group traveled a total of nearly 6,000 kilometers, over the course of twenty days, to visit widespread and often remote schools and woredas in six regions and the Addis Ababa City Administration. FINDINGS AND CONCLUSIONS This section is organized into subsections representing the four evaluation issues outlined in the SOW. Findings and conclusions are provided for each issue section. A separate section containing recommendations and lessons learned follows. A. APPROACH, INPUTS, AND RESULTS: FINDINGS Component 1: Strengthened Pre-service Teacher Education FIGURE 3. COMPONENT 1 OUTPUTS: 2009–2014 Output # Indicator Target Actual 2.1a Percent of functioning ITRCs 100% 22.2% 2.1b Percent of functioning LCUs in CTEs 100% 67.6% 2.1c Number of linkage primary school teachers who received and completed a face-to-face training related to instruction (Math, EGR, Instructional Kits and ScHBs) 14,119 16,785 2.1d Number of linkage primary school principals and deputy principals who received and completed face-to-face training related to instruction and instructional leadership 2,000 1,809 8 2.1e Number of CTE instructors trained 2,066 1,585 FIGURE 4. COMPONENT 1 RESULTS: 2009–2014 Result # Baseline Target Actual 1.4c Percent of primary schools with functioning Teacher Study Groups 15.6% 100% 65.0% Within this component, IQPEP sought to build the capacity of the CTEs and linkage schools as learning institutions. This involved multiple activities over the life of the contract including building the professional capacities of the CTE administrators, instructors, and students in their subject matter knowledge and pedagogical skills, particularly as they related to the linkage schools with which they were connected. The particular projects promoted were based on an assessment of needs, conducted jointly by the CTE and IQPEP staffs. The concept of bringing together local schools and teacher training is an excellent idea and is based on the Professional Development School (PDS) model used in many countries today. The Linkage Coordinating Units (LCUs) at the CTEs were critically important in attempting to tie PDS theory with practice. In 67.6% of the CTEs the LCUs were successfully structured—given their independent nature in Ethiopia, the adoption of the LCUs by two-thirds of the CTEs can be considered evidence of some level of success, even though IQPEP did not reach the 100% target. The training and leadership abilities of the LCU coordinators appear to be the key factors in the success of this approach to developing both the commitment of the CTE and successfully working with the linkage schools. Regrettably, however, announcing that a school is a linkage school does not guarantee that its teachers, learning activities, management, and other components are of sufficiently high quality to serve as an example for others to follow. In other countries, PDSs have sought to bring together the best teaching staffs and top principals, and also serve as an observable model for student teachers, in-service teachers, and principals. Unfortunately, this evaluation found little evidence of this practice in Ethiopia. The IQPEP funding for the Information Technology Resource Centers (ITRCs) and CTE Centers of Excellence (CoEs) was refocused after two years during a mid-course shift in priorities. Trainings in the second year in E-Lesson Development, Educational Technology and Student Record Management and Information System (SRMIS) were conducted and IT equipment was delivered to nine newly established ITRCs. While some training, equipment and materials were still provided to existing centers, only 26% of the CTEs have seen fit to continue maintaining the ITRCs without outside funding. The ITRCs that remain do not appear to be functioning at a high level, although the CoEs in special needs education (Sebata) and in publishing (Debre Berhan) appear to be meeting real needs in the system, despite the mid-term decision in IQPEP to discontinue any regular funding for them. In its final study of ITRCs, MERA found them to be inadequate on almost all measures, including lack of infrastructure, materials, and activities for students. Visits to ITRCs during the evaluation found them to be seriously lacking in working computers and poorly maintained; also, the visits revealed little evidence that CTE students or faculty made significant use of them. Although not all ITRCs were visited, the evaluation team felt confident in concluding that ITRCs had not served the CTEs well, and that there was no real commitment on the part of the CTEs to embed them into the institution or maintain them. Similar to the LCUs, the Staff Development Units (SDUs) were tasked with providing needs-based training for 8,429 CTE instructors and 133 deans and coordinators. Deans and other CTE administrators were trained in transformational leadership, strategic planning, project design and implementation, management, and monitoring and evaluation (i.e. topics similar to those offered to MOE, RESB, and Woreda officers). College deans spoke highly of the planning and 9 management components of the training; many stated that it was the only training they had received as CTE administrators. CTEs in Ethiopia and throughout most of Africa are generally recognized as maintaining the status quo—institutions and their deans are seldom known for their transformational leadership. During the mid-term redesigning of IQPEP’s CTE interventions, activities focusing on students were recommended and trainings on guidance and counseling, life skills, educational technology, peer study groups, and library usage were offered to 11,676 students. IQPEP was also instrumental in supporting student research and action research working with Linked Primary Schools (LPSs). In IQPEP’s final year, 2,093 students who were low performers in chemistry, math and physics received eight hours of tutorials in the sciences. These various trainings were generally well received by students, and requests were made to continue them. Unless these topics and experiences are mainstreamed into the CTE curriculum, there is little indication that they will continue once funding ends. As stated above, the 400 linkage schools connected to the 30 CTEs were an important component of IQPEP. Given the mostly theoretical nature of the curriculum at the CTEs (this is true of teacher training in many countries) the development of linkages was vitally dependent on the LCUs, and usually on the appointment of a trained coordinator. The key development in the linkage schools themselves was the development of Teacher Study Groups (TSGs). These TSGs were modeled, indirectly, on the successful Japanese model of Lesson Study, in which teachers meeting biweekly exchange subject matter knowledge and pedagogical methods, while more experienced teachers work with student teachers and those new to the profession. From a baseline of only 15.6%, the percentage of IQPEP-assisted schools with TSGs increased to 65% over the course of the project. While Lesson Study has been practiced in Japan for many years, it has taken decades for other countries to successfully adopt the model. If CTEs enabled Ethiopia to embed TSGs in all schools, it would be one of the first countries in Africa to do so. TSGs—as assisted by IQPEP—contribute to teachers’ professional development through the provision of self-instructional kits, subject-matter-specific supplementary teachers’ handbooks, early reading modules, and other instructional materials. TSGs are organized around subject matter or pedagogical issues as appropriate and meet bi-weekly for two hours after school at the convenience of group members. IQPEP further supported the TGSs by providing small grants to cover miscellaneous costs. Questions of sustainability have to do with whether Ethiopian primary teachers have the knowledge, skills, and commitment to keep such a program going without externally provided instructional materials, i.e. with nothing but internal incentives to keep them going. EGRW was added to the IQPEP program in its second year, and included the establishment of reading centers in all LPSs, books and writing materials, and supervisory visits to LPS centers. IQPEP did not achieve its target of reaching 85% of the schools; centers were only established in 46% of the schools between years three and five of the project. The importance of the Reading Centers and their emphasis on Early Grade Reading (EGR) is recognized by USAID as a top priority. IQPEP became the initial vehicle for making EGR a major focus of the Ethiopian educational system. Results from the first (2009– 10) to the second (2012–13) Early Grade Reading Assessment (EGRA) were encouraging in that they rose, although the scores of proficiency were low in both second and third grades. The results on the third iteration of the test, 2014, were encouraging in that intervention schools retained most of the gains from the EGRA assessment of 2012–13, although they still fell below their set goals. The intervention schools also performed at or above comparison schools in all seven testing areas. Evaluation team visits to schools confirmed reports of IQPEP schools rapidly changing how they were dealing with the low proficiency levels, even though these were not yet showing up on EGRA results. Via the Reading for Ethiopia’s Achievement Developed (READ) project begun in 2013 and extending for five years, USAID has prioritized and allocated resources to EGRW for the foreseeable future. 10 IQPEP invested significant effort on the printing, reprinting, and translation of training materials, including subject matter handbooks (16,479), self-instructional kits (79,128), EGRW modules (27,385), instructional leadership modules (4,927), CTE instructors’ capacity-building modules (4,927), and CTE management-capacity-building modules (930). These items were produced after carrying out needs assessments; both the materials and the face-to-face trainings in which most of them were used were well received. In reviewing most of the documents, evaluators found them to be generally well-written documents, often authored by national and/or international experts. Active Learning (AL) and Continuous Assessment (CA) were two of the major pedagogical thrusts of IQPEP, not only in the CTEs, but in the linkage, cluster, and satellite schools. The evaluation team found considerable evidence that teachers in IQPEP funded-schools had adopted many aspects of AL and CA as part of their pedagogical repertoire. Component 2: Enhanced In-service Teacher Training FIGURE 5. COMPONENT 2 OUTPUTS: 2009 – 2014 Output # Indicator Target Actual 2.2a Number of (cluster/satellite) primary school teachers completing cluster￾based training programs 37,600 80,347 2.2b Number of primary principals who received and completed a package of training through face-to-face training related to instruction and instructional leadership 4,646 10,112 2.2c Number of WEO Officers trained in instruction and instructional leadership 400 774 2.2d Number of RSEB/CAEB officers trained in instruction and instructional leadership 116 162 2.2e Number of WCRCs established and strengthened 314 65 2.2f Number of SCRCs equipped and strengthened 443 255 2.2g Number of SCRC supervisors trained 443 445 FIGURE 6. COMPONENT 2 RESULTS: 2009 – 2014 Result # Indicator Target Actual 1.1a % of grade 2 students in USAID-supported primary schools who are proficient in reading (in medium of instruction) 35% 3.2% 1.1b % of grade 3 students in USAID-supported primary schools (CTE Linkage and cluster/satellite who are proficient in reading; in medium of instruction) 50% 11.5% 1.3b Average % of teaching time using active learning methods in USAID (CTE linkage/cluster/satellite) primary schools 85% 72.1% 1.3c Average percent of primary school teachers using formative continuous assessment 85% 74.5% 1.4a Percent of functioning SCRCs 80% 48.8% 1.4b Percent of primary schools with functioning Reading Centers 85% 46.0% 1.5a School Survival Rate to grade five 65% N/A 1.5b Total # of students enrolled in USAID-assisted primary schools N/A N/A Under Component 2 come all the trainings and materials offered to teachers already in the schools. As with Component 1 however, there is overlap between pre-service and in-service. Linkage school teachers fit into this overlapping category, as they received much of their training from CTE instructors, who had themselves been trained as TOTs by IQPEP staff. These CTE TOTs (452) offered training to linkage/cluster/satellite school teachers on science, EGRW (four modules), formative CA, and Self￾Instruction Kits (SIKs), with other training for principals, department heads, WEOs, and other administrators on capacity building and Instructional Leadership (IL). In its first year, IQPEP initiated a large-scale materials development project, developing three new SIKs—Action Research (AR), TALULAR (Teaching and Learning Using Locally Available Resources), and 11 Understanding and Managing Student Behavior)—along with an instructional leadership module and, in the second cycle, science manuals in the three sciences. After validation and field testing, the materials were translated and became part of the training program. EGRW modules were also written and translated. Throughout the five years of the program, previous and newly developed Subject Matter Handbooks (SMHBs), SIKs, and leadership modules were printed, used for face-to-face training and TSG meetings, and distributed to schools, WCRCs, WEOs, Zonal Education Department (ZEDs), and RSEBs. This is another example of sustainability, in that tens of thousands of documents in English and mother tongues are now in schools and classrooms throughout the CTEs and woredas. Whether the MOE/GEQIP II will make use of them or republish them for the rest of Ethiopian schools remains to be seen. Cluster-based training was used to bring a package of training through a cluster approach to CTE￾related linkage/cluster and satellite schools. It was aimed at 37,600 teachers, but 80,347 teachers actually went through at least some of the training modules. These trainings involved SIKs on active learning methods for grades 1–4 and 5–8. Sample learning activities for grades 5–8 covered topics such as CA, gender issues, managing large classes to promote active learning, managing student behavior, TALULAR, and AR; math and science handbooks were also included. Another large group to receive training were the primary school principals, who completed face-to-face training related to instruction and instructional leadership. WEO officers were given this same package, as were a number of RSEB/CAEB officers. As stated earlier, the importance of instructional leadership by principals cannot be overstated: they are the key individuals in the reform of primary education in Ethiopia. The evaluation team heard many of them testify that these trainings were “the first training I have ever had in how to properly supervise my teachers on their instruction,” and “for the first time I have learned how to lead through involving my teachers, rather than doing things dictatorially.” WEOs were another group targeted by IQPEP; their roles included supervising principals and visiting and supporting schools in their woredas. Most had been promoted from principal or directly from the classroom, without any experience or instruction on the important supporting or supervisory roles they play. With their involvement extending beyond instructional leadership into most of the same academic and classroom trainings that teachers and principals attended, they came to understand, for the first time, how to carry out their responsibilities. Also for the first time, both principals and WEOs were on the same page as their teachers. This made educational reform much more attainable, as most of the front-line administrators no longer opposed such things as AL and CA. IQPEP again exceeded its targets, as WEOs recognized the value of attending the workshops. RSEBs and CAEBs were also targeted, with 162 being trained. Although these administrators were farther from the classroom, their support of the reforms was greatly enhanced by going through the same actual training as those they supervised farther down the pyramid. As stated earlier, potentially one of the most valuable contributions of IQPEP to Ethiopian education has been its success in preparing a large number of TOTs. A total of 104 trainers participated in national TOTs, who in turn trained 5,880 regional TOTs to conduct training through the School Cluster Resource Centers (SCRCs), and for principals at the regional level. National TOTs were drawn from many groups including CTE instructors, RSEBs/CAEBs, ZEDs, curriculum experts; at the regional level they were drawn for ZEDs, WEOs, SCRC supervisors, and key teachers. This model is exemplary in the fact that it drew talent from many sources, and thus cut across all levels of the educational system. Perhaps for the first time, classroom teachers became part of the leadership group. National TOTs conducted face-to-face training on EGRW, while regional TOTs conducted trainings for teachers in SIKs and for math and science. These were all part of the 80,347 teachers trained. It is often said that one of the best ways to master a subject is to teach it. Undoubtedly, this large group of trainers now has a much greater mastery of their subjects and of pedagogy than if they were passive recipients. 12 The impact of IQPEP training on EGRA had a positive effect as EGRA scores rose between the first and second iterations, and—as can be seen in Figures 7 and 8—those impacts were generally retained in the third iteration in 2014. In addition, intervention schools, as seen in Figures 9 and 10, outperformed the comparison schools in almost all EGRA tests. The positive effects of the EGRA training conducted by IQPEP were not as great as might have been hoped, perhaps partially due to the fact that in three of the schools visited by the evaluation team not a single EGRW-trained teacher remained. In eight other schools more than one trained grade 1–3 teacher had left—due either to promotions within the civil service but outside of education, or through promotions to higher grades—leaving the newest and untrained teachers to carry out the much harder task of early grade literacy training. This challenge of teacher turnover and of teacher promotion is a problem that the new Reading for Ethiopia’s Achievement Developed (READ) program must face. FIGURE 7. IQPEP-PMP AND EGRA INTERVENTION SCHOOLS IN PERCENT FIGURE 8. IQPEP-PMP AND EGRA INTERVENTION SCHOOLS READING FLUENCY PATTERN 13 FIGURE 9. GRADE 2 TRENDS IN MEAN SCORES FIGURE 10. GRADE 3 TRENDS IN MEAN SCORES AL was a major focus of IQPEP in-service education. Activity-Based Education (ABL) has a long history in the developed world. AL emphasizes: student-centered learning; student choice in learning activities; learning corners in math, science, reading, and social sciences; children’s leadership activities in the classroom and school; cooperative and small group learning; individualized learning; mastery learning; continuous assessment to track achievement in each subject on a daily basis; the use of mathematics manipulatives; student writing and art work posted around the room; flexible promotion in recognition of the fact that children learn different subjects at their own pace; a strong emphasis on reading age￾appropriate books in the mother tongue; children’s writing and production of books; and a range of other educational innovations that characterize good early childhood classrooms around the world. This approach has also been instituted in countless other classrooms in poorer countries and states such as the Escuela Nueva of Colombia; ABL in Tamil Nadu, India; New Bilingual schools of Guatemala; and Breakthrough to Literacy throughout Southern and Eastern African countries. While AL can and does have some of these same components, in the Ethiopian setting it predominantly involves children working or sitting in groups, and the teacher asking questions that students respond to. Any activity, even mental activity, can be part of AL, but the AL practiced in the Ethiopian classroom is not as full as that found in most ABL settings. Initially, the evaluation team felt that little true activity was going on; after delving deeper the team concluded that the moves made by AL in Ethiopia are possibly a first step in breaking down the Chalk Talk, copying off the board, and students being nothing but passive learners which has dominated the system for generations. Also, as the classrooms of Ethiopia become fully resourced with children’s books, science materials, paper and writing materials, art supplies, math manipulatives, and games, and as class sizes drop to 40 or less, it is likely that Ethiopian primary teachers will be able to move towards a full ABL program. FIGURE I I. A CTIVE LEARNING IN THE CLASSROOM 140 120 100 80 60 40 20 0 How often do you use active learning in the classroom, teachers (n= 180) Every day A few times a week A few t imes a month A few times a year Almost never FIGURE 12. BARRIERS TO DOING ACTIVE LEARNING 140 120 100 80 60 40 20 0 Barriers to Doing Active Learning in the Classroom, Teachers (n= 173) The figu res above indicate that while most teachers report doing AL in their classrooms every day, all reported multiple challenges to doing so. The most common reported challenges were too many pupils, not enough materials, or poor fac ilities. Evaluation interviews revealed that knowledge about AL was 14 15 either limited, or—in many cases—quite good but lacking the conditions and support to practice AL and CA in the classroom. FIGURE 13. USE OF ACTIVE LEARNING (AL) METHODS (% OF TIME) IN PRIMARY SCHOOLS Type of Assessment Biology Chemistry Physics Mathematics Mother Tongue Aggregate Average Active Learning (AL) methods in primary schools 69.0% 68.9% 68.9% 74.3% 74.6% 72.1% Source: MERA (2014) A Progress Report on Indicators of Use of Active Learning Methods and Formative Continuous Assessment in USAID-assisted Primary Schools. Addis Ababa: IQPEP, p. 12. One of the encouraging signs of AL was the visible presence of many science teachers who had taken IQPEP science trainings, TALULAR, and who were using the science handbooks for ideas on how to activate their students—despite having no laboratories, no chemicals, and no equipment, the science teachers were moving beyond those lessons into new and creative ways to teach science. The IQPEP science kits added somewhat to activating these classrooms, though evaluators did observe that some kits had never been opened or used. It is impossible to say in either the context of Ethiopian AL, or the broader ABL definitions from abroad, what is an appropriate percentage of time given over to active learning. That IQPEP schools in Ethiopia have moved beyond the old model is sufficient for now. Perhaps rather than claiming that 85% of the time is spent on AL, a better description would be “time-on-task,” indicating that students are meaningfully involved in learning tasks at all times. Such tasks may be individual or small- or large-group, and may have components of mental, affective or physical involvement. The second most common words in our interviews, after “active learning,” were “formative continuous assessment (FCA).” After initially concluding that the practice was not fully in line with true continuous assessment, the evaluators agreed that it is a positive step in the right direction. The introduction of a variety of monitoring and supervision tools will likely continue this positive trend towards continuous assessment of what children are learning on a regular, even daily basis and using a wide range of written and other measures. TSGs were discussed in the pre-service section, but an important element of the TSGs was the 2,094 key classroom teachers who received special training at the regional level to get TSGs functioning in their schools. This was in addition to a similar number of principals who received training on how to start and maintain TSGs at the local school level. Each school received a minimum of three copies of all printed SIKs, modules and SMHBs, in addition to one copy for each trainee to use when conducting their TSGs. As teachers become better educated and have more practice with peer-to-peer training, such extra materials may not be as necessary; teachers that were interviewed, however, stated that these items were indispensable when they learned the process. A major intervention to try to improve on low proficiency levels in reading was the development of Reading Centers. Schools were provided with supplementary storybooks, mobile library shelves, personal blackboards, alphabet sorts, chalk and stationary. This was done in all 2,215 schools directly aided by IQPEP and 400 LPSs. Significant gains were made from 0% Reading Centers in year 3 to 46% in year 5, although the target of 100% proved too ambitious. Though a welcome addition to any school, the Reading Centers appeared to be too few to have a significant effect on EGRA reading scores. Some centers visited by the team had only 30 small storybooks for up to 1,000 students in grades 1–3. Early grade classes could come to the center for one period per week. Another innovation of IQPEP was to organize and equip School Cluster Resource Centers (SCRCs). Via IQPEP, 443 centers were established, with coordinators trained on IL and SIKs. These coordinators were to check on the level of transfer of training in all the cluster and satellite schools, and assist those teachers having difficulties. Once again the target of centers was not reached, but from a 0% baseline, 48% of the clusters were found to have functioning centers equipped with IT and other equipment such as photocopiers. It was the perception of the evaluation team that- although the schools in which the centers were located received benefits-distance and communication difficulties made the SCRCs not terribly useful to other schools in the cluster, particularly in rural areas. This was found in 12 different schools visited by the evaluation team. WCRCs amounted to a total of 157, with the additional SCRCs making up the difference in regions that decided not to have WCRCs. They were equipped similarly to the SCRCs with a desk computer, HP Laser Jet printer, and photocopier, in addition to necessary materials to serve the machines. It was the evaluation team's observation that these also didn't function well for several reasons: teachers had little access to the equipment; maintenance was poor; there was a turnover of trained personnel; and there was a lack of separate rooms for the WCRCs, where teachers might come to develop lesson material. These centers did not appear to be a priority of the woredas, as only 19% had functioning WCRCs, compared to a goal of I 00%. FIGURE 14. CENTER COORDINATO RS' PERCEPTIO N O F IQ PEP IMPACT O N TEACHING 25 20 IS 10 s 0 Pe r ceptions of Impact on Prima ry School Teaching, Center Coordinators (n=36) A lot of Medium A little No Impact Don't impact impact impact know Center Coordinators were very enthusiastic about IQPEP inputs and their impact on teaching. Findings from this study indicate however that the schools in which centers are located benefit, but that the benefits do not extend throughout the network of schools which the centers are supposed to serve. Principals and teachers in every "non-center" school indicated that they did not receive the same benefits as the schools where the centers were located. Furthermore, EGRA results show that students in schools where CRCs are located score higher in reading. Compone nt 3: Decent ra lized Pla nning and Manage m ent FIGURE 15. COMPO NENT 3 OUTPUTS: 2009 - 20 14 Out put# Indicator Target Actual 2.3a Number of school principals trained 10,000 10,11 2 2.3b Number of MOE officers trained 54 21 2.3c Number of regional education/city administration officers trained 432 399 2.3d Number of woreda education officers trained 5,000 5,0 15 16 17 2.3e Number of Kebele Education and Training Board members trained 10,000 10,017 2.3f Number of WEOs that have established PMIS with support of USAID 200 200 IQPEP made significant contributions to the decentralization of Ethiopian education, and could become a model through GEQIP II, for the rest of the nation. Prior to IQPEP’s intervention in the training of WEOs, these critically important education officials had received very limited training for their roles. School Improvement Programs (SIPs) were mandated for all schools and woredas, but very little had been done to bring them about. Many RESB/CAEB officials had had little or no training, and often minimal management or supervisory experience before being thrust into their larger roles in the system. Kebele Education and Training Board (KETB) members had had little technical support in fulfilling their community educational mandates, and principals were too often promoted from the classroom, with little or no training in planning and management, let alone instructional leadership. IQPEP was mandated to offer a complete package through face-to-face training of 5,000 woreda education officers. From a baseline of only 5%, it managed to train 5,015, delivered in 13 training modules, on topics identified through a needs assessment. Trainings were over an eight-day period and conducted by trainers from their region. The modules were translated in Afan Oromo, Tigrigna, and Somali. The original modules had been developed under one of the previous projects, and their use under IQPEP is evidence that well done materials have a shelf-life greater than any given project. A total of 67 TOTs participated in a TOT workshop before delivering them to the over five thousand WEOs. Even though IQPEP reached its numerical goal for training 5,000 woreda officials, 67.3% of the woredas produced adequate annual plan documents. Along with WEOs, capacity building directed at school principals was a major key to the success of IQPEP. The he program managed to meet its ambitious target of 10,000. As with all IQPEP activities, training participants were disaggregated by gender, in order to assure as large a population of women participants as possible. IQPEP took 12 modules that had been developed under the EQUIP2 predecessor project, which were reviewed, revised, translated and validated before being put to use in the training of principals. The addition of TOTs, in this case for principals, again formed a major contribution of IQPEP to possible future programs for GEQIP II with 66 TOTs involved in the training of fellow principals. One of the critical issues again raised by meeting the deliverable of training 10,000 principals was the pressure of the contract to meet high target numbers. Utilizing materials that had been used by a previous project likely contributed to meeting this goal. IQPEP also met its deliverable of training 10,000 KETB members from at least 2,000 kebeles. Materials were reviewed, revised, validated and translated into Ormiffa, Tigrigna, and Somali before being printed and distributed for training KETB members. As with other mass trainings by IQPEP, these appear to have been well received by the recipients. An impressive aspect of IQPEP’s decentralization activities was the positive working relationship between the RESB/CAEB staff and the IQPEP regional staffs. They appeared to work closely together on areas of mutually defined need. Modules were prepared to build skills in planning, management, policy￾making and analysis, conflict resolution, proposal writing, project preparation and implementation, and transformative leadership. There were also modules on communication, team building, gender mainstreaming, and budgeting, as well as appreciative inquiry with ideas and tools for positive thinking and assertiveness. National experts, many from Addis Ababa University, led five-day trainings for mid￾level managers and seven-day trainings for senior-level professionals. Eventually, a total of 399 of a targeted 432 RESB/CAEB officials were trained. One of the most noteworthy aspects of IQPEP was the establishment of a computerized Personnel Management Information System (PMIS) in 200 WEOs, and providing training and facilities for the same 18 woredas. IQPEP was able to establish functioning systems in all of the participating woredas. It was not just a hardware project, however, as it involved modifying software, user guides, and data collection manuals, and then delivering the package in a carefully phased manner throughout the five years of the project. It was this phased approach to the project that likely led to the high success rate of PMIS. In every region, from one to 13 woredas were brought online each year, and lessons learned in previous years were dealt with before the next set of woredas received their hardware and training. Functional features training was conducted for 564 officers in the focus woredas, and at the end of the training the computers, printers, UPSs, and other materials were handed over to the woredas for implementation. Trainings were practical and hands-on with careful attention paid to each trainee’s level of accomplishment. The MERA evaluations were equally practical, with PMIS staff being asked to actually perform certain functions. In addition to the basic training, PMIS staff held rehabilitation training on new or modified technical features and new software. This follow-up training was one of the few instances found by the evaluation team in which central and/or regional staff were involved in follow-up training, lending again a possible answer to the broader question of whether one-off, broad training is sufficient, or whether it is better to start small, grow and adapt, and then follow up on a regular basis. Granted, it is perhaps easier to program such an in-depth approach when there are inherent needs and payoffs such as those in MIS than in more abstract areas such as AL, but that does not necessarily negate the need for follow-up training or supervision of any programmatic component. At the end of the start-up process, woreda officials knew—for the first time—the education, training, salary, and posting of all its employees. A Student Record Management Information System (SRMIS) was an activity to improve the accuracy and efficiency of producing college (CTE) student academic reports. Training was provided for 32 registrars in the eight newly established CTEs and two additional colleges. Field visits were made by IQPEP IT staff to assist in installing and implementing new hardware and software. IQPEP also provided limited commodities to RSEBs/CAEBs and the MOE. The turnover problems in all parts of the educational system also plagued the PMIS and SRMIS programs, as it led to time-consuming and costly training of new staff. Some of this could have been dealt with, if the system rewarded trained people for remaining in place, but too often people with technical skills were hired away by other GOE departments or even within the educational system. As with most programs that involve expensive hardware, the MIS programs too often suffered from poor maintenance, intermittent electricity, insecurity of hardware, and delays in obtaining replacement parts. Overall, however, the PMIS is a program that can be replicated across the educational system, and as with all aspects of IQPEP, there is now a cadre of trainers to assist in that process. Component 4: Improved Gender Equity and Participation FIGURE 16. COMPONENT 4 RESULTS: 2009 – 2014 Result # Indicator Baseline Year IV or V Actual 1.6a Percentage of girls in USAID-assisted primary schools 48.5% IV 48.6% 1.6b Average grade promotion rate of female pupils in USAID-supported primary schools (grades 1–7) 80.4% IV 77.7% 1.6c Grade point average of female students in USAID-assisted CTEs 2.4 V 2.5 1.6d Percent of primary schools with functioning Girls’ Education Advisory Committees (GEACs) in USAID-supported primary schools 14.6% V 47.0% 1.6e Number of CTEs with functioning Girls’ Clubs 0 V 7 1.6f Number of CTEs with functioning Gender Units NBL V 11 1.6g Number of female teachers trained in management and leadership NBL V 4,795 The Gender Equity and Participation component was implemented through a partnership with Pact/Ethiopia. As can be seen in Figure 16, there were a range of activities measured throughout the project. While the percentage of girls in USAID-assisted primary schools remained constant throughout 19 the project—almost at the hoped-for 50% mark—the grade promotion rate of female pupils actually fell over the course of the project. In Ethiopia, girls have historically dropped out in greater numbers than boys in primary schools. In this evaluation, however, we found the reverse to be true in many rural communities, where more boys dropped out to work on the family farms or herd livestock. The interventions by Pact/IQPEP were designed to improve access, persistence and success of girls at the primary level and at the CTEs through a range of interventions. Girls’ Education Advisory Committees (GEACs) were to be established in 2,615 focus primary schools, and were charged to work closely with parent teacher associations to increase girls’ enrollment and improve retention and girls’ achievement levels. Starting with a 14.6% baseline, the percentage of primary schools with GEACs grew to 47% by year 5 of the program. The GEACs activities involved needs assessments; manuals, the translation of materials, the distribution of 5,595 copies; the orientation of 2,434 principals, including 249 female principals; workshops primarily for principals; 303 strategy development workshops; and life skills orientation workshops. The workshops included working with the community around gender issues, gender discrimination, life skills and the importance of educating girls. Other topics were academic support for girls through tutoring programs and the importance of sanitary towels and separate toilets for female students. House-to-house visits for girl dropouts were planned, as were negotiating with parents and relatives to prevent early marriage of school-age girls. To ensure a secure and supportive environment for girls preparing to be teachers, Girls’ Clubs (GC’s) were to be established and functioning in 22 CTEs. A manual was developed to assist in their formation. However, GCs were established in 27 CTEs, due to the increase in the number of CTEs over time. Out of the 27 GCs, only seven were found to be functional. The CTEs reported a total of 1,890 club members, representing 70 members per college. A range of activities were found in the clubs: community forums, tutoring, mentoring, awards for high performing students, induction programs, peer teaching and learning on life skills and study skills, gender violence workshops, reproductive health, and HIV/AIDs materials. IQPEP provided minimal support, but both MERA assessments and observations made by the evaluation team found impressive results. Gender units were formed in 11 of 30 CTEs and GU coordinators were appointed and trained. The coordinators that met with the evaluation team appeared to be deeply committed and talented women who praised IQPEP for its support for gender-focused activities in their CTEs. An important gender activity was the encouragement of male involvement. Gender inequality is a social problem with injustices impacting both males and females, so the IQPEP gender team organized a three￾day workshop on how to involve men and boys in the programs. Women instructors at the CTEs were taught gender-responsive pedagogy and appreciative inquiry, and then were encouraged to carry out AR of issues that affect female trainees’ performance and achievement at the college and out in the schools. A critically important program was to build the capacity of female teachers in leadership and management. Pact/IQPEP had a goal of 3,000 female primary teachers preparing for leadership and actually reached 4,795. The handbook, originally only in Amharic, was translated into Afan Oromo, Somali, and Tigrigna. TOTs (45 male and 35 female) were trained, and then participants were chosen based on leadership, planning and management, and an academic qualification above first degree. The results of the training were almost immediate as 30 graduates of the workshop received offers to become principals. While this is a small total in a large system, the promotion of women so rapidly after the workshops provided evidence of some system responsiveness. IQPEP sought to incorporate gender in the policies of all institutions with which it worked, holding a series of workshops and carrying out the MERA policy study on gender. Throughout the five-year life of the project, IQPEP sought to have a fair representation of women in all of its training programs. Its gender team produced a booklet titled Some Strategic Ideas to Enhance the Professional Development of 20 Females in the Education Sector, and distributed it among all the different groups and levels of the educational system with whom IQPEP worked. Component 5: Monitoring, Evaluation, Research, and Analysis (MERA) FIGURE 17. COMPONENT 5 OUTPUTS: 2009 – 2014 Output # Indicator Target Actual 2.4a Number of policy studies conducted, written up, and disseminated 4 4 2.4a Number of National Learning Assessments supported 2 2 - Number of MERA Monitoring and Evaluation Reports - 24 MERA completed four policy studies, one per year for the final four years of IQPEP. The topics for the studies were based on the expressed needs of the RSEBs and the MOE, and a technical committee made up of IQPEP and MOE staff oversaw the process. The first topic chosen was gender, which led to Exploring Policy Practice Gaps of Female Leadership in the Ethiopian Education System. The study took place in all nine regional states and the two city administrations, with a roughly equal number of males (1,139) and females (944) participating. Among the study’s major findings: the issue of female leadership has not been properly conceptualized nor has it been fully addressed within the educational system; female leaders only make up 11.2% of the lower educational leadership and 9.5% at the upper levels; female leaders generally secure more positions through competition at the lower levels than at upper levels where political appointments dominate; there is only a single policy provision on female leadership in the Education and Training Policy of Ethiopia and it is too general to target action; gender equity prevails more in “advanced” regions at the implementation level; gender-role stereotyping and negativity play important parts in discouraging female educators to become educational leaders; and female leaders are often urged to work in distant and difficult locations. The School Improvement Program (SIP): Its Implementation, Challenges and Policy Implications assessed the extent of SIP implementation, seeking factors that led to success or failure. The major concerns of the SIP were teaching-learning, a safe school environment, leadership and management, and community participation; these concerns were studied in 26 woredas across all regions and city administrations. The study found that the entire planning process had improved, but that there were problems of poor communication and inadequate key facilities and physical infrastructure. Factors Affecting the Success of Teachers and Educational Personnel in Implementing Educational Reform in Ethiopian Schools looked at the New Education and Training Policy and found inaccessibility to masses, lack of equity, curriculum irrelevance, low quality, and high wastage. Curriculum reform, continuous professional development (CPD), and redefining the language of instruction were seen as issues to be addressed. The study found that: regions had different understandings of school reform; principals understood the reforms better than other educational personnel; most informants had positive attitudes toward the reforms; and the SIP was better understood than other reform components. Education for All: A Study of Factors Keeping Young Children out of Schools was the final policy study and outlined the reasons 2.5 million children are out of school in Ethiopia. While progress has been made in keeping children in schools, the study found significant differences between various communities in their involvement and commitment to education, including the demand for girls’ education. Poverty and the need for children to work in manual labor, farming, or trade affected schooling rates and early marriage, teen pregnancy, and religious education were other factors. Schools factors such as shortages of laboratories and libraries, the absence of water, crowded classrooms, and inadequate materials also contributed to dropout rate. MERA assisted with the fourth and fifth National Learning Assessments (NLAs) to analyze the learning achievements of students and compare students on a range of factors. MERA also did a comparative study of IQPEP and non-IQPEP schools, based on the national data for the fourth NLA in 2011. IQPEP 21 assisted with the fifth NLA conducted this year, but results will not be available until the fall. The challenges faced by MERA/IQPEP on the NLA had to do with the competency of data collectors, the inaccessibility of some woredas, and timely completion of assessment activities. The recruitment of data collectors from all levels of the educational system helped to build capacity, as did training workshops that MERA held. While there was no deliverable required, the largest task for the MERA group in IQPEP was conducting annual impact assessments of key program activities. This resulted in 23 documents over the five years, including studies of the Woreda PMIS functioning, the adequacy of school management, the adequacy of the woreda annual plans and functioning of the WCRCs, the use of AL methods and FCA of primary teachers, the use of AL methods by CTE instructors, key activities in the CTEs, and various activities in the primary schools. Among the IQPEP components studied over the years were: EGRA, CTE, Decentralized Planning, Quality of Teaching, PMIS, School Management, AL, FCA, Personnel Management, SCRCs, GEACs, TSGs, RCs, LCUs, ITRCs, GUs, and GCs. The original plans for MERA, including many of the monitoring instruments were prepared at the time the original contract with AED was signed. This has made it possible for MERA to prepare a PMP, which is also included in this evaluation. The PMP includes baseline data from 2009–10 and each year’s targets, actual accomplishments, and a cumulative accomplishment’s percentage for years 2–5. At the request of the evaluation team, MERA also prepared a total targets and total actual completions list for the output factors in the PMP. Additional figures on project outputs and results are presented in Annex 7. A. APPROACH, INPUTS, AND RESULTS: CONCLUSIONS Pre-Service Training 1. The linkage school concept achieved its goal of linking the CTEs to the school setting and was critically important to the success of IQPEP. If the linkage schools become institutions in which outstanding teachers model best practices, then the idea will come to full fruition. 2. IQPEP trained 16,785 linkage primary school teachers in math, EGR, Instructional Kits and ScHBs. The satisfaction level of the teachers with the training and materials was very high. 3. Only 67% of the CTEs had functioning LCUs, which doesn’t bode well for the long-term sustainability of the concept. IQPEP made strong efforts to get all CTEs to appoint and have the coordinators trained, but were not successful in achieving their 100% goal. 4. It is possible that the training of TOTs, many of them CTE instructors, was IQPEP’s greatest and most sustainable contribution. These instructors have taken their experience and the materials back into their CTE classrooms, and are now ready to take what they have learned to primary teachers across the country. 5. The guidance and counseling, life skills, peer mentoring and other IQPEP-funded programs are unlikely to continue unless incorporated into the CTEs’ ongoing programs and curriculum. 6. The ITRCs appeared to have been a waste of precious funds and human resources by IQPEP. Only 22.2% of them were functioning at the end of the program; those that were functioning appeared not to have maintained the equipment in working order, nor were they making good use of the ITRCs for teaching and learning. 7. IQPEP/MERA reports had active learning at high levels in the CTEs. It is possible that this is the result of the instructors utilizing what they had learned and practiced as TOTs. 8. Instructional leadership training, along with basic planning and management skills, were generally well received by deans and department chairs at the CTEs and very highly rated by principals of the linkage schools. 22 9. The inputs to the CTEs were pretty minimal in comparison to the rest of IQPEP, and it could be concluded that without seriously affecting the curriculum, it was not transformational in any real sense. In-service Training 1. Among the largest and best received of all of IQPEPs’ initiatives were the many training workshops. The goal of 37,600 teachers completing a package of training was reached. In total, 80,347 teachers, completed some in-service training, but there were insufficient details on how many completed each training. Teachers exhibited high to very high satisfaction levels with their workshops and pled with the evaluators to do what they could to keep them going. 2. IQPEP met its goals for teachers completing a package of training and also had large numbers of teachers who participated in one or more workshops, but not the whole package. 3. Good teacher training is valuable, but with such large numbers of teachers to train, there was little time or money for follow-up. Mitigating this to some extent was the excellent, well received training of 10,112 principals, almost double the targeted number. These principals were given not only valuable planning and management skills, but also trained in instructional leadership, so that principals can now assist their teachers in teaching and learning. 4. In addition to principals, WEOs and RESB/CAEB officials were also trained in instructional leadership. This is an excellent mechanism for bringing about and sustaining educational reform. Teachers are best supported by people above them who have been through the same training and are able to understand the situation. 5. Tens of thousands of handbooks, modules, SIKs, and other materials published in English and four national languages are now in the hands of teachers and in principals’ offices, and can be used within their own schools or can be used by schools and woredas not part of IQPEP. Instructional materials developed by prior to this project were revised, validated, and translated, saving additional time and funds. 6. While the technology and instructional materials in the SCRCs were unanimously welcomed, it tended to be used primarily by the school in which it was located for reasons of communication, distance, and travel. This was found in interviews in 6 of 8 woredas and 20 of 26 schools visited throughout the country 7. Reading Centers are excellent conceptually and the evaluators were impressed with the training of the coordinators. However, to have a lasting impact on EGRW, there needs to be a much greater intervention in reading materials than 30 books for a school of 2,000 students. One period a week in a reading center is insufficient for almost any child. 8. Teacher Study Groups are an exciting and internationally successful approach to empowering teachers to do peer-to-peer sharing. With the TSGs receiving assistance from the principals and being supplied with many of the instructional materials, they have a real possibility of making a long￾term difference at comparatively little cost. 9. Definitions, scope and practice of active learning and continuous assessment are somewhat limited in the Ethiopian context, but there is no question that classroom instruction and assessment have begun a major change for the better. Decentralized Planning and Management 1. The PMIS at the woreda level was a carefully planned, phased approach, which led to high levels of success in almost every woreda. The IQPEP woredas were the first in the country to have complete records on all their employees, and have thus established a replicable model for the nation. 2. Principals in IQPEP schools actually produce SIPs and annual work plans, something that GEQIP I evaluators seldom found in the broad range of Ethiopian schools. These workshops have shown that even under-trained principals can, with short-term training produce workable SIPS. 23 3. Leaders at all levels were trained in instruction and instructional leadership, which appears to pay off well, as classroom teachers receive similar messages from supervisors, particularly when they try to teach strategies which are new to Ethiopian schools. 4. Principals spoke of their becoming participatory and even transformative leaders, while praising IQPEP training in planning and management as the only time they had much idea of how to do their jobs. 5. Training principals, WEOs and RESB/CAEB administrators in many of the topics that their teachers had been trained in, is helping produce a coherent supervisory system that appears more open to changes in the classroom. 6. Turnover of trained staff is a problem in most educational systems, but with Ethiopian teachers able to move throughout the public sector, some IQPEP schools have experienced complete staff turnover in one year. Without new training and follow-up it is hard to maintain current gains. In supervisory and technical positions, staff are continually changing and there needs to be a plan to train and retrain, in addition to policy changes. 7. Regional and City Board staff worked very closely with their IQPEP counterparts in almost every region of the country. The central MOE professionals and the IQPEP central staff did not appear to have the same level of trust or involvement, with the exception of those working on gender issues. 8. Inadequate educational supervision and poor resource management and maintenance were singled out as areas of greatest weakness. With a system of training educators at all levels in a similar planned and coherent way, and administrators in management techniques, there is now a possibility that these problems will be mitigated. Gender 1. As mandated, Pact/IQPEP broadly succeeded in bringing gender issues to the forefront at all levels of the educational system. 2. Despite strong efforts, the gender activities did not substantially affect either the percentage of girls or the number of girls dropping out of primary school. Broad societal and economic factors make this a long-term effort, but Pact/IQPEP has made a solid start. 3. While the evaluation team met some exceptionally committed gender unit coordinators at CTEs that had developed a wide range of well-received programs, the majority of CTEs did not participate in many meaningful ways. The gender program was not budgeted for large expenditures and this may be the reason that more CTEs did not choose to participate. 4. Sustainability may also be due to the level of commitment observed in interviews. The GU coordinators were generally working with small budgets, while still impacting CTE students. Female and male teachers in the primary schools expressed real pride in the girl’s room and efforts to provide safe environments for girls. 5. Pact/IQPEP made a concerted effort to bring gender issues to the fore at all levels of the educational system and appeared to do this in a way that did not offend, but rather enlisted them in efforts to bring about gender equity. 6. Changing gender roles and bringing about gender equity is a multi-generational effort, but the materials, school-based programs, manuals, and workshops began a nationwide dialogue on the issues involved. 7. There are likely significant differences between those CTEs with and without programs and those primary schools with and without girls’ clubs, girls’ rooms, sanitary facilities and other interventions. These differences, however, have not yet been evaluated, but on broad macro-issues like school attendance and dropouts, there is, as yet, no evidence of significant differences. 8. The central MOE gender experts were by far the most positive and satisfied by IQPEP’s contributions of any MOE group, and overall satisfaction with the gender programs is very high. 24 Monitoring, Evaluation, Research, and Analysis (MERA) 1. MERA, like the rest of IQPEP, was a prolific part of the program, producing both high quality evaluations of on-going parts of the project and high quality policy studies for the MOE. It achieved its major goals of four policy studies and support for two NLAs. Policy studies are not usually a high priority of MOEs, so external projects are needed to provide funding and expertise. 2. The annual assessments of IQPEP’s many components provide a solid history of the program, but some of the instruments prepared at the start of the project needed changes and adjustments over the 5 years. It was critical to have these inputs on a regular basis, with evidence related to targets. 3. MERA maintained high targets in most areas, but it could be questioned as to whether it might have been better to choose more realistic targets. On the other hand, if an area was not coming even close to its annual and final targets, it was evidence that something might be seriously wrong. 4. Definitions of active learning and formative continuous assessment were somewhat limited, but high (70–80%) usage rates were found in MERA studies. 5. The policy studies, done in conjunction with the MOE, provided important, albeit not surprising, findings and data. Policy makers’ not making use of the findings is a problem in many countries. 6. The EGRA reports done by MERA helped to make early childhood reading and writing a national priority. In the preliminary findings of its third study, MERA/IQPEP raised questions about the effects of efforts to-date to break the cycle of failure in basic reading and writing. 7. Financial and professional support of the 2011 and 2014 National Learning Assessment were critical in bringing the results of those studies to policy makers and the general public. B. PROGRAM MANAGEMENT: FINDINGS IQPEP created a two-tier structure: central and regional offices, which were generally well resourced. The magnitude of the IQPEP project outputs—tens of thousands trained and over 100,000 materials distributed—demonstrates in and of itself that the program management was highly efficient in achieving program objectives. The program was managed tightly and centrally. Regional education offices and CTEs stated that they could not spend a single birr without permission of the central office. Given the tight timelines and massive number of deliverables, this was likely necessary. The central office did not appear to have close ties to the REBs and central MOE, with the exception of top officials, while the regional IQPEP offices appeared to work closely with the RSEBs. The RSEBs had strong feelings that the program established a highly systematic, transparent, and efficient system, which could serve as a good working model to be benchmarked by other similar partners. In fact, it was noted that there were only fine lines between the regional IQPEP staff and REB staff, which led to the integration of the project activities with that of the REBs. The transition from AED to FHI360 in the project mid-term appeared to go quite smoothly, as IQPEP staff in Ethiopia and program backstops in the U.S. both remained in place. The REBs and IQPEP staffs worked well together at almost all steps in the process. While 54 MOE staff were targeted for training, only 21 actually went through the face-to-face training. With the exception of the gender officers in the MOE, there appeared to be a strong antipathy towards IQPEP on the part of six of the seven offices of the MOE staff interviewed. While evaluators were unable to uncover all the reasons for the apparent difference, evaluators did find that—when compared to other levels of the system—the MOE staff expressed frustration, claiming that IQPEP did not consult them on most activities. IQPEP in turn said that some MOE staff refused invitations to participate in planning meetings, evaluations, or trainings. Part of the difficulty was likely the turnover of staff at the central MOE, so that there was insufficient continuity to offer either trainings or keep long-term meaningful working relationships. The antipathy, however, was palpable among several MOE officials, and must be noted as a concern for any project, particularly one located within the MOE buildings. FIGURE 18. IQPEP STAFF SATISFACTION AS EMPLOYEES OF IQPEP Overa ll Satisfaction on a Sca le of I to S, IQPEP Staff (n=26) 16 ----------------~ 14 - ---- 12 -1----- 10 -1----- a - ---- 6 4 2 0 5 4 3 2 IQPEP staff reported high overall satisfaction as employees of IQPEP, but a number of consistent criticisms were expressed. Most staff were generally unhappy with t he level of pay and per diem as most claimed they were below the level of other NGO-funded projects. The evaluators have not independently verified this claim. Staff also felt an insufficient amount of support was afforded them in the area of training for their own self-improvement, especially after the transition to FHl360. Finally, staff expressed frustrat ion about the centralized decisio n-making struct ure of IQPEP management, complaining that it made mid-level managers irrelevant. FIGURE 19. IQPEP STAFF RATING OF MANAGERIAL SUPPORT IQPEP Staff Rating of Managerial Support (n=26) 16 ~---------------~ 14 12 10 8 6 4 2 0 5 4 3 2 25 FIGURE 20. IQPEP PROVISION OF EDUCATIONAL SUPPORT TO STAFF IQPEP Staff Rating of Training and Educational Support to Staff (n=26) 10 ....-~~~~~~~~~~~~~~~~- 9 -1---~~~---~---~~~~~~~~~~- 8 +------- 7 +------- 6 +------- 5 --- -11 4 3 2 0 5 4 3 2 FIGURE 21. IQPEP AND HUMAN RESOURCE NEEDS IQPEP Staff Rating of Project's Ability to Meet Human Resource Needs (n=I 5) 7 ....-~~~~~~~~~~~~~~~~- 6 4-~~~--~--~~~~~~~~~~­ s ------11 4 +--<----- 3 2 0 5 4 3 2 FIGURE 22. IQPEP AND FINANCIAL NEEDS IQPEP Staff Rating of Project's Ability to Meet Financial Needs (n=2 I) 12 ....-~~~~~~~~~~~~~~~~- 10 +-----~~~~~~~~~~~~~~- 8 6 4 2 0 5 4 3 2 26 FIGURE 23. IQPEP AND MATERIAL NEEDS IQPEP Staff Rating of Project's Ability to Meet Material Needs (n=20) 10 -.--~~~~~~~~~~~~~~~~~~ 9 8 7 6 5 4 3 2 0 5 4 3 2 The tables above indicate high ratings among IQPEP staff in managerial support and in meeting human resource, financial, and material needs of the program. Staff have complaints about a lack of sufficient support for their own training and education. Additional figures on IQPEP management issues are presented in Annex 7. B. PROGRAM MANAGEMENT: CONCLUSIONS I. The IQPEP contractor (previously AED and as of the third year FHI 360) in Ethiopia has been found to be adequately organized and structured as it enabled the program to meet or exceed most objectives. The amount of output that was completed by the project was immense, and management should be commended for carrying out such a large and complex project. Its relationship with the host government had room for improvement, however, especially with national MOE officials. 2. The delineation and delegation of each staff member's responsibilities, from the lowest to the top management level, was clear and most staff felt that the necessary support was provided to carry out implementation. 3. As a result of the high degree of management efficiency, the project was fully implemented, with a few previously planned inputs cancelled due to lack of results. This has allowed evaluators to get a good feel for the efficacy of the program design, which was found to be effective in reaching results in some areas, but not in others. 4. IQPEP management has been only moderately successful in building team capacity, and its ability to organize and encourage staff to work towards a common objective was rated moderate to good. 5. The evaluators did not do a cost-benefit analysis of the project, and so the findings on IQPEP's management, organization, and use of its resources (human, financial, material and time) to tackle program priorities in a timely and efficient manner is based on staff perceptions. 27 C. HOST GOVERNMENT SATISFACTION: FINDINGS FIGURE 24. HOST GOVERNMENT SATISFACTION Host Government Satisfaction, Average Score on a Scale of 1-5 (n=4 I 3) 5.00 ~----------------- 4.50 +---- - ---1'--,1----1 4.00 +----- 3.50 3.00 2.50 2.00 I.SO 1.00 0.50 0.00 FIGURE 25. PERCEPTIONS OF PROGRAM IMPACT Perceptions of Impact Within Own Sphere, All Respondents (n=226) 180 ~-----------------~ 160 +-r---w--------------- 140 120 100 80 60 40 20 0 A lot of Medium A little No Impact impact(4) impact (3) impact (2) (I) Don't know (0) From the 635 interviews with teachers, CTE instructors, W EO supervisors, REB staff, Kebele Boards and PT As, throughout 6 regio ns and o ne city administration of t he country, evaluato rs can generalize that the closer one got to the bottom of the educational pyramid, the greater was the satisfaction wit h IQPEP. All groups- with the exception of central MO E personnel and o ne RESB staff member- rated IQPEP as having a high to very high impact. IQPEP had and continues to have strong support at every 28 RESB, and W EOs have benefitted greatly from the program and look on it as a model for the thousands of other woredas and schools in the areas of management and leadership, supervision, finances, planning, PMIS, annual planning and other components in which they were trained by IQPEP. The principals and teachers in almost all the schools visited by the evaluation team were highly sat isfied with the relevance of the interventions to improve the quality of education through modules, handbooks, and manuals; the competence and dedication of facilitators; the efficiency of the services rendered; and the continuous backstopping support provided by the IQPEP regional and central staff. Additional figures on host government satisfaction and sustainability are presented in Annex 7. FIGURE 26. SU STAINABILITY H ow Likely Will IQPEP A ctivit ies Cont inue After Project Closes, All Respondents (n=392) 200 ~------------- 180 +-...... iiiiiiii..------------- 160 140 120 100 80 60 40 20 0 Very likely It's possible Not likely don't know C. HOST GOVERNMENT SATISFACTION: CONCLUSIONS I. The majority of program participants and stakeholders report that IQPEP has responded to the needs of the education system. 2. In interviews, the regional and woreda education officials felt a modest sense of ownership of the project in terms of their own professional abilit ies to continue the project, especially around training and access to quality materials. However, in terms of the financial ability of the system to maintain the levels of training and material provision that was achieved under IQPEP, there is much skepticism. 3. School directors, teachers, woreda and regional education officials rate the IQPEP trainings extremely high and report a high level of relevance to their needs. 4. IQPEP activities and outputs have been integrated into init iatives of the Government at CTEs, cluster centers, woredas, regional and national levels in terms of professional ability to carry out similar training and support. RECOMMENDATIONS AND LESSONS LEARNED I. USAID has been involved in broad-based education projects in Ethiopia since 1996, with BESO I and II and EQUIP2, all with AED as the contractor/recipient. This continuity was an advantage in meeting country requests for broad educational involvement, but also served as a constraint as the design contained few pilot projects to fi nd new ways of meeting the many challenges. 29 30 2. There are aspects of IQPEP which should be sustained and spread throughout the nation: PMIS, TSGs, instructional leadership by principals and WEOs, a large cadre of trained TOTs, linkage schools, expanded Reading Centers, less teacher-centered classrooms and gender-focused programs, to name a few. 3. Interviews with classroom teachers and principals convinced the evaluation team of the real appreciation and satisfaction for the attention they received from IQPEP, and the concern many of them feel that perhaps now no one will be looking out for them. IQPEP used teachers as Trainers and TOTs on a par with other educators. This is an exceptionally powerful message to send. 4. The linkage school connections to CTEs model is used in countries around the world and can lead to reforms in both settings. Theory and practice are brought together, particularly when the training institution provides a coordinator and the linkage schools become genuine models of the best in teaching and learning. 5. The IQPEP project as designed was not carefully focused, but rather attempted to try to cover too many components of the educational system too lightly. This was most evident in the pre-service teacher training at the CTEs, where money, time, and generally poorly focused attention were spent and in the end there was not much to show for it. 6. While the IQPEP staff are to be commended on meeting a few very large targets (80,347 classroom teachers, 10,112 principals, 5,015 WEOs, and 10,017 KETBs) it became impossible to do any meaningful follow-up, something mentioned by regional and local educational staff and by IQPEP staff at all levels. Regular follow-up is a must for any successful intervention to a chance at sustainability. 7. The PMIS at the woreda level was one of the distinct successes of IQPEP. We believe that its success was largely due to its starting small, then growing in a phased and incremental manner, while constantly testing and improving the model. 8. The Teacher Study Groups (TSG) received little project money and effort, but may well have the greatest long-term effect on teaching-learning. Supplying teachers and schools with instructional material and giving them time for peer-to-peer interaction on lessons could well be one of the best things coming out of IQPEP. 9. The program has done a major service to the Ethiopian educational system by developing a large cadre of Trainers and TOTs. These individuals can form the core of a nationwide in-service training system, as they now have the skills and knowledge needed. These trainers come from among the best of classroom teachers, principals, WEOs, RESB/CAEBS, CTEs, and universities. 10. Reading Centers were a very positive innovation, but any serious attempt to deal with the issues raised by EGRA must involve much more than a small number of books, as low as 30 early readers for a school of 2000 children. Classroom libraries supplied with numerous age appropriate books in the mother tongue could make a dramatic difference. It has been estimated that such books could be produced at a cost of 20 birr per book. Tens of thousands of schools worldwide have children, teachers, and the community write books costing almost nothing. 11. Policy studies can and do have an effect. The Gender policy study helped with the successes that Pact/IQPEP had in raising difficult gender issues in the schools and society. The EGRA studies conducted by IQPEP have led to a national effort, supported by USAID, to confront the issue of early grade illiteracy. 12. MERA’s use of baseline studies, followed by targets to be reached, is a good model. This is particularly true if the targets are realistic and more of the goals are outcomes, instead of just inputs or outputs. 13. The holistic approach to supervision was inherent in IQPEP, so that principals, WEOs, zonal officers and RESB/CAEBs were all trained not only to function in their specific jobs, but also to have been through training on instructional leadership and supervision, and even at times on subject matter itself. 31 14. Unless the problem of turnover is alleviated, the government or some other donor is going to have to be constantly re-training teachers, principals, WEOs, PMIS staff and others in the system. 15. Needs assessments are critical to not only designing training or materials that educators need, but also in getting buy-in from the stakeholders. IQPEP did this well. 16. In Ethiopia, and any other country for that matter, gender must always be a central concern. Pact/IQPEP set a high standard and showed how—with a comparatively small budget—a well￾focused gender program can make a difference in the lives of boys and girls, women and men in not just the schools but in society in general. It is also a multi-generational struggle, not a one-off activity, and in 10 or 20 years, Ethiopia will look back on this period as the one where gender equity made real gains. See Annex 8 for a detailed analysis matrix of all IQPEP components with corresponding challenges and recommendations for each. 1 ANNEXES 2 ANNEX 1: FIELD RESEARCH SCHEDULE ANNEX 3: RESEARCH METHODOLOGY MATRIX Evaluation Type of Answer/ Methods for Data Collection Sampling or Data Analysis Questions Evidence Needed Selection Methods Approach Data Source(s) Method I. Approach, Inputs and Yes/No IQPEP HQ and regional Frequency distributions of Results: What are the Program's x Description staff and MIS; MoE, semi-structures interviews and overall effectiveness and results to- Comparison Regional, and Woreda Desk review of IQPEP MIS, Purposeful sampling of surveys; qualitative codi ng and date. Explanation Officials; CTE officials, reports, and assessments; Semi- access ble IQPEP content analysis of field notes faculty; primary school structured interviews; surveys; CTEs, woredas and and interview transcripts; principals and school focus group discussions. schools stratified by triangulation of evaluator leaders, teachers, and geography students; PT As, school (urban/rural). collected data with existing boards, and Kebeles. MIS and assessment data. 2. Program Management: How Yes/No Frequency distributions of efficient is the management of x Description IQPEP HQ and regional Desk review of IQPEP MIS, Director and Deputy semi-structured interviews; IQ PEP in achieving the Program Comparison staff, documents, and reports, and assessments; Semi- Directors, available qualitative coding and content objectives. Explanation MIS. structured interviews. regional staff. analysis of field notes and interview transcripts; triangulation of evaluator collected data with existing MIS and assessment data. 3. Host government x Yes/No IQPEP HQ and regional satisfaction: Has the project x Description staff and MIS; MoE, Frequency distributions of responded to the needs of Comparison Regional, and Woreda Purposeful sampling of semi-structures interviews and beneficiaries, are relevant Explanation Officials; CTE officials, Desk review of IQPEP MIS, access ble IQPEP surveys; qualitative coding and government officials satisfied with faculty; primary school reports, and assessments; Semi- CTEs, woredas and content analysis of field notes the performance of the project, and principals and school structured interviews; surveys; schools stratified by and interview transcripts; have project activities been leaders, teachers, and focus group discussions. geography triangulation of evaluator integrated into relevant government students; PT As, school collected data with existing institutions at the national, regional, boards, and Kebeles. (urban/rural). MIS and assessment data. and local levels. 4. Lessons learned: What are the Yes/No IQPEP HQ and regional Frequency distributions of constraints to project x Description staff and MIS; MoE, semi-structures interviews and implementation and achievement of Comparison Regional, and Woreda Desk review of IQPEP MIS, Purposeful sampling of surveys; qualitative codi ng and goals and objectives. Explanation Officials; CTE officials, reports, and assessments; Semi- access ble IQPEP content analysis of field notes faculty; primary school structured interviews; surveys; CTEs, woredas and and interview transcripts; principals and school focus group discussions. schools stratified by triangulation of evaluator leaders, teachers, and geography students; PT As, school (urban/rural). collected data with existing boards, and Kebeles. MIS and assessment data. 12 13 ANNEX 4: DATA COLLECTION TOOLS IQPEP Final Evaluation MoE/Regional Education Official Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Region 4. Woreda 5. Town/Village 6. Name of official 7. Title 8. How long have you been in this position? ________(years) 9. Gender: M / F 10.Tell me about the IQPEP activities in which you were involved?  Planning & management training (a)  PMIS (b)  Teacher education colleges (c)  Linkage schools/pedagogical units (d)  Cluster Resource Centers (e)  Reading Centers (f)  GEAC/Gender clubs (g) Other: 11.How much of an impact has IQPEP had on the ability of the staff here in your office to carry out the overall mission of the Regional Office? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 14 12.How much of an impact has IQPEP had on the ability of the staff here in your office to support the Woreda Education Offices? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 13.How much of an impact has IQPEP had on pre-service teacher training at the colleges of teacher education? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 14.How much of an impact has IQPEP had on in-service teacher training? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 15.How much of an impact has IQPEP had on the quality of teaching in primary schools? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 15 16.How much of an impact has IQPEP had on student achievement in early grade reading and writing? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 17.How much of an impact has IQPEP had on student achievement in math and science? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 18.How much of an impact has IQPEP had on girl’s education and empowerment? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 19.To what extent have there been barriers to implementing IQPEP activities in the Woreda Education Offices? No barriers A few barriers Many barriers Don’t know Explain: 16 20.To what extent have there been barriers to implementing IQPEP activities in the colleges of teacher education? No barriers A few barriers Many barriers Don’t know Explain: 21.How useful have IQPEP planning and management training been for you and other Regional Office staff? Not useful Somewhat useful Very useful Did not participate Explain: 22.The USAID/IQPEP program has been implemented for 5 years and is now closing. (A) Are there plans to continue the initiatives of the project after it closes, and (B) what is the likelihood that they will be implemented? YES NO Don’t know Not likely It’s possible Very Likely Don’t know Explain: 23.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction with the IQPEP project? 1 2 3 4 5 Explain: 17 24.Of the many activities of IQPEP, which in your experience are the best practices. 25.Anything else we should know about IQPEP that we haven’t talked about? 18 IQPEP Final Evaluation Woreda Education Official Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Region 4. Woreda 5. Town/Village 6. Name of official 7. Title 8. How long have you been in this position? ________(years) 9. Gender: M / F 26.Tell me about the IQPEP activities in which you were involved?  Planning & management training (a)  PMIS (b)  Teacher education colleges (c)  Linkage schools/pedagogical units (d)  Cluster Resource Centers (e) Other: 27.How much of an impact has IQPEP had on the ability of the staff here in your office to carry out the mission of the Woreda Education Office? No impact A little impact Medium Impact A lot of impact Don’t know 28. Explain: 29.How much of an impact has IQPEP had on the ability of Regional Education Office staff to support the activities of the Woreda Education Offices? 19 No impact A little impact Medium Impact A lot of impact Don’t know Explain: 30.How much of an impact the PMIS support had on the functioning of the Woreda Office? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 31.How much of an impact has IQPEP had on the ability of Woreda Education Offices to support principals and school leaders? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 32.How much of an impact has IQPEP had on the ability of the Woreda Education Offices to improve the quality of teaching in their primary schools? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 33.How much of an impact has IQPEP had on in-service teacher training? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 20 34.How much of an impact has IQPEP had on the quality of teaching in primary schools? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 35.How much of an impact has IQPEP had on student achievement in early grade reading and writing? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 36.How much of an impact has IQPEP had on girl’s education and empowerment? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 37.To what extent have there been barriers to implementing IQPEP activities in the Woreda Offices? No barriers A few barriers Many barriers Don’t know Explain: 21 38.To what extent have there been barriers to changing teacher practices and promoting “active learning?” No barriers A few barriers Many barriers Don’t know Explain: 39.How useful have IQPEP planning and management training been for you and other staff in the Woreda Education Office? Not useful Somewhat useful Very useful Did not participate Explain: 40.The USAID/IQPEP program has been implemented for 5 years and is now closing. (A) Are there plans to continue the initiatives of the project after it closes, and (B) what is the likelihood that they will be implemented? YES NO Don’t know Not likely It’s possible Very Likely Don’t know Explain: 41.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction with the IQPEP project? 1 2 3 4 5 Explain: 22 42.Of the many activities of IQPEP, which in your experience are the best practices. 43.Anything else we should know about IQPEP that we haven’t talked about? 23 IQPEP Final Evaluation Teacher Education College Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Name of College 4. Region 5. Woreda 6. Town/Village 7. Name of official 8. Title 9. How long have you been in this position? ________(years) 10. Gender: M / F 44.Tell me about the IQPEP activities in which you were involved?  Planning and management training  Linkage schools  Staff Development Units  Gender clubs Other: 45.How much of an impact has IQPEP had on the ability of Regional Education Office staff to support the activities of the Colleges of Teacher Education? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 46.How much of an impact has IQPEP had on the instructional practices of the instructors at this college? 24 No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 47.How much of an impact has IQPEP had on the ability of this college to improve the quality of teaching in primary schools? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 48.How much of an impact has IQPEP had on student achievement in early grade reading and writing? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 49.How much of an impact has IQPEP had on student achievement in math and science? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 50.How much of an impact has IQPEP had on girl’s education and empowerment? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 25 51.To what extent have there been barriers to implementing IQPEP activities in this college? No barriers A few barriers Many barriers Don’t know Explain and how you know: 52.To what extent have there been barriers to changing teacher practices and promoting “active learning?” No barriers A few barriers Many barriers Don’t know Explain and how you know: 53.How useful has IQPEP planning and management training been for the deans and other leaders of this CTE? Not useful Somewhat useful Very useful Don’t know Explain and how you know: 54.How useful have the pedagogical units in linkage schools been for the teachers in those schools? Not useful Somewhat useful Very useful Don’t know Explain and how you know: 26 55.To what extent have the number of linkage schools increased from the original number supported by IQPEP? None A few Many Don’t know Explain and how you know: 56.The USAID/IQPEP program has been implemented for 5 years and is now closing. (A) Are there plans to continue the initiatives of the project after it closes, and (B) what is the likelihood that they will be implemented? YES NO Don’t know Not likely It’s possible Very Likely Don’t know Explain: 57.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction with the IQPEP project? 1 2 3 4 5 Explain: 58.Of the many activities of IQPEP, which in your experience are the best practices. 59.Anything else we should know about IQPEP that we haven’t talked about? 27 28 IQPEP Final Evaluation School Leader/Principal Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Region 4. Woreda 5. Town/Village 6. Name of official 7. Title 8. How long have you been in this position? ________(years) 9. Gender: M / F 10. Name of School 60.School Type:  Woreda Cluster Resource (a)  School Cluster Resource (b)  Satellite (c)  Linkage (d) 61.Tell me about the IQPEP activities in which you were involved?  Planning & management training (a)  TOT (b)  Pre-service training (c)  In-service training (d)  Early Grade Reading (e)  GEAC (f)  Teacher Study Groups (g) Other: 62.How much of an impact has IQPEP had on YOUR ability to be an effective school leader? No impact A little impact Medium Impact A lot of impact Don’t know 29 Explain the impact and how you know: 63.How much of an impact has IQPEP had on the instructional practices of the teachers here at this school? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 64.How much of an impact has IQPEP had on student achievement in early grade reading and writing? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 65.How much of an impact has IQPEP had on student achievement in math and science? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 66.How much of an impact has the GEAC had on girl’s education and empowerment? No impact A little impact Medium Impact A lot of impact NA Explain the impact and how you know: 30 67.If this school has a cluster resource center, how much of an impact has it had on teaching in this school? No impact A little impact Medium Impact A lot of impact NA Explain the impact and how you know: 68.If this is a linkage school, how much of an impact has the pedagogical center had on this school? No impact A little impact Medium Impact A lot of impact NA Explain the impact and how you know: 69.To what extent have there been barriers to implementing IQPEP activities in this school? No barriers A few barriers Many barriers Don’t know Explain the impact and how you know: 70.To what extent have there been barriers to changing teacher practices and promoting “active learning” and “continuous assessment.” No barriers A few barriers Many barriers Don’t know List the barriers and how you know: 31 71.How useful has IQPEP planning and management training been for principals and school leaders? Not useful Somewhat useful Very useful Did not participate Explain the impact and how you know: 72.The USAID/IQPEP program has been implemented for 5 years and is now closing. (A) Are there plans to continue the initiatives of the project after it closes, and (B) what is the likelihood that they will be implemented? YES NO Don’t know Not likely It’s possible Very Likely Don’t know Explain: 73.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction with the IQPEP project? 1 2 3 4 5 Explain: 74.Of the many activities of IQPEP, which in your experience are the best practices. 32 75.Anything else we should know about IQPEP that we haven’t talked about? 33 IQPEP Final Evaluation Cluster Resource Center/Pedagogical Center/Reading Center/Science Center Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Region 4. Woreda 5. Town/Village 6. Name of interviewee 7. Title 8. How long have you been in this position? ________(years) 9. Gender: M / F 10. Name of School 76.Center Type:  Woreda Cluster Resource (a)  School Cluster Resource (b)  Pedagogical Center (c)  Reading Center/Corner (d)  Science Lab (e) Other: (f) 77.How much of an impact has this Center had on the quality of teaching in this schools? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 78.How much of an impact has this Center had on in-service teacher training? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 34 79.How much of an impact has this Center had on student achievement in early grade reading and writing? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 80.How much of an impact has the Woreda Cluster Resource Center had on girl’s education and empowerment? No impact A little impact Medium Impact A lot of impact Don’t know Explain the impact and how you know: 81.To what extent have there been barriers to implementing the Woreda Cluster Resource Center activities in the Woreda schools? No barriers A few barriers Many barriers Don’t know Explain: 82.To what extent have there been barriers to changing teacher practices and promoting “active learning?” No barriers A few barriers Many barriers Don’t know Explain: 35 83.The USAID/IQPEP program has been implemented for 5 years and is now closing. (A) Are there plans to continue the initiatives of the project after it closes, and (B) what is the likelihood that they will be implemented? YES NO Don’t know Not likely It’s possible Very Likely Don’t know Explain: 84.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction with the IQPEP project? 1 2 3 4 5 Explain: 85.Of the many activities of IQPEP, which in your experience are the best practices. 86.Anything else we should know about IQPEP that we haven’t talked about? 36 IQPEP Final Evaluation IQPEP Staff Interview Protocol This interview is part of the final evaluation of the USAID/IQPEP project. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are also interested in the management of IQPEP. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 1. Name of Interviewer 2. Date 3. Region 4. Town/Village 5. Name of staff 6. Title 7. How long have you been in this position? ________(years) 8. Gender: M / F 9. Indicate the IQPEP activities in which you were involved?  In-service Teacher Training (a)  Pre-service Teacher Training (b)  MERA (c)  Planning and management (d)  Gender (e)  Operations (f) Other: (g) 10.Tell me about your duties and responsibilities: 11.Tell us what you feel are IQPEP’s greatest strengths? 12.Tell us what you feel are IQPEP’s greatest weaknesses? 37 13.On a scale of 1 to 5, with 1 being lowest, rate the effectiveness of your managers to support you in carrying out your roles and responsibilities? 1 2 3 4 5 Explain: 14.On a scale of 1 to 5, with 1 being lowest, rate the extent to which the organizational structure of IQPEP was appropriate to meet its programmatic goals and objectives? 1 2 3 4 5 Explain: 15.Do you have a job description that delineates in detail the responsibilities of your position? YES NO Don’t know If yes, please provide evidence of a detailed job description. Provided and detailed Provided and not detailed Not provided 16.Have there been instances when your ability to fully implement an initiative or meet programmatic targets were hampered by insufficient project support? YES NO Don’t know Explain: 38 17.On a scale of 1 to 5, with 1 being lowest, rate the project’s efforts to increase your knowledge and skills in order to more effectively carry out the roles and responsibilities of your position. 1 2 3 4 5 Explain: 18.On a scale of 1 to 5, with 1 being lowest, rate the project’s efforts to support or improve the staff’s ability to work together. 1 2 3 4 5 Explain: 19.On a scale of 1 to 5, with 1 being lowest, rate the project’s ability to respond in a timely and efficient manner to the human resource needs required to meet programmatic goals and targets. 1 2 3 4 5 Explain: 20.On a scale of 1 to 5, with 1 being lowest, rate the project’s ability to respond in a timely and efficient manner to the financial needs required to meet programmatic goals and targets. 39 1 2 3 4 5 Explain: 21.On a scale of 1 to 5, with 1 being lowest, rate the project’s ability to respond in a timely and efficient manner to the material needs required to meet programmatic goals and targets. 1 2 3 4 5 Explain: 22.On a scale of 1 to 5, with 1 being lowest, rate the project’s support to you in your capacity to document, collect, and manage data about the activities you oversee. 1 2 3 4 5 Explain: 23.On a scale of 1 to 5, with 1 being lowest, rate the project’s ability to respond effectively and efficiently to changing conditions and unforeseen challenges. 1 2 3 4 5 Explain: 40 24.What is the likelihood that IQPEP initiatives and activities will continue after the project closes, given limited resources? Not likely It’s possible Very Likely Don’t know Explain: 25.On a scale of 1 to 5, with 1 being lowest, how would you rate your overall satisfaction as an employee of IQPEP? 1 2 3 4 5 Explain: 41 IQPEP Final Evaluation Teacher Survey This survey is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 15-20 minutes to complete. 1. Your Name 2. Date 3. School name (where you teach) 4. School Region 5. School Woreda 6. School Town/Village 7. Classes you teach 8. Subjects you teach 9. Gender: M / F 10.How many years you been a teacher? ___________________ 11.How many years have you been at your current school? 12.Check the box next to the IQPEP activities in which you participated?  Planning & management training (a)  Teacher education colleges (b)  Linkage schools/pedagogical units (c)  Cluster Resource Centers (d)  Reading Centers (e)  GEAC (f)  Teacher Study Groups (g) Other: (h) Many of the following questions ask that you rate your answer on a scale of 1 to 5, with 1 being low effect and 5 being the highest effect. Check “NA” if you did not participate in the activity or don’t know. 13.If you work in a linkage school, how would you rate the effectiveness of the pedagogical units in linkage schools to improve the quality of teaching in your school? 1 2 3 4 5 NA 42 14.If you work in a school with a cluster resource center, how would you rate the effectiveness of the cluster resource center to improve the quality of teaching in your school? 1 2 3 4 5 NA 15.How would you rate the effectiveness of the teacher study groups to improve the quality of teaching in your school? 1 2 3 4 5 NA 16.How would you rate the effectiveness of the reading centers to improve the quality of teaching early grade reading and writing? 1 2 3 4 5 NA 17.How would you rate the effectiveness of the reading centers to improve the quality of teaching math and science? 1 2 3 4 5 NA 18.How would you rate the effectiveness of the GEAC to address gender equity and girl’s empowerment in your school? 1 2 3 4 5 NA 19.How often do you use “active learning” teaching techniques in your classes? Almost never A few times a year A few times a month A few times a week Every day NA 20.Is it sometimes challenging to use “active learning” techniques in your classroom? YES NO DON’T KNOW 21.Check any of the following that are barriers to doing “active learning” in your classroom:  Too many pupils (a)  Not enough materials (b)  Do not understand how to do it properly (c)  Not enough time (d)  The children don’t like it (e)  Parents don’t like it (f)  Not enough support from the principal (g)  Poor classroom facilities (h) 43 Other (i) 22.If you work in a school with a cluster resource center, how would you rate the effectiveness of it to support to teachers in your school? 1 2 3 4 5 NA 23.If you work in a linkage school, how would you rate the effectiveness of the Colleges of Teacher Education to support teachers your school? 1 2 3 4 5 NA 24.How effective has the IQPEP activities been in raising student achievement, especially in reading and writing? 1 2 3 4 5 NA 25.The USAID/IQPEP program has been implemented for 5 years and is now closing. Are there plans to continue the initiatives of the project after it closes? YES NO Don’t know 26.If yes, what is the likelihood that these continuation plans will be implemented given limited resources? Not likely It’s possible Very Likely Don’t know 27.Please rate your overall satisfaction with USAID? IQPEP activities and initiatives in your school. 1 2 3 4 5 NA 28.Is there anything else that you think the evaluators should know about the IQPEP project? 44 45 IQPEP Final Evaluation Teacher Focus Group Discussion Date: Time start: Time end: Facilitator Name: Note-taker name: Region, Woreda, Village: Number of males: Number of females: 1. Before we begin, we would like to introduce ourselves… 2. Introduction: This discussion is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate your performance. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This survey will take about 1 hour to complete. 3. Now, one at a time, everyone please say your first name, where you teach, and the classes and subjects you teach. 4. Raise your hand if you are familiar with IQPEP? (Choose a person with their hand up) What can you tell me about it? (Repeat once or twice) 5. Now raise your hand if you have participated in any IQPEP activities either at your school, or another school, or at a teacher education college? (Choose a person with their hand up). Tell me about the activity (Repeat two or three times). 6. As some of you may know, one of the goals of IQPEP is to promote “Active Learning” in the classroom. Can anyone explain to me what active learning is? 7. We understand that doing active learning in the classroom can be challenging. Raise your hand if you do active learning in your classroom regularly. (Choose a person with their hand up). Tell me about what you do (Repeat two or three times). 8. Among those of you who didn’t raise your hand, can you tell me some of the barrier you face to doing active learning in the classroom? 9. Is there anything else you think we should know about the IQPEP project? 10. Do you have any questions for us? Thank you very much for spending your time with us today. 46 IQPEP Final Evaluation Student Focus Group Discussion Date: Time start: Time end: Facilitator Name: Note-taker name: Region, Woreda, Village: Number of males: Number of females: 1. Before we begin, we would like to introduce ourselves… 2. Introduction: This group interview is part of the final evaluation of the IQPEP project, a project to improve primary schools funded by the United States. We are very grateful for your time and appreciate your help. What we are interested in is your views about the way your teachers teach. We hope you will give us as much information as possible, and also respect the opinions and experiences of others. We will not share what you tell us here with anyone or put your names in our report. Nothing that you say to us will affect your grade or standing in this school; this is strictly between us. This survey will take about 1 hour to complete. 3. Now, one at a time, everyone please say your first name, how old you are, and what class you are in. 4. Raise your hand if you think you have an excellent teacher right now? By excellent, I mean a teacher from whom you really learn a lot, more than other teachers you’ve had. (Repeat once or twice) 5. A) Now I’m going read a list of classroom activities, and for each I’ll ask you to raise your hands if your teachers do this activity regularly in the classroom. That means at least a few times a month. (record number that raise hands) B) Based on the responses above, use follow-up questions to clarify the students’ responses.  Teacher copies text from a book onto the chalkboard, and then students copy it into their notebooks  Students write on the chalkboard  Students work in small groups together  Students repeat in unison what the teacher reads from the board  Students sit at their desks and work individually on an assignment  Students make learning materials posters, charts, or other visuals to present to the class  The teacher calls on students by name to answer questions  The Teacher takes the students to do activities outside or in another location beside the classroom  Students discuss a classroom topic amongst themselves  The teacher encourages both boys and girls to participate  Students are given a problem, and they must try to figure out the answer without the teacher’s help  Students work individually from a textbook  Students give presentations to the class  Students play games that involve what they are learning in class 6. Raise your hand if you have participated in the GEAC activities? Tell me about what you did. Thank you very much for spending your time with us today. 47 IQPEP Final Evaluation Kebele/PTA Focus Group Discussion Date: Time start: Time end: Facilitator Name: Note-taker name: Region, Woreda, Village: Number of males: Number of females: 1. Before we begin, we would like to introduce ourselves… 2. Introduction: This interview is part of the final evaluation of the USAID/IQPEP project, a project to improve primary schools in Ethiopia. This evaluation will not determine if the project will continue, nor is it to evaluate performance of parents, teachers, or students. It is only to learn about what went well, and what the challenges were. We are very grateful for your time and appreciate your help. We hope you will give us as much information as possible. The information you provide to us here is confidential. Please know that we will not share your name with anyone outside the research team, or put your name in our report without your permission. This discussion will take about 1 hour to complete. 3. Now, one at a time, everyone please say your first name, your role in the Kebele/PTA, and how long you have been a member. 4. Tell me a little about the activities of this group. In what ways do you support schools and teachers? 5. Raise your hand if you participated in any training provided by USAID/IQPEP. (Choose a person with their hand up.) Tell me about what you learned, and if it was useful. (Repeat 2 to 3 times) 6. Raise your hand if you believe that IQPEP has helped to improve the way teachers teach in your school(s). (Choose a person with their hand up.) Tell me how the teaching has improved and how you know. (Repeat 2 to 3 times). Thank you very much for spending your time with us today. ANNEX 5: IQPEP PMP OUTPUT INDICATORS, 2009-2014 CUMULATIVE Performance Indicator Output 1: 2.lStrengthened Pl'f!-service Teac:htr Education Output 2: 2.2 Enhanced ln,·service Teacher Professional Development Performance lndlcMor Output 3: 2.30ecentralized Educational Planning and Management Output4: 2.4 Supporting the Educational System by Enhancing Rese.arch -· _,, -·· -IV ..... .,.., •. -.v 2t09-10 201CHI 20!1·12 20U·l1 20U·l4 l l ! l ! l ! l ! Name of lrdlcator l ;; f i! i f i! i ~ ~ ;, ~ !c H " ~ l ~ l { H ~ 1 H ~ 1 H ~ l { ~1 ~ /( !! ~ {! ~ ~1 !! ~ E ~ {! ~ ~1 ~ ~ ~1 ;; • 3 6 I af i I ~ § Ji ~ < ~ 2.1~ Pcrccn1 of functioning !nronmuion '"'"' 16-1 Technology Rcsomcc Centers (lTRC$) - - - - - J0.0% 6'7% 9.6 9.6 85.0% 23.~ 27.4 19.4 100.0% 22.2 2.t b Percent of functioning ink~c 68.,. 68• Coordina1ini:r: Unit§ (LCUs) in C11!s - - - - - - - '11).(l!I 30.0% .,. d l .9 IS.llll 7).6% 86.6 66.8 ""'"" ''·' 2.le Number of Linka~e prin1ary school teachers who received mid compkted a f11(¢-IO-fACC trAining rcbtcd ~ ins;tnaction - - - 2.442 1.904 18 0 180 l.877 ·~ .. 1S6.4 1'26.) il.200 4,961 1l8.J. 122.9 l.600 ... ,. l07J 1189 U,119 16, 785 (tvfath, EGR, lns1ruc1io·1al Ki1s and Sclllh) 2.ld Number of Linkage prirn:uy school principals arnl dc1xa1y p incipals ''ho received and complete( a face-lo.face - - - 400 130 82.5 82.5 l!OO .. , 55.9 6'.8 l!OO 761 95.4 77.0 - 26s··· 90.S 2.000 1,809 training rcfote-d to instnction and in.i;tn1ctionol lcR hi., 2.le NuinbcrofCTE ir1strudors trained N8l - - - - - - 2()66 1487 72.0 72.0 - 91 76.7 - 2066 1585 2 . .2~ Number of (dust