i INDONESIA: PRIORITAS MID-TERM EVALUATION FINAL Report May 2015 This report was produced for review by the United States Agency for International Development (USAID) under Task Order PRIORITAS Mid-term Evaluation. It was prepared by JBS International. The authors are: Frank Dall, Lynn Losert, Agung Purwadi, Bandi, Sandra Triatmoko, and Yanti Yulianti Maskoen . ii Indonesia: PRIORITAS Mid-term Evaluation Frank Dall, Team Leader Agung Purwadi Bandi Lynn Losert Sandra Triatmoko Yanti Yulianti Maskoen The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii Table of Contents ACRONYMS AND ABBREVIATIONS .................................................................................... V EXECUTIVE SUMMARY ...................................................................................................... VII EDUCATION AT A GLANCE: INDONESIA......................................................................... XI SECTION I: INTRODUCTION .................................................................................................. 1 Objectives of the Evaluation ...................................................................................................................... 1 Methodology .............................................................................................................................................. 2 Constrains in Undertaking the Evaluation ................................................................................................. 4 Structure of the Report............................................................................................................................... 4 SECTION II: BACKGROUND ................................................................................................... 5 Education in Indonesia .............................................................................................................................. 5 USAID Assistance to Basic Education ...................................................................................................... 6 The PRIORITAS Project ........................................................................................................................... 7 Donor Involvement in the Sector ............................................................................................................... 7 SECTION III: FINDINGS ........................................................................................................... 8 Question I: To what extent has PRIORITAS achieved its stated mid-point objectives and outcomes in a timely and effective manner? ..................................................................................................................... 8 Question II: What aspects of PRIORITAS are proving most and least effective to improve access to quality education in Indonesia for each of the PRIORITAS project goals? ............................................ 12 Question III: To what extent have PRIORITAS stakeholders benefited from the project’s activities and what specific value has been added? ....................................................................................................... 23 Question IV. To what extent are the PRIORITAS project’s resources being implemented and managed efficiently and cost effectively? ................................................................................................................ 28 Question V. What evidence is there to indicate that student reading and reading comprehension skills have improved as a result of PRIORITAS interventions? ........................................................................ 31 Question VI. What evidence is there to show that the PRIORITAS project’s activities and results are making progress toward sustainability and replication after project completion? ................................. 35 Question VII. How effective has each component of the PRIORITAS project been in improving gender equality among students in schools? ....................................................................................................... 37 Question VIII. Is PRIORITAS on track in terms of meeting its overall end-of-project goals? ................ 39 SECTION IV CONCLUSIONS ................................................................................................. 43 SECTION V RECOMMENDATIONS ..................................................................................... 45 ANNEXES ................................................................................................................................... 48 iv ANNEX I: Terms of Reference for the Mid-term Evaluation of USAID PRIORITAS .......................... 49 ANNEX II: List of Documents Referenced ............................................................................................. 53 ANNEX III: Interview Guides ................................................................................................................. 59 ANNEX IV: Survey Instruments ............................................................................................................. 69 ANNEX V: Classroom Observation Form, Measurement Criteria, and Rating Scale ............................ 79 ANNEX VI: List of Interviewees and Contacts ...................................................................................... 81 ANNEX VII: Field Travel Schedule ....................................................................................................... 94 ANNEX VIII: Classroom Observation Data ........................................................................................... 98 ANNEX IX: Selected Additional Survey Data ...................................................................................... 100 ANNEX X: School Interview Data ....................................................................................................... 101 Figures and Tables Figure 1: PRIORITAS Activities Impacting Education Quality in District Schools.................... 25 Figure 2: Parents' Participation in School Activities .................................................................... 27 Table 1: Interviewees by Stakeholder Group .................................................................................. 3 Table 2: PRIORITAS Performance Indicators: Objective 1........................................................... 8 Table 3: PRIORITAS Performance Indicators: Objective 2......................................................... 10 Table 4: Performance Indicators: Objective 3 .............................................................................. 11 Table 5: Observation Scores by Province ..................................................................................... 31 Table 6: Primary Observation Scores: Math and Science ............................................................ 34 Table 7: Jr. Secondary Observation Scores: English and Indonesian ........................................... 35 v ACRONYMS and ABBREVIATIONS AL Active Learning BAPPEDA Regional Government Planning Board BAPPENAS National Development Planning Board BERMUTU Better Education Through Reform Management and Universal Teacher Upgrading, a recent World Bank project BOS Biaya Operasi Sekolah: School Operational Budget funded by the Central Government Bupati Principal Government Officer Responsible for District or City CTL Contextual Teaching and Learning DBE Decentralized Basic Education DC District Coordinator Dewan Pendidikan Education Board DFAT Department of Foreign Affairs and Trade, branch which delivers Australian aid DINAS Pendidikan Education Office of Local Government (District and Provincial Levels) EDC Education Development Center, PRIORITAS implementing partner EGRA Early Grade Reading Assessment FASDA Project District Facilitators GOI Government of Indonesia IR Intermediate Result JSS Junior Secondary School K13 Curriculum 13: the National Curriculum developed in 2013. Kesra Menko Kesra: Coordinating Ministry for People’s Welfare KKG Teachers’ Working Group (secular primary schools) KKKS Principals’ Working Group (K3S) LPMP Lembaga Penjaminan Mutu Pendidikan (Provincial Quality Assurance Institute) M&E Monitoring and Evaluation Madrasah Islamic School MBE Managing Basic Education MBS School-based Management: training for principals, teachers, and communities MGMP Subject Teacher Working Group (junior secondary schools) MI Madrasah Ibtidaiyah (Islamic Primary School) MOEC Ministry of Education and Culture MORA Ministry of Religious Affairs vi MTs Madrasah Tsanawiyah (Islamic Junior Secondary School) MOU Memorandum of Understanding PAKEM Indonesian acronym for ‘Active, Creative, Effective and Joyful Learning’ PISA Program for International Student Assessment PRIORITAS Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators and Students, current USAID Basic Education Project REDI Regional Economic Development Institute, a private Indonesian research company RENSTRA Strategic Plan RKS School Work Plans RPP Teacher Lesson Plan RTI Research Triangle Institute, PRIORITAS implementing partner S1 Sarjana 1 (Degree Equivalent to Bachelor’s Degree) SD Primary School SMP Junior Secondary School TIMSS Trends in Mathematics and Science Study TTI Teacher Training Institutes UIN Universitas Islam Negeri (State Islamic University) UNICEF United Nations Children’s Fund USAID United States Agency for International Development WB World Bank WSD Whole School Development vii EXECUTIVE SUMMARY Background This is the mid-term evaluation of the USAID basic education project, Prioritizing Reforms, Innovations, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS). PRIORITAS is the third of three consecutive USAID projects since 2003 to focus on improving quality of basic education in Indonesia. This project builds on the successes of the previous projects to increase capacity of multiple stakeholders in the education system leading to institutionalization of good practices. PRIORITAS works to improve education governance and management, enhance pre- and in-service teacher training, and revitalize stakeholder involvement to improve coordination, planning, and funding for education delivery. PRIORITAS supports USAID’s education programmatic focus in Indonesia of strengthening teaching and learning processes in basic education reading, math, and science. Purpose With a goal of improved access to quality basic education in primary and junior secondary schools in Indonesia, PRIORITAS is currently nearing the third year of implementation in a five year timeframe. The Indonesian USAID Mission requested an evaluation of PRIORITAS to assess the mid-term performance of the project against the stated objectives and timeline. Conducted by a JBS Evaluation Team of four national and two international evaluators, the evaluation sought to determine the project’s effectiveness and efficiency to date. The results provided stakeholders with evidence-based information as the basis for recommendation of refinements in project implementation to guide PRIORITAS to a successful end achievement of project goals. The evaluators assessed mid-term achievement through an examination of the project’s three components: 1. Strengthened pre- and in-service teacher development programs to increase quality of instruction in classrooms; 2. Improved education management and governance of schools and districts for improved support to classroom delivery of education; 3. Improved coordination of communication, information-based planning and policy￾making, feedback, and use of financial and human resources at all system levels to improve delivery to the lower levels. The evaluation had two main foci: an examination of the changes at the district level which play a pivotal role in the provision and management of education services to schools and changes at the school and classroom levels which lead to improved student performance. Methodology: The JBS Evaluation Team used a mixed methods approach for data collection, employing both qualitative and quantitative research methods which included a literature review, key informant interviews, focus group interviews, on-site visits, quantitative data collection, and classroom observations. The eight questions posed in the Scope of Work provided the evaluation framework. The data collection focus areas carefully examined program outputs and outcomes, programmatic, management, financial implementation issues, the perceptions of stakeholders on current achievements, and the value added by the project. The comprehensive design of this evaluation included visits to five of the seven project provinces, ten project districts, 82 schools, viii 76 classrooms, and ten Teacher Training Institutes as well as interviews with more than 630 principals, teachers, lecturers, government officials, school committees, and project staff members. In partnership with a private Indonesian research firm, the Regional Economic Development Institute (REDI), two quantitative face-to-face surveys were administered in all 43 project districts containing Cohort 1 and 2 schools. One survey was administered to eight identified individuals in each of the district government and education offices (DINAS),1 while a second survey was carried out in three households in each district to assess parental attitudes and perceptions of the effects of PRIORITAS on increased community participation in local schools and improved education provision for their children. General Findings Based on analysis of data gathered from all collection methods, the Evaluation Team determined that overall progress of PRIORIAS was good and mostly on-track but improvement rates varied across provinces and stakeholders. Specifically, the Team determined that PRIORITAS is having a tangible impact on beneficiaries at the district, sub-district, and school levels, the areas where the project focuses most resources and efforts. The impact is specifically noticeable in the improved attitudes and behaviors of principals, teachers, students, and communities. The Team noted the changes in teacher and student behaviors in classrooms due to Active Learning (AL) methodologies but countered this with the note that the real impact on learning achievement as a result of these methodologies was difficult to assess at this time. Furthermore, the evaluation highlighted findings of the following key project stakeholders: 1. Teacher Training Institutes (TTIs). The Active Learning methodology was well-received by Teacher Training Institutes for use by lecturers and students. Pre-service preparation of student teachers has improved but TTIs are having difficulty initiating in-service training delivery at the district and school levels. 2. Teacher working groups (KKG, MGMP). These groups have become more energized by PRIORITAS efforts and are actively engaged in in-service provision. They would benefit from more consistent and adequate financial support from district and school budgets. 3. Teachers: Across the board, teachers were enthusiastic about the changes they see in student learning behaviors as an effect of Active Learning. Measurement of learning as a result of AL methods is still difficult and teachers are generally not well-versed in or comfortable with the wide array of assessment and instructional options available under Active Learning. 4. Principals. Schools have benefitted positively after principals received PRIORITAS School￾Based Management (MBS) training. When principals do double duty as PRIORITAS project facilitators, they have difficulty responding to the responsibilities of both PRIORITAS and their full-time jobs; PRIORITAS training however does enhance their capacities as principals. 5. Project Facilitators: They are one of the most effective interventions by PRIORITAS and have had a positive influence on most project stakeholders. They seem to be, however, overwhelmed by their numerous PRIORITAS responsibilities which they forsake in favor of their own regular jobs (as teachers, principals, and supervisors) in the face of time issues. 1 Information was collected from one member of the local parliament, the Head of Education, Division Head of Primary Education, Division Head of Junior Education, Curriculum Advisor, and the Coordinator of Supervisors as well as a member of MORA education department and the Coordinator of Facilitators. ix 6. School Committees and Communities. School Committees have increased their involvement with schools after participation in MBS training even though the law restricting financial support from parents to schools sometimes restrains communities from making financial contributions to their local institutions. 7. Provincial and district DINAS and MORA. These two levels and divisions of government have increased inter-communication to some degree especially in agreement of the need for improved teacher quality. However, officials in both offices feel they have not been involved with PRIORITAS planning and design to the extent they would have liked. Additionally, PRIORITAS has made limited attempts to bring provincial stakeholders (MORA, DINAS, Teacher Training Institutes (TTI), and Provincial Quality Assurance Institutes (LPMP)) together to explore collaborative ways to address in-service training provision. 8. Central MOEC, MORA and KESRA. The PRIORITAS goal and objectives are in harmony with Government of Indonesia (GOI) education priorities; several project good practices may be included in the future national strategic plan. Some government officials however feel they have had limited involvement in the planning and implementation of the project. As requested by USAID, the Evaluation Team also responded to specific topics of inquiry noted here: • Reading: Children can read as assessed by the Early Grade Reading Assessment (EGRA) in 2013 but have comprehension difficulties. Schools seem to be largely supporting the PRIORITAS focus on development of a ‘reading culture’ but teachers in classrooms were generally not seen to be applying reading instruction to improve student comprehension. • Math and Science Learning: Observations of lessons showed that teachers seem to be using AL methods more effectively in math and science, two subjects strongly focused on by PRIORITAS. The math and science curricula however are still weak and need to be better integrated with the AL methodology. • Gender: Gender does not have a strong focus in either PRIORITAS or with the GOI. Gender issues are only superficially acknowledged in classrooms for example through the use of mixed seating arrangements of boys and girls. Otherwise, educators seem to have limited awareness of gender application in classroom instruction. The growing issue of lower rates of boys’ participation in education (e.g., increased early drop-out and decreased academic performance) is only slowly impacting some educators’ awareness. • Inclusive education (Special education): Similar to gender, this is not a strong focus in either the project or within the government though many teachers noted the presence of special needs children in classrooms and the absence of technical expertise to educate them. • Potential for Sustainability: The Evaluation Team noted several items which they considered potentially sustainable. Considering that PRIORITAS is the third consecutive USAID project to focus on building capacity in individuals and the system, the possibility of continuation of these elements after termination of USAID support is a positive trend. • At the GOI national level, several ministers noted the potential of future funding for national expansion of some PRIORITAS good practices. • At the district level, officials have confirmed commitment to financial support for continuation of facilitator training and use. • Increased ownership of education change processes was observed at the local government level. x • Schools are more pro-active in the use of their own resources to support quality improvement. • With increased confidence and commitment, some School Committees and communities are now lobbying local governments for more funding to strengthen education reform. Recommendations: The Evaluation Team made many recommendations in each of the identified stakeholder categories in acknowledgement of the diversity of needs and varying rates of progress in PRIORITAS’ provinces and districts. The numerous recommendations offer a range of choices for consideration by project leadership. PRIORITAS will need to prioritize implementation of recommendations that have the most impact on the greatest number of project stakeholders and are within the capacity of the project to achieve before project completion in 2017. Only a limited number of recommendations are noted here; the complete list is available in Section Five of the Evaluation Report.  Facilitators: Provide refresher training in the more difficult AL topics such as authentic assessment, gender-balanced instructional strategies, and higher order questioning. This will address teachers’ requests and assist them to expand and refine their use of active learning methodologies in classrooms.  Gender: Modify the gender unit in Module II to be more reflective of the different cultural environments in provinces and districts. As a socially-constructed concept, gender cannot be a “one size fits all” solution, rather needs to reflect the surroundings that it’s applied in.  General: Respond to school-level requests for additional training in: librarianship, office administration, and the design and development of appropriate teaching aids to assist in expanding capacity of school-based education personnel.  Inclusive Education: Develop a unit on instructional techniques for teachers, principals, and parents to assist special needs children already mainstreamed in local schools. This may be included in the AL training.  KKGs and MGMPs: Use KKGs/MGMPs as the main vehicle for in-service teacher training as they are local and provide immediate access to a ‘community of good practice’ for teachers.  Literacy and Language: Upgrade and lengthen the language teaching and learning skills elements in the English and Bahasa Indonesian language training components currently applied in the PRIORITAS training modules. • Principals: Provide a stronger leadership component in MBS training to improve school management effectiveness and efficiency. Currently, MBS training emphasizes organizational and management skills while principals could benefit from training that builds leadership and character capacities.  Reading Culture: Provide more reading and reference books for schools actively engaged in supporting a school reading culture in order to expand the variety and levels of available books.  Teacher Training Institutes: Provide TTI lecturers with refresher training in AL methods, micro-teaching, classroom observation, and teacher assessment in order to ingrain these competencies to both TTI staffs and new student teachers. This will serve to institutionalize active learning methodologies and build TTI capacities. xi EDUCATION AT A GLANCE: INDONESIA Primary Junior Secondary MORA MOEC Population under 14 years old (yo), 2014 estimate2 27,6% Unemployment rate [%]3 6,25% Educational expenditure [% of GDP]4 3,6% Literacy rate: 15 year old and older 20142 94,75% Total number of enrollment (2011)5 30.932.441 11,933,560 13% 87% Total number of schools (2011)4 169,897 48,912 17% 83% Total number of teachers (2011)4 1,856,330 825,032 20% 80% Gross enrollment rate (GER) 20132 107.71% 85.96% NA NA Net enrollment rate (NER) 20132 95.59% 73.88% NA NA School participation rate (SPR) 20132 98.42% (7-12 yo) 90.81% (13- 15 yo) NA NA SPR girls 20132 98.56% (7-12 yo) 91.67% (7-12 yo) NA NA SPR boys 20132 98.13% (7-12 yo) 89.62% (7-12 yo) NA NA Drop-out rate, primary, 20116 NA NA 0.18% 1.61% Drop-out rate, junior sec., 20115 NA NA 0.06% 1.41% Attending primary education: least PCE* 22.1% NA NA NA Attending primary education: maximum PCE** 28.8% NA NA NA Pupil/teacher ratio (all teacher) 16 14 10 17 PISA 2012 score (junior secondary) 5 • Mathematics NA - 357 360 • Reading NA - 380 376 • Science NA - 363 366 * School participation rate (SPR) for 7-12 years old equals the number of students aged 7-12 years old (in any level of education) divided by number of population aged 7-12 years old. ** PCE: Per Capita Expenditure 2 National Development Planning Agency Indonesia, National Statistics Office Indonesia , UNFPA: 2010-2035 Indonesia Population Projection. 3 National Office of Statistics, Indonesia. 4 Data: worldbank.org/indicator/SE, XPP, TOTL, GP, ZS downloaded 4 April 2015. 5 Ministry of Education and Culture. 6 OECD and ADB (2015) Education in Indonesia: Rising to the Challenge. 1 SECTION I: INTRODUCTION Objectives of the Evaluation This study is the mid-term evaluation of the USAID basic education project, Prioritizing Reforms, Innovations, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS). PRIORITAS is the third of three consecutive USAID projects since 2003 to focus on improving quality of basic education in Indonesia. The project is designed to build on previous successes of increasing capacity of multiple stakeholders in the education system leading to institutionalization of good practices. PRIORITAS works with the Ministry of Education and Culture (MOEC), the Ministry of Religious Affairs (MORA), teacher training institutes (TTI) and other in-service training providers, district education offices, school staffs, and communities to improve coordination, planning, resourcing, and training for education delivery. PRIORITAS supports USAID’s education strategic focus on strengthening teaching and learning processes in basic education reading, math, and science in Indonesia. PRIORITAS has three project objectives which support the improvement of quality education: • Strengthened capacity of selected pre-service and in-service training providers to increase instructional capacity of teachers in classrooms leading to improved student achievement; • Improved capacity of provincial, district, and school leaders to improve education management and governance of schools leading to increased support for teacher development and student learning; • Strengthened coordination of communication, information-based planning and policy￾making, feedback, and use of financial and human resources at all system levels to improve education delivery to schools. The purpose of this mid-term evaluation was to assess the progress of implementation of PRIORITAS, examining the current performance of the project against the stated objectives and timeline to improve quality of basic education in primary and junior secondary schools. As per the request of USAID, the evaluation measured the degree to which the project objectives were achieved and identified positive and negative factors which affected the achievement of project goals. The evaluation reviewed programmatic, management, and financial processes to determine constraints which affect current programming. In response to these, practical recommendations are made to facilitate increased efficiency and impact of the project in the next two, and perhaps final years of implementation. Additionally, the evaluation assessed the project’s current achievements to determine the potential for sustainability. As a mid-term evaluation, PRIORITAS performance was assessed from the beginning of the project until the current mid-point of implementation, approximately two and a half years. The evaluation was guided by eight questions requested by USAID in the original evaluation Scope of Work.7 1. To what extent has PRIORITAS achieved its stated mid-point objectives and outcomes in a timely and effective way? 2. Is PRIORITAS on track in terms of meeting its overall end-of-project goals? 7 USAID Indonesia PRIORITAS Mid-term Evaluation Scope of Work, Annex I. 2 3. What aspects of PRIORITAS are proving most and least effective in improving access to quality education in Indonesia for each of the project goals? 4. To what extent have PRIORITAS stakeholders benefitted from the project’s activities and what specific value has been added? 5. To what extent are PRIORITAS resources being implemented and managed efficiently and cost effectively? 6. What evidence is available to indicate that student reading and reading comprehension skills have improved as a result of PRIORITAS interventions? 7. What evidence is available to show that PRIORITAS project activities and results are making progress towards sustainability and replication after the project is completed? 8. How effective has each component of PRIORITAS project been in improving gender equality among students in schools? Methodology The evaluation was conducted over a period of four months from January to April, 2015 by a team of two international and four national consultants. The complexity of such a large decentralized project as PRIORITAS required the collection of data from multiple sources to ensure a wide range and depth of information for triangulation and analysis. Data collection therefore took place both in Jakarta and in the field in targeted PRIORITAS provinces and districts with a wide variety of stakeholders. Data-gathering areas of focus looked at program outputs and outcomes, programmatic, management, and financial implementation issues, perceptions of stakeholders on current achievement and value added by the project, and the degree to which PRIORITAS mid-term targets were being met. The evaluation team used six basic methods to obtain data. These included: 1. A literature review: the Team reviewed an extensive array of PRIORITAS literature including work plans, quarterly reports, PRIORITAS schools’ yearly plans and budgets, and other project documents as well as relevant studies and research in the field, MOEC documents, and donor studies (List of Documents Referenced, Annex II); 2. One-on-one interviews with key stakeholders including representatives from the central government, PRIORITAS project, provincial and district DINAS offices, and schools (Data Collection Guides, Annex III); 3. Focus group discussions with single groups of stakeholders (principals, teachers, school committees, supervisors, lecturers, and facilitators) (Data Collection Guides, Annex III); 4. Two quantitative surveys conducted in all 43 Cohort 1 and 2 districts (Survey Tools, Annex IV); 5. Site observations of Cohort 1 schools in ten districts in five provinces to verify visible evidence of project outputs, e.g., reading spaces, environmental changes, and community engagement; 6. Observations of classroom lessons to assess teacher delivery of math, science, or language lessons (Classroom Observation Guide, Annex V). Additionally, the Team had proposed to conduct a ‘mini-reading assessment’ with a sample of grade three students to measure children’s ability to read and comprehend grade level text. 3 However, USAID told the Team at the initial evaluation planning session that it did not have to conduct this assessment. The two quantitative instruments, a District Education Survey and a Household Survey, were administered independently by an Indonesian research company, the Regional Economic Development Institute (REDI). The company worked closely with the Evaluation Team to identify relevant questions and develop the survey instruments first in English and then in Bahasa Indonesian. After field testing the instruments in a PRIORITAS district in East Java, the company sent trained enumerators to all 43 PRIORITAS Cohort I and II Districts to administer the instruments. The surveys were administered face-to-face and data were collected electronically by tablet. For the District Education Survey, enumerators contacted one member of the district parliament and the following members of the District Education Office: the Head of Education, Division Head of Basic Education, Division Head of Junior Education, Curriculum Advisor, and the Coordinator of Supervisor. Additionally, a member of the MORA Education office and the Coordinator of Facilitators were interviewed. All respondents were knowledgeable of the PRIORITAS project; data were collected from a total of 344 respondents. The Household Survey was administered to three heads of households randomly selected from families situated close to PRIORITAS schools in each of the project’s 43 districts; families had to have children attending a project school. A total of 129 parent respondents were interviewed in this survey. During the evaluation, the Evaluation Team made contact with more than 630 stakeholders (Table 1) including: national government staffs in multiple departments of the Ministry of Education and Culture (MOEC), the Ministry of Religious Affairs (MORA), and the Ministry for People’s Welfare (MENKO KESRA), representatives of those ministries at the provincial and district and governor’s offices, District Education Board members, university staffs, central, provincial, and provincial and district PRIORITAS project staffs, donor representatives, school supervisors, principals, teachers, and school committee members (List of Interviewees and Contacts, Annex VI). In the course of the fieldwork, the evaluation team visited a total of 82 PRIORITAS Cohort 1 schools in five provinces (Banten, Aceh, North Sumatra, South Sulawesi, and East Java) and ten districts: Serang, Aceh Jaya, Bener Meriah, Labuhan Batu, Medan, Makassar, Blitar, Maros, Pamekasan, and Bantaeng (Field Schedule, Annex VII). The schools consisted of 55 public schools and 27 madrasahs (both public and private). Additionally, 12 of the schools (eight primary and four junior secondary) were comparison schools at which baseline data were collected in the beginning of the project in 2012. Also, 12 of the 82 schools were ‘good practice’ schools or Lab schools, affiliated with TTIs as locations for pre-service preparation and demonstrations of good teaching practices. Table 1: Interviewees by Stakeholder Group Government officials: central Government officials: provincial and district 14 48 Government officials total 62 Teachers 229 Principals 75 University staffs 70 Provincial facilitators District facilitators 23 93 Facilitators total 116 Donor members 6 LPMP members 37 School committee members 19 PRIORITAS staff 19 USAID staff 3 TOTAL 636 4 Constrains in Undertaking the Evaluation A number of constraints were operating during the undertaking of this evaluation which may have inadvertently affected the findings of the Team. First, the scale and geographic spread of the PRIORITAS project is very large across a vast archipelago of many thousands of islands. Despite best efforts to visit many schools and classrooms, the Team was realistically able to visit less than one percent of project schools within the limited time of the evaluation. Additionally, a limited time at each school allowed the Team to only form impressions of PRIORITAS affects. Moreover, the presence of the Evaluation Team interfered with best efforts to try to document "normal" classroom behaviors in schools. By extension, the sample of PRIORITAS schools visited had to be ad hoc rather than random in view of the need to travel between widely-separated schools and still arrive within school hours. The Team relied heavily on PRIORITAS province and district coordinators to arrange visits and provide guidance to project schools which by necessity and choice limited the randomness of school selection. The Team insisted however on including comparison schools within the sample as a basis on which to judge the extent of PRIORITAS' effects. Despite best efforts to meet with relevant provincial and district government and education personnel, access to them was sometimes difficult given their own busy schedules and availability. As a substitute, the Evaluation Team sometimes had to meet with second or third tier staff members who may not have had the best knowledge of the Governor’s or Head of Division’s perspective on PRIORITAS’ efforts in education. Lastly, with all expectations of seeing clear signs of PRIORITAS progress, the Team was unprepared to find so many education personnel who had previously been exposed to “active learning” (called PAKEM in the field, a term which is itself a holdover from UNICEF efforts in the 1990s) and school-based management training from previous projects, donors, and the DINAS. For this reason, the Team felt that the specific effects of PRIORITAS activities were not always easy to distinguish from the possible lingering effects of previous projects and programs. Structure of the Report The report is organized into five main sections. This introduction section provides an overview of the objectives of the evaluation along with the eight questions specifically requested by USAID to guide the evaluation of the current progress of PRIORITAS. Additionally, this section presents the methodology the Evaluation Team followed to conduct the task as well as the constraints which might have influenced the rigor and reliability of the evaluation findings. Section Two provides background information on the current state of education in Indonesia as the basis for USAID involvement in the sector leading to a description of Agency’s current basic education project, PRIORITAS. Section Three presents the evaluation findings, addressing each of the eight questions posed by USAID; this section reports the evidence used to determine the project’s current progress. Section Four provides the Evaluation Team’s conclusions about PRIORITAS laid out in a list of eleven points. The report finishes with Section Five which provides an extensive list of recommendations to assist the PRIORITAS team to make informed decisions about the last two years of project implementation. Data collection tools, instruments, sites, and contacts are contained in Annexes I to VII while Annexes VIII to X contain select data from the classroom observations, surveys, and the interviews with school-level individuals. 5 SECTION II: BACKGROUND Education in Indonesia The fourth most populous country in the world, Indonesia is a land of unfulfilled promise and immense challenge as it strives to become a regional leader and global player. In a country that has roughly 55 million students, three million teachers, and 236,000 schools, the education system is under intense pressure to produce the human resources necessary to drive economic growth and regional aspirations. Education is central to the Indonesian Government’s development agenda; education spending has increased significantly since 2000. The government is mandated by the Law on National Education (No. 20/2003) to allocate 20 percent of expenditures on education to ensure free education to all citizens. The long-term goal of the government is to build quality human resources capable of playing a significant role in achieving the country’s aim to be a global- and knowledge-based economy at the regional level by 2020 and at the international level by 2025. To achieve those long-term goals, shorter-term goals are focused on increasing the availability and affordability of education services, improving educational quality and relevance, and ensuring equality and universality of access to quality services. While progress has been made, achievement is uneven. Primary net enrollment reached near universal levels by 2005. Lower secondary completion rates rose from 63 percent to 78 percent between 2002 and 2012; these accomplishments are in stark contrast however to upper secondary education which 45 percent of the eligible population attends.8 Junior secondary enrollments, the gateway to higher levels of learning, remain a key challenge for the country; only 55 percent of children from low income families are enrolled and this level remains under-invested.9 As of 2014, tertiary education had a gross enrollment of 12 percent.10 Sub-standard education quality persists across the system with issues of quality of teachers, ineffective teaching and learning methodologies, and poor management and governance of education delivery, impacting system results. An Early Grade Reading Assessment (EGRA) conducted in 2014 with 4,800 second graders across four grouped “regions,” found that children tended to read at relatively high levels; only 5.9 percent of tested children were classified as non￾readers.11 Comparing Indonesian student performance on an international level, results of the Program for International Student Assessment (PISA) over the years show that Indonesia is one of only eight participant countries (of 65) whose students’ reading results in the PISA have improved significantly between 2000 to 2009 (8.4 %) while the achievement gap between the highest and lowest performing students nationally was consistently narrowed.12 Conversely, results in the 2009 assessment showed that the country’s 15-year-olds ranked 57th (out of 65 8 Overseas Development Institute (2014) Towards Better Education Quality: Indonesia’s Promising Path. London, England:ODI. 9 World Bank (2014) World Bank and Education in Indonesia: Research on Education in Indonesia. http://www.establishmentpost.com/aec-2015-indonesian-education-reform/ 10 Ibid World Bank (2014) 11 RTI (2014) Indonesia 2014: the National Early Grade Reading Assessment (EGRA) and Snapshot of School Management Effectiveness (SSME) Survey. Jakarta: USAID Indonesia. 12 Ibid Overseas Development Institute (2014). 6 participant countries) in reading and 59th in mathematics, highlighting weaknesses in math, science, and technology, the skills of the 21st century.13 At the apex of the government’s efforts to improve student achievement is a focus on the quality of the teaching force: the selection, training, management, and incentivizing teachers. The Teacher and Lecturer Law in 2005 (UU14/2005) set into place a comprehensive program of reform to raise the quality of the teaching corps. As the foundation for the law, a conceptual framework for teacher development included the key components of certification, continuous professional development, and appraisal and career development.14 Certification brought with it a doubling of teachers’ incomes. As of 2013, approximately two thirds of Indonesia’s teachers had attained four year degrees for certification. 15 Teacher certification reforms however have not yet demonstrated an impact on upgraded teacher classroom skills and commensurate improved student outcomes as found in recent research by the World Bank and the Ministry of Education and Culture.16 Despite the law, provision of a comprehensive, viable system of continuous teacher development and in-service delivery remains fragmented and ineffective. Decentralization of authority has placed responsibility for education planning, provision, and financing with District Education Offices as autonomous entities. Capacity of local education authorities to effectively manage the process of improving education delivery at school level however is as yet limited. Linkages and coordination between provincial and district offices, training providers, and schools are weak; teacher deployment, on-going teacher quality improvement and monitoring, and resource management are seriously affected. The sheer numbers of teachers who need professional development through in-service training and certification coupled with weak systems and management capacity makes upgrading of teacher quality and resultant improved student performance a huge challenge in the country. USAID Assistance to Basic Education USAID has a long and close partnership with Indonesia, a rising economic power and growing regional leader in the Pacific. Economic growth has exceeded six percent in recent years while democratic governance strengthens across the vast archipelago. With a population of 240 million, however, Indonesia is home to about 40 million people still living below the international poverty line of $1.25 a day.17 Decentralization of services, generally a positive democratic development, has not evened out access to health and education. Within this setting, USAID assistance to Indonesia is directed at advancing the country’s national and global development goals by focusing on strengthened governance, improved service delivery especially to the most vulnerable, and collaborative achievement in science, technology, and innovation. USAID has a history of providing assistance to quality improvement of Indonesian basic education. From 2003 until 2005, USAID sponsored the Managing Basic Education project 13 Cited in RTI (2014). 14 Chang, M.C., S. Shaffer, S. Al-Samarrai, A. Ragatz, J. de Ree, and R. Stevenson (2014) Executive summary: Teacher Reform in Indonesia: The Role of Politics and Evidence in Policy Making. Jakarta, Indonesia: World Bank. 15 As quoted in H. Abbas, ‘Inconvenient truths about teacher certification program.’ Jakarta Post, April 27, 2013. http://www.thejakartapost.com/news/2013/04/27/inconvenient-truths-about-teacher-certification-program.html 16 Ibid World Bank (2014). 17 USAID Indonesia (2013) Investing in Indonesia: A stronger Indonesia advancing national and global development. Country Development Cooperation Strategy. Jakarta, Indonesia: USAID. 7 (MBE) which prioritized building capacity for local government service delivery. This four year project worked closely with local stakeholders to plan, manage, and implement basic education. MBE resulted in changes in classroom environments, improved school management, and better teacher performance through use of active learning methodologies. MBE was followed by the three-part Decentralized Basic Education (DBE) project, DBE 1, 2, and 3, which began in 2005. Partnering with primary and junior secondary schools in more than 50 districts in seven provinces, the DBEs continued the focus on improved quality of decentralized education provision. Collectively, the projects improved education accountability and oversight, enhanced pre- and in-service teacher training and certification programs, and supported new initiatives in Kindergartens, inclusive education, and tertiary professional development. The PRIORITAS Project With the end of the DBEs in December, 2012, the U.S government sought to consolidate gains made in the sector and build on the foundation of improved capacity of local actors left by previous projects. The project, Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) was designed to continue the focus on decentralized education provision. Partnering with the Government of Indonesia, the overall goal of the five-year $83.7 million project is to expand access to quality basic education. In this role, PRIORITAS continues to focus on building system capacity at the provincial, district, and school levels leading to strengthened teaching and learning processes at the classroom level. A project focus on literacy, science, and math aligns with USAID’s global education strategy and the efforts to build an educated workforce for the 21st century. Donor Involvement in the Sector The Indonesian education sector has seen committed and long-term involvement of donors, focused on building system capacity across the board to improve quality of education. The World Bank’s recent six-year project, BERMUTU, worked with the GOI to inset a teacher professional management system for up-grading teachers to national certification levels. The project supported a system of university accreditation and supplied grants to stimulate and support the work of teacher and principal working groups (KKGs and KKKSs). Development assistance from Australia, currently under the Department of Foreign Affairs and Trade (DFAT), has undergone a radical re-organization in delivery of education aid by out-sourcing responsibilities to national entities and some Australian private sector organizations. The organization’s reduced education support focuses on improving teacher and school management capacity of principals and grants to madrasah for self-improvement. UNICEF’s activities focus on provision of services to the most vulnerable and marginalized women and children in Indonesia. The Agency’s future thrust will improve early childhood education services in Papua where work with local NGOs has already begun. In interviews, all three agencies mentioned the positive contributions of previous USAID programs to improve decentralized delivery of education through focused attention on district representatives, principals, and teachers. All requested closer cooperation and coordination with USAID in program planning and activities in the areas of common interest to build technical capacity of the GOI leading to sustainability of good practices. 8 SECTION III: FINDINGS Responses to USAID’s eight questions from the original Scope of Work are contained in this section. Findings reported here are based on analysis of all the data gathered from six collection methodologies; sources are cited in footnotes and located in the Annexes. Question I: To what extent has PRIORITAS achieved its stated mid-point objectives and outcomes in a timely and effective manner? From a field perspective, a review of PRIORITAS baseline evaluation data presented in the Performance Management Plans (PMP) is necessary in order to determine the degree of achievement from the start of the project to the current mid-point. A determination on timeliness and effectiveness of achievement of mid-term objectives however was difficult without more information on end-of-project targets which the Evaluation Team was not able to locate. Table 2 presents the 2012 baseline information and the most recent 2014 measurement as the basis for analysis of project progress to date. The framework for analysis has two dimensions: 1) Commentary and 2) Field observation. Table 2: PRIORITAS Performance Indicators: Objective 1 Performance Indicators (From the project PMP) Baseline (2012) 2 years (2014) % Increase Objective 1: Strengthen capacity of pre- and in-service training provision, leading to improved capacity of primary and junior secondary school (jss) teachers in reading, math, and science instruction. Early grade teachers demonstrated good practices in teaching and assessment and reading materials used. IR2 Early grade teachers demonstrate good practices in teaching and assessing reading Partner 13.0% 66.5% 53.5% Comparison 16.0% 37.7% 21.7% IR618 Early grades reading materials are regularly used. Partner 21.7% 50.0% 28.3% Comparison 24.3% 39.4% 15.1% All teachers demonstrate good practices in teaching and assessment and reinforce reading skills. IR1 Teachers demonstrate good practice in teaching and assessment. Partner Primary 23.9% 60.7% 36.8% Comparison Primary 26.8% 28.9% 2.1% Partner JSS 18.4% 58.0% 39.6% Comparison JSS 28.5% 29.3% 0.8% IR3 Teachers of all subjects support the development and reinforcement of subject reading skills. Partner Primary 8.7% 53.1% 44.4% Comparison Primary 10.9% 32.9% 22.0% Partner JSS 8.7% 42.0% 33.3% Comparison JSS 14.5% 28.7% 14.2% Outputs: Student Achievement IR5 Students demonstrate positive learning behaviors. Partner Primary 16.7% 80.7% 64.0% Comparison Primary 19.9% 51.4% 31.5% Partner JSS 16.9% 82.0% 65.1% 18 IR7: Students’ performance in district and/or national examination improves. This IR is no longer used as per the PRIORITAS Director of M&E. 9 Comparison JSS 26.6% 54.8% 28.2% IR8a Early grade students demonstrate that they can read and understand the meaning of grade-level text. 50.4% 71.1% 20.7% IR8b Performance of students in grades 4 and 5 in reading, writing, math, and science improved. Gr 4 Reading Partner 43.1% 47.1% 4.0% Comparison 41.1% 44.1% 3.0% Grade 4 Writing Partner 42.5% 44.4% 1.9% Comparison 39.8% 40.0% 0.2% Grade 4 Math Partner 40.6% 43.7% 3.1% Comparison 40.7% 43.1% 2.4% Grade 4 Science Partner 35.5% 42.3% 6.8% Comparison 33.0% 40.5% 7.5% IR9 Performance of students in grade 8 in reading, writing, math, and science improved. Grade 8 Reading Partner 63.9% 69.7% 5.8% Comparison 65.9% 68.0% 2.1% Grade 8 Writing Partner 50.4% 52.5% 2.1% Comparison 47.4% 46.9% -0.5% Grade 8 Math Partner 33.4% 43.8% 10.4% Comparison 32.6% 40.7% 8.1% Grade 8 Science Partner 38.6% 42.3% 3.7% Comparison 38.2% 40.6% 2.4% Leadership, working groups, dissemination, and GOI participation IR16 Instructional leadership in schools is improving. Partner Primary 10.9% 35.6% 24.7% Comparison Primary 2.9% 26.9% 24.0% Partner JSS 2.9% 26.9% 24.0% Comparison JSS 15.9% 23.4% 7.5% IR17 Teacher working groups are more effective and good quality training is being improved. 35.6% Partner KKG 31.1% 60.9% 29.8% Comparison KKG 44.6% 54.0% 9.4% Partner MGMP 30.9% 30.9% 0% Comparison MGMP 27.65 39.6 12.0% IR19 Project programs are disseminated in line with QA standards. 1.3% IR20 Total amount of non-USG funds (in US$) used to disseminate the project programs 649,351 (2013) 1,096,7 96. 371% TTI Activities in supporting active learning implementation.19 IR10 Lecturers in TTIs implement active learning. 40.%% IR12 TTIs offer a more practice-oriented practicum. 3.7% Commentary: Objective 1 A review of the data for Objective 1 in Table 2 reveals that nearly all the IRs listed show a percentage of improvement, in some cases to a large degree. For the most part, only the student achievement outputs do not show significant degrees of improvement at this time although the Evaluation Team understood from project staff that measurement of student performance would take place shortly after the performance of the mid-term evaluation. The Team reacted specifically to results of IRs 2 and 5 as the objectives being measured involve many complex 19 This indicator is currently being measured by PRIORITAS. Most recent data are not available at time of writing. 10 classroom behaviors which would reasonably take longer than one year to improve to the levels shown in the table. This led the Team to wonder how these many complex teaching behaviors were measured by the project. Interviews and discussions with project field M&E staffs did not always provide satisfactory answers to the many questions of the Team concerning the measurement of complex learning-teaching behaviors in the Indonesian classroom context. Field Observation During the observation of 76 classes, the Team saw little evidence of systematic assessment of reading and the use of more than conventional methods to monitor and correct reading errors in students. In only several cases of lessons observed did the Team note teacher inputs to correct student reading errors; approximately ten teachers were seen to apply some kind of assessment to work done by students. While positive learning behaviors of children were frequently observed in and outside of classrooms in most schools visited, the Team was however unable to assess performance improvements of children in math, science and Bahasa as no baseline criteria was available from which to compare performance in classrooms. The Team did see and was impressed however by the strong instructional leadership shown in many schools, especially where principals had been trained by PRIORITAS and were functioning as facilitators. Table 3: PRIORITAS Performance Indicators: Objective 2 Objective 2: Improved education management at the provincial and district level leading to improved human and financial resources to school. Baseline (2012) 2 years (2014) % Increase School Level 2R1 Schools produce annual budget plans in a transparent and participative manner. Partner Primary 17.4% 22.2% 4.8% Comparison Primary 14.1% 23.9% 9.8% Partner JSS 11.6% 32.8% 21.2% Comparison JSS 27.5% 26.6% -0.9% 2R2 Increased percentage of schools that involve parents in teaching and learning and improving school environment (Prim school only). (Previously was: ‘increased parent and community participation in activities which focus on teaching and learning and/or improving the school environment.’) Partner Primary 27.2% 65.5% 38.4% Comparison Primary 30.4% 44.3% 13.9% 2R3 School managers initiate activities to create a school reading culture. Partner Primary 30.4% 82.2% 51.8% Comparison Primary 33.7% 61.4% 27.7% Partner JSS 17.4% 73.1% 55.7% Comparison JSS 23.2% 35.9% 12.7% District Level20 2R4 Districts use the personnel management tool for improving the efficiency of the education system. - 2R5 Districts develop needs-based in-service training plans and 87.0% 20 Data analysis for this level is currently underway by PRIORITAS. 11 collaborate with provincial training providers to implement these plans. (Previously was: Districts collaborate with provincial training providers to implement their in-service training plan.)21 2R6 Districts use financial analysis to allocate more resources to quality improvement. - 2R7 Districts have better reading programs. (Previously was: Districts have improved EMIS.)22 21.7% Commentary: Objective 2 The inclusion of data from participating partner and non-partner madrasas at the primary and junior secondary levels are not obvious in Table 3. Disaggregation of data should assist to demonstrate important performance differences between secular and religious schools. Future project plans to measure changes in institutional behaviors may need to take into account the impact of political variables on the different education levels and ministry institutions especially with regards to the district decentralization process. Performance differences among schools may lead to a better understanding of the changes taking place at the provincial and district levels and suggest appropriate project interventions leading to increased institutionalization of practices. Field Observation The Team interviewed principals to discuss management, planning, and budgeting issues and were quite impressed. Generally, principals who had received MBS training and mentoring had prepared yearly plans and budgets for their schools. The increased involvement of parents and School Committees in school development activities was probably one of the most positive results observed from MBS training. Parent members of School Committees were very enthusiastic and committed to working with principals to improve their schools. In districts managed by effective and committed District Education Heads, for example in Bener Meriah, Bantaeng and Blitar, the Team saw some evidence of district use of PRIORITAS teacher deployment and financial management tools and guidance. The Team was impressed by the generally high level of performance of teachers and principals seen in state-supported primary and junior secondary madrasas. Table 4: Performance Indicators: Objective 3 Objective 3: Strengthened coordination between all levels of the Education System, improving linkages to involve all key education institutions for increased planning, provision, and evaluation. Baseline (2012) 2 years (2014) % Increase Provincial Level23 3R1 Provincial government coordinates the management and provision of education staff development. 3R2 Provincial government channels funds for educational staff development. 3R3 Provincial government holds Public Policy Forums to consult on policies and plans for improvement in education. 6 21 Information taken from Baseline Monitoring Report. No further data available at this time. 22 Information taken from Baseline Monitoring Report. No further data available at this time. 23 The project Director of M&E claimed that an internal disagreement over this indicator was underway because the project did not work at the provincial level. The M&E Department has ceased to measure this indicator now. 12 (Previously was: Provincial government shares information and recommendations on the implementation of educational policies.) Commentary: Objective 3 A challenge is always present when measuring complex behavioral and institutional variables such as those in Objective 3. Provincial government variables may only become relevant when the project begins more intense engagement with stakeholders at this government level. These indicators should challenge PRIORITAS’ M&E professionals to think beyond the need to measure output variables to the perspective of measuring complex process variables. Developing alternative measures of institutional process variables may need more rigorous and time￾consuming qualitative approaches requiring a different level of M&E experience and expertise. In any case, assessing behavioral change at the provincial level is an important goal in a project such as PRIORITAS which is focused on improving system performance. At a minimum, baseline information should have been collected as the foundation to measure future change. Field Observation The Team visited and interviewed senior officials in five provinces; in two provinces, District BUPATIs were met. In three of five provinces, senior provincial level officials had more than a passing knowledge and appreciation for PRIORITAS’ work. Where education was a top provincial priority, officials commented positively on the need to get provinces more involved with the project. In Aceh and South Sulawesi, provincial officials were contemplating earmarking funds to adopt and adapt the best of PRIORITAS practices to meet province-wide needs for basic education quality improvements. The Team found strong support in only one provincial office for the project’s inputs to help districts resolve their teacher deployment and shortage problems. Question II: What aspects of PRIORITAS are proving most and least effective to improve access to quality education in Indonesia for each of the PRIORITAS project goals? The Evaluation Team conducted multiple interviews, focus group discussions, and observations with all relevant PRIORITAS stakeholders, including government officials from three ministries at the central, provincial, and district levels, PRIORITAS staffs, service providers, school staffs, and community members. In building a system, all stakeholders play a role; PRIORITAS made commendable efforts to include all relevant actors in project activities. The information the Team received was comprehensive but in many cases, overlapping. In responding to this question therefore, the groups identified under each objective are not necessarily all that have input to it, rather they are the stakeholders considered by the Team to have the most bearing on achievement of the PRIORITAS objectives. In each case, the Team noted which factors were considered most and least effective to attainment of the objective. Objective 1: Strengthened capacity of pre- and in-service training provision, leading to improved capacity of primary and junior secondary teachers in reading, math, and science instruction. From the Central Government officials down to supervisors and principals in schools, all stakeholders echoed the need to “improve the quality of teachers.” While pre-service training was clearly understood to be the responsibility of teacher training institutes, the responsibility for 13 provision of in-service training was not so clearly understood. The Evaluation Team found that multiple forms and topics of the training were used inter-changeably and understood by various stakeholders as a means to “improve the quality of teachers,” i.e. fulfil in-service training needs of teachers. The organizations and groups identified here were judged in terms of factors most and least useful to date for improving the instructional capacities of teachers. Teacher Training Institutes The Evaluation Team visited ten PRIORITAS partner training institutes, both in public and religious-based universities. All provided pre-service training. Some also stated that they participated in “in-service training,” which most often took the form of providing national certification coursework on campus to uncertified teachers. At each university, the Team made a short visit to the Rector for protocol purposes. The Dean of Education was asked to discuss the institution’s vision of improved quality of education, issues in provision of both pre-service and in-service teacher professional development, and the effects of PRIORITAS to improve the capacity of the institution as a service provider. Additionally, in each institution, the Team conducted a focus group with lecturers who were involved with PRIORITAS to hear their opinions of the project’s contribution to building personal and institutional capacity. Most effective: • University staff appreciated the training and mentoring support to lecturers from PRIORITAS in active learning methodologies. This had a twofold effect in making university lecturers better educators in their own right as well as improving the capacity of the pre-service training department to produce better future teachers. Lecturers also noted that the active learning methodology was now being applied within and across university departments. For example, the State Islamic University of North Sumatra has 32 lecturers involved with PRIORITAS while the State University of Makassar has more than 50 lecturers in different departments trained in PRIORITAS AL methods. • Lecturers emphatically endorsed the tied training-mentoring process as a very beneficial practice learned from PRIORITAS. Interviews with university lab and partner schools revealed that university lecturers have increased their mentoring of practice teachers when they are out in classrooms. Additionally, lab school principals reported that they have adopted the practice of mentoring their own staffs. • Local schools understand the importance of better-prepared teachers to deliver improved teaching and learning in classrooms. Several universities noted that more local schools are now approaching them to partner in technical assistance for teacher professional development. • Designation of some lecturers as PRIORITAS provincial facilitators allows these individuals to become highly competent training providers of active learning techniques to peers and pre￾service students as well as MOEC, DINAS, and MORA representatives. They have the potential to form the nucleus of future university consulting services for in-service teacher development. Least Effective • The Evaluation Team consistently found no contact or services received from TTIs in outlying districts or provision of in-service training services to schools outside of their own localized partner schools (only UNI Makassar has a satellite campus in another town thereby 14 offering a wider base of services.). PRIORITAS has not yet assisted TTIs to expand their reach for service provision although several TTIs mentioned that they do have Memorandum of Understandings (MOU) with district education offices, creating the potential for future service as district trainers. Several of the TTIs mentioned that they would like to be, and thought the goal of the TTI was to be, a service provider but were not certain how to initiate such an off-campus service. • The Evaluation Team found minimal cross-over of subject specialist training within the focus on Active Learning done in pre-service training preparation of primary education teachers despite the PRIORITAS focus on math, science, and literacy; therefore primary teacher development seems to include minimum preparation in these necessary academic topics. Outside of UIN Sunan Ampel in Surabaya, Evaluators found little evidence of a strong focus on literacy and teacher preparation in reading. • The Classroom Active Research (CAR) intervention of PRIORITAS seems to be too lightly and thinly applied to have a solid impact on improvement of individual TTIs’ capacity to do research or apply results in teacher preparation. Provincial Quality Assurance Institute (LPMP) The Evaluation Team interviewed LPMP staff in each of the five provinces during field work. LPMPs function as training providers for teacher quality improvement as well as train principals and supervisors for Curriculum 13 (K13) and national certification. As per their mandate, LPMPs are not able to work in schools; therefore this precludes their ability to perform requisite mentoring of teachers in classrooms and monitoring of quality of teacher improvement at school level. This limitation is in direct contradiction to PRIORITAS’ efforts to strengthen a decentralized system of training provision. As verified by numerous interviews at the school level, LPMPs have a limited scope to reach a significant quantity of teachers with training. Most effective: • LPMPs value PRIORITAS training Modules I and II for their focus on active learning and have adopted them into the institutes’ Curriculum 13 training curricula. This may assist to institutionalize good practices of active learning and sound school management, enabling greater dissemination of these practices to a wider audience in individual provinces. Least effective: • According to an early work-plan, PRIORITAS originally planned to consider LPMPs as training service providers along with TTIs at the provincial level but seems to have had little to do with them since the project began, having trained only a limited number of instructors in Modules I and II in each institute (e.g. 6 of 24 in Sumatra and 4 of 27 in East Java). • PRIORITAS seems to have done little to include or facilitate LPMPs in discussions to improve training provision to the district level. Several LPMPs noted they felt left out of PRIORITAS activities but were eager to join in if invited. • After training of LPMP instructors, PRIORITAS did not provide follow-up mentoring, thus limiting the exposure LPMP instructors had to sound practices of good training methodology. Teacher Working Groups: KKGs (primary) and MGMPs (junior secondary) The KKGs and MGMPs appear to have a key role in improving the quality of teachers. While a comparative assessment of increased activity of the KKGs and MGMPs was not in the 15 Evaluation Team’s Scope of Work, the frequency of positive responses the Team received during interviews of increased KKG/MGMP activity prompted Evaluators to consider the potential they have as major mechanisms for teacher training service delivery. Most effective: • The attendance of PRIORITAS facilitators as trainers and resource specialists at KKGs and MGMPs was commonly cited by interviewees as an effective practice. This allowed all teachers collectively to hear, discuss, and receive specific topic knowledge from the facilitators. KKGs and MGMPs thereby become mechanisms for dissemination of good practices across school clusters. • Prompted by the School-based Management (MBS) training, several principals noted they would encourage increased action from lack-luster KKGs (or try to revive dead ones!). In the meantime, many principals and teachers reported the initiation of ‘mini-KKGs’ i.e. internal KKGs, to share good practices among all teachers at a single school. This was essential when not all teachers at a school were trained by PRIORITAS and the principal wanted the whole teaching staff to benefit from AL training. • Educators from universities to school level and at the District Education Offices understood and commented on the value of these groups to support teachers at schools. Teachers and principals from more than 80 percent of observed schools noted that relevant KKGs or MGMPs were operating regularly, in many cases as a result of PRIORITAS intervention.24 Teachers As the largest stakeholder group in the PRIORITAS project, the Evaluation Team spent much time interviewing and observing teachers in classrooms. The Team observed 76 mathematics, science, and language lessons taught by teachers in grades one to nine25 and conducted group interviews with more than 200 teachers of project primary and junior secondary schools. The most and least beneficial contributions these individuals made to the system are noted here. Most effective: • Teachers were almost universally positive about the Active Learning methodology promoted by PRIORITAS even in cases where individuals were familiar with these methods from many years before. Those who were willing to adapt and adopt active learning methods seemed to have not only renewed their own interest in teaching but stimulated increased interest among students to learn. Where AL methods were being applied in math and science for example, students seemed to exhibit fuller enthusiasm and engagement in learning. • While teachers generally were not able to prove increased learning achievement as a result of students using active learning techniques, teachers were equally convinced that engagement in active learning had increased students’ motivation and ability to learn. They anticipated better academic scores in the future. • Mentoring after training was generally noted by teachers as an effective practice and the one that distinguished PRIORITAS active learning training from other previous PAKEM efforts. • Teachers who were well-supported by strong leadership of principals were most likely to make an effort to change their teaching methods by adopting the AL methods they had learned during PRIORITAS training. 24 School Interview Data, Annex X. 25 Classroom Observation Tool, Annex V. See Annex VIII for results of the observations. 16 • Some mathematics lessons observed in the early grades of one to three did reflect the PRIORITAS math-for-life practical approach which seemed to have a positive impact on student enthusiasm. • The most effective trainers and mentors of teachers (i.e. facilitators) were themselves teachers. Many teachers noted that supervisor-facilitators or principal-facilitators were not always competent or willing to demonstrate examples of active learning techniques. Least effective: • The Evaluation Team saw very little emphasis given to literacy instruction in primary schools. A review of the 12 integrated lessons seen in grades one to three did not reveal any special attempt on the part of teachers to instruct reading with the use of progressive teaching-reading methods. Language classes generally (both Bahasa Indonesian and English) were poorly taught using outmoded methods of language instruction. • While science was generally observed to be well-taught by teachers trained to teach science, in some cases teachers seemed to be confused by the recent Ministerial order to revert back to the use of the 2006 curriculum from the recent 2013 version. This change, coupled with teachers’ lower competency in science, may have accounted for the many observed lessons being taught from textbooks. An absence of laboratories and equipment exacerbated this tendency to teach science theoretically and not practically. • The Evaluation Team was frequently at a loss to know which indicators were being used to assess AL implementation.26 In over 80 percent of the lessons seen, the most tangible evidence of AL methods in use was in the organization of students (by groups) and the presence of student work on the walls. The Team felt that generally teachers either had not been exposed to nor were comfortable using the diverse array of techniques and possibilities available in active learning methodologies; authentic assessment and higher order questioning seemed to pose the most difficulties for teachers. • Newly-trained teachers in many districts complained that they were not being mentored with sufficient frequency or quality following PRIORITAS training; therefore teachers were not able to receive the immediate assistance they needed to resolve methodological problems. • Based on observation of teacher classroom behaviors, the Evaluation Team had difficulty distinguishing the specific effects of PRIORITIS training from previous PAKEM training due to the impact of prior efforts using similar methodologies. Some teachers in all provinces claimed they received training and knowledge from other projects and programs implemented for example by Save the Children, UNICEF, DFAT, USAID/DBE, the World Bank, and the DINAS. • An overriding weakness of the intervention being applied by PRIORITAS is the limited focus on validating learning achievement gains after application of AL methods. The use of the national examination to measure learning gains from AL interventions stimulated concerns of some educators but teachers in classrooms feel they are not competent nor allowed to use alternative methods of student assessment. Objective 2: Improved education management and governance at the provincial and district level leading to improved human and financial resources to schools. 26 See the Classroom Observation form to view the list of indicators and the process used by Evaluators to measure AL in classroom observations. Annex V. 17 Analysis in this section was based on information from interviews with the following stakeholders: provincial and district DINAS, provincial and district MORA, principals, supervisors, communities, and project facilitators. The survey of District Education Offices, accompanying the qualitative data collection for this report, makes a deeper examination of the increased capacity of district education officers as a result of PRIORITAS efforts.27 Provincial and district DINAS The Evaluation Team conducted in-depth interviews with senior DINAS officials in all five of the target provinces as well as in ten project districts. Most effective: • In all the provinces visited, provincial and district DINAS representatives testified to having slightly more inter-cooperation across levels now. The Evaluation Team was unsure however if this was as a result of PRIORITAS or the current tendency in the education sector in anticipation of the law in 2016 giving provincial offices more responsibility. Senior officers in the Aceh, Sulawesi and East Java provincial offices said they contribute some funding to the district level for education construction and infrastructure development and scholarships for socially-marginalized students, confirmed by the district DINAS heads of education in most of the districts visited. • District DINAS officers expressed appreciation for the frequent visits and contacts by PRIORITAS staff. Certainly establishment of close working relationships by PRIORITAS district and provincial facilitators with the DINAS District Heads of Education and their staff builds capacity and contributes to successful achievement of district program objectives. Least effective: • Provincial offices complained that PRIORITAS hadn’t shared the planning of important project activities with them prior to implementation. The East Java Provincial Office’s Department of International Affairs went so far as to state that continuation or extension of PRIORITAS activities in the Province would depend on a separate MOU with that office. • Some degree of frustration was expressed in four provincial education offices over limited sharing of PRIORITAS activities in their provinces. Contact at that level seemed to be limited to occasional invitations to attend “Showcases” and other ad hoc meetings. Provincial and District MORA As with the DINAS, the Evaluation Team visited provincial and district MORA offices in five of the target provinces and ten districts. The Team conducted in-depth interviews with senior MORA officials including directors, commissioners, and senior supervisors in education offices. Most effective: • The MORA Offices appreciate the active learning methodologies and school-based management training they got from PRIORITAS. Project training has improved principals’ planning capacities and engagement with communities, vital to the survival of the many private madrasahs which do not receive extensive government support. 27 Results of the survey are reported in an accompanying document: REDI (2015) PRIORITAS Mid-term Evaluation: District Education and Household Surveys Report. Surabaya, Java. 18 • District MORA Offices are more active in school improvement activities than Provincial Offices as the district level is the nexus of PRIORITAS focus. More direct contact of PRIORITAS at this level seems to have stimulated more activity and closer direct links with schools. Least effective: • PRIORITAS seems to be less involved with the MORA than with the DINAS offices. Even though MORA provincial and district offices are invited to participate in PRIORITAS events the same as DINAS offices are, follow-on support to MORA does not appear to have the same value as with the DINAS. Provincial offices especially feel more removed from the center of activities at the District Office level. • Only a few MORA supervisors have been trained by PRIORITAS and only a limited number have become project facilitators. The MORA seems to have greater issues with supervisors including limited numbers, poor training, and huge workloads (number of schools per supervisor); the limited attention by PRIORITAS may contribute to the Ministry’s sense that it cannot adequately address these issues. Supervisors Supervisors are a potentially important group of stakeholders in the Indonesian education system. They fill the gap between District Education Offices and the schools where they should play a critical role in managing and monitoring education provision. Quoted the Dean of Education at the State Islamic University Sanan Ampel, “Supervisors are the ‘extended hand of the TTI’ to support teachers.” However, their mention in interviews almost always caused a reaction leading the Evaluation Team to understand that their role and usefulness in the system are sometimes questionable. An ad hoc sample of school supervisors based in District DINAS offices or at the schools they were visiting was interviewed by the Evaluation Team in situ. Most effective: • PRIORITAS has made excellent use of competent supervisors in the role of facilitators (FASDA). The Evaluation Team observed and was told often that supervisors were best positioned to carry out combined roles of supervisor s and facilitators because of the similarity of their responsibilities and their flexibility to do both functions effectively. PRIORITAS facilitator training enhances their understanding and competency to do their supervision jobs. • The selection of women supervisor s as FASDA seems to be making a positive impact at the subject teaching level where many primary and junior secondary school teachers are women. Some women teachers expressed more confidence in requesting help from women FASDA than from men supervisors. • Former principals who are now supervisors seemed to be most effective as MBS facilitators after they receive training. Their experience as principals gives them better professional engagement and understanding of the needs of fellow principals when they facilitate MBS. Least effective: • Senior supervisors who hadn’t taken part in PRIORITAS training or other modern teaching methods seem to be less respected by teachers and principals in PRIORITAS schools trying to adopt new methods to improve the quality of teaching and learning. 19 • Supervisors who are not PRIORITAS-trained appear to have less understanding of the need to observe complete classrooms lessons as a method of teacher quality assessment. Principals Principals are key individuals in PRIORITAS’ efforts to strengthen school management and governance. In all of the 82 schools visited, interviews of one to two hours each were held with either a principal, a deputy principal, or an acting principal. Most effective: • Principals exposed and trained in PRIORITAS methods are more competent to lead their schools toward quality change than principals who do not implement or have not received training. Many principals noted that as a result of MBS training, they feel more capable as school managers and leaders, are more concerned with teachers’ needs in classrooms, have increased transparency with community and teaching staff, and have completed the budget and planning to include yearly teacher training. • Principals who were trained as facilitators to mentor MBS were more apt to make timely governance and management changes in their own schools. • When principals were helped and mentored by experienced facilitators who were, or had been, principals themselves, they professed to better understanding of the MBS training and were more apt to implement school management improvements as a result. • Many principals made great progress in engaging school community committees after MBS training, leading to increased community participation and contributions to PRIORITAS schools. Least effective: • Principals who were also facilitators had difficulty doing both their jobs to the fullest. Some said they were not doing justice to their schools by spending valuable time away helping other principals solve their school problems. Additionally, schools that had to operate temporarily without their principals seem to go against PRIORITAS’ aim to improve school government and management. • Principals claimed they weren’t visited frequently enough by facilitators and PRIORITAS project staff; they wanted more attention and support to fully understand and implement MBS training. Time, location, and resources impacted regular facilitation visits to principals. • Many principals testified that they were unable to observe their teachers in classrooms to the frequency or duration recommended by MBS training because of limits on their time. Few were able to observe a full lesson from start to finish. Facilitators Facilitators are the glue which binds PRIORITAS together across the provincial, school, and community levels. The Evaluation Team met and interviewed many facilitators, both in their positions as PRIORITAS trainers and in their daily roles as supervisors, principals, and teachers. Additionally, the Team held focus group discussions with groups of facilitators, separated by topic (MBS or AL), position, or grade level which proved very rewarding and provided important information about this important project role. 20 Most effective: • The use of facilitators builds capacity in the education system. Facilitators gained in confidence and proficiency in their own professional positions. They provide expertise in their respective places of employment, whether as teachers, principals, or lecturers, and will continue to do so after the project has ended. • Their selection from university lecturers, LPMP instructors, supervisors, principals, and teachers ensured that all grades and ranks of expertise in the education system were tapped into enabling facilitation of the project to all levels and stakeholders. Their selection and use will assist to institutionalize the knowledge and skills gained from PRIORITAS within the education sector. • Evaluators often heard that the selection of local facilitators ensured that they understood education contexts and issues thus enabling them to act more flexibly within each of their geographical areas of responsibility. • Facilitators felt that the comprehensive training and mentoring they initially received from PRIOIRTAS prepared them well for the important roles they had to perform. Least effective: • Facilitators’ ability to do two demanding jobs at once was often questioned. Facilitators themselves noted that in reality they sometimes had to downplay the facilitator’s job to give more attention to their professional duties. • The exhausting and extensive rounds of PRIORITAS training means that facilitators are always busy training or mentoring. Many facilitators noted that participants, themselves included, had difficulty absorbing and assimilating training information due to the fast pace and comprehensive content of training. • Facilitators’ competency was sometimes questioned. Teachers regularly mentioned that facilitators were not always willing or able to demonstrate the application of AL methods when requested in trainings. • The role of facilitators does not appear in the organizational structure of the education system although their value to the system is well understood at all levels of the government. PRIORITAS is not seen as advocating strongly with relevant stakeholders in target provinces for continuation of the position after project end. Communities/School Committee (SC) Strong community engagement at target schools seems to be one of the successes of PRIORITAS. The Evaluation Team was not tasked with a comparative assessment of their increased engagement; however, based on the frequency of positive reports from the range of stakeholders across the education system, communities have become major players in many schools. The Team met many SC members at schools and often had the opportunity to talk independently with parents during school visits. Additionally the Team held formal interviews with 19 school committee chairs. Most effective: • In 70 percent of the schools visited, the important role of School Committees in school development was a key topic of discussion with principals or with chairpersons of the SCs themselves. Both parties as well as PRIORITAS field staff were unanimous in observing that 21 the revival of community participation in schools seemed positively correlated with PRIORITAS MBS training. • PRIORITAS training assisted SCs to understand how they could effectively and legally contribute to their schools. In the majority of government schools, financial donations were voluntary and limited to what parents could afford. Community members also contributed in￾kind or labor to assist in school beautification and refurbishment. Less frequently, parents contributed to major school construction projects of toilets, student refectories, classrooms, and an occasional on-campus Mosque. • In several instances, SC chairs mentioned they provided oversight to ensure that principals were in their schools and teachers were in their classrooms. • In some rural areas, for example in Aceh Jaya, Bantaeng, and Madura Island districts, SCs claimed they also performed an important broader community liaison role between semi￾literate rural communities and the district DINAS education office or other local government departments. Least effective: • As with all training, absence of consistent and on-going support lessens the impact of training. Principals in schools where communities are not active or weakly involved noted that they needed follow-up support from PRIORITAS to assist them to revitalize SCs. Objective 3: Strengthened coordination between all levels of the education system, improving linkages to involve all education stakeholders in education planning, provision, and evaluation. For this objective, the Evaluation Team focused on the effects PRIORITAS has had at building linkages and informing policy and planning from the central MOEC and MORA level down to the provincial and district levels. Central MOEC and MORA Interviews with senior MOEC, MORA, KESRA, and BAPPENAS directors in Jakarta revealed that at the highest levels, knowledge of PRIORITAS has had a positive impact. Most effective: • The goals and objectives of PRIORITAS are harmonized with government priorities in education and several will be reflected in the next five-year national education plan which emphasizes a reformed and stronger education system at all levels of schooling. Some of the intersecting priorities between PRIORITAS areas of influence and government aims are: • Curricular assessment and change, inclusion of character building; • Better teacher management; • Decentralized and regulated TTIs; • Improved quality and performance of madrasah; • Improved financial management and governance within schools. • PRIORITAS has increased dialogue and awareness among critical government individuals in the MOEC, MORA, and KESRA around good practices in teacher improvement, teacher distribution, and student active learning. PRIORITAS’ good practices will provide the evidence needed to convince the GOI to go to scale with good practices beyond 2017. 22 • PRIORITAS has provided leadership and technical assistance for the rationalization of teacher distribution, a problem that has needed attention since 1999. • PRIORITAS efficiency in carrying out training and program implementation activities as planned is appreciated by government officials. • PRIORITAS efforts to get district offices more involved in training and change have set a good example for national dissemination and replication, one that is endorsed by several top MOEC officials. • MORA senior education coordinators were very positive about PRIORITAS’ efforts and are appreciative of the technical assistance provided by the project. Modules 1 and 2, MBS, and facilitator training are very relevant to MORA needs to improve education quality in madrasahs. • Showcases at the national and provincial levels are effective means to promote the PRIORITAS program to top officials especially since their limited travel budgets often preclude them from accessing the districts and schools to see PRIORITAS in action. • PRIORITAS training, especially for individuals such as the facilitators, assists to fulfill an identified need for trained human resources and experienced professionals to carry out education reforms in the future. Least effective: • Senior MOEC and MORA directors expressed the need to be more involved in PRIORITAS future planning and implementation activities. MORA directors claimed that throughout PRIORITAS they weren’t given equal access to funding and other training resources. • PRIORITAS is not seen to be very pro-active on the issue of gender, an issue which is more acute for boys who are not performing as well as girls at all levels of the education system. • In a similar vein, PRIORITAS is not very actively engaged in inclusive education. The Evaluation Team raised this issue at the regional, district and school levels and found little evidence of increased access to quality education by special needs children as a result of PRIORITAS efforts. Provincial and District Stakeholders Most effective: • Improved teacher quality and school management and organization are the commonly￾expressed needs across provincial and district stakeholders; PRIORITAS efforts in these areas draw the most appreciation from stakeholders. Most stakeholders express the need for these components to be taken to scale. • PRIORITAS efforts have made some inroads to improved communication and coordination between provincial and district offices although the rate of progress is very dependent on the willingness of leaders at each level to be engaged as well as efforts of PRIORITAS provincial and district coordinators to promote these relationships. Provincial offices for the most part seem to be more willing to engage with district offices than the district offices seeking cooperation with the provinces. Least effective: • Decentralization and the autonomy of district governance restrain PRIORITAS efforts to promote more cooperation across levels. 23 • The Evaluation Team found limited efforts by PRIORITAS to encourage closer cooperation between provincially-based stakeholders such as the Provincial DINAS, MORA, TTIs, and LPMPs. Question III: To what extent have PRIORITAS stakeholders benefited from the project’s activities and what specific value has been added? PRIORITAS was intended to build on previous USAID project successes to strengthen system capacities in teacher classroom performance, school management, and community support. Interviews with senior staffs, directors, and other stakeholders at MOEC, KESRA and MORA headquarters as well as at provincial, district, school, and community levels captured stakeholders’ interest and reasons for involvement in PRIORITAS. Their views highlight the benefits gained from participation in PRIORITAS. Central Ministries MOEC, a prime beneficiary and long-term supporter of USAID education projects, was positive in stating it had and continues to benefit in a number of important ways from PRIORITAS. • PRIORITAS has demonstrated that with modest expenditure and well-organized and designed training mainly at the district level, a positive impact could be made on improving the quality of teaching and learning in classrooms. • The PRIORITAS training model28 is having an impact on improving individual capacities. If evaluated positively by MOEC in the future, the Director General proposed it as a potential national model beyond the current few beneficiary districts for accelerating the improvement of classroom teaching methods. Going to scale would be supported by national funding. • The impact being made by a cadre of trained PRIORITAS facilitators was a definite improvement over what MOEC already has in place. The facilitator model could serve as a model for improving the functions of supervisors and trainers in other MOEC training units. • PRIORITAS has provided important leadership in addressing the serious teacher distribution problem that exists nationwide. PRIORITAS’ timely research and technical assistance in this area is beneficial, especially in districts where teacher distribution is proving to be a politically-sensitive issue and difficult to resolve. • PRIORITAS’ emphasis on improved school management linked to better classroom management and greater parental and community involvement fits the new Minister’s priority for making schools better resourced and more accountable to stakeholders. • The government places the most value on PRIORITAS technical assistance. As amplified by several high-level MOEC individuals, the government doesn’t need more money; rather it needs technical expertise and access to skills and knowledge in a timely manner. Benefits to KESRA were less clear but reactions were no less positive. As the agency holding the MOU with USAID, KESRA played an important role in the initiation and administration of the agreement. Without a significant national responsibility for provision of educational services however, the agency has not had much to do with PRIORITAS since the project’s inception. The senior KESRA MOU coordinating staff member did mention that PRIORITAS seemed to be having a strong impact at the district level on training teachers, a very positive contribution to national quality improvement in education. She also noted that her understanding of the good 28 This was highlighted by numerous stakeholders as “training-followed-by-mentoring.” 24 practices of the project has influenced her participation in the regulation of policies that affect student active learning, the promotion of reading and reading materials, student creativity, and teaching improvement. MORA directors and senior staff in Jakarta were positive about the project benefits of PRIORITAS but were unanimous in stating that they needed more help and financial support to enable implementation to assist transformation of MORA schools into effective centers of teaching and learning. The new PRIORITAS training methods were effective but not all MORA supervisors and teachers benefitted, especially the contract teachers in private madrasahs who needed the most help. The MORA highlighted the following benefits from PRIORITAS: • Exposure to a more effective training methodology; • Guidance to encourage a reading culture in MORA schools; • Support to principals for better management of MORA schools with special emphasis on managing teachers more effectively; • A focus on the importance of student-centered learning and teaching methods; • A focus on increased engagement of parents and communities to support MORA schools. Provincial Governments The five Provincial Governments represented by DINAS and Governors Offices were on the whole knowledgeable and positive about PRIORITAS. In the Governor’s Offices in Aceh, North Sumatra, and East Java, senior education officers praised PRIORITAS, noting that project aims intersected with their Governors’ top prioritization of improvement of education quality. Assurances were given that Governors would find funding to continue the best aspects of the project, especially the teacher training and facilitator services, when PRIORITAS ceases to be supported by USAID. Taking both aspects to scale would be an important first step in supporting good practices beyond 2107. Other important benefits from the project included: • Encouragement of students to want to learn and read more; • Increased community involvement in their schools; • Increased attention to the special needs of poorer schools in remote parts of the province; • Increased attention to the large numbers of untrained teachers in primary and secondary schools who need professional improvement; • Guidance to provinces for a better understanding of teacher distribution issues and creative solutions such as multi-grade teaching in small schools to rationalize the more effective allocation and use of teachers (as in East Java); • Focused attention on the need to refurbish and improve school infrastructure and appearance; • Assistance to provincial governments to identify and support educational priorities. District Governments District administration, where much of the project’s impact was focused, expressed near unanimous appreciation of efforts to raise awareness of the need to improve learning and teaching quality of district basic education services. PRIORITAS was making a positive and observable difference. District Heads of Education drew attention to the way schools were better managed and run, teachers were more enthusiastic about what they did, and students were more engaged and absorbed in the learning process. 25 The most impactful aspects of the project deemed to be sustainable beyond 2017 were the well￾organized and effective training methods and the newly-created corps of facilitator-trainers. Both components have filled important gaps at the district level and are seen to be a pivotal driving force for changes at both the primary and junior secondary school levels. District governments seem to consider these two components worth future investments. In Aceh Jaya, Bener Meriah, Maros, and Labuhan Batu districts for example, where supervisory staff are ineffective because of poor job performance, district stakeholders showed some determination to retrain supervisors with the skills they were seeing being successfully applied by facilitators. In Blitar, the newly appointed District Head of Education who himself had received facilitator training, recruited and appointed a younger group of supervisors who he said would be retrained using the PRIORITAS facilitator training. District BUPATIs and district level parliamentarians available for interviews were well informed about the positive contributions to quality improvement of their districts’ basic education services. As assessed by the district education survey of these individuals, many PRIORITAS activities have had a positive impact on quality improvements in their schools. The three interventions considered most beneficial were in-service teacher training, teacher mapping and budget allocations, and training of school principals, although many other activities received positive ratings (Figure 1). Three BUPATIs, in agreement with their DINAS Heads of Education, stated they would work with local Parliaments to set aside sufficient funding for continuation of PRIORITAS good practices in teacher and facilitator training and school management after the project’s completion date in 2017. Figure 1: PRIORITAS Activities Impacting Education Quality in District Schools REDI District Education Survey, 2015 26 District BUPATIs in Aceh Jaya, Bantaeng, and Blitar who had visited schools in their districts were strong supporters of PRIORITAS and had observed improvements in classroom management, teacher performance, and student learning. In South Sulawesi and in East Java, DINAS were funding district-wide school-level dissemination of good practices including mentoring. In other districts, governments were employing the PRIORITAS guidance on teacher distribution which would lead to the solution of a “supposed” teacher shortage problem. The savings generated could be channeled into the purchase of learning resources and other priorities. Schools and Communities Educators in all districts praised PRIORITAS efforts with active learning methodology and school-based management. From interviews with principals and teachers, nearly 90 percent of them rated active learning as the most effective PRIORITAS activity and the same percentage noted that student behaviors had changed to became more actively involved with learning at school.29 For the most part, educators placed high value on Modules I and II to build capacity of principals and teachers in schools. A selection of comments from teachers, principals and parents, too numerous to recount by topic, illustrates the range of what was learned and appreciated from PRIORITAS and more importantly, shows how education in project schools is beginning to change: • All teachers are well prepared with their lesson plan before teaching and approved by principal… All teachers can teach in an easier manner, they are more enthusiastic. … Teachers and students are more active and creative…. Teachers become more disciplined and get to school on time….Teachers use teaching aids more…. Teachers have become more disciplined in preparing instruments for class management …lesson plans and teaching aids. Parents can discuss anything with teachers about the progress of their children. Principals had many positive comments about PRIORITAS’s MBS training and the impact it has made on both on their own performance as leaders and on their schools. They now spend more time carrying out school planning and budgeting and reaching out to involve both teachers and parents in important school decisions. Nearly 80 percent of them noted that they had completed their school’s one year plan to include teacher training support since PRIORITAS training and nearly 70 percent had increased their interaction and involvement with the School Committee.30 Eighty two percent had initiated a school reading culture. They paid more attention to the need to beautify their schools and carry out timely maintenance. • The principal is more active and concerned regarding teaching and learning. She has a schedule to do class observation, 6 classes per semester. She checks teacher attendance every morning. The RPP and display are in every class. And reading corners are available in every class. • Before PRIORITAS there was no budget to support CTL [contextual teaching and learning]. After PRIORITAS, in 2014 [the principal] allocated 10% of the total budget to support CTL and in 2015 allocated 25%. He had no idea before PRIORITAS. The School plan and budget plan can be seen by teachers and committee. 29 School Interview Data, Annex X. 30 Ibid 27 78 84 26 78 86 77 44 67 83 62 64 62 2 9 A. MORE VISIT TO SCHOOL B. ATTEND SCHOOL COMMITTEE MEETING C. ATTEND READING TRAINING D. MEET TEACHERS MORE FREQUENT E. MEET WITH HEADMASTER F. VOLUNTEER TO HELP G. CONTRIBUTE MONEY H. INCREASED READING ACTIVITIES WITH CHILDREN I. DISCUSSION WITH CHILDREN ABOUT LEARNING… J. ATTEND SCHOOL ACTIVITIES K. SUGGEST IDEAS IN PLANNING L. INVOLVE IN SCHOOL ACTIVITIES M. NO CHANGE IN SCHOOL INVOLVEMENT N. OTHERS Increased participation of Parents in School Activities since PRIORITAS Training (%) N=129 • Before PRIORITAS she was reluctant to observe teachers in the classroom. Teachers did not respect her and always commented negatively when observed. Now, they are aware of each other’s roles and responsibilities. Community involvement and School Committee revival were some of the clearest and most significant benefits emerging from PRIORITAS training as attested to by the more than 80 percent of educators interviewed who noted active and on-going involvement of School Committees. 31 If MBS has made one significant contribution it is in re-energizing School Committees and reinvigorating parental involvement in schools, especially in rural communities. When surveyed, parents noted that they had increased their attendance at school committee meetings, were more likely to talk with their children about what they had learned at school, and met with the school headmaster and teachers more often (Figure 2). A significant proportion of the SC leaders the Team met claimed that because of PRIORITAS MBS training, they were now more aware of the need for communities to claim ownership of their schools. They were clear that benefits of training had been two sided. On the one hand, principals were now more aware of the need to include parents and SCs in all decision making. On the other, SCs were now more aware of the need to be more involved and spend more time on school campuses. Figure 2: Parents' Participation in School Activities REDI District Education Survey, 2015 In one Bantaeng primary school, the SC chair claimed that he now came to school every day to check if the principal had arrived on time and was doing his job. Previously, the same principal had often been absent, running his grocery business. In Bener Meriah district, another SC head said he came into his school as often as he could to make sure all teachers where in their classrooms teaching. He kept his own register of teachers to make sure he had the evidence to 31 Ibid School Interview Data 28 show the head when her teachers were absent. In Blitar, in one of the oldest primary schools, founded by the Dutch, the SC chair had been re-elected to the committee for the last 22 years. With other parents, he had witnessed significant improvements on the school campus: new classrooms, toilets, a library and a head teacher’s office, school gates, a mosque and many other school improvements, all paid for mainly through parent contributions or labor. A final significant benefit seen at the school level is exactly what PRIORITAS was supposed to do, improve access to quality basic education. The evaluation did not complete any data analysis of student achievement or attendance in PRIORITAS target schools. However, the following anecdotes from educators who spend their days and hours in schools with children tell much about the benefits PRIORITAS is having on students: • Students gained the capability of finding the needed information by themselves and interpreting the information found for their learning purposes. They put forth their opinion, explain their mind systematically orally and in writing, and conduct team-work so that they become more capable in learning to live together. • They gained more understanding of lesson content - these are shown in their answer to daily problem-solving sessions [in which] some of the answers have even surprised teachers in their depth of understanding on the concept. • One hundred percent of [primary] graduates are accepted into their favourite junior secondary school and the school attracts children from other villages to study here due to its reputation although their homes are far away….The children are independent and more confident. • Students are less absent. Only the ill students are absent. • Ninety eight percent of grade one children can read and write after they graduate. Question IV. To what extent are the PRIORITAS project’s resources being implemented and managed efficiently and cost effectively? The JBS Evaluation Team focused on gaining a better understanding of the district, school, and classroom level effects of PRIORITAS interventions on decentralized educational services. This meant less time available for more than cursory visits to PRIORITAS provincial and regional offices. Consequently, without enough time to carry out a systematic financial analysis of the project’s effective use of resources, evidence for this aspect was limited to a few proxy indicators extracted from data accumulated over five weeks of field interviews and observations and meetings with selected PRIORITAS headquarters staffs. Meetings with PRIORITAS field staff, institutional stakeholders, and local level beneficiaries led the team to note and highlight the following trends, strengths, and deficiencies in the project’s distribution and use of project resources. • Three years into the life of a five year project, training and training events were still occupying over 70 percent32 of the PRIORITAS field staffs’ time, energy and resources. District level project staff, teachers, and principals complained that a strenuous cycle of training cut into time for provision of mentoring and other support activities to teachers and principals in schools, casting some doubt on the efficiency of the current PRIORITAS training strategies and schedules. Additionally, the volume and frequency of training events 32 Author estimate. 29 may be diverting needed project resources away from support to school level activities such as mentoring, especially in poorer schools where resource shortages exist. • The Team received comments from principals, teachers, and facilitators about the limited time they had to absorb information between trainings due to the high frequency of training events. Poorly absorbed and applied training content at the school level seemed to contradict claims made on the part of project trainers that PRIORITAS training was efficient and effective. • The JBS Team was surprised to find that an inaugural national workshop, planned and organized by the PRIORITAS East Java provincial office to launch Module 3, was being hosted and run in a five star hotel. The Evaluation Team surmised that the combined travel and living expenses incurred in support of this event may have been excessive and did not seem to be the best and most cost-effective use of PRIORITAS’ resources. • District Coordinators, facilitators, principals and teachers in nearly every district visited testified to having insufficient financial resources to support activities like teacher cluster meetings, district level FASDA training, mentoring trips, MBS training events, and travel allowances to working group meetings (KKG and MGMP) which required district wide travel. A request for more PRIORITAS support for travel and other training was frequently heard. A better distribution of limited project resources to meet such requests might be a more cost-effective use of resources as attendance at all of these events supports continued training and building capacity of school-level personnel. • In an estimated 60 percent 33 of the classes observed, teachers were limited in implementing the full range of AL activities, especially group-work, because they lacked adequate resources to meet the increased amount of materials or aids required for active learning activities. The Team often heard reports from teachers and principals of having to buy teaching materials with their own funds. A small PRIORITAS-supported in-school learning materials support fund would be a cost-effective way of providing more classroom materials and encouraging a more thorough implementation of AL methods in classrooms. • District Coordinators interviewed in Aceh Jaya, Blitar and Maros were frustrated by the difficulty of carrying out their multifunction jobs in the face of inadequate administrative support and insufficient travel funding. They were expected to carry out an increasingly complex and demanding schedule of school visits with a small allowance. DCs drew attention to the need for more administrative staff to carry out functions involving school visits, dissemination and replication activities, oversight for FASDA training, and a list of other PRIORITAS-related activities in a growing number of partner, comparison, and some laboratory schools. This burden will become more onerous with the induction of Cohort 3 schools and the start of a new round of intensive training. The transfer of resources from generously-resourced Provincial Offices to under-supported District Offices would improve cost-effectiveness if the project focus is on improving the delivery of better quality management, teaching and learning to the district and school levels. • The Team observed that Monitoring and Evaluation (M&E) activities were led and coordinated from PRIORITAS provincial offices. When interviewing project M&E officers, the Evaluation Team learned that they mostly travel to districts once a year to collect 33 Author calculation based on classroom observations. 30 difficult-to-verify quantitative data from District DINAS offices. The Team saw little or no evidence of more frequent monitoring procedures being implemented at the provincial or district levels leading the Team to question the rigor of project monitoring. The Evaluation Team was not able to determine if the yearly M&E procedures carried by PRIORITAS staff meet current USAID recommendations for rigorous and frequent project monitoring. PRIORITAS’ provincial program resources deserve to be better allocated to improve the quality and frequency of M&E procedures in line with best USAID evaluation practices. Project monitoring and evaluation should include more in-class observations, capacity assessments, and more rigorous behavioral analyses at the district institutional levels. • When the Team financial specialist met with PRIORITAS staff in Jakarta to access the data needed for a more comprehensive assessment of the project’s cost-effectiveness she was told the following: The information … wanted was in TRAINET, which contained detailed data on training and training beneficiaries to the district level. The information in TRAINET is more about headcounts. This would seem to indicate that a more comprehensive project cost analysis is 1) quite an involved process, at least more involved than a mid-term evaluation can accomplish, and 2) a cost analysis of project implementation and therefore of the implementing partner is not possible, at least at this time. In addition, PRIORITAS staff explained that the budget is split between the prime contractor, RTI and a project sub-contractor, EDC, each of which has its own dedicated budget manager. RTI’s Head Office consolidates the financial information for both RTI and EDC expenditures and submits it to USAID. When asked to share a summary of the project budget and expenditure for the preceding two and a half years (ending 31 Dec. 2014), the Team was told that to do so, each partner would need time to obtain clearances to get access to the information required from the RTI office in Jakarta and the EDC office in the USA. • During the ten FASDA Focus Group Meetings convened in five provinces, Team members were informed often by participants that they were not receiving their FASDA stipends and travel allowances on time. This was causing hardship and inefficiencies to complete FASDA responsibilities in a timely manner. To get a better understanding of the problem, the Evaluation Team went to the PRIORITAS Office in Jakarta to discuss the issue with project management and received the following response: FASDAs have one-year contracts with PRIORITAS and stipend payments are due monthly. To get paid, each FASDA has to submit a monthly report that has to be approved by his/her Provincial Coordinator. If there are payment delays these may be caused by: (1) FASDAs submitting reports late, so the reports get paid during the following month. (2) RTI has to deduct prepayment of tax on payments of honoraria of FASDAs and sometimes they give RTI wrong taxpayer numbers; (3) Payments are sometimes made at the FASDAs instructions to their bank accounts, which they seldom check. So, they are often not aware that payments had actually been made. (4) Sometimes, FASDAs are confused as to which period a payment is for and on checking discover that their payment schedules were in fact correct. 31 This explanation would seem to lay the fault for late payments almost entirely on the FASDA themselves although the Team heard the complaint of late payments in every province. This led Evaluators to question the placement of all the blame for late or missed payments only on the shoulders of facilitators in the field. Question V. What evidence is there to indicate that student reading and reading comprehension skills have improved as a result of PRIORITAS interventions? Measuring Reading Achievement The only hard data on student reading comprehension currently available in PRIORITAS target schools are from the project’s implementation of the Early Grade Reading Assessment (EGRA).34 EGRA was conducted in November 2013 as a baseline assessment and in May 201435 as a monitoring procedure in a selection of comparison and Cohort 1 and Cohort2 schools in relevant partner provinces. The first assessment was applied to 3,574 grade three students out of a total population of 7,437 grade three students in the Cohort 2 schools targeted. The first and only EGRA assessment so far involved teachers and selected USAID PRIORITAS partner TTI lecturers and students who had been involved in the previous DBE years. Overall, EGRA findings for Cohort 1 schools (Grade 2) and Cohort 2 schools (Grade 3) coincide in that, while each sample shows well-developed reading skills, many Indonesian children don’t understand everything they read or hear in the Bahasa Indonesia language. The Evaluation Team interviews with teachers and classroom observations support this overall conclusion. Additionally, Evaluators who carefully observed ten Bahasa language classes and ten thematic classes in grades one to three saw little or no evidence of teachers correcting students’ oral reading errors and few instructional activities in language classes which assisted students to learn to read. Only the next EGRA assessment cycle can provide harder evidence to support real achievement in reading comprehension in the grades targeted. The May 2014 PRIORITAS Cohort 2 Grade 3 assessment drew attention to the following trends (in bold italics below) which deserve to be compared to findings from the classroom observations. Sampled children on the Island of Java are reading at a better level than their peers elsewhere. The JBS Team’s observation in East Java schools (in Blitar and Pamekasan districts) seem to back up this conclusion in that lessons observed in language teaching, math and science received a larger percentage of higher scores in terms of implementation of active teaching-learning methods (Table 5). Sixty percent of observed East Java schools received A and B scores, more than the total number of these scores in classes observed in the other three provinces.36 34 RTI International. (2013d) Baseline Monitoring Report, Vol. 3: An assessment of Early Grade Reading: How Well Children are Reading. PRIORITAS Report. Research Triangle Park, N.C.: PRIORITAS USAID/Indonesia. 35 RTI international (2014d) Baseline Monitoring Report, Vol. 3: Assessment of Early Grade Reading: How Well Children are Reading in cohort 2 Districts. PRIORITAS Report. Research Triangle Park, N.C.: PRIORITAS USAID/Indonesia. 36 Evaluators rated class observations on a scale of A to D, with A the highest score. Annex V. Aceh N. Sumatra A-B = 22% A-B = 0 C-D = 77% C-D = 100% S. Sulawesi E. Java A-B = 44% A-B = 60% C-D = 55% C-D = 40% Table 5: Observation Scores by Province 32 Girls in sampled schools are significantly out-performing boys in all subjects. In schools in most districts visited, the Team observed girls more often taking the lead in group work than boys. Teachers sometimes commented that their girls were performing better than most boys in most subjects. Girls also seemed more motived and engaged in learning than boys in many classrooms. Teachers who had participated in PRIORITAS Module II were exposed to a gender unit. In response to an interview question about gender in the classroom, teachers would often note that “girls and boys have equal treatment from the teachers; they have same rights and opportunities in the classroom.” Evaluators understood this to mean that teachers’ awareness of gender issues was allowing girls more opportunities to participate in classroom work; Evaluators noted however that this now seemed to be in detriment to boys’ participation in some classes. Children in rural schools are reading at lower levels than their peers in urban schools. In the poor rural areas visited such as in Bener Meriah, Banda Aceh, Madura Island, Labuhan Batu, and Bantaeng, teachers and students in classrooms seemed to be performing or achieving at slightly lower levels. Several contributory factors were shared by principals and teachers including high poverty levels, low levels of parental literacy, lower levels of resources in schools, and children who spoke local dialects or languages who struggled to read and write in Bahasa Indonesian. Children in project partner schools are outperforming students in non-partner schools. The Team did observe a tendency for partner schools to perform better than other non-partner schools in reading. While a rigorous reading assessment was not part of the evaluation task, the Team observed and heard anecdotal evidence on which to form impressions of possible reading abilities in partner and non-partner schools. For example, in non-partner schools, none of the 12 schools out of the total number of 82 visited had facilitated a reading culture (e.g., active libraries, reading spaces, a concern for the number of books available, an impression of student reading ability, etc.). Reading corners were not in evidence in classrooms and a silent reading time during the school day was not a standard activity; therefore children were not involved in more time with books.37 Evaluators understood therefore that reading was not a focus in non￾PRIORITAS schools. On the other hand, in partner schools, the Team heard of many proxy indicators used as the basis for ‘measuring’ reading literacy (but not necessarily comprehension) during interviews with educators. Several are noted here: ‘98 percent of grade one students could read and write after graduation’ (SDN1 Kebonduren, Blitar), “Now 60 percent of this MIN’s students pass the entrance test (reading of Quran) to junior secondary school.” (MIN Merduati, Banda Aceh.) and “Students gained the capacity to find the needed information by themselves after reading their text book and going to the school library to find relevant books” (SMP Sampoineit, Aceh Jaya). On the other hand, the Team did observe some non-partner schools, especially GOI-supported junior secondary madrasahs which were seen to be performing better than some of the partner government schools visited. The following contributing factors may be having an impact. Some madrasahs are well managed and hold students to a higher degree of discipline and personal responsibility. Teachers and principals have now been better trained than before and are beginning to make a difference in school performance. Likewise, in some comparison schools, an unexpected high level of teacher performance and student response may be due to the cumulative effects of other trainings received from previous PAKEM programs. 37 More time with books in the hands of children is one of the recognized ‘5 Ts’ of literacy, factors which have proven necessary to increase literacy abilities of children. 33 The Team visited 30 madrasas and 52 government primary and junior secondary schools. The Team was often positively surprised to observe better-led and managed schools in many madrasas where both PRIORITAS and non-PRIORITAS trained teachers were observed to be using active learning methods in science, math, and language. School stakeholders were convinced that well-performing madrasahs were a consequence of good school leadership, a clear vision of school excellence, a higher level of teacher commitment and motivation, and stronger support from the community. Children with access to books at home score higher on all subtasks than those that do not have access to books. The JBS Team did not include a visit to parents in individual households, but did interview 19 parent members of School committees. None of the parents interviewed discussed their roles as reading mentors or the role played by books in homes but noted that parents were contributing to the “reading culture” in their schools by building reading gardens, donating books, and refurbishing libraries. Responses in the Household Survey seem to indicate that parents had increased their involvement with their children’ reading activity by 67 percent (Figure 2). Additionally, 84 percent of parents (out of 129) noted that their children’s schools now possess more books as a result of PRIORITAS intervention; 88 percent noted that their children now read more than before PRIORITAS assistance.38 Observed Signs of a Growing School Reading Culture In the 82 schools visited, the team saw tangible evidence of a growing reading culture in schools. 1. Reading corners in 36% (30) classes observed; 2. Active attempts to restore or build library facilities in about 13% (10) of Junior Secondary and Primary Schools; 3. Outdoor reading areas (reading parks or gardens) in 20% (17) of the schools; 4. Parental or school committee book contributions in 13% (10) of schools; 5. A request for more books from teachers, principals, students and parents in 30% of the schools visited; 6. Rural mobile library services paid for by the district DINAS in one rural school on Madura Island and plans of several other schools in Bener Meriah and elsewhere to submit proposals to District DINAS for visits by mobile libraries. 7. Setting aside silent reading time during school hours or in class noted in 45% (37) of the schools.39 The tangible factors supporting reading in the targeted schools are insufficient to verify measurable reading and comprehension achievement at this time. The awaited next EGRA assessment may shed a more objective light on the importance of PRIORITAS’ contribution to reading comprehension in the Indonesian schools. Achievement in Math, Science, and Bahasa Indonesia An analysis was done by the Team comparing results obtained from student testing carried out in 2014 by PRIORITAS evaluators and qualitative information gathered during the evaluation. Information collected during observations of grade one to nine Bahasa, English, math, and science lessons provides some basis for the assessment of the current status of classroom learning in the four subjects areas cited. 38 Household Survey data, Annex IX. 39 All data are from interviews with principals and teachers and school observations: evaluators’ calculations. 34 • At the Primary Level, the 2014 PRIORITAS test results revealed the following: Grade four students tested for Bahasa Indonesia had average scores of 37 percent for reading and 36 percent for writing. Reading test scores were similar for partner and comparison schools whereas scores for writing in partner schools were substantially higher than in comparison schools (38percent compared to 33 percent). Thirteen percent of children in partner schools and 20% in comparison schools wrote nothing. From this, the test concluded that many children in grade four partner schools had difficulty in comprehending meaning in what they read. The Team’s observations of ten Bahasa Indonesian language classes in primary schools showed few teachers making any effort to correct and help students with their reading. In the majority of language classes seen, instruction was traditionally book-centered or taught by rote chanting. This suggested a need to train language teachers in more progressive language teaching methods such as interactive drill-based modern language teaching/ learning methods. • The Grade 4 mathematics test showed that the overall average score was 39 percent for partner schools and 36 percent in comparison schools for this test. Little difference was evident in scores between partner and comparison schools. Areas where students had difficulties were in recognizing the value of both decimal and simple fractions and operations with fractions. Students also scored poorly on questions requiring problem solving and creativity in their answers. The Team’s observation of ten math classes showed an encouraging trend. The classes observed were relatively well-taught and may reflect better method preparation in mathematics and better qualified math teachers in that subject. Observation scores tended to be average (‘C’) or above with three lessons rated as excellent (‘A’) (Table 6). Active group work sessions and the use of the natural environment to practice measurements, calculate distances, and apply mathematics to real life situations were observed • The Grade 5 test for Science learning revealed that the overall average score was 33.6%. Little difference was seen in scores between partner and comparison schools. Some differences were evident in the way children handled multiple choice questions and questions requiring problem solving, application of concepts and deductive thinking. In the latter aspects, students had considerable difficulties responding. JBS Team observers saw ten primary science lessons. Some were well-taught and all rated at a ‘C’ or above (Table 6). Real evidence was apparent that some science teachers had grasped and were applying AL methods in their lessons, using practical aids and hands-on group work. Students appeared to be enthusiastic and well-engaged. In lessons where teachers appeared to be poorly trained in science instruction, classes were book-centered and more passive teaching and learning approaches were observed. • At the Junior Secondary Level, the 2014 Language Tests revealed the following: The Bahasa Indonesia reading and writing tests applied in grade eight in 2014 gave a baseline of 64 percent for reading, and 46 percent in writing. Comparison schools scored slightly higher Table 6: Primary Observation Scores: Math and Science Classes Science Math Total A 2 3 5 B 3 2 5 C 3 4 7 D 2 1 3 35 than partner schools in reading comprehension. Partner schools scored higher on the writing test. Between 23 percent and 35 percent of students scored poorly or very poorly in the writing test in the sentence-writing segment, the quality of ideas, spelling and punctuation and in the handwriting segments. Fifty percent had difficulty writing paragraphs. The Team’s observation of language teaching at this level included 12 English Language and 2 Bahasa Indonesian lessons. Although totals numbers of scores are too small on which to base a rigorous conclusion, in both cases language learning methods left a lot to be desired and classroom observations were generally rated poorly (Table 7). Especially notable was the lack of intervention from teachers in correcting reading, speaking, and grammar errors and deficiencies. Little indication was seen of any special effort to help students improve their comprehension skills in either language apart from some written work which observers were not able to assess for quality or comprehension level. Additionally, few English teachers spoke English well. English lessons provided few or no opportunities for learners to practice speaking the language, a key objective in any language class. Question VI. What evidence is there to show that the PRIORITAS project’s activities and results are making progress toward sustainability and replication after project completion? For purposes of this field evaluation, the JBS team adopted the following forward-looking definition of sustainability, identifying six indicators as they are applied to the observed replicable and sustainable educational changes presented below. Sustainability is where the following elements are perceived to be in place: A. Local ownership of the process and responsibility for outcomes is ultimately with stakeholders. B. Benefits are maintained and advanced after external development assistance has been completed. C. External inputs lessen over time approaching zero. D. Rate of local participation in activities is constant or increasing. E. Outcomes advance the quality of education. F. Distribution of benefits is equitable. The concept of dissemination, frequently used in PRIORITAS plans and training, is not easily defined as it is often confused with what is better described as replication or the process of ‘transferring, reproducing, or duplicating an activity, process, or construct in its original form without change or modification.’40 The definition of dissemination, used here, will serve to clarify what the Team understands to be the process as applied in PRIORITAS, “The process where the programs or innovations developed under the PRIORITAS project are implemented independently by others, beyond the original project sites, using their own resources.”41 To 40 Author definition of replication: F Dall. 41 From Cannon, R. et. al. (2014) Dissemination and Sustainability of DBE and PRIORITAS Programs. Research Paper. Jakarta: PRIORITAS and USAID Indonesia. Table 7: Jr. Secondary Observation Scores: English and Indonesian English Indonesia Total A B C 3 1 4 D 9 1 10 36 analyze the connection between the dissemination, replication and sustainability concepts and processes in place that may result in sustainable outcomes, a three phase model was used: • Implementation (Socialization, roll-out and advocacy). • Dissemination (Transfer, replication and scaling-up). • Sustainability (Mainstreaming, institutionalization and sustainability). Field observations and interviews in Jakarta and five PRIORITAS provinces and ten PRIORITAS Cohort 1 districts targeted for this evaluation provide the data used to highlight potentially sustainable and replicable practices. The following outcomes at the government, district, school, and community levels were identified. Relevant elements from the list above (A to F) are noted at the end of each outcome to substantiate the Evaluation Team’s determination of sustainability. Central Government Level • Successful PRIORITAS training, mentoring, and facilitation activities are aligned with national government policies and plans that support a need for basic education quality reforms (A, C, D, E). • Commitment to improve the quality of basic education is reflected in local district government policies and funding. (A, D, E). • PRIORITAS training methods are accepted at the highest government levels as potentially worthy of adoption and funding for national education training purposes. (A, B, C. E). • National recognition is given to PRIORITAS module content and training methods and considered worthy of support and dissemination even after project closure in 2107. (A, B, D, C, E, F). The District Level • Commitment to disseminate the best aspects of PRIORITAS is reflected in the majority of the district education and financial plans seen by the Team (A, B, D, F). • Increased levels of education expertise are observed in DINAS PENDIDIKAN offices and in teachers and principals in schools (A, D, F). • Six District parliaments are setting aside funding to support PRIORITAS training in schools and facilitators at the district and sub-districts levels (B, A, D, F). • The MBS Module is recognized as effective to improve the way principals manage and organize their schools and is considered worthy of support after PRIORITAS closes in 2107 (A, B, D, E, F). • A move is seen away from external donor-led education reform to education change owned and led by local governments, districts, schools and communities (A, B, D, C, E, F). • PRIORITAS has raised awareness among DINAS that the future of basic education quality is a district level responsibility to be shared among teachers, principals, parents, and students (A, B, C, D, E, F). • PRIORITAS’ research into more pragmatic teacher deployment strategies is making an impact on education policy and planning at district and provincial levels. (A, B, D, E, F). School Level • Schools are more aware of the need to be pro-active in the use their own resources for the improvement of quality of the basic education services they are responsible for (A, D, E, F). 37 • In 95 percent of the schools visited, visible evidence of significant improvements were seen in school buildings, school environments, learning spaces, and common areas such as libraries, reading corners, eating places, staff rooms, and classrooms. (A, D, E, F). • Evidence of a strong “osmosis effect” was seen where a few teachers implementing PRIORITAS innovations and changes at one school were influencing teachers in the same school who in turn would do the same for teachers they met socially or at KKG/MGMP cluster meetings (A,B, F). • Frequent reports were received from parents, students, and SC members of improved teacher attitudes and more engaging behaviors resulting in higher student motivation and participation in and out of classrooms. (D, E, F). • Eighty percent of the schools visited had initiated reading activities through implementation of focused reading in classrooms and free reading time, the building or improvement of libraries, and the creation of reading corners and spaces. (A, B, D, E, F).42 The Community Level • The revitalization and energizing of School Committees are having a strong positive influence on principal, teacher, and student performance in schools (A, B, D, F). • Increased community responsibility and involvement in learning initiatives have resulted in more focus on reading and cleaner and better organized school premises (A, B, D, F). • A more frequent and active parent presence is felt in schools; some parents now feel responsible for monitoring teachers and principals to make sure they do their jobs well (A, B, D). • Parents and teachers more frequently lobby the BUPATI, the District Head of Education and local parliament for educational improvement and more financial support to schools (A, B, D, E, F). The JBS Evaluation Team is on the whole positive about the potential for replication and sustainability of key PRIORITAS activities and contributions despite observing only nominal evidence at this mid-term stage in the project’s implementation. Some of the central government, district, and school level reactions highlighted above support the Team’s general assumptions that over time, some of the contributions to basic education change being made by PRIORITAS will be adopted and adapted especially at the local district and community levels. The Team felt most encouraged that certain elements such as project training methods and content, the use of facilitators, and better school management are beginning to make an institutional footprint that could insure their longer term sustainability. Question VII. How effective has each component of the PRIORITAS project been in improving gender equality among students in schools? After seven weeks of collecting data in all locations, the Evaluation Team came to two conclusions about gender in the education system. Education delivery at the school-level has become feminized. The teaching staffs at most schools the Evaluation Team visited are women (in a 73 to 27 percent ratio for all teachers in site schools). This follows a global trend in education that primary teachers are predominantly female. Some of the schools visited had no male teaching staff at all, a fact that the Team felt 42 School Interview Data, Annex X 38 was unfortunate in that children lacked male role models from which to learn appropriate behaviors and attitudes. When present, men were most often sports and religion teachers, rather stereotypical positions for men. Half of the principals at observed schools were women and by all appearances were as effective as their male counterparts. The Team deemed this to be a good trend as it allows women access to a leadership career path in the sector; however, the Team met no women in the top District and Provincial leadership roles, similar to the situation in School Committees where 98 percent of the leaders are men. The second conclusion that seems to be developing in the Indonesian education sector is a tendency for boys at all levels to be achieving lower performance and enrollment rates than girls, a case of a “reverse-gender” trend. Girls in many schools seemed to be doing better on examination results, working better in classrooms, and transitioning at higher rates to the next education level. The Evaluation Team often observed girls leading and performing most of the tasks in classroom group activities while boys appeared most often to be involved as “spectators.’ This reverse gender issue is coming to the forefront as evidenced by the reactions of several Deans of Educations at Teacher Training Institutes to increased rates of female students in higher education. They feel that the issue is sufficiently pressing to warrant research into the social and economic factors which drive boys to under-perform in the education system. The Evaluation Team drew the following conclusions about the effectiveness of each PRIORITAS component to improve gender equity in schools. Objective #1: Strengthen capacity of pre- and in-service training provision, leading to improved capacity of primary and JSS teachers in reading, math, and science instruction Most effective: • PRIORITAS included a gender unit in Module 2 which served as in-service training guidance for teachers, principals, and community members at the school-level. Lecturers at universities as well as LPMP instructors were also exposed to this module. The training modules are popular and have been modified as pre-service curricula at several of the partner Teacher Training Institutes and subsumed into the training curricula at several LPMPs. This will assist to disseminate and institutionalize information on gender into teacher preparation and training and among educators and communities on a wider scale. • Teachers in many schools and madrasas have reacted to guidance in the PRIORITAS Module by mixing boys’ and girls’ seating arrangements in classrooms. Many state that ‘boys and girls are given equal treatment and opportunities’ in the classroom. Least effective: • Implementation of gender into active learning methodologies in schools is sometimes simplistic. Mixed seating is a superficial response to gender awareness in the classroom. As noted in observations by the Evaluation Team, girls often did most of the actual work in the group activities. Meanwhile, many teachers, depending on their own un-conscience gender bias, chose a majority of girls or boys when acknowledging responses and picking group activity reporters. The degree of sensitivity necessary to integrate gender into classroom instruction takes time to acquire. PRIORITAS project managers and coordinators themselves may not even be aware of the issues as they operate with their own “gender blinders.” 39 • PRIORITAS seems to have given little additional guidance or input specifically to gender issues in the education system at any level or in institutes responsible for teacher development. • PRIORITAS show no acknowledgement the looming problem of reverse gender in the sector and the impact this is having on the low male achievement and increased drop-out rates. Objective #2: Improved education management and governance at the provincial and district level leading to improved human and financial resources to schools Most effective: • PRIORITAS selects and engages women and men equally in positions of leadership and management in project staffing as well as in project partnerships with schools, institutions, and government offices. • Principals and School Committee leaders have benefitted from the MBS training to look for solutions to inequalities in school access. The Team heard several times of community heads and principals advocating with parents to send their girls to school and addressing issues of boys’ non-involvement in education. Least effective: • PRIORITAS has not engaged institutions such as Teacher Training Institutes or District Education Offices to research gender issues in the education system. Raising awareness of the presence of the issue is a necessary first step. Objective #3: Strengthened Coordination between all levels of the Education system, improving linkages to involve all education stakeholders in educational planning, provision, and evaluation Least effective: • Gender has not been given the focus it should have at the central levels of the system to build capacity and raise awareness of issues impacting the education system. As noted by the Director of the Center for Teacher Professional Development at central MOEC, gender is not being emphasized enough in PRIORITAS. She believes cooperation between her agency and PRIORITAS is needed to develop a separate gender training agenda. Question VIII. Is PRIORITAS on track in terms of meeting its overall end-of-project goals? The three end-of-project objectives which are the focus of this mid-term evaluation are assessed from two perspectives: • As a result of the Team’s observations during visits to schools, interviews with key informants, and visits to stakeholder institutions in the field; • From a review of PRIORITAS reports, studies, records and data. Objective #1: Strengthened capacity of pre-and in-service training provision to improve the capacity of primary and junior secondary teachers in reading, math and science. Visits and interviews carried out with staffs in five Islamic and five government Teacher Training Institutes revealed that Islamic institutions seemed generally more determined to adopt 40 and use PRIORITAS methods at the internal staff development level and in their delivery of pre￾service teacher training. Less commitment was apparent in some of the government universities visited. The majority of both Islamic and government TTIs were still not able to completely apply PRIORITAS methods to change the manner in which practicums were being carried out. Evidence suggests that the notion of off-campus mentoring and visits to schools by academic staff are still difficult to adopt and implement in the observed university training culture. If the ultimate test of increased capacity of teachers is changes to learning in classrooms, then classroom observations may be the best indicator of PRIORITAS’ impact. Positive signs of improvement in the teaching of math and science subjects were observed while less progress seems to have been achieved in the teaching of languages. Little evidence was observed to indicate that teachers were using anything more than conventional methods when teaching reading and writing in nearly all of the lessons observed. In only a few of the 76 lessons observed was there evidence of teachers helping students with their reading skills. The two groups making the most difference to the way schools are managed and teachers are performing in classrooms are PRIORITAS-trained FASDAs and principals. Interviews with 116 FASDAs and 75 principals revealed that one of the most effective interventions put into place by PRIORITAS was well-focused and organized training. Most PRIORITAS training beneficiaries in the field are enthusiastic about the Module I and II training they received to which they attribute improved capacity in carrying out their roles. Especially significant was the MBS training received by principals and the active learning training received by teachers. Some criticism about frequency and speed of PRIORITAS training was heard; critical comments about poor follow-up, insufficient mentoring, and weak post-training support were also recorded. The PMP and quarterly project documents thoroughly record in detail the many training events, school level tests, and other project planning and management activities. Project reports support the Team’s observation that the project has been and continues to be heavily engaged in training activities, the volume of which may be overwhelming the need to pay more attention to quality rather than quantity concerning actions which should be happening in classrooms. Objective #2: Improved education management and governance at the provincial and district levels to improve human and financial resources in schools. School and district DINAS-level observations and interviews revealed the following positive trends: Principals trained in PRIORITAS MBS methods were doing better jobs as school managers and were more focused on the needs of their teachers, students and communities. They were, in general, more attentive to the need for improving their school plans and budgets and most were happy with the assistance they were receiving from FASDAs to carry this out. Attention was frequently drawn to the overall improvement in the way schools are now being organized and managed with increased transparency and input from stakeholders. Improved principal management and mentoring of teachers was another positive outcome of MBS training. At the District, PRIORITAS training seemed to be making positive contributions especially where DINAS Heads and staff had themselves been beneficiaries of project training. DINAS educators were generally very supportive of what PRIORITAS was doing to improve education 41 quality; they assured Evaluators that they and their BUPATIS would make every effort to find local resources to continue the best PRIORITAS activities after 2017. Where district level leadership had been removed due to malfeasance, a leadership vacuum at this key level was preventing the effective up-take and facilitation of PRIORITAS interventions. The impact of MBS training on School Committees and communities was positive and significant. The 19 SC members interviewed all spoke highly of the impact MBS training had made on their committees and on their principals. In some instances, SCs which had almost ceased to function had been revitalized and were now playing active roles in supporting principals and teachers to improve schools, both as learning institutions and green environments. Attention to gardens and surroundings, the refurbishment of classrooms, teachers and principals’ offices and work on libraries, reading corners and quiet areas were ample evidence of active community participation to improve school learning and teaching. A better understanding of parental and community responsibility for the creation of reading cultures both in schools and in homes was another positive outcome of PRIORITAS’ campaign to involve parents in schools. A wealth of written evidence exists in PRIORITAS field reports and the project newsletter, PRIORITAS PENDIDIKAN to support the Team’s observations and conclusions above. The sheer volume of reports and news items precludes their full inclusion in this report; however, on reading almost all of the newsletters the Team could access, some project activities and achievements could not be confirmed or verified in the field when relevant individuals or institutions were interviewed. Objective #3: Strengthened coordination between all levels of the education system, by improving linkages to all education stakeholders in education planning provision and evaluation. Interviews and discussions held with senior MOEC, MORA and KESRA staff in Jakarta revealed a keen interest and awareness at the national level of the importance of projects like PRIORITAS to help the GOI improve the overall quality of basic education. The need to bring international donor partners closer together and coordinate parallel efforts under one umbrella was mentioned by both senior MOEC planners and by donors including the World Bank, DFAT, and UNICEF. The need for technical expertise of the highest quality and less emphasis on funding was clearly stated. The GOI emphasized that it has sufficient funds to carry out the changes wanted in the national education system but not enough experienced technical know￾how to implement these well. Discussions with donor partners in Jakarta were supportive and very positive. The World Bank, DFAT, and UNICEF were all aware of PRIORITAS and project achievements but requested closer cooperation and coordination of program planning and activities in the areas of common interest such as basic education. The World Bank’s Education Section went so far as to say that when it begins the next basic education project, currently under discussion with the GOI, close program coordination with USAID would be desirable and in the interest of Indonesia. UNICEF’s new Education Chief in Jakarta testified that previous cooperation with USAID projects such as DBE had been positive; however he thought that UNICEF’s future thrust in Indonesia would be more on improvement of early childhood education for poor provinces like 42 Papua where work with local NGOs was already starting. DFAT’s new thrust would be increasingly toward bolstering programs to improve educational quality in line with the focus of PRIORITAS; however, DFAT, functioning under a radical reorganization, would not be directly involved in program implementation in situ, rather plans to out-source responsibilities to national entities and some Australian private sector organizations. Senior directors at MORA headquarters in Jakarta voiced similar satisfaction with PRIORITAS efforts but felt the project favored GOI schools and teachers over MORA schools. MORA is now fighting to maintain its independence from GOI controls in basic education service but needs more financial assistance from the GOI, donors and projects like PRIORITAS to stay independent. MORA education contributes a moral, ethical and spiritual dimension to Indonesian education which is apparently not being achieved through secular GOI education; many parents are finding madrasah education attractive as evidenced by increasing madrasah enrollments. In five provinces, keen interest was expressed within BUPATIs’ offices to know more about PRIORITAS. In Aceh, East Java and North Sumatra, senior provincial education officers testified that their priority was to improve the quality of educational services at all levels. Their knowledge of PRIORITAS led them to believe that when the project ends in 2017, the provincial planning offices and their parliaments would set aside sufficient funding to continue project training and other relevant aspects to scale such as the role of FASDA. Provincial educators were generally frustrated by the level of controls given to districts over the hiring, firing and recruitment of teachers, principals, and other key personnel and hoped for more responsibility in the future from MOEC and the current Jakarta government over these functions. The next PRIORITAS phase would benefit from forging closer ties with provincial education offices especially with the imminent implementation of Parliamentary Decree 23 transferring responsibility of senior secondary schools and special education to the provincial level. PRIORITAS reports, the PMP, and other sources of written information do not give much detail about project contacts with stakeholders at the GOI senior levels presumably because discussions at this level were/are of a more confidential nature. However, ample examples are noted in the PRIORITAS PENDIDIKAN of officially organized and hosted “show cases” and other awareness-raising events to which senior government officials were invited and had attended. In each province, at least one provincial showcase event was held to disseminate information about PRIORITAS’ work in that province. A national “show case” meeting was successfully held in Jakarta to which all senior directors and heads of partner ministries and international donor agencies were invited. This confidence and effort to “build bridges” at the senior levels of PRIORITAS still need more attention. 43 SECTION IV CONCLUSIONS The JBS Evaluation Team presents the following conclusions of the Mid-term Evaluation of PRIORITAS project. Based on analysis of data gathered from all collection methods, the Team has determined that overall progress of PRIORIAS is good and mostly on-track but progress rates vary across provinces and stakeholders. Some areas have made better progress than in others as noted below. • PRIORITAS is having a positive, palpable impact on beneficiaries at the district, sub-district, and school levels. The impact is especially noticeable in the improvement of attitudes and behaviors of principals, teachers, students, and parents. • The most obvious progress is in the teaching behaviors and attitudes in classrooms with the use of active learning methodologies, effective use of school-based management training of principals, and the use of trained facilitators to carry out training and mentoring of project stakeholders, the three areas where PRIORITAS has placed most of its resources and efforts to date. • The focused and organized training process and the content of Modules I and II are widely appreciated and used by beneficiaries both in the field and at the national level. The modules for example can now be found in the training curricula of Teacher Training Institutes and the LPMPs. PRIORITAS has become well-known for the training- mentoring process, continuously commented on by stakeholders as the most beneficial and impactful in delivery of active learning methodologies. • The real impact on learning achievement is difficult to assess given the lack of reliable methods for measuring learning resulting from an application of active learning methods. Indicators of active learning seem superficial (e.g. group seating and wall displays) without providing a measurement of real change in learning. Individuals who are responsible to monitor classroom teaching and learning processes, for example facilitators, principals, and supervisors, themselves do not understand the measurement of learning. Few teachers seem to conduct classroom-level assessment of students, e.g. authentic assessment. Improvements in student achievement are measured by national examination for which the required learning process does not align with the methodology of active learning. • Little evidence was seen in classrooms that reading comprehension is receiving the attention it needs from teachers to improve student performance. Evaluators saw almost no evidence of attention to comprehension, understanding concepts, and application of problem solving by teachers in any subject. • Project schools are clearly benefitting from better management and organization as a result of practical skills acquired by principals and communities in MBS training. • School committees and parents are making a positive impact on school quality through financial or in-kind contributions and cultivation of ownership and pride in schools and student achievements. Evaluators saw positive evidence of school committee dedication and 44 generosity to improvement of school environments and the delivery of better quality education. • PRIORITAS spends significant resources and energy on training. The training regime could be improved through re-design and attention to frequency, length, and reduction in volume of information given at one time. Teachers, principals, and FASDA appreciate the training but are overwhelmed with the speed and quantity of information given at one time. Many individuals mentioned the difficulty in absorbing so much information at once and recommended that training cycles be spaced out over a longer time. • Project-supported facilitators received many positive reviews and are highly appreciated for the personal attention given to teachers and principals through provision of training and mentoring. Additionally, trained facilitators build individual and system capacity and improve expertise across multiple levels of the education system. • The impact made on Teacher Training Institutions by PRIORITAS is uneven but encouraging. Many lecturers have benefitted from active learning techniques which seem to be making a positive impact on university staffs’ instructional methodologies. Lecturers feel this has positively impacted pre-service teacher preparation. Evaluators heard many positive comments about improved quality of student teachers in the practicum component although improvements could be made in the frequency and quality of mentoring. Teacher Training Institutes are not yet able to provide in-service training since they do not know how to initiate and deliver it at the district level. TTIs also have a relatively narrow geographic footprint and cannot access more distant districts. • The once-yearly collection of project monitoring data seems ineffective given the size and complexity of the project. M&E should provide more targeted information and attention to increased capacity of individuals and the system rather than just address quantitative measurement. Monitoring should reflect the social, political, and institutional processes that affect PRIORITAS’ impact on change in government institutions at the national, provincial, and district levels. 45 SECTION V RECOMMENDATIONS The Evaluation makes the following recommendations for consideration to increase the impact of PRIORITAS in the next two years of implementation. Recommendations were winnowed from the many made by project stakeholders during data gathering as well as suggested by the Evaluation Team based on observed PRIORITAS and Indonesian needs. Recommendations are arranged under common alphabetized headings but are not given in order of priority of implementation. Recommendations may be too numerous for all to be implemented before the project’s end date in 2017; project leadership and USAID will have to decide which recommendations will have the most impact in the time remaining. Facilitators • Pay facilitator honoraria on time and in adequate amounts to ensure they can travel to schools regularly and are rewarded for their time. If FASDA activities are carried out over a longer period of time, payments should be made in a larger lump sum from the Jakarta office at clearly agreed intervals. • Work with facilitators and District Offices to develop structured schedules so their own jobs do not suffer in their absence and they still have time to dedicate to the demanding schedule of a FASDA. Teachers for example, could teach in the mornings and use afternoons for facilitator work. Agreements with District Offices allowing educators to become facilitators should include discussions and assurances of job security, continuation of contributions towards retirement, and development of career pathways for advancement in the system; these considerations will be especially important if governments continue to fund the position in the long-term. • Reduce the number of schools within the responsibility of each FASDA. Suggest that mentoring responsibilities take place after school hours. Recruit and use facilitators to work in schools belonging to the same cluster to reduce their travel times between activities. • Recruit and train supervisors as MBS facilitators to ameliorate the effects of removing principals from their schools. Principals who are not in their schools send the wrong messages about creating a strong culture of leadership and well-managed schools. • Provide refresher training in the more difficult topics such as authentic assessment, gender￾sensitive teaching strategies, and higher order questioning. These elements were identified as the most difficult to implement in active learning during classroom observations and interviews. Gender and Inclusive Education • Expand gender training for principals, teachers, and School Committees to increase awareness and provision of gender in school policy, planning, and practice. • Modify the gender unit in Module II to be more reflective of different cultural contexts in provinces and districts. • Provide technical assistance to provincial governments actively engaged in development of education services for children with special needs. Assistance should focus on teacher training, school management, funding, and equipment and facility needs in preparation for the increased role of provinces in special education management in the future. 46 • Develop a unit on teaching techniques for teachers, principals, and parents to assist special needs children already mainstreamed in local schools. This could be included in the A-L training. • Support Classroom Action Research at universities to explore gender issues through well￾designed social research carefully examining school and community factors which affect the current trend of boys’ poor performance and low achievement in schools. General suggestions for training • Follow up requests from schools for library management training, training for design and development of appropriate teaching aids, and office administrative skills training. • Consider more focused Information and Communication Technology (ICT) training for schools with identified needs such as office administration or teachers to increase access to resources and information online. • Provide schools with an Active Learning manual/handbook or a teacher’s guide with a CD that can be kept in the teachers’ room as a reference. These can be used as individual resources or to provide relevant guidance at KKG/MGMP meetings. Principals • Work closely with principals in MSB training to identify appropriate categories of BOS funds that support teachers in classrooms. This could include funds to buy teaching-learning materials, resources or travel to KKG/MGMP meetings, more books, or hiring of specific subject specialists to give training in math, science, and language. • Provide a stronger leadership component in MBS training for principals to increase the effectiveness and efficiency of school management. MBS training has an emphasis on organizational and management skills while principals could benefit from training that builds leadership and character capacities (e.g. visioning, ethics, personal needs assessment, etc.). • Provision of more focused training for principals was suggested in the following areas:  Teacher performance evaluation skills;  Increased understanding and support of Contextual Teaching and Learning;  Increased ability to activate and guide community proposal writing. Provincial and District Offices • Facilitate meetings with motivated heads of each level to identify ways Provincial offices can put funds into teacher quality improvement. This could be for example, through provision of grants to schools, ‘adoption’ of under-resourced schools to pay for teacher training or materials, or expansion of dissemination efforts. District input would assist to identify those schools needing support. The agreement between Provincial and District Offices should also include identification of monitoring responsibilities possibly through partnerships with training service providers. Reading: • Work with schools, communities, local governments, and the private sector to provide more readers and reference books for both primary and junior secondary schools that have been actively engaged in supporting a reading culture. Many books at schools are seen to be well￾used but tattered and worn; a wider variety of grade/age appropriate books are needed. 47 • Enhance and improve the literacy language training elements in the existing training modules with special attention given to training language teachers in the use of modern interactive language teaching methods. Services of private providers may be useful to expand the timeframe and geographic coverage of such training. Teacher Support • Use KKGs/MGMPs as the main vehicle for in-service teacher training as they are local and provide immediate access to a ‘community of practice’ for teachers. Advocate with Districts and schools to explore funding schemes to support KKG/MGMPs. This assistance is especially needed for madrasah and contract teachers who do not have access to sufficient BOS funding. Focus stakeholders’ attention on building a mechanism for consistent provision of qualified trainers and materials to ensure these groups provide the best quality in-service training to teachers. • Explore avenues with schools for increased funding to buy or make more teaching-learning materials. Implementation of active learning techniques requires more materials and teachers can always use new teaching aids. • Schedule all project training, especially that involving teachers and principals, to take place during out-of-school hours or on school holidays to reduce unnecessary disruption to school schedules. • Include honoraria teachers in all trainings where possible. They form the bulk of many teaching staffs at many schools but receive limited support to improve quality. Teacher Training Institutes • Provide lecturers with AL refresher training on methodology. Individuals who have not been trained by the project should be included; that way PRIORITAS can also provide some needed quality assurance to the training process. • Support lecturers to provide “micro-teaching” classes/presentations in lab or partner schools so teacher training becomes more school-focused and practical. This would benefit both lecturers who have not had a lot of exposure to classroom environments and children and for student teachers who are learning how to teach. 48 ANNEXES ANNEX I: Terms of Reference for the Mid-term Evaluation of USAID PRIORITAS ANNEX II: List of Documents Referenced ANNEX III: Interview Guides ANNEX IV: Survey Instruments ANNEX V: Classroom Observation Form, Measurement Criteria, and Rating Scale ANNEX VI: List of Interviewees and Contacts ANNEX VII: Field Travel Schedule ANNEX VIII: Classroom Observation Data ANNEX IX: Select Survey Data ANNEX X: School Interview Data 49 ANNEX I: Terms of Reference for the Mid-term Evaluation of USAID PRIORITAS TITLE Mid-Term Performance Evaluation of USAID/Indonesia’s Prioritizing Reform, Innovation, Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) Program. I. DESCRIPTION AND STATEMENT OF WORK The USAID PRIORITAS Program USAID’s involvement in improving the quality of basic education in Indonesia began with the Managing Basic Education (MBE) Project in 2003. This four-year $10M activity resulted in visible changes in classroom environments, improved school management and better teacher performance through use of active learning methodologies. The project also enhanced parental and community involvement in school activities. MBE was followed by three separate Decentralized Basic Education Projects (DBE 1, 2 and 3) begun in 2005 and finished in December 2011. Collectively, the DBE program has improved education accountability and oversight by districts and schools, enhanced pre- and in-service teacher training and certification programs and supported new initiatives in kindergartens, inclusive education and tertiary professional development. This project, Prioritizing Reform, Innovation and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) will further consolidate gains made under the DBE programs and will focus on capacity building and quality improvements for teacher training institutions (TTI)43 and for non-academic, in-service teacher training providers. PRIORITAS will also build the capacity of MOEC, MORA and pre- and in-service teacher training organizations to coordinate, plan and train. In addition to teacher training, PRIORITAS will improve education management and governance in schools and districts. The components of PRIORITAS are included below; each of these components is necessary to support the overall improvement of quality education. • Strengthen the capacity of selected pre-service teacher training institutes and in-service teacher training organizations to produce skilled primary and junior secondary teachers, competent and practiced in active learning methodologies with enhanced capability to teach reading, math and science. • Strengthen and expand provincial and district capacity to improve education management and governance from the school level up particularly as it relates to improving revenue streams to directly support teacher development and improved learning. • Strengthen coordination and feed-back systems across all levels of the GOI decentralized education system and key educational institutions. Currently, the five-year project is in the third year of implementation. This current phase of PRIORITAS is working with 16 Teacher Training Institutions and the Provincial Quality 43 For the purposes of this document, teacher training institutes (TTI) will be a generic term which includes both those college which were founded specifically to train teachers and university education departments. 50 Institutes, over 1,000 primary and junior secondary schools and madrasah in the 43 new USAID PRIORITAS partner districts, reaching 13,000 teachers and 180,000 students to improve access to quality education for children in Indonesia. In addition, PRIORITAS is working across 89 partner districts and regional governments in Aceh, North Sumatra, Banten, West Java, Central Java, East Java, and South Sulawesi. The project provided grants to two local organizations in Papua to work at the school level improving the capacity of teachers and school management. II. PURPOSE OF TH EVALUATION The purpose of this mid-term evaluation is to assess the performance of the PRIORITAS project on the quality and relevance of basic education in primary and junior secondary schools in Indonesia. More specifically, the evaluation will assess the progress of the PRIORITAS project in terms of achieving their primary goals of: 1) improving the quality and relevance of teaching and learning in schools through pre and in-service training; 2) developing better management and governance in schools and districts; 3) supporting better coordination within and between schools, teacher training institutions (TTIs) and government at all levels. This evaluation will measure the degree to which these three goals have been met and the contributing factors that have been responsible for, or detracted from, the achievement of these goals. The evaluation will identify programmatic, management, and/or financial obstacles and challenges affecting program implementation and recommend changes in program or management strategies to increase the efficiency and impact of the program. The evaluation will cover the PRIORITAS project performance up to the mid-point of the project (approximately 2.5 years of project implementation). Finally, this evaluation will provide an initial assessment of the sustainability of project’s achievements and the factors that have contributed to or detracted from the sustainability of project’s achievements to USAID Indonesia as well as the GOI. III. EVALUATION QUESTIONS The contractor shall provide evaluation services sufficient to achieve the objectives set forth above. Specifically, the evaluation should address the questions that follow. Offerors are encouraged to propose additional evaluation sub-questions for review. 1. To what extent has PRIORITAS achieved its stated mid-point objectives and outcomes in a timely and effective manner? 2. Is PRIORITAS on track in terms of meeting its overall end-of-project goals?44 3. What aspects of PRIORITAS are proving most and least effective in improving access to quality education in Indonesia, for each of the PRIORITAS project goals? 4. To what extent have PRIORITAS stakeholders45 benefited from the project’s activities and what specific value has been added? 5. To what extent are the PRIORITAS project’s resources being implemented and managed efficiently and cost effectively? 6. What evidence is there to indicate that student reading and reading comprehension skills have improved as a result of PRIORITAS interventions? 7. What evidence is there to show that the PRIORITAS project’s activities and results are 44 The PRIORITAS program seeks to (1) improve the quality and relevance of teaching and learning in schools through pre￾and in-service training; (2) develop better management and governance in schools and districts; (3) support better coordination within and between schools, teacher training institutions (TTIs) and government at all levels. 45 PRIORITAS stakeholders include: Students, teachers, parents, community members, university partners, provincial and district officials, Menko Kesra, MOEC and MORA. 51 making progress towards sustainability and replication after the project is completed? 8. How effective has each component of the PRIORITAS project been in improving gender equality among students in schools? The contractor shall present evaluation findings to substantiate answers to these evaluation questions; findings that are based on facts, evidence, and data. Findings should be specific, concise, and supported by quantitative and qualitative information that is reliable, valid, and generalizable. Recommendations must be action-oriented, practical and specific. IV. METHODOLOGY As part of the technical proposal the Offerors must outline evaluation methodology or mix of methodologies for answering r the questions above that is effective and cost efficient. The contractor will examine both quantitative and qualitative approaches and a combination of secondary (existing) and primary (new) data during the course of the evaluation. Offerors are encouraged to include the following stakeholders in the evaluation: PRIORITAS Stakeholders: • USAID/Indonesia PRIORITAS team members • RTI International and local partners • MOEC, MORA and KEMENKO KESRA officials of relevant units • Heads of provincial and district education offices (minimum 4 provinces out of 8, and 3 districts within each province46) • Teacher Training Institution (TTI) lecturers and managers • School principals and supervisors, teachers, training facilitators • Students, parents, and girls/boys • Donor agencies The contractor must follow the guidance published in the January 2011 document entitled, “USAID Evaluation Policy.” In particular, the contractor should carefully review section 5 entitled, “Evaluation Requirements.” One example of the many points highlighted in this article, where available, the evaluation should use sex-disaggregated data and incorporates attention to gender relations in all relevant areas. (See Appendix 2 for more details) V. COMPOSITION OF THE EVALUATION TEAM There are two key positions outlined in this RFP, the Evaluation Team Leader and the Evaluation Expert. Offerors must provide CVs for these positions. Beyond these two key positions the offeror is encouraged to propose at least two professional Indonesian experts in basic education focusing on teacher competency and school-based management and any kind of personnel structure deemed appropriate to conduct the work outlined in this SOW. Evaluation Team Leader: The Team Leader should possess graduate-level degree (Ph.D. or master’s degree, or Indonesian equivalent S3 or S2) in education, the social sciences, or a related 46 Papua is the eighth PRIORITAS province. Work in this province began in early 2014; the evaluation will focus on the seven target districts but will review the approach in Papua rather than the progress towards implementation. 52 relevant field. The Team Leader should also have a minimum of five years of working experience with basic education evaluations, ten years of working experience with basic education activities and prior experience working in Indonesia. The offerors must provide an example of an evaluation report written by the team leader. Evaluation Expert: The Evaluation Expert should possess graduate-level degree (Ph.D. or master’s degree) in education (or relevant field) and a minimum of seven years of planning and evaluating education assistance projects. S/he must also have specific skills in evaluation methodology and planning, including demonstrated experience in developing evaluation methodologies and managing teams in primary data collection. Experience working in Indonesia is preferred as well as skills in gender analysis. The offerors must include a sample evaluation plan for a similar evaluation completed by the evaluation expert. The full composition of the evaluation team and the roles of the Evaluation Team Leader, the Evaluation Expert and other team members should be defined and delineated in the Technical Proposal. VI. LIST OF DOCUMENTS FOR EVALUATION TEAM TO REVIEW The following are a list of documents that will be forwarded to the Evaluation Team for review prior their arrival in Indonesia. • Modules, tools, and training materials developed by the PRIORITAS program • Monitoring and Evaluation reports including Early Grade Reading Assessment (EGRA) • Quarterly reports and Annual report by the PRIORITAS program • Annual Work Plan by the program • Dissemination and Sustainability Study of DBE and PRIORITAS programs • Scope of Work of the PRIORITAS program including Amendments to the Scope of Work • Final Evaluation of the Decentralized Basic Education Program. • The Agency’s recent “USAID Evaluation Policy” report and ADS 203 53 ANNEX II: List of Documents Referenced References in English ACDP Indonesia (2014) “Education Reform, the Nation’s Big Homework.” Kompas, August 28, 2014 Cannon, Robert, R. Arlianti, and I. Riu. (2014). Dissemination and Sustainability of DBE and PRIORITAS Programs. Research Paper. Jakarta: PRIORITAS USAID/ Indonesia. Chang, M.C., S. Shaeffer, S. Al-Samarrai, A. Ragatz, J. de Ree, and R. Stevenson (2014) Teacher Reform in Indonesia: the Role of Politics and Evidence in Policy Making. Executive Summary. Jakarta: World Bank. Evans, D, S. Tate, L. Losert, N. Noni, M. B. Wello, A. Purwadi, N. A. Bimo, T. Savitri, and A. Setiadi (2012) Final Evaluation of USAID/Indonesia’s Decentralized Basic Education (DBE) Project. Jakarta: USAID/Indonesia. Evans, D., S. Tate, R. Navarro, and M. Nicolls (2009) Teacher Education and Professional Development in Indonesia: A Gap Analysis. Jakarta: JBS International and USAID/Indonesia. Heyward, M., R. Cannon, and Sarjono (2011) Implementing School-based Management in Indonesia, RTI Press No. OP-0006-1109. Research Triangle Park, NC: RTI Press. Overseas Development Institute (2014) Towards Better Education Quality: Indonesia’s Promising Path. London, England. PRIORITAS. (April 2013 – September 2014). PRIORITAS PENDIDIKAN Editions 3, 5, 7, 8. Jakarta, Indonesia: USAID PRIORITAS. Raihani (2007) “Education Reforms in Indonesia in the Twenty-first Century.” International Education Journal, 2007, 8(1)m 172-183. http://iej.com.au RTI International. (May 2012 – October 2014). Quarterly Reports Number 1 - Number 10. USAID PRIORITAS Reports. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012a). Annual Report (May–September 2012). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012b). Collaborative Governance and Management Needs Assessment. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012c). Inclusive Education Policy Study. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012d). Laporan Analisis Situasi Gender Terkait Permasalahan Proses Belajar dan Prestasi Murid (Report on Gender Situation Analysis Related to Student Learning and Achievement). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012e). The Role of the Province in Governing and Managing Basic Education. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2012f). Year One Detailed Work Plan May 7, 2012–September 30, 2013 and Indicative Five Year Plan. USAID PRIORITAS (USAID Prioritizing Reform, Innovation, and Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students) Document. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2013a). Annual Report (October 2012–September 2013). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. 54 RTI International. (2013b). Baseline Monitoring Report, Volume 1 (Revised Version): Assessing the Impact of the USAID PRIORITAS Program on Teacher Training Institutes (TTIs), Districts and Schools. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2013c). Baseline Monitoring Report, Volume 2: Assessing the Impact of the USAID PRIORITAS Program on Student Performance in Bahasa Indonesia, Mathematics, and Science. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2013d). Baseline Monitoring Report, Volume 3: An Assessment of Early Grade Reading—How Well Children are Reading. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2013e). Report on Gender Situation Analysis Related to Student Learning and Achievement. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2013f). Year 2 Detailed Work Plan May 1, 2013-April 30, 2014 and Indicative Five Year Plan. PRIORITAS Document. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014a). Annual Report (October 2013–September 2014). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014b). Baseline Monitoring Report, Volume 1: Assessing the Impact of the USAID PRIORITAS Program on Schools in Cohort 2 Districts. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014c). Baseline Monitoring Report, Volume 1: Assessing the Impact of the USAID PRIORITAS Program on Student Performance (Baseline Assessment of a Sample of Students in Primary and Junior Secondary Schools in Cohort 2 Partner Districts). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014d). Baseline Monitoring Report, Volume 3: An Assessment of Early Grade Reading - How Well Children are Reading in Cohort 2 Districts. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014e). Collaborative District Governance and Management Needs Assessment— Cohort 2. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014f). Draft Report on Teacher Distribution (Penataan & Pemerataan Guru). PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014g). Second Round Monitoring Report: Assessing the Impact of the USAID PRIORITAS Program on Schools in Cohort 1 Districts. PRIORITAS Report. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014h). USAID PRIORITAS Performance Management Plan. PRIORITAS Document. Research Triangle Park, NC: PRIORITAS USAID/ Indonesia. RTI International. (2014i). Year 3 Detailed Work Plan May 1, 2014 - April 30, 2015 and Indicative Five Year Plan. PRIORITAS. Research Triangle Park, NC: USAID PRIORITAS. USAID/ Indonesia (2013) Investing in Indonesia: A Stronger Indonesia Advancing National and Global Development. Country Development Cooperation Strategy. Jakarta: USAID. References in Indonesian Balitbang Provinsi Sumatera Utara. (2014). Kajian Distribusi Guru Di Sumatera Utara oleh Tim Peneliti, Jaringan Penelitian dan Pengembangan Pendidikan dan Kebudayaan Provinsi Sumatera Utara. Medan, Provinsi Sumatera Utara: Badan Penelitian dan Pengembangan Provinsi Sumatera Utara. 55 Bupati Bener Meriah. (2014). Peraturan Bupati Nomor 07 Tahun 2014 Tentang Penataan dan Pemerataan Guru Pegawai Negeri Sipil (PNS) Di Lingkungan Pemerintah Kabupaten Bener Meriah. Takengon: Kabupaten Bener Meriah. Bupati Labuhanbatu. (2013). Draft Peraturan Bupati Labuhanbatu Nomor … Tahun 2014 Tentang Program Akselerasi Peningkatan Mutu Pembelajaran Pendidikan Dasar Di Kabupaten Labuhanbatu. Kota Rantau Prapat: Kabupaten Labuhanbatu. Bupati Labuhanbatu. (2013). Keputusan Bupati Labuhanbatu Nomor 420/226/DISDIK/2013 tentang Penetapan Fasilitator Daerah Mitra USAID PRIORITAS tingkat Kabupaten Labuhanbatu. Kota Rantau Prapat: Kabupaten Labuhanbatu. Bupati Labuhanbatu. (2015). Keputusan Bupati Labuhanbatu Nomor 440.441/18/DISDIK/ 2015 tentang Pembentukan Tim Pengembangan dan Pendampingan Program Akselerasi Peningkatan Mutu Pembelajaran Pendidikan Dasar Di Kabupaten Labuhanbatu. Kota Rantau Prapat: Kabupaten Labuhanbatu. Bupati Maros. (2014). Peraturan Bupati Maros Nomor 49 Tahun 2014 Tentang Penggunaan Dana Sharing Pendidikan Gratis Pemerintah Kabupaten Maros. Maros: Kabupaten Maros. Bupati Maros. (2014). Keputusan Bupati Maros Nomor 729/KPTS/421/II/2014 Tentang Penunjukan Fasilitator Daerah Tingkat SD/MI dan SMP/MTs Program USAID PRIORITAS Kabupaten Maros Tahun 2014. Maros: Kabupaten Maros. Dinas Pendidikan Kabupaten Maros. (2015). Surat Tugas Nomor: 800/035/Disdik/2015 tentang fasilitator daerah ditugaskan untuk melakukan kegiatan pendampingan praktik yang baik dalam pembelajaran dan Manajemen Sekolah Implementasi Modul II USAID PRIORITAS. Maros: Dinas Pendidikan Kabupaten Maros. Dinas Pendidikan Kabupaten Aceh Jaya. (2015). Rencana Kerja dan Anggaran Satuan Kerja Perangkat Daerah Tahun Anggaran 2015. Calang: Kabupaten Aceh Jaya. Dinas Pendidikan Kabupaten Bener Meriah. (2014). Profil Pendidikan Kabupaten Bener Meriah Tahun Pelajaran 2012/2013 dan 2013/2014. Takengon: Dinas Pendidikan Kabupaten Bener Meriah. Forum Fasda Maros. (2015). Jadwal Pendampingan Sekolah Hasil Pelatihan Modul II, Kecamatan Turikale dan Bantimurung Tahun 2015. Kota Maros: Forum Fasda Maros. Forum Fasda Maros. (2015). Sekolah Sasaran Pendampingan Hasil Pelatihan Module II. Kota Maros: Forum Fasda Maros. Forum Fasda Maros. (2015). Jadwal Kelompok Kerja Guru (KKG) Kecamatan Turikale Kabupaten Maros – Semester Genap Tahun Pelajaran 2014/2015. Kota Maros: Forum Fasda Maros. Gubernur Sulawesi Selatan. (2013). Keputusan Gubernur Sulawesi Selatan, No.188.4/PD.8/1440/2013 Tentang Penetapan TK/RA, SD/MI/ SMP/SMK/MA Piloting (Model) Penyelenggara Inklusif Propinsi Sulawesi Selatan. Makassar: Pemerintah Provinsi Sulawesi Selatan. Kepala Dinas Pendidikan Kabupaten Labuhanbatu. (2014). Draft Keputusan Kepala Dinas Pendidikan Kabupaten Labuhan Batu Nomor 800/…….Sekr.2/2014 tentang Pengangkatan dan PenetapanTim Pengembangan dan Pendampingan USAID PRIORITAS, Akselerasi Dinas Pendidikan Kabupaten Labuhanbatu Tahun Anggaran 2014. Kota Rantau Prapat: Pemerintah Kabupaten Labuhanbatu. LPMP Jawa Timur. (2014). Best Practice Pelaksanaan Kegiatan Sekolah Berbasis Standar Nasional Pendidikan (SB-SNP) Provinsi Jawa Timur Tahun 2014. Dibuat Oleh Lembaga Penjaminan Mutu Pendidikan (LPMP), Badan Pengembangan Sumber Daya Manusia Pendidikan dan Kebudayaan dan Penjaminan Mutu Pendidikan, Kementerian Pendidikan dan Kebudayaan. Surabaya: LPMP Jawa Timur. 56 MIN Padang Bulan. (2015). Rincian Kertas Kerja Satker T.A. 2015, Unit Kerja: MIN Padang Bulan Kab Labuhan Batu. Kecamatan Rantau Utara, Kabupaten Labuhanbatu, Sumatera Utara: MIN Padang Bulan. MIN Urung Kompas. (2014). Profil MIN Urung Kompas. Kecamatan Rantau Selatan, Kabupaten Labuhanbatu, Sumatera Utara: MIN Urung Kompas. MIN Urung Kompas. (2015). Draft Anggaran MIN Urung Kompas Tahun 2015. Kecamatan Rantau Selatan, Kabupaten Labuhanbatu, Sumatera Utara: MIN Urung Kompas. MTs Al Ittihad Aek Nabara. (2015). Draft Rencana Kegiatan Madrasah (RKM). Kecamatan Bilah Hulu, Kabupaten Labuhan, Sumatera Utara: MTs Al Ittihad Aeh Nabara. MTs Al Ittihad Aek Nabara. (2015). Profil MTs Swasta Al Ittihad Aek Nabara. Kecamatan Bilah Hulu, Kabupaten Labuhan, Sumatera Utara: MTs Al Ittihad Aeh Nabara. MTs DDI Alliritengae. (2014). Laporan Bulanan: December 2014. Kecamatan Turilake, Kabupaten Maros, Sulawesi Selatan: MTs DDI Alliritengae. MTs DDI Alliritengae. (2014). Rencana Kegiatan dan Anggaran Madrasah (RKAM) MTs DDI Alliritengae Tahun Pelajaran 2014/2015 . Kecamatan Turilake, Kabupaten Maros, Sulawesi Selatan: MTs DDI Alliritengae. MTs DDI Alliritengae. (2014). Rencana Kerja Tahunan Madrasah (RKTM) MTs DDI Alliritengae Tahun Pelajaran 2014/2015 . Kecamatan Turilake, Kabupaten Maros, Sulawesi Selatan: MTs DDI Alliritengae. MTs Muhammadiyah Bantaeng. (2014). Kegiatan Rutin MTs Muhammadiyah Bantaeng. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: MTs Muhammadiyah Bantaeng. MTs Muhammadiyah Bantaeng. (2015). Rencana Kegiatan dan Anggaran Sekolah (RKAS) Tahun Anggaran 2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: MTs Muhammadiyah Bantaeng. MTsN Jambewangi. (2015). Profil MTsN Jambewangi. Kecamatan Selopuro, Kabupaten Blitar, Jawa Timur: MTsN Jambewangi. MTsN Jambewangi. (2015). Rincian Kerjtas Kerja Satker T.A. 2015, Unit Kerja: MTsN Jambewangi. Kecamatan Selopuro, Kabupaten Blitar, Jawa Timur: MTsN Jambewangi. MTsN Lamno. (2013). Laporan Bulanan: September 2013. Lamno, Kabupaten Aceh Jaya, Aceh: MTsN Lamno. MTsN Lamno. (2015). Rincian Kertas Kerja Satker T.A. 2015, Unit Kerja: MTsN Lamno Kab. Aceh Jaya. Lamno, Kabupaten Aceh Jaya, Aceh: MTsN Lamno. MTsN 2 Medan. (2014). Profil MTsN 2 Medan Tahun Pembelajaran 2014/2015. Medan, Sumatera Utara: MTsN 2 Medan. MTsN 2 Medan. (2013). Rencana Kerja Madrasah (RKM), Rencana Program dan Rencana Kerja Tahunan (RKT) Tahun Pembelajaran 2013/2014. Medan, Sumatera Utara: MTsN 2 Medan. MTsN Pademawu. (2015). Laporan Bulanan: Maret 2015. Kecamatan Pademawu, Kabupaten Pamekasan, Jawa Timur: MTsN Pademawu. SDN 1 Pakalu I. (2015). Laporan Bulanan: Januari 2015. Kecamatan Kalabbirang, Kabupaten Maros, Sulawesi Selatan: SDN 1 Pakalu I. SDN 1 Pakalu I. (2013). Rencana Kerja Tahunan Sekolah (RKT) SDN 1 Pakalu I Tahun Pelajaran 2013/2014. Kecamatan Kalabbirang, Kabupaten Maros, Sulawesi Selatan: SDN 1 Pakalu I. 57 SDN 7 Letta. (2015). Dokumen Pelaksanaan Anggaran Satuan Kerja Perangkat Daerah Tahun Anggaran 2015: BOS SD No. 7 Letta. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: SDN 7 Letta SDN 7 Letta. (2015). Draft Rencana Kegiatan dan Anggaran (RKAS) Tahun Anggaran 2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: SDN 7 Letta SDN 7 Letta - Gugus. (2014). Jadwal Program Kerja: Kelompok Kerja Kepala Sekolah (K3S) Semester I Tahun Pelajaran 2014-2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: Gugus SDN 7 Letta SDN 7 Letta. (2015). Laporan Bulanan: Januari 2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: SDN 7 Letta SDN 7 Letta - Gugus. (2014). Program Kegiatan Kelompok Kerja Guru (KKG) Semester I Tahun Pelajaran 2014-2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: Gugus SDN 7 Letta SDN 7 Letta. (2014). Rincian Penggunaan Gratis Tahun Anggaran 2014/2015. Kecamatan Bantaeng, Kabupaten Bantaeng, Sulawesi Selatan: SDN 7 Letta SDN 15 Jawi-Jawi. (2013). Rencana Kegiatan & Anggaran Sekolah Tahun 2015. Kecamatan Bantimurung, Kabupaten Maros: SDN 15 Jawi-Jawi. SDN 15 Jawi-Jawi. (2013). Rencana Kerja Tahunan & Rencana Kegiatan Anggaran Sekolah Tahun Ajaran 2013/2014. Kecamatan Bantimurung, Kabupaten Maros: SDN 15 Jawi-Jawi. SDN 15 Jawi-Jawi. (2015). Rencana Pelaksanaan Pembelajaran (RPP) Matematika Kelas VI/Semester 2 Tahun 2015 oleh Musdalipah, S.Pd. Kecamatan Bantimurung, Kabupaten Maros: SDN 15 Jawi￾Jawi. SD Inpres Tappanjeng. (2015). Program Anggaran Pembelajaran Tahun Anggaran 2015. Kecamatan Bantaeng, Kabupaten Bantaeng: SD Inpres Tappanjeng. SD Inpres Tappanjeng. (2015). Rencana Kerja Tahunan (RKT) Dana BOS Tahun Anggaran 2015. Kecamatan Bantaeng, Kabupaten Bantaeng: SD Inpres Tappanjeng. SDN Jalmak 1. (2015). Laporan Bulanan: Januari 2015. Kecamatan Pamekasan, Kabupaten Pamekasan, Jawa Timur: SDN Jalmak 1. SDN Kalipang 01. (2015). Rencana Pelaksanaan Pembelajaran (RPP) Mata Pelajaran Lingkungan, Tema 6. Lingkungan Bersih, Sehat, dan Asri Kelas I/Semester II oleh Puji Rahayu, S.Pd. Kecamatan Sutojayan, Kabupaten Blitar: SDN-SN Kalipang 01. SDN Kalipang 01. (2014). Rencana Kerja Tahunan (RKT) Tahun 2014-2015. Kecamatan Sutojayan, Kabupaten Blitar: SDN-SN Kalipang 01. SDN Kalipang 03. (2014). Profil Sekolah. Kecamatan Sutojayan, Kabupaten Blitar, Jawa Timur: SDN Kalipang 03. SDN Kebonduren 01. (2015). Rencana Pelaksanaan Pembelajaran (RPP) Mata Pelajaran Matematika Kelas V Semester II Tahun Ajaran 2014/2015 oleh Semiyati, S.Pd. Kecamatan Ponggok, Kabupaten Blitar: SDN Kebonduren 01. SDN Kebonduren 01. (2015). Rencana Kegiatan dan Angaran Sekolah (RKAS) Tahun Anggaran 2015. Kecamatan Ponggok, Kabupaten Blitar: SDN Kebonduren 01. SDN No. 116879 Sukamulia. (2015). Laporan Bulanan: Januari 2015. Kecamatan Bilah Hulu, Kabupaten Labuhanbatu, Sumatera Utara: SDN No. 116879 Sukamulia. 58 SDN Pademawu Barat 02. Laporan Bulanan: Februari 2015. Kecamatan Pademawu, Kabupaten Pamekasan, Jawa Timur: SDN Pademawu Barat 02. SDN Pademawu Timur 2. (2014). Rencana Anggaran Pendapatan dan Belanja Sekolah (RAPBS) Tahun Ajaran 2014/2015. Kecamatan Padewmawu, Kabupaten Pamekasan: SDN Pademawu Timur 2. SD Swasta No. 1 HKBP. (2014). Pakta Integritas, Dewan Pendidikan dan Komite Sekolah/Madrasah SD Swasta No. 1 HKBP Lintongnihuta. Kelurahan Lintong Nihuta, Kabupaten Humbang Hasudutan: SD Swasta No. 1 HKBP. SMPN 1 Rantau Utara. (2015). Laporan Bulanan: Januari 2015. Kecamatan Rantau Utara, Kabupaten Labuhanbatu, Sumatera Utara: SMPN 1 Rantau Utara. SMPN 1 Rantau Utara. (2014). Rencana Anggaran Pendapatan dan Belanja Sekolah (RAPBS) Tahun: 2014. Kecamatan Rantau Utara, Kabupaten Labuhanbatu, Sumatera Utara: SMPN 1 Rantau Utara. SMPN 1 Sanankulon. (2015). Paparan Visi & Misi dan Kemajuan Sebagai Mitra USAID PRIORITAS. Kecamatan Sanankulon, Kabupaten Blitar: SMPN 1 Sanankulon. SMPN 1 Sanankulon. (2015). Profil Sekolah. Kecamatan Sanankulon, Kabupaten Blitar: SMPN 1 Sanankulon. SMPN2 Makassar. (2014). Buletin Sekolah Edisi Perdana Oktober 2014. Kota Makassar: SMP Negeri 2 Makassar. SMP Negeri 2 Makassar. (2014). Daftar Keadaan Guru SLTP. Kota Makassar: SMP Negeri 2 Makassar. SMPN2 Makassar. (2015). Rencana Pembelajaran untuk IPA Kelas II/Semester Genap – Energi Dalam Sistem Kehidupan. Kota Makassar: SMP Negeri 2 Makassar. SMPN2 Makassar. (2015). Rencana Pembelajaran untuk IPA Kelas II/Semester Genap – Pengertian Suhu dan Termometer. Kota Makassar: SMP Negeri 2 Makassar. SMPN 4 Bantimurung. (2014). Keputusan Kepala Sekolah Menengah Pertama Negeri 4 Bantimurung, Kabupaten Maros Nomor 006/106.1/SMP.06/KP/2014. Kecamatan Bantimurung, Kabupaten Maros: SMPN 4 Bantimurung. Tim PPG Kota Medan. (2014). Laporan Analisis Pemetaan untuk Penataan & Pemerataan Guru (PPG) Di Kota Medan (Menuju Implementasi Peraturan Walikota Nomor 54 Tahun 2012 Tentang Penataan & Pemerataan Guru PNS Di Kota Medan. Medan: Tim PPG Kota Medan. USAID PRIORITAS Aceh. (Agustus 2013 – December 2014). SEURAMOE PRIORITAS Edisi V, VI, VII, VIII, IX. Banda Aceh, Aceh: USAID PRIORITAS. USAID PRIORITAS Jawa Timur. (2014). SWARA PRIORITAS Edisi No. 07 Tahun 2014. Surabaya, Jawa Timur: USAID PRIORITAS. USAID PRIORITAS Sumatera Utara. (2014). KABAR PRIORITAS Edisi No. 05 Tahun 2014. Medan, Sumatera Utara: USAID PRIORITAS. USAID PRIORITAS Sumatera Utara. (2015). Paparan tentang Implementasi Program di Provinsi Sumatera Utara. Medan, Sumatera Utara: USAID PRIORITAS. UPTD Pendidikan Kec. Bantimurung. (2014). Pemberitahuan Tentang Program Kerja K 3 S Wilayah I Semester II Tahun Pelajaran 2013/2014. Bantimurung, Kabupaten Maros: UPTD Pendidikan Kecamatan Bantimurung. Walikota Binjai. (2014). Peraturan Walikota Binjai No 22 Tahun 2014 Tentang Penyelenggaraan Pendidikan Inklusif di Kota Binjai. Kota Binjai: Pemerintah Kota Binjai 59 ANNEX III: Interview Guides District DINAS and MORA District Office The objective of this interview is to obtain opinions of District DINAS and MORA personnel on PRIORITAS performance to strengthen education through a) improved teacher preparation, b) stronger education management and governance, and c) increased cooperation among province, district, and school education actors. 1. What is your involvement with PRIORITAS? 2. What activities from PRIORITAS are most useful to improve education in your schools? 3. How is your own professional knowledge and expertise improved as a result of involvement with PRIORITAS? 4. How have you used PRIORITAS training to improve teacher deployment and training in your District? 5. How do you work with the Provincial Office, TTIS, and LPMPs as a result of PRIORITAS? 6. What good practices from PRIORITAS are you disseminating in your district? 7. Have PRIORITAS inputs, supports, and training to your office been adequate and on schedule to help your work? 8. What more support do you need from PRIORITAS to continue education improvements in your district? 9. Are there any additional points or comments that you would like to make about PRIORITAS not mentioned here? MORA 10. What activities from PRIORITAS are most useful to improve education in your Madrasah? 11. How have you improved your personal knowledge and skills from participation with PRIORITAS? 12. How do you work together with the DINAS and TTI to improve quality of education? 13. Have PRIORITAS inputs, supports, and training to your office been adequate and on schedule to help your work? 14. What more support do you need from PRIORITAS to continue education improvements in your district? 15. Are there any additional points or comments that you would like to make about PRIORITAS not mentioned here? 60 District Facilitators The objectives of this interview are to collect opinions from district facilitators about the PRIORITAS program on: 1) the importance of your role in PRIORITAS to improve teaching and learning; 2) the effectiveness of the project activities to improve reading, science, and maths, and 3) the effectiveness of PRIORITAS to strengthen teacher development, stronger school governance and management, and coordination among TTIs, LPMPs, province, district, and schools. 1. How long have you been a district facilitator? Please tell us briefly about your own role and work as a district facilitator. 2. How effective is PRIORITAS training and support to increase capacity of education personnel, including your own, at the province/district/school levels? 3. Which PRIORITAS interventions are most effective to improve the quality of teaching and learning in target schools? 4. How has PRIORITAS changed teacher behavior in the classroom? 5. How do students respond to the new learning approaches to reading, science, and math applied in classrooms? 6. How has PRIORITAS changed behavior of principals to improve education in schools? 7. How has PRIORITAS improved community involvement with schools? 8. Has PRIORITAS been equally effective in public schools and Madrasa? Rural and urban schools? 9. How have District Education Offices improved their support to teacher training and teacher deployment as a result of PRIORITAS? 10. Have Teacher Training Institutions and LPMPs support to in-service teacher training improved since PRIORITAS? 11. What changes have you seen in attitudes and actions to address gender and inclusive education at the district and in schools as a result of PRIORITAS? 12. Should facilitator positions continue after PRIORITAS to support teaching and learning? How can the position of Facilitator be continued after PRIORITAS? 13. What feedback would you give to PRIORITAS to improve the position of Facilitator? 14. Are there any additional points or comments that you would like to make about PRIORITAS? 61 Jr Secondary Teachers The objective of the group interview is to obtain information from primary teachers on the effectiveness of PRIORITAS to improve a) teacher preparation and training, b) school management and governance, and c) collaboration among education stakeholders. 1. Please tell us how you were involved in PRIORITAS. 2. What PRIORITAS activities are most effective to improve teaching and learning in your classroom? 3. How effective is PRIORITAS to change your own teaching ability? What skills or knowledge did you improve as a result of involvement in PRIORITAS? 4. Is teaching and learning improving in your school? What is the evidence that students are learning? 5. How do students respond to the Contextual Learning approach to reading, science, and math applied in classrooms? 6. What are the problems with applying the Contextual Learning approach? 7. How has PRIORITAS strengthened principals’ ability to manage the school? Does he provide you with help in the classroom? 8. What PRIORITAS good practices should be shared with other teachers and schools? 9. How has the involvement of TTIs and LPMPs helped you to improve teaching and learning process? 10. How does PRIOIRTAS help you to address gender and inclusion issues in your school? 11. How is the community/School Management Committee involved at your school since PRIORITAs? 12. Any other comments you would like to make about PRIORITAs not already covered here? 62 LPMP The objective of this interview is to obtain information on the perspectives and opinions of LPMPs and P4TKs on a) the performance of PRIORITAS and b) the effectiveness of project support to increase your institution’s capacity and input to teacher preparation, school governance, and improved coordination between provincial, district, and school levels. 1. Please tell us about the role of your institution to improve the quality of teachers and school management and governance in schools. 2. What PRIORITAS activities were you involved in? 3. What are your impressions of PRIORITAS efforts to improve teacher quality and student performance in classrooms in schools? 4. How has PRIORITAS strengthened the capacity of your LPMP to provide quality assurance to schools in your district? 5. How has PRIOIRITAS strengthened your personal professional knowledge and skills to do your job? 6. How effective is PRIORITAS assistance to link your institution with Teacher Training Institutes, Provincial and District Education Offices and schools to address in-service teacher training? 7. Has PRIORITAS assistance improved the capacity of your LPMP to use data in the EMIS? 8. How many of your staff serve as provincial or district facilitators? How relevant are the facilitators to your institution’s role of providing quality assurance? 9. How successful are PRIORITAS training modules and mentoring to improve your institution’s capacity to provide in-service training? 10. What gender and inclusion issues are you aware of in education? How do you address these issues through service to schools? 11. Are there any other topics related to PRIORITAS and the work of your institution that have not been discussed here that you would like to add? 63 MOEC, MORA, and KESRA Name: ____________________________ Title: _________________________________ Department: _______________________ Ministry: _____________________________ Interviewer: Date: The objective of this interview is to obtain information on the perspectives and opinions of government officials on a) their sense of ownership in the PRIORITAS program, b) the usefulness/effectiveness of PRIORITAS to improve quality of education, and c) PRIORITAS inputs to policy dialogue and education innovation. 1. To begin, please tell us what you know about PRIORITAS. 2. How effective has PRIORITAS been to assist your office to improve education quality? 3. Was MOEC/MORA/KESRA involved in the planning of PRIORITAS from the beginning? What kinds of inputs did central government bodies give to the planning of PRIORITAS? 4. What is your opinion about PRIORITAS efforts to improve teacher deployment and teacher development in Indonesia? 5. From your own knowledge or experience, what PRIORITAS activities have the most impact on national policy? For MORA 6. Has PRIORITAS improved education quality at Madrasah? 7. How is PRIORITAS programming effectively engaging MORA in the improvement education in Madrasah? 8. What changes does MORA contemplate undertaking at the provincial and district levels as a result of participation in PRIORITAS activities? For All 9. What gender and inclusive education issues are present in the education system? What does your office do to address them? 10. What PRIORITAS activities should be sustained after the project ends? What support can your office give to this effort? 11. What other issues about PRIORITAS do you think are important that haven’t been discussed? 64 Primary Principals Sub-district/District/Province: ______________________________________ School Name: ____________________________ Public School ___ Madrasah ___ Number of years at current post: Highest academic degree: Management training Yes ___ No ___ We are conducting a mid-term evaluation of PRIORITAS. We are collecting information to assess effectiveness of project activities to improve a) school governance and management, b) teacher instructional ability, and c) collaboration among education actors. We would like to have your opinions on PRIORITAS. 1. Which PRIORITAS activities did you participate in? 2. What is your opinion of PRIORITAS assistance to your school? 3. How was your school selected to be PRIORITAS’ a partner school? 4. Which of the following PRIORITAS modules do you feel were most effective to improve your school management skills and knowledge? 5. How has your understanding of quality education improved as a result of PRIORITAS? Do teachers at your school teach better as a result of what they learned in PRIORITAS? 6. What changes are evident in student achievement and participation in literacy, science, and math as a result of PRIORITAS efforts? 7. What plans have you made to improve teacher training in your school as a result of PRIORITAS? What funding has been allocated? 8. How has involvement of Teacher Training Institutes, the LPMP, and the Provincial Education Office improved to address in-service teacher training? 9. How has the capacity of KKGs improved as a result of PRIORITAS? School supervisors? 10. What support do you give to dissemination of good practices? 65 Primary Teachers The objective of the group interview is to obtain information from primary teachers on the effectiveness of PRIORITAS to improve a) teacher preparation and training, b) school management and governance, and c) collaboration among education stakeholders. 1. Please tell us how you were involved in PRIORITAS. What activities did you participate in? 2. What PRIORITAS activities are most effective to improve teaching and learning in your classroom? 3. What evidence is available to indicate that teaching and learning are improving in your school? What is the evidence that student reading and comprehension skills are improving? 4. How effective is PRIORITAS to change your own teaching ability? What skills or knowledge did you improve as a result of involvement in PRIORITAS? 5. How do students respond to the new learning approaches to reading, science, and math applied in classrooms? 6. How has PRIORITAS strengthened principals’ ability to provide you with help in the classroom? What examples can you give to illustrate this? 7. Have you been personally involved in school planning and management? 8. How has the involvement of TTIs and LPMPs helped you to improve teaching and learning process? 9. How does PRIOIRTAS help you to address gender and inclusion issues in your school? 10. How is the community/School Management Committee involved at your school as a result of PRIORITAs? 11. What additional activities would continue to improve teacher capacity and student learning until PRIORITAS ends? 12. Are PRIORITAS resources, training, and service adequate and delivered on time to help you in your classroom? 13. Any other comments you would like to make about PRIORITAs not already covered here? 66 Provincial DINAS Director/staff and MORA The objective of this interview is to obtain opinions of Provincial DINAS and MORA personnel on PRIORITAS performance to strengthen education through a) improved teacher preparation, b) stronger education management and governance, and c) increased cooperation among province, district, and school education actors. 1. What is your involvement with PRIORITAS? 2. What are the most significant issues your office deals with to improve education in the province? 3. What would you like to see change to allow your office more involvement in education provision at the schools? 4. How has PRIORITAS been effective to improve education in the province? 5. What plans have you made to improve in-service teacher training as a result of PRIORITAS activities? What funding have you set aside for in-service teacher training? 6. What plans have you made to improve teacher deployment as a result of PRIORITAS activities? 7. What will you do to disseminate PRIORITAS good practices? 8. Has communication and interaction between the Central Ministry, Provincial, and District governments changed since PRIORITAS? 9. How has your interaction with Teacher Training Institutes and the LPMP changed since PRIORITAS? How do you interact with them? 10. What gender and inclusion issues are you aware of in education in your province? What actions do you take to improve these issues? 11. Are there any additional points or comments that you would like to make about PRIORITAS? MORA Ask the relevant questions from above in addition to: 12. How do you work together with the DINAS to improve quality of education? 13. How has education in MORA schools improved as a result of PRIORITAS? 67 Junior Secondary School Principals We are conducting a mid-term evaluation of PRIORITAS. We are collecting information to assess effectiveness of project activities to improve a) school governance and management, b) teacher instructional ability, and c) collaboration among education actors. We would like to have your opinions on PRIORITAS. 1. What PRIORITAS activities did you participate in? 2. What is your opinion of PRIORITAS assistance to your school? 3. What skills and knowledge of your own improved as a result of PRIORITAS? What activities were most helpful to you? 4. How has the PRIORITAS strengthened your planning and management capacity at your school? 5. What plans have you made to improve teacher in-service training as a result of PRIORITAS? What have you done to increase funding to teacher training? 6. Have you employed the Teacher Deployment Strategy of PRIORITAS? Is it successful at your school? 7. How has teacher behavior in the classroom changed as a result of what they learned in PRIORITAS? 8. What changes are evident in student achievement and participation in literacy, science, and math as a result of PRIORITAS efforts? 9. How has the capacity of MGMPs improved as a result of PRIORITAS? School supervisors? 10. Have you had more involvement with the Teacher Training Institutes and LPMPs for in￾service teacher training since PRIORITAS? 11. Has the District Office become more involved in provision of in-service teacher training to your school? 12. In your opinion, what activities of PRIORITAS should be shared in other schools in Indonesia because they are effective and improve the education the students receive? 13. What do you do in gender and inclusive education with the district and in your school as a result or PRIORITAS? 14. Is there any other information that you would like to add about PRIORITAS not mentioned here? 68 Teacher Training Institutes (TTI) The objective of this interview is to obtain information on the perspectives and opinions of TTI personnel who work with PRIORITAS to improve education through a) improved teacher preparation, b) strengthened education management and governance, and c) increased cooperation among province, district, and school education actors. 1. Please tell us about the involvement of your TTI with PRIORITAS. What activities are you involved in? 2. How has the involvement of your institution in teacher training and school quality improved as a result of PRIORITAS? 3. How has your professional knowledge and skill improved as a result of PRIORITAS involvement? 4. In your opinion, which PRIORITAS activities are most useful to improve teaching and learning in the classroom? 5. How have pre-service teacher education and the practicum for student teachers changed as a result of PRIORITAS? 6. How involved is your institution in provision of in-service teacher training? 7. How do lecturers from your TTI gain an understanding of teacher needs in the classroom to train pre-service students? How often do you visit schools and classrooms? 8. What is your involvement in the development of curriculum for reading/literacy, math, and science? What progress has been made in the new curricula for these subjects? 9. Does your institution have any involvement with the LPMPs, provincial, and district offices to improve in-service teacher training? Which relationships (between which levels) are becoming stronger and more productive? How can these relationships be sustained? 10. Do you participate in the Classroom Action Research? How useful is it to improve pre￾service preparation? 11. What gender and inclusion issues are you aware of in education? What actions do you take to improve these issues? 12. Are there any additional comments you would like to make about PRIORITAS? 69 ANNEX IV: Survey Instruments The Surveys in the formats here are representational only. The actual Surveys differed in appearance and lay-out with the inclusion of an individual respondent identification section in each category. Additionally, the Surveys were administered in Bahasa Indonesian. District Education Officials Survey This survey will be conducted with at least six of the following nine officials located at the District Offices in each of the 43 Cohort 1 and 2 PRIORITAS districts: Bappeda, Head of District Education Office, MORA representative, local parliament (DPRA) member: Chairman of the Commission, Director of Sub-division Primary Education, Director of Sub-division Jr. Secondary Education, Director of Curriculum Division (KASI), Coordinator of Supervisors, and Coordinators of Facilitators. Member of Local Parliament USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What PRIORITAS activities are or have you been involved in? tick all that apply Teacher Mapping Teacher Deployment Analysis Teacher Cost Analysis Budget Allocations Teacher Training School Quality Improvement Strategy Planning Monitoring and Evaluation Data collection and use Showcase workshops 2. Has PRIORITAS benefitted you to plan and budget for education in your district? Y/N/Don’t know 3. Has PRIORITAS increased your knowledge of quality improvement needs in education? Y/N/Don’t know 4. Has PRIORITAS improved the quality of teaching and learning in District schools? Y/N/ Don’t know 5. Have you been more involved in school-level activities as a result of PRIORITAS? Y/N 6. Do you participate in PRIORITAS district policy workshops? Y?N 7. Do you participate in PRIORITAS national policy workshops? Y/N 8. Have you made new policies or plans in the District One-year or Four-year strategies to address specific issues in education in your district a result of PRIORITAS? Y/N/ if no, skip to #10 9. What new policies or plans have you made in your district to address specific education issues as a result of PRIORITAS? Tick all that apply. Teacher deployment Additional direct funding to needy schools Improved school supervision Improved teacher quality (training, certification, etc.) Improved school infrastructure and internal processes (calendars, instructional time, libraries, technology, etc.) Improved literacy (re-direct funding to literacy, books, libraries, teacher instruction, etc.) Increased community involvement (in planning and management, contributions, etc.) 70 Promotion of quality teachers and principals to senior positions Inclusive education and gender (teacher training, mapping of students, special materials, etc.) Other? 10. Have you supported dissemination of good practices to other schools and districts from PRIORITAS activities? Y/N/Don’t know 11. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 12. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know District Heads of Education (including Sub-Division Heads of Primary and Junior Secondary) USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What PRIORITAS activities are or have you been involved in? tick all that apply Teacher Mapping Teacher Deployment Analysis Teacher Cost Analysis Budget Allocations Teacher Training School Quality Improvement Strategy Planning Monitoring and Evaluation Data collection and use Showcase workshops Dissemination activities Contextual Teaching-Learning (CTL) School-based Management Active-learning (PAKEM) training School committees/community involvement Best practices replication 2. Has PRIORITAS given you valuable knowledge and tools to help you plan and budget for education needs in schools? Y/N/Don’t know 3. Has PRIORITAS increased your knowledge of quality improvements needed in your schools to improve education? Y/N/Don’t know 4. Has PRIORITAS improved the quality of teaching and learning in District schools? Y/N/ Don’t know 5. Have you been more involved in school-level activities as a result of PRIORITAS? Y/N 6. Has your department made or up-dated the One-year or Four-year strategic plans to include better teacher deployment in the district? Y/N/Don’t know 7. Has the deployment of teachers in the district improved as a result? Y/N/Don’t know 8. If not, what has stopped you from using PRIORITAS information to deploy teachers better in your district? Tick all that apply Don’t understand PRIORITAS PPG guidelines Don’t have good data from schools No time to be involved Limited coordination among other relevant planners 71 Political objections to moving teachers Financial constraints Personal objections from teachers to move them Re-distribution not necessary Don’t know 9. Has your department made or up-dated the One-year or Four-year strategic plans to include in-service training of district teachers? Y/N/Don’t know (if ‘yes,’ skip to #11) 10. If not, what holds up planning to include in-service training for teachers in your district? Tick all that apply. Don’t understand PRIORITAS guidance Don’t have good data from schools Teachers are busy in classrooms Limited coordination among other relevant planners No budget to do so KKG/MGMPs are of poor quality School principals are not knowledgeable School principals do not have time. Good facilitators/trainers not available No assistance from TTIs No assistance from LPMP Don’t know Other 11. Has your professional knowledge and expertise to plan and budget for teacher training/professional development improved as a result of PRIORITAS? Y/N 12. Has your ability to use data for decision-making about teacher training needs and teacher distribution in your district improved? Y/N 13. Which PRIORITAS activities have had the most impact to improve education quality in district schools? Tick all that apply Literacy teaching and learning (includes instruction, curriculum dev., materials, assessment) Science teaching and learning (same as above) Math teaching and learning (same as above) Dissemination and sharing good practices In-service teacher training Interaction within schools Budget Management Interaction with TTIs Teacher distribution Interaction with LPMPs School principal training Interaction with Provincial Offices School supervisor training Inclusive education and gender training KKG or MGMP training Other 14. Have your schools received assistance from any Teacher Training Institute (TTI)? Y/N/Don’t know (if ‘no’ ‘don’t know,’ skip to # 17 15. Has assistance from TTIs been beneficial to your school? Y/N/Don’t know 16. What inputs from TTIs do your schools receive? Tick all that apply. 72 In-service teacher training Teacher practicum facilitation Good practice school selection and support Student-centered and classroom learning activities Curriculum development Literacy and reading support Mentoring/ refresher training Science support School planning Math support Other? Inclusive education and gender support 17. Has involvement of the Provincial Office of Education increased to support teacher training and school quality improvement? Y/N/Don’t know 18. Are PRIORITAS district facilitators useful to improve quality of education in your district? Y/N/Don’t know 19. Should the position of district facilitator be continued after PRIORITAS is finished? Y/N/Don’t know (if ‘no’ ‘don’t know’ skip to #21) 20. What support could you offer from your office to continue district facilitators after PRIORITAS is finished? Recruitment Transportation Oversight Coordination Management Performance Evaluation Salary Training 21. Have you supported dissemination of good practices to other schools and districts from PRIORITAS activities? Y/N/Don’t know 22. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 23. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know Curriculum Division (KASI) USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What curriculum development activities do you participate in with PRIORITAS? Tick all that apply Math In-service training Science Teacher training Literacy/reading Monitoring School-based management Module development 2. Has PRIORITAS been helpful to improve school quality through curriculum development? Y/N/Don’t know 3. Has PRIORITAS helped you to improve your own curriculum development skills? Y/N 4. Which curricula do you feel are most needed by teachers and principals in schools in your district? Tick all that apply Math In-service training 73 Science Teacher training Literacy/reading Monitoring School-based management Module development Other? 5. Have TTI lecturers been helpful to improve curricula for teachers? Y/N/Don’t 6. Have P4TKs been helpful to improve curricula for teachers? Y/N/Don’t know 7. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 8. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know MORA Representative USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What PRIORITAS activities are or have you been involved in? tick all that apply Teacher Mapping Teacher Deployment Analysis Teacher Cost Analysis Monitoring and Evaluation Teacher Training (Literacy, math, science) School Quality Improvement School based management Showcase workshops Data collection and use National/provincial/district policy workshops Other 2. Has PRIORITAS increased your own knowledge of quality improvement needs in education? Y/N/Don’t know 3. Has PRIORITAS improved the quality of teaching and learning in Madrasah schools? Y/N/ Don’t know 4. What PRIOIRITAS activities have been most useful in Madrasah to improve education? tick all that apply. Teacher Mapping Teacher Deployment Planning Teacher training (PAKEM) Monitoring and Evaluation Teacher Training (Literacy, math, science) School Quality Improvement School based management Showcase workshops Data collection and use Dissemination of good practices Curriculum modules (literacy, math, science) Participation of Teacher Training Institutes 5. Are PRIORITAS district facilitators useful to improve education in your Madrasah? Y/N/Don’t know 6. Have you supported dissemination of good practices to other Madrasah from PRIORITAS activities? Y/N/Don’t know 74 7. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 8. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know Coordinator of Supervisors USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What PRIORITAS activities are or have you been involved in? tick all that apply Teacher Mapping Teacher Deployment Analysis Teacher Cost Analysis Budget Allocations Teacher Training (PAKEM) School Quality Improvement Teacher Training (literacy, math, science) Monitoring and Evaluation Data collection and use Showcase workshops Supervisor training Good practices training Strategy Planning Other 2. Has PRIORITAS improved your ability to coordinate district supervisors? Y/N/Don’t know 3. In what areas has your ability improved to coordinate district supervisors? Tick all that apply More understanding of improved quality education More understanding of literacy, math, and science More understanding of teacher needs to improve instruction More understanding of classroom needs to help teachers More understanding of budgeting for teacher training More coordination with other education planners Other Don’t know 4. Has PRIORITAS improved the ability of school supervisors to support teaching and learning in your schools? Y/N/ Don’t know 5. Do school supervisors generally offer good support to district schools? Y/N/Don’t know 6. In what areas have supervisors improved their ability to support the teachers? Tick all that apply Teacher training Subject: science Teacher Mentoring Monitoring Instructional leadership Learning assessment Subject: literacy TTI partnerships Subject: math School management Community Participation Instructional technology Other Good practice dissemination 7. What limits school supervisors from offering good support to schools in the district? Tick all that apply 75 Limited funds for travel between schools Poor understanding of new education methods (e.g. PAKEM, technology) Limited understanding of literacy, math, and science Poor understanding of quality education Not enough training Limited motivation Limited understanding of role and responsibility Other 8. Have you been more involved in school-level activities as a result of PRIORITAS? Y/N 9. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 10. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know Coordinator of Facilitators USAID has requested a mid-term evaluation of the PRIORITAS education project. This survey is part of the evaluation and will collect information from all Cohort I and 2 PRIORITAS District Offices. The objective of the survey is to obtain information on the perspectives and opinions of multiple government officials in district offices on a) the performance of PRIORITAS to date, b) the effectiveness of project support to increase your institution’s capacity to improve education quality, and c) efforts of PRIORITAS to improve coordination vertically and horizontally among education actors to strengthen teaching and learning process. 1. What PRIORITAS activities are or have you been involved in? tick all that apply Teacher Mapping Teacher Deployment Analysis Teacher Cost Analysis Budget Allocations Teacher Training (PAKEM) School Quality Improvement Teacher Training (literacy, math, science) Monitoring and Evaluation Data collection and use Showcase workshops School based Management Good practices training Dissemination/replication activities Module development Other Provincial/district policy workshops 2. Are PRIORITAS district facilitators useful to improve quality of education in your district? Y/N/Don’t know 3. In what role are facilitators most valuable? Tick all that apply As trainers As mentors For dissemination of good practices For support to schools in quality improvement (PAKEM, technology, etc.) For content knowledge (literacy, math, science) Training module development To connect District Offices to schools Other 76 Don’t know 4. Should the position of district facilitator be continued after PRIORITAS is finished? Y/N/Don’t know 5. Who do you think could replace the position of facilitator when PRIORITAS is finished? Tick all that apply School principals School supervisors TTI lecturers LPMP representatives Head teachers KKG/MGMP No need to replace Don’t know 6. Have you been more involved in school-level activities as a result of PRIORITAS? Y/N 7. Do PRIORITAS inputs, supports, and training follow national policy and government goals of education? Y/N/Don’t know 8. Are PRIORITAS inputs, supports, and training to your office sufficient and given in a timely manner? Y/N/Don’t know 77 PRIORITAS Household Survey We are conducting a mid-term evaluation of the PRIORITAS project. For this evaluation, we are collecting information from a small number of households with children who attend PRIORITAS schools. We would like to know your opinion of your children’s school and changes in your children’s attitude about school since PRIOIRTAs support. Also, we would like to ask you about your participation with your children’s school now since PRIORITAs. PRIORITAS District: ____________________________________________________________ Name of School: ________________________________________________________________ Head of Household or Spouse Name: ________________________________________________ Single Parent: Yes / No Education Level of Respondent: Primary / Junior Secondary / Upper Secondary / Higher Education / None Employed: Yes / No Monthly Income: Below IDR 1,000,000 per month IDR 1,000,000 – IDR 5,000,000 per month IDR 6,000,000-IDR 10,000,000 per month Above IDR 10,000,000 per month. Which school do your children attend? ______________________________________________ Number of Children in School 1/2/3/4/5/more What grades are your children in? __ 1 __2 __3 __4 __5 __ 6 __ 7 __ 8 __ 9 The sex of the children in school: boys: girls: Do you have any special needs/disabled children? Y/N Do the special needs children attend the local school? Y/N 1. Do you know about PRIORITAS? Y/N 2. Do you feel PRIORITAS has improved the school your children attend? Y/N 3. How? Tick all that apply Teachers are present in classrooms all or most of the time. Teachers are active in the classroom. Teachers instruct more reading, science, and math. Teachers are friendly and welcome parents to the school. Principal is at the school all or most of the time. Principal is active at the school. Principal is friendly and welcomes parents to the school. Principal invites parents to visit the school. School looks attractive. School has more books and materials. Classrooms look nice and cheerful. School management committee helps plan school activities. I am asked my opinion of what activities to do at the school. I know more about school plans and budget. My children are learning more at school. Other? 78 4. Have you seen changes in your children’s attitude toward school since PRIORITAS? Y/N 5. What changes? Tick all that apply. Is happy to go to school. Is absent from school less often. Talks positively about school. Shares with parents about activities at school. Does homework regularly. Does better on tests. Reads more. Participates in school activities more. Writes more. Spends more time at school (after-school activities, ‘hangs out’ at school, etc.). Passive about school. Doesn’t want to go to school. Doesn’t talk about school. Other? Don’t know. 5. What activities do you participate in more often with the school since PRIORITAS? Tick all that apply. Visit the school more often. Attend school committee meetings. Attend training on reading (or other subjects) from PRIOIRTAS. Meet with my children’s teachers more often to discuss about school. Meet with the school principal. Volunteer my help at the school. Give money to the school. Involve with reading activities with my children more often. Discuss with my children more often about what they learn at school. Attend planned school activities (showcases, workshops, etc.) Give suggestions to planning and organizing school activities Involve in school activities No change in participation at school. Other? 6. How satisfied are you with your children’s school since PRIORITAS support? Scale 1-5 79 ANNEX V: Classroom Observation Form, Measurement Criteria, and Rating Scale A. Classroom Observation Form Sub-district/District/Province: _______________________________________ School Name: ______________________________________________ Level/Class Grade: ___________________ Subject Matter: _______________ Teacher Name: __________________________ Total Number of Students: ________ Start Time: ________ Completion Time: ______ Male: _____ Female: _____ 1. Classroom Appearance a. Size: _____________________________________________________________________ b. Lighting: __________________________________________________________________ c. Furniture: _________________________________________________________________ d. Neatness: _________________________________________________________________ 2. Classroom Management a. Seating arrangement: ________________________________________________________ b. Teacher position: ___________________________________________________________ c. Between-lesson organization: _________________________________________________ d. Attendance procedure: _______________________________________________________ 3. Lesson Presentation a. Clear objectives: ___________________________________________________________ b. Lesson plan: ______________________________________________________________ c. Methodology: _____________________________________________________________ d. Activities: ________________________________________________________________ 4. Learning Feedback a. Questions & Answers: _____________________________________________________ b. Written Response: _________________________________________________________ c. Oral Response/Discussion: __________________________________________________ 5. Lesson Closure a. Recapitulation: ___________________________________________________________ b. Student Response: _________________________________________________________ c. Learning Outcome Review __________________________________________________ 6. Evidence of Student Involvement or Initiative: ___________________________________ 7. General observations: ________________________________________________________ 8. A description of lesson activities and the time dedicated to each listed on the back of the form. 80 B. Classroom Observation Measurement Criteria and Rating Scale Classroom observations took place by experienced classroom teacher/observers at each school visited. Observation visits were unannounced so that teachers could not prepare in advance specific lessons to be observed. Observers attended the lesson in its entirety, ensuring that they were able to assess teachers’ presentation of a well-organized and instructive lesson as they had been trained to do in PRIORITAS active learning. Observers completed the Classroom Observation Form (above) based on what was seen in the lesson. Based on the observations notes, observers rated classroom instruction practices against 14 indicators of observable teacher and student behaviors listed here. 1. Appropriate Q&A use. 2. Students working together. 3. Student /teacher discussions. 4. Relevant student/teacher black board use. 5. Students reading aloud. 6. Written work either individually or in groups. 7. A well-prepared lesson plan obviously seen. 8. Teacher guidance/ mentoring of groups and individuals. 9. Desks arranged in clusters to enhance group work. 10. Presentation of group work efforts during class. 11. Wall diagrams and visual aids available and in use. 12. Clear objectives at the beginning of class. 13. Clear review of what was learned at the end of the lesson. 14. Positive student and teacher interactions during class. One point was given for each indicator seen to take place in the lesson. Points were tallied and the lesson was given a ‘grade’ of A, B, C, or D based on the number of points accumulated. The rating scale is as follows: • A = 12-14 indicators seen in the lesson • B = 9-11 indicators seen in the lesson • C = 5-8 indicators seen in the lesson • D = 1-4 indicators seen in the lesson Teachers were given negative points for the following extreme classroom behaviors: • Teacher not responding to bad student behaviors: for example, physical or verbal abuse between students or students extremely off-task with loud or inappropriate behavior, • Unacceptable teacher behaviors: for example, sitting at the desk the majority of the lesson, using cell phones in class, or ignoring extreme student needs/behaviors. Classroom Observation data are presented in Annex 8. 81 ANNEX VI: List of Interviewees and Contacts NAME POSITION INSTITUTION/Organization 1. Lawrence W. Dolan, Ph.D. 2. Jona Lai 3. Mimy M. Santika Education Officer Workforce Development Officer Education Specialist USAID Indonesia 1. Stuart Weston 2. Lynne Hill 3. Ajar Budi Kuncoro 4. Ujang Sukandi 5. Wiwit Sri Aryati 6. Mark Heyward 7. Feiny Sentosa 8. Ruwiyati Ahmadi 9. Rifki Rosyad 10. Sri Wahyuni 11. Ridwan Ibrahim 12. Agus Marwan 13. Agus Prayitno 14. Ely Djulia 15. Jamaruddin 16. Mohamad Azmi 17. Silvana 18. Nur Kholis 19. Triana Chief of Party Teaching & Learning Advisor University & Stakeholder Specialist Teacher Education Specialist-JSS Gender Specialist Gov’ & Manag’t Advisor, DCOP Technical Coordinator with GOI Provincial Coord- Banten DC Aceh Jaya Province Coord -Aceh Provicince Coord – Medan WSD Specialist-Medan TTO Primary - Medan Provicince Coord - Makasar DC Maros, Sulsel Province Coord – Jatim TTI Development Specialist TTI Specialist, East Java DC Blitar PRIORITAS 1. Prof. Dr. Patta Bundu 2. Dian Wahyuni, 3. Ferry Yulmarino 4. Hamid Muhammad 5. Dr. Unifah Rosyidi 6. Suharno Science Module Developer Deputy Director of Teacher Kepala Bidang Penjaminan Mutu Director General of Basic Educaton Deputy Director of s Section Head Teachers Prof Dev MOEC 1. Dr. Femmy Eka Kartika 2. Putri, Assistant to the Deputy for Basic Education, Early Childhood Education and Community Education PMK 1. Drs. Subandi Sardjoko Education Director BAPPENAS 1. Drs. A. Syafi’i 2. Dra. Yeni Sulsirawati 3. Dra. Rini Susilowati 4. Drs. Nurul Islam 5. Mustofa Fahmi Ka.Sub.Dit. PTK Ka.Si. Pendidikan Ka.Si. Tendik Ka.Si. Pengawas Ka.Si. Kepala Madrasah MORA 1. Nick Clinch Operations Manager DFAT 1. Widodo Suhartoyo 2. Nabendra Dahal Education Specialist Chief Education and Adi UNICEF 1. Andy Ragatz 2. Susiana Iskandar 3. Ratna Kesuma Senior Education Specialist Senior Eduaction Specialist Senior Operation Officer World Bank 1. Dr. Naf’an Tarihoran M.Hum 2. Dr. Yudi Juniardi 3. Dr. Hepsi Nindiasari 4. Siti Aisah, M.Hum 5. Maman Fathurahman, 6. Udi Samanhudi Dean of Education Faculty Vice Dean of Education Head of Math Lab English Lecturer Head of Math Education English Lecturer UNTIRTA Banten 82 7. John Pahamzah Head of Language Program 1. Prof. Dr. H. Farid Wajdi Ibrahin Rektor UIN Ar-Raniry Banda Aceh 1. Azhari, Asisten II Gubernur-Aceh Aceh Province 1. Monawati 2. Intan Safiah 3. Cut Khairunisak 4. Fauzy 5. M. Yamin 6. M. Husni 7. Nurhaidah 8. Rosma Elly 9. Asiah MD 10. M. Hasan 11. Alfiati Syafrina Lectures/Facilitators University of Syiah Kuala 1. Nida Kurniati 2. Rita Purnama Sari 3. Sukiarni 4. Ainul Mardyah Usman 5. Rusydi 6. Teuku Husni Instructors LPMP Aceh 1. Abu Bakar Edu. Quality Enhancement Unit Aceh Jaya District 1. Irnayati SPd 2. Isnanidar SPd JSS Supervisor PS supervisor Education Office-Aceh 1. Prof. Dr. Ir. Samsul Rizal, 2. Drs. Sulaiman M.SE Rektor Ka. Prodi University of Syiah Kuala￾Aceh 1. Ahmad Wany 2. Bandi Jr Secondary Supervisor Vice Head of Education MORA ACEH 1. Salhadi K.SE Ka.Sie. Pendidikan Luar Biasa MOEC Bener Meriah 1. Drs. Daud Pakeh Ka. MORA MORA Aceh Jaya 2. Syamsul Ka.Sie Pendidikan Madrasah 1. Baihaqi Zaka SPd Ka. Bid. Dikmen MOEC Aceh Jaya 1. Dr. Mardianto M.Pd Vice Dean I - FITK IAIN Sumut 1. Prof. Dr. Abdul Hamid , K. Dean of Technic Faculty Universitas Negeri Medan 1. H Machyuzar Nasution Director of Al-Azhar Foundation Al-Azhar, Medan 1. Drs. H. Soritva Harahap 2. Dr. Azizah Hanim 3. M.Hum 4. H. Solehuddin, 5. Halimatussa Diyah 6. Dahyar Husein 7. Drs. H. M. Ghozali KaBId Pendidikan Madrasah Kasi Kesiswaan KaBid Kelembagaan Kasi Pendidikan dan tenga Kependidikan Kasi Kurikulum dan Evaluasi Kasi Sarana Prasarana MORA District-Medan 1. Sarimpunan Ritonga 2. Basrin Siregar Head Ka.Sie PLB MOEC Labuhanbatu District￾SUMUT 1. Supri Harahap 2. Hamzah Harahap Ka.Sie Kurikulum SMP Ka.Sie Kurikulum SD MOEC Sumut 1. Naimah SPd Kepsek SDN 024772 Binjai District 83 1. Dr. Bambang Winarji M.Pd 2. Dra. Sriwandayani Harahap M.Pd 3. Drs. M. Yakub M.Pd 4. Drs. Abror M.Pd 5. Drs. Syahdian M.Si 6. Dra. Juliah, M.Pd 7. Ajizah Siregar M.Pd 8. Neni Juli Astuti ST. M.Si 9. Okda KA 10. Jogi Sumarlan 11. M. Yakub M.Pd 12. Reinhard Gultom 13. Mahyun Hadi 14. Suwarni M.Pd 15. Jasawitten Brando Purba 16. M. Faisal Syamir S.Sos 17. Arsenal 18. Syandian 19. Misuryanti 20. Taufikurrahman Ginting 21. Jonnedi Trainers LPMP Sumut 1. Burhan T. 2. Muh. Ruli Gunawan 3. Rahmaniar 4. Nur Aulia Hafid. 5. Muh. Abdul Makki 6. Ainun Farida Trainers LPMP Sulsel 1. Toni Satria 2. Salamun Chief Quality Mapping Head of LPMP LPMP East Java 1. H. Ali Yafid Sag, Mag 2. DR. H. Wahyudin Hakim 3. Hj. Tirtawati Sag, 4. Hj. Ida Nurrahmah 5. H. Masykur 6. H.Faturrahman KaBid Pendidikan Madrasah KaSie Kurukulum dan evaluasi KaSie Kesiswaan KaSie Sarana Prasarana KaSie Kelembagaan dan SIP KaSie Pendidikan MORA Province Makasar 1. Drs. H.A. Salam Soba, 2. Drs. Zain 3. Drs. Rusdi 4. Dr. Husniati Pawelloy Head of Education Office Ka.Sie Inklusif Ka.Sie SLB Chief Social Cultural Coop. MOEC Province Makasar 1. Prof. Dr. H. Arismunandar, 2. Ahmad Syawaluddin Rector Head of PGSD Program UNM Makassar 1. Drs. Ashar Salam Ka.Bid Kurikulum MOEC District - Maros 1. Dr. Ir. H. Syamsu Alam Ka. Dinas Pendidikan MOEC District - Bantaeng 1. Prof.Dr. Warsono, 2. Prof. Dr. Djodjok Supardjo 3. Basri Rektor Partnership and Academic Affair Dosen Biologi UNESA Surabaya 1. Silvia 2. Dr. Erman, 3. Prof. Siti Amin 4. Ganes Gunansyah 5. Harmanto 6. Purwanto 7. Sumarno 8. Ulhaq Zuldi Province Facilitator 84 1. Dodong M 2. Adji Arnowo 3. Ali Afandi 4. Esti Marti Goenawati 5. Dyah Lestariningsih Divison Head Head of Sub Division Head of Monev Staff of Intenational Institution Staff of Monev Division of International Cooperation- Jawa Timur 1. DR. Harun MSI. 2. Drs. Nuryanto Kepala Dinas Pendidikan Ka.Bid.TK/SD/PK MOEC Jawa Timur 1. Fitri Hilmiyati 2. Siti Aisah, 3. Ila Amalia, 4. Uyu Mu'awwanah, 5. Apud, 6. Emilia, 7. Siti Solihah, 8. Dini, 9. Anis, Province Facilitator IAIN Serang 1. Apik 2. Ismail 3. Moh. Sahur SH Ketua Komisi IV Ketua Komisi I Anggota Komis IV DPRD Pamekasan 1. Dr. H. Abdullah A’la 2. Dr. H. Ali Mudlofir Rektor Lecturer IAN Sunan Ampel 1. Mr Willy Augusta 2. Drs. MYusuf Suhartono, 3. Drs. Slamet Goestiantoko. DinasPlanning Deputy Head Head of District Education Office. Secretary Pamekasan District 1. Drs. Totok Subihandono, M.Si. 2. Drs. Muhajirin Head of District Education Office Ka.Bid TK/SD MOEC Blitar 85 1. Insofah, 2. Hasan Basri, 3. Aini 4. Ihsan 5. Kemalawati 6. Bustaman 7. Mizanna 8. Yusnizar 9. Hasmiati 10. Raamah 11. Fitri Alfizah Ama 12. Asbidar 13. Harmaini 14. Rinawati S. 15. Nurmita 16. Marlini 17. Nurmawan 18. Varla Yusnila 19. Edwar 20. M. Siddiq 21. Iin Syahri 22. Rianda Prastia 23. Sukiran 24. Muklis Ismail 25. Layna Yanti 26. Arlina 27. Mariyem 28. Lahri Aswita 29. Masdi 30. Masrura District Facilitator Bener Meriah District; Aceh 1. Lili Kurniati 2. Hardi 3. Salvina 4. Nana Ariani 5. Trisna 6. M. Hardi 7. Lili Kurniati 8. Dra. Salvina 9. Trisno 10. Nana Ariana District Facilitator Labuhan Batu District Sumut 1. Drs. Daitin Tarigan 2. Dr. Rahmad Husein 3. Dra. Meida Nugrahalia 4. Dr. Maurice 5. Lala Jelita Ananda 6. Rika SPd. M.Hum 7. Dra. Inayah Hanum, 8. Aida Fitriani Sitompul S 9. Trisnawati Hutagalung 10. Ahmad Shafwan S. Pulungan 11. Sabani 12. Drs. Wildan Lubis Province Facilitator UNIMED Medan 86 1. Dra. St. Johar Nonci 2. Dr. Andi Makkasau 3. Widya Karmilasari A 4. Nurhaedah Arifin 5. Syamsiah B Province Facilitator UNM Makasar 1. Nurhayati 2. Ika Putriana 3. Dra. Nursinah 4. Nurmi 5. Maimi 6. Shaumiati UIN Alaudin Makasar 1. H. Hamka Hasan 2. Muhammad Dalil 3. Irwan 4. Dra. Hj. Hasnah 5. Muhsin 6. Syarifuddin 7. Aryani R 8. Kasmiatang 9. H. Arsyad 10. Sariman 11. Burhanuddin 12. Nur Ridawati 13. Hamsir 14. M Akis 15. Samsu Alam 16. Maulida 17. Hyrawati 18. Irwan 19. Abdul Azis 20. Salwati 21. Hajrah K 22. Hj. Ummiati S 23. Nurcahya 24. Irlidiya 25. Alimuddin Assegaf District Facilitator Maros District Sumsel 87 1. Takdirmin 2. Nurdayana 3. St. Rohani 4. Baharuddin 5. Pammusu 6. Sitti Sulaeha 7. Suarni 8. Andi Haeriyanti 9. Safruddin 10. St. Syamsurya Yusup 11. Nurhaeni 12. Syamsir 13. Nurjanah 14. Siti Marhama 15. M. Hasbi 16. Habar 17. Kasminah 18. Nur Faidah 19. Rohani Sag 20. Zenab SPdI 21. Pratanita SPd District Facilitator Bantaeng District Sumut 1. Mu'azimah 2. Sumarni SPd 3. Musdalifah 4. Sastri Dahlia 5. Rahmadi 6. Siner Jaya SPd 7. Anis Zohriah District Facilitator Blitar District Jatim 1. Drs. Nasruddin ZZ Principal; ACEH SDN 54 Banda Aceh 2. Hilmiyati, Sag MIN Merduati, Banda Aceh 3. Dra. Sumiati. MIN Suka Damai, Bener Meriah 4. Syakbann Nur Spd MIN Lampahan, Bener Meriah 5. Darusallam,. SDN 1 Lampahan, Bener Meriah 6. Arlina, SDN 2 Lampahan, Bener Meriah 7. Riyandi, MTs Negeri Janarata, B.M. 8. M. Yusuf SDN Hakim Wih Ilang, B.M. 9. Bu Isnaini SDN Pondok Gajah, B.M. 10. H Darwin, SDN Tunyang, Bener Meriah 11. Kasno Sag MTsN Lampahan, Bener Meriah 12. Sabardi SPd SMPN 4 Timang Gajah, B.M. 13. Dra. Nurvawela MIN Kota Makmur, B.M. 14. Isnaini SDN Pondok Gajah, B.M. 15. Syarifah Usmawidah, MIN Teunom, Aceh Jaya 16. Nun Faridah, SDN 2 Teunom, Aceh Jaya 17. Sri Indrayati SMPN 1 Sampoinit, Aceh Jaya 18. Hasri Eddy MTsN Lamno, Aceh Jaya 19. Nuriah, SDN 4 Calang, Aceh Jaya 20. Syahruddin SMPN1 Krueng Sabe, Aceh Jaya 21. Sutaiya MIN Dayah Baroh, Aceh Jaya 88 22. Yusni SPd SDN 3 Calang, Aceh Jaya 1. Ahyar Principal; SUMUT MIN Medan Tembung, Medan 2. Drs. Darwis Nasution SDN 064037, Kota Medan, 3. Juniati Siregar, SMPN 35 Medan Tembung, 4. Dra. Nursalimi Mag MTsN 2 Medan Tembung, 5. Asnawati, SDN 118252, Labuhan Batu 6. Faradiba, SDN 116879, Labuhan Batu 7. Umar Tanjung, SMPN 2, Labuhan Batu 8. Karyadi MTs Al -Ittihad, Labuhan Batu 9. Hj. Ruhaya hasibuan, SDN 112147, Labuhan Batu 10. Hj. Sarifah, MIN Padang Bulan, L.B. 11. Samriati, SDN 112162 Labuhan Batu 12. Faridatul Hikmah SDN 112139 Labuhan Batu, 13. Sugiarto, SMPN 1, Labuhan Batu 14. Kamal Tanjung, MTSN Labuhan Batu 15. Seriati Silaban, SDN 066045 Medan Helvetia, 16. Rahmadi SMPN 20 Medan Marelang, 17. Drs. Agus Tono, SMP Al -Azhar, Kota Medan 1. Dra. Hj. Rosdiana Amir, Principal, SULSEL SMPN 2 MAKASSAR 2. Hj. Andi Nensih, SDN39 KASSI, MAROS 3. Hj. Salasiah, SDN 216 INPRESS, MAROS 4. Nur Ridawati, MIN Maros Baru, Maros 5. Marjan, MTs DDI Alliritengae, Maros 6. Hj. Hasibah, SDN 15, JAWI -JAWI, MAROS 7. Hj. Saniah,. SDN1 Pakalu I, Maros, 8. Drs. Sariman, SMPN 4, MAROS 9. Roslaini, MIS Nahdatul Ulunm Maros 10. Muhammad Idris, SMPN 1, BANTAENG 11. Harlina, SDN11 Sarroanging, Bantaeng 12. Rosbiah, SDN 7 Letta, Bantaeng 13. Hasna Syahadat, SDN INPRES T, BANTAENG 14. Dra. Hj. Aidah Pakkanna MTs Muhammadiyah, Bantaeng 15. Hj. Rosmiati Nengsi, SDN INPRES,Bantaeng 16. Muh. Basri MIS Ma’arif , Bantaeng 1. Sariati, Principal, JATIM SDN KONANG 2, Pamekasan, 2. Puji Santoso, SMPN 7, Pamekasan 3. Edi Purnomo, MIN Konang, Galis, Pamekasan 4. Mustafa, SDN Jalmak I Pamekasan 5. Syamsul Hadi SPMN1 Pademawu , Pamekasan, 6. Mujab, SDN 2 Pademawu T, Pamekasan 7. Suparno, SDN 5 Pademawu T,Pamekasan, 8. Imam Rahadi, SDN Pademawu B2, Pamekasan, 9. Abdul Qadimul Azal, SMPN 3 Pademawu, Pamekasan, 10. Sholeh Suadi, MTsN Pademawu, Pamekasan 11. Masrifah, Principal , JATIM SDN 1 Kebonduren, BLITAR 89 12. H. Asnal Khurori, MI Plus M’Bacem, BLITAR 13. Drs. Purwanto SMPN 3, Nglegok, Blitar 14. Gatot Sutrisno, SMPN 3, Srengat, BLITAR 15. Sugianto SMPN 1 Kecamatan, Blitar 16. Drs. Muawinul Huda, MTsN Jambewangi, Blitar 17. Witarti Prasiwi SDN 1 Kalipang, BLITAR 18. Sutiani SDN 1 K.Bunder, BLITAR 19. Retno Pangastuty, SDN Kalipang3, Blitar 20. Moh. Ihsanudin, MI Miftahul Huda, Blitar 1. Eni 2. Waecih 3. Ratu 4. Reni 5. Dedi Teachers, Banten MTs Ciruas - Banten 1. Husna 2. Mukhlis Teachers, Aceh SDN 54 Banda Aceh 1. Sudarni 2. Yuliani 3. Sugito 4. Maranaek MIN Suka Damai – Aceh 1. Hadidjah, 2. Nurhaini, 3. Fitraini 4. Jamhuriah, 5. Hermawati, 6. Mariana, 7. Wirasmi MIN Lampahan, Bener Meriah, Aceh 1. Ika 2. Insofah 3. Erna SDN 2 Lampahan, Bener Meriah, Aceh 1. Asbidar 2. Munawarah 3. Masnidar 4. Suharni 5. Sulastri 6. Fatimah M 7. Siti Aisyah 8. Idah 9. Mariani SDN 1 Lampahan, Bener Meriah, Aceh 1. Fitri Hartanti 2. Esi Zenilis 3. Winsyiah 4. Seidi M Nurdin 5. Suryanum 6. M. Syahril 7. Endang Teachers, Aceh MTs Negeri Janarata, Bener Meriah 8. Usmar SDN H. W. Ilang, Bener Meriah 9. Nun 10. Suamah 11. Karmawati SDN Tunyang, Bener Meriah 90 12. Ismiyati 13. Nurmizayah SPd 14. Bahniar SPd 15. Nurhayati SDN Tunyang, Bener Meriah 16. Mulyadi 17. Nailus Saadah 18. Jufri 19. Salwati 20. Aisah MIN Teunom, Aceh Jaya 21. Yusman AMd 22. Lia Susanti SPd SMPN Sampoinit, Aceh Jaya 23. Asnida SPd 24. Marlina SPd 25. Umikalsum MTs. Lamno, Aceh Jaya 26. Saudah A 27. Erlinawati 28. M. Nasir 29. Hayatun Uswati 30. Putri SDN 2 Teunom, Aceh Jaya 31. Nur Maulina Spd 32. Roza Ovita Spd 33. Rosna Amd 34. Ainul Mardiah Spd 35. Iriani Spd SMPN1 Krueng Sabe, Aceh Jaya 36. Misdarwati 37. Fauzi Ansari 38. Eviyanti SDN 4 Calang, Aceh Jaya 39. Novi Andriani 40. Asmini 41. Darmati 42. Raudah Fahriani 43. Raudah Fahriani SDN Calang, Aceh Jaya 1. Sri Rejeki 2. Erlina Purba Teachers, Sumut SDN 066045 Medan Helvetia 3. Khalidah M.Hum 4. Anda 5. Nurjani MTsN 2 Medan, Medan 6. Sahnan 7. Fakhrrurozi SMP Al-Azhar Kota Medan 8. Nur Hamidah 9. Irma Surayani 10. Nuraida 11. May Darlis MIN Medang Tembung 12. Anna Farida 13. Raisah Putri Sitanggang 14. Sari Kumala Dewi SDN 064037 Medan Tembung 15. Ida Rohani Nasution 16. Cahaya Murni Sari SDN 064037 Medan Tembung Kota Medan 17. Destri Lubis 18. Resdi Ernawati Simaibang 19. Ahmad Dahlan Siregar SMPN 35 Medan Tembung, Kota Medan 91 20. Raudah 21. Syarifah HR 22. Rustiana Harahap 23. Abdul Rahman H 24. Syarifah Mutiara SMPN 20 Medan Marelang, Kota Medan 25. Gelora Ginting SPd 26. Amnizar, SPd 27. Holila, SPd 28. Samidah SPd 29. Cici Suwarsih 30. Eki Andriani Primary Lab School, Kota Medan 31. Putri 32. Nurjanah 33. Rosnaria 34. Dahniar 35. Gustriani Dewi 36. Magdalena Simbolon SDN 118252 Bilah Hulu, Labuhan Batu, Sumut 37. Paidana Rambe SPd SDN 116879 Labuhan Batu, 38. M Yusuf ST 39. Hamiah Hafsah, MTsN AlIttihad, Labuhan Batu, 40. Ana Siregar 41. Asroini SDN 112147, Labuhan Batu, 42. Fidilah Bahar 43. Nirmala 44. Masliana 45. Darmawati 46. Sisni Ernawati 47. Zaitun Nizar SMPN 1, Rantau Selatan, Labuhan Batu, Sumut 48. Mayanti Siagian 49. Aisyah 50. Sumarti 51. Arpiah 52. Seremina Batubara 53. Hj. Yusnalin 54. Jamilah SDN 112162 Rantau Prapat, Labuhan Batu, Sumut 55. Leni Maria MTsN R Prapat, Labuhan Batu, 56. Raihana 57. Rahayuningsih 58. Meri Syahfitri MIN Padang Bulan, Labuhan Batu, Sumut 1. Harmidah 2. Rosmawati 3. M. Nawir 4. Enni 5. Hamriah 6. Rahmini 7. Mira 8. Kurniawati 9. Rahmini 10. Mira 11. Kurniawati Teacher,Sumsel 92 12.Rahmah 13.Mulyana 14.Desi 15.Siti Ramlah 16.Baiduri SDN 216 Inpres Kassi - Maros 17.Hj. Asnawati 18.Hj. Aminah 19.Hj. Musdalifah 20.Nur Cahaya 21.Pitrawati SDN 15 Jawi Jawi - Bantimurung - Makasar 22. Harding 23. Ausag Husah 24. Hamdana 25. Hamsinah 26. Habibah 27. Rahmawati 28. Mulyanti SMPN 4 Bantimurung - Makasar 1. Khalikul Bari 2. Siti Maimona 3. Suranti Teacher, Jatim MIN Konang, Pamekasan 4. Haerus Saleh SPd SDN Konang 2, Pamekasan 5. 1. Usamatul Azizah 6. Samsul Arifin 7. Widodo Prayitno 8. Sutarji SMPN 7 Pamekasan 9. Kustinah 10. Hj. Rukmiati Ningsih 11. Sumaryani SPd SDN Pademawu 2, Pamekasan 12. Jufriadi SPd SDN Pademawu 5, Pamekasan 13. Titik Rismiwati SPd 14. Evan Yuliana SPd 15. Norohana Ekawati SPd SMPN 3 Pademawu, Pamekasan 16. Tiwik Suliyani 17. Sri Lumintu SDN Kalipang 3, Blithar 18. Umi Ngabibah 19. Ketut Ganggiwati SDN Kebon Duren 1, Blitar 20. Asro'in 21. Niswatun Hunsa 22. Siti Nur Hidayah 23. Dendys Darmawan MI Bacem, Blitar 24. Dra. Lina 25. Aminah SPd SMPN 3 Nglegok, Blitar 26. D. Puguh Santosa 27. Sukema 28. Astuti 29. Puguh Santosa 30. Dewi Nasakih 31. Sriyani SMPN 3 Srengat 32. Suharyanto SPd 33. Nur Alfi Yuliati Spsi SDN Kalipang 01, Blitar 93 Total number of interviewee by stakeholder group 1. Government officials: central 14 2. Government officials: provincial and district 48 3. Government officials total 61 4. Teachers 229 5. Principals 75 6. University Staffs 70 7. Provincial Facilitators 23 8. District Facilitators 93 9. Facilitators total 116 10. Donors 6 11. LPMPs 37 12. School Committees 19 13. PRIORITAS 19 14. USAID 3 15. TOTAL 636 1. Muhammad Arysad 2. Azahari Muslim 3. Haji Tiro 4. Drs. H. Abd. Gani 5. Sulaiman 6. Abdul Rosyid 7. Haji Mansur 8. Hj. Baso Parampang 9. Hj. Ali Muddin 10.Drs. Saliman, 11.Hj. Misnadi 12. Haji Mohammed Qosim 13.Sukardi 14.Mohadi 15.Nono Hasihono, 16. Marjani Adiprajitno 17. Mulyana 18. Hadi Sunarto 19. Fathuroman School Committee Members MIN Lampahan 2, B.M. MIN Lampahan, B.M. MIN Maros Baru, Maros, MTs Alliritengae, Maros MIS Nahdatul Ulung, Maros, SDN 39, Maros SDN11 Sarroanging, Bantaeng SDN 7 Letta, Bantaeng MIS Ma’arif Cedo, Bantaeng SDN Jalmak I, Pamekasan SPMN 1 Pademawu, Pamekasan, MTsN Pademawu, Pamekasan SDN Pademawu B02, Pamekasan SDN Kebonduren 3, Blitar MTsN Jambewangi, Blitar SMPN 1, Blitar SDN Kalipang 03, Blitar MI Miftahul Huda Kedungbunder, Blitar SDN 1 Kedung Bunder, Blitar 94 ANNEX VII: Field Travel Schedule Date Institutions/Stakeholders District/Province 15 Jan 2015  USAID PRIORITAS Jakarta, DKI Jakarta 19 Jan 2015  Coordinating Ministry of People Development and Culture (Kemenko PMK)  The Agency of Human Resources Development on Education and Culture and Education Quality Assurance, MOEC.  Center for Improvement of Teachers and Education Profession, MOEC Jakarta, DKI Jakarta 20 Jan 2015  Ministry of National Planning and Development (BAPPENAS)  Directorate of Madrasah Education, MORA  Center of Data Processing (DAPODIK), MOEC Jakarta, DKI Jakarta 22 Jan 2015  Directorate General of Basic Education, MOEC Jakarta, DKI Jakarta 26 Jan 2015  Center for Teachers Profesion Development, MOEC Jakarta, DKI Jakarta 28 Jan 2015  SD Negeri Ciruas 4  MTsN Ciruas (Madrasah)  Universitas Tirtayasa (UNTIRTA)  IAIN Sultan Maulana  Province Education office, MOEC  Serang District Education Office, MOEC  Province MORA Office  Serang District MORA Office  LPMP Banten Office Serang, Banten 1 Feb 2015  USAID PRIORITAS Aceh Province Team Banda Aceh, Aceh 2 Feb 2015  Universitas Syiah Kuala (UNSYIAH)  IAIN Ar-Raniry  Aceh Education Development Board (TKPPA) / MPD  LPMP Aceh Office  MORA Province Office  PGSD Campus, UNSYIAH Banda Aceh, Aceh 3 Feb 2015  MIN Merduati  SDN 54 Banda Aceh, Aceh 4 Feb 2015  MIN Suka Damai  MIN Lampahan  SDN Tunyang  SMPN 2 Timang Gajah  MORA District Office  School Supervisors Bener Meriah, Aceh  SDN 1 Lampahan  SDN 2 Lampahan 95 5 Feb 2015  MTSN Lampahan  SMPN 4 Timang Gajah  District Education office  School Supervisors Bener Meriah, Aceh 6 Feb 2015  MTSN Janarata  SDN Hakim Wih Ilang  MIN Kota Makmur  SDN Pondok Gajah  District Facilitators (FGD) Bener Meriah, Aceh 9 Feb 2015  MIN Teunom  SDN 2 Teunom  SMPN 1 Sampoinit  MTsN Lamno  Aceh Jaya District MORA Office Aceh Jaya, Aceh 10 Feb 2015  MIN Dayah Baroh  SDN 3 Calang  SDN 4 Calang  SMPN 1 Krueng Sabe.  Aceh Jaya District Education Office Aceh Jaya, Aceh 11 Feb 2015  USAID PRIORITAS North Sumatera Team Medan, North Sumatera 12 Feb 2015  Universitas Negeri Medan (UNIMED)  UIN Sumatera Utara  Province Education Office  MORA Province Office  Governor Office  LPMP Medan, North Sumatera 13 Feb 2015  MIN Medan Tembung  SDN 064037 Medan Tembung  SMPN 35 Medan Tembung  MTsN 2 Medan Tembung  Medan District Education office  Medan District MORA Office Medan, North Sumatera 14 Feb 2015  Meeting with USAID PRIORITAS Team (PRIORITAS Office). Medan, North Sumatera 16 Feb 2015  SDN 118252 Bilah Hulu  SDN 116879 Bilah Hulu  SMPN 2 Bilah Hulu  Mts Al-Ittihad  District Facilitators Labuahan Batu (FGD) Labuhan Batu, North Sumatera. 17 Feb 2015  SMP1 Rantau Selatan  MIN Padang Bulan Rantau  SDN 112147 Rantau  MIN Urung Kompas Rantau Selatan  Labuhan Batu District Education Office  Labuhan Batu District MORA Office Labuhan Batu, North Sumatera 18 Feb 2015  SDN 112162 Rantau  SDN 112139 Rantau  SMPN 1 Rantau Utara  MTsN Rantau Prapat  Head of District Labuhan Batu Office Labuhan Batu, North Sumatera 96 20 Feb 2015  SDN 066045 Medan Helvetia  SMP Al-Azhar Medan  SMP 20 Medan Marelang  SDN 101774 Medan Marelang Medan, North Sumatera 23 Feb 2015  SMPN 2, Makassar  MIS Al Abrar  Universitas Negeri Makassar (UNM)  UIN Alaudin Makassar  Province Education Office  Province MORA Office  LPMP South Sulawesi Office Makassar, South Sulawesi 24 Feb 2015  SDN 39 Kassi  MIN Maros Baru  SDN 216 Inpres Kassi  MTs DDI  District Maros Education Office  District Maros MORA Office Maros, South Sulawesi 25 Feb 2015  SDN 15 Jawi-Jawi  SDN 1 Pakalu 1  SMPN 4 Bantimurung  MIS Nahdatul Ulum  District Facilitators Maros (FGD) Maros, South Sulawesi 26 Feb 2015  SMPN 1 Tompobulu  SDN 11 Sarroangin  MIS Borong Kapala  SDN 7 Letta  Bantaeng District Education office  Bantaeng District MORA Office Bantaeng, South Sulawesi 27 Feb 2015  SDN InpresTappanjeng  MTs Muhammadiyah  SD Inpres Mattoanging  MIS Cedo  District Facilitators Bantaeng (FGD) Bantaeng, South Sulawesi 2 Mar 2015  Unversitas Negeri Surabaya (UNESA)  UIN Surabaya (UINSA)  Province Education Office  Province MORA Office Surabaya, East Java 3 Mar 2015  Pamekasan Head of District Office Pamekasan, East Java 4 Mar 2015  SDN Konang 2 Galis  SMPN 7 Pamekasan  SDN Jalmak I  SPMN 1 Pademawu  Pamekasan District Education Office  Pamekasan District MORA Office  Pamekasan Parlement (DPRD) Office Pamekasan, East Java 5 Mar 2015  SDN Pademawu Barat 2  MIN Konang  SDN Pademawu Timur 2  SDN Pademawu Timur 5  Pamekasan District Facilitators (FGD) Pamekasan, East Java 97 6 Mar 2015  SMPN Pademawu 3  MTsN Pademawu Pamekasan, East java 9 Mar 2015  SDN Kebonduren 1  SDN Kebonduren 3  MIs Bacem  SDN Bagelenan 2  Blitar District Education Office  Blitar District MORA Office Blitar, East Java 10 Mar 2015  SMPN 1 Sanankolun  SMPN 3 Nglegok  MtsNJambewangi  SMPN 3 Srengat  Blitar District Facilitators (FGD) Blitar, East Java 11 Mar 2015  SDN Kalipang 1  SDN Kalipang 3  SDN Kedungbunder 1  MI Kedungbunder Blitar, East Java 18 Mar 2015  World Bank Indonesia Office Jakarta, DKI Jakarta 19 Mar 2015  DFAT (Australian Aid) Indonesia Office  UNICEF Indonesia Office Jakarta, DKI Jakarta 98 ANNEX VIII: Classroom Observation Data I. Number of Classes Observed by Subject, Education Level, and Province II. Number and Percentage of Lessons Observed by Subject. SUB. ACEH N SUM S SULA E JAVA TOT SUBJ. ACEH N SUM S SULA E JAVA BANT TOT Grand Totals Math 1 4 5 Math 2 1 2 5 10 Science 1 2 3 2 8 Science 1 1 2 10 Lang 2 1 3 Lang 5 1 1 7 10 Themati 1 3 4 Themati 2 2 3 7 11 Civics 0 Civics 1 2 1 4 4 English 1 1 English 0 1 SUB ACEH N SUM S SULA E JAVA TOT SUB ACEH N SUM S SULA E JAVA BANT TOT Grand Totals Math 1 1 1 3 Math 1 2 2 5 8 Science 1 1 2 Science 1 2 2 5 7 Lang 2 2 Lang 0 2 Civics 0 Civics 1 1 1 English 3 2 2 2 9 English 2 1 3 12 Total 76 Jr Secondary: Set 1 Jr Secondary: Set 2 Primary: Set 1 Primary: Set 2 46 30 Total 4 4 7 6 21 Total 10 4 6 5 0 25 Total 5 5 2 4 16 Total 1 5 3 4 1 14 SUB Aceh North Sumatera South Sulawesi East Java Banten Total % Math 2 2 6 0 0 10 22% Science 1 3 4 2 0 10 22% Bahasa 7 1 1 1 0 10 22% Thematic 3 0 2 6 0 11 24% Civics 1 2 0 1 0 4 9% English 0 0 0 1 0 1 2% Total 14 8 13 11 0 46 100% A. Primary 99 III. Grade Distribution of Classroom Observations by Subject SUB Aceh North Sumatera South Sulawesi East Java Banten Total % Math 1 2 2 3 0 8 27% Science 2 2 0 3 0 7 23% Bahasa 0 2 0 0 0 2 7% Thematic 0 0 0 0 0 0 0% Civics 0 0 1 0 0 1 3% English 3 4 2 2 1 12 40% Total 6 10 5 8 1 30 100% B. Junior Secondary SCORES Math Science Bahasa English Civics Thematic Total % A 3 2 1 2 8 17% B 2 3 3 4 12 26% C 4 3 4 1 5 17 37% D 1 2 3 1 2 9 20% Total 10 10 10 1 4 11 46 100% Grade Distribution of Classroom Observations: Primary SCORES Math Science Bahasa English Civics Thematic Total % A 1 1 3% B 4 3 7 23% C 3 3 1 4 1 12 40% D 1 1 8 10 33% Total 8 7 2 12 1 0 30 100% Grade Distribution of Classroom Observations: Junior Secondary 100 ANNEX IX: Selected Additional Survey Data Household Survey 1. Parents’ Perception of Changes in Their Children’s Behavior and Attitude toward School since PRIORITAS Involvement 2. Parent’s Perception of Changes in their Schools since PRIORITAS Involvement 88 95 86 91 92 84 88 84 73 67 2 0 0 0 22 A. GLAD TO GO TO SCHOOL. B. OFTEN PRESENT AT SCHOOL (RARE TO BE… C. POSITIVE TALKING ABOUT SCHOOL D. SHARE WITH PARENTS ABOUT ACTIVITIES AT… E. FINISH OFF HOMEWORK REGULARLY F. FINISH OFF THE TEST BETTER G. MORE READING H. MORE ACTIVE TO INVOLVE ACTIVITIES AT SCHOOL I. MORE WRITING J. SPENDING MORE TIME AT SCHOOL (AFTER… K. PASIVE ABOUT SCHOOL L. UNWILLING TO GO TO SCHOOL M. NO TALKING ABOUT SCHOOL N. DON'T KNOW O. OTHERS Parents' Perceptions of Changes in Children’s Behavior and Attitude toward School since PRIORITAS (%)? N=129 88 97 88 91 91 94 91 85 97 84 92 87 74 50 87 31 A. TEACHER PRESENCE IN CLASS ALMOST EVERYTIME B. TEACHERS ARE ACTIVE IN CLASS C. TEACHER MORE TEACHED LITERACY, MATH,… D. TEACHERS AND PLEASANT TO WELCOME… E. HEADMASTER PRESENTS AT SCHOOL ALMOST… F. HEADMASTER IS ACTIVE AT SCHOOL G. HEADMASTER IS PLEASANT TO WELCOME… H. HEADMASTER INVITES PARENTS TO COME TO… I. SCHOOL LOOKS ATTRACTIVE. J. SCHOOL POSSES MANY BOOKS AND TEACHING… K. CLASSROOMS LOOKS BEAUTIFUL AND FUN L. SCHOOL COMMUNITY MANAGEMENT ASSIST… M. I WAS ASKED FOR MY OPINION ABOUT WHAT… N. I KNOW MUCH ABOUT PLAN AND BUDGET OF… O. MY CHILDREN KNOW MANY THINGS AT SCHOOL P. OTHERS Parents' Perception of Changes in Schools since PRIORITAS Involvement (%) N=129 101 PRIORITAS Provinces F M Read Culture Tchr Support 1 yr plan Comm Part. Yes No Yes No Yes No Yes No Tchr Disc AL meth Less Plan Class rm mgt Math Sci Teach Aids Impr'd scores Active learn Enjoy ment Crea tive Expr's Opin More readi ng Aceh, N=22 57% 43% 77% 36% 68% 64% 43% 57% 5% 95% 64% 36% 82% 17% 45% 82% 50% 68% 22% 82% 41% 50% 41% 68% 47% 53% 87% 37% 62% 43% 45% 55% 54% 46% 65% 35% 100% 31% 87% 44% 100% 56% 68% 25% 75% 63% 36% 85% 54% 92% 85% 19% 81% 73% 26% 87% 13% 85% 50% 36% 86% 28% 78% 64% 57% 78% 30% 70% 77% 44% 94% 83% 7% 93% 7% 93% 100% 0% 91% 8% 89% 55% 33% 83% 33% 100% 44% 78% 16% 77% Ave: 49% 51% 82% 43% 79% 69% 73% 27% 81% 18% 89% 47% 81% 32% 90% 65% 35% 75% Principals LPMP K13 TTI Trng Sch Comm KKG/MGMP Changed student Behaviors Involvement with: N. Sumatra, N=20, 4 comp schools S. Sulawesi, N= 18, 4 comp East Java, N= 22, 4 comp schools Increased activities since PRIORITAS Most effective PRIORITAS activities ANNEX X: School Interview Data Explanation The matrix presents data on the effectiveness of PRIORITAS interventions based on information collected from interviews with principals, teachers, and school committee members in project schools. The frequency with which specific topics of inquiry were mentioned by respondents was counted and responses were averaged. The Evaluator coded, recorded, and analyzed the information manually. As is the nature of qualitative information, responses were highly subjective as was the Evaluator’s judgment on coding of the information; therefore results presented here should be considered illustrative of tendencies rather than acknowledged as rigorous fact. Sections blue, red, and green (top headings) specifically concerned effects of PRIORITAS; therefore the common denominator used to calculate averages was the numbers of project schools visited minus the number of comparison schools (noted in the left column as ‘comp schools’) as comparison schools were not directly affected by PRIORITAS treatments. Only in the yellow section which assessed all schools’ on-going involvement with any of the listed agencies was the total number of schools visited used as the denominator. Blank spaces in any section indicate a low response rate (too few to represent substantial frequencies) or lack of specificity of respondents’ answers to enable calculation. Legend for Topics of Inquiry • Brown: Principals: Indicates sex of respondents • Blue: Increased activity of principals in specific areas as a result of PRIORITAS MBS training:  Development of a reading culture, 102  Support to teachers through observation, faculty meetings, more materials, advising, etc.,  Transparent development of the one year school plan, and  Engagement with communities. • Yellow: On-going or noticeable involvement of schools with the following groups:  LPMPs for training of K13,  In-service training by Teacher Training Institutes,  Active engagement of School Committees, and  Active functioning of KKGs or MGMPs. • Red: Respondents’ opinion of the most effective PRIORITAS activities or areas where progress had been made at project schools:  Teacher discipline (e.g. daily presence at school and in classroom, on-time school/classroom arrival, timely preparation of lesson plans, timely development of appropriate learning materials, etc.) (This item was mentioned often in Aceh so Evaluators decided to enquire about it individually in the next provinces also but the trend didn’t develop further.),  The use of active learning or being more actively involved in presentation of learning to students (while the next four topics are considered to be part of the ‘active learning’ methodology, each was mentioned often by respondents or specific information on the topic was requested by USAID that Evaluators decided to treat each one separately),  Preparation of lesson plans,  Better classroom management,  More focus on math and science, and  The development and use of teaching-learning materials. • Green: Noticeable changes in student behaviors by teachers and principals:  Improved academic scores (as qualified by educators, improvements for the most part were minor but important enough for them to point out as a promising trend),  Actively involved in learning, working together, exploring for information, using multiple sources, etc.  Enjoying learning and appearing happy to be at school,  Increased creativity in completing assignments, developing materials and classroom projects, etc.  Being confident to express opinions, report out group work, respond in class, etc., and  Reading more, often as a response from principals and teachers to schedule assigned reading times during school hours.