August 2017 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Management Systems International (MSI). Indonesia Prioritizing Reform, Innovation, Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) Project Final Performance Evaluation of USAID/Indonesia Prioritizing Reform, Innovation, Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) Project Final Report Contracted under AID-497-C-16-00006 Prepared for USAID Indonesia Prepared by Management Systems International 200 12th St. South, Suite 1200, Arlington, VA 22202 USA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation i TABLE OF CONTENTS List of Tables and List of Figures ................................................................................................ iii List of Acronyms, Abbreviations, and Terms ............................................................................ iv Executive Summary ...................................................................................................................... 1 Background ..................................................................................................................................................................... 1 Purpose ............................................................................................................................................................................ 1 Methodology and Limitations ..................................................................................................................................... 2 General Findings ............................................................................................................................................................ 2 RINGKASAN EKSEKUTIF........................................................................................................... 9 Latar Belakang ................................................................................................................................................................ 9 Tujuan ............................................................................................................................................................................. 9 Metodologi dan Keterbatasan ..................................................................................................................................10 Temuan Umum............................................................................................................................................................10 1. Project Background ................................................................................................................ 17 1.1 National Education Context .............................................................................................................................. 17 1.2 Project Background.............................................................................................................................................. 17 2. Evaluation Purpose & Questions ........................................................................................... 18 2.1 Evaluation Purpose ............................................................................................................................................... 18 2.2 Evaluation Questions ........................................................................................................................................... 19 3. Evaluation Methods &Limitations ......................................................................................... 19 3.1 Evaluation Process, Methodology & Data Resources .................................................................................. 19 3.2 Data Analysis ......................................................................................................................................................... 24 3.3 Limitations of the Evaluation ............................................................................................................................. 25 4. Findings, Conclusions &Recommendations .......................................................................... 26 4.1 Competency and Skills of Pre-Service &In-Service Teachers and TTIs ................................................... 26 4.2 Capacity & Initiative of Schools, Communities & District Governments for the Management of Schools .................................................................................................................................................................... 43 4.3 Capacity and Initiative of Local Governments for Coordinating Education Resource Management49 4.4 Program Sustainability ......................................................................................................................................... 55 4.5 National-Level Benefits from PRIORITAS ...................................................................................................... 58 Annex I: Final Evaluation of USAID PRIORITAS Statement of Work .................................. 63 1. Introduction ........................................................................................................................................................... 63 2. Activity Description ............................................................................................................................................. 63 3. Background and Results Framework ............................................................................................................... 64 4. Evaluation Purpose, Intended Use, and Audience ........................................................................................ 66 5. Evaluation Questions ........................................................................................................................................... 66 6. Gender Considerations....................................................................................................................................... 67 7. Data Collection and Analysis Methods ........................................................................................................... 67 8. Deliverables ........................................................................................................................................................... 68 9. Reporting and Dissemination ............................................................................................................................ 69 10. Team Composition .............................................................................................................................................. 70 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation ii 11. USAID Participation ............................................................................................................................................. 71 12. Logistics and Scheduling ...................................................................................................................................... 71 14. Budget ..................................................................................................................................................................... 72 Annex II: Map of USAID PRIORITAS Partners ....................................................................... 73 Annex III: Analytical Framework for the PRIORITAS Final Evaluation: Getting to Answers74 Annex IV: PRIORITAS Evaluation: Sampling Design and Detailed Fieldwork Schedule ..... 77 Annex V: Data Collection Instruments ..................................................................................... 81 FGD Guide – District Level Officials ...................................................................................................................... 81 Interview Guide – District Facilitator .................................................................................................................... 83 Interview Guide – TTI Leader.................................................................................................................................. 85 Interview Guide – TTI Facilitator ............................................................................................................................ 87 Interview Guide – Head of UPTD .......................................................................................................................... 89 Interview Guide – School Supervisor (SD/MI and SMP/MTs) .......................................................................... 91 Interview Guide – School Principal ......................................................................................................................... 93 Lesson Observation in Classroom Guide ............................................................................................................. 95 FGD Guide – Teachers at School ........................................................................................................................... 97 FGD Guide – School Committee Members ......................................................................................................... 99 FGD Guide – School Students ............................................................................................................................... 101 Interview Guide – Whole School Development Program Coordinator of PRIORITAS ......................... 103 Interview Guide – Governance and Management Coordinator of PRIORITAS ........................................ 105 Interview Guide – TTI Coordinator of PRIORITAS ......................................................................................... 106 Interview Guide – Training Facilitator/Coordinator of PRIORITAS ............................................................ 107 Interview Guide – GOI Coordinator of PRIORITAS ....................................................................................... 108 Online Survey Instrument – Head of TTI............................................................................................................ 110 Online Survey Instrument – TTI Facilitator ........................................................................................................ 114 Annex VI: Disclosure of any Conflicts of Interest .................................................................. 118 Annex VII: Sources of Information .......................................................................................... 126 List of Document Review ........................................................................................................................................ 126 List of Interview and FGD ....................................................................................................................................... 117 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation iii LIST OF TABLES AND LIST OF FIGURES LIST OF TABLES Table 1: Percentage of Teachers and Students Implementing Active Learning in Classrooms ............... 28 Table 2: Percentage of Teachers Implementing Active Learning Methods .................................................. 29 Table 3: Percentage of Students Demonstrating Positive Learning Behaviors ............................................ 29 Table 4: Summary of PRIORITAS Test Results in Primary Schools (By Type of Schools) ....................... 31 Table 5: Summary of PRIORITAS Test Results in Primary Schools (By Gender) ...................................... 31 Table 6: Summary of PRIORITAS Test Results in Secondary Schools (By Type of Schools) .................. 32 Table 7: Summary of prioritas test Results in Secondary Schools (By Gender) ......................................... 32 Table 8: Project Partners and Beneficiaries ......................................................................................................... 59 LIST OF FIGURES Figure 1: PRIORITAS Results Framework ........................................................................................................... 18 Figure 2: Percentage of Schools Implementing Transparent and Participative School Budget Management................................................................................................................................................ 44 Figure 3: Accumulative Budget Allocation in APBD 2014 for Education Program Activities by several PRIORITAS Partner Districts ................................................................................................................. 50 Figure 4: Percentage of PRIORITAS District Partners having a PKB plan based on Needs Analysis and Cooperation with Capacity-Building Service Providers .......................................................... 51 Figure 5: Estimated Budget and Time Requirements for Local Governments to Train All Teachers ... 52 Figure 6: Allocated Budget for Education Sector as Percentage of Total APBD in Selected PRIORITAS Partner Districts ................................................................................................................. 53 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation iv LIST OF ACRONYMS, ABBREVIATIONS, AND TERMS APBD AnggaranPendapatandanBelanja Daerah (District Budget) AusAID Australian Agency for International Development (now DFAT) BAPPEDA BadanPerencanaan Pembangunan Daerah (Regional Development Planning Board) BAPPENAS BadanPerencanaan Pembangunan Nasional (National Development Planning Board) BERMUTU Better Education through Reform Management & Universal Teacher Upgrading Project BKD BadanKepegawaian Daerah (District Human Resource Board) BOS Bantuan Operasional Sekolah (Government-Allocated School Operational Assistance Fund) Bupati Regent (District Head) CTL Contextual Teaching and Learning DAPODIK Data PokokPendidikan (MOEC’s national web-based Education Management Information System) DBE Decentralized Basic Education Project DC District Coordinator DCOP Deputy Chief of Party Dewan Pendidikan District Education Board DFAT Department of Foreign Affairs and Trade (Australia) Fasda Fasilitator Daerah (District Facilitator) FGD Focus Group Discussion FSU Florida State University GLS Gerakan Literasi Sekolah (School Literacy Movement) GOI Government of Indonesia HDI Human Development Index IAIN Institut Agama Islam Negeri (State Islamic Institute) ICT Information and Communication Technology IDR Indonesian Rupiah (currency) IE Inclusive Education IKIP Institut Keguruan dan Ilmu Pendidikan (Teachers’ Training College) IT Information Technology JSS Junior-Secondary School Kabupaten (Kab.) Local government administrative area; a district or regency KII Key Informant Interview KAK Kerangka Acuan Kerja (Terms of Reference) USAID/Indonesia PRIORITAS Project: Final Performance Evaluation v KKG Kelompok Kerja Guru (Teacher Working Group at regular primary schools) KKM Kelompok Kerja Madrasah (School Principal Working Group for Islamic schools) KKKS Kelompok Kerja Kepala Sekolah (School Principal Working Group) KKPS Kelompok Kerja Pengawas Sekolah (School Supervisor Working Group) Kota Local government administrative area; a city or municipality LAPIS Learning Assistance Program for Islamic Schools LPMP Lembaga Penjaminan Mutu Pendidikan (Education Quality Assurance Institute) LPTK Lembaga Pendidikan Tenaga Kependidikan (also Teacher Training Institute, TTI) M&E Monitoring and Evaluation MESP Monitoring & Evaluation Support Project (USAID/Indonesia) Madrasah Islamic School MBE Managing Basic Education Project MEDP Madrasah Education Development Project MENKO KESRA Coordinating Ministry for People’s Welfare; now MENKO PMK MENKO PMK Coordinating Ministry for Human Development and Culture MGMP Musyawarah Guru Mata Pelajaran (Local Association of Secondary Teachers; grouped by subject matter) MGMPS Musyawarah Guru Mata Pelajaran Sekolah (Local Association of Secondary Teachers; grouped by subject matter, school-based) MI Madrasah Ibtidaiyah (Islamic Primary School) MKKS Musyawarah Kerja Kepala Sekolah (Secondary School Principal Working Group) MOEC Ministry of Education and Culture MOHA Ministry of Home Affairs MORA Ministry of Religious Affairs MORHE Ministry of Research, Technology and Higher Education MOU Memorandum of Understanding MTs Madrasah Tsanawiyah (Islamic Junior-Secondary School) MUSRENBANG Musyawaran Perencanaan Pembangunan (Development Planning Congress) P4TK Pusat Pengembangan dan Pemberdayaan Pendidikdan Tenaga Kependidikan (Teacher and Education Staff Development and Empowerment Center) Paguyuban Parents Association PAKEM Pembelajaran Aktif, Kreatif, Efektifdan Menyenangkan (Active, Effective and Enjoyable Learning) PC Provincial Coordinator PGRI Persatuan Guru Republik Indonesia (National Teacher Union) PKB Pengembangan Keprofesian Berkelanjutan (Continuing Professional Development) PMP Performance Monitoring Plan USAID/Indonesia PRIORITAS Project: Final Performance Evaluation vi PPG Pendidikan Profesi Guru (Teacher Professional Education Program) PPG Penataandan Pemerataan Guru (Teacher Deployment Program) PPL Praktik Pengalaman Lapangan (Teaching Practicum for Teacher Candidate) PRIORITAS Prioritizing Reform, Innovation and Opportunities for Reaching Indonesia’s Teachers, Administrators and Students Project PTK Penelitian Tindakan Kelas (Classroom Action Research) Renstra Rencana Strategis (Five-Year Educational Development Plan) RKAS Rencana Kegiatan Anggaran Sekolah (School Budget Plans) RKS Rencana Kerja Sekolah (School Plans) RPJMN Rencana Pembangunan Jangka Menengah Nasional (National Medium-Term Development Plan) SBM School-Based Management SD Sekolah Dasar (Primary School) Sekolah Rujukan MOEC’s model/reference school. SIMDIKDAS Sistem Informasi Manajemen Pendidikan Dasar (Basic Education Information Management System) SMA Sekolah Menengah Atas (Senior Secondary School) SMP Sekolah Menengah Pertama (Junior Secondary School) SOP Standard Operating Procedure SOW Statement of Work TK Taman Kanak-kanak (Kindergarten) TOT Training of Trainers TPM Team Planning Meeting TTI Teacher Training Institute (also Lembaga Pendidikan Tenaga Kependidikan, LPTK) UIN Universitas Islam Negeri (State Islamic University) UINSA Universitas Islam Negeri Sunan Ampel UINWS Universitas Islam Negeri Walisongo UKG Ujian Kompetensi Guru (Teacher competency test) UM Universitas Negeri Malang (State University of Malang) UN National Student Examinations UNIMED Universitas Negeri Medan (Medan State University) UNNES Universitas Negeri Semarang (State University of Semarang) UPTD Unit Pelayanan Teknis Daerah (Regional Technical Service Unit) USAID United States Agency for International Development WSD Whole-School Development YLAI Yayasan Literasi Anak Indonesia USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 1 EXECUTIVE SUMMARY Background The final performance evaluation of USAID Prioritizing Reforms, Innovations and Opportunities for Reaching Indonesia’s Teachers, Administrators and Students (PRIORITAS) Project took place from February to May 2017.PRIORITAS is the last of three consecutive USAID/Indonesia basic education projects, beginning in 2003, to assist Indonesia to improve the quality of basic education. The USAID Managing Basic Education (MBE) project ran from 2003–2007 and the three simultaneous USAID Decentralized Basic Education projects (DBE1, 2 and 3) operated from 2005–2011. MBE, DBE and PRIORITAS shared similar program characteristics: school community participation; working directly in local systems to build commitment at the provincial and district levels; intensive training with follow-up mentoring; provision of technical assistance rather than funding; and content based on evolving Government of Indonesia (GOI) education policies and innovations. PRIORITAS, operating from 2012–2017, works intensively in 53 districts in nine provinces, including three districts in Papua and West Papua. PRIORITAS aims to support the overall USAID goal of expanded access to quality basic education for children in Indonesia. This project has three components: 1. Strengthened pre- and in-service teacher development programs to increase quality of instruction in classrooms. 2. Improved education management and governance of schools and districts for improved support to classroom delivery of education. 3. Improved coordination of communication, information-based planning and policymaking, feedback and use of financial and human resources at all system levels to improve delivery to the lower levels. Purpose The primary purpose of this final performance evaluation is to inform USAID, GOI counterparts, partners of PRIORITAS and other donors about the results PRIORITAS has achieved at the school, district and provincial levels; the implications of those results on the quality and relevance of basic education in Indonesia; and the practices and lessons from PRIORITAS that relevant stakeholders can sustain to continue increasing access to quality basic education in Indonesia. This evaluation sought to find answers to the following five research questions specified by USAID: 1. How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? How has PRIORITAS strengthened the institutions (Teacher Training Institutes) to support these teachers? 2. How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS improved in developing/supporting accountable and participatory school management? 3. How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management with focus on teacher deployment, teacher continuing professional development, and educational funding? USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 2 4. How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level? 5. How have national government agencies (MOEC, MORA, and MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects? Methodology and Limitations The evaluation team used a mixed-methods approach to the PRIORITAS final evaluation, quantitative and qualitative. Quantitative data were collected through an online survey sent to Teacher Training Institutes (TTIs) and analysis of secondary performance monitoring data from the project. Qualitative data were mostly gathered during key informant interviews (KIIs) and focus group discussions (FGDs) with informants at the national, district and school levels, and at PRIORITAS partner TTIs. The evaluation team also conducted a large number of classroom lesson observations at project partner schools. The evaluation team identified the following limitations to the evaluation SOW and implementation:  Small number of districts and schools sampled, compared to the large number of project locations and partners.  A primarily qualitative approach, which might not enable the evaluation team to calculate the net effects of PRIORITAS impacts (but would provide greater insights into why the project worked or did not work in given situations).  Presence of previous or current similar donor programs in the study area, making it difficult to precisely identify unique impacts of PRIORITAS in some cases.  Presence of evaluators during the administration of classroom observations, introducing bias (Hawthorne effect).  Unavailability of senior decision-makers from MOEC, MORA and MORHE to serve as key informants during the evaluation. General Findings The evaluation sought to find answers to the five research questions specified by USAID. The full evaluation report contains the team’s findings, conclusions and recommendations for each question. A summary of the evaluation results is presented below in this executive summary. Question 1 How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? How has PRIORITAS strengthened the institutions (TTIs) to support these teachers? This section of the evaluation report has two parts. The first addresses the delivery of active learning methodology, including the PRIORITAS reading program; the second covers the PRIORITAS interventions at TTIs. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 3 Active Learning Methodology PRIORITAS developed three initial sets of modules, with a fourth added later, that the project used to train teachers of three core subjects – mathematics, science and Indonesian language – in the Active, Effective and Enjoyable Learning methodology, known by the Indonesian acronym PAKEM. These trainings reached at least six teachers in each primary school and 15 teachers in each junior-secondary school, per project plans. The evaluation shown positive response from schools and district governments in implementing the PAKEM methodology, resulting to teachers using PAKEM in subjects extended from the three core subjects, including English and social sciences. Project activities were disseminated to wider audiences through teacher working groups (KKG/KKM or MGMP). While the project itself disseminated the PRIORITAS modules, the full spread of these methods occurred as the result of enthusiasm and support from school and district leaders. This dissemination was dependent on the availability of financial resources, such as the district budget (known by the Indonesian acronym APBD) and the Government-Allocated School Operational Assistance Fund (BOS). A large majority of all categories of informants interviewed by the evaluation team in PRIORITAS schools reported that positive, observable educational benefits had resulted from teachers adopting active teaching and learning methods followed by mentoring. Classroom observations and FGDs with teachers and students confirmed this finding. But the evaluation team noted that despite PRIORITAS interventions, older teachers and teachers of non-core subjects are rarely using active teaching methods. PRIORITAS monitoring and evaluation (M&E) data demonstrated considerable improvements in student performances on mathematics, science and Indonesian language tests conducted by PRIORITAS in partner schools from 2013–2016. The data also showed that, in general, female students scored higher than male students. Students in FGDs stated that after teachers implemented PAKEM, the students could absorb mathematics and science subjects more easily. In addition, the lesson observations revealed that female and male students received equal opportunities to participate in learning activities. The evaluation team recommended that schools continue mentoring on the PRIORITAS package of trainings in active learning methodologies to ensure that teachers receive the technical support they need to implement this model effectively. It is also important to continue strengthening the teacher working groups (KKG/KKM and MGMP) as a sharing forum to innovate and disseminate the PRIORITAS program, and enhance teachers’ competencies and skills in implementing active learning methodologies. PRIORITAS provided a special focus on reading development. The project’s teacher training modules included segments for early grade reading, developing interest in reading and reading comprehension. Many informants reported that the culture of reading introduced by PRIORITAS was generally useful and popular, and that it worked well in partner schools. PRIORITAS also provided many books, including leveled readers, to schools. From these interventions, project data indicated more than 5 million direct and indirect beneficiaries of the PRIORITAS reading program. Therefore, innovations and school partnerships with the district library, mobile library or other reading programs should be sustained and are strongly recommended for future reading projects. Teacher Training Institutes Regarding TTIs, key informants stated that PRIORITAS’ efficient organization of training processes was widely appreciated, and that all TTIs sampled in this evaluation now use the project-supported content. PRIORITAS was acknowledged to have developed the knowledge and skills of teacher educators and teacher candidates to use student-centered and innovative teacher training methodologies. This result was enabled through agreements between the project and 17 partner TTIs, including both state and Islamic institutions; and by working with 31 other TTIs through consortia in provinces. Technical USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 4 support from PRIORITAS has also enhanced the capacity of TTIs to be service providers. Various memoranda of understanding (MOUs) have been developed between TTIs and provincial or district governments or schools for efforts aimed at improving the quality of teachers. To build TTI capacities and continue the successes of PRIORITAS, it is important for TTIs to have internal affirmative programs to disseminate active learning methods across their departments and to broaden their coverage beyond science, math and Indonesian language to realize greater impacts at the university level. The evaluation team recommends that PRIORITAS partner TTIs continue their partnerships with consortia members, partner schools and lab schools to sustain effective practices after the project ends. Given the benefits of PRIORITAS at the TTI level, GOI should consider adopting the best practices of PRIORITAS training to improve the quality of pre-service training at TTIs – that is, following up training with mentoring and provision of technical assistance. Question 2 How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS improved in developing/supporting accountable and participatory school management? In a large majority of partner schools, PRIORITAS interventions were effective in improving capacity for school management and participatory governance. Interventions were also useful to schools that reported having fully adopted inclusive, school-based management before PRIORITAS began. Moreover, the evaluation team found that project interventions were particularly effective in strengthening community participation in school governance.Informants reported positive changes in the accountable and participatory management of their schools as a result of PRIORITAS interventions.Improvements included: (1) more involvement and better support from paguyuban (parent associations), parents and committees; (2) better use of BOS funds; and (3) communities that were empowered to participate more in their schools as a direct result of committee members having been included in PRIORITAS training events. Teachers and parents interviewed by the evaluation team stated that they communicate more frequently and more easily since PRIORITAS worked in their schools. These findings are consistent with PRIORITAS monitoring data, which show that more schools are now implementing transparent and participative school management. PRIORITAS has built a culture of school community and parents about the importance of planning for quality education. The project has provided a better understanding of the roles of principals, teachers and school committees in school management and governance. Most informants in schools met by the evaluation team claimed that teachers are now involved in the management teams dealing with school development. Schools invited parents to share any updated information related to the school, when schools needed parents’ support or during school budget (RKAS) and school plan (RKS) development. The project also encouraged community participation to support the quality of education. Parents have become more engaged with schools and have supported school with various needs (e.g., money, materials and labor), especially with the existence of paguyuban. As an entity, the paguyuban structure falls under the school committee and they supported school activities in many ways, most significantly active learning and reading programs. They provided teaching materials and aids, as well as furnishings and books for reading corners. Where school committees need support to participate fully in school governance or to take on new responsibilities that are devolved to them, local governments should adopt an inclusive training approach such as the one taken with PRIORITAS, wherein training is provided not only for school principals and staff, but for school committee and community members, alumni, etc. These trainings will help all committee members understand the roles and actions needed to improve the quality of education. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 5 The evaluation team used a broad interpretation of “media” to include PRIORITAS’ communication products and mass media coverage about project activities and methodologies, as observed by informants. PRIORITAS developed significant communication and outreach programs using videos, best practices books, website resources and other materials. Evaluation informants and survey respondents stated that mass media had frequently run stories about PRIORITAS activities; that media stories had raised public awareness about educational issues; and that informants had utilized PRIORITAS media (e.g., website, a newsletter, books) to learn about PRIORITAS best practices. Future donor or GOI education projects are recommended to adopt the PRIORITAS communication approach–capitalizing on online, social, and mass media to disseminate program information and materials to a broad audience of stakeholders. Such interventions should be tied to desired project results and outcomes, with specific impact-related indicators. Question 3 How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management, with a focus on teacher deployment, teacher continuing professional development, and educational funding? This section of the report primarily addresses the PRIORITAS Teacher Deployment Program (PPG) and the related Teacher Continuing Professional Development program (PKB) that each district attempts to fund and manage for its teachers. PRIORITASPPG and PKB were designed to help local governments address the undersupply of civil service teachers and the abundance of temporary teachers, as well as the inappropriate distribution of civil service teachers among urban and rural schools. Evaluation informants commended PRIORITAS for assisting local governments to assess existing teacher deployments, and to make more proportional redistributions using innovative new policies. Additionally, PRIORITAS district partners have successfully implemented PPG and PKB – duties prescribed under Law 14/2005, Government Regulation 74/2008 and the joint regulation of five ministries. Regarding PPG, a total of 40 new policies related to the deployment of teachers were reported to have been adopted by PRIORITAS partner districts in the evaluation sample. These new policies included introducing multi-grade classes, grouping a number of schools into one, rotating teachers from school to school and recruiting new teachers. To improve teacher capacity, several PRIORITAS partner districts allocated more than IDR 25 billion in their 2014 budgets. The allocations of budgetary support for disseminating PRIORITAS practices illustrates an improvement in coordination among district-level authorities. The key authorities – Dinas Pendidikan (Office of District Education), Badan Kepegawaian Daerah (Regional Human Resources Board), Dinas Pendapatan Daerah (Office of District Revenues) and Badan Perencanaan Pembangunan Daerah (Regional Development Planning Board, or BAPPEDA) – now have a better understanding of the strategic issues confronting their education sectors. This understanding has led to strengthened coordination and has led to the education sector receiving more financial support. The evaluation team found that this coordination will likely be sustained, since the relevant budgeting processes take place through a nationally applied and coordinated development planning system, known by the Indonesian acronym MUSRENBANG. The combination of local capacity building and working with TTIs as service providers also strengthened coordination between TTIs and local government as users, and this coordination should eventually enhance the sustainability of PRIORITAS’ efforts. This result stems from the approach by PRIORITAS to develop local human resources to be PPG/PKB trainers. Participants trained by the project included those currently serving as teachers, principals, school supervisors, university lecturers and Education USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 6 Quality Assurance Institute (Indonesian acronym: LPMP) trainers. Local governments will be able to mobilize these resources to assist in sustaining their PPG and PKB. The combination of local governments as beneficiaries and TTIs/LPMPs acting as service providers is a lesson learned: all entities can cooperate in support of quality education. Despite all of those efforts to improve PPG and PKB, a majority of informants told evaluators that they could not confidently predict whether PRIORITAS methods would be utilized indefinitely after support from PRIORITAS ends. They cited a number of obstacles to sustaining these practices, including interference by local politicians in the allocation of future educational funding and ongoing difficulties using computer-based information systems for PPG. These challenges echo the findings in a 2015 study on teacher deployment in Indonesia by USAID. Regarding PKB, informants highlighted the magnitude of local governments’ financial challenges to pay for programs to develop professional skills for all of their teachers. The evaluation team noted, however, the PRIORITAS legacy that each district now has many PRIORITAS-trained teachers, principals qualified to serve as facilitators, school supervisors, university lecturers and LPMP trainers who could be mobilized to assist in sustaining future PKB activities. The evaluation team notes that local governments must regularly update their teacher data and implement PPG and PKB to help ensure the delivery of quality education and maintain quality coordination. The team recommends that local governments continue to seek assistance from TTIs that have the capacity to facilitate PPG and PKB processes. The team also recommends that the central government play its critical roles (as stated in Law No. 23/2014 on Local Government) by providing technical assistance to PRIORITAS partner and non-partner districts, continuing the roles previously filled by PRIORITAS staff. These lessons learned from PRIORITAS PPG and PKB initiatives can be utilized by central government to improve PPG and PKB in non-PRIORITAS districts. In this regard, central government should coordinate with local government entities and use locally available capacities – that is, TTIs – to help local governments develop and implement PPG and PKB. This support could begin in provinces with a higher human development index (HDI) scores, because they generally employ higher numbers of education personnel. Question 4 How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level? The evaluation team found that some districts had included elements of PRIORITAS interventions in their district regulations or annual plans and budgets. Following technical assistance from PRIORITAS, the43 partner districts that held local elections in late 2015 developed their five-year strategic plans in 2016 and included a total of US$87 million to continue in-service teacher training and dissemination of good practices over the next five years. These plans and budgets represent a formal, legal basis to continue implementing PRIORITAS programs in these districts. In addition, the local governments have disseminated the project’s best practices using their own funds to as many as 36,300 schools, reaching over 232,600 teachers and school principals, and benefitting approximately 12.5 million students. Local governments spent US$8.3 million for replication in other schools, demonstrating an effort by PRIORITAS stakeholders to sustain project approaches. Some districts have progressed beyond planning. Maros, South Sulawesi, is one example where the bupati (regent, or district head) issued regulations stipulating that educational budgets should cover the dissemination of PRIORITAS school management and active teaching and learning best practices. Maros has budgeted to train more than 2,000 teachers, school principals and school committee members at non-PRIORITAS schools. Another example is the PRIORITAS district of Lumajang in East Java, where the local government included funding for dissemination of PRIORITAS best practices in its education budget. Other PRIORITAS districts in East Java, including Batu and Blitar, have also funded unique USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 7 programs to disseminate PRIORITAS interventions. Pinrang District in South Sulawesi has a unique education regulation that stipulates that for teachers to be promoted to be a school principal, or a school principal to be promoted to supervisor, they must have formerly been a Decentralized Basic Education (DBE) master teacher, PRIORITAS facilitator or facilitator for another donor program. More generally, the evaluation team found that PRIORITAS district facilitators are now being used as resource persons for implementing Dinas Pendidikan programs. At the school level, evaluators were told that a number of factors and methods will be used to sustain PRIORITAS practices in schools, including steps to ensure that the transition to active teaching and learning is continued. Positive forces for sustainability include: the familiarity that many teachers have already developed with active teaching and learning; inclusion of these methods in school annual plans and budgets; and efforts of working groups (KKG, KKM, and MGMP working group for teachers; KKKS, MKKS, and KKKM working groups for principals; and KKPS working groups for school supervisors) to carry on PRIORITAS methods and to improve coordination among local education stakeholders: teachers, school principals and school supervisors. However, interviews with teachers (in total 38 teachers) in three primary schools, one junior secondary school, and two madrasah revealed that they rarely attended KKG/KKM/MGMP meetings. Some teachers argued that they were busy and had no time to attend the meeting or the KKG/KKM/MGMP was not active (there were no meetings at all) while others argued that the materials delivered in the meeting were unattractive so they preferred not to come. In these schools, dissemination of PRIORITAS activities did not occur at KKG/KKM/MGMP level, rather it happened at the school level. There are indications of sustainability at the district and school levels, stemming from the commitment of local leaders to continue PRIORITAS-supported practices. These indications include incidences of activity planning, budget allocation and community participation. School principals and district heads play critical roles in creating the enabling environment for PRIORITAS sustainability. They have the authority to ensure sufficient resources are allocated to develop and implement strategic planning. In addition, school supervisors can take the place of PRIORITAS district facilitators in supporting teachers. Parents and the communities at large, through paguyuban and school committees, serve as both pressure and support groups for schools and teachers to continue applying methodologies and approaches introduced by PRIORITAS. However, the sustainability of PRIORITAS district-level interventions, or those of any USAID or donor education projects, cannot be guaranteed by the hopes of local education officials alone. To ensure that all PRIORITAS districts incorporate PRIORITAS programs into their future regulations, plans and budgets, PRIORITAS stakeholders must systematically apply advocacy tools of information, encouragement and pressure to local political leaders. In addition, future GOI efforts and donor-assisted basic education projects may need to place greater emphasis on the role of local educator professional groups to sustain and disseminate interventions. District and stakeholder leaders should therefore allocate sufficient resources to support activities of the KKG, KKM, MGMP, and KKKS. The evaluation team further recommends that future education programs emphasize training of school supervisors, as their positions are key to ensuring schools’ accountability in the delivery of quality education, including providing support to enhance the functioning of KKG, KKM, and MGMP. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 8 Question 5 How have national government agencies (MOEC, MORA, and MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects? PRIORITAS activities aligned with GOI programs and regulations, including the project’s results in building the capacity of human resources in 53 district partners, 995 primary schools, 457 junior-secondary schools, 17 TTIs and 31 consortia TTIs. In total, project data show that PRIORITAS provided benefits to 29,409 teachers, 8,497 higher education lecturers, 478,612 learners in partner schools, and 45,685 student teachers. Project interventions to improve the quality of teachers, teaching processes and learning outcomes contributed positively to the national application of Curriculum 2013.The PRIORITAS reading program supported the Ministry of Education and Culture (MOEC) school literacy movement, as stated in Ministry Decree 23/2015 on Character Building. MOEC has designated PRIORITAS’19 partner districts as models for literacy movement. The PPG program has assisted local governments to perform mandates as stated in the Law 14/2005, Government Regulation 74/2008 and the joint regulation of five ministries. Recognizing PRIORITAS’ effective methodologies and approaches, MOEC invited the project to train representatives of 33 provincial LPMPs and MOEC’s sekolahrujukan (model schools) MOEC officials also expressed interest in adopting PRIORITAS training modules and employing PRIORITAS facilitators to conduct teacher trainings. In addition, the Ministry of Religious Affairs (MORA) has allocated IDR 30 billion to be distributed to 200 KKM in the 90 partner district over three years (2017 to 2019), and IDR 1 billion per year for three years to nine TTIs. Similarly, officials from the Ministry of Research and Higher Education (MORHE) welcomed PRIORITAS to share information about comparative performances of partner TTIs. While acknowledging the positive alignment between PRIORITAS and GOI efforts in basic education development, the evaluation team found that the delayed formal agreement between USAID/Indonesia and PRIORITAS with the project’s primary counterpart ministry, MOEC, hampered the proactive participation and support from staff of several MOEC departments. For the GOI and project stakeholders to gain the greatest benefits from PRIORITAS over the long term, the evaluation team recommends that the project hand over all PRIORITAS products through a formal process or event that involves national-level decision-makers. The team also recommended that the GOI take proactive steps to adopt PRIORITAS methodologies and approaches, as they have yielded positive results in partner districts. Further, the MOEC Secretary General’s Office or another relevant office should consider a stronger focus on coordination to ensure coherence among donor-funded projects, and to provide better support to partner agencies. A renewed focus on coordination will provide a foundation for the GOI to actively participate in project implementation and improve policy linkages between Jakarta and supported regions. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 9 RINGKASAN EKSEKUTIF Latar Belakang Evaluasi kinerja akhir Proyek USAID Prioritizing Reforms, Innovations and Opportunities for Reaching Indonesia’s Teachers, Administrators and Students (PRIORITAS) (Mengutamakan Pembaharuan, Inovasi dan Kesempatan bagi Guru, Tenaga Kependidikan, dan Siswa) berlangsung dari bulan Februari hingga Mei 2017. PRIORITAS merupakan bagian terakhir dari tiga proyek pendidikan dasar USAID/Indonesia yang dilaksanakan berturut-turut, dimulai pada tahun 2003, untuk membantu Indonesia dalam meningkatkan kualitas pendidikan dasar. Proyek USAID Managing Basic Education (MBE), atau Mengelola Pendidikan Dasar, berjalan dari tahun 2003–2007 dan tiga proyek USAID Decentralized Basic Education atau Pendidikan Dasar Terdesentralisasi (DBE1, 2 dan 3) beroperasi sejak tahun 2005–2011. MBE, DBE dan PRIORITAS memiliki karakteristik program yang sama: partisipasi komunitas sekolah; bekerja langsung di sistem lokal untuk membangun komitmen di tingkat propinsi dan kabupaten; pelatihan intensif yang ditindaklanjuti dengan mentoring; penyediaan bantuan teknis daripada bantuan dana; dan berbasis konten tentang kebijakan dan inovasi pendidikan Pemerintah Indonesia yang terus berkembang. PRIORITAS, yang beroperasi sejak tahun 2012–2017, bekerja secara intensif di 53 kabupaten di sembilan propinsi, termasuk tiga kabupaten di Papua dan Papua Barat. PRIORITAS bertujuan untuk mendukung keseluruhan tujuan USAID atas perluasan akses terhadap pendidikan dasar yang berkualitas bagi anak-anak di Indonesia. Proyek ini memiliki tiga komponen: 1. Memperkuat program pengembangan guru pra-jabatan dan guru dalam jabatan untuk meningkatkan kualitas pengajaran di kelas. 2. Peningkatan manajemen pendidikan dan tata kelola sekolah dan kabupaten untuk meningkatkan dukungan pada pemberian kelas pendidikan. 3. Peningkatan koordinasi komunikasi, perencanaan dan pembuatan kebijakan berbasis informasi, tanggapan dan penggunaan sumber daya keuangan dan sumber daya manusia di semua tingkat sistem guna meningkatkan pemberian pendidikan ketingkat yang lebih rendah. Tujuan Tujuan utama dari evaluasi kinerja akhir ini adalah untuk menginformasikan kepada USAID, jajaran Pemerintah Indonesia, mitra PRIORITAS dan pendonor lainnya mengenai hasil PRIORITAS yang telah berhasil di sekolah baik di tingkat kabupaten dan propinsi; implikasi dari hasil tersebut terhadap kualitas dan relevansi pendidikan dasar di Indonesia; dan praktik serta pelajaran dari PRIORITAS bahwa pemangku kepentingan terkait dapat terus melanjutkan peningkatan akses terhadap pendidikan dasar berkualitas di Indonesia. Evaluasi ini berusaha mencari jawaban atas lima pertanyaan penelitian berikut ini yang ditentukan oleh USAID: 1. Bagaimana kompetensi dan keterampilan guru pra-jabatan dan guru dalam jabatan yang didukung PRIORITAS, meningkat dalam memberikan metodologi pembelajaran aktif? Bagaimana PRIORITAS memperkuat institusi (Lembaga Pendidikan Tenaga Kependidikan) untuk mendukung para guru tersebut? 2. Bagaimana kapasitas dan inisiatif sekolah, komunitas (termasuk orang tua, komite sekolah, media) dan pemerintahan kabupaten yang didukung PRIORITAS meningkat dalam mengembangkan/mendukung manajemen sekolah yang akuntabel dan partisipatif? USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 10 3. Bagaimana kapasitas dan inisiatif pemerintah daerah yang didukung PRIORITAS meningkat dalam mengkoordinasikan manajemen sumber daya pendidikan dengan berfokus pada penyebaran guru, pengembangan profesional guru secara berkelanjutan, dan pendanaan pendidikan? 4. Bagaimana hasil dan praktik PRIORITAS dapat dipertahankan oleh lembaga dan institusi mitra di tingkat regional? 5. Bagaimana lembaga-lembaga pemerintahan nasional (Kemendikbud, Kemenag, dan Kemenristekdikti) mendapat manfaat dari PRIORITAS? Bagaimana mereka akan melanjutkan dan menangani aspek keberlanjutan? Metodologi dan Keterbatasan Tim evaluasi menggunakan pendekatan metode beragam untuk evaluasi akhir PRIORITAS, secara kuantitatif dan kualitatif. Data kuantitatif didapatkan melalui survei daring yang dikirim ke Lembaga Pendidikan Tenaga Kependidikan (LPTK) dan analisa data pemantauan kinerja sekunder dari proyek. Data kualitatif sebagian besar dikumpulkan pada wawancara informan utama dan diskusi kelompok terarah (focus group discussions / FGD) dengan informan ditingkat nasional, kabupaten dan sekolah, dan di LPTK mitra PRIORITAS. Tim evaluasi juga melakukan sejumlah besar observasi pelajaran di kelas di sekolah mitra proyek. Tim evaluasi mengidentifikasi keterbatasan berikut untuk evaluasi Ruang Lingkup Kerja dan implementasinya:  Hanya sejumlah kecil kabupaten dan sekolah yang dijadikan contoh, dibandingkan dengan jumlah yang besar dari lokasi dan mitra proyek.  Pendekatan kualitatif utama, yang mungkin tidak memungkinkan tim evaluasi untuk menghitung hasil bersih dari dampak PRIORITAS (namun akan memberikan wawasan lebih dalam mengapa proyek tersebut berhasil atau tidak berhasil dalam situasi tertentu).  Adanya program donor serupa yang sebelumnya atau sekarang ada dibidang pendidikan, dan mengakibatkan sulitnya untuk mengidentifikasi secara tepat dampak unik dari PRIORITAS dibeberapa kasus.  Kehadiran pengevaluasi selama administrasi observasi kelas, membawa bias (efek Hawthorne).  Tidak tersedianya para pengambil keputusan senior dari Kemendikbud, Kemenag dan Kemenristekdikti untuk menjadi informan utama selama evaluasi. Temuan Umum Evaluasi ini berusaha mencari jawaban atas lima pertanyaan penelitian yang ditentukan USAID. Laporan evaluasi yang lengkap memuat temuan, kesimpulan dan rekomendasi tim atas setiap pertanyaan. Ringkasan dari hasil evaluasi adalah sebagaimana tertuang dibawah ini didalam Ringkasan Eksekutif ini. Pertanyaan 1 Bagaimana kompetensi dan keterampilan guru pra-jabatan dan guru dalam jabatan yang didukung PRIORITAS, meningkat dalam memberikan metodologi pembelajaran aktif? Bagaimana PRIORITAS memperkuat institusi (LPTK) untuk mendukung para guru tersebut? Bagian dari laporan evaluasi ini memiliki dua bagian. Yang pertama membahas penyampaian metodologi pembelajaran aktif, termasuk program membaca PRIORITAS; yang kedua mencakup intervensi PRIORITAS di LPTK. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 11 Metodologi Pembelajaran Aktif PRIORITAS mengembangkan tiga set modul awal, dengan modul keempat ditambahkan kemudian, yang digunakan proyek untuk melatih para guru atas tiga mata pejalaran inti – matematika, ilmu pengetahuan alam dan bahasa Indonesia – dalam metodologi Pembelajaran Aktif, Kreatif, Efektif dan Menyenangkan, dikenal dengan singkatan PAKEM. Pelatihan ini diikuti setidaknya enam guru disetiap sekolah dasar dan 15 guru disetiap sekolah menengah pertama, sesuai rencana proyek. Evaluasi menunjukkan respon positif dari sekolah dan pemerintah kabupaten atas pelaksanaan metodologi PAKEM, yang menghasilkan para guru menggunakan PAKEM dalam mata￾mata pelajaran diluar dari tiga mata pelajaran inti, termasuk Bahasa Inggris dan ilmu pengetahuan sosial. Kegiatan proyek disebarluaskan ke khalayak lebih luas melalui kelompok kerja guru (KKG/KKM atau MGMP). Sementara proyek itu sendiri menyebarluaskan modul PRIORITAS, penyebaran penuh dari metodologi ini terjadi sebagai hasil dari antusiasme dan dukungan para pemimpin sekolah dan kabupaten. Penyebarluasan ini bergantung pada ketersediaan sumber daya keuangan, seperti anggaran kabupaten (dikenal dengan singkatan APBD) dan Dana Bantuan Operasional Sekolah Yang Dialokasikan Pemerintah (BOS). Sebagian besar dari semua kategori informan yang diwawancara oleh tim evaluasi di sekolah PRIORITAS memberikan laporan bahwa manfaat pendidikan yang positif dan dapat diamati dihasilkan dari para guru yang mengadopsi metode pengajaran dan pembelajaran aktif yang ditindaklanjuti dengan mentoring. Observasi di kelas dan FGD dengan para guru dan siswa membenarkan temuan ini. Tetapi tim evaluasi mencatat bahwa terlepas dari intervensi PRIORITAS, guru yang berusia lebih tua dan guru mata pelajaran non-inti jarang yang menggunakan metode pengajaran aktif. Data pemantauan dan evaluasi (monitoring and evaluation / M&E) PRIORITAS menunjukkan peningkatan yang cukup tinggi atas kinerja siswa pada ujian matematika, ilmu pengetahuan alam dan bahasa Indonesia yang dilaksanakan oleh PRIORITAS di sekolah-sekolah mitra dari tahun 2013–2016. Data juga menunjukkan bahwa, secara umum, siswa perempuan memperoleh nilai lebih tinggi dari siswa laki-laki. Para siswa diFGD menyatakan bahwa setelah guru menerapkan PAKEM, para siswa dapat menyerap pelajaran matematikan dan ilmu pengetahuan alam lebih mudah. Selain itu, hasil observasi kegiatan belajar mengajar mengungkapkan bahwa siswa laki-laki dan perempuan menerima kesempatan yang sama untuk berpartisipasi dalam kegiatan belajar. Tim evaluasi merekomendasikan agar sekolah meneruskan paket pelatihan PRIORITAS dengan mentoring dalam metodologi pembelajaran aktif guna memastikan para guru menerima bantuan teknis yang mereka perlukan untuk melaksanakan model ini secara efektif. Merupakan hal yang penting juga untuk terus memperkuat kelompok kerja guru (KKG/KKM dan MGMP) sebagai forum berbagi untuk menginovasi dan menyebarluaskan program PRIORITAS, dan meningkatkan kompetensi dan keterampilan guru dalam melaksanakan metodologi pembelajaran aktif. PRIORITAS memberikan fokus khusus pada pengembangan membaca. Modul pelatihan guru proyek mencakup segmen membaca usia dini, mengembangkan minat membaca dan pemahaman membaca. Banyak informan melaporkan bahwa budaya membaca yang diperkenalkan PRIORITAS umumnya bermanfaat dan populer, dan berhasil di sekolah mitra. PRIORITAS juga menyediakan banyak buku, termasuk buku-buku untuk membaca secara berjenjang (leveled reading), untuk sekolah-sekolah. Dari intervensi tersebut, data proyek mengindikasikan terdapat lebih dari 5 juta penerima manfaat langsung dan tidak langsung dari program membaca PRIORITAS. Oleh karenanya, inovasi dan kemitraan sekolah dengan perpustakaan kabupaten, perpustakaan bergerak atau program membaca lainnya harus dipertahankan dan sangat direkomendasikan untuk proyek membaca di kemudian hari. Lembaga Pendidikan Tenaga Kependidikan Terkait LPTK, para informan utama menyatakan bahwa penyelenggaraan proses pelatihan PRIORITAS yang efisien sangat dihargai, dan semua LPTK yang dijadikan contoh dalam evaluasi ini sekarang menggunakan konten USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 12 yang didukung proyek. PRIORITAS diakui telah mengembangkan pengetahuan dan keterampilan guru pendidik dan calon guru untuk menggunakan metodologi pelatihan guru yang berpusat pada siswa dan inovatif. Hasil ini tercapai dengan adanya kesepakatan antara proyek dengan 17 mitra LPTK, termasuk lembaga negari dan institusi Islam; dan dengan bekerja sama dengan 31 LPTK lainnya melalui konsorsium di propinsi. Bantuan teknis dari PRIORITAS juga telah meningkatkan kapasitas LPTK untuk menjadi penyedia layanan. Berbagai Nota Kesepakatan telah dibuat antara LPTK dan pemerintah propinsi atau kabupaten atau sekolah atas usaha-usaha yang bertujuan untuk meningkatkan kualitas para guru. Untuk membangun kapasitas LPTK dan melanjutkan kesuksesan PRIORITAS, adalah penting bagi LPTK untuk memiliki program afirmatif internal untuk menyebarluaskan metode pembelajaran aktif keseluruh departemen mereka dan memperluas cakupan mereka diluar mata pelajaran ilmu pengetahuan alam, matematika dan bahasa Indonesia untuk mewujudkan dampak yang lebih luas ditingkat universitas. Tim evaluasi merekomendasikan agar LPTK mitra PRIORITAS melanjutkan kemitraan mereka dengan anggota konsorsium, sekolah mitra dan sekolah laboratorium guna mempertahankan praktik-praktik yang efektif setelah proyek berakhir. Mengingat manfaat PRIORITAS ditingkat LPTK, Pemerintah Indonesia harus mempertimbangkan cara mengadopsi praktik terbaik dari pelatihan PRIORITAS untuk meningkatkan kualitas pelatihan pra-mengajar di LPTK - yaitu, menindaklanjuti pelatihan dengan mentoring dan penyediaan bantuan teknis. Pertanyaan 2 Bagaimana kapasitas dan inisiatif sekolah, komunitas (termasuk orang tua, komite sekolah, media) dan pemerintah kabupaten yang didukung oleh PRIORITAS meningkat dalam mengembangkan/mendukung manajemen sekolah yang akuntabel dan partisipatif? Disebagian besar sekolah mitra, intervensi PRIORITAS efektif dalam meningkatkan kapasitas untuk manajemen sekolah dan tata kelola yang partisipatif. Intervensi juga bermanfaat untuk sekolah yang dilaporkan telah sepenuhnya mengadopsi manajemen berbasis sekolah inklusif sebelum dimulainya PRIORITAS. Bahkan, tim evaluasi menemukan kalau intervensi proyek efektif secara khusus untuk memperkuat partisipasi komunitas dalam tata kelola sekolah. Para informan melaporkan adanya perubahan positif di dalam manajemen sekolah mereka yang akuntabel dan partisipatif sebagai hasil dari intervensi PRIORITAS. Peningkatan-peningkatan tersebut mencakup: (1) keterlibatan dan dukungan yang lebih dari paguyuban (asosiasi orang tua), orang tua dan komite; (2) penggunaan dana BOS yang lebih baik; dan (3) komunitas yang diberdayakan untuk berpartisipasi lebih di sekolah mereka sebagai hasil langsung dari anggota komite yang diikutsertakan pada acara-acara pelatihan PRIORITAS. Para guru dan orang tua yang diwawancara tim evaluasi menyatakan mereka jadi lebih sering dan mudah berkomunikasi sejak diterapkannya PRIORITAS di sekolah mereka. Temuan-temuan ini konsisten dengan data pemantauan PRIORITAS, yang menunjukkan lebih banyak sekolah sekarang yang menerapkan manajemen sekolah yang transparan dan partisipatif. PRIORITAS telah membentuk budaya komunitas sekolah dan orang tua mengenai pentingnya perencanaan atas pendidikan yang berkualitas. Proyek telah memberikan pemahaman yang lebih baik atas peran kepala sekolah, guru dan komite sekolah dalam manajemen dan tata kelola sekolah. Sebagian besar informan di sekolah yang ditemui tim evaluasi menyatakan bahwa para guru sekarang terlibat dalam tim manajemen yang berurusan dengan pengembangan sekolah. Pihak sekolah mengundang para orang tua untuk berbagi informasi terkini terkait sekolah, ketika sekolah membutuhkan dukungan orang tua atau pada saat penyusunan angggaran sekolah (RKAS) dan perencanaan kegiatan sekolah (RKS). Proyek juga mendorong partisipasi komunitas untuk mendukung pendidikan yang berkualitas. Para orang tua menjadi lebih terlibat dengan sekolah dan membantu sekolah untuk berbagai keperluan (misalnya dana, bahan/peralatan dan tenaga kerja money), terutama dengan adanya paguyuban. Sebagai suatu badan, struktur paguyuban berada di bawah komite sekolah dan mereka mendukung kegiatan sekolah dalam berbagai cara, yang USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 13 paling signifikan program pembelajaran aktif dan program membaca. Mereka menyediakan materi pengajaran dan bantuan, serta perabotan dan buku-buku untuk pojok membaca. Ketika komite sekolah membutuhkan dukungan untuk berpartisipasi sepenuhnya dalam tata kelola sekolah atau untuk mengambil tanggung jawab baru yang diserahkan kepada mereka, pemerintah daerah seharusnya mengadopsi pendekatan pelatihan inklusif seperti yang diambil oleh PRIORITAS, dimana pelatihan diberikan tidak hanya untuk kepala sekolah dan pegawai, tetapi juga untuk komite sekolah dan anggota komunitas, alumni, dll. Pelatihan tersebut akan membantu semua anggota komite memahami peran dan tindakan yang diperlukan untuk meningkatkan kualitas pendidikan. Tim evaluasi menggunakan interpretasi “media” yang lebih luas untuk mencakup produk komunikasi PRIORITAS dan liputan media massa terkait kegiatan dan metodologi proyek, sebagaimana diamati oleh para informan. PRIORITAS mengembangkan program komunikasi dan penjangkauan menggunakan video, buku-buku praktik terbaik, sumber daya situs web dan bahan-bahan lainnya. Para informan evaluasi dan responden survei menyatakan bahwa media massa telah sering membuat cerita tentang kegiatan PRIORITAS; bahwa cerita-cerita media telah meningkatkan kesadaran publik mengenai isu pendidikan; dan bahwa para informan telah menggunakan media PRIORITAS (misalnya situs web, newsletter, buku) untuk mempelajari praktik-praktik terbaik PRIORITAS. Kegiatan donor atau proyek pendidikan Pemerintah Indonesian yang akan datang disarankan untuk mengadopsi pendekatan komunikasi PRIORITAS – memanfaatkan media online, media sosial dan media massa untuk menyebarluaskan informasi dan materi program ke khalayak luas para pemangku kepentingan. Intervensi tersebut harus dikaitkan dengan hasil proyek yang diinginkan, dengan indikator terkait dampak secara spesifik. Pertanyaan 3 Bagaimana kapasitas dan inisiatif pemerintah daerah yang didukung PRIORITAS meningkat dalam mengkoordinasikan manajemen sumber daya pendidikan, dengan berfokus pada penyebaran guru, pengembangan keprofesian guru yang berkelanjutan, dan pendanaan pendidikan? Bagian laporan ini khususnya membahas Program Penyebaran Guru (PPG) PRIORITAS dan program terkait Pengembangan Keprofesian Guru Berkelanjutan (PKB) dimana setiap kabupaten berusaha untuk mendanai dan mengelola para gurunya. PPG dan PKB PRIORITAS dirancang untuk membantu pemerintah daerah dalam menangani kekurangan pasokan guru pegawai negeri sipil dan banyaknya guru temporer, serta distribusi guru pegawai negeri sipil yang tidak tepat antara sekolah di perkotaan dan di pedesaan. Para informan memuji PRIORITAS yang telah membantu pemerintah daerah dalam menilai penyebaran guru-guru yang ada, dan membuat pendistribusian ulang menggunakan kebijakan baru yang inovatif. Sebagai tambahan, mitra kabupaten PRIORITAS telah berhasil melaksanakan PPG dan PKB – dengan tugas sebagaimana diatur dalam UU No. 14/2005, Peraturan Pemerintah No. 74/2008 dan peraturan bersama lima kementerian. Terkait PPG dilaporkan bahwa terdapat total 40 kebijakan baru terkait dengan penyebaran guru yang telah diadopsi oleh mitra kabupaten PRIORITAS seperti yang dicatat pada waktu evaluasi. Kebijakan baru ini mencakup pengenalan kelas multi tingkat, pengelompokan sejumlah sekolah menjadi satu, pemutaran tugas guru dari satu sekolah ke sekolah lain dan perekrutan guru-guru baru. Untuk meningkatkan kapasitas guru, beberapa mitra kabupaten PRIORITAS mengalokasikan lebih dari Rp 25 miliar dalam anggaran 2014 mereka. Alokasi dukungan anggaran untuk menyebarluaskan praktik-praktik PRIORITAS menggambarkan peningkatan dalam koordinasi antar pihak otoritas tingkat kabupaten. Pihak otoritas utama – Dinas Pendidikan, Badan Kepegawaian Daerah, Dinas Pendapatan Daerah dan Badan Perencanaan Pembangunan Daerah (BAPPEDA) – saat ini telah memiliki pemahaman yang lebih baik atas isu strategis yang dihadapi sektor pendidikan mereka. Pemahaman ini menimbulkan koordinasi yang lebih kuat dan menyebabkan sektor pendidikan menerima banyak bantuan finansial. Tim evaluasi menemukan bahwa koordinasi ini sepertinya akan dipertahankan, karena proses USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 14 penganggaran yang relevan terjadi melalui sistem perencanaan pembangunan yang diterapkan secara nasional dan terkoordinasi, yang dikenal dengan singkatan MUSRENBANG. Gabungan pengembangan kapasitas lokal dan bekerja sama dengan LPTK selaku penyedia layanan juga memperkuat koordinasi antara LPTK dan pemerintah daerah selaku pengguna, dan koordinasi ini pada akhirnya harus meningkatkan keberlanjutan dari usaha-usaha PRIORITAS. Hasil ini berasal dari pendekatan PRIORITAS untuk mengembangkan sumber daya manusia lokal untuk menjadi pelatih PPG/PKB. Para partisipan dilatih oleh proyek mencakup mereka yang saat ini menjadi guru, kepala sekolah, pengawas sekolah, dosen universitas dan pelatih Lembaga Penjaminan Mutu Pendidikan (LPMP). Pemerintah daerah akan dapat memobilisasi sumber daya tersebut untuk membantu mempertahankan PPG dan PKB mereka. Gabungan pemerintah daerah sebagai penerima manfaat dan LPTK / LPMP yang bertindak sebagai penyedia layanan adalah suatu pembelajaran: semua badan dapat bekerja sama dalam mendukung pendidkan yang berkualitas. Terlepas dari semua usaha untuk meningkatkan PPG dan PKB tersebut, mayoritas informan menyampaikan kepada pengevaluasi kalau mereka tidak dapat dengan yakin memperkirakan apakah metode PRIORITAS akan digunakan tanpa batas waktu setelah dukungan dari PRIORITAS berakhir. Mereka mencontohkan sejumlah kendala untuk mempertahankan praktik ini, termasuk campur tangan dari para politisi lokal dalam alokasi dana pendidikan dikemudian hari dan kesulitan yang sedang dihadapi menggunakan sistem informasi berbasis komputer untuk PPG. Tantangan ini mengikuti temuan pada studi penyebaran guru di Indonesia oleh USAID di tahun 2015. Mengenai PKB, informan menyoroti besarnya tantangan keuangan pemerintah daerah untuk membayar program pengembangan keterampilan bagi semua guru mereka. Tim evaluasi mencatat, meskipun demikian, warisan PRIORITAS terlihat disetiap kabupaten sekarang memiliki banyak guru yang telah dilatih PRIORITAS, kepala sekolah berkualitas untuk bertindak sebagai fasilitator, pengawas sekolah, dosen universitas dan pelatih LPMP yang dapat dimobilisasi untuk membantu mempertahankan kegiatan PKB di kemudian hari. Tim evaluasi mencatat bahwa pemerintah daerah harus melakukan pemutakhiran data guru mereka secara teratur dan menerapkan PPG dan PKB untuk membantu memastikan pemberian pendidikan berkualitas dan menjaga kualitas koordinasi. Tim menyarankan pemerintah daerah untuk tetap terus mencari bantuan dari LPTK yang memiliki kapasitas untuk memfasilitasi proses PPG dan PKB. Tim juga merekomendasikan agar pemerintah pusat menjalankan peran pentingnya (sebagaimana ternyata dalam UU No. 23/2014 mengenai Pemerintahan Daerah) dengan menyediakan bantuan teknis kepada para mitra dan non-mitra kabupaten PRIORITAS, melanjutkan peran yang sebelumnya dilakukan oleh staf PRIORITAS. Pembelajaran dari inisiatif PPG dan PKB PRIORITAS dapat dimanfaatkan oleh pemerintah pusat untuk meningkatkan PPG dan PKB di kabupaten￾kabupaten non-PRIORITAS. Terkait hal ini, pemerintah pusat harus berkoordinasi dengan lembaga pemerintah daerah dan menggunakan kapasitas daerah yang tersedia – yakni, LPTK – untuk membantu pemerintah daerah mengembangkan dan melaksanakan PPG dan PKB. Dukungan ini dapat dimulai di tingkat propinsi dengan nilai Indeks Pembangunan Manusia (IPM) yang lebih tinggi, karena umumnya mereka mempekerjakan lebih banyak personil pendidikan. Pertanyaan 4 Bagaimana hasil dan praktik PRIORITAS dapat dipertahankan oleh badan dan institusi mitra ditingkat daerah? Tim evaluasi menemukan beberapa kabupaten telah memasukkan elemen intervensi PRIORITAS dalam peraturan kabupaten mereka atau rencana kerja dan anggaran tahunan. Mengikuti bantuan teknis dari PRIORITAS, 43 mitra kabupaten yang menyelenggarakan pilkada di akhir 2015 mengembangkan rencana strategis lima tahun mereka di 2016 dan memasukkan anggaran sejumlah US$87 juta untuk melanjutkan pelatihan guru dalam jabatan dan penyebarluasan praktik yang baik selama lima tahun ke depan. Rencana dan anggaran ini mewakili dasar hukum formal untuk melanjutkan pelaksanaan program PRIORITAS di kabupaten-kabupaten tersebut. Selain itu, pemerintah daerah telah menyebarluaskan praktik terbaik dari proyek menggunakan dana mereka sendiri kesebanyak 36.300 sekolah, mencapai lebih dari 232.600 guru dan kepala sekolah, dan memberi keuntungan USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 15 untuk sekitar 12,5 juta siswa. Pemerintah daerah menghabiskan dana US$8,3 juta untuk membuat replika di sekolah-sekolah lain, menunjukkan upaya pemangku kepentingan PRIORITAS untuk mempertahankan pendekatan proyek. Beberapa kabupaten telah melewati tahap perencanaan. Maros, Sulawesi Selatan, adalah satu contoh dimana bupatinya mengeluarkan peraturan yang mengatur anggaran pendidikan harus mencakup penyebarluasan manajemen sekolah PRIORITAS dan praktik terbaik dari pengajaran dan pembelajaran aktif. Maros telah menganggarkan pelatihan untuk 2.000 guru, kepala sekolah dan anggota komite sekolah di sekolah-sekolah non￾PRIORITAS. Contoh lainnya adalah kabupaten PRIORITAS Lumajang di Jawa Timur, dimana pemerintah daerah memasukkan dana untuk penyebarluasan praktik terbaik PRIORITAS dalam anggaran pendidikannya. Kabupaten PRIORITAS lainnya di Jawa Timur, termasuk Batu dan Blitar, juga telah mendanai program khusus untuk menyebarluaskan intervensi PRIORITAS. Kabupaten Pinrang di Sulawesi Selatan memiliki peraturan pendidikan yang unik yang mengatur guru yang dipromosikan menjadi kepala sekolah, atau kepala sekolah yang dipromosikan menjadi pengawas, harus sebelumnya pernah menjadi guru pelatih Pendidikan Dasar yang Terdesentralisasi (DBE), fasilitator PRIORITAS atau fasilitator program donor lainnya. Lebih umum, tim evaluasi menemukan bahwa fasilitator kabupaten PRIORITAS saat ini menjadi nara sumber untuk melaksanakan program-program Dinas Pendidikan. Ditingkat sekolah, para pengevaluasi diberitahu bahwa sejumlah faktor dan metode akan digunakan untuk mempertahankan praktik PRIORITAS di sekolah, termasuk langkah-langkah guna memastikan transisi kepengajaran dan pembelajaran aktif terus dijalankan. Kekuatan positif untuk keberlanjutan mencakup: kebiasaan pengajaran dan pembelajaran aktif yang telah dikembangkan banyak guru; memasukkan metode ini dalam rencana dan anggaran tahunan sekolah; dan upaya-upaya kelompok kerja (kelompok kerja untuk guru KKG, KKM, dan MGMP; kelompok kerja untuk kepala sekolah KKKS, MKKS, dan KKKM; dan kelompok kerja untuk pengawas sekolah KKPS) untuk melanjutkan metode PRIORITAS dan untuk meningkatkan koordinasi antar pemangku kepentingan pendidikan daerah: guru, kepala sekolah dan pengawas sekolah. Namun, hasil wawancana dengan guru-guru (total 38 guru) ditiga sekolah dasar, satu sekolah menengah pertama, dan dua madrasah mengungkapkan bahwa mereka jarang menghadiri pertemuan KKG/KKM/MGMP. Beberapa guru berargumen kalau mereka sibuk dan tidak memiliki waktu untuk menghadiri pertemuan tersebut atau KKG/KKM/MGMP-nya tidak aktif (tidak menyelenggarakan pertemuan sama sekali), sementara yang lainnya berargumen bahwa materi yang diberikan pada pertemuan tidak menarik sehingga mereka memilih untuk tidak hadir. Di sekolah seperti ini, penyebarluasan kegiatan PRIORITAS tidak terjadi di tingkat KKG/KKM/MGMP, melainkan terjadi ditingkat sekolah. Terdapat indikasi keberlanjutan di tingkat kabupaten dan sekolah yang berasal dari komitmen pemimpin daerah untuk melanjutkan praktik yang didukung PRIORITAS. Indikasi tersebut mencakup adanya perencanaan kegiatan, alokasi anggaran dan partisipasi komunikasi. Kepala sekolah dan bupati memainkan peran penting dalam menciptakan lingkungan yang mendukung keberlanjutan PRIORITAS. Mereka memiliki kewenangan untuk memastikan cukupnya sumber daya yang dialokasikan untuk membangun dan melaksanakan perencanaan strategis. Selain itu, pengawas sekolah dapat berperan sebagai fasilitator kabupaten PRIORITAS guna mendukung guru. Orang tua dan masyarakat luas, melalui paguyuban dan komite sekolah, berperan sebagai kelompok penekan dan pendukung sekolah dan guru untuk terus menerapkan metodologi dan pendekatan yang diperkenalkan PRIORITAS. Namun demikian, keberlanjutan dari intervensi PRIORITAS ditingkat kabupaten atau setiap proyek pendidikan USAID atau donor, tidak dapat dijamin hanya dengan harapan dari para pejabat pendidikan daerah. Guna memastikan semua kabupaten PRIORITAS telah memasukkan program PRIORITAS kedalam peraturan, rencana dan anggaran mereka di kemudian hari, para pemangku kepentingan PRIORITAS harus secara sistematis menerapkan perangkat informasi advokasi, dorongan dan tekanan kepada para pemimpin politik daerah. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 16 Sebagai tambahan, upaya-upaya Pemerintahan Indonesia dan proyek pendidikan dasar yang dibantu pendonor dikemudian hari mungkin perlu memberikan penekanan lebih besar pada peran dari kelompok profesi pendidik daerah untuk mempertahankan dan menyebarluaskan intervensi. Pemimpin kabupaten dan pemangku kepentingan oleh karenanya harus mengalokasikan sumber daya yang cukup untuk mendukung kegiatan KKG, KKM, MGMP, dan KKKS. Tim evaluasi selanjutnya merekomendasikan program pendidikan ke depannya menekankan pada pelatihan untuk pengawas sekolah dimana posisi mereka merupakan kunci untuk memastikan akuntabilitas sekolah dalam memberikan pendidikan yang berkualitas, termasuk menyediakan dukungan untuk meningkatkan fungsi KKG, KKM, dan MGMP. Pertanyaan 5 Bagaimana lembaga-lembaga pemerintahan nasional (Kemendikbud, Kemenag, dan Kemenristekdikti) memperoleh manfaat dari PRIORITAS? Bagaimana mereka akan melanjutkan dan menangani aspek keberlanjutan? Kegiatan PRIORITAS diselaraskan dengan program dan peraturan Pemerintah Indonesia, termasuk hasil proyek dalam membangun kapasitas sumber daya manusia di 53 mitra kabupaten/kota, 995 sekolah dasar, 457 sekolah menengah pertama, 17 LPTK dan 31 konsorsium LPTK. Secara total, data proyek menunjukkan bahwa PRIORITAS memberikan manfaat kepada 29.409 guru, 8.497 pengajar tingkat pendidikan lebih tinggi, 478.612 pelajar di sekolah mitra, dan 45.685 guru siswa. Intervensi proyek untuk meningkatkan kualitas guru, proses pengajaran dan hasil pembelajaran yang dikontribusikan secara positif kedalam penerapan Kurikulum Nasional 2013. Program membaca PRIORITAS mendukung gerakan keaksaraan sekolah Kementerian Pendidikan dan Kebudayaan (Kemendikbud), sebagaimana tercantum dalam Keputusan Menteri No. 23/2015 mengenai Pembangunan Karakter. Kemendikbud telah menunjuk 19 mitra kabupaten PRIORITAS sebagai model percontohan untuk gerakan keaksaraan. Program PPG telah membantu pemerintah daerah dalam melaksanakan mandat yang dinyatakan dalam UU No. 14/2005, Peraturan Pemerintah No. 74/2008 dan peraturan bersama lima kementerian. Mengakui metodologi dan pendekatan PRIORITAS yang efektif, Kemendikbud mengundang proyek untuk melatih LPMP dan sekolah rujukan Kemendikbud di 33 propinsi. Pejabat Kemendikbud juga menyatakan minat mereka dalam mengadopsi modul pelatihan PRIORITAS dan mempekerjakan fasilitator PRIORITAS untuk pelaksanaan pelatihan guru. Selain itu, Kementerian Agama (Kemenag) telah mengalokasikan Rp. 30 miliar untuk didistribusikan ke 200 KKM di 90 kabupaten mitra selama tiga tahun (2017 hingga 2019), dan Rp. 1 miliar per tahun untuk tiga tahun kesembilan LPTK. Hal yang sama, pejabat dari Kementerian Riset Teknologi dan Pendidikan Lebih Tinggi (Kemenristekdikti) menyambut PRIORITAS untuk berbagi informasi mengenai komparasi kinerja para mitra LPTK. Sementara mengakui penyelarasan yang positif antara PRIORITAS dan upaya Pemerintah Indonesia dalam pengembangan pendidikan dasar, tim evaluasi menemukan bahwa kesepakatan formal yang tertunda antara USAID/Indonesia dan PRIORITAS dengan kementerian rekanan utama proyek, Kemendikbud, menghambat partisipasi aktif dan dukungan dari staf beberapa departemen Kemendikbud. Untuk Pemerintah Indonesia dan pemangku kepentingan proyek untuk mendapatkan manfaat terbesar dari PRIORITAS dalam jangka panjang, tim evaluasi menyarankan proyek mengalihkan semua produk PRIORITAS melalui proses atau acara formal yang melibatkan para pengambil keputusan di tingkat nasional. Tim juga merekomendasikan agar Pemerintah Indonesia mengambil langkah proaktif untuk mengadopsi metodologi dan pendekatan PRIORITAS, sebagaimana telah memberikan hasil positif di kabupaten mitra. Selanjutnya, Kantor Sekretaris Jendral Kemendikbud atau kantor relevan lainnya harus mempertimbangkan fokus yang lebih kuat pada koordinasi guna memastikan keselarasan antara proyek-proyek yang didanai donor, dan untuk memberikan dukungan lebih baik bagi lembaga-lembaga mitra. Fokus atas koordinasi yang diperbaharui akan memberikan dasar bagi Pemerintah Indonesia untuk berpartisipasi aktif dalam pelaksanaan proyek dan meningkatkan keterkaitan kebijakan antara Jakarta dan daerah-daerah yang didukung. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 17 1. PROJECT BACKGROUND 1.1 National Education Context Indonesia’s has the third-largest national education system in Asia, after China and India. In 2016, the data center of the Ministry of Education and Culture (MOEC) outlined the numerical dimensions of its basic education sector, which is the context of this report – almost 36 million students being taught by 2.2 million principals and teachers in 185,200 schools. The sector’s size, geographical spread and traditions have provided many challenges for Indonesia’s education reformers. These have included transitions from a traditional pedagogy built around rote memorization of facts to modern, active teaching and learning, and the steady devolution of many educational administration functions to individual education districts and schools. This report evaluates the substantial assistance to navigate those transitions by a USAID/Indonesia education project working with many districts and schools. 1.2 Project Background The USAID PRIORITAS project commenced in 2012. It was the last of three consecutive USAID Indonesia education projects that began in 2003 to assist Indonesia to improve the quality of basic education. The Managing Basic Education (MBE) Project ran from 2003-2007, and the three simultaneous Decentralized Basic Education (DBE) projects (DBE 1, 2 and 3) operated from 2005 to 2011. MBE, DBE and PRIORITAS shared similar program characteristics: school community participation and working directly in local systems to build commitment at the provincial and district levels; intensive and ongoing training, with follow-up mentoring provided; technical assistance rather than funding provided by the projects; and project content that was based on and followed evolving Indonesian government education policies and innovations. Collectively, the three projects covered six of the most heavily populated Indonesian provinces, plus some smaller ones; influenced education leaders in numerous districts and schools; and upgraded the professional skills of tens of thousands of school teachers. Using a whole-school development (WSD) approach, PRIORITAS sought to involve all school residents, including teachers, principals, school committees and communities in efforts to improve teaching capacities and delivery, and to improve school management and community support. PRIORITAS focused on building capacity at the provincial, district and school levels to improve the quality of teaching and learning. The project was meant to build on the accomplishments of the previous USAID projects to increase the capacity of stakeholders in the Indonesian education system to institutionalize and disseminate good practices. The project was also intended to support GOI innovations in education governance and management; help improve pre-service and in-service teacher training; and foster stakeholder involvement in coordinating and planning school education and managing funding. Specifically, from 2012 to 2017, PRIORITAS worked with 53 districts in nine provinces, including three districts in Papua and West Papua. The project engaged less intensively with another 40 districts that were USAID DBE activities previously supported. PRIORITAS aimed to:  Extend the capacity of district and provincial governments to plan, manage and finance education services.  Improve education human resource management in the provinces and districts, with a focus on teacher deployment.  Upgrade teacher capacity to implement active and contextual learning, and make teaching and learning in primary and junior-secondary schools more interesting, relevant and effective. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 18  Encourage parental and community involvement in schools, particularly supporting children in the early grades.  Develop participatory and accountable school management that focuses on improving learning outcomes.  Help prepare new and in-service teachers by supporting Teacher Training Institutes (TTIs) with course development and lecturer capacity building that emphasized best practices. FIGURE 1: PRIORITAS RESULTS FRAMEWORK PRIORITAS worked with all levels of government to expand evidence-based decision-making, communication and coordination. At the central level, the project supported Indonesia’s MOEC and Ministry of Religious Affairs (MORA) in introducing a new 2013 curriculum and developing a reporting tool that increases transparency and accountability in funding. At the province and district levels, the project supported strategic planning by coordinating with stakeholders from local government offices and councils. In addition, the project aimed to improve linkages among the schools, districts, provinces, TTIs and Indonesia’s national government. 2. EVALUATION PURPOSE & QUESTIONS 2.1 Evaluation Purpose The primary purpose of this final performance evaluation is to inform USAID, GOI counterparts, PRIORITAS’ partners and other donors about the results PRIORITAS has achieved at the school, district and provincial levels; the implications of those results on the quality and relevance of basic education in Indonesia; and the Goal: Expanded Access to Quality Basic Education IR 1.1 Strengthened Instruction in Schools Sub IR 1.1.1 More Effective Pre-Service Programs Sub IR 1.1.2 More Effective In-Service Programs IR 1.2 Improved Education Management and Governance Sub IR 1.2.1 Strengthened Capacity at the School Level Sub IR 1.2.2 More Effective District-based Management IR 1.3 Strengthened Coordination Between All Levels of GOI and Key Education Institutions Sub IR 1.3.1 Greater Capacity to Inform National Policy Sub IR 1.3.2 Greater Capacity to Build Linkages Sub IR 1.3.3 Greater Capacity for Staff Development Sub IR 1.3.4 Greater Capacity to Advocate for Education (Funding) USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 19 practices and lessons from PRIORITAS that relevant stakeholders can sustain to continue increasing access to quality basic education in Indonesia. 2.2 Evaluation Questions This evaluation sought to find answers to the following five research questions provided by USAID: 1. How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? How has PRIORITAS strengthened the institutions (Teacher Training Institutes) to support these teachers? 2. How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS improved in developing/supporting accountable and participatory school management? 3. How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management with focus on teacher deployment, teacher continuing professional development, and educational funding? 4. How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level? 5. How have national government agencies (MOEC, MORA, MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects? The evaluation team believes the detailed information in this report will demonstrate to USAID and other audiences the extent to which PRIORITAS interventions achieved the project’s goals; and how partners could use the experiences and lessons from PRIORITAS to improve their programs and policies to enhance basic education in Indonesia in the future. 3. EVALUATION METHODS & LIMITATIONS 3.1 Evaluation Process, Methodology & Data Resources MSI’s evaluation standard operating procedures (SOPs) were applied to the PRIORITAS evaluation to ensure that the evaluation team focused effectively and strategically from the start on key analytical issues. The SOPs define step-by-step timelines for the pre-planning work required to manage an effective evaluation process. The SOPs identified the actions to be completed in Indonesia prior to the evaluation team’s mobilization, including a background literature review, identification of key stakeholders and development of an evaluation schedule. The evaluation schedule included a list of focus activities to evaluate, budget allocations and a geographic profile of the areas to analyze. USAID/Indonesia’s Monitoring & Evaluation Support Project (MESP) organized and managed the services of an independent team to conduct the evaluation as defined in the statement of work (SOW). The evaluation began with the team planning meeting (TPM) process on February 27 at the MESP office in Jakarta. All evaluation members, relevant MESP staff and representatives of USAID’s program and technical offices participated. Representatives of the MOEC, MORA and Ministry of Research, Technology and Higher Education (MORHE) also attended sessions. The PRIORITAS evaluation team continued TPMs through the week, presenting final site selection and travel plans to USAID and ministry officials on March 6. TPM meetings covered best evaluation practices, such as: using MSI’s Getting to Answers Matrix as part of a structured process to develop the evaluation’s detailed methodology; the importance of findings, conclusions and recommendations matrices; the USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 20 need to conduct analysis on an ongoing basis; and a clarification of team roles and responsibilities. The evaluation team also identified both quantitative and qualitative methodologies to address the evaluation questions, including (but not limited to) document and literature review, analysis of existing PRIORITAS monitoring data, key informant interviews and site visits to schools, district education offices and provincial teacher training institutes. Following fieldwork, data were synthesized and analyzed using the framework of the Getting to Answers Matrix (see Annex I: Statement of Work) to answer the evaluation questions and sub-questions. Individual items within each data collection instrument were mapped against each of the five research questions guiding this evaluation. The evaluation team also developed detailed data analysis and report-writing plans (with draft report outlines) and shared these with USAID teams for initial comments. The following subsections describe key elements of the methodology. 3.1.1 Document and Literature Review Below is a non-exhaustive list of project documents reviewed by the evaluation team: 1. PRIORITAS Annual Progress Reports (2012 to 2016) 2. PRIORITAS contract and contract modifications 3. PRIORITAS Performance Monitoring Plan (PMP) or Monitoring & Evaluation Plan (October 2014) 4. Map of PRIORITAS activities and partners 5. Midterm Evaluation Report (2015) 6. Midterm Evaluation SOW 7. Monitoring reports on the assessment of the impact of the USAID PRIORITAS program on schools 8. DBE Impact Evaluation Report, including annexes (2012) 9. DBE Impact Evaluation SOW 10. Relevant studies conducted by PRIORITAS, including dissemination and sustainability study of DBE and PRIORITAS programs, teacher deployment reports, continuing professional development, gender study, lab and good practice schools, local government’s role and interests assessments, best practices books, impact on teacher training institutes, etc. 11. Additional information on the USAID PRIORITAS website: www.prioritaspendidikan.org. The evaluation team also collected documentation throughout the province, district and school-level fieldwork. A complete list of documentation reviewed for this final performance evaluation is in Annex VII: Sources of Information. 3.1.2 Collection and Analysis of Secondary Data The evaluation team extensively used any and all data available that was pertinent to the evaluation questions. Over its five years, PRIORITAS has conducted many performance measurement studies and collected a large amount of data related to activity objectives. These include:  Early Grade Reading Assessment (EGRA) and student assessments in Indonesian language, mathematics and science, conducted in a sample of partner and comparison primary schools.  Student assessment in Indonesian language, mathematics and science in a sample of partner and comparison junior-secondary schools. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 21  Surveys of teaching and management practices administered in a sample of partner and comparison primary and junior-secondary schools. Baseline data collection took place in 2012, followed by annual performance indicator data collection. The PRIORITAS M&E team plans to submit its final indicator data to USAID in 2017. In addition to the regular monitoring database, the project also collected basic statistics about partner schools using the Project Data Management System (PDMS), which also has district-level data on basic education statistics (e.g., rates of student enrollment, transition, dropouts). An analysis strategy linking secondary data to the primary evaluation questions was developed during the TPM. The quantitative analysis explored the performance trends for key indicators in each of the selected districts and schools in this evaluation. Analysis of project secondary data is aimed to support the qualitative findings. To analyze the project secondary data, the evaluation team applied two methods. First, the team calculated descriptive statistics using frequencies, means and variances to summarize general characteristics of the data, with a particular focus on indicators of teaching and management practices and student outcomes (from student tests in PRIORITAS M&E activities). The team then disaggregated these indicators by partner vs. non-partner districts, cohorts 1, 2 and 3, types of school (secular/madrasah) and respondent characteristics (gender and grade). Second, the team also conducted comparative analyses to identify significant differences between respondents by key school characteristics using contingency table analysis (cross-tabs). This analysis is a modification of baseline and midline analyses conducted by the PRIORITAS. In addition, the team performed multiple regression analysis to determine the degree to which teaching practices predicted variation in key student outcomes such as achievement in Indonesian language, mathematics and science. 3.1.3 Online Survey The purpose of the online survey was to gather information from the heads of Teacher Training Institutes (TTIs) and lecturers (TTI facilitators) on their satisfaction with the PRIORITAS program, and to hear their opinions of how their strategic plans will fare after the project ends (with a particular focus on continuation, replication and sustainability of PRIORITAS activities). The evaluation team targeted the key informants from 17 TTI partners to answer this survey. The team created the online survey using Survey Gizmo, and conducted pilot testing of the instrument prior to departure for fieldwork. Key informants had one month to complete the survey. The response rate was 55 percent (165 of 303). Each indicator was scored on a four-point Likert scale: strongly disagree (1); disagree (2); agree (3); and strongly agree (4); a “do not know” option was also available. The team calculated descriptive statistics such as item-level frequencies. 3.1.4 Key Informant Interviews The evaluation team conducted in-depth interviews or focus group discussions with key informants at central, provincial, district and school levels. The individuals and institutions listed below are identified as key stakeholders, and thus important informants. The provincial and district level sample, as well as criteria for school selection within each district, were finalized with USAID/Indonesia prior to fieldwork. The evaluation team developed semi-structured interview protocols for each key informant (see Annex V: Data Collection Instruments). The key criteria for province and district-level informants is previous involvement in and familiarity with the PRIORITAS program. The team was careful to maintain gender balance in FGDs with students and teachers.  USAID/Indonesia: Education and Program Offices personnel USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 22  Implementing partners: Research Triangle Institute (RTI), Education Development Center (EDC), and World Education (WE) teams, including: o Chief of party (COP), deputy COP, M&E staff, governance and management advisor & specialist, TTI program coordinator, teacher training coordinators for primary and junior-secondary, whole-school development program coordinator and GOI coordinator  Central level stakeholders potentially include: o MOEC, MORA and MORHE officials  Province-level informants (through interviews and online survey) include: o TTIs (heads, lecturers) o PRIORITAS provincial coordinators  District-level informants include: o PRIORITAS district coordinators o District facilitators o District Education Office o District Religious Affairs Office o School supervisor for junior-secondary school (SMP and MTs) o Teacher working group for junior secondary school (MGMP) o Civil service agency at sub-national level (Badan Kepegawaian Daerah) o Development planning agency at sub-national level (Badan Perencanaan Pembangunan Daerah) o Local parliament (Education Commission and Budget Commission) o Education council  Sub-District Level: o Head of the Regional Technical Service Unit (Indonesian acronym: UPTD) o School supervisor for primary school (SD and MI) o Teacher working group for primary school (KKG and KKM)  School Level (both TTI-supported and PRIORITAS-supported): o School principal o Teacher o Student o School committee o Classroom observations (two classes, Indonesian language, mathematics, science or social studies) USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 23 3.1.5 District, Province and School Selection The fieldwork included school visits, interviews and focus group discussions and direct classroom observations in selected districts in North Sumatra, East Java, Central Java and South Sulawesi. The team selected the provinces and districts it visited in collaboration with USAID. Evaluation fieldwork emphasized a qualitative approach to data collection, hence the team used a purposive sampling strategy. The evaluation team did not use random sample selection for data collection, but drew a sample of activity sites and partners that reflects the diversity of locations and institutions where PRIORITAS has worked. As this is a performance – not an impact – evaluation, the sample did not include comparison schools. However, the team was careful to include schools within DBE partner districts that received less￾intensive, indirect support from PRIORITAS. Key informants were also asked to compare professional development and other opportunities provided by PRIORITAS with existing government-sponsored efforts to better understand the relevance and added value of PRIORITAS activities. Provinces were selected purposively based on:  Geographic variation across Indonesia;  PRIORITAS cohort status (1, 2 or3); and  Presence of DBE partner districts. Districts were selected purposively based on:  Participation in the midterm evaluation;  MOUs with TTIs;  Presence of TTI-supported schools;  Participation in teacher deployment program; and  Geographic accessibility. Districts in the sample included:  North Sumatra: Medan, Labuhan Batu, Serdang Berdagai, Deli Serdang.  East Java: Surabaya, Malang, Kota Batu, Blitar, Lumajang.  South Sulawesi: Makassar, Maros, Pare-Pare, Pinrang,  Central Java: Semarang, Batang, Pekalongan. The evaluation team comprised eight individuals, and two teams of four performed site work. Between March 12 and April 2, 2017, team A visited North Sumatra and East Java, while team B visited Central Java and South Sulawesi. The final field movement plan is contained in Annex I: Statement of Work. The evaluation team finalized the list of schools to visit in each district prior to commencement of fieldwork, in consultation with the PRIORITAS project team. School selection was based on set criteria. Within each district, two to four schools were purposively selected for data collection. Criteria included:  Good practice or low-performing, as ranked by PRIORITAS project data;  Level (primary, secondary); USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 24  TTI-supported and PRIORITAS;  Type (general/secular, madrasah); and  Locality (urban, rural). The final school list and individuals interviewed per district (by position title to maintain confidentiality) is in Annex VII: Sources of Information. 3.2 Data Analysis The final evaluation employed both qualitative and quantitative approaches to answer the evaluation questions. These approaches included:  KIIs;  Classroom observations;  Document analysis;  Secondary data analysis of activity M&E data; and  Online survey of TTI-related stakeholders. The evaluation team validated qualitative findings by triangulating different data collection methods and inputs from key informants. To minimize bias and increase the response rate in the limited online survey, the team administered measures including instrument pre-testing among evaluation team members and follow-up calling during online data collection. The team conducted several types of data analysis based on the evaluation design. These included: 3.2.1 Document Review  Comparisons between the original project design, implementing partner (IP) agreements, Performance Monitoring Plan (PMP) and midterm evaluation findings to identify potential strengths, weaknesses and modifications of the program. The team compared these to the findings from the current evaluation.  PRIORITAS program reports and documents: annual and quarterly reports; technical briefs; midterm evaluation report; final reports on teacher deployment, PRIORITAS TTIs and others as available; project tools and standards.  Review of essential regulatory documents (national and regional level) to support evaluation findings and develop conclusions and recommendations. 3.2.2 Qualitative Data Analysis A framework for data analysis was developed during the TPM and refined throughout fieldwork (see the Getting to Answers Matrix in Annex I: Statement of Work). This framework specified the main topics to be analyzed, as organized by the evaluation questions. Data entry templates for all qualitative data collection activities were developed, ensuring standardization across the evaluation team. These templates and accompanying guidance focused on how to translate (from Indonesian to English) and summarize interview notes and assure that the field interviews covered all pertinent points at the end of each week of fieldwork. The two data collection teams scheduled weekly calls to check on challenges or discuss other issues related to data collection activities and ensure a standardized response across the evaluation team. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 25 Quantitative Data Analysis The team’s quantitative data analysis included the following steps:  Comparison of district and school level data at the start, midline and end of PRIORITAS.  Descriptive statistics on classroom observation data collected during fieldwork.  Regression analysis using PRIORITAS monitoring data predicting variation in student achievement in key subjects (targeted by the PRIORITAS program) and teaching practices.  Descriptive analysis of data from the evaluation online survey of TTIs staff.  Analysis of PRIORITAS progress toward indicators from the project’s reports and PMP. Finally, wherever possible, the evaluation team triangulated or validated the qualitative findings using methods such as interviews with key informants. 3.3 Limitations of the Evaluation The evaluation team identified the following limitations to the evaluation SOW and implementation:  Small number of districts and schools sampled, compared to the large number of project locations and partners.  A primarily qualitative approach, which might not enable the evaluation team to calculate the net effects of PRIORITAS impacts, but would provide greater insights into why the project worked or did not work in given situations.  Presence of evaluators during the administration of classroom observations, introducing bias (Hawthorne effect).  Unavailability of all invited senior decision-makers from MOEC, MORA and MORHE to serve as key informants during the evaluation. The following is a brief description of two of the limitations noted above: the Hawthorne effect and the presence of other donor-funded basic education projects in PRIORITAS locations. 3.3.1 Presence of Evaluators during Classroom Observation The presence of evaluators generated some Hawthorne bias effects at PRIORITAS schools. These appeared to have been caused by possible misunderstandings on the part of supervisors, principals, teachers, students and members of school committees regarding the purpose of the evaluation, and the preparations to be made for demonstrating positive results during classroom lesson observations. The observed Hawthorne effects were:  Evaluators sometimes felt they were observing rehearsed demonstrations of active teaching and learning. The teacher might then reveal that this wasn’t done for every lesson, or on every day (this was later confirmed by FGD informants). Evaluators requested that PRIORITAS staff not take them only to PRIORITAS “good-practice schools.” The PRIORITAS team noted this request and attempted to address it by identifying additional schools for the evaluation team to visit.  In response to this effect, evaluators also devised better explanations (e.g., reassuring teachers this was not an assessment of their performance) and some methods to minimize the Hawthorne bias caused by their presence, including returning to the same classrooms after lesson observations had ended, and observing other classrooms from outside. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 26 3.3.2 Presence of Previous or Similar Donor Programs in the Study Area Informants at some PRIORITAS partner districts and schools in East Java, Central Java and South Sulawesi reported that they had been influenced by MOEC programs, and by other international donor education programs, as follows:  MOEC curriculums and policies that promoted active teaching and learning, literacy and school-based management and governance had been implemented in PRIORITAS partner districts and schools before PRIORITAS began in 2012. It appeared to evaluators that this may have occurred with varying degrees of commitment and success among individual schools.  Informants reported the previous or existing presence in PRIORITAS districts of other donor programs that promoted active teaching and learning or literacy and school-based management and governance. Donor programs specifically reported by informants were the Asian Development Bank Madrasah Education Development Project (MEDP, 2007–2012); Australia’s Education Partnership with Indonesia – the AusAID Learning Assistance Program for Islamic Schools (LAPIS, 2005-2010) and the Department of Foreign Affairs and Trade (DFAT) Professional Development for Education Personnel Program (ProDEP, 2014-2019), Japan International Cooperation Agency Program for Enhancing Quality of Junior Secondary Education (2009-2013), and the World Bank Better Education through Reformed Management and Universal Teacher Upgrading Project (BERMUTU, 2007–2013). The evaluation team acknowledges the presence and possible influences of these other projects and notes that the activities, results, and outcomes from PRIORITAS stand as described in this evaluation report, in previous evaluations, and in the project’s own progress reports. 4. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 Competency and Skills of Pre-Service & In-Service Teachers and TTIs Research Question 1: How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? How has PRIORITAS strengthened the institutions (TTIs) to support these teachers? 4.1.1 Pre-Service and In-Service Teachers PRIORITAS worked at the school and TTI levels to strengthen instruction in schools. The project developed training modules and materials for both in-service and pre-service teacher training delivery through the WSD approach. The project developed WSD training modules based on the training materials from similar former projects, such as DBE and MBE. PRIORITAS staff developed the modules through consultation with specialists, TTI partners and government stakeholders before testing and finalizing them. From2012 to 2015, three sets of modules on WSD were developed, covering teaching and learning management and methodologies, school management and community participation. The project placed a special focus on reading and literacy outcomes for students. These modules were delivered in project partner schools and TTI lab schools. They were also adapted for use in TTI pre-service programs and made available on request as resources for in-service training. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 27 4.1.1.1 Findings A. Active Teaching and Learning PRIORITAS encapsulated USAID’s programmatic focus in Indonesia of improving the quality of teaching and learning in basic education through active learning methods. School-level stakeholders stated that Active, Creative, Effective and Joyful Learning, known by the Indonesian acronym PAKEM, has existed since 2007, in accordance with the ministerial regulation (Peraturan Menteri Pendidikan Nasional No.41/2007). However, until the PRIORITAS project, many teachers struggled with consistent implementation of the PAKEM method across subjects such as Indonesian language, mathematics and science. Teachers stated that PRIORITAS training helped them implement lesson plans systematically. Using active learning as the foundation for PRIORITAS training events was much appreciated by the majority of these informants. For example, more than 90 percent of school committee members and about 95 percent of teachers and school principals interviewed by the evaluation team supported the PRIORITAS program and reported positive changes in teachers’ abilities to deliver active learning methodology. Respondents noted that learning materials were given to school principals, teachers, school committees, supervisors and others so that they were able to see the learning process from the perspectives of students and teachers, and to create a learning environment that supported active learning. According to school committee members, school principals and teachers interviewed for this evaluation, such positive changes in teachers’ capacities to implement active learning methods occurred, at least in part, due to PRIORITAS, as illustrated by this KII statement: “There is a shifting paradigm among teachers after their teaching method training. [Previously] teachers would believe that they are the source of knowledge in the class. It put pressure on them, especially if they were seen as a superman who knew all the answers in the classroom. Nowadays, teachers see themselves as facilitators for students to reach their maximum ability.” School supervisors, heads of Regional Technical Service Units (UPTD) and district facilitators stated a noticeable change occurred in the teaching and learning process in the classroom after the teachers participated in the PRIORITAS training and received additional mentoring and guidance from the program. Teachers and students developed increasing familiarity and comfort level with active learning methodologies. Teachers’ experience in the classroom shifted from that of a speaker to that of a facilitator of student learning. Teachers also used a variety of tools and props to help students achieve a better understanding of the subject matter. Teachers encouraged students to find their own way in solving a problem. As illustrated by the quotes below, teachers, school directors, supervisors and students noted that active learning had made teaching and learning more joyful experiences. “After the program was introduced, the teaching method became more varied; for example, teaching and learning held in the school yard. Every Saturday, teachers and the school principal conduct teaching￾learning evaluation for the week. Teachers also discussed their lesson plans for the next week. I think the quality of education should not only be measured from students’ scores, but also from the process.”– School supervisor, Central Java “Teachers deliver the topics with lots of practice or experiments. Now, learning is more joyful, as students have the opportunity to practice, to work on something, or to discuss.” – Secondary school student, Central Java Students interviewed in all schools stated that substantial changes in teaching and learning practices have occurred since PRIORITAS began. Atmospheres in partner schools are different now, as indicated by various changes. For example, classrooms have been transformed into group-seating arrangements; students’ work is USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 28 displayed in the classroom; students are more engaged in class activities, group discussions and experiments (participatory learning); teachers and students use teaching aids more frequently; more evidence points to a stronger reading culture, including reading corners, silent and guided reading, resume-writing activities and use of books donated by USAID; and teachers are using reflection sessions at the end of lessons. These findings are supported by the results of the online survey of TTI personnel, in which respondents stated that active teaching and learning is the most successful PRIORITAS program, followed by mentoring and literacy. For this evaluation, the team conducted 89 classroom lesson observations across 48 schools (primary and junior secondary schools: combination of state and private and of regular and islamic schools); and PRIORITAS and TTI-supported schools), to gather real-time data on teachers’ abilities to implement active learning methodologies and better understand actual teaching and learning practices. Nearly all of the teachers (86 of 89) observed by the evaluation team had completed Modules 1, 2 and 3 of the PRIORITAS training package. Even though lesson observations were also designed to capture reading lessons, the evaluation team found only six reading lessons during its observations. Table 1 shows the findings about classroom observations. Classroom observations revealed that on average, more than 80 percent of teachers had implemented active teaching (e.g., using varied methods of interaction and engagement with students, and moving around the classroom during the lesson). Further, evaluators saw student involvement and engagement in class activities in 86 percent of lessons they observed (e.g., conducting observations or experiments during group projects). However, only 54 percent of teachers were observed using tools – such as running records or observation reports – to measure student achievement during the classes. In general, these lesson observation results confirm the trends demonstrated in PRIORITAS monitoring data, which showed more teachers delivering active learning methodologies in 2016 compared to 2014 (see Table 2). Regarding students’ learning behavior, project data showed the number of students showing positive learning behavior increased between 2014 and 2016 (see Table 3). In addition, project data as well as evaluation team observations showed that in Grades 4, 5, 6 and 8, the most commonly demonstrated aspect of active teaching being implemented was organizing classrooms to facilitate interactive learning. Examples included displaying students’ work and using teaching aids effectively. Teachers were observed conducting a combination of whole class, group and individual teaching during the classes, and engaging in interactive dialogues with students. Students also expressed their feelings and opinions during observed classes. These findings show that students were engaged in their tasks and producing work independently. TABLE 1: PERCENTAGE OF TEACHERS AND STUDENTS IMPLEMENTING ACTIVE LEARNING IN CLASSROOMS Indicator Sample (N) # observed % observed Teacher organizes physical classroom to facilitate interactive learning 89 74 83% Teacher uses a mix of whole class/group/ partner and individual work 89 81 91% Teacher asks non-recall questions and allows students time to answer 82 53 65% Teacher uses varied learning approaches 89 74 83% Teacher uses tools to gather data about student achievement 79 43 54% Teacher moves around the room, observing and assisting students 89 80 90% Students are engaged in their task 88 71 81% Students undertake activities that involve problem solving 89 79 89% Students’ work is the result of their own thinking 89 74 83% Students express their feelings and opinions 86 55 64% Students participate in cooperative activities, e.g., experiments/discussion 88 76 86% Source: PRIORITAS Final Evaluation Data, 2017 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 29 TABLE 2: PERCENTAGE OF TEACHERS IMPLEMENTING ACTIVE LEARNING METHODS Indicator Academic Year 2014- 2015 2015- 2016 2016- 2017 Teacher organizes the classroom to facilitate interactive learning 82% 88% 87% Teacher uses mix of learning approaches: open-ended tasks 70% 78% 79% Teacher uses mix of learning approaches: environment 31% 40% 43% Teacher uses mix of learning approaches: learning aids 65% 72% 73% Teacher works with groups of students 73% 80% 77% Teacher works with individual student 65% 77% 78% Teacher asks non-recall questions 66% 73% 75% Teacher allows students time to answer 58% 67% 69% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 The evaluation team’s classroom observations revealed a gender-balanced and individualized approach to teaching and learning. Even though female students outnumbered male students in the classrooms observed, teachers gave equal opportunities to all students. Teachers asked non-recall questions alternately between female and male students, and gave time for each to answer. Teachers also provided individual support to students who needed help, again without bias to student gender. Most observed classrooms have implemented group-seating arrangements. In some cases, groups were selected randomly by the teachers, while in others the teachers selected group members purposively. The most common consideration observed was to distribute students equally by gender. Another approach observed was to mix students with different characteristics across groups (i.e., stronger and weaker students; active and passive students). This latter approach was aimed to encourage students’ collaboration, to increase their empathy through peer teaching or tutor sebaya and to motivate students to compete in fair competition. TABLE 3: PERCENTAGE OF STUDENTS DEMONSTRATING POSITIVE LEARNING BEHAVIORS Indicator Academic Year 2014-2015 2015-2016 2016-2017 The majority of studentsare engaged in their tasks 81% 87% 87% Students undertake activities that involve problem solving 72% 78% 78% Students’ works are the results of their own thinking 75% 81% 81% Student express their feelings/opinionsduring lessons 55% 68% 69% Students are active in group discussion 71% 79% 76% Students are active in conducting experiments 24% 32% 35% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 Further, school residents reported that lessons taught using active learning methodologies were more easily absorbed by students compared to lessons taught using conventional methods. All interviewed teachers, school principals, and students reported that students who experienced active learning had become more courageous in expressing their ideas, and were more confident speaking in front of their class. Students also reported that they now had closer relationships with their teachers. While these informants were generally unable to demonstrate a clear link between improvements in student achievement (for example, higher test scores for national examinations) and active learning techniques, all USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 30 school-level informants interviewed believed that active learning methods had increased students’ motivation, and thus their ability to learn. These findings replicate those of the PRIORITAS midterm evaluation. In that study, it was reported that “teachers were almost universally positive about the active learning methodology promoted by PRIORITAS, even in cases where individuals were familiar with these methods from many years before. Those who were willing to adapt and adopt active learning methods seemed to have not only renewed their own interest in teaching but stimulated increased interest among students to learn.” PRIORITAS monitoring data indicate improvements in student performance in mathematics, science, and Indonesian language across years Tables 4A and 4B, and 5A and 5B show the average percentage scores of students (against the maximum possible score) on tests conducted by PRIORITAS, presented by gender and type of schools. In general, students in partner schools scored better than students in comparison schools. In partner schools, primary student scores on math and science tests increased by 12 percent and 13 percent, respectively, while secondary student scores increased by 20 percent and 7 percent, respectively, from 2013 to 2016. For Indonesian language, students in all partner schools performed better in 2016 than 2013, with the greatest improvements in reading scores. By gender, female students scored higher than male students in partner schools. These differences were most marked in reading and writing, where females performed considerably better than males. In mathematics and science, the differences were relatively small. The evaluation team conducted correlation and regression tests of teaching practices variables and student performance using project monitoring data. The results of the Spearman correlation test showed that the combination of good practices in teaching and student assessments were significantly associated with higher student performances in mathematics, science and Indonesian language. The evaluation team also ran a regression analysis of indicator data on teaching practices and students’ performance to identify any causality. Even though the team found a correlation between these variables, the causality effect between teaching practices and student performances was low, as indicated by the low variance in the model. The evaluation team believes this low variance most likely occurred because many factors often strongly associated with student performance are not available in the project’s monitoring data, for example: family factors, such as parenting practices;1 teacher experience, educational background and characteristics;2 and school-level environment, such as school facilities.3 1 Considine, G. and Zapala G, 2002.The Influence of Social and Economic Disadvantage in the Academic Performance of School Students in Australia. Journal of Sociology.38(2), pp.129-148., and Xia, N. 2010.Family Factors and Student Outcomes. Dissertation. Pardee RAND Graduate School. United States of America. Available at http://www.rand.org/content/dam/rand/pubs/rgs_dissertations/2010/RDND_RGSD256.pdf. 2 An, X., Hannum, E.C., and Sargent, T., 2008. Teaching Quality and Student Outcomes: Academic Achievement and Educational Engagement in Rural Northwest China. Gansu Survey of Children and Family Papers.5(2), pp.309-334., Kyriakides, L., Creemers, B.P.M., and Antoniou, P., 2009. Teacher Behavior and Student Outcomes: Suggestions for Research on Teacher Training and Professional Development. Teaching and Teacher Education.25, pp.12-23. 3 Leithwood, K. and Jantzi, D., 2000. The Effect of Transformational Leadership on Organizational Conditions and Student Engagement with School. Journal of Education Administration.38(2), pp.112-129., and Webster, B.J. and Fisher, D.J., 2003. School-level Environment and Student Outcomes in Mathematics. Learning Environment Research. 6(3), pp.309 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 31 TABLE 4: SUMMARY OF PRIORITAS TEST RESULTS IN PRIMARY SCHOOLS (BY TYPE OF SCHOOLS) Academic Year By Comparison (C) and Partner (P) School Mathematics: Grade 4 Science: Grade 5 Reading: Grade 4 Writing: Grade 4 C P C P C P C P 2012-2013 40% 41% 33% 36% 41% 43% 40% 42% 2013-2014 37% 39% 33% 34% 37% 37% 34% 39% 2014-2015 43% 45% 40% 42% 44% 46% 40% 42% 2015-2016 42% 47% 40% 42% 49% 53% 40% 47% 2016-2017 48% 53% 41% 49% 49% 56% 40% 49% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 TABLE 5: SUMMARY OF PRIORITAS TEST RESULTS IN PRIMARY SCHOOLS (BY GENDER) Academic Year By Gender Mathematics: Grade 4 Science: Grade 5 Reading: Grade 4 Writing: Grade 4 M F M F M F M F 2012-2013 40% 41% 34% 35% 40% 44% 37% 44% 2013-2014 36% 40% 33% 34% 34% 39% 32% 40% 2014-2015 43% 46% 41% 42% 42% 48% 37% 46% 2015-2016 43% 47% 39% 42% 47% 55% 38% 49% 2016-2017 49% 52% 45% 47% 50% 56% 40% 49% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 While the project achieved significant successes, the consistent application of PAKEM methods by teachers in PRIORITAS partner schools often faced constraints. For example, informants reported that implementing PAKEM methods required time and funds to produce or acquire teaching aids and learning materials. Many school directors and teachers reported facing financial constraints or had limited experience with creative design to purchase or develop teaching aids or media. In addition, school principals and school supervisors reported that active teaching was sometimes difficult for teachers to do, especially older teachers, who were in some cases resistant to changing their teaching styles. Such teachers may have continued to use lecturing methods because it was easier for them, or because they had memorized their teaching materials after using them for many years. While the PRIORITAS approach did not require a technology component in active learning; however, some teachers felt the need for additional support for technology use in order to enrich their delivery of active learning methods – for example, the use of computers, laptops, and projectors in teaching. Technology was often used in partner schools to expand and enrich lessons, and to conduct research on class topics for or by students. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 32 TABLE 6: SUMMARY OF PRIORITAS TEST RESULTS IN SECONDARY SCHOOLS (BY TYPE OF SCHOOLS) Academic Year By Comparison (C) and Partner (P) school Mathematics: Grade 8 Science: Grade 8 Reading: Grade 8 Writing: Grade 8 C P C P C P C P 2012-2013 27% 29% 40% 41% 66% 64% 47% 50% 2013-2014 32% 34% 36% 39% 64% 66% 43% 50% 2014-2015 34% 37% 43% 44% 68% 70% 47% 52% 2015-2016 35% 38% 41% 43% 69% 70% 47% 49% 2016-2017 43% 49% 43% 48% 70% 73% 52% 59% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 TABLE 7: SUMMARY OF PRIORITAS TEST RESULTS IN SECONDARY SCHOOLS (BY GENDER) Academic Year By Gender Mathematics: Grade 8 Science: Grade 8 Reading: Grade 8 Writing: Grade 8 M F M F M F M F 2012-2013 33% 25% 41% 40% 63% 67% 44% 53% 2013-2014 32% 34% 38% 37% 62% 67% 41% 51% 2014-2015 34% 36% 44% 43% 66% 71% 45% 53% 2015-2016 37% 36% 42% 42% 67% 71% 42% 53% 2016-2017 47% 46% 44% 46% 69% 74% 49% 61% Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 Overall, 80 percent of students interviewed said that active learning has operated well in their classrooms, per the reasons and responses described above. The 20 percent who stated that active learning does not always run smoothly described the following challenges: some teachers ask students to engage in group discussions but do not provide explanations or guidance to the students; teachers are unable or unprepared to innovate, especially older teachers and teachers of non-core subjects such as religious subjects or citizenship; classrooms are excessively noisy during group discussions; or “free-rider” students do not share in the work or they cheat. Another challenge teachers said they faced is the difficulty to manage aggressive students during class activities, especially male students. In one school, several female teachers experienced verbal harassment and impolite behavior from students while teaching. According to the teachers’ statements, sometimes students did not understand the essence of active learning. From the students’ perspectives, active learning means more freedom in the classroom. Therefore, teachers need support to handle such students and to strengthen the character education of students. Both school principals and teachers told the evaluation team that local values must be considered in implementing active learning methods. Two Islamic schools that the evaluation team visited had an issue regarding gender at the beginning of PRIORITAS implementation. The schools felt that female and male students should be separated while sitting. Some teachers and parents complained about the group-seating arrangement that was usually adapted in active learning. This issue was resolved after school principals invited parents to meet and explained to them about active learning, where the methods implemented by teachers were intended to improve learning process. The principals reassured the parents that the schools would still respect the religious and local values in the school and the community. PRIORITAS also noted these issues in the Rapid Study it conducted in 2016. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 33 In addition to training principals, teachers and school committee members, PRIORITAS interventions included training for school supervisors about improving student learning, strengthening school management, and partnerships between schools and their communities. It was expected that with this knowledge the supervisors would perform their support functions better, and focus their efforts on advancing learning processes in schools. Supervisors were also expected to understand the indicators of learning performance, reading cultures and school management. During training, supervisors were also taught how to make informed observations in schools to quickly see what progress had been made. By knowing the strengths and weaknesses of the schools they supervise, it was expected that supervisors could then develop better programs to help their schools become more advanced. In addition, if supervisors did not personally possess the technical skills required to help their schools, they were meant to identify other useful resources. Apart from interviewing school supervisors, the evaluation team conducted interviews with heads of UPTDs of Dinas Pendidikan, and with District Facilitators. In total, 38 supervisors, eight heads of UPTD and 29 district facilitators were interviewed. These informants consistently reported that the training provided by PRIORITAS was excellent and differed from previous training programs to which teachers had been exposed. For example, training participants not only listened to the lectures and used materials presented by trainers, but also actively participated in discussions and practiced applying lesson content together. Consistent with midterm evaluation findings, the final evaluation team found that 85 percent of supervisors and teachers noted that those “who were well-supported by strong leadership of principals were most likely to make an effort to change their teaching methods by adopting the active learning methods they had learned during PRIORITAS training.” After PRIORITAS training events for teachers, participants received ongoing assistance from local facilitators who visited their schools. As noted during the midterm evaluation of PRIORITAS, “mentoring after training was generally noted by teachers as an effective practice and the one that distinguished PRIORITAS active learning training from other previous PAKEM efforts.” The current evaluation also found that PRIORITAS training was a helpful resource for teachers as they began the practical implementation of active learning methodologies and faced day-to-day challenges that may not have been addressed during the training. The following quote illustrates this: “Training topics provided by PRIORITAS had actually already been covered in previous training by the Office of Education, such as CBSA (Active Students Learning Method), but the training provided by the Office of Education was not accompanied by teaching practice in the classroom or by supervision by facilitators, as is done in PRIORITAS programs. The strength of PRIORITAS training is that there is lots of practicing, plus mentoring by the facilitators after the training.” – Primary school supervisor, South Sulawesi When PRIORITAS began, mentoring was arranged for up to 20 visits per project district facilitator. At that time, the mentoring was one-on-one. Facilitators would observe teachers in their classrooms and provide feedback to improve their practices. The content of the mentoring was based on what had been trained in the modules. Halfway through the project, teachers became more advanced. Even though one-on-one mentoring was still possible, it was not recommended due to the high cost. As a result, PRIORITAS shifted to mentoring through KKG/M or MGMP as forums to share knowledge and learn through lesson study approach. Through this approach, one or two of the teachers could teach the lesson based on the collectively developed lesson plan, while the others observed and provided feedback to improve the lesson. This step was also taken by PRIORITAS to enable schools to sustain approaches after the project ends (Quarterly Report, 2017).School principals, supervisors and teachers interviewed by the evaluation team revealed that mentoring and disseminations have taken place regularly through KKG/M or MGMP, at least monthly. Despite the requirement of KKG/M or MGMP monthly meetings, as stated in the ministerial regulation (Peraturan Menteri Agama No.60/2015) and technical guidance of teacher working groups, some schools – especially madrasah and rural schools – conducted these meetings less frequently. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 34 The project has supported the dissemination of good practices in both PRIORITAS and former DBE districts. This activity involved district-level management programs and school-level training (i.e., school-based management, teaching and learning methods) to non-partner districts and non-partner schools. Between 2013 and 2016, dissemination activities covered more than 171,000 education professionals benefitting over 6.9 million additional students. These were jointly funded activities between PRIORITAS, local government and schools themselves (PRIORITAS Annual Report, 2016). It was recorded that more than US$6.1 million was allocated by local governments to disseminate the project’s best practices. In addition, schools and district governments initiated their own dissemination activities through teacher working groups and district-level training. However, in some instances, those dissemination activities did not cover all teachers or did not include all necessary training materials. All school directors and district-level education officials interviewed for this evaluation reiterated that dissemination of the PRIORITAS modules was dependent on the enthusiasm and support of school leadership and district-level officials, and on the availability of financial resources such as the district budget (known by the Indonesian acronym APBD) and the Government-Allocated School Operational Assistance Fund (BOS). The evaluation team found that older teachers and teachers from non-core subjects rarely used active teaching methods, in large part because the PRIORITAS training focused on three core subjects. Further, the majority of teachers interviewed in this evaluation requested further training materials as well as guidance on using active learning methodologies to support them and their peers in adapting active learning methodologies for non-core subjects. B. Supporting Reading and Literacy One of the goals of PRIORITAS was to improve the reading abilities of students and to develop a strong reading culture in schools. The project directly aligned with the MOEC’s subsequent School Literacy Movement (Indonesian acronym: GLS). In addition to increasing students’ interest in reading and reading skills, the literacy movement is expected to improve the morality of students, as stated in Ministerial Regulation No. 23/2015). PRIORITAS provided reading books to help teachers apply what they learn in project trainings. In 2014, a partnership between PRIORITAS and Yayasan Literasi Anak Indonesia (YLAI) was implemented to produce leveled reading books for use in early grade classrooms across Indonesia. The books consist of 75 titles, 6 big books, and 6 teacher guides. This support included reviewing and revising the books developed by YLAI, introducing the books during trainings and distributing the books to schools. In 2016, PRIORITAS provided both partner and non-partner schools with leveled reading books, training on how to use the books and mentoring by facilitators to improve the ability of teachers to use the books. More than 8 million reading books were distributed to more than 13,000 schools. The evaluation team identified the important roles that principals and supervisors have in managing the inculcation of a reading culture in schools. Principals play a role in planning, preparation and provision of facilities and human resources, and developing and monitoring the program in their schools. Supervisors coordinate the needs of schools with the district education office and catalyze dissemination of good practice among their schools. This situation is illustrated by a quote from a school supervisor in Central Java: “This school became the first winner of the library competition in 2016. It was the school's success in managing the school library to become an attractive place to read. Regarding the reading program, there is a schedule to read in the library for each class. The school is also working with the regional library to lend books to schools within a certain timeframe. If all the books have been read, the school will call the regional library to collect the books and provide other books to the school library. As for increasing students' interest in reading, it is still in process. We have to encourage students to want to read. The school seeks to provide books that can attract students to read. The library display makes students feel at home in the library. Some teachers use the library as student’s place of learning.” USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 35 School residents interviewed by evaluators reported that the culture of reading that PRIORITAS introduced generally works well in partner schools. Most schools have sought to create a good environment for reading by increasing the use of school libraries, creating reading corners or reading parks and implementing a timetabled reading schedule (i.e., guided reading and silent reading). Each school also familiarizes students with reading by specifying times just for reading books, usually 15 minutes before the class starts. Some schools have been running special programs for students who are slow readers, improving their literacy by integrating reading into other learning activities. At least seven schools regularly monitor the progress of students’ reading skills through the use of reading journals. Students enter all reading-related activities into their journals, which are then given to teachers and signed by teachers or parents. Two of the schools visited had a reading program for parents. Parents were encouraged to read books provided especially for them in the reading parks. Parents with the most frequent reading activity would be rewarded. School principals stated that this program is aimed to establish parents as reading role models for their children. Besides leveled reading books from PRIORITAS, parents also provided books through donations. Parents, students and teachers worked together to create reading corners and furnish them. Supervisors, heads of UPTD and district facilitators, however, also stated that the implementation of the reading culture activities did not always proceed smoothly. Five of 48 partner schools (10 percent) visited by the evaluation team had no reading corners inside or outside classrooms, other than in libraries. In eight schools visited (17 percent), reading corners had no storybooks, only school textbooks, or they lacked books that were appropriate for the ages and grades of students attending that school. Evaluators also observed that in at least two instances, school libraries did not have enough storybooks or knowledge books. And three visited schools kept the books provided by PRIORITAS in boxes because principals and teachers were afraid students would damage the books. Large majorities of students met by the evaluation team said the reading program has operated well (80 percent) and that reading corners were available (90 percent). About 20 percent of the students interviewed said the reading program still not running well. For example, in one school in East Java, reading activities were not regularly scheduled. About 10 percent of students interviewed at schools in South Sulawesi and Central Java said that they had no reading corners in their classrooms. This due either to limited classroom space to create reading corner or lack of supports from parental association or school committee. The creation of a reading culture in schools by PRIORITAS was one of the most frequently cited changes directly attributed to the project. The following quotation illustrates what the evaluation team found in most schools the team visited: “Our class has a reading corner, tree of knowledge and silent reading activity. At first, we did not realize that it was encouraging us to read books. We only did it because we were told to by our school, but we feel the amount of time for us to read on each day is increasing.” – Primary school student, North Sumatra PRIORITAS has documented evidence of the success of its reading program, including a 2015 expression of support by MOEC to institute 15-minute reading sessions before classes start every day to encourage the reading culture. At the sub-national level, MOEC has designated 19 partner districts with good examples of literacy programs to serve as national models of district literacy. MOEC gave acertificate of appreciation to PRIORITAS and the partner districts in March 2017. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 36 Best Practices: Literacy Program A secondary school in Lumajang District holds a regular literacy program every Monday, Friday and Saturday called “Morning Reading Mania,” or MORENA. The school has made innovations to increase students’ interest and motivation, integrating reading, writing and listening activities into the program. On Mondays and Saturdays, students read any books they want and write summaries. On Fridays, students summarize books that contain messages of wisdom. Students are also encouraged to display their summaries on the wall magazine and present them to their class. Students with the best performances are appointed as Duta Baca (ambassador of reading). Each month, a different ambassador is chosen from each grade. Students feel honored to be chosen: “We feel the pride in being the ambassador of reading because this is the highest appreciation from the school,” said one student. 4.1.1.2 Conclusions 1. From the qualitative findings, it is clear that the training provided by PRIORITAS has helped teachers understand how to apply PAKEM methods. The unique training package, most importantly for lesson plan development, classroom management, and regular mentoring by facilitators has enhanced teachers’ capacities to deliver active teaching and learning in their classrooms. Together with these successes, challenges remain in sustaining PAKEM approaches after PRIORITAS ends. Teachers will require more support and mentoring from supervisors to maintain quality active teaching methods. In addition, enrichments in teaching methods and materials (e.g., innovative and low-cost teaching aids; issues related to managing aggressive students or students who misbehave) are needed to support teachers in implementing continuous innovations in active teaching. 2. From the project’s secondary data, it can be concluded that the active learning methodologies promoted by PRIORITAS have enhanced student performance in mathematics, Indonesian language and science. Test results for Grades 4, 5 and 8 showed student performance improvements in those subjects from 2013–2016. There was a statistically meaningful – albeit small – association between teaching practices and students’ learning outcomes. 3. The evaluation team concludes that the PRIORITAS reading and literacy program has been successful and implemented well, even as the degree to which the program operates in each school varies. Reading corners were available in most schools, and some schools even built reading parks. Further, MOEC has adapted the best practices of the PRIORITAS reading program across Indonesia through the implementation of a 15-minute reading session every day at schools. Some schools were able to implement both reading and writing activities together and to monitor students reading skills with students’ reading journals, while some schools enriched their reading corners through contributions from the community and student art work. However, other schools visited in this evaluation faced obstacles in implementing the reading program, mostly due to the limited availability of books or lack of physical space set aside for reading corners. 4.1.1.3 Recommendations The following recommendations relate to the competency and skills of pre-service and in-service teachers: 1. School supervisors and school principals should routinely mentor and supervise teachers to maintain the quality of active learning. In addition, schools should continuously partner with TTIs to ensure that mentoring will take place after project ends. Continued mentoring on the PRIORITAS package of trainings in active learning methodologies is needed to ensure that teachers receive the technical support they need to implement this model effectively. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 37 2. School principals should regularly discuss issues related to classroom management and student behavior in check-in meetings with their teachers, to help create a culture where members of the teaching staff feel comfortable to share challenges they are experiencing and to brainstorm solutions. Such issues should also be raised with the school management committee, to ensure that parents are aware of behavioral issues with their children, and so constructive solutions can be co-created. 3. Principals and teachers within a school or a region should take the initiative to arrange trainings for themselves, without waiting for national or local government support. These educators could jointly fund their own training using part of their professional incentives, and could invite former PRIORITAS facilitators to deliver the trainings. 4. District governments or other agencies should conduct refresher trainings for teachers to update their competencies in teaching and to accommodate the most current educational issues. This support will increase teachers’ capacities to address differing individual student needs, including issues of character and gender equality. 5. Teachers, supported by school supervisors, should continue to strengthen the roles of teacher working groups as a sharing forum to innovate and disseminate PRIORITAS approaches. Particular attention should be paid to areas where teachers faced challenges. For example, targeted sessions could be held for older teachers or non-core subject teachers addressing challenges these teachers face in implementing active learning methodologies. 6. District governments should standardize dissemination activities, including materials and training delivery mechanisms. Toward this end, local governments could recruit PRIORITAS-trained facilitators and experts to follow up and advocate for best practice dissemination activities that could enhance the sustainability of PRIORITAS. Such an approach will have implications for quality assurance of dissemination activities. 7. Principals and teachers should continue the reading practices that PRIORITAS has developed. The evaluation team recommends continued school partnerships with district libraries, mobile libraries, and other reading programs. Innovation programs – such as rewards for students with the best performance in literacy-related tasks, or the combination of reading and writing programs – should be maintained to enhance students’ motivation to read. 4.1.2 Teacher Training Institutes 4.1.2.1 Findings A primary PRIORITAS objective was to build the capacity of TTIs to provide improved, high-quality training programs for pre-service and in-service teachers. The project achieved this objective by developing the knowledge and skills of teacher educators to use student-centered and innovative training methodologies, and by increasing their access to and use of training curricula, resources and facilities. To improve the quality of pre￾service training, PRIORITAS worked with 17 partner TTIs in nine provinces (two to three per province), comprising a combination of state universities and Islamic institutions, as well as one TTI in West Papua Province. Direct beneficiaries included lecturers and students in partner TTIs and in-service trainees. PRIORITAS also worked with 31 TTIs through consortia in partner provinces. PRIORITAS signed MOUs with each TTI to accommodate the uniqueness of each collaboration. PRIORITAS partnerships with TTIs had four main activity areas: strengthening class-based pre-service training, strengthening practical pre-service training, developing the role of TTIs as service providers and disseminating good practices within partner institutions. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 38 A. Strengthening Classroom-Based Pre-Service Training To carry out its program of strengthening classroom-based pre-service training in partner TTIs, PRIORITAS conducted four main activities: 1. Adapted training modules for pre-service training. Over four years of implementation, PRIORITAS developed three sets of modules for use in teacher and education personnel training. These modules addressed topics of active learning and school-based management (SBM). Although the modules were originally designed for in-service training of teachers in clusters and working groups, the content was regarded as suitable to be incorporated into TTI pre-service courses. PRIORITAS trained more than 1,800 lecturers from TTIs in the good practices of teaching and learning, and in SBM (Annual Report, 2016).The evaluation online survey showed that 98 percent of TTI respondents were impressed with the benefits provided by PRIORITAS. One TTI director stated that for lecturers working in the TTI, attending PRIORITAS training yielded two benefits – innovation in how individual teachers learned their trade and improvement in the institute’s curriculum. 2. Provided resource materials to support 2013curriculum based on LPTK and GOI directives. The project developed modules in seven subjects: (1) Early Grades Reading and Literacy, (2) Upper Grades Literacy (5-6), (3) Primary Mathematics, (4) Primary Science, (5) Junior-Secondary Literacy, (6) Junior-Secondary Mathematics and (7) Junior-Secondary Science. 3. Trained TTI pedagogy lecturers. Trainings were provided on the adapted modules and pre-service curriculum materials. These trainings involved 341 facilitators and 1,713 teacher trainers. Materials were adapted from project in-service materials introduced to pedagogy lecturers of partner TTIs. These materials were also used in dissemination training for TTI consortia partners. 4. Established a partnership between Florida State University (FSU) and the State University of Semarang (UNNES).This partnership began in September 2014 to: (1) develop curricula and courses for pre-and in-service teacher training for early grade reading and literacy; (2) develop and pilot supporting training and classroom materials; and (3) roll out the courses and materials to other TTIs. The initiative involved four professors from FSU and 16 lecturers from UNNES to develop 12 units of teacher training modules in early grade reading and writing. FSU professors completed several visits to UNNES – the first in 2015, to review the implementation of eight reading course units and to implement four remaining units. In 2016, FSU professors returned to develop monitoring and evaluation instruments. Two of the UNNES lecturers interviewed by evaluators said they were very appreciative of this program and had learned a lot from working with foreign professors in drafting the modules. The lecturers noted that all 16 modules were completed and ready for publishing by FSU. The rollout to another 16 TTIs began in October 2016. B. Strengthening Practical Pre-Service Training To strengthen practical pre-service training, PRIORITAS conducted three main activities: (1) develop lab, partner and “good practice” schools; (2) develop improved practicum programs; and (3) implement a program of Classroom Action Research (CAR). 1. Develop lab, partner and “good practice” schools. PRIORITAS’ partner TTIs used 147 schools as laboratories for piloting new teaching approaches during the project. The project supported TTIs and their lab schools through regular meetings and ongoing monitoring. These lab schools received the same in￾service package as PRIORITAS’ district partner schools for teaching and learning and for SBM, delivered by LPTK facilitators. Developing lab and good practice schools was meant to provide a context for practical pre-service training, and such schools were jointly selected, trained and mentored. One TTI lecturer USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 39 noted that, “Good practice schools supported pre-service training. For mentoring, planning and assistance, PRIORITAS staff and facilitators made regular one-day visits to those schools.” Lecturers emphatically endorsed the combined training-mentoring process as a very beneficial practice learned from PRIORITAS. Interviews with TTI facilitators revealed that university lecturers have increased their mentoring of teaching practice when they are in classrooms. They further said that lab school principals have adopted the practice of mentoring their own staff. During an interview with the evaluation team, the MORHE Director of Learning stated that PRIORITAS actions to strengthen TTI pre-service and in-service programs were aligned with MORHE’s programs on standardizing student-centered learning. The director expected that details about the best practices of active learning, CAR and PPG (all key PRIORITAS models) implemented at the top three TTI partners would be supplied to the directorate for possible inclusion in their programs. 2. Develop improved practicum programs. In its last three years, PRIORITAS trained pedagogy lecturers, teachers and school principals in lab and partner schools to develop improved practicum programs through workshops on developing materials to enrich the PPG curriculum. The workshops focused on preparing lesson plans, microteaching that included active learning, reflective journals, higher-order questioning and worksheets, authentic assessments, peer teaching practices, portfolios and follow-up planning. PRIORITAS also developed materials to support the PPL and PPG in these topics. The rector of one PRIORITAS partner TTI told evaluators that the institute had adopted several models from the project: “We kept all the documents, including the CDs, which were useful for our lecturers. Many USAID projects have the same objectives as we do for teaching and learning, such as how teachers at the primary school can become professional teachers and how to teach students effectively.” 3. Implement a CAR program. The CAR program was organized to improve the quality of teaching in classrooms-based action research. The program ran from 2014 to 2015 and was organized by TTI lecturers and teachers supported by project facilitators. The CAR team comprised a TTI lecturer from a practice teaching department, a TTI project facilitator and two teachers selected from lab and partner schools. PRIORITAS conducted 20 provincial workshops to enable 32 CAR teams (there were two CAR teams per partner TTI) to meet for planning, data collection, writing and rewriting the research process. This program was described in PRIORITAS documents as having been highly successful and effective, even as it reached a relatively small number of teachers in partner schools. In addition to the listed activities, in 2015 the USAID Program to Extend Scholarships to Achieve Sustainable Impacts (PRESTASI) collaborated with PRIORITAS to send 50 personnel from TTIs, MOEC, MORA and MORHE to visit Michigan State University to learn about best practices in pre-service teacher preparation. Nearly all online survey respondents (95 percent) from TTIs stated that PRIORITAS has improved the competency of lecturers and teacher candidates in active learning. One of the challenges in TTIs is the excessive focus on theory and relatively little focus on the practical aspects of teaching. According to respondents, after receiving benefits from PRIORITAS, their TTIs are more practice-oriented in their teaching. Respondents also affirmed the effectiveness of activities introduced by PRIORITAS, ranking them in the following order, starting with the most successful activities: (1) active learning; (2) mentoring by facilitators; (3) literacy program; (4) SBM and community participation program; (5) teaching practicum for teacher candidates (Praktek Pengalaman Lapangan, PPL); (6) CAR; and (7) teacher deployment program. C. Developing the Role of TTIs as Service Providers According to the PRIORITAS 2016 Annual Report, the project developed “the capacity of TTIs to provide improved quality training programs both for teachers in training and for serving teachers, by developing the knowledge and skills of teacher educators to use student-centered and innovative training methodologies, and USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 40 by increasing their access to and use of quality training curricula, resources, and facilities. “PRIORITAS also supported TTIs, provincial and district governments and LPMPs by improving human resources planning and budget allocations related to in-service training, Three inter-related PRIORITAS initiatives contributed to the objective of strengthening the role of TTIs as service providers: (1) collaborating in the design and delivery of in-service training in project partner schools; (2) strategic business planning; and (3) developing partnerships with district governments and MORA. 1. For the design and delivery of in-service training, TTI representatives were involved in designing and implementing an in-service training program on whole-school development in 1,243 partner schools (842 primary schools and 401 junior-secondary schools in 50 districts). PRIORITAS hired TTI facilitators and personnel to design and write modules, deliver them in South Sulawesi and East Java, and then review their implementation. During the evaluation visit to South Sulawesi and East Java, TTI informants in these two provinces reported that TTI facilitators are still involved in training and mentoring at teachers’ working groups and MGMPs. 2. To support TTI strategic business planning, PRIORITAS conducted a national-level strategic business planning workshop in Jakarta in 2015 to discuss opportunities and challenges in developing TTIs as service providers. The project followed up with a series of provincial and district-level workshops to support planning and budgeting of continuing professional development (CPD) programs. At these workshops, TTI and district representatives discussed how TTIs could support districts as co-service providers (trainers, consultants) to implement their plans to disseminate PRIORITAS activities. 3. To further ensure sustainability of the program, PRIORITAS conducted workshops with 45 of its 90 districts in preparing their education sector strategic plans. As a result, most of these districts have incorporated CPD plans into their strategic plans, thus guaranteeing funding for the next five years (Annual Report, 2016).PRIORITAS took a similar approach with MORA for madrasah teachers. In the planning workshops, TTIs, districts and MORA jointly prepared budgets and implementation plans for teacher in-service training programs. At least 249 relevant MOUs or formal partnerships resulted between partner TTIs and districts, schools, MORA offices or other agencies. During FGDs with facilitators at five TTIs, some participants told evaluators that their institutions were ready to operate as service providers, while others were not sure. Their uncertainty stemmed from the legal status of their TTIs(Badan Hukum Milik Negara or Badan Layanan Umum, a state company), which may not permit the institutes to play this role. One facilitator confirmed that a business center at his TTI had been established in anticipation of the coming activities. Another TTI had signed 22 MOUs with the Dinas Pendidikan in East Java. Results from the online survey showed more than 90 percent of respondents agreeing that PRIORITAS has increased the capacity of TTIs as service providers. The evaluation team found that, as a result of PRIORITAS training, all partner TTIs were prepared in several substantial and positive ways to become service providers for in-service training. TTIs that already have legally registered companies have made more progress compared with those who are trying to operate as institutions. Project documents suggest that at least 2,500 participants were trained under this initiative to help TTIs become service providers. In terms of the sustainability of the TTI program, there are three main platforms:  Replication or adaptation of PRIORITAS programs;  Dissemination to spread the impact to wider audiences; and  Partnerships by TTIs as follow-up service providers. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 41 The online survey revealed that 43 percent of TTI respondents had adapted PRIORITAS ideas (i.e., active learning and reading programs) into their classroom teaching. The survey further showed that 84 percent of TTIs had integrated PRIORITAS ideas into their PPL (pre-service) and PLPG-PPG (in-service) curricula. For example, a TTI Vice Rector in East Java told the evaluation team: “[In the past, PPL] was conducted internally, and we had not yet started with partner schools. We now have 80 partner schools, and 10 are in partnership with PRIORITAS. Before joining with USAID, we conducted all our activities within the university. Since we started the partnership we now focus on “in/out/in/out.” Students receive theory, then go to the field and observe and practice. Using this pattern, students start practicing to become teachers. For PPL,in the past it was for two months, but now it is for four months.” D. Dissemination of Good Practices from Partner Institutions The intention of this dissemination program is for good practices and systems established in partner TTIs to be disseminated to others. PRIORITAS put this into practice in four ways: 1. Worked through consortia of partner TTIs in each province. Thirty-one consortia associated with 17 partner TTIs were identified as provincial partners. TTI consortia members were also routinely included in many PRIORITAS activities, the project providing only facilitators and training materials. A host TTI was responsible for planning and funding all other training components. In meetings with TTIs, evaluation key informants said they work with a group of non-partner TTI through consortia to share information and disseminate PRIORITAS good practices. PRIORITAS held meetings with consortium partners twice per year in each province. Online survey results show that 90 percent of TTI respondents are willing to continue to partner with consortia and disseminate PRIORITAS ideas after the project ends. 2. Showcased events of good practices. PRIORITAS held meetings and talk shows in each province and at the national level to provide forums for sharing progress, successes and outcomes from the project trainings, workshops and activities. Events included displays from partner universities and lab schools that highlighted examples of improved teacher planning and professional development, as well as examples of students’ work showing active learning approaches. Talk shows included panels of representatives from TTIs, Dinas Pendidikan and GOI ministries. Participants highlighted progress and discussed issues and challenges in improving the quality of education through more effective teaching and learning.1 School and local government informants in Lumajang District reported that the three-day showcase event held there in 2016 attracted huge attention. Audiences from non-partner schools and non-partner districts in East Java attended this event. Teachers and district facilitators interviewed told the evaluation team that the showcase events they held or attended increased their knowledge, primarily in the area of teaching innovations. 3. Collaboration with other TTIs. PRIORITAS facilitated the development of professional relationships between project TTIs and other provincial TTIs. These relationships have provided opportunities for sharing information, planning upcoming activities and promoting professional discussions related to academic programs. 4. Disseminated training programs. PRIORITAS provided facilitators and materials to disseminate training to additional lecturers and teachers, with host TTIs being responsible for training planning and funding. At least 2,500 participants were trained by TTI partners working as service providers. Host TTIs selected training materials that suited their needs, and TTI facilitators provided a customized program during the three-day trainings. Per the KIIs and FGDs with TTI informants, some TTIs have 1Heyward, 2016. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 42 conducted internal dissemination independently by adapting some or all PRIORITAS modules, depending on the needs and capability of each TTI. Going forward, the continuation of the TTI-support programs faces some challenges. Survey respondents and key informants from three TTIs visited by the evaluation team stated that the main challenges are decisions by university policymakers, coordination and communication among university stakeholders and financial resources. Therefore, lecturers need additional support to ensure program sustainability, especially to provide active learning materials and to address the financial, policymaking and coordination issues they highlighted. For example, it requires funds to continuously collaborate with partner schools and lab schools, and to provide additional mentoring and refresher trainings to update lecturers’ knowledge. Additionally, it will be critical for university and department leaders to adapt the programs institutionally. 4.1.2.2 Conclusions 1. PRIORITAS’ quality content and efficient organization of training processes were widely appreciated by TTIs and are now being used by all partner institutes. The training-mentoring process – frequently judged by stakeholders to be valuable and effective for delivery of active learning methodologies – had two major impacts. It made university lecturers better educators and it improved the capacity of pre￾service training divisions to produce better future teachers. 2. PRIORITAS’ partnerships with TTIs produced successful training programs for increasing the pedagogical capacity of nearly 2,500 TTI lecturers of teaching and learning and SBM. At all TTIs, student teachers are now demonstrating good practices in teaching during PPL courses. To various degrees, all TTIs are now ready to be training providers, and almost all TTIs have established a wide range of MOUs with district governments and Dinas Pendidikan. 3. TTI partners have the capacity to facilitate in-service and pre-service trainings that meet local needs. This capacity is illustrated by ability of TTIs to conduct both types of trainings for TTI consortia members, teachers and district education staff by adjusting and modifying the PRIORITAS modules to fit local needs and availability of resources. TTIs' abilities to continue to conduct such trainings will depend on locally available capacity and funding. 4.1.2.3 Recommendations 1. To institutionalize active teaching and learning methodologies and build TTI capacities, partner TTIs should develop and implement internal affirmative programs to disseminate active learning methods to all departments within their institutes. 2. Partner TTIs should broaden their coverage beyond science, math and Indonesian language to have greater impact at the university level. 3. TTIs should continue their partnerships with consortia members, partner schools and lab schools, to sustain the best practices after PRIORITAS ends. 4. To realize a broader impact on Indonesia’s TTIs, future donor and GOI projects should include lower￾tier and private TTIs as partners. 5. GOI should consider formally adopting the best practices of PRIORITAS training to improve the quality of pre-service training at TTIs. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 43 6. To foster sustainability, local governments should incorporate CPD (Continuous Professional Development) plans into local government strategic plans. This step will help ensure the provision of required funding over the next five years. 4.2 Capacity & Initiative of Schools, Communities & District Governments for the Management of Schools Research Question 2: How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS improved in developing/supporting accountable and participatory school management? The background to this question is a transition by MOEC from centralized to decentralized school administration, which has been underway since 2003. This transition has involved regulating and encouraging government-owned schools and their communities to adopt SBM; and the provision of BOS grant programs, and District Assistance (BOS BOS Daerah), which enable schools to manage their own finances. Although SBM and BOS have become well-known school administration methods throughout Indonesia, there has been considerable variation across districts and schools in the capacity and readiness to fully adopt this type of accountable, transparent, and participatory school management. 4.2.1 Transparent and Participative School Management 4.2.1.1 Findings A detailed statistical analysis of findings derived from interviews and FGDs shows:  89 percent of informants reported that positive changes had occurred in the way schools and budgets are managed, and the ways in which community participation influences the governance of schools, as a result of PRIORITAS interventions;  3 percent of informants reported that negative changes had occurred; and  8 percent of informants reported that no changes had occurred. The most frequent improvements noted by school informants in FGDs and interviews were: (1) more involvement and better support from paguyuban (parent associations), parents and committees; (2) better use of BOS funds; and (3) communities empowered to participate more in their schools as a direct result of committee members having been included in PRIORITAS trainings. These findings are consistent with recent PRIORITAS project performance data, which show that more schools are now implementing transparent and participatory school budget management. The trend is consistent between cohorts across years (see Figure 2). Respectively, 57-65% of primary schools and 49-52% of secondary schools implemented good SBM practices in 2016. These percentages were higher than they were in 2014. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 44 FIGURE 2: PERCENTAGE OF SCHOOLS IMPLEMENTING TRANSPARENT AND PARTICIPATIVE SCHOOL BUDGET MANAGEMENT Source: PRIORITAS Monitoring and Evaluation Project Data, 2016 PRIORITAS has provided its stakeholder groups with a better understanding of the role of principals, teachers, and school committees in school management and governance. The project also encouraged community participation to support the quality of education. Now, schools work closely with teachers, parents, and school committees. Most informants in all schools claimed that teachers are now involved in the management teams dealing with school development. They said teachers and parents were involved in developing school plans and budgets through regular meetings. Schools invited parents if there was any updated information related to the school, or if schools need parents’ support. Parents have become more engaged with schools and have supported school with various needs, such as money, materials, and labor. However, the existence of “saber pungli” made some school principal cautious. Those principals said they worried that if their schools received money from parents, it could be classified as “pungutan,” or illegally collected money, even though parents gave the money voluntarily. 22% 35% 11% 40% 40% 32% 65% 60% 57% 0% 20% 40% 60% 80% Cohort 1 Cohort 2 Cohort 3 Transparent and participative school budget management in primary school 2014 2015 2016 30% 26% 14% 49% 44% 29% 40% 55% 52% 0% 20% 40% 60% 80% Cohort 1 Cohort 2 Cohort 3 Transparent and participative school budget management in secondary school 2014 2015 2016 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 45 The quotes that follow illustrate that an overwhelming majority (89 percent) of informants reported positive changes from SBM. “Through school-based management training, PRIORITAS has built a culture of school community, parents and alumni around the importance of planning for quality education. The Office of Education has asked all schools to use the PRIORITAS school strategic plan formats as a reference for developing their plans.” – District official, South Sulawesi “Previously, community participation was hidden; now it is transparent. It is because of USAID. We now understand that the three pillars – school, committee, and parents – should work hand-in-hand.” – School committee member, North Sumatra “The involvement of the school committee with the school is increasing. The number of meetings they attend is increasing. RKAS, school development, and school visits put the committee and parents on a busy schedule, especially after they attended the PRIORITAS workshop that encouraged them to participate more.” – School principal, Central Java Only 3percent of informants reported that they were less involved in school management or involved on a limited basis. Some parents emphasized the importance of leadership roles (school principal as leader) to implement participatory and transparent school management. Seven percent of informants reported that no changes had occurred. These responses mostly came regarding schools where good SBM practices existed before PRIORITAS began, usually at “A-team” schools (sekolah unggulan). Evaluators noted that in some of the schools they visited, school committees were dominated by teachers. They did not include parents, members of the community or school alumni, as was normal in most other schools. In addition, more school committee leaders and members were male, while paguyuban members were mostly female (mothers). As an entity, the paguyuban structure falls under the school committee. Per the FGDs and KIIs, the paguyuban supported school activities in many ways, most importantly in active learning and reading programs. They provided furnishings and books for reading corners as well as teaching materials and aids. Teachers and parents the evaluation team interviewed stated that they communicate more frequently and easily now, since PRIORITAS was implemented in their schools. 4.2.1.2 Conclusions 1. PRIORITAS interventions were effective in improving management and participatory governance in partner schools. Project interventions were also effective for some schools that reported having fully adopted inclusive SBM before PRIORITAS began in 2012. 2. PRIORITAS interventions were effective in strengthening community participation in school governance. The project deliberately involved school committee members, parents and others in SBM training activities. In many cases, these were unprecedented opportunities for them to work with school principals and education officials to gain a better understanding of their duties and responsibilities. 3. PRIORITAS’ approach of inclusive training appears to be a best practice model. The steady devolution of authority to individual Indonesian schools has been an unfamiliar experience for school principals and communities alike. They have required assistance to wield their new authorities effectively, and to be individually responsible for improving the quality of education in their schools. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 46 4.2.1.3 Recommendations 1. Where school committees need support to participate fully in school governance or to take on new responsibilities that are devolved to them, local governments should adopt an inclusive training approach such as the one taken with PRIORITAS, wherein training is provided not only for school principals and staff, but for other stakeholders such as school committee and community members, alumni, etc. These trainings will help all committee members understand the roles and actions needed to improve the quality of education. 2. While the frequency of transparent management is increasing in PRIORITAS school districts, the potential remains for poor financial management practices by school administrators. Therefore, school administrators should understand and act in accordance with the situation that a school committee is now an integral part of school governance, not a separate, independent entity. 4.2.2 Media Programs This topic refers to the substantial program of PRIORITAS media and communications activities using a wide variety of print and non-print media. The responsible PRIORITAS project team members explained that these activities occurred in two main areas:  Various types of communications methods and media for internal audiences such as PRIORITAS schools, districts, partner TTIs, trainers and staff, as well as others directly associated with the project.  Communications media for external audiences not directly associated with the project, distributed through local, regional and national news media outlets. PRIORITAS has developed communication and outreach programs, such as good practices and reference books, project videos and a newsletter that is available on the PRIORITAS website. The newsletter covered information and best practices of the project related to teaching and learning, SBM, teacher deployment and reading programs. The purposes of PRIORITAS media and communications activities were described as:  Documenting and disseminating good practices to raise public awareness about achieving quality education.  Supporting training activities by producing videos and training materials and modules.  Strengthening accountable and participatory school management. Following discussions with USAID to better define the scope of the media activity evaluation, the evaluation team took a broad perspective in documenting the media’s role in PRIORITAS. Data collection methods were designed that asked informants for their observations about if, where and when they heard or saw any mention of PRIORITAS in mass media, and what had been observed; whether journalists ever visited their schools or districts, and under what circumstances; and whether they ever access PRIORITAS communication products. Two collection methods were used to collect a variety of qualitative data: 1. A standard open question about PRIORITAS media was included in online survey of informants at all partner TTIs. This question required informants to provide a written response. 2. The same question was included in most face-to-face interviews and FGDs conducted in districts, sub￾districts and schools covered by the evaluation. More information about this survey is presented under Research Question 4 in this section of this report. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 47 4.2.2.1 Findings Analysis of both methods showed significant variation in informant responses. Some answered the question specifically; others also included unsolicited comments about mass media in general. Many provided general observations, but a significant number provided considerable detail regarding the media types, outlet names and specific stories they had seen. A number included comments about the role local news media played in ensuring that school budgetary management is transparent. Overall, answers revealed that many informants had responded positively to the types of PRIORITAS media they had observed. A smaller number had responded negatively or had not observed any PRIORITAS media. PRIORITAS communication products contributed to the quality and reach of project trainings, offering an alternative channel to disseminate project materials and ideas to broader audiences. Sixty project videos were produced to be used in training and advocacy activities. Through September 2016, a total of 6,096 news stories about PRIORITAS activities appeared in at least 279 different newspapers and other printed media. Similarly, through September 2016, the PRIORITAS website received 528,820 unique visitors and 6,620,654hits. PRIORITAS also published newsletters in each project province from September 2015 to October 2016 and distributed them to ministries, non-governmental organizations, partner TTIs, partner schools, media and other stakeholders (Annual Report, 2016). Informants and survey respondents stated that mass media had frequently run stories about PRIORITAS activities; that media stories had raised public awareness about educational issues; and that informants had used PRIORITAS media (e.g., website, newsletter, books) to learn about project best practices. In the large number of positive answers, the words “inspiring,” “inspirational,” “very positive,” “positive impact” and “motivational” were frequently used. Some qualified these expressions by thoughtfully describing what they meant regarding the usefulness of PRIORITAS media for broadly disseminating news about its own activities and, more generally, for raising public awareness about educational issues, as illustrated by the quotes that follow: “I read about USAID in Jawa Pos and other papers – my impression is the news could inspire good teaching and learning practices in Indonesia.” – Lecturer, TTI informant “I have read news about PRIORITAS. The content is good. The community can understand more about the importance of improving teaching [to improve] the quality of education.” – Lecturer, TTI informant “The news about PRIORITAS has inspired schools and other districts who have not been exposed to the program to follow or replicate it.” – Lecturer, TTI informant “Jawa Pos newspaper, radio, TVRI, PGRI magazine and social media put out a lot of news about literacy, especially the coverage about the literacy award.”– School principal, East Java At the same time, a small number of informants responded less positively about the impacts of the project’s media interventions, as illustrated by the following quotes: “The spread of the news has not reached the non-partner schools really intensively, so it doesn’t have a real impact.” – Lecturer, TTI informant “In this district, the media has not been informed specifically about [PRIORITAS]. The media does not really understand it. Even if there is coverage of PRIORITAS activities, they do not understand the content or essence. Instead of asking questions about the PRIORITAS activity, they ask about something else. In my opinion, the media has not yet explored and exposed the information that could be disseminated.” –School supervisor, North Sumatra USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 48 TTI respondents and sub-district and school-level informants often included comments relating to school management and governance. Some made general comments about the role of news media as a watchdog over accountable and participatory school management. These comments revealed their concerns that school administrators may either mishandle BOS funds or may ask parents to make additional payments when education in government schools is meant to be free. Informants implied that local news media are very interested in these topics: “We have had many journalists come to this school to ask a lot of questions about the school budget, including BOS.” – School principal, East Java “We are very careful when we want to invite parents to socialize the school program, especially if we want to ask them to make financial contributions. After the all the news about saber pungli[a task force to eliminate extortion in schools], we are afraid that local journalists will come because they heard rumors that the school collects money from parents.” – Parent, East Java Some informants revealed that there could be a special arrangement between a school and the local news media. One type of arrangement the evaluation learned about was where schools had purchased a “subscription” to a local newspaper. They could then enjoy a mutually beneficial arrangement to have stories published about the school. 4.2.2.2 Conclusions 1. PRIORITAS media and communications activities used many types of tools to communicate with the project’s internal and external audiences, and for multiple purposes. These media channels proved to be an effective means to publicize PRIORITAS activities and best practices and raise public awareness on basic education issues. However, given the possible wide range of the meaning of media and the indirect connection between project media activities and PRIORITAS’ desired outcomes, it is difficult to assess a specific impact of media on education governance. In addition, informants reported that journalists were usually more interested in reporting on schools’ scandals involving corrupt or unfair practices, instead of their good education practices. 2. There is a general expectation that all news media, including PRIORITAS’ own media, should educate the public about the management and quality of education. It was also generally observed that the media plays an important role in monitoring school activities, especially the correct implementation of SBM. In some ways, media and schools are expected to collaborate on these issues. 4.2.2.3 Recommendation Future donor or GOI education projects are recommended to adopt the PRIORITAS communication approach – capitalizing on online, social, and mass media to disseminate program information and materials to a broad audience of stakeholders. Such interventions should be tied to desired project results and outcomes, with specific impact-related indicators. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 49 4.3 Capacity and Initiative of Local Governments for Coordinating Education Resource Management Research Question 3: How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management with focus on teacher deployment, teacher professional development, and educational funding? 4.3.1 Findings This section primarily addresses the PRIORITAS teacher deployment program (PPG) and the teacher continuing professional development program (PKB), which each district attempts to fund and manage for its teachers. PRIORITAS developed a PPG to help governments address the undersupply of civil service teachers and abundance of temporary teachers, as well as the inappropriate distribution of civil service teachers among urban and rural schools. PRIORITAS provided PPG support to partner districts through three types of workshops that were led by provincial facilitators and trainers, and university lecturers. The workshops were for developing manpower planning skills in (1) data analysis to assess existing teacher deployments using the DAPODIK information system; (2) policy analysis to establish the strategic issues related to PPG and to prepare supporting policies based on data analysis; and (3) implementing PPG as a manifestation of mandates provided by government laws and regulations. To collect information about the PPG program, evaluators conducted interviews with school supervisors and the heads of local UPTD and FGDs with district education stakeholders. The FGDs usually included representatives of district education offices, district religious affairs offices, regional development planning boards and district human resources boards, as well as local parliamentary representatives and members of local education boards. The most extensive and relevant qualitative data came from FGDs for district stakeholders. Coincidentally, more than 80 percent of UPTD heads attended these FGDs. Few relevant responses were collected during the individual interviews with heads of UPTD and school supervisors. Evaluators recorded statements during the FGDs that commended the PRIORITAS PPG, because it had assisted local governments in assessing teacher deployments and making better proportional redistributions. One FGD participant noted, “The distribution of teachers in this district is now good, and this is because of the PRIORITAS PPG.” PRIORITAS district partners reported having applied at least one policy that had been developed during PRIORITAS PPG workshops. These new policies included introducing multi-grade classes, grouping multiple schools into one, rotating teachers from school to school and recruiting new teachers. In all, 40 new policies related to the deployment of teachers were reported to have been adopted by PRIORITAS partner districts in the evaluation sample. Partner districts also responded positively to PRIORITAS educational programs by committing to disseminate them broadly to non-PRIORITAS schools. For this purpose, PRIORITAS partner districts in one province collectively allocated more than IDR 17 billion in their 2014-2016 budgets for PRIORITAS approach dissemination. The project’s own monitoring data also shows that several partner districts collectively allocated more than IDR 40 billion in their 2014 budgets for education program activities. Of this total, more than 50 percent, or IDR 25 billion, was allocated for PKB programs (see Figure 3). USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 50 FIGURE 3: ACCUMULATIVE BUDGET ALLOCATION IN APBD 2014 FOR EDUCATION PROGRAM ACTIVITIES BY SEVERAL PRIORITAS PARTNER DISTRICTS Source: PRIORITAS Project Monitoring Data (2017) Other project monitoring data show that more than 70 percent of partner districts had prepared teacher training plans based on conducting needs analyses; and 53 percent and 72 percent of them, respectively, had entered cooperation arrangements with TTIs and LPMPs for conducting PKB (see Figure 4). These significant budget allocations were developed through coordination among local development agencies, including Dinas Pendidikan, Badan Kepegawaian Daerah, Dinas Pendapatan Daerah and BAPPEDA. In one district FGD, a BAPPEDA representative explained how the education sector receives support: “BAPPEDA has always supported education programs, as we have a better understanding of the issues and needs due to our participation in PRIORITAS activities, including monitoring of education activities at the school level.” 2.2 6.1 9.2 25.0 - 5.0 10.0 15.0 20.0 25.0 30.0 Reading Skills Reading Culture KKG/MGMP Trainings for Teachers in Rp billion Education Program Activities USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 51 FIGURE 4: PERCENTAGE OF PRIORITAS DISTRICT PARTNERS HAVING A PKB PLAN BASED ON NEEDS ANALYSIS AND COOPERATION WITH CAPACITY-BUILDING SERVICE PROVIDERS Source: PRIORITAS Project Monitoring Data (2017) However, local government officials frequently told evaluators that they could not confidently predict if using PRIORITAS PPG methods would continue indefinitely. They cited the various challenges they faced: limited funding, limited human resources, and other political commitments of local leaders. These challenges echo the findings in a 2015 study on teacher deployment in Indonesia by USAID.1 One FGD participant stated: “PRIORITAS trained only a small number of schools, but the district is left responsible for training the rest of the schools. This will require not only huge funding and considerable human resources, but also a strong commitment from the local government.” Informants told the evaluation team that politics always influenced the ways that local education sectors could be developed, and that elected governors and mayors made their own decisions. One informant highlighted the sensitivity around political commitment within the education sector in Indonesia, stating: “There is no governor’s regulation about the future the of PRIORITAS program in our district, because it is a political matter connected with local government elections. However, we are very positive that the PRIORITAS program will be included in 2018.” Data provided by PRIORITAS partner districts in one province during PPG workshops, and analyzed by PRIORITAS, exposed the magnitude of the financial challenges faced by local governments to pay for PKB programs that develop the professional skills of teachers. The funding requirement ranges from IRD 3 billion to 54billion to train all of the teachers in each district (see Figure 5). Further analysis of the data using the recent unit cost of teachers’ training applied in each district showed that funding limitations of local governments would mean they required between 5 and more than 40 years to train all of the teachers in each district (see Figure 5). 1RTI International (2015). Teacher Deployment in Indonesia: Challenges and Solutions. 86 74 53 72 14 26 48 28 0 20 40 60 80 100 Has developed in-service training In-service training is based on need analysis TTI as trainer LPMP as trainer Percentage Yes No USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 52 FIGURE 5: ESTIMATED BUDGET AND TIME REQUIREMENTS FOR LOCAL GOVERNMENTSTO TRAIN ALL TEACHERS Source: Provincial office of USAID PRIORITAS (2017) A further challenge faced by PRIORITAS partner districts is the competition education programming faces with other development priorities. As Figure 6 illustrates, the percentage of education funding (covering education program activities and salaries) in total APBD for eight districts shows that 75 percent had allocated more than 20 percent of their development funds to the education sector. However, the large majority of this funding was for salaries, with only 4 to 9 percent going toward education program activities in these districts. During one district FGD, local officials told the evaluation team of the difficulties the district will face if they continue trying to use the PRIORITAS PPG after the project concludes: "In 2017, we were planning to use the SIMDIKNAS information system, but we cannot because the DAPODIK system is being upgraded. We still need support from USAID PRIORITAS to continue developing our capacities for analyzing the SIMDIKNAS tool for teacher deployments.” 54 37 27 24 18 16 15 15 11 11 11 10 7 5 3 0 5 10 15 20 25 30 35 40 45 0 10 20 30 40 50 60 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15 Years in Rp billion PRIORITAS District Partners Budget Requirement Times Requirement USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 53 FIGURE 6: ALLOCATED BUDGET FOR EDUCATION SECTOR AS PERCENTAGE OF TOTAL APBDIN SELECTED PRIORITAS PARTNER DISTRICTS Source: PRIORITAS Project Monitoring Data (2017) Another FGD statement revealed concerns about inadequate local capacity to continue implementing education programs at a standard similar to PRIORITAS: “The Dinas Pendidikan cannot take on the full monitoring responsibilities required to take the place of the PRIORITAS District Coordinator. The Dinas is overwhelmed and busy with responsibilities. Taking over PRIORITAS monitoring would mean not fulfilling other current responsibilities.” 4.3.2 Conclusions 1. The allocations of budgetary support for disseminating PRIORITAS practices illustrates the way PRIORITAS stimulated an improvement in coordination between district-level authorities. The key authorities – Dinas Pendidikan, Badan Kepegawaian Daerah, Dinas Pendapatan Daerah, and BAPPEDA – now have a better understanding of the strategic issues confronting their education sectors. Such improved understanding was further enhanced through their active participation in PRIORITAS programs– e.g., joint project monitoring. These agencies now coordinate more, resulting in larger budget allocations for education. This coordination will likely be sustained, since budgeting processes run through a nationally applied coordination system called MUSRENBANG. 2. The combination of local capacity building and working with TTIs as service providers enhanced coordination between TTIs and local government as users. This coordination should help sustain PRIORITAS’ efforts, in part because the approach developed local human resources to be PPG/PKB trainers. Trainees include teachers, principals, school supervisors, university lecturers and LPMP trainers. Local governments will be able to mobilize these individuals to continue their PPG and PKB. 3. PRIORITAS improved the capacity of its partner local governments to distribute teachers and better plan their deployment – duties mandated by Indonesian law and regulations. PRIORITAS improved local capacity using DAPODIK data to assess regional teacher deployment situations, use assessment results to develop socially and technically sound policies that met local needs, and implement PPG policies in accordance with the strategic issues in each region. 45% 39% 43% 16% 61% 49% 31% 20% 5% 5% 6% 4% 4% 9% 4% 5% 0% 10% 20% 30% 40% 50% 60% DP 1 DP 2 DP 3 DP 4 DP 5 DP 6 DP 7 DP 8 Percentage of total APBD PRIORITAS District Partners % of Education Budget (including Salaries) to APBD % of Education Budget (excluding Salaries) to APBD USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 54 4. Local politics can present funding challenges for the education sector. Local leaders possess the legal authority to set development priorities, affecting how funds are distributed to each sector. 5. In general, PRIORITAS did not just provide a “do it yourself” handbook, but facilitated the development of a system in accordance with local needs to improve the capacity of officials to implement PPG and PKB. The strength of the PRIORITAS PPG and PKB approaches rested on the combination of workshops with follow-up technical advice and mentoring. PRIORITAS always carried out TA and mentoring immediately after a workshop was completed, and when follow-up activities were being implemented. 4.3.3 Recommendations 1. Local governments should continue to seek assistance from TTIs that have the capacity to facilitate PPG and PKB processes. The team also recommends that the central government play its critical roles (as stated in Law No. 23/2014 on Local Government) by providing technical assistance to PRIORITAS partner and non-partner districts, continuing the roles previously filled by PRIORITAS staff. These lessons learned from PRIORITAS PPG and PKB initiatives can be utilized by central government to improve PPG and PKB in non-PRIORITAS districts. In this regard, central government should coordinate with local government entities and use locally available capacities – that is, TTIs – to help local governments develop and implement PPG and PKB. This support could begin in provinces with a higher human development index (HDI) scores, because they generally employ higher numbers of education personnel. In addition, province-based PPG systems will provide opportunities to redistribute personnel across districts within a province. 2. Regarding PKB, the evaluation team strongly recommends that stakeholders in PRIORITAS districts continue to adopt the project’s dissemination model, wherein teacher trainings take place in KKG/Ms and MGMPs – at schools and cluster schools. The relevant parties have shown their enthusiasm to implement PKB, and governments continue allocating funding to support KKG/M and MGMP activities. 3. District and provincial education stakeholders should demonstrate the political will to reduce costs, exploring models of teacher training beyond cluster meetings and in-school dissemination, to find acceptable, lower-cost approaches for their regions. One possible model is Pendidikan dan Latihan (Diklat) Mandiri, or self-initiated training, where teachers in a region jointly fund their own training by paying invited trainers or lecturers, thereby reducing PKB costs. The association for supervisors of early grade education, known as PAUD, experienced a 50 percent reduction in training costs by adopting Diklat Mandiri for its members.1Diklat Mandiri significantly reduces logistical costs, since participants do not require accommodation, and training materials can be procured with funds contributed by participants. Further reductions of costs can be achieved if local governments support Diklat Mandiri by providing training facilities free of charge. 4. School supervisors should be an essential part of ensuring that principals and teachers adequately perform the duties assigned to them. All efforts to improve management of education in districts should include capacity development for school supervisors, particularly on distance management to help supervisors support their remote schools in resource-constrained environments. 1 Policy Paper: Capacity Building on Monitoring and Evaluation for Early Grade Education Supervisor. Prepared by Budi Rahardjo for the World Bank, 2014.Unpublished. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 55 4.4 Program Sustainability Research Question 4: How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level? USAID defines sustainability as “the continuation of benefits after major assistance has been completed. While ultimate responsibility for sustained benefits often rests with the local stakeholders, the operational principle of sustainability requires that it be incorporated from the start when selecting a program during the Country Development Cooperation Strategy process or designing a subsequent project.”1 The purpose of the sustainability program is to continue, adapt and advance good practices after development assistance has been completed. 4.4.1 Findings 4.4.1.1 Sustainability at the District Level At the district level, the sustainability of PRIORITAS interventions will be influenced by various factors; one of these will be the attitudes of elected district heads – mayors and bupatis. A district human resources board (BKD) member told evaluators that the political influence wielded by the district leader was considerable: “In this district, so many policies that are introduced are based on the political interests of the leader. Some programs that already run well may stop because of a change of leadership. Education programs may be replaced or entirely removed, even though politics should not interfere in educational matters. This makes us worry, because every time there’s an election, we are concerned that good educational programs will be discontinued after the district leadership changes.” This political influence reaffirms findings of a 2014 study on dissemination and sustainability conducted by USAID.2 To sustain PRIORITAS interventions, some districts have included elements of project approaches in their district regulations, the regulations of governors or mayors or in district annual plans. As reported, 43 partner districts that held local elections in 2015 developed new five-year strategic plans (rencana strategis or renstra) in 2016, with the assistance from PRIORITAS. Those districts have allocated around US$87 million for implementing and disseminating good practices over the next five years. In addition, 13 district regulations were promulgated to support implementation of plans and policies related to continuous professional development. These steps mean that those districts have formal or legal obligations to implement PRIORITAS programs. Some districts have progressed beyond planning. PRIORITAS documents showed that dissemination had been implemented using funding from local governments and schools, as well as project funds. Over the life of the project, local governments and schools contributed more than US$6.14 million to disseminate activities, supported by US$1.39 million of project funds. In 2014, the Bupati of Maros, South Sulawesi, issued multiple regulations stipulating that educational budgets should cover the dissemination of PRIORITAS school management and active teaching and learning best practices. The subsequent training targeted1, 523 teachers, school principals and school committee members at non-PRIORITAS schools. Another regulation formally appointed 15 PRIORITAS district facilitators for primary schools and 15 facilitators for junior-secondary schools. In 2015, the Bupati of Maros also provided funds for disseminating PRIORITAS best practices by training 516 teachers and school principals at non-PRIORITAS schools. Another example is the PRIORITAS district of Lumajang in East Java, where the local government included funding for dissemination of PRIORITAS best practices in the district education budget. Between 2014 and 2016, this district had the largest budget in East 1 Cannon, R., Arlianti, R., & Riu, I. (2014). The dissemination and sustainability of DBE and USAID PRIORITAS Programs: A research report prepared for USAID PRIORITAS. Jakarta Indonesia. Page 11. 2 Cannon, R., Arlianti, R., & Riu, I. (2014). The dissemination and sustainability of DBE and USAID PRIORITAS Programs: A research report prepared for USAID PRIORITAS. Jakarta Indonesia. Page 11. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 56 Java for disseminating PRIORITAS practices. Other PRIORITAS districts in East Java, including Batu and Blitar, also funded unique programs to disseminate PRIORITAS interventions. The PRIORITAS district of Pinrang in South Sulawesi, which was also a USAID DBE district, is another example. Evaluators were told that nine sub-districts in Pinrang had received dissemination training on PRIORITAS modules 1, 2 and 3, and it was expected that dissemination would be extended to other sub-districts in 2017. Pinrang has a unique education regulation that stipulates that for teachers to be promoted to be a school principal, or for a school principal to be promoted to supervisor, they must have formerly been a DBE master teacher, a PRIORITAS facilitator or a facilitator for another donor program. Additionally, PRIORITAS district facilitators are also now being used as resource persons for implementing Dinas Pendidikan programs more generally. Regarding PRIORITAS districts that had not regulated or funded such dissemination, evaluators were told during FGDs that some are optimistic that PRIORITAS approaches will be incorporated after 2017. For example, even though at the time of discussion there were still no regulations in place to support the thought, a district education officer from PRIORITAS district of Pekalongan, Central Java, said: “We are very optimistic that PRIORITAS [approaches] will be continued in Pekalongan because all district stakeholders have acknowledged that it has improved the quality of education here. At present, there is no regulation about this because of the coming local government elections; but we are very positive that in 2018 the PRORITAS program will be included in the district annual plan.” 4.4.1.2 Sustainability at the School Level School principals told evaluators during interviews that a number of methods will be used to sustain PRIORITAS good practices in their schools. Noticeably, at least four different methods were highlighted: 1. Many principals stated that PRIORITAS practices will be sustained partly because teachers have become accustomed to them, and partly because the teachers will continue to be encouraged by principals to develop lesson plans based on active teaching and learning classroom methods until they are completely familiar with them. One principal said: “The active learning methods will certainly continue in this school because teachers have become accustomed to them.” Another principal noted that, “I often come into classes to see what is going on. I see the teachers conducting active teaching and learning, using relevant teaching aids and, at the end of the lesson, conducting evaluation and reflection. So, I believe PRIORITAS methods will always be used by the teachers because they are now used to them.” Another principal said, “We ask all teachers to have lesson plans based on active teaching and learning methods.” These points were echoed during evaluation FGDs with teachers, for example: “Almost all teachers expressed their promise to implement the active teaching model practices because they are very useful and students like them. Besides that, the PRIORITAS program is in line with Curriculum 2013.” School committee informants made similar comments during their FGDs: “Many of them will use the PRIORITAS ideas even though the project has ended, because they have already got the advantages from implementing the PRIORITAS program, especially when they see the involvement of parents in school activities, and paguyuban participation took us even further than before.” 2. Many principals have developed annual plans that incorporate PRIORITAS methods. These plans were created by school development teams, which include parents and teachers, to guide the ways principals will conduct school activities. School supervisors also support and participate in the processes to provide necessary technical assistance. One principal said: “All the requirements for effective teaching are included and budgeted for in our annual plan. The needs of each class have also been discussed with each class organization for parents (Paguyuban Kelas).”Some principals guaranteed that active learning practices will continue to exist: “Because up to this moment there have been lots of good outcomes USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 57 that can be seen. Our sustainability strategy now is that at the beginning of every semester we provide each teacher with a package of teaching materials based on the requirements each earlier submitted during planning meetings.” This opinion was reinforced by informants during teacher and school committee FGDs: “Some teachers and school committee members in each school have been asked by the principal to form a school development team (Tim Pengembang Sekolah/TPS). One TPS task is to develop the school’s annual plan. In addition, in the past, teachers normally just copied their lesson plans from their colleagues or copied them from the internet. Now the teachers are competent and able to develop their own lesson plans.” 3. Some principals told evaluators that their schools will use local teachers’ working groups (KKG/M and MGMP) to sustain PRIORITAS methods. The teachers used these groups to share knowledge, skills and experiences, especially about PRIORITAS methods. A principal said: “Our KKG/Ms are very effective for transferring knowledge about active teaching and learning methods to teachers who were not trained under PRIORITAS. Teachers who have received training share their experiences and demonstrate their abilities. This motivates other teachers to do the same. Our school is also often visited for comparative studies by other schools that are keen to learn about active teaching and learning. They have been interested in the colorful displays of students’ work, and wanted to imitate these in their schools.” In the same vein, a teacher made the following points in an FGD: “We usually used KKG/M or MGMP to share our knowledge, skills, and experiences. It does not have to be profound, but any knowledge or experiences are important and useful to improve the quality of teaching and learning.” Another added: “Our MGMPs should become active learning clinics, where teachers, ex-facilitators and supervisors can meet regularly to discuss issues related to active teaching and learning.” 4. Principals told evaluators they also use their principals’ working groups (KKKS/MKKS and KKM) for sharing information and experiences related to PRIORITAS programs. One principal said: “In our KKS, we have periodically discussed the principles and problems of introducing SBM, and empowering school communities to participate more in our schools.” A teacher in another FGD stated: “In the future, strengthening our KKKS or MKKS to sustain PRIORITAS methods should be considered, and if possible, this should be included in the district’s annual plan.” 4.4.2 Conclusions 1. There are clear indications of sustainability of PRIORITAS approaches at both the district and school levels, including the commitment of leaders, presence of activity planning and relevant regulations, budget allocations, and community participation. Leaders – school principals and district heads (bupatis and mayors) – play critical roles in creating enabling environments for program sustainability. They have the necessary authorities to ensure that acceptable resources are allocated to develop and implement strategic activity planning. In addition to these political leaders, school supervisors occupy a key position in program sustainability by taking the place of district facilitators in providing support to teachers. Parents and the community at large, through paguyuban and school committees, play supporting roles by encouraging schools and teachers to continue applying methodologies and approaches introduced by PRIORITAS. 2. PRIORITAS interventions most likely will be sustained in some districts in the evaluation sample – a result made more certain by the inclusion of project approaches in annual plans and budgets. In other districts without such formal measures, PRIORITAS approaches may be carried on based on the interest and perceived benefits by political leaders. In all cases, future political interests of these leaders will be a factor. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 58 3. PRIORITAS-supported schools in the evaluation sample were using a variety of methods to sustain PRIORITAS interventions: using the momentum generated by teachers having become accustomed to the methods; incorporating PRIORITAS practices into annual school plans; using local teachers’ working groups (KKG/Ms and MGMPs) to disseminate practices; and using local working groups for school principals (KKKS) to share knowledge, skills and experiences. Using KKG/Ms and MGMPs as education forums helps improve coordination among stakeholders, including teachers, principals and supervisors. 4.4.3 Recommendations 1. The sustainability of PRIORITAS district-level interventions, or those of any future donor education projects, cannot be guaranteed by the optimism of local education officials alone. PRIORITAS partners and stakeholders at the district level should encourage local political leaders to incorporate PRIORITAS programs into their future regulations, plans and budgets. 2. Future education projects should place greater emphasis on the important roles that local educator professional groups can play to sustain and disseminate program interventions. Similarly, education officials at district and province levels should allocate sufficient resources to support activities of local teachers working groups (KKG and MGMP) and principals working groups (KKKS), including providing resource persons from TTIs or other relevant institutions. 3. The evaluation team understands the limited time and resources of school supervisors. Per KeMenPAN￾RB regulation No. 21/2010 and MOEC regulation No.143/2014, school supervisors should play a central role in giving direction and guidance to educators and in evaluating the performance of school principals, teachers and administrative staff. Therefore, future education projects should place greater emphasis on training local school supervisors– as was done by PRIORITAS – and actively using them as mentors and program disseminators, since they are key to ensuring accountability at the school level in the delivery of quality education which includes provision of support to enhance the functioning of local teachers’ working groups. Similarly, local governments must ensure that school supervisors can perform their supervisory and technical assistance duties. The evaluation team recommends that districts conduct local assessments of school supervisory functions and capacities, focusing on such factors as the ratio of supervisorsper schools and geographic area to be covered. 4.5 National-Level Benefits from PRIORITAS Research Question 5: How have national government agencies (MOEC, MORA, MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects? 4.5.1 Findings The evaluation team collected information and data for this section from a number of sources, particularly KIIs with MOEC and MORA officials at the national and provincial levels and national MORHE officials, as well as from PRIORITAS reports. PRIORITAS was implemented under the auspices of the Assistance Agreement signed in late 2015 between USAID and the Coordinating Ministry of People’s Welfare, which is now the Coordinating Ministry of Human Development and Culture. In early 2017, an Implementation Agreement was signed between USAID and MOEC. At the district level, PRIORITAS entered into MOUs with each local government. PRIORITAS invested its resources in building capacity of human resources in each district partner. On average, the project provided training in the active learning methodology to 15 teachers of each secondary school partner, six teachers of each primary school partner and five education personnel comprising school principals and school supervisors. PRIORITAS also provided training in SBM to each school principal and three members USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 59 of school committees – head, treasurer and one additional member. Within state-owned partner TTIs, more than 300 institute facilitators benefited from PRIORITAS training. Table 5 shows the enormous number of partners and beneficiaries from the five years of PRIORITAS implementation. National government officials also attended several PRIORITAS training programs and associated activities. Those programs and activities included showcases, workshops, a national policy dialogue on teacher deployment, a national training-of-trainers (TOT) workshop on strategic planning for TTIs and a national conference on action research in classrooms. TABLE 8: PROJECT PARTNERS AND BENEFICIARIES Partner Beneficiaries Cohort 1 Cohort 2 Cohort 3 Papua and West Papua TTI and Lab and Partner Schools Total Districts 23 20 7 3 53 Primary schools 368 321 111 97 98 995 Junior-secondary schools 184 160 57 0 56 457 Teachers in partners schools 11,103 1,008 3,601 240 4,385 29,409 Learners in partners schools 173,092 161,520 62,694 6,848 74,458 478,612 Teacher Training Institutes 17 17 Consortia TTIs 31 31 Higher education lectures 8,497 8,497 Student teachers 43,685 43,685 Source: USAID PRIORITAS Annual Report, October 2015-September 2016 The 2016 Annual Report notes that PRIORITAS provided significant technical assistance to the GOI over the project’s five years, in particular to MOEC and MORA. At MOEC’s request, PRIORITAS at various times trained representatives of 33 provincial LPMPs and trained MOEC’s sekolah rujukan (reference schools). Based on a request from MORA, the project conducted a pilot program in East Java to support districts and the provincial MORA office to prepare budgets and plans for PPG and PKB. As a result, MORA allocated IDR30 billion (US$2.3 million) to be distributed to 200 madrasah working groups (KKM) in the 90 partner districts over the three years from 2017 to 2019, and IDR 1 billion per year for three years to nine TTIs, including the seven partner TTIs. During interviews with senior MOEC officials in Jakarta, evaluators were told that PRIORITAS had collaborated for two years with MOEC’s Board of Culture and Education Human Resources Development (BPSMPK). This cooperation enabled the synchronization of PRIORITAS with MOEC’s 2013 curriculum, while also improving the knowledge within LPMPs about active teaching and learning and SBM. In addition, MOEC officials told evaluators that PRIORITAS district facilitators were being actively recruited to work as LPMP facilitators in 32 provinces. One MORA official interviewed for this evaluation had made efforts to disseminate PRIORITAS approaches within the ministry because he valued the high quality of project modules for improving madrasah education. He had arranged 40 teachers to be trained using the modules, to lay a foundation for broader dissemination within MORA. Unfortunately, this official had no further information about the fate of his plans, because following a departmental restructuring, he was transferred to another position within the ministry. The evaluation team noted PRIORITAS’ need to co-exist with other donor-funded technical assistance and capacity building projects, which also required significant GOI attention. For example, a project from the Australian aid program under DFAT to improve Indonesia’s educational policymaking and planning required considerable commitments from MORA’s budget and human resources. Coordinating and harmonizing the projects of multiple donor agencies can be a challenge for the government at all levels. One FGD participant USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 60 from BAPPEDA stated: “We were approached by three different funding agencies intending to implement almost similar programs operating in overlapping areas. Most of them require sharing budget from local resources. Harmonizing programs at the central [GOI] level would help local government to coordinate implementation of the programs at the field level.” One MOEC official raised a related point about the need to formalize donor project relationships with GOI counterparts, noting: “We would like to learn what improvements have been made in schools and what approaches have been adopted from PRIORITAS, but we are not sure if we can conduct the research because our department has not entered into an MOU with PRIORITAS.” Despite these challenges, MOEC officials expressed interest in adopting PRIORITAS training modules and employing PRIORITAS facilitators to conduct teacher trainings. However, they first needed to assess the suitability of the PRIORITAS modules, because MOEC had already developed its own course content for similar teacher trainings.1 A MORHE official interviewed by the evaluation team expressed interest in and appreciation for the contributions made by PRIORITAS to TTIs. The official invited PRIORITAS to share the information it possessed about the comparative performance of PRIORITAS partner TTIs, in the hope that MORHE would be able to choose suitable training plans for all Institutes. MORHE is in the process of issuing a ministerial decree to support TTI capacity to adopt and implement active learning – a step that would further dissemination of PRIORITAS approaches. As noted in Section 4.1.2, Teacher Training Institutes (TTIs), the TTIs are now, to varying degrees, ready to serve as training providers. They have established a wide range of MOUs with district governments and Dinas Pendidikan. Some TTIs are not ready because their legal status (BHMN or BLU state company)does not permit them to obtain a license to provide in-service training. Overall, the demonstrated benefits of PRIORITAS teacher training should have a positive impact on good teaching practices nationwide, particularly once MOEC and MORA are able to adopt project practices. The evaluation team also interviewed MORA education officials at the provincial level. In general, these officials were very positive about PRIORITAS’ efforts, and were appreciative of the technical assistance provided by the project. They said that the active learning methodologies, SBM modules and facilitator training were relevant to MORA’s need to improve the quality of education in madrasah. They also said PRIORITAS trainings had improved the planning abilities of principals and their ability to engage with their communities. Having these skills was essential for the survival of the many private madrasah, which do not receive extensive government support. Overall, PRIORITAS activities aligned with GOI education programs. In addition, PRIORITAS improved the capacity of local governments to fulfill their roles, as required by Law No. 23/2014 on Local Government.The PRIORITAS reading culture program has supported MOEC’s school literacy movement, as stated in MOEC Decree No.23/2015 on Character Building. MOEC designated 19 PRIORITAS partner districts as models for other districts to follow, after seeing how the project had supported the literacy movement in partner schools. To support this expanded reading program, PRIORITAS provided more than 13,000 packages of leveled reading books to partner schools and TTIs and non-partner schools. PRIORITAS PPG was another intervention that will likely provide direct benefits to GOI agencies. As described under Section 4.3 on Research Question 3, the PPG helped partner districts redistribute teachers while complying with the minimum standards for educational services, but without recruiting new education personnel. By getting some realistic policy options (tested in PRIORITAS districts) into the National Medium￾Term Development Plan (Indonesian acronym: RPJMN) and MOEC’s Renstra, the project helped MOEC enable 1During a presentation of these final evaluation findings in Jakarta on May 19, 2017, the Director General for Primary and Secondary Education stated that MOEC requires at least one year to adopt new methodologies and approaches. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 61 districts better distribute teachers, for example through multi-grade teaching, mobile teachers, school mergers, reassignments, etc. 4.5.2 Conclusions 1. Considered from the perspectives described above, it is clear that PRIORITAS has provided national￾level benefits. MORA, MOEC and MOHRE officials acknowledged and appreciated the value of PRIORITAS interventions and approaches and they affirmed that PRIORITAS activities aligned with GOI policies to improve the quality of teaching and learning outcomes, and for the national application of Curriculum 2013. 2. Notwithstanding the broad appreciation of PRIORITAS interventions by GOI agencies, the belated formal agreement between PRIORITAS/USAID Indonesia and the direct technical ministry (MOEC) hampered the GOI’s proactive participation and the support from various MOEC departments.1 This was compounded by the fact due to changes of key personnel in the ministries from organizational restructuring, continued support for PRIORITAS cannot always be confirmed. 3. During the course of PRIORITAS, the GOI’s ability to coordinate and harmonize the interventions of multiple donor programs in the basic education sector was limited by available time and resources of involved agencies and institutions. Aid requires governments to allocate sufficient resources to weave together approaches as well as lessons from different programs. Effective program harmonization is the collective responsibility of each counterpart agency and donor. 4. Coordinating and harmonizing the projects of multiple donor agencies can be a challenge for the government at all levels. One FGD participant from BAPPEDA stated: “We were approached by three different funding agencies intending to implement almost similar programs operating in overlapping areas. Most of them require sharing budget from local resources. Harmonizing programs at the central [GOI] level would help local government to coordinate implementation of the programs at the field level.” 4.5.3 Recommendations 1. To ensure the central government receives maximum benefits from PRIORITAS interventions before the project concludes, the PRIORITAS team should handover to the GOI of all the results and products from the five-plus years of project activities. While some project monitoring data and comprehensive district-level information on the availability of active learning and SBM facilitators has been provided to technical ministries, personnel turnover within those ministries has limited the institutional uptake up of these materials. The evaluation team recommends some form of formal project closing event that includes decision makers at the central government level. If resources are available, a national workshop involving selected PRIORITAS partner districts, TTIs and LPMPs should also be considered. Participants could make presentations about the concrete benefits they have received, how they have adopted methodologies and approaches introduced by PRIORITAS – particularly the training modules – and what support might be required for GOI ministries to fully adopt PRIORITAS programs. 2. For line ministries such as MOEC, MORA and MORHE to assume ownership of future education programs from USAID or other donors, and to facilitate better coordination, future programs should sign a formal memorandum of understanding directly with the ministry or ministries at the earliest stage possible. In addition, the MOEC Secretary General’s Office or another relevant office should consider a 1 During the presentation of final evaluation findings in Jakarta on May 19, 2017, the Director General for Primary and Secondary Education stated that one supporting factor of sustainability is an early MOU between donor and recipient. In fact, in early stage of project implementation, an Assistance Agreement was made with Menko PMK, which may reduce the ownership of technical ministries such as MOEC and MORA. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 62 stronger focus on coordination to ensure coherence among donor-funded projects, and to provide better support to partner agencies. A renewed focus on coordination will provide a foundation for the GOI to actively participate in project implementation and improve policy linkages between Jakarta and supported regions. 3. Per the project design, PRIORITAS was far more heavily engaged with educators at district levels than with national-level agencies. This approach improved local government capacities per their responsibilities under Law No. 23/2014. At the same time, the district-level focus limited opportunities to bolster the central government’s role per Law 23, to provide technical assistance and perform supervision. The evaluation team recommends that future education programs in Indonesia funded by USAID or other donors more evenly balance their engagement between the districts and the central government.Adjusting this balance will facilitate and encourage active participation from counterpart agencies, and will improve coordination, collaboration and policy linkages between regions and the center. 4. PRIORITAS approaches, methodologies and products have been implemented in a large number of schools, districts and provinces for more than five years. The GOI should seriously consider adopting those approaches, methodologies and products nationwide to maximize the benefits of the project to Indonesia’s educational systems. For example,the GOI could accelerate quality education through adopting PRIORITAS PPG in non-PRIORITAS districts, while at the same time reducing pressure on the national budget (APBN) to pay for local teacher remuneration. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 63 ANNEX I: FINAL EVALUATION OF USAID PRIORITAS STATEMENT OF WORK 1. Introduction The USAID/Indonesia Education Office seeks to carry out a final performance evaluation of its Prioritizing Reform, Innovation, Opportunities for Reaching Indonesia’s Teachers, Administrators, and Students (PRIORITAS) activity. PRIORITAS supports USAID’s programmatic focus in Indonesia of strengthening teaching and learning processes in the basic education subjects of Reading and Literacy, Math, and Science; developing better management and governance; and supporting better coordination among the stakeholders. USAID wishes to evaluate the extent to which PRIORITAS has met its overarching objective to support efforts to expand access to quality basic education in Indonesia. In line with the definition of performance evaluation (ADS 201.3.5.12), the PRIORITAS evaluation will address descriptive and normative questions related to activity objectives, namely: (i) enhance pre- and in-service teacher training; (ii) improve education governance and management; and (iii) enhance stakeholder involvement to improve coordination, planning, and funding for basic education delivery. As further described in this Statement of Work (SOW), the final evaluation is framed within three themes of activity results, learning, and sustainability. The USAID/Indonesia M&E Support Project (MESP) will organize and manage the services of an independent team to conduct the evaluation as defined in this SOW. 2. Activity Description PRIORITAS is the third of three consecutive USAID activities since 2003 that have worked to improve the quality of basic education in Indonesia. PRIORITAS, which runs from 2012-2017, builds on the successes of previous activities to increase capacity of stakeholders in the education system, leading to institutionalization of improved practices. The overall goal of the five-year, US$88.2 million, PRIORITAS activity is to expand access to quality basic education. In this role, PRIORITAS focuses on building system capacity at the provincial, district, and school levels, leading to strengthened teaching and learning processes at the classroom level. PRIORITAS works intensively in 53 districts in 9 provinces, including 3 districts in Papua & West Papua. The activity also engages less intensively with another 40 districts that were supported by the predecessor USAID Decentralized Basic Education (DBE) activity. See Annex 1 of this SOW for a map of PRIORITAS partners. PRIORITAS counterpart institutions include teacher training institutes (TTIs), partner schools, and national and subnational governments. Partner collaboration includes development of training modules, TTI capacity building, support to in-service teacher institutions such as LPMP (Provincial Quality Assurance Institute), developing capacities of school and district education offices for better informed decision-making, and others. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 64 3. Background and Results Framework 3.1 Background Indonesia has made progress in providing access to basic education, and the country is now close to achieving universal primary education. Enrolment rates in junior secondary school have also risen, but remain below those of neighboring countries such as Malaysia and Thailand. Strong progress has also been made towards targets for providing qualified teachers and good-standard classrooms and teaching materials across the country. Despite this progress, there remain critical gaps and variation in the quality of learning that students receive across the country, and overall student achievement is low.1 USAID PRIORITAS aims to support the overall USAID goal of expanded access to quality basic education for children in Indonesia. The project is working closely with national and local partners to: (1) improve the quality and relevance of teaching and learning in schools through pre-service and in-service training; (2) develop better management and governance in schools and districts; and (3) support better coordination within and between schools, teacher training institutes (TTIs) and government at all levels. The development hypothesis is as follows: expanded access to quality basic education will be achieved by (1) strengthening pre- and in-service teacher training programs so that more and better trained teachers are working in more classrooms, resulting in more schools offering a higher quality of instruction; (2) improving education management and governance of schools and districts, resulting in teachers receiving more and improved support to assist them to teach better; and (3) strengthening coordination at all levels of GOI agencies and education institutions, thereby improving communication including the use of media to educate public regarding the importance of quality basic education, information-based planning and policy making, feedback, and better use of financial and human resources across a decentralized system. 3.2 Results Framework The PRIORITAS Results Framework is presented in Figure 1, below: 1 OECD/Asian Development Bank (2015), Education in Indonesia: Rising to the Challenge USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 65 Exhibit 1: PRIORITAS Results Framework Over its five years, PRIORITAS has conducted many performance measurement studies and collected a large amount of data related to activity objectives. These include: • Early Grade Reading Assessment (EGRA) and student assessments in Indonesian language, Mathematics, and Science, conducted in a sample of partner and comparison primary schools. • Student assessment in Indonesian language, mathematics, and science in a sample of partner and comparison junior secondary schools. • Surveys of teaching and management practices administered in a sample of partner and comparison primary and junior secondary schools. Baseline data collection took place in 2012, followed by annual performance indicator data collection. The PRIORITAS team will submit its final indicator data to USAID in early 2017. In addition to the Regular Monitoring Database, the project also collects basic statistics about partner schools using the Project Data Management System (PDMS) which also has district-level data on basic education statistics (e.g., rates of student enrollment, transition, dropouts). 3.3 Existing Information Sources The following is a non-exhaustive list of project documents available to be reviewed by the evaluation team: 1. PRIORITAS Annual Progress Reports (2012 to 2016) 2. PRIORITAS contract and contract modifications 3. PRIORITAS PMP or Monitoring & Evaluation Plan (October 2014) 4. Map of PRIORITAS activities and partners 5. Midterm Evaluation Report (2015) USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 66 6. Midterm Evaluation SOW 7. Monitoring reports on the assessment of the impact of the USAID PRIORITAS program on schools 8. DBE Impact Evaluation Report, including Annexes (2012) 9. DBE Impact Evaluation SOW 10. Relevant studies conducted by PRIORITAS including dissemination and sustainability study of DBE and PRIORITAS program, teacher deployment reports, continuing professional development, gender study, lab and good practice schools, local government’s role and interests assessments, best practices books, impact on teacher training institutes, etc. 11. Additional information on the USAID PRIORITAS website: http://www.prioritaspendidikan.org. 4. Evaluation Purpose, Intended Use, and Audience 4.1 Evaluation Purpose The primary purpose of this final evaluation is to inform USAID, GOI counterparts, PRIORITAS’ partners, and other donors about the results PRIORITAS has achieved, about the impacts of those results on the quality and relevance of basic education in Indonesia, and about the practices and lessons from PRIORITAS that can be sustained by relevant stakeholders to continue increasing access to quality basic education in Indonesia. 4.2 Evaluation Audiences and Intended Uses • Pre-service teacher training institutes and in-service teacher education will use the evaluation findings to improve modules and approaches for improving the quality of teaching in Indonesia. • Participating schools (including dissemination schools), parents and community at large, and media will use the evaluation findings to strengthen school management and governance, and to expand awareness and implementation of high-impact learning and teaching practices in reading and literacy, math, and science, as well as school management and governance. • Government agencies, particularly Local Education Offices (Dinas Pendidikan Propinsi dan Kabupaten/Kota) and Local Religious Affairs Offices, MOEC, MORA and MORHE will use the evaluation findings to strengthen management, coordination, and feedback systems across all levels of the GOI’s decentralized education system. • Donor agencies will use the evaluation findings to learn best practices, lessons learned, and approaches used by the project to strengthen their program implementation, project design, monitoring and evaluation. • Implementing partners (RTI, EDC, and World Education) will use the evaluation findings to learn what worked and did not work in implementing the program in order to achieve the goal and objectives stated in the contract. • USAID will use the evaluation findings to generate lessons learned for basic education activity design in Indonesia or elsewhere, including identifying effective methods to measure performance and impacts of basic education initiatives such as PRIORITAS. 5. Evaluation Questions To guide this evaluation, USAID has identified the following questions: 1. How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? How has PRIORITAS strengthened the institutions (Teacher Training Institutes) to support these teachers? USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 67 2. How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS improved in developing/supporting accountable and participatory school management? 3. How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management with focus on teacher deployment, teacher continuing professional development, and educational funding? 4. How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level? 5. How have national government agencies (MOEC, MORA, MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects? By obtaining answers to these questions, USAID and other audiences hope to learn how PRIORITAS interventions have resulted in the improved capacity of teachers in teaching and learning through pre- and in￾service training, better management and governance in schools and districts, and better coordination among schools, teacher training institution, and government agencies. They will also know more about how the experiences and lessons of PRIORITAS have been and could potentially be used by partners to improve their programs and policies. 6. Gender Considerations USAID PRIORITAS has made a number of efforts to address the gender dimensions of basic education delivery into its activity components. For example, the PRIORITAS training program included a gender unit in Module 2, which served as in-service training guidance for teachers, principals, and community members. In addition, PRIORITAS conducted a gender study in response to a recommendation in the midterm evaluation. The final evaluation will examine gender issues primarily through analyzing disaggregated indicator data, and through information provided through the course of answering the Evaluation Questions through key informant interviews and group discussions. 7. Data Collection and Analysis Methods The final evaluation will employ both qualitative and quantitative approaches to answer the evaluation questions. These approaches will include: • Key informant interviews (KIIs) • Class observations • Document analysis • Secondary data analysis of activity M&E data • Possible limited surveys (online) of selected partner groups The evaluation team will validate qualitative findings by triangulating different data collection methods and inputs from key informants. There will be no specific methodology to validate quantitative findings, given the priority on qualitative findings. However, in order to minimize bias in the use of the limited online survey, some measures such as instrument pre-testing and follow-up calling during online data collection will be administered. Limitations of the evaluation include: • The small number of districts and schools sampled, compared to the large number of project locations and partners. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 68 • A primarily qualitative approach may not enable the evaluation team to calculate the net effects of PRIORITAS impacts, but will provide greater insights into why the project worked or did not work in given situations. • The presence of previous or other similar donor programs in the study area may make it difficult to evaluate the specific impacts of PRIORITAS. • The presence of evaluators during the administration of classroom observation may introduce bias (Hawthorne effect). Annex 2 presents an illustrative analytical matrix that identifies data sources, collection methods, and analysis methods for each of the evaluation questions. This matrix will be finalized by the evaluation team in its Design Proposal following the initial team planning meeting with USAID in Jakarta. Evaluation fieldwork will emphasize a qualitative approach to data collection, hence a purposive sampling strategy will be used. The evaluation team will not use random sample selection for data collection, but rather will draw a sample of activity sites and partners that reflects the diversity of locations and institutions where PRIORITAS has worked. Draft criteria for selecting fieldwork locations and partners include those listed below. These criteria and a detailed implementation plan – including travel schedule and arrangements – will be developed as part of the Design Proposal submitted by the evaluation team. Draft Criteria for Selecting Fieldwork Locations and Interview Subjects • Diversity of PRIORITAS districts and regions, with a balance of locations within and outside Java. • The evaluation team will visit all PRIORITAS districts in Papua and West Papua provinces. Activity interventions in these provinces used different mechanisms compared to the rest of the country, including grants under contract and direct assistance by the IP. • Priority given to PRIORITAS core districts, but also including previous DBE districts that received lower-intensity, more indirect support from PRIORITAS. • Balance among activity school/district cohorts (1, 2, 3, urban and rural areas), TTI partners. • Balance between the number of districts to be visited with the required time frame to complete the evaluation report, as detailed in Table II below (Gantt chart). • Other criteria, as agreed with USAID. 8. Deliverables The detailed deliverables and deliverables schedule for this PRIORITAS Final Evaluation are indicated in Table I, below. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 69 Table 1: Illustrative Deliverable and Estimated Due Date Deliverable Estimated Due Date 1. Evaluation Design Proposal, including description of the evaluation methodology, drafts of data collection instruments and sampling plan, as relevant Beginning March 2017 2. Final Evaluation Design Mid-March, after USAID inputs on proposed Design 3. Draft Evaluation Report June 2017 4. Oral presentation(s) of key findings, conclusions and recommendations from Draft Evaluation Report for USAID and implementing partners June or July 2017 5. Final Evaluation Report, including Executive Summary and recommendations in Bahasa Indonesia Two weeks after receiving USAID’s feedback on Draft Evaluation Report. Plan July 2017 6. Oral presentation(s) of key findings, conclusions and recommendations (both in Bahasa Indonesia and English) from Final Evaluation Report for USAID, MOEC, MORA, MORHE, implementing partners, donors, and broader community July 2017, per availability of GOI and IP participants 7. Datasets will be submitted to USAID as required under ADS 579 (in digital format) July 2017 All documents and reports will be provided electronically to USAID no later than the dates indicated in the approved Design Proposal. All qualitative and quantitative data will be provided in electronic format to USAID in a format consistent with Automated Directives System (ADS) 579 requirements. All oral presentations will include a formal presentation with slides delivered both electronically and in hard copy for all attendees. 9. Reporting and Dissemination The format of the Final Study Report will follow, as applicable, USAID guidelines set forth in the USAID Evaluation Report Template (http://usaidlearninglab.org/library/evaluation-report-template) and the How-To Note on Preparing Evaluation Reports (http://usaidlearninglab.org/library/how-note-preparingevaluation￾reports). Study team members will be provided with USAID’s mandatory statement of the evaluation standards they are expected to meet, shown in the text box below. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 70 USAID EVALUATION POLICY CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT • The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. • Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. Prior to the submission of the Design Proposal, the evaluation team will discuss with USAID whether its preliminary dissemination plan for this study indicates other deliverables that should be prepared, such as a summary and recommendations in Bahasa Indonesia. Such additions as agreed with USAID will then be included in the Design Proposal. 10. Team Composition The evaluation team will to have seven people, consisting of two international experts (including an international team leader) and five Indonesian experts. This composition will enable the team to split into two groups to conduct fieldwork across all target regions of Indonesia. Positions will include:  Team Leader, international  Evaluation/education specialists, one international and one Indonesian  Basic education specialists respectively with strong background in teaching and learning methodologies, and school management and governance as well as community participation, two Indonesian  Data analyst, one Indonesian; and research assistant, one Indonesian USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 71 Collective required qualifications for the overall evaluation team include the following:  Strong knowledge of Indonesia, both the basic education sector with strong background in teaching and learning methodologies, education management and governance structures, and community participation. Experience in multiple geographic areas, including Papua, is a plus.  Team leader: knowledge of USAID education programming and experience with evaluation of education activities, including successful experience leading evaluation teams.  Knowledge of qualitative evaluation school-based management (teaching and learning, school management and governance, as well as community participation) practices.  Survey design and administration skills.  Classroom observation skills.  Data analysis skills.  Ability to communicate in Indonesian.  Excellent writing and inter-personal communication skills. 11. USAID Participation Regular communication between the evaluation team and the designated USAID Activity Manager will be essential to the successful execution of the PRIORITAS evaluation. The evaluation team will keep USAID apprised of changes and developments that necessitate any significant decision-making or modification of the approved evaluation design. Possible USAID participation in the data collection phase of the evaluation will be discussed in the Team Planning Meeting (TPM), prior to the start of fieldwork. 12. Logistics and Scheduling 13.1.Logistics Support MESP will provide logistical and administration support for the team i.e., travel arrangements, accommodations, meeting spaces, et al. USAID and the PRIORITAS IP will provide list of stakeholders, introductions to program counterparts and stakeholders, background documents, activity reports, and all performance monitoring data. 13.2.Scheduling Table II below presents a Gantt chart for evaluation implementation. A final schedule, including specific deliverable due dates, will be included in the evaluation team’s Design Proposal. USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 72 Table II. Performance Evaluation Timeline Task/ Deliverable Dec 2016 Jan 2017 Feb Mar Apr May June July Evaluation SOW development Document review; initial meetings w/ USAID, IP Team Planning Meeting & Evaluation Design drafting Submission of Evaluation Design Early Mar Fieldwork In-country USAID debriefs, as requested Secondary data analysis of M&E data Analysis & report writing Submission of draft Evaluation Report End June Revision of Evaluation Report Submission of Final Evaluation Report Delivery of findings to counterparts and partners TBD TBD The MESP team and evaluation team will be responsible for procuring all logistical needs related to this study, such as work space, transportation, printing, translation, and any other forms of communication, as well as delivery of findings to counterparts and partners. USAID will offer assistance as appropriate in providing list of stakeholders, introductions to partners and key stakeholders as needed, and will ensure the provision of data and supporting documents as required. 14. Budget MESP will prepare a draft detailed budget for the evaluation following comments on this SOW by USAID. A final detailed budget will be included in the Design Proposal for USAID’s approval. 73 ANNEX II: MAP OF USAID PRIORITAS PARTNERS 74 ANNEX III: ANALYTICAL FRAMEWORK FOR THE PRIORITAS FINAL EVALUATION: GETTING TO ANSWERS Evaluation Questions Data Required Data Sources Data Collection Methods Data Analysis Methods 1. How have the competency and skills of pre-service and in-service teachers supported by PRIORITAS improved in delivering an active learning methodology? -How has PRIORITAS strengthened the institutions (Teacher Training Institutes) to support these teachers?  Competency and skills of pre-service and in￾service teachers in delivering an active learning  TTI’s capacity and competency in delivering in-service and pre-service teacher training  Project M&E data Report on TTI  Results of key informant interviews and focus group discussion (FGD) conducted at schools, districts, and TTIs  Result of limited online survey.  Annual project M&E carried out in all of PRIORITAS sampled schools  FGD/interviews with teachers, school principals, students, school committees, school supervisors, district facilitators, sub-district officers  Classroom observations  PRIORITAS project staff interviews.  Key informant interview with district and TTI facilitators  Online survey with TTIs.  Document review (MOUs, project strategy on partnership)  Analysis of quantitative data derived from project M&E data and online survey  Analysis of qualitative data, including data triangulation 2. How have the capacity and initiative of schools, community (including parents, school committees, media) and district governments supported by PRIORITAS  Improvement of management and governance at school level  School committees participation in school  Project PRIORITAS Annual Report  Good Practices Book  M&E Data  News from media  Analysis of secondary data  Document review  Key informant interviews and FGD with teachers, school principals, students, school committees,  Descriptive analysis of quantitative data  Analysis of qualitative data 75 improved in developing/supporting accountable and participatory school management? management and governance  Reports on governance and management - i.e. Collaborative Governance and Management Needs Assessment school supervisors, district facilitators, sub-district officials (e.g. UPTD), district officers (DinasPendidikan, MORA, Bappeda, BKD, DPRD, Board of Education)  PRIORITAS Project staff interviews (e.g. media) 3. How have the capacity and initiative of local governments supported by PRIORITAS improved in coordinating education resource management with focus on teacher deployment, teacher continuing professional development, and educational funding?  Data on education sector development planning at district level, particularly on teacher deployment, teacher continuing professional development, and educational funding  Report on Investigation into Teacher Deployment and Teacher Continuing Professional Development Program in Indonesia, especially from PRIORITAS partner districts  Report on allocation of funds for dissemination of teacher training  Analysis of secondary data  Key informant interviews/ FGD with school supervisors, district facilitators, district officers (Dinas Pendidikan and MORA, Bappeda, BKD, DPRD, Board of Education)  Interviews/FGDs with PRIORITAS staff  Descriptive analysis  Analysis of qualitative data 4. How are PRIORITAS’ results and practices being sustained by partner agencies and institutions at the regional level?  Data on education stakeholders continue using and adopting PRIORITAS initiatives  Latest Quarterly (Dec 2016)  Annual Report  2014 Report on Dissemination and Sustainability of DBE and USAID PRIORITAS Program  Project annual monitoring data  Document review  Analysis of secondary data  Key informant interviews and FGD with district facilitators, district officers (Dinas Pendidikan and MORA, Bappeda, DPRD)  Interviews with PRIORITAS project staff  Interview with USAID  Descriptive analysis  Analysis of qualitative data 76 5. How have national government agencies (MOEC, MORA, MORHE) benefited from PRIORITAS? How will they continue and address sustainability aspects?  Initiatives of all the government agencies to adopt and disseminate PRIORITAS results  Report on dissemination of Pembelajaran and MBS by MOEC  Decree on pilot of Reading Culture in 19 districts  MoU with MORA as a legal basis for dissemination of PRIORITAS initiatives by UINs  Decree on assignment of PRIORITAS District facilitators as the Provincial facilitators and resource persons for K13  Document review (e.g. national regulations, decrees)  Key informant interviews with MOEC, MORA, MORHE,  Interview with PRIORITAS project staffs  Interview with USAID  Descriptive analysis  Analysis of qualitative data. 77 ANNEX IV: PRIORITAS EVALUATION: SAMPLING DESIGN AND DETAILED FIELDWORK SCHEDULE SAMPLING DESIGN: Province District Mid￾term Eval location Cohort DBE partner district Primary school Secondary school TTI supported school 1 2 3 General or Madrasah General or Madrasah 1 Primary & 1 JSS North Sumatra Medan (TTI)* -- -- -- -- -- 0 0 2 LabuhanBatu √ √ -- -- -- 2 2 0 SerdangBerdagai -- -- -- √ -- 2 2 0 Central Java Semarang (TTI) -- -- -- -- -- 0 0 2 Pekalongan -- -- √ -- -- 2 2 0 Batang -- √ -- -- -- 2 2 0 East Java Malang (TTI) -- -- -- -- -- 0 0 2 Kota Batu -- -- -- √ -- 2 2 0 Lumajang -- -- √ -- -- 2 2 0 Blitar √ √ -- -- -- 2 2 0 South Sulawesi Makassar (TTI) -- -- -- -- -- 0 0 2 Maros √ √ -- -- -- 2 2 0 Pinrang -- -- -- -- √ 2 2 0 Pare Pare -- -- √ -- -- 2 2 0 Total 20 20 8 48 78 Detailed Fieldwork Schedule – Team A (North Sumatra and East Java) TEAM A Sun 12 Mar Mon 13 Mar Tues 14 Mar Weds 15 Mar Thurs 16 Mar Fri 17 Mar Sat 18 Mar Travel from Jakarta to North Sumatra  Transfer to Labuhan Batu (8 hours by train)  2 SMP/MTs in Labuhan Batu  District level interviews/ FGD  2 MI in LabuhanBatu  Sub district level interviews/FGD in Labuhan Batu  Transfer to Serdang Berdagai (7 hours)  2 SMP/MTs in Serdang Berdagai  District level interviews/FGD in Serdang Berdagai  2 SD/MI in Serdang Berdagai  Sub district level interviews/ FGD in Serdang Berdagai  Transfer to Medan Sun 19 Mar Mon 20 Mar Tues 21 Mar Weds 22 Mar Thurs 23 Mar Fri 24 Mar Sat 25 Mar  Entry data/compiling  1 SD/MI & 1 SMP/MTs TTI supported schools in Deli Serdang (of one TTI, either UN Medan or UIN Sumut)  2 TTI interviews at Universitas Negeri Medan & UIN Sumatera Utara Travel From Medan to Surabaya (East Java)  TTI interview at UIN Sunan Ampel Surabaya  Transfer to Malang (2 hours)  1 SD/MI & 1 SMP/MTs Supported TTI in Malang  TTI Interview at Universitas Negeri Malang  2 SMP/MTs in Kota Batu  District level interviews/FGD in Kota Batu  2 SD/MI in Kota Batu  Sub district level interviews/FGD in Kota Batu Sun 26 Mar Mon 27 Mar Tues 28 Mar Weds 29 Mar Thurs 30 Mar Fri 31 Mar Sat 1 April  Transfer to Blitar (2-3 hours)  Entry data/compiling  2 SMP/MTs in Blitar  District level interviews/FGD in Blitar  Bali’s Day of Silence and Hindu New Year (school off day)  Entry data/compiling  2 SD/MI in Blitar  Sub district level interviews/FGD in Blitar  Transfer to Lumajang (5-6 hours)  2 SMP/MTs in Lumajang  District level interviews/FGD in Lumajang  2 SD/MI in Lumajang  Sub district level interviews/FGD in Lumajang  Transfer to Surabaya (4-5 jam) 79 TEAM A Sun 12 Mar Mon 13 Mar Tues 14 Mar Weds 15 Mar Thurs 16 Mar Fri 17 Mar Sat 18 Mar Sun 2 April Back to Jakarta Detailed Fieldwork Schedule –Team B (South Sulawesi and Central Java) TEAM B Sun 12 Mar Mon 13 Mar Tues 14 Mar Weds 15 Mar Thurs 16 Mar Fri 17 Mar Sat 18 Mar Travel from Jakarta to Makassar (South Sulawesi)  2 TTI interviews at Universitas Negeri Makassar & UIN Sultan Alauddin  1 SD/MI & 1 SMP/MTs TTI supported schools in Makassar  2 SD/MI in Maros  Sub district level interviews/FGD in Maros  District level interviews/FGD in Maros  2 SMP/MTs in Maros  Transfer to Pare-pare (4 hours)  2 SD/MI in Pare￾pare  Sub district level interviews/FGD in Pare-pare  Entry data/compiling Sun 19 Mar Mon 20 Mar Tues 21 Mar Weds 22 Mar Thurs 23 Mar Fri 24 Mar Sat 25 Mar  Entry data/compiling  2 SMP/MTs in Pare-pare  District level interviews/FGD in Pare-pare  2 SD/MI in Pinrang  Sub district level interviews in Pinrang  District level interviews/FGD in Pinrang  2 SMP/MTs in Pinrang  Transfer to Makassar (4 hours) Travel From Makassar to Semarang (Central Java)  2 TTI interviews at Universitas Negeri Semarang & UIN Walisongo Semarang  1 SD/MI & 1 SMP/MTs TTI supported schools in Semarang  Transfer to Batang (2 hours) Sun 26 Mar Mon 27 Mar Tues 28 Mar Weds 29 Mar Thurs 30 Mar Fri 31 Mar Sat 1 April 80 TEAM B Sun 12 Mar Mon 13 Mar Tues 14 Mar Weds 15 Mar Thurs 16 Mar Fri 17 Mar Sat 18 Mar  Entry data/compiling  2 SMP/MTs in Batang  District level interviews/FGD in Batang  Bali’s Day of Silence and Hindu New Year (school off day)  Entry data/compiling  2 SD/MI in Batang  Sub district level interviews/FGD in Batang  Transfer to Pekalongan (2 hours)  2 SMP/MTs in Pekalongan  District level interviews/FGD in Pekalongan  2 SD/MI in Pekalongan  Sub district level interviews/FGD in Pekalongan  Transfer to Semarang (2-3 hours) Back to Jakarta 81 ANNEX V: DATA COLLECTION INSTRUMENTS FGD Guide – District Level Officials District Education Office, District Religious Affairs Office, Regional Development Planning Board, Regional Human Resources Board, District Education Council, and Local Parliament (Education Commission and Budget Commission) 60 minutes FGD ID: Evaluator ID: FGD date: Town: District: Province: Interviewee name: Interviewee gender M/F: Interviewee position title: Interviewee HAS / HAS NOT personally participated in PRIORITAS training (circle one): 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Time interview commenced: Time interview concluded: 1 (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2 (RQ1) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they may be used in your school/madrasah? Give some examples of active teaching and learning that teachers may use in school/madrasah. Interviewer’s notes in English: 3 (RQ1) (Bahasa Indonesia) What are your views about the school and budget management methods promoted by PRIORITAS, and the ways that they are being used in schools/madrasahs in your district?. 82 Give some examples of PRIORITAS school and budget management methods that you have observed principals and school committees using in schools/madrasahs in your district. Interviewer’s notes in English: 4 (RQ 3) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS reading program, and how it is being used in schools/madrasahs in your district? Interviewer’s notes in English: 5 (RQ2) (Bahasa Indonesia) What are your opinions about PRIORITAS efforts to improve teacher deployment and teacher professional development in your district? Interviewer’s notes in English: 6 (RQ4) (Bahasa Indonesia) What are your opinions about the usefulness of PRIORITAS in supporting your district to improve the quality of education? Interviewer’s notes in English: 7 (RQ4) (Bahasa Indonesia) Provide some of your predictions about concrete ways that the educational and management methods, and materials developed under PRIORITAS will continue to be promoted in schools/madrasahs in your district after the project ends Interviewer’s notes in English: 8 (RQ5) (Bahasa Indonesia) Could you think of any examples of challenges or obstacles your district may have promoting the educational and management methods, and materials developed under PRIORITAS in schools/madrasahs after the project ends? Interviewer’s notes in English: 9 (Bahasa Indonesia) Please give us any examples of the impact of PRIORITAS activities on policies at district level, for example in your own offices? Interviewer’s notes in English: 10 (RQ2) (Bahasa Indonesia) Are there any further comments or observations about PRIORITAS that you might like to make? Interviewer’s notes in English: 11 (RQ2) (Bahasa Indonesia)I’d like to now ask you a question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your district Interviewer’s notes in English: 12 (Bahasa Indonesia) The interview is now concluded. Your comments today have been valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion statement inserted here) 83 Interview Guide – District Facilitator 30 minutes Interview ID: Evaluator ID: Interview date: Name & level of school: Type of school (circle): PRIORITAS-supported, or TTI-supported Town/sub-district: District: Province: Interviewee name: Interviewee gender M/F: Interviewee position title: Interviewee HAS / HAS NOT personally participated in PRIORITAS training (circle) Interviewee’s length of career service in years. Time interview commenced: Time interview concluded: 1. (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2. (RQ1) (Bahasa Indonesia) What type of training have already been provided by PRIORITAS for in-service teachers? Interviewer’s notes in English: 3. (RQ1) (Bahasa Indonesia) How has PRIORITAS strengthened your capacity to improve the quality of education? Give some examples from school that you are mentoring. Interviewer’s notes in English: 4. (RQ2) (Bahasa Indonesia) Provide some of observation or examples of improvement in community participation a result of school’s involvement with the PRIORITAS project? Give some examples of how this has occurred in school that you are mentoring. Interviewer’s notes in English: 5. (RQ4) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they may be used in school that you are mentoring? Give some examples of active teaching and learning that your teachers use in school that you are mentoring. Interviewer’s notes in English: 6. (RQ 4) (Bahasa Indonesia) What are your views about the school and budget management methods promoted by PRIORITAS, and the ways that they are being used in schools/madrasahs in your district?. Give some examples of PRIORITAS school and budget management methods that you have observed principals and school committees using in schools/madrasahs in your district. Interviewer’s notes in English: 7. (RQ 1) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS reading program, and how it was used in school that you are mentoring. Interviewer’s notes in English: 84 8. (RQ4) (Bahasa Indonesia) Provide some of your predictions about concrete ways that the educational ideas, methods, and materials developed under PRIORITAS will continue to be used in school that you are mentoring after the project ends. Interviewer’s notes in English: 9. (Bahasa Indonesia) Are there any further comments or observations about PRIORITAS that you might like to make? Interviewer’s notes in English: 10. (RQ2) (Bahasa Indonesia) I’d like to now ask you a question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Interviewer’s notes in English: 11. (RQ2) (Bahasa Indonesia) Do you know of journalist visiting schools you supervise to collect information about PRIORITAS? Interviewer’s notes in English: 12. (Bahasa Indonesia) The interview is now concluded. Your comments today have been valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion statement inserted here) 85 Interview Guide – TTI Leader 1. NAME OF INSTITUTION : 2. NAME OF INTERVIEWEE : 3. DISTRICT/CITY : 4. PROVINCE : 5. DAY/DATE : 6. EVALUATOR : 1 (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2 What is your overall opinion /impression about PRIORITAS programs? (RQ.2) 3 What are the major programs of PRIORITAS targeting improvement of in-service teachers? Provide examples (e.g. specific activities conducted, numbers of trainees, timing of activities, training materials distributed, etc.). (RQ.1) 4 What are the major PRIORITAS activities/programs targeting the improvement of pre-service teachers? Provide examples (e.g. specific activities conducted, numbers of trainees, timing of activities, training materials distributed, etc.). (RQ.1) 5 Is there any special program offered by PRIORITAS to strengthen your INSTITUTION to improve the quality of pre-service and in-service teachers? (RQ.3,RQ.2) Please provide specific examples (e.g. specific activities conducted, numbers of trainees, timing of activities, training materials distributed, etc.) 6 Describe how the PRIORITAS project has impacted the capacities/skills of the following groups. Please provide examples for each only IF the PRIORITAS project has had an impact:  Your institution  lecturers admin  students or pre-service teachers  learning and teaching process  library  laboratory  others (please describe) (RQ.1, RQ.2, RQ.5) 7 How would your institution sustain /replicate/continue the PRIORITAS programs? What’s your strategic plans to do it when the project is over and USAID does not provide support? Please provide specific examples, focusing on In-service and pre-service training, coordination with GOI and financial support. (RQ.4) 86 8 With your institution, what are the most/least successful PRIORITAS activities? Please provide specific examples and your reasons. (RQ.2) 9 Anything else you would like to say about PRIORITAS, from different point of views/perspectives? (RQ.1, RQ.2, RQ.3, RQ.4, RQ.5) 10 (Bahasa Indonesia) Our discussions about PRIORITAS are now concluded. Your comments today have been valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion statement inserted here) 87 Interview Guide – TTI Facilitator 1 NAME OF INSTITUTION : 2 NAME OF INTERVIEWEE : 3 DISTRICT/CITY : 4 PROVINCE : 5 DAY/DATE : 6 EVALUATOR : (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (- ) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 1 As a TTI facilitator, what is your overall impression with the programs/ trainings provided by PRIORITAS? (RQ.2) 2 How did PRIORITAS train TTI lecturers to be facilitators (provide specific examples of the timing of activities, the content of the activities, the materials distributed, and your opinions on the quality of these activities)? (RQ. 1, RQ.4) 3 How has PRIORITAS improved the capacity of pre-service and in-service teachers in teaching reading, math and science? (e.g. one example is applying active learning techniques). Provide specific examples. (RQ.1) 4 How has PRIORITAS improved the capacity of pre-service and in-service teachers in school management and governance? Provide specific examples. (RQ.1, RQ.2, RQ.3) 5 How has PRIORITAS improved the capacity of school directors/leaders in school management and governance? Provide specific examples. (RQ.2, RQ.3) 6 How has PRIORITAS supported the capacity of school stakeholders to improve accountability and participatory school management? (RQ.2, RQ.3) 7 How has PRIORITAS supported the capacity and initiative of local government to improve coordination education resource management focusing on teacher deployment and development? (RQ.3, RQ.5) 8 Her/his observations about news media coverage of PRIORITAS? (RQ.2, RQ.3, RQ.4, RQ.5) 9 How can schools, local GOI/TTIs/MORA sustain/replicate/continue the PRIORITAS programs in the future, without support and funding from USAID? Do you know of any examples where PRIORITAS programs are planned and implemented without USAID support (e.g. by the GOI)? Please describe this in detail. (RQ.4, RQ.5) 88 10 What will the challenges be for schools, local GOI/TTIs/MORA to continue/sustain/replicate PRIORITAS programs without support and funding from USAID? (RQ.4, RQ.5) 11 What resources or support does your agency need to continue PRIORITAS activities, once USAID ends the program? (RQ.4, RQ.5) 12 What are the most successful PRIORITAS activities/programs to: 13 To the best of your knowledge as a facilitator, what are the least successful PRIORITAS programs in those aspects, if any? Provide specific examples to support your opinion. (RQ.1, RQ.2, RQ.2) 14 Anything else you would like to say about PRIORITAS from different perspective? (RQ.1, RQ.2, RQ.3, RQ.4, RQ.5) 15 (Bahasa Indonesia) Our discussions about PRIORITAS are now concluded. Your comments today have been valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion statement inserted here) 89 Interview Guide – Head of UPTD 30 minutes Interview ID: Evaluator ID: 02 Interview date: March 17, 2017 Name & level of school: SDN 2 Soreang Type of school (circle): PRIORITAS-supported Town/sub-district: Soreang District: Pare-pare Province: South Sulawesi Interviewee name: Supini Interviewee gender M/F: M Interviewee position title: Head of UPTD in SoreangSubdistrict Interviewee HAS personally participated in PRIORITAS training (circle) Interviewee’s length of career service in years. 4 years Time interview commenced: Time interview concluded: 1 (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2 (RQ1) (Bahasa Indonesia) What are your opinions about any type of PRIORITAS training, training materials, and follow up in your school by PRIORITAS facilitators after the training? Give some examples from your school. Interviewer’s notes in English: 3 (RQ2) (Bahasa Indonesia) Have the management methods in your school and institution changed as a result of your school’s involvement with the PRIORITAS project? Give some examples of how this has occurred in your school. Interviewer’s notes in English: 4 (RQ2) (Bahasa Indonesia) Can you think of any examples of improvement in community participation as a result of your school’s involvement with the PRIORITAS project? Interviewer’s notes in English: 5 (RQ1) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they may be used in your school? Give some examples of active teaching and learning that your teachers use in your school. Interviewer’s notes in English: 6 (RQ4) (Bahasa Indonesia) Can you think of any examples of challenges or obstacles to using active teaching and learning methods in your school, now or in the future? Interviewer’s notes in English: 7 (RQ1) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS reading program, and how it was used in your school. Interviewer’s notes in English: 8 (RQ4) (Bahasa Indonesia) Provide some of your predictions about concrete ways that the educational ideas, methods, and materials developed under PRIORITAS will continue to be used in your school and institution after the project ends. 90 Interviewer’s notes in English: 9 (RQ3) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS teacher development, teacher professional development and educational funding, and how it support improving quality education. Interviewer’s notes in English: 10 (Bahasa Indonesia) Are there any further comments or observations about PRIORITAS that you might like to make? Interviewer’s notes in English: 11 (RQ2) (Bahasa Indonesia) I’d like to now ask you a question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Interviewer’s notes in English: 12 (RQ2) (Bahasa Indonesia) Do you know of journalist visiting your schools or institution to collect information about PRIORITAS? Interviewer’s notes in English: 13 (Bahasa Indonesia) The interview is now concluded. Your comments today have been valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released.(Audio recording conclusion statement inserted here) 91 Interview Guide – School Supervisor (SD/MI and SMP/MTs) 30 minutes Interview ID: Evaluator ID: 07 Interview date: March 16, 2017 Name & level of school: SMPN 22 Bantimurung Type of school (circle): PRIORITAS-supported Town/sub-district: Bantimurung District: Maros Province: South Sulawesi Interviewee name: Muhammad Dahlan Interviewee gender M/F: M Interviewee position title: School supervisor Interviewee HAS personally participated in PRIORITAS training (circle) Interviewee’s length of career service in years. 2 years Time interview commenced: Time interview concluded: 1. (Audio recording commencement statement inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2. (RQ1) (Bahasa Indonesia) What are your opinions about any type of PRIORITAS training, training materials, and follow up in schools/madrasahs that you supervise by PRIORITAS facilitators after the training? Give some examples from schools/madrasahs that you supervise. Interviewer’s notes in English: 3. (RQ2) (Bahasa Indonesia) Have the management methods in your school/madrasah changed as a result of school/madrasah’s involvement with the PRIORITAS project? Give some examples of how this has occurred in schools/madrasahs that you supervise. Interviewer’s notes in English: 4. (RQ2) (Bahasa Indonesia) Can you think of any examples of improvement in community participation a result of school/madrasah’s involvement with the PRIORITAS project? Give some examples of how this has occurred in schools/madrasahs that you supervise. Interviewer’s notes in English: 5. (RQ4) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they may be used in schools/madrasah that you supervise? Give some examples of active teaching and learning that your teachers use in schools/madrasahs that you supervise. Interviewer’s notes in English: 6. (RQ4) (Bahasa Indonesia) Can you think of any examples of challenges or obstacles to using active teaching and learning methods in schools/madrasahs that you supervise, now or in the future? Interviewer’s notes in English: 7. (RQ1) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS reading program, and how it was used in schools/madrasahs that you supervise. Interviewer’s notes in English: 92 8. (RQ4) (Bahasa Indonesia) Provide some of your predictions about concrete ways that the educational ideas, methods, and materials developed under PRIORITAS will continue to be used in your school/madrasah after the project ends. Interviewer’s notes in English: 9. (RQ3) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS teacher development, teacher professional development and educational funding, and how it support improving quality education. Interviewer’s notes in English: 10. (Bahasa Indonesia) Are there any further comments or observations about PRIORITAS that you might like to make? Interviewer’s notes in English: 11. (RQ2) (Bahasa Indonesia) I’d like to now ask you a question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Interviewer’s notes in English: 12. (RQ2) (Bahasa Indonesia) Do you know of journalist visiting schools you supervise to collect information about PRIORITAS? Interviewer’s notes in English: 13. (Bahasa Indonesia) The interview is now concluded. Your comments today have been valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion statement inserted here) 93 Interview Guide – School Principal 30 minutes Interview ID: Evaluator ID: Interview date: Name & level of school: Type of school (circle): PRIORITAS-supported, or TTI-supported Town/sub-district: District: Province: School principal’s name: Principal’s gender M/F: Principal’s position title: Principal (circle) HAS NOT personally participated in PRIORITAS school manager training Principal’s career length of service in years. Time interview commenced: Time interview concluded 1. (Audio recording commencement marker inserted here) (Bahasa Indonesia) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2. (RQ1) (Bahasa Indonesia) What are your opinions about any type of PRIORITAS training, the training materials, and follow up in your school by PRIORITAS facilitators after the training? Give some examples from your school. Interviewer’s notes in English: 3. (RQ2) (Bahasa Indonesia) Have any management or budgeting methods in your school changed as a result of your school’s involvement with the PRIORITAS project? Give some examples of how this has occurred in your school. Has the School Committee become more involved? Interviewer’s notes in English: 4. (RQ1) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they are being used in your school in Bahasa Indonesia, Mathematics, Science, and Reading classes? Give some examples of active teaching and learning that your teachers used/use/will use in your school. Interviewer’s notes in English: 5. (RQ1) (Bahasa Indonesia) Can you think of any examples of challenges or obstacles to using active teaching and learning methods in your school, now or in the future? Interviewer’s notes in English: 6. (RQ1) (Bahasa Indonesia) Provide some of your observations about the PRIORITAS reading program, and how it was used in your school. Interviewer’s notes in English: 7. (RQ1) (Bahasa Indonesia) Provide some of your predictions about concrete ways that the educational ideas, methods, and materials developed under PRIORITAS will continue to be used in your school after the project ends. 94 Interviewer’s notes in English: 8. (all RQs) (Bahasa Indonesia) Are there any further comments or observations about PRIORITAS that you might like to make? Interviewer’s notes in English: 9. (RQ2) (Bahasa Indonesia) I’d like to now ask you a question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. Interviewer’s notes in English: 10. (Bahasa Indonesia) The interview is now concluded. Your comments today have been very valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released.(Audio recording conclusion marker inserted here) 95 Lesson Observation in Classroom Guide SPECIFICATIONS  Number of evaluators observing each lesson: 1  Number of lessons to be observed in each school: 2 separate classes observed simultaneously each by 1 evaluator.  Observation duration: duration of class, 45 minutes  Essential characteristics of lesson to be observed: a lesson on Bahasa Indonesia, Mathematics, Science, or Reading being taught by a teacher trained under PRIORITAS.  Observation purpose: to determine if good practices in teaching and learning as defined and promoted by PRIORITAS are occurring or not, and if students do or do not demonstrate the positive learning behaviors as defined and promoted by PRIORITAS.  Observation format & style: unobtrusive, silent, seated at rear of classroom.  Language used by observers: all observation instrument items and notes are written in English.  Observation items: evaluators will determine if any of the good teaching and learning practices encouraged by PRIORITAS are observed generally occurring or not generally occurring: o The PRIORITAS definition of good teaching practices is:  Organizing the physical classroom to facilitate interactive learning (furniture, teaching aids, displays)  Using a mix of whole class/group/ partner and individual work with students  Asking non recall questions and allowing students time to answer  Using varied learning approaches (other than lecturing and text book) such as giving open ended tasks, using the environment and using learning aids  Using tools to gather data about student achievement  Moving around the room, observing and assisting students to complete their tasks o The PRIORITAS definition of good reading teaching practices is:  Providing specific instruction appropriate to the learner to build word knowledge and teach word analysis  Providing opportunities for students to engage in sustained reading activities to practice their reading skills  Creating a literacy-rich classroom environment  Checking students comprehension on what they are reading  Reading aloud to students/asks students to read aloud using a range of materials to enhance children’s print and phonological awareness  Conducting regular and purposeful monitoring of children’s progress in reading o The PRIORITAS definition of students’ positive learning behaviors is:  Students are engaged in their task (not easily distracted)  Students undertake activities which involve problem solving  Students’ work is the result of their own thinking (e.g. written in their own words)  They express their feelings and opinions during lessons or ask questions (verbally)  They participate in cooperative activities such as experiments or discussion  Collation, analysis, & reporting of completed observations in English: TBD Evaluation instrument follows on next page. 96 Observation ID: Evaluator ID: Observation date: Name & level of school: Type of school (circle): PRIORITAS￾supported, or TTI-supported Town/sub-district: District: Province: DETAILS OF CLASS OBSERVED Teacher’s name & gender (M or F) Subject taught (tick) Grade level Is teacher PRIORITAS trained? YES NO Bahasa Indonesia Mathematics Science Other (describe) Time observation commenced: Time observation concluded: GOOD TEACHING PRACTICE CHECKLISTS (Please TICK one RH column if observed) (RQ1) TEACHING – GENERAL (BAHASA INDONESIA, MATHEMATICS, SCIENCE, OR OTHER) Items to observe Observed generally occurring Not observed generally occurring Organizing the physical classroom to facilitate interactive learning (furniture, teaching aids, displays) Using a mix of whole class/group/ partner and individual work with students Asking non recall questions and allowing students time to answer Using varied learning approaches (other than lecturing and text book) such as giving open ended tasks, using the environment and using learning aids Using tools to gather data about student achievement Moving around the room, observing and assisting students to complete their tasks (RQ1) TEACHING – READING (COMPLETE ONLY IF READING IS BEING TAUGHT) Items to observe Providing specific instruction appropriate to the learner to build word knowledge and teach word analysis Providing opportunities for students to engage in sustained reading activities to practice their reading skills Creating a literacy-rich classroom environment Checking students comprehension on what they are reading Reading aloud to students/asks students to read aloud using a range of materials to enhance children’s print and phonological awareness Conducting regular and purposeful monitoring of children’s progress in reading. (RQ1) STUDENTS’ POSITIVE LEARNING BEHAVIOURS CHECKLIST Items to observe Students are engaged in their task (not easily distracted) Students undertake activities which involve problem solving Students’ work is the result of their own thinking (e.g. written in their own words) They express their feelings and opinions during lessons or ask questions (verbally) They participate in cooperative activities such as experiments or discussion 97 FGD Guide – Teachers at School 45-60 minutes FGD ID: Evaluator IDs: FGD date: Type and name of school: Type of school (circle): Town/sub-district: District: Province: Teacher name Subjects taught if SMP Grade level taught Career length of service in years PNS YES or NO Gender M or F: Types of PRIORITAS training/s attended Time FGD commenced: Time FGD concluded: 1 (Audio recording commencement marker inserted here) (Bahasa Indonesia) My name is (-) and my colleague’s name is (-). We’re assisting with a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify any impacts of PRIORITAS for your school, and identify possible ways PRIORITAS activities will be replicated and sustained in your schools after the project ends. I will suggest some discussion topics, and invite you to discuss them openly and honestly. There are no right or wrong comments or answers. Your identities are protected, and in our report to USAID your names will not be linked to any comments you make today. We will use a recording device to record your comments. Do I have your permission to now proceed with a recorded discussion which may take about 45-60 minutes? 2 (RQ1) (Bahasa Indonesia) What are your opinions about PRIORITAS trainings, training materials, and follow up in your schools by PRIORITAS facilitators and by your own school managers after the training? a. Ask the teachers to provide examples of the usefulness for them of the activities at training events. b. Ask the teachers to provide examples of post training follow-up by PRIORITAS staff that occurred in their schools, and how frequently they occurred. Evaluator’s notes in English: 3 (RQ1) (Bahasa Indonesia) What are your views about the active teaching and learning methods promoted by PRIORITAS, and the ways that they were used/are used/will be used in your schools in Bahasa Indonesia, Mathematics, Science, and Reading classes? Evaluator’s notes in English 4 (RQ1) (Bahasa Indonesia) What are some examples of PRIORITAS active teaching and learning methods that you or other teachers use in your school for teaching all subjects? If they use active learning and teaching methods, ask if they’ve noticed any changes in student learning outcomes, such as in academic and non-academic performance (critical thinking, collaboration, leadership) Evaluator’s notes in English: 5 (RQ1.) (Bahasa Indonesia) Can you think of any examples of challenges or obstacles to using PRIORITAS active teaching and learning methods in your schools now or in the future? Evaluator’s notes in English: 98 6 (RQ1.) (Bahasa Indonesia) Can you think of any examples of the usefulness for you in your classrooms of any kind of PRIORITAS assistance and support? Evaluator’s notes in English: 7 (RQ1.) (Bahasa Indonesia) What are some of your observations about the PRIORITAS reading program that was introduced to your schools? Evaluator’s notes in English: 8 (RQ1.) (Bahasa Indonesia) What are some of your predictions about how the educational ideas, methods, and materials developed under PRIORITAS will continue to be used in your schools after the project ends. Evaluator’s notes in English: 9 (RQ1) (Bahasa Indonesia) Dibawah PRIORITAS, KelompokKerja Guru (KKG)/Musyawarah Guru Mata Pelajar (MGMP) mungkinmenjadi forum untukberbagi ide danmetodedalampelaksanaanmetode PRIORITAS di sekolahAndadansekolah lain. ApapendapatAndamengenaikegunaan KKG/MGMP dalammencapaihaltersebut? Evaluator’s notes in English/Indonesian: 10 (all RQs) (Bahasa Indonesia) Any there any general comments or observations about PRIORITAS that you might like to make? Evaluator’s notes in English: 11 (RQ2.) (Bahasa Indonesia) I’d now like to ask you a final question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. Evaluator’s notes in English: 12 (Bahasa Indonesia) Our discussions about PRIORITAS are now concluded. Your comments today have been very valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion marker inserted here) 99 FGD Guide – School Committee Members 30-40 minutes FGD ID: Evaluator IDs: FGD date: Name & level of school: Type of school (circle): Town/sub-district: District: Province: School Committee member names Length of involvement with school (in years) Occupation Gender M or F: HAS or HAS NOT attended PRIORITAS training/s attended Time FGD commenced: Time FGD concluded: 1 (Audio recording commencement marker inserted here) (Bahasa Indonesia) My name is (-) and my colleague’s name is (-). We’re assisting with a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify any impacts of PRIORITAS for your school, and identify possible ways PRIORITAS activities will be replicated and sustained in your school after the project ends. I will suggest some discussion topics, and invite you to discuss them openly and honestly. There are no right or wrong comments or answers. Your identities are protected, and in our report to USAID your names will not be linked to any comments you make today. We will use a recording device to record your comments. Do I have your permission to now proceed with a recorded discussion which may take about 30-40 minutes? 2 (RQ1) (Bahasa Indonesia) The principal, and some of the teachers at your school, perhaps all of them, have been attending PRIORITAS training activities for some time. Were any of you aware that this has been occurring? Did you attend any PRIORITAS training? Does anyone know what all this training was for? (Explain it was to enable teachers to use better methods for teaching Bahasa Indonesia, mathematics, science, and reading; and also to help improve management and budget planning at the school.) Evaluator’s notes in English: 3 (RQ2) (Bahasa Indonesia)ThePRIORITAS project has been promoting a new type of school management method that is more accountable, and more participatory. In this respect, can you provide examples of ways your school committee has become more involved with managing the school? Evaluator’s notes in English: 4 (RQ2) (Bahasa Indonesia) Can you give any examples of good or bad changes you’ve observed in the management and financial budgeting methods used in your school since the time it became involved with the PRIORITAS project? Do you think the changes occurred because of PRIORITAS, or because of other influences on the school? (for example, from a district Education Office, or from the committee itself) Evaluator’s notes in English: 5 (RQ1) (Bahasa Indonesia) What are your opinions about the active teaching and learning methods promoted by PRIORITAS, and the ways that they were used/are used/will be used in your school for Bahasa Indonesia, Mathematics, Science, or Reading classes? Evaluator’s notes in English 100 6 (RQ1) (Bahasa Indonesia) Can you think of any examples of challenges or obstacles to using PRIORITAS active teaching and learning methods in your school now or in the future? Evaluator’s notes in English: 7 (RQ1) (Bahasa Indonesia) Can you provide any examples of the usefulness for your school of any kind of PRIORITAS assistance and support for any aspect of the school? Evaluator’s notes in English: 8 (RQ1) (Bahasa Indonesia) What are some of your observations about the PRIORITAS reading program that was introduced to your school? Evaluator’s notes in English: 9 (RQ1) (Bahasa Indonesia) What are some of your predictions about how the educational ideas, methods, and materials developed under PRIORITAS will definitely continue to be used in your school after the project ends. Evaluator’s notes in English: 10 (all RQs) (Bahasa Indonesia) Any there any general comments or observations about PRIORITAS that you might like to make? Evaluator’s notes in English: 11 (RQ2) (Bahasa Indonesia) I’d now like to ask you a final question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. Evaluator’s notes in English: 12 (Bahasa Indonesia) Our discussions about PRIORITAS are now concluded. Your comments today have been valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion marker inserted here) 101 FGD Guide – School Students 30-40 minutes FGD ID: Evaluator IDs: FGD date: Name & level of school: Type of school (circle): Town/sub-district: District: Province: Student name Age Grade level Gender M or F: Time FGD commenced: Time FGD concluded: 1 (Audio recording commencement marker inserted here) (Bahasa Indonesia) My name is (-) and my colleague’s name is (-). We’re assisting with a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify any impacts of PRIORITAS for your school, and identify possible ways PRIORITAS activities will be replicated and sustained in your school after the project ends. I will suggest some discussion topics, and invite you to discuss them openly and honestly. There are no right or wrong comments or answers. Your identities are protected, and in our report to USAID your names will not be linked to any comments you make today. We will use a recording device to record your comments. Do I have your permission to now proceed with a recorded discussion which may take about 30-40 minutes? 2. (RQ1) (Bahasa Indonesia) Some of your teachers, perhaps all of them, have been attending PRIORITAS training activities for some time. Were any of you aware that this has been occurring? Does anyone know what this training was for? (Say it was to enable teachers to use better methods for teaching Bahasa Indonesia, Mathematics, Science, and Reading.) 3. Evaluator’s notes in English:(RQ1) (Bahasa Indonesia) The PRIORITAS methods for teaching Bahasa Indonesia, Mathematics, Science, and Reading are meant to involve you much more in your own learning, instead of the teacher just talking, and you just passively listening and memorizing facts. For example, during training your teachers were encouraged to invite you to ask many different kinds of questions during classes, and help you to find possible answers to those questions by yourselves. Can you think of any examples where changes like these have been happening during your classes? Evaluator’s notes in English 4. (RQ1) (Bahasa Indonesia) Your teachers who attended PRIORITAS training activities were encouraged to use other different ways for teaching Bahasa Indonesia, Mathematics, Science, and Reading. For example, nowadays they might arrange for you to often learn together in small groups; or provide you with many different kinds of learning projects, models, and experiments; or ask you to design and make your own. Can you think of any examples where changes like these have been happening during your Bahasa Indonesia, Mathematics, Science, and Reading classes? Evaluator’s notes in English: 5. (RQ1) (Bahasa Indonesia) Can you think of any recent examples where you have felt that learning Bahasa Indonesia, Mathematics, Science, and Reading has become easier, more interesting, or even exciting for you? 102 Evaluator’s notes in English: 6. (RQ1) (Bahasa Indonesia) During the PRIORITAS project your school became involved in a special program to encourage you to read books. Can you think of any examples where you have been aware that this was happening (e.g. ‘reading corners’)? Can you think of examples when you have become more interested in reading books? Can you remember having read any books given to your school by PRIORITAS? Evaluator’s notes in English: 7. (all RQs) (Bahasa Indonesia) Any there any general comments or observations about the PRIORITAS project that you might like to make? Evaluator’s notes in English: 8. (RQ2) (Bahasa Indonesia) I’d now like to ask you a final question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. Evaluator’s notes in English: 9 (Bahasa Indonesia) Our discussions about PRIORITAS are now concluded. Your comments today have been valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion marker inserted here) 103 Interview Guide – Whole School Development Program Coordinator of PRIORITAS 60 minutes Interview ID: Evaluator ID: Interview date: PRIORITAS CENTRAL OFFICE: RATU PLAZA, JL. JEND. SUDIRMAN JAKARTA DKI: Coordinator’s name: Gender M/F: Position title: Whole School Development Program Coordinator Coordinator’s professional background (brief summary): Time interview commenced: Time interview concluded: 1. (Audio recording commencement marker inserted here) My name is (-) and I’m helping conduct a final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your school after the project ends. I wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions I ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. I will use a recording device to record your comments. Do I have your permission to now proceed with a recorded interview which should take about 30 minutes? 2. (RQ1) To start this interview, please briefly tell us what your duties are as the Whole School Development Program Coordinator for PRIORITAS. Please break down your duties into your main areas of responsibility. (ARE SCHOOL MANAGEMENT ACTIVITIES INCLUDED?) Interviewer’s notes in English: 3. (all RQs) We’d like to know something about your personal involvement in implementing and promoting the PRIORITAS whole school development program: a. Briefly describe the ways you have been personally involved in designing and implementing PRIORITAS training activities for PRIORITAS field facilitators, schools, school principals, school teachers, and others involved with the schools. Interviewer’s notes in English b. How frequently did you personally visit PRIORITAS schools, for example to meet with school staff to discuss aspects of whole school development, and obtain feedback to better inform your decisions? Interviewer’s notes in English c. How frequently did you personally visit district education offices in PRIORITAS Districts and Sub-districts, for example to meet with district education offices staff and supervisors to monitor developments and obtain feedback to better inform your decisions? Interviewer’s notes in English 4. How frequently did you personally visit province-level education office in PRIORITAS provinces? Can you give some examples of such meetings? How frequently do you personally visit or meet with officials at relevant national-level government departments in Jakarta? Can you give some examples of such meetings? Interviewer’s notes in English 5. (RQ1) In general, what do you think have been the main strengths and weaknesses of the various types of PRIORITAS training activities, the training materials, and training follow up that you supervised? a. Main general strengths 104 b. Main general weaknesses Interviewer’s notes in English: 6. (RQ1) What do you think have been the main strengths and weaknesses in the ways PRIORITAS￾supported schools have attempted to adopt the active teaching and learning methods promoted by PRIORITAS for teaching Bahasa Indonesia, Mathematics, Science, and Reading? a. Main strengths b. Main weaknesses Interviewer’s notes in English: 7. (RQ1) In your estimate, what percentage of all PRIORITAS-supported schools have been generally successful in adopting for the whole school (i.e. by most teachers) the active teaching and learning methods promoted by PRIORITAS for teaching Bahasa Indonesia, Mathematics, Science, and Reading? Interviewer’s notes in English: 8. (RQ1) Thinking only of schools that have had difficulty adopting the active teaching and learning methods promoted by PRIORITAS, please provide examples of the challenges or obstacles to successful adoption that have been experienced by those schools. Interviewer’s notes in English: 9. (RQ1) Provide some of your general observations about the main strengths and weaknesses of the PRIORITAS reading program that was introduced to PRIORITAS schools. a. General main strengths b. General main weaknesses Interviewer’s notes in English: 10. (RQ1) Provide some of your predictions about concrete ways that the educational ideas, methods, and materials that were developed under PRIORITAS will continue to be used in schools after the project ends. Interviewer’s notes in English: 11. (all RQs) (Bahasa Indonesia) Are there any further comments or observations about any aspect of the PRIORITAS whole school development program that you might like to make? Interviewer’s notes in English: 12. (RQ2) I’d like to now ask you a question about an unrelated topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Can you provide examples of when you saw news reporters and journalists covering PRIORITAS activities in schools or districts? Interviewer’s notes in English: 13. (Bahasa Indonesia) The interview is now concluded. Your comments today have been very valuable and useful. Thank you for providing your time to assist us to make a comprehensive evaluation of PRIORITAS. The USAID evaluation report of PRIORITAS will later be publicly released. (Audio recording conclusion marker inserted here) 105 Interview Guide – Governance and Management Coordinator of PRIORITAS Interview ID: Evaluator ID: Interview date: PRIORITAS CENTRAL OFFICE: RATU PLAZA, JL. JEND. SUDIRMAN JAKARTA DKI: Coordinator’s name: Gender M/F: Position title: Coordinator’s professional background (brief summary): Time interview commenced: Time interview concluded: 1. In the PRIORITAS program, how is governance and management defined? a. What are the key activities related to governance and management? Do these activities vary by level – e.g. national, provincial, district, sub-district and school? Please provide some examples. b. What are the primary objectives of these activities? Do these objectives vary by level – e.g. national, provincial, district, sub-district and school? Please provide some examples. 2. Which institutions (at national, provincial, district, sub-district, and school levels) have been supported and/or received training from PRIORITAS? a. Please provide some examples of the types of institutions at different levels that have received support or training. b. Please provide some examples of the types of trainings and subjects of trainings that PRIORITAS has provided to each of those institutions. c. What types of outcomes or changes have you seen as a result of PRIORITAS support and/or training? Please provide specific examples. d. Are there any examples where no change has occurred? Why do you think that is? 3. What are the differences between the trainings provided by PRIORITAS and trainings provided by the Government? Provide specific examples. Are there any similarities between PRIORITAS trainings and Government trainings? Please provide specific examples. 4. Our next question focuses on the sustainability of the PRIORITAS program after it closes. In line with the PRIORITAS objective to “improve capacities of local institutions”, which institutions have been involved in coordinating, organizing or managing PRIORITAS trainings? a. How has the PRIORITAS program prepared these institutions to continue key activities after the program closes? Provide specific examples. b. Are there particular provinces, districts where partners are already using training materials to improve education sector governance and management? Please provide specific examples. Why do you think this is occurring in some districts/provinces and not others? What factors are key to sustainability and replication of PRIORITAS activities? c. What PRIORITAS activities are most likely to be sustained, replicated beyond the program? Why is this? Please provide some specific examples. d. Which PRIORITAS activities are least likely to be sustained, replicated beyond the program? Why is this? Please provide some specific examples. 5. Are there any other issues related to the governance and management component of the PRIORITAS program that are important to discuss? 106 Interview Guide – TTI Coordinator of PRIORITAS Interview ID: Evaluator ID: Interview date: PRIORITAS CENTRAL OFFICE: RATU PLAZA, JL. JEND. SUDIRMAN JAKARTA DKI: Coordinator’s name: Gender M/F: Position title: Coordinator’s professional background (brief summary): Time interview commenced: Time interview concluded: 1. Could you just brief us the overall PRIORITAS programs and their objectives with the selected TTIs? 2. One of the objectives of this program is strengthening the capacity of TTIs to produce skilled primary and secondary teachers. Could you explain, in detail, what the PRIORITAS programs are and how they were delivered towards this objective? What are the outcomes of these programs for: teachers, TTIs (heads, facilitators)? 3. The other objective is strengthening the capacity of TTIs to deliver quality active learning in reading, math and science. Could you explain in detail the PRIORITAS activities towards this objective? a. In your opinion how effective were these activities? What specific effects did these activities have on the skills and competencies of pre-service teachers? Why/How? b. Of the TTI in general? Why? c. What were the challenges your team experienced in implementing these activities? 4. What has been the TTIs response and role in carrying out these activities? 5. How have PRIORITAS activities and best practices been disseminated across TTIs and schools? 6. What do you expect from the TTI after the project ends? a. Will TTIs be able to sustain certain PRIORITAS activities without USAID support? Which ones? b. Why do you think this is? c. What are some of the differences between TTIs that are able to play a strong implementation role compared with those TTIs that are less effective in this way? 7. What can TTIs do to ensure that PRIORITAS activities can continue (be sustained) after USAID ends their support? (e.g. train more teachers or collaborate more closely with district or provincial government units) 107 Interview Guide – Training Facilitator/Coordinator of PRIORITAS Interview ID: Evaluator ID: Interview date: PRIORITAS CENTRAL OFFICE: RATU PLAZA, JL. JEND. SUDIRMAN JAKARTA DKI: Coordinator’s name: Gender M/F: Position title: Coordinator’s professional background (brief summary): Time interview commenced: Time interview concluded: 1. Please describe the structure and content of the facilitator trainings for PRIORITAS. What has your role been in supporting/managing/organizing these trainings? 2. What are the primary purposes of the facilitator training? Who do you target to participate in these training? How do you select facilitators? What are the characteristics of an effective or good facilitator? 3. How was the training of trainers for facilitators conducted? 4. Please describe the training programs for JSS-level and primary-level facilitators. a. Were the trainings for JSS-level and primary-level facilitator structured in the same way? b. Was the content the same? c. What were the differences? 5. Our next question is about sustainability. What do you expect from the facilitator trainings after the project ends? a. Are there institutions that are well positioned to continue the facilitator trainings? Which ones? How has PRIORITAS prepared these institutions to sustain these trainings after the project ends? b. What are some of the differences between institutions that are well positioned to sustain the facilitator trainings, or other key PRIORITAS activities, versus those institutions that are not able to do so? 6. What has PRIORITAS done so far in terms of working with different levels of the GOI to ensure key activities of PRIORITAS are continued after the project closes? a. Which PRIORITAS activities are more likely to be continued after USAID closes the project? Why? b. Which PRIORITAS activities are least likely to be continued after USAID closes the project? Why? 108 Interview Guide – GOI Coordinator of PRIORITAS Interview ID: Evaluator ID: Interview date: PRIORITAS CENTRAL OFFICE: RATU PLAZA, JL. JEND. SUDIRMAN JAKARTA DKI: Coordinator’s name: Gender M/F: Position title: Coordinator’s professional background (brief summary): Time interview commenced: Time interview concluded: 1. Could you please tell us, who is the person in charge from the Directorate of Primary Education and Development or Directorate of Secondary Education and Development on PRIORITAS Program? Who have you worked most closely with in your role as Coordinator with the GOI? 2. Based on your experience, do these individuals follow the activities of PRIORITAS program intensively? If yes, why?And If no, why? 3. Could you tell us, what supports (technical, administrative, other) you received from the Directorate of Primary Education and Development or Directorate of Secondary Education and Development on PRIORITAS Program? 4. Are there any PRIORITAS target schools selected by MOEC as district reference schools? How were these schools selected? What additional support do these schools receive? 5. What obstacles do you have in involving the Directorate of Primary Education and Development or Directorate of Secondary Education and Development on PRIORITAS Program? a. Please provide specific examples. b. What activities has PRIORITAS conducted to strengthen coordination with the GOI? c. How effective have these activities been? Why or why not? 6. In 2012, the PRIORITAS project started to train teachers on K13. However, in 2015 the MOEC also conducted K13 training the teachers. a. What are the differences between these training programs? b. What are the areas of overlap? c. Has any effort been made to synchronize the K13 training strategy and materials? Why or why not? 7. Do you have any strategy to continue or sustain the PRIORITAS district facilitators to keep working as district facilitators after the PRIORITAS project closes this year? Please describe this strategy. How was it developed? Who must you coordinate closely with in order to implement this strategy? 8. Are there other PRIORITAS activities that are likely to be sustained or continued after the project closes? If yes, which ones? Why? If no, why? 9. In 2016 the Directorate of Education Personnel has a program called “Strengthening of District Facilitator”. 109 a. Can you tell me about this program? b. How does this program complement PRIORITAS? 10. In early 2017, MOEC introduce new National Standard of Teacher Competency. a. Can you describe these standards? b. How are these standards related to the standards developed by the PRIORTIAS program (e.g. Whole School Program)? 11. Is there anything else you would like to discuss in relation to your role with PRIORITAS? 12. Is there anything else you would like to discuss in relation to the successes and challenges PRIORITAS has experiences in coordinating with the GOI? Thank you so much for your time! 110 Online Survey Instrument – Head of TTI (Teacher Training Institution) Background: This online survey is part the final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your institution after the project ends. We wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions we ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. Directions: We request you to answer all the questions. For the questions with a response scale, please tick the answer based on your degree of agreement or disagreement. For open-ended questions, please type your answer in the provided box. Once you finish, please click the “DONE” button and your responses will be automatically sent to us. This online survey should take about 30 minutes to answer. Name of institution: Town/sub-district: District: Province: Interviewee name: Interviewee phone number: Interviewee email address: Interviewee gender: MALE or FEMALE (please tick) Interviewee HAS or HAS NOT personally participated in PRIORITAS training (please tick) Interviewee’s length of career service in your institution: …….. months Practice in TTI Stro ngly Agr ee Ag re e Disag ree Stron gly Disag ree Do Not Know 1. Lecturers in your TTI demonstrate models and techniques of active learning behaviors 2. 3. Please give some examples of active teaching and learning that your lecturers use in your institution ……………………………………………… 4. 5. TTI offers a more practice-oriented practicum after receiving support from the PRIORITAS program 6. 7. TTI offers a more practice-oriented practicum compared with other types of training institutions 8. 9. Please give some examples on how practice-oriented practicum implemented in your institution ………………………………………………… 10. 11. TTI integrates PRIORITAS project training materials and programs into pre-service teacher education curricula 12. 111 13. TTI integrates PRIORITAS project training materials and programs into in-service teacher education curricula 14. 15. Please give some examples on how your institution integrates PRIORITAS project training materials and programs into in-service and pre-service teacher education 16. Curricula ………………………………………………… Satisfaction Very Satis fied Sat isfi ed Dissa tisfied Very Dissa tisfied Do Not Know 17. Has PRIORITAS project impacted the capacities/skills of your institution? 1 YES 2 NO 18. 19. If YES, please rate your satisfaction on the capacities/skills of your institution 20. 21. Please give some examples on why PRIORITAS project has impacted the capacities/skills of your institution ………………………………………………… 22. 23. If PRIORITAS project has not impacted the capacities/skills of your institution, please provide the reasons ………………………………………………… 24. 25. Has PRIORITAS project impacted the capacities/skills of your lecturers, most importantly in active learning? 1 YES 2 NO 26. 27. If YES, please rate your satisfaction on the capacities/skills of your lecturers 28. 29. Please give some examples on why PRIORITAS project has impacted the capacities/skills of your lecturers ………………………………………………… 30. 31. If PRIORITAS project has not impacted the capacities/skills of your lecturers, please provide the reasons ………………………………………………… 32. 33. Has PRIORITAS project impacted the capacities/skills of your students (in-service and pre-service teachers)? 1 YES 2 NO 34. 35. If YES, please rate your satisfaction on the capacities/skills of your students 36. 37. Please give some examples on why PRIORITAS project has impacted the capacities/skills of your students ………………………………………………… 38. 39. If PRIORITAS project has not impacted the capacities/skills of your institution, please provide the reasons ………………………………………………… 40. 41. PRIORITAS trainings and modules promote active learning and teaching 42. PRIORITAS trainings and modules promote school-based management 43. 44. PRIORITAS trainings and modules are relevant to the needs and challenges faced by PRIORITAS partner schools 45. 112 46. If the PRIORITAS trainings and modules are not relevant with the schools’ needs and challenges, please provide the reasons 47. …………………………………………….. 48. PRIORITAS trainings and modules promote gender-equality 49. What is your overall opinion /impression about PRIORITAS programs? ………………………………………………… 50. 51. With your institution, what are the most effective/successful PRIORITAS activities? Please provide specific examples and why these are most effective/successful ………………………………………………… 52. With your institution, what are the least successful PRIORITAS activities? Please provide specific examples and why these are least effective/successful ………………………………………………… Sustainability Stro ngly Agr ee Ag re e Disag ree Stron gly Disag ree Do Not Know 53. PRIORITAS project supports your TTI to function effectively as a hub for continuing professional development 54. 55. Your TTI should collaborate with provincial government/district government/other institutions related to in-service training plan 56. 57. Your TTI should work together with provincial and district government in education resource management, especially for teacher deployment program Readiness Stro ngly Rea dy Re ad y Not Read y Stron gly Not Read y Do Not Know 58. Please rate your readiness in continuing PRIORITAS activities after the project ends 59. 60. What plans that you have in order to sustain/replicate/continue the program after the project ends? ………………………………………………… 61. 62. What were the major challenges or obstacles in implementing your plans after the project ends? ………………………………………………… 63. 64. What are some of your predictions about how the educational ideas, methods, and materials developed under PRIORITAS will continue to be used after the project ends? ………………………………………………… 65. 66. I’d now like to ask you a final question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. ………………………………………………… 113 67. 68. Please provide any further comments/suggestions you have related to the PRIORITAS project 69. ………………………………………………… Our survey about PRIORITAS is now concluded. Your comments today have been very valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. We will contact you in the future if we need to make some clarifications on your responses. 114 Online Survey Instrument – TTI Facilitator (Teacher Training Institution Lecturer) Background: This online survey is part the final evaluation of the USAID PRIORITAS education project, which will end soon. The purposes of the evaluation are to identify impacts of the project on your school, and identify possible ways its activities will be replicated and sustained in your institution after the project ends. We wish to ask you some questions about these topics, and invite you to answer them as honestly as you can. There are no right or wrong answers to the questions we ask. Your identity will be protected, and in our report to USAID your name will not be linked to any comments you make today. Directions: We request you to answer all the questions. For the questions with a response scale, please tick the answer based on your degree of agreement or disagreement. For open-ended questions, please type your answer in the provided box. Once you finish, please click the “DONE” button and your responses will be automatically sent to us. This online survey should take about 30 minutes to answer. Name of institution: Town/sub-district: District: Province: Interviewee name: Interviewee phone number: Interviewee email address: Interviewee gender: MALE or FEMALE (please tick) Interviewee HAS or HAS NOT personally participated in PRIORITAS training (please tick) Interviewee’s length of career service in your institution: …….. months Practice in TTI Stro ngly Agr ee Ag re e Disag ree Stron gly Disag ree Do Not Know 1. As a lecturer, you demonstrate active learning behaviors 2. 3. Please give some examples of active teaching and learning models/techniques that you use in your institution ………………………………………………… 4. 5. Your institution offers a more practice-oriented practicum after receiving support from the PRIORITAS program 6. 7. Your institution offers a more practice-oriented practicum compared with other types of training institutions 8. 9. Please give some examples on how practice-oriented practicum implemented in your institution ………………………………………………… 10. 11. Your institution integrates PRIORITAS project training materials and programs into pre-service teacher education curricula 12. Your institution integrates PRIORITAS project training materials and programs into in-service teacher education curricula 115 13. 14. Please give some examples on how your institution integrates PRIORITAS project training materials and programs into in-service and pre-service teacher education curricula ……………………………………………… Satisfaction Very Satis fied Sat isfi ed Dissa tisfied Very Dissa tisfied Do Not Know 15. Has PRIORITAS project impacted the capacities/skills of your institution? 1 YES 2 NO 16. 17. If YES, please rate your satisfaction on the capacities/skills of your institution 18. 19. Please give some examples on why PRIORITAS project has impacted the capacities/skills of your institution ……………………………………………… 20. 21. If PRIORITAS project has not impacted the capacities/skills of your institution, please provide the reasons ……………………………………………… 22. 23. Has PRIORITAS project impacted your capacities/skills, most importantly in active learning? 1 2 YES 2 NO 24. 25. If YES, please rate your satisfaction on your capacities/skills 26. 27. Please give some examples on why PRIORITAS project has impacted your capacities/skills ………………………………………………… 28. 29. If PRIORITAS project has not impacted your capacities/skills, please provide the reasons ……………………………………………… 30. 31. Has PRIORITAS project impacted the capacities/skills of your students (in-service and pre-service teachers)? 1 YES 2 NO 32. 33. If YES, please rate your satisfaction on the capacities/skills of your students 34. 35. Please give some examples on why PRIORITAS project has impacted the capacities/skills of your students ……………………………………………… 36. 37. If PRIORITAS project has not impacted the capacities/skills of your institution, please provide the reasons ……………………………………………… 38. 39. PRIORITAS trainings and modules promote active learning and teaching 40. 41. PRIORITAS trainings and modules promote school-based management 42. 43. PRIORITAS trainings and modules are relevant to the needs and challenges faced by schools in your district 44. If the PRIORITAS trainings and modules are not relevant with the schools’ needs and challenges, please provide the reasons ………………………………………………… 45. 116 46. PRIORITAS trainings and modules promote gender-equality 47. 48. What is your overall opinion /impression about PRIORITAS programs? ………………………………………………… 49. 50. With your institution, what are the most effective/successful PRIORITAS activities? Please provide specific examples and why these are most effective/successful ………………………………………………… 51. 52. With your institution, what are the least successful PRIORITAS activities? Please provide specific examples and why these are least effective/successful ………………………………………………… Sustainability Stro ngly Agr ee Ag re e Disag ree Stron gly Disag ree Do Not Know 53. PRIORITAS project supports your TTI to function effectively as a hub for continuing professional development 54. 55. Your TTI should collaborate with provincial government/district government/other institutions related to in-service training plan 56. 57. Your TTI should work together with provincial and district government in education resource management, especially for teacher deployment program Readiness Stro ngly Rea dy Re ad y Not Read y Stron gly Not Read y Do Not Know 58. Please rate your readiness in continuing PRIORITAS activities after the project ends 59. 60. What plans that you have in order to sustain/replicate/continue the program after the project ends? ………………………………………………… 61. 62. What were the major challenges or obstacles in implementing your plans? ………………………………………………… 63. 64. What are some of your predictions about how the educational ideas, methods, and materials developed under PRIORITAS will continue to be used after the project ends? ………………………………………………… 65. 66. I’d now like to ask you a final question about a different topic. What are your observations about the coverage that PRIORITAS activities have received in the news media, for example in newspapers and online news, and during television and radio broadcasts? Provide examples of when you saw news reporters and journalists covering PRIORITAS activities in your school or district. 67. Please provide any further comments/suggestions you have related to the PRIORITAS project ……………………………………………… 117 Our survey about PRIORITAS is now concluded. Your comments today have been very valuable and useful for our evaluation of the project. Thank you for providing your time to assist us. The USAID evaluation report of PRIORITAS will later be publicly released. We will contact you in the future if we need to make some clarifications on your responses. 118 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST 119 120 121 122 123 124 125 126 ANNEX VII: SOURCES OF INFORMATION List of Document Review 1. Statement of Work (SOW) Final Evaluation 2. Map of PRIORITAS Partners 3. DBE & PRIORITAS Final Dissemination Study Report 10 Sep 2014 4. Performance Management Plan (PMP) - October 2014 5. LPTK Program Power Point Slides I. DBE Impact Evaluation 1. DBE Impact Evaluation SOW 2. DBE Impact Evaluation, Volume I FINAL 3. DBE Impact Evaluation Annexes, Volume II FINAL II. PRIORITAS Annual Reports 1. PRIORITAS Annual Report FINAL 2012 2. PRIORITAS Annual Report FINAL 2013 3. PRIORITAS Annual Report FINAL 2014 4. PRIORITAS Annual Report FINAL 2015 III. PRIORITAS Contracts 1. Modification No. 03 to Contract No. AID-497-C-12-00003 2. PRIORITAS Contract No AID-497-C-12-00003 IV. PRIORITAS District and Province Assessment Reports 1. Role of Provincial Study Oct 2012 2. Cohort 1 District Collaborative Needs Assessment Dec 2012 3. Cohort 2 District Collaborative Needs Assessment Sept 2014 4. Cohort 3 District Collaborative Needs Assessment April 2015 V. PRIORITAS Midline Monitoring of Schools 1. 201504 - Cohort 1 Midline Monitoring of Schools 21 April 2015 2. 201603 - Cohort 2 - Midline Monitoring of Schools 7 April 2016 Vol 1 VI. PRIORITAS Mid-Term Evaluation 1. PRIORITAS Mid-Term Evaluation SOW 2. FINAL PRIORITAS Mid-Term Evaluation Report 3. Gender Study from Mid-Term Prioritas 17 June 2016 VII. TTI Reports 1. PRIORITAS Partner District and TTI Selection Report 2. Report on USAID PRIORITAS TTI Program Draft Final Dec 2016 VIII. Partnership UNS - Florida State University (videos available) Pendahuluan/Introduction 1. Pengantar Perkembangan Literasi Kelas Awal - pdf Power Point Slides 2. Bahasa Lisan - pdf Power Point Slides 3. Kesadaran Fonologi - pdf Power Point Slides 4. Konsep Cetak & Kesadaran Alfabet - pdf Power Point Slides 5. Membaca Kata - pdf Power Point Slides 127 6. SAS - pdf Power Point Slides 7. Kelancaran Membaca - pdf Power Point Slides 8. Kosakata - pdf Power Point Slides 9. Membaca Pemahaman - pdf Power Point Slides 10. Membaca Mandiri - pdf Power Point Slides 11. Menulis - pdf Power Point Slides 12. Tematik - pdf Power Point Slides IX. Quarterly Reports 1. PRIORITAS Quarter 1 Report July 2012-Final 2. PRIORITAS Quarter 2 Report Final 12 Oct 2012 3. PRIORITAS Quarter 3 Report Final 4. PRIORITAS Quarter 4 Report_15April2013-FINAL 5. PRIORITAS Quarter 5 Report FINAL X. Teacher Deployment in Indonesia 1. Teacher Deployment and Continuing Professional Development Program - Study 071216 2. 2016 Policy Brief PPG - Small schools 3. 2016 Policy Brief PPG Cost-Benefit Analysis 4. Teacher Deployment in Indonesia - Policy Brief 5. Teacher Deployment - Challenges and Solutions, Nov 2015 (Cover) 6. Teacher Deployment - Challenges and Solutions, Nov 2015 (Substance) XI. News Mapping and Media A good practices sharing: Working with Media Power Point Slides XII. Regional Regulation and MOU 1. MOU UINSA with City of Surabaya, 2015 2. Peraturan Bupati/Perbup (regional regulation) Blitar, East Java about Multigrade Teaching, 2014 3. Peraturan Bupati/Perbup (regional regulation) Labuhan Batu, North Sumatra about Acceleration of Basic Education Quality Improvement, 2014 4. Peraturan Walikota/Perwali (regional regulation) Batu, East Java about Teacher Deployment, 2015 5. Peraturan Daerah/Perda (regional regulation) Serdang Berdagai, North Sumatra 2016 6. MOU SMPN 03 Srengat, East Java with kelompok sudut baca desaMaron, a reading community at Srengat Sub District, Blitar, 2017 7. MOU SMPN 03 Srengat with JaTim Bank (East Java, Regional Bank), 2016 8. MOU SMPN 03 Srengat with Blitar, East Java2016 117 List of Interview and FGD Province District School/TTI Student Teacher School Committee School Supervisor School Principal Sub District Level District Level Provincial Level GOI Prioritas USAID Total Respondent East Java Batu 16 16 MI Al Hidayah 7 4 n/a n/a 1 n/a 12 SMP Muhammadiyah 8 6 7 1 n/a 1 n/a 15 SMPN 06 Batu 9 7 2 1 1 n/a 20 Sumbergondo 2 10 5 5 n/a 1 n/a 21 Blitar 10 10 MTsN Jambewangi n/a n/a 1 1 n/a n/a 2 SDN Kalipang 01 10 6 n/a 1 1 n/a 18 SDN Kalipang 03 n/a 5 3 1 1 1 11 SMPN 03 Srengat 8 6 2 n/a 1 n/a 17 Lumajang 12 12 SDN Kutorenon 01 9 8 4 1 1 2 25 SDN Kutorenon 03 10 n/a 3 n/a 1 n/a 14 SMPN 03 Lumajang 12 7 n/a 1 1 n/a 21 SMPN 04 Lumajang 7 3 2 1 1 n/a 14 Malang & Surabaya 9 9 SDN Pisang Candi 1 7 7 3 1 1 n/a 19 SMPN Lab UM 10 n/a 2 n/a 1 n/a 13 118 Province District School/TTI Student Teacher School Committee School Supervisor School Principal Sub District Level District Level Provincial Level GOI Prioritas USAID Total Respondent North Sumatera Labuhan Batu 24 24 MIN Padang Bulan 7 4 3 1 1 n/a 16 Urung Kompas n/a 2 n/a 1 n/a n/a 3 MTsN Rantau Prapat 4 n/a 7 n/a 1 n/a 12 SMPN 2 Bilah Hulu n/a 6 n/a 1 1 n/a 8 Medan 9 9 SDN 101774 6 6 3 n/a 1 1 17 SMP Al Azhar 6 n/a 2 n/a 1 n/a 9 Serdang Berdagai 9 9 MTs Alwasliyah 27 6 8 2 1 1 n/a 18 SDN 102020 8 3 3 1 1 1 17 SMPN 03 Perbaungan 6 9 2 1 1 n/a 19 SDN 108293 Perbaungan 6 2 1 1 1 11 119 Province District School/TTI Student Teacher School Committee School Supervisor School Principal Sub District Level District Level Provincial Level GOI Prioritas USAID Total Respondent Central Java Semarang 4 4 MIN Sumber Rejo 4 2 2 1 n/a n/a 9 MTsN 2 Semarang 5 6 1 1 1 n/a 14 Batang 3 3 MI Muhammadiyah Karangasem 4 2 3 1 1 n/a 11 SDN 13 Karangasem n/a 3 1 n/a 1 n/a 5 SMPN 1 Subah 9 9 2 1 1 n/a 22 SMPN 7 Batang 5 n/a 1 1 1 n/a 8 Pekalongan 7 7 MTs 45 Wiradesa 5 4 1 1 1 n/a 12 SDN 01 Kampil 6 3 5 n/a 1 n/a 15 SDN Bener 5 1 n/a 1 1 n/a 8 SMPN 2 Wiradesa n/a 5 2 1 2 n/a 10 120 Province District School/TTI Student Teacher School Committee School Supervisor School Principal Sub District Level District Level Provincial Level GOI Prioritas USAID Total Respondent South Sulawesi Makasar 10 10 MI Al Abrar 4 4 2 1 1 n/a 12 MTsN 1 Makasar 6 10 2 1 1 n/a 20 UNM 0 Maros 15 15 MIN Maros Baru n/a 6 2 1 1 n/a 10 SD 111 Inpres 5 6 3 1 1 1 17 SMPN 1 Maros 9 5 n/a 1 1 n/a 16 SMPN 2 Bantimurung 5 6 1 1 1 n/a 14 Pare-Pare 1 1 MTs DDI Lil Banat 9 6 1 1 1 n/a 18 SDN 02 Soreang 4 5 2 1 1 1 14 SDN 35 6 3 n/a 1 1 1 12 SMPN 4 Pare-pare 6 3 1 1 1 n/a 12 Pinrang MI DDI Pinrang Barat 4 10 5 1 1 n/a 21 Mattiro Bulu 6 4 2 1 1 1 15 Mattiro Sompe 9 6 3 1 1 n/a 20 SMPN 4 Pinrang 6 6 1 1 1 n/a 15 USAID/Indonesia PRIORITAS Project: Final Performance Evaluation 121