Theory of Change August 2011 May 22, 2014 This publication was produced by Amex International for the United States Agency for International Development under Contract No. AID-RAN-I-00-09-00008/AID-167-TO-14-00003. Written by John R. Swallow, Team Leader and Vlera Kastrati, Basic Education Specialist, with the participation of Edita Bucinça, Administrative Assistant. MID-TERM PERFORMANCE EVALUATION OF THE KOSOVO BASIC EDUCATION PROGRAM (BEP) FINAL REPORT MID-TERM PERFORMANCE EVALUATION OF THE KOSOVO BASIC EDUCATION PROGRAM (BEP) FINAL REPORT DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i CONTENTS Acronyms............................................................................................................................................................................. ii Executive Summary ........................................................................................................................................................... 1 Background .......................................................................................................................................................................... 3 Methodology ....................................................................................................................................................................... 4 Data Collection Methods ............................................................................................................................................ 5 Limitations of the Study ............................................................................................................................................... 6 Findings and Conclusions ................................................................................................................................................. 7 Findings Related to Question #1 ............................................................................................................................... 8 Component One: Enhance School Management Capacities in the Decentralized Environment .......... 8 Component Two: Strengthen the Classroom-Based Assessment of Learning Outcomes................... 12 Component Three: Improve the In-Service Professional Development of Teachers ........................... 14 Findings Related to Question #2 ............................................................................................................................. 17 Findings Related to Question #3 ............................................................................................................................. 18 Findings Related to Question #4 ............................................................................................................................. 18 Findings Related to Question #5 ............................................................................................................................. 19 Findings Related to Question #6 ............................................................................................................................. 21 Findings Related to Question #7 ............................................................................................................................. 22 Findings Related to Question #8 ............................................................................................................................. 22 Recommendations ........................................................................................................................................................... 22 Special Concerns for USAID/Kosovo ......................................................................................................................... 25 Critical Thinking Skills and Interactive Teaching and Learning......................................................................... 25 Technology ................................................................................................................................................................... 25 Indicators Utilized by USAID/Kosovo and BEP ................................................................................................... 26 Annexes ............................................................................................................................................................................... 1 Annex A: List of Persons Met and/or Interviewed .................................................................................................... 2 Annex B: List of Documents Reviewed ...................................................................................................................... 6 Annex C: Anonymous Questionnaires ....................................................................................................................... 8 Annex D: Scope of Work ............................................................................................................................................ 30 ii ACRONYMS A-EGRA Albanian-Early Grade Reading Assessment AfL Assessment for Learning BEP Basic Education Program CLC Community Liaison Coordinator COP Chief of Party DCOP Deputy Chief of Party EGRA Early Grade Reading Assessment EMIS Education Management Information System FHI 360 Family Health International 360 GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GOK Government of Kosovo ICT Information and Communication Technology (also IT) JOC Joint Oversight Committee KEC Kosovo Education Center KESP Kosovo Education Strategic Plan LF Learner Facilitator M&E Monitoring and Evaluation MED Municipal Education Department MEST Ministry of Education, Science and Technology MLF Master Learning Facilitator MOU Memorandum of Understanding NGO Non-governmental Organization PD Professional development PDC Professional Development Center PMP Performance Monitoring Plan SBA School-based Assessment SDP School Development Plan SBPD School-based Professional Development SBASHK Kosovo Teacher Union SSTC Student Support Technician Club TA Technical Assistance TD Teacher Development TOR Terms of Reference USAID United States Agency for International Development 1 EXECUTIVE SUMMARY The purpose of this mid-term performance evaluation is to provide USAID/Kosovo with an objective external assessment of its Basic Education Program (BEP). This assessment covers programmatic activities implemented from August 30, 2010, through the present, and provides findings and recommendations for USAID to consider during the program's remaining implementation period (the BEP cooperative agreement runs through September 30, 2015). BEP is being implemented by Family Health International (FHI 360). This evaluation will enable USAID/Kosovo and Kosovo's Ministry of Education, Science and Technology (MEST)—as well as other public and private organizations—to judge how effective BEP has been in assisting schools, Municipal Education Departments (MEDs), and MEST itself, in implementing and complying with reform measures established by the Government of Kosovo (GOK) to improve education at the pre-university level. For this mid-term evaluation, AMEX International was asked by USAID/Kosovo to address eight key questions. To address these questions, AMEX sent a team of two evaluators to Kosovo from March 18 to April 7, 2014 (approximately three weeks). Prior to and during the trip, the evaluators reviewed key documents and online information relevant to BEP, including FHI 360 work plans and progress reports; USAID, MEST, and GOK policy documents; statistical data; and other related documents and websites. The team also developed an evaluation methodology for collecting data during their trip to Kosovo. Qualitative data collected in Kosovo included unstructured individual and group interviews; focus groups with teachers; site visits to rural and urban schools and MED offices in various municipalities; and meetings with USAID, FHI 360, the MEST, and other international donors. Quantitative data was collected via anonymous surveys administered to individual teachers, school directors, and staff at the MEDs, MEST, and the Kosovo Education Center (KEC). The data and analysis obtained is presented in the Findings section of this report and serves as a basis for concluding how well the BEP is meeting its objectives, identifying deficiencies, recommending ways to correct deficiencies during the program's remaining implementation period, and formulating similar initiatives in the future. The evaluation team found that much has been accomplished in the BEP, particularly with respect to program outputs (e.g., the quantity of teachers and school directors trained). The evaluators consider FHI 360’s leadership and personnel to be high-quality, dynamic, and energetic; they have interacted routinely and effectively with the MEST, MEDs, teachers, school directors, and other international donor organizations. The quality of BEP training courses, training materials, and especially BEP trainers has universally received high ratings and levels of appreciation. The evaluation team’s interviews and focus group discussions brought forth positive comments regarding the usefulness of the BEP’s school-based, classroom-centered assessment trainings for students and teachers, as well as the skills and techniques that reading teachers are acquiring and putting into practice. Supported by the responses in the anonymous questionnaires, the interviews and focus group discussions also provided evidence that the BEP is valued for the quality of its training and its trainers; its contribution toward professional development of teachers and a more participatory, relaxed classroom environment; its creation of training and organizational structures for improving school management capacity; and its introduction of technology into schools. With respect to deficiencies, a frequent complaint heard from teachers and school directors was the failure to provide certificates to teachers participating in "cascade" trainings at the schools. (In contrast, the teachers who receive the first-level three-day or two-day BEP-sponsored training promptly obtain 2 their certificates). Not receiving certificates after investing time and effort into acquiring new skills and techniques is causing disappointment and loss of morale, especially since these certificates can impact their chances for job security and/or career advancement. Technology and the provision of equipment to BEP-supported schools are major elements in the program. However, the teacher focus groups and site visits revealed that the technology—despite being greatly appreciated—is probably not being used, and should probably be discontinued in favor of other low-cost, low-tech learning tools. As requested in the Scope of Work, actionable recommendations that can be implemented during the remaining life of the project are provided as part of this evaluation. These recommendations are discussed in detail in the Findings section of this report; briefly, they are: 1. Ensure that certificates are promptly awarded to teachers who complete BEP trainings; 2. Create an online BEP resource for information sharing; 3. Promote greater use of effective yet low-cost learning tools and supplementary materials, while also reducing reliance on higher technology equipment; 4. Emphasize the importance of reading, the benefit of parents reading with their children, and the use of low-cost, easy-to-make reading materials for young children; 5. Designate a responsible person, preferably a teacher, in each school to coordinate with the PDC, and vice versa; 6. Integrate competent citizens into municipal committees to work with MED officials in hiring the most qualified candidates for school director and teaching positions; 7. Provide high-quality, targeted training to school psychologists, especially in areas of Roma population; 8. Provide more BEP training kits, especially in large schools, and hands-on training so more teachers know how to use them; 9. Ensure the presence of all three BEP components in each participating school; 10. Expand the Assessment for Learning (AfL) training from two to three days; 11. Encourage MED and MEST personnel to visit schools more frequently (at least once per year); 12. Provide more post-training monitoring and mentoring, especially in rural schools; and 13. Provide PDCs with more than one projector. Additionally, the evaluation team developed recommendations that—although they could be piloted during the current program—may serve as ideas, or lessons learned, for possible follow-on projects to support education reform in Kosovo. These include: 1. Find ways to further stimulate and encourage use of the PDCs to bolster the MEDs’ role in implementing Pre-University Education Law; 2. Assist Kosovo in collecting and writing up its native literature, i.e., folk tales, fairy tales, fables, interesting information about Kosovo’s flora and fauna, geography, histories of notable Kosovars (artists, writers, scientists, etc.) and other literature appealing to children; 3. Consider ways to mitigate inequities between students who have considerable exposure to technology at home and students with little or no such at-home exposure; and 4. Achieve a better balance between numerical and non-numerical indicators in order to better measure quality, relevance, and usefulness of program elements. 3 BACKGROUND The Basic Education Program (BEP) formally began on August 30, 2010 when USAID/Kosovo signed a cooperative agreement with the Academy for Education Development (AED), which was acquired approximately nine months later by Family Health International (FHI 360). BEP is a five-year, USAID-funded education initiative designed to benefit all public primary and lower secondary schools in Kosovo. As of May 2014, 16 more months remain until the termination of the five￾year cooperative agreement. BEP aims to strengthen the capacity of Kosovo’s teachers and schools to provide relevant skills for its students. Its overarching goal is to strengthen the Government of Kosovo’s (GOK) institutional capacity in the education sector and improve the quality of primary education. BEP’s motto is “Developing students’ 21st century skills with schools and communities.” The BEP program objectives consist of three interlocking “components”: 1. Enhance School Management Capacities in the Decentralized Environment; 2. Strengthen the Assessment of Learning Outcomes; and 3. Improve In-Service Teacher Training. BEP is integrated closely with the ongoing implementation of the Kosovo Education Strategic Plan (KESP 2011 – 2016)1 adopted in September 2011, which includes a vision, objectives, and measures, and covers the human and financial resources required to improve education performance using a sector-wide approach. KESP's focus on basic primary and lower secondary education (grades 1 – 9) mirrors the programmatic focus of BEP in the three components of school management, assessment of learning outcomes, and teachers’ professional development and licensing. The BEP assists the GOK's process of decentralizing management and planning responsibilities to individual municipalities and schools. As foreseen in the Law on Education in Municipalities (2008),2 the municipalities’ role, among others, is "training educators and other professional staff in accordance with guidelines, principles and standards promulgated by the MEST.” This has been closely linked to establishing Professional Development Centers (PDCs) and developing the capacity of Municipal Education Departments (MEDs) to monitor, evaluate, and support schools. Moreover, BEP has focused on increasing the management capacities of school directors and governing bodies. With the new Kosovo Curriculum Framework3 currently being piloted, the BEP seeks to align its teacher professional development efforts with the new competence-based curriculum and assessment system via the numerous training courses offered to teachers. MEST introduced the teacher's licensing process in 2009 and created a system that not only provides permits to teachers to continue to work in the system but also offers opportunities for career advancement. According to the administrative instruction of April 2010, there is a set of criteria and standards that MEST has established for teachers’ qualifications, development of the system for licensing, and accreditation of the programs for teacher's professional development; there has also been development of a payment system focusing on teacher's performance and career advancement. The State Council for Teachers Licensing is responsible for accrediting trainers, licensing teachers,4 and developing and monitoring licensing policies and practices. 1 Ministry of Education, Science and Technology (MEST), Kosovo Education Strategic Plan 2011–2016, Prishtina 2011 2 Ministry of Education, Science and Technology (MEST), Law on Education in Municipalities, Prishtina 2008 3 Ministry of Education, Science and Technology (MEST), Kosovo Curriculum Framework, Prishtina 2011 4 MEST Administrative Instruction (5/2010) Teachers Licensing; retrieved from http://www.masht￾gov.net/advCms/documents/UA_Licencimi_i_Mesimdhenesve.pdf 4 The training offered by BEP has been recognized and accredited by MEST, and the certification of trained teaching staff is in line with the ongoing licensing process. The BEP included MEST in its planning stages. Specifically, the first component of the BEP seeks to improve the management skills of school directors; school boards; and the MEDs in the areas of planning, school management, and quality assurance. The second BEP component aims to improve the capacity—at the local, municipal, and central levels—to develop and implement new school-based (internal) and potentially national (external) assessments tied to the new curricula. Hopefully, this would support the establishment of an effective and reliable system for assessing student learning outcomes, and hence better primary-level education. The program's third component aims to assist MEST in providing in-service teacher training reforms. These intended reforms included certification requirements, supporting the MEST’s new Teachers’ Licensing foundation and establishing a program for teachers that would provide for continuous professional development. Although FHI 360 leads the implementation of the BEP on behalf of USAID, the Kosovo Education Center (KEC), an NGO, has been sub-contracted to manage the program's third component. FHI 360 and KEC also partner with MEST, and align their activities to USAID's educational reform priorities and to European Union (EU) standards, in support of Kosovo’s effort to join the EU. Investment of program funds across the three program components has been roughly equal, except that the third component related to teacher training, licensing, and professional development has incurred slightly higher costs due to higher educational technology expenses. Also, activities related to the first and second components are being implemented in some schools where the third component is not. Lastly, the table below provides some basic, cumulative statistics in four major categories that FHI 360 has tracked since the initiation of the project in late 2010. These statistics show that the program has progressively expanded during its initial three years in most, or arguably all, categories. TABLE 1: BEP YEARLY STATISTICS BEP Category 2010-11 2011-12 2012-13 2013-14 (until March 31) Schools actively involved 58 277 416 437 Teachers and School Directors trained 277 6,411 15,243 17,290 Professional Development Centers established 17 24 25 25 Schools which received BEP equipment 48 73 201 267 *These statistics are cumulative year-by-year, with latest data from March 31, 2014. METHODOLOGY AMEX reviewed USAID/Kosovo’s Statement of Work and assembled a strong evaluation team to carry out the proposed assessment of BEP activities. The team was led by an educator and international development consultant with more than 10 years of relevant experience, direct knowledge of basic education and gender issues within an international development context, and expertise in quantitative and qualitative research methods. A local educational researcher with extensive practical knowledge of Kosovo’s basic education reform agenda, particularly as it relates to decentralization, curriculum, teacher training, and gender integration issues was also part of the team. A local administrative assistant also supported the team with its external evaluation. The team worked to identify qualitative changes in opinions, satisfaction rates, and attitudes of key stakeholders and beneficiaries, paying close attention to 5 evidence related to successful and less successful outcomes, and applying sound professional judgment to reach conclusions and lessons learned based on appropriate evidence. As set forth in USAID ADS 203, this performance evaluation focused on descriptive and normative questions, such as what a project has achieved, how it is being implemented, how it is perceived and valued, and whether expected results are occurring. USAID/Kosovo specifically posed eight questions to be answered as part of this mid-term performance evaluation; the approach taken is consequently based around gathering and reviewing data for answering the eight evaluation questions. To this end, the evaluation team engaged project staff, implementing partners, GOK/MEST staff, regional and local education officials, representatives of other donor groups, local stakeholders, teachers, and students. The eight evaluation questions incorporate the BEP objectives; cross-cutting themes; and other considerations such as coordination with other donors, possible course corrections, lessons learned, etc. The eight questions are: 1. How has the Kosovo Education System in Kosovo been strengthened and benefited from the implementation of BEP’s three (3) program assistance components? To what extent has the program met its three stated objectives and how effective have the program's interventions been in achieving the program’s stated objectives? Why or why not? 2. How have the Ministry of Education and Municipal Education Departments (MEDs) in municipalities where BEP is active strengthened their gender diversity as a result of the implementation of BEP’s activities? To what extent has the program strengthened gender diversity through its three stated objectives? Why or why not? 3. How successful is BEP in integrating non-Albanian ethnic minorities in its Programming? To what extent has the program reached out to them? To what extent has the program reached out to Serb minority? Why or why not? 4. What is the current capacity and timeline for municipalities to become effective in assuming prescribed responsibilities under the Pre-University Education Law? How much has BEP utilized on the job training and coaching with partner municipalities (in comparison to classical training) and what are the concrete results on the ground? 5. How effectively does BEP coordinate activities with other USAID and other donors’ programs? 6. Based on the review of BEP’s implementation and results, what recommendations are there for possible future USAID programming and/or other donors or governments in improving Kosovo education system? What recommendations are there for supporting the implementation of the reform process? 7. Apart from current coordination, in what other ways can BEP collaborate with other ongoing USAID programs? 8. Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? What are recommendations and lessons learned? DATA COLLECTION METHODS The evaluation relied on a mixed-methods approach; data was gathered to generate detailed knowledge about the magnitude and performance of the BEP project, to measure accountability and benefit, and to inform future activities. To obtain this data, five different, albeit complementary methods of data collection were utilized:  document review  unstructured in-person interviews of key informants (individual and group)  teacher focus group discussions 6  site visits to participating schools, MEDs, and the MEST  anonymous questionnaires for individuals representing different BEP stakeholder groups Five different anonymous questionnaires were developed for teachers, school directors, MED personnel, MEST authorities, and KEC personnel, respectively. Several common questions were maintained throughout all of the questionnaires; most of the five groups had seven questions, although two questions were not asked of KEC personnel and one was not asked of teachers. (The questionnaires, in Albanian and English, are found in Annex C.) These questions were based on a scale of 1 to 5—with 1 being “very low” or “not at all” and 5 being “very high” or “very good.” In addition to the questions, respondents were provided with a rank-order question at the end of the questionnaire: respondents were asked to rank the 14 BEP elements/activities from 1 to 14, from the least valuable or least important element (1), to the most valuable or important element (14). In total, 115 teachers, 18 school directors, six MED officials, five KEC personnel, and two MEST officials completed the questionnaires and returned them to the evaluation team. During focus group discussions, site visits, and key informant interviews, the evaluation team probed for possible shortcomings, gaps, weaknesses, or other problems with each element of the BEP. This method was employed even when positive data or comments were being provided to the evaluation team. LIMITATIONS OF THE STUDY The main limitation of this mid-term performance evaluation can be attributed to the amount of time allotted for such a large, decentralized, complex, and multi-faceted national program serving many beneficiaries. The in-country component of the evaluation was conducted March 18 – April 7 and involved extensive travel to schools, municipalities, and education offices, as well as meetings, interviews, and focus groups with program participants and key informants throughout Kosovo. In all, the team had approximately 18 days in-country to conduct field work, arrange meetings, brief USAID staff, collect and analyze data, and produce a draft report. While Kosovo is not a large country, the limited time for in-country evaluation activities did not allow the team much room for error in securing interviews with key informants most acquainted with BEP activities. However, some school and MED officials were unavailable during the team’s site visits because of schedule conflicts. The team also recognizes the possibility that some of the school directors and teachers may not have participated in, or completed, the focus group discussions, in-depth interviews, and questionnaires with a maximum level of calm, insight, and reflection. For example, an interview with the Director of Pre-University Education at the MEST—a key informant, who is also the BEP coordinator for Component 1—lasted only 22 minutes (nevertheless, a good amount of useful information was produced by this meeting despite its brief duration). Additionally, because the evaluation team spent much of its time out in the field, some key informants based in Pristina were out of the country or otherwise unavailable for in-person interviews during the few days that the team was present in the capital, and the team was not able to conduct follow-up interviews over the phone. (One such key informant was the Director of the MEST Assessment Unit, who is also BEP Coordinator for Component 2 and thus a key source for evaluating the strengthening of classroom-based assessment of learning outcomes). Another area not addressed in this report is the participation of Serb schools in the BEP. The team did not address the program’s effectiveness in Serb schools because of security concerns and questions about the extent of their participation in the program. The team had to rely on Serb-associated comments from its literature review, interviews, and focus group discussions. These limitations on methodology are acceptable given that the assigned task was to conduct a mid￾term evaluation to identify, with a minimal level of effort, where the BEP was having its greatest success, where it should focus its resources during the remaining implementation period, and what problems 7 should be addressed or rectified to improve the program’s final results. Much useful original and primary data were discovered by the evaluation team. Also, the team raised or uncovered questions and topics which can be further investigated as the report’s recommendations are addressed. FINDINGS AND CONCLUSIONS The findings and conclusions contained in this report were developed based on a comprehensive analysis of all data collected for this evaluation. The team’s analysis looked primarily for reoccurring themes such as, qualitative statements that coincided between independent sources, common opinions and statements reported across multiple schools, municipalities, or other entities (e.g., MEST, KEC, etc.). Themes that generated strong consensus in the teacher focus groups were also taken into consideration, and the team noted any exceptional positive or negative comments received during the interviews, focus groups, site visits, and other meetings/interactions. Quantitatively, the numerically coded responses from individual questionnaires were combined according to group (i.e., teachers, school directors, MEDs, and KEC) in order to calculate the average response to each question (scaled from 1 to 5), including the rank-order question (scaled from 1 to 14). When relevant, these average response values from the questionnaires were analyzed according to response group to determine the degree to which the numerical results supported (or failed to support) the qualitative evidence. Following this analysis, the findings with the strongest support and programmatic relevance were then considered by the evaluation team, independent of any other input, in the context of developing the recommendations requested by USAID/Kosovo in Evaluation Questions 6 and 8. The numerical averages obtained from the anonymous questionnaires are summarized below and are also cited throughout the report to convey perceptions unique to a particular group, or to specific elements of the BEP. Table 2 below shows the average response values to the anonymous questionnaires by group and question/component. TABLE 2. RESPONSES TO BEP ANONYMOUS QUESTIONNAIRES BEP Elements/Components Teachers School Dirs. MEDs KEC Degree of help BEP is actually giving to decentralization 4.0 4.2 3.7 3.6 Quality of teacher training and professional development 4.1 4.3 4.0 4.1 Value of current contributions of the PDCs 3.8 4.1 4.5 3.8 Quality of BEP technical assistance 3.5 3.6 3.7 4.4 Municipal capacity to improve education 3.5 3.3 3.8 2.4 School management and school directors’ improvement 4.0 3.7 Teachers sharing teaching practices and methods 3.8 3.3 3.0 *Based on a scale of 1 to 5 (with 1 rated very low and 5 very high) Table 3 below shows the average scores that each group provided when asked to rank the 14 BEP components. TABLE 3. RESPONDENTS’ RANKINGS OF BEP ELEMENTS AND COMPONENTS BEP Elements/Components Teachers School Dirs. MED KEC School-based teacher training 8.3 10.4 11.3 13.6 National standards in assessment for learning 8.4 9.2 7.5 10.2 21st century technology classrooms 9.4 8.8 7.3 9.4 Institutional strengthening and capacity development of 8.1 9.5 8.2 8.4 8 MEST, MEDs, KEC, etc. Developing standards for school management 7.5 6.6 9.2 8.0 Student councils and student clubs 6.1 6.3 5.5 6.8 Reading activities involving students and parents 7.2 6.9 5.2 4.0 Training of school directors 7.2 8.4 10.0 10.4 School-based facilitators in various subject areas 7.4 9.2 6.0 8.8 Professional development centers 7.7 7.4 10.5 9.2 School boards and citizen/parent involvement in education 7.4 8.8 8.2 8.4 Gender and gender diversity 7.4 5.9 7.3 4.6 Integration of non-Albanian ethnic minorities 7.3 6.3 6.5 4.2 Other components or elements 6.3 7.3 5.8 1.0 *Based on a scale of 1 to 14 (with 1 being least valuable/important and 14 most valuable/important) The numerical results of the rank-order question shows which activities are least-valued and which are more highly valued, or deemed significant. In some cases the results are counter-intuitive. For instance, as a group, teachers value school-based training less than all other groups, even though the teachers are the beneficiaries of the training. Likewise, among the four groups, school directors gave the lowest average ranking regarding the importance of developing standards for school management. Generally, the highest rankings were given to: national standards in assessment for learning; school-based teacher training; and institutional strengthening and capacity development of the MEST, MEDs, KEC, and other entities. FINDINGS RELATED TO QUESTION #1 How has the Kosovo Education System in Kosovo been strengthened and benefited from the implementation of BEP’s three (3) program assistance components? To what extent has the program met its three stated objectives and how effective have the program's interventions been in achieving the program’s stated objectives? Why or why not? The findings for Question 1 are discussed below by each of the program's three components. COMPONENT ONE: ENHANCE SCHOOL MANAGEMENT CAPACITIES IN THE DECENTRALIZED ENVIRONMENT The first component of the BEP includes elements related to:  Establishment of Professional Development Centers (PDCs) in each Municipality  Training of School Directors to Effectively Manage Schools  Increased Abilities of Municipal Education Departments (MEDs) to Improve Education and Manage Schools  School Boards, Student Councils, Student Clubs, and Parent Associations BEP initiated its activities soon after the GOK enacted new laws regarding the decentralization of education. Under these new decentralization laws, the management and quality of education in schools would become the responsibility of municipalities and local school directors. However, the municipalities and school directors lacked the operating systems and human resources to strengthen and monitor the quality of schools and support teachers with training and professional development. Also, several persons interviewed, including some FHI 360 personnel, informed the evaluation team that, in general, school managers do not have a good reputation in Kosovo, particularly since the role of school directors increased along with the greater school autonomy created by decentralization. 9 Thus, FHI 360's initial BEP activities involved providing quality, relevant training to school directors, assisting municipalities by giving appropriate training for their education staff, and in establishing Professional Development Centers (PDCs). School Management, Municipal Education Departments, and PDCs Initially, 17 municipalities expressed an interest in participating in the BEP, leading the BEP Chief of Party to sign Memorandums of Understanding with the mayors of these 17 municipalities. The municipalities swiftly selected venues for the PDCs and 21st Century Classrooms. The PDCs would be managed by an official within the Municipal Education Departments (MEDs) and would include computers and other facilities for training, while the 21st Century Classrooms would be utilized for technology classes so that students could participate in practical projects encompassing everyday electronics and mechanics in addition to computer-aided design, control technology, and robotics. The evaluation team’s document review and interviews, including interviews with international donor representatives, highlighted that the greatest challenge for school directors was/is to plan (well). The school director training and, to an extent, the school development plan (SDP) established comparable standards and capacity levels. The 18 school directors5 that completed the anonymous questionnaire gave their highest scores to the question of the quality of their training under the BEP (score = 4.3) and the degree to which the BEP is helping to decentralize education in Kosovo (score = 4.2). School directors rated the current contributions of the PDCs in municipalities only slightly less (score = 4.1), and when asked “To what extent do you feel you have improved as a School Director because of the BEP?” the directors' responses averaged 4.0—also a high rating. The school directors gave a comparatively lower rating (score = 3.3) when asked to judge the increase in municipalities' capacity to improve education. On this same question, KEC personnel were more skeptical (score = 2.4). In light of the findings, the MEDs appear to be the weakest link, and thus an occasional bottleneck occurs between the schools and their communities, on the one hand, and the MEST, on the other. This finding is based on qualitative data characterizing the MEDs as ineffectual and/or unsuccessful thus far at increasing their capacities since the 2004 Law on Primary and Secondary Education in Kosovo was enacted. Also, some felt that there has been little attention or provision for training in line with municipalities’ new roles under educational decentralization. As a result, the MEDs are viewed as unqualified for the significant tasks given to them. While school directors and many teachers appeared hesitant to raise the matter, some informants suggested that the MEDs were sometimes staffed with inappropriate, unqualified, political appointees. The concern about nepotism and/or unqualified MED staff was consistently confirmed by international donor representatives, FHI 360 personnel, and various consultants interviewed. However, some interviewees stated that the Pristina and Mitrovica municipalities are attempting to monitor the recruitment of school directors via the participation of various representatives from NGOs, parents associations, and other groups to avoid hiring irregularities, nepotism, and/or purely political appointments. The MEDs have developed three-year plans to carry out training for their municipalities in coordination with the BEP. Generally, the PDCs are located in a large, geographically centralized school (the evaluation team visited a PDC in one municipality, however, that was located in the MED offices). Nevertheless, several directors of large schools complained that the PDCs were not housed in their schools, recognizing in part that having the PDC at their school would make it more accessible to that particular school’s large number of teachers. Regardless of whether or not PDCs are housed in particular schools, or are housed in or near MED premises, the evaluation team’s observations suggest that the PDCs are being under-utilized. One exception, or at least one exception, would be the PDC housed at the Green School in Pristina. This PDC appeared to be the most actively used, not only by its 5 Of the 18 school directors surveyed, two were deputy school directors and three were administrative officers who were knowledgeable about the BEP. 10 own students and teachers but also by teachers from numerous nearby schools. On our field visits to both smaller rural and larger urban schools around Kosovo, teachers in several focus groups mentioned they had received good training in ecology and English at the Green School PDC. The evaluation team anticipated there would be both formal and informal means of information exchanges (i.e., sharing of experiences and lessons learned among the MEDs in the country), given the heightened roles of municipalities for pre-university education reform and to improve education in Kosovo. The evaluation team asked several MED Directors and PDC Coordinators about this, two of whom attested that they had some contact with one or two neighboring municipalities, but the impression was that it was a personal connection and not something formal; the evaluation did not find any strong evidence—neither past nor present—of information sharing or institutional contact/coordination between the MEDs. Also, when interviewed, several directors of schools that are neither small nor rural complained that their schools had not been visited by anyone from MED or MEST offices in three years. School, Student, and Parent Organizations and Associations The Law on Primary and Secondary Education in Kosovo that was passed in 2004 called for greater parent participation, requiring that each publicly-funded school have a School Board comprised of three parent representatives, five teacher representatives, and—for schools at Levels 2 and 3 (secondary)— one pupil representative. The Law also mandated that each School Board establish a Pupil Council (hereinafter called student council) consisting of at least one student from every class, elected by secret ballot. The functions of the student council were/are to “work towards the improvement of the learning environment, working conditions, and welfare interests of pupils and to make representation on these matters to the Director and to the School Board.” The Law, in effect, prompted enhanced civil society action and citizen participation in public affairs, specifically for the involvement in, and improvement of, primary and secondary public education. As an additional activity partially prompted by the BEP, but also responding to student desires, several student extra-curricular organizations (e.g., technology clubs, ecology clubs) and activities were formed in a number of BEP-supported schools. The evaluation team was only able to conduct limited research regarding the progress of School Boards, Parent Associations, and Student Councils. When asked, most school directors said that the School Boards were operating fairly well and were helpful with respect to assisting in the operation and management of the school. From the few comments made regarding Parent Associations, they appear to be less active than they potentially could be, but at least they are established and their roles and responsibilities are understood. This represents a distinct accomplishment compared to previous years. It appears that only a limited number of parents participate in the school boards and parent associations. This limited participation also extends to parental involvement regarding their children’s academic progress, as well as overall interest in school reform issues. At the Naim Frasheri School in Zaskok, the evaluation team asked for an impromptu meeting with members of the student council and was able to meet with the head of the council (a male) and two council members (female). They reported that their council is composed of 15 students and that they meet outside of normal class time. They reported having a pilot project related to personal hygiene, an anti-smoking campaign, a student volunteer group, an Independence Day exhibit, and a regular production of leaflets on various topics. They also mentioned they have positive discrimination in this school, with the aim to assist children of ethnic minorities, and thus conform to Kosovo laws and national policies. The school director at Naim Frasheri also informed the team that the school’s student council sponsors an end-of-school-year party for all students and staff. Lastly, an interview with the new BEP Chief of Party, along with a review of the BEP documents, revealed that many student councils in 11 BEP-assisted schools have anti-bullying campaigns. Another positive student activity was significant participation in the Green Clubs to address the problem of litter in both rural and urban neighborhoods. With respect to the anonymous questionnaire’s rank-order question, nearly every group—with the exception of KEC personnel—ranked student councils and student clubs at the least valuable/least important end of the spectrum. Reading activities involving students and parents also ranked very low. In particular, student councils and student clubs was the lowest ranked item among the teachers’ group (average ranking was 6.1), which was comprised of 115 total respondents. More revealing was the fact that the teachers also gave a low ranking to reading activities involving students and parents; this was surprising given that the underlying purpose of this BEP subcomponent is to encourage parental involvement in learning and school reform issues. Both this activity and the training of school directors scored a below-average ranking among the teachers’ group, suggesting that the teachers place comparatively less value on these BEP initiatives. At the MEST, the evaluation team met only briefly with the Director of Pre-University Education, who is the BEP coordinator at the MEST for activities conducted under Component 1. The Director called the BEP one of the MEST’s strongest partner initiatives and asserted that the BEP is good for meeting the great challenge and responsibility of decentralization and also for supporting school management. The Director is also an active member—and lead MEST member—of the BEP’s Joint Oversight Committee (JOC), which meets each quarter to provide direction and oversight to the BEP. The Committee is composed of MEST, USAID, and FHI 360 personnel, as well as representatives from other GOK entities and donor organizations. When asked about the quality of BEP technical assistance, the Director replied: “…it is at a very high level. This is not only my opinion, but what almost everyone says from almost all the schools I hear from. BEP’s trainers are highly evaluated by school directors, and they are adept at creating skills among local trainers … BEP was the first time school directors had to meet a certain standard or certification. A great manager is a necessity to have a good school.” The Director went on to say that the MEST has two important criteria: (1) student-centered schools; and (2) critical thinking skills. The MEST views the child not just as an object, but also a subject who has some responsibility for her or his own actions and academic performance. Another strong point for BEP, he emphasized, is its school-based professional development: “BEP has done a great job of coordination and working with others.” The Director believes the successful completion of school director training can be one criterion in the hiring of qualified school directors. This would be particularly relevant in Kosovo, which has no degree for school management and administration. The MEST Director had several recommendations that the evaluation team took under consideration. Going forward, the MEST Director would like to see the BEP process deepen and extend to all schools as he is concerned about the sustainability after the BEP ends. He suggested a possible management support structure after the BEP ends, similar to a structure and organization Slovenia has created, which he evidently became acquainted with on a visit to that country. He also recommended that psychologists and pedagogues be re-instated in schools in grades 1-9 (this was included in the team’s list of recommendations below) and that each school director and deputy director be properly trained and certified. Lastly, the evaluation team included activities such as “classroom makeovers” and parent and community involvement under Component One. Classroom makeover activities appear to become more popular each year and—based on FHI 360 annual results, interviews with school directors, and a high average ranking in the teachers’ questionnaires—also appear to be a return on invesment. This BEP initiative involves students, teachers, parents, and the community in turning a traditional classroom into a more child-friendly, attractive learning environment. In 2012 – 2013, 50 schools were selected for support for 12 materials costing less than US$ 500 per school. In-kind and voluntary financial contributions provided by communities amounted to over 70 percent of the total value of the renovations. According to FHI 360, MED, and school personnel, many BEP schools renovated more than one classroom, and several schools renovated the entire school such that 112 classrooms were renovated during the year. RTK, the national TV channel, broadcast six 30-minute TV programs on the initiative (all videos are available on the BEP website). Note: Technology and the provision and utilization of technological equipment and supplies for PDCs and schools are commonly grouped under Component One. However, the evaluation team’s findings and thoughts on technology will be found in a later section of this report related to special concerns. COMPONENT TWO: STRENGTHEN THE CLASSROOM-BASED ASSESSMENT OF LEARNING OUTCOMES The second component of the BEP includes elements to:  Assist MEST to Develop National Standards for School-Based Assessment (SBA)Training of School Directors to Effectively Manage Schools  Provide In-Service Professional Development for Teachers and School Directors in SBA and Assessment Ethics  Provide Professional Development to Selected Educators in Test Construction and Develop an Item-Bank  Develop the Capacity of the MEST Assessment Unit to Meet Its Responsibilities The Launch and First year of Classroom-Based Assessment, including a Code of Ethics During the first three years of the BEP, personnel from FHI 360, KEC, and MEST, along with trainers and school facilitators, were busy with training and intensifying classroom-based student assessment efforts. As the BEP was being conceptualized, requests for continuous, quality, classroom-based assessment of student learning came from the MEST and its personnel during discussions with USAID. MEST personnel spoke of utilizing classroom-based assessment to heighten the quality of the teaching and learning process, allow students and their teachers to gauge student progress individually and as a group, and enhance the progress of individual students. To align with the MEST, FHI 360 quickly moved into action with a BEP assessment team to conduct needs analyses in sample schools, train selected teachers in a three-day Assessment for Learning (AfL) course, and conduct appropriate feedback and reporting. In the BEP, AfL is defined as what the teacher does in every lesson to assess if the students are learning (i.e., teachers ask themselves, or one or more students, which students might not understand the lesson, what might they not understand, and how can they assist them to learn better). Interviews and focus groups revealed that AfL benefits include higher self-confidence among students, better quality class discussions, and enhanced professional relationships and discourse among teachers. In 2011, BEP worked with MEST officials, teachers, and school directors to develop national standards for school-based assessment, which were then tested in schools. Soon thereafter BEP and MEST worked together to formulate a consolidated Code of Ethics, which was accepted by teachers, school directors, and MEST officials. The Code of Ethics was/is designed to inform teachers, school directors, parents, and students of their rights and obligations during the assessment process and advise them in making appropriate ethical choices. The Code enumerates the basic principles of ethical conduct for these four principal groups. The school facilitators’ training course of six days in two stages has been integrated with practical application of AfL in the classrooms. There has been a two-day course on AfL and the application of these AfL methods in classrooms, plus a three-day course on facilitation of learning standards. Master 13 Learning Facilitators (MLFs) provide participants with formative feedback. To successfully complete the AfL course, teachers must attend two-day workshops, have their lesson plans reviewed and assessed by a facilitator, and prepare and deliver lessons using AfL. During the 2012 – 2013 year, it was reported that a total of 4,871 teachers and school directors attended the AfL courses, a total of 256 certificates were presented to assessment school facilitators, and 2,728 certificates to teachers who met the criteria for certification. (Note: the large variance between the reported number of attendees and the number of certificates received or presented may be due to school directors’ failure to deliver certificates, which is explained elsewhere in this report.) National standards in assessment for learning was one of 14 items or elements that BEP respondents were asked to rank from 1 (least) to 14 (most) valuable or important in the anonymous questionnaires. School directors and teachers both ranked national standards as relatively important while MED officials gave this item a slightly lower, but nonetheless strong, indication of importance. Overall, the rankings indicate a strong level of support that all three groups of educators have for national standards as a means of improving student learning and educational outcomes. When asked, “To what degree, if any, do you feel you have improved as a teacher because of the BEP?” teachers reported “much” improvement in their classroom skills as a result of the BEP. Qualitatively, the classroom-based student assessment is generally well-regarded for its introduction of improved, more relevant, sensible, and relaxed teaching and learning, as well as for the more enlightened philosophy, behavior, techniques and skills acquired by teachers. The BEP standards for assessment have become standardized and used nationwide, for which USAID and FHI 360 deserve credit. Inherent also is that the assessment process is school-based, in effect, incorporating both formative and summative measures. The director of a large school in Gjakova remarked, “BEP, in great part through its classroom-based student assessment, has helped with critical thinking skills and changed the attitudes and behaviors of students and teachers.” The evaluation team’s interviews and focus group discussions elicited how transformative and positive BEP’s school-based, classroom-based assessment training, manuals, and mentoring have been for students and teachers. A teacher in a focus group declared, “Students were accustomed to grading, not verbal discussion and such collegiality with the teacher. Relationships between teachers and students are more relaxed and helpful.” Regarding reading skills, FHI 360 and MEST personnel told the evaluation team how little students were understanding and comprehending as judged by the A-EGRA test. However, reading teachers and school directors were enthusiastic about the skills and techniques reading teachers were acquiring in their training and putting into practice, including with classroom￾based student assessment. In a focus group of teachers in Ferezi, in a large school where half of the student body is Roma and the other half ethnic Albanian, the school psychologist spoke for most of her colleagues, stating: “We teachers have a very high regard for assessment, and how it aids and follows up on student performance. Students know where they stand academically, it gives them confidence, and it makes relations between teachers and students more relaxed and free-flowing.” Other strengths of classroom-based student assessment were made evident via interviews and teacher focus groups. First, the AfL courses and training are school-based, thus the facilitators are always nearby and at hand. Second, teachers feel the AfL course improves their teaching, including in the ways outlined above. Third, teachers are able to give students immediate and frequent feedback. And fourth, the BEP (in particular, one BEP consultant—Mark Zellman—was cited) helped the MEST Assessment Unit to construct tests and test items, which demonstrates fulfillment of the fourth element of Program Component 2 (“Develop the Capacity of the MEST Assessment Unit to Meet Its Responsibilities”). Along with the evidence of success, the evaluation team probed for possible weaknesses with the BEP classroom-based student assessment component. A few potential weaknesses were detected. 14 Although praised by teachers in nearly every focus group, some said they do not fully understand how to best implement classroom-based student assessment. Although unable to gain further detail, the evaluation team believes—as some teachers aluded to this—that teachers are potentially hesitant to ask questions on the topic. It would thus appear that encouraging teachers to engage their school facilitators in discussions about assessments would lead to improvements in teaching and student learning. Another gap detected is that certification comes from school directors and school facilitators, sometimes making it difficult for the BEP to obtain a copy of the teachers’ names, have the document stamped by the school director, and sent to the BEP. More cooperation is needed on this; FHI 360 said they were attempting to enlist the assistance of the PDCs on this matter. Another shortcoming is that the AfL training is limited to only two days, and this was emphasized by various teachers in the focus groups conducted during school visits. The evaluation team also asked the FHI 360 Assessment Officer in Pristina about this, and she agreed with the teachers that the AfL training should be longer than two days. Unfortunately, the evaluation team was unable to interview the Director of the MEST Assessment Unit, who is also the BEP Coordinator for Component 2, and a member of the JOC; his perceptions on how BEP has helped the Assessment Unit to meet its responsibilities, or to assist in test construction and developing an item-bank, were thus not evaluated. COMPONENT THREE: IMPROVE THE IN-SERVICE PROFESSIONAL DEVELOPMENT OF TEACHERS The third component of the BEP includes elements to:  Provide Relevant Instructional Materials and Equipment to PDCs and Schools  Instill Relevant Skills in Students and Support the New Curriculum  Increase the Number of Qualified Trainers and Training Providers  Strengthen Municipal-Level Capacities to Plan and Organize Professional Teacher Development Activities  Provide Educational Technology to Targeted Schools  Increase Opportunities for Teachers to Participate in Peer-Support, Professional Network Organizations, and Research From the BEP’s launch in late 2010 to the present, the professional development of teachers has increased steadily. As of March 31 of this year, 17,133 teachers, representing 437 schools, had received the BEP training—some of them in more than one BEP-offered course. Teacher Development (TD) in the BEP has been closely planned with MEST officials to assist the ongoing Kosovo education reform. Two very important reform initiatives relate to the professional development of teachers: teacher licensing and the new curriculum policy. The BEP training aligns with the MEST goal of implementing the new competence-based curriculum, which focuses more on results-oriented learning and teaching. Additionally, the BEP training is accredited by MEST, thereby meeting the requirement for training provision and certification of teachers in step with the ongoing teacher licensing process while also providing credit for licensing and career advancement. According to the BEP coordinator at MEST, nine training courses have thus far been accredited by MEST, and two more are being developed and prepared for certification. The courses provided cover mathematics, science, IT, English language, reading, technology, language arts, learner-centered classrooms, environment, and assessment. The evaluation team found evidence that FHI 360’s implementation of this component of the BEP is having positive effects and improving the training capacity of Kosovo’s education system in several ways. First, the composition of stakeholders responsible for the development of BEP-related training is not driven by a homogenous group as was the practice previously. As mentioned by KEC, FHI 360 and MEST representatives, the working groups included the MLFs, BEP coordinators, teachers, international 15 consultants and, in some cases, university professors. Second, the training content is piloted and evaluated before being submitted for accreditation. This approach assists the evaluation of training content by the teachers and thus ensures that content is well matched to the needs of the schools. Third, the training is aligned with the new curriculum framework, as well as results-oriented learning and teaching. With respect to decentralization, the school-based approach in component three has been well planned. The development of a hierarchy of change agents to support the school-based approach from national, regional, and municipal to school levels with set roles and responsibilities in all levels establishes a system that is effective and sustainable. The expert working teams, regional MLFs, municipal PDCs, and school-based-level facilitators have set roles and responsibilities to support professional development. Training is provided to develop their capacities. The focus groups supported the finding that the training for school-based facilitators was well-designed and its provisions well-planned. In the anonymous questionnaires, the teachers’ group response score was 3.8 to the question, “To what extent is in￾service teacher development improving as a result of BEP?” The response of the school directors to this same question was 4.1, while MED officials and KEC personnel both averaged a response score of 4.0. All of these groups therefore believe, more or less, that there has been “much” improvement in teacher development as a result of the BEP. The BEP also provides standards for TD through the Development of Standards for Facilitation of Professional Development, well-structured guidelines on the before, during, and after training phases. The TD training in the BEP lasts three days. The framework is a significant step towards the standardization of school-based professional development of teachers that is sustainable and serves for present and future training as mentioned by many school-based facilitators. The school-based facilitators are also supported by the Framework for the Certification and Practice of Facilitators that sets standards and criteria for the selection, responsibilities, training, capacity building, and certification of the school based facilitators in an effort to standardize the school based professional development delivery. Moreover, the School Based Professional Development Framework: Guide for schools, MED and MEST is a guiding document for the implementation of professional development activities at the school and enables facilitators to take initiatives into their own hands. In the anonymous questionnaires, teachers gave a score of 4.1 (on the 5-point scale) when asked about the quality of teacher training and professional development under the BEP. KEC personnel also gave it a 4.1 score. Teachers in the focus groups said the training is high-quality and the length sufficient, except for one school where it was only offered once. Overall, the teacher focus group discussions on the implementation of training were positive and noted that the impact on students is visible, particularly with the techniques provided in the training. Many teachers in focus groups in Pristina (the Dardania School), Gjakova (Zekirija Rexha School) and Dragash (Fetah Sylejmani School), among others, suggest that "classes are more attractive and challenging for children" and are moving closer to being "student￾centered classrooms.” Teachers in all nine focus groups also remarked that the manuals for implementation of training content for each course were very useful, suggesting that they provide support to the planning and implementation of the training content in classrooms. In addition to the manuals, the BEP has provided teachers in each subject one kit per school to support the implementation. Many teachers, particularly in larger schools, said that more than one training kit per subject is necessary. The evaluation team also carried out classroom observations in English language and IT classes. There was evidence of good interaction between teachers and students, skill-oriented practices, and classes that move beyond traditional lecturing. Furthermore, the team was exposed to teachers’ portfolios that show evidence of teaching plans, as well as photographs and work of students in the classrooms that demonstrate evidence of change toward student-centered classrooms. Interestingly, the 115 teachers 16 who completed anonymous questionnaires essentially gave an average score of “much” (3.8) to the question of “to what extent do teachers share teaching practices and methods among themselves?” Nonetheless, implementation of training at the school level has many challenges. Many teachers complained to evaluation team members that they have waited three months, eight months, and even as much as 18 months without receiving certification or any feedback on the portfolios and plans they submitted after taking the school-based training. Thus, despite frequently working on the weekends and learning new skills and techniques, the teachers have yet to be formally recognized for their efforts. Under MEST regulations, teachers must successfully undergo 100 hours of training over a five-year period to maintain their teaching license, setting aside the issue of qualifying for a promotion. Needless to say, this situation appears to be causing angst, disappointment and loss of morale among teachers. School directors play a key role in this common complaint. Many teachers said that school directors do not always follow through on their responsibility to make sure that the teachers receive their certificates, and thus accreditation, from training received at the school level. However, this appears not to be the case for the “original” two or three days of training generally held off-site. This issue is very pertinent to BEP components 2 and 3, especially for 3. In this sense, the “cascade” approach or Training￾of-Trainers (ToT) model is problematic. Upon further research and inquiry, the evaluation team discovered there are several reasons for this lack of action, or delay in the process, on the part of the school directors. One is that school directors and the school-level facilitator must certify this school-based training. Sometimes, for whatever reason, school directors fail to certify the list and ensure that it gets sent to FHI 360 personnel, who after verifying and certifying it, then send it to the MEST for its final certification and registration. Another reason is that, on occasion, the school-level facilitators wait for more or all of their colleagues to formulate their portfolios and plans before submitting the entire notebook of individual portfolios. Additionally—at one school—the evaluation team was told that the entire large folder containing every teachers’ work was lost before it could be sent to FHI 360 for verification. Unfortunately, there were no duplicate copies of this work. Other challenges were also revealed. The Dardania school focus groups revealed that, in bigger schools where the number of students per class reaches 40 or more, the training implementation is problematic in terms of class management and application of techniques. Also, despite the support of technology in PDCs, the training content is too advanced for the schools in Kosovo, given the reality of a lack of equipment and didactic materials, since the majority of classrooms have only a blackboard and chalk. Despite the efforts of the BEP to set guidelines and a framework to follow up on the training in the schools, focus group discussions show very limited follow up on teachers' work. There was no evidence of regular and systematic mentoring and monitoring, and this was mentioned in most of the focus groups. (FHI 360 trainers who provided the original training should also follow up with monitoring and mentoring activities.) Lastly, information from field visits, interviews with PDC coordinators, and focus groups with teachers are all collected and integrated into BEP training as well as that of other training providers. This training is principally on weekends. Nevertheless, focus group discussions and interviews with PDC coordinators revealed far less than full usage of PDCs on weekdays, mainly because of the lack of coordination at the school level. One exception was the Fetah Sylejmani School in Dragash where coordination was handled well, but this planning and coordination of activities and lessons were not widely used in other schools. There are examples where teachers are locked out of the PDCs, because some schools reserve the PDCs mainly for IT classes. The PDC coordinator at the municipal level has no voice in—nor apparently cooperation with—the school, leaving the coordination of school activities entirely in the hands of the school. A certified development program for the local PDC coordinators has been provided and the duties and roles were set to coordinate and organize training events, participate in selecting participants, 17 develop training materials, and support participants in planning amongst others through various tools like checklists, schedules, plans etc. The Municipal PD Coordinators also supposedly function as monitoring and evaluation resources for the BEP. This is the stated role; however, the MEDs and PDCs oeperations and utility appear to have room for improvement in this respect. FINDINGS RELATED TO QUESTION #2 How have the Ministry of Education and Municipal Education Departments (MEDs) in municipalities where BEP is active strengthened their gender diversity as a result of the implementation of BEP’s activities? To what extent has the program strengthened gender diversity through its three stated objectives? Why or why not? The Kosovo education system and regulations, on their own and in many cases with helpful boosts from the BEP, seem to be doing a good job of addressing USAID concerns regarding gender diversity.6 USAID recognizes the serious gender gaps in education management and the high drop-out rate for girls and encourages inclusive and equitable education strategies throughout the BEP activities.7 Participation numbers collected by FHI 360 demonstrate integration of women and men in the BEP training. As for quantitative outputs, women outnumber men in a number of the BEP-sponsored training courses. Of the total number of teachers and school directors trained under the BEP, 60% are female and 40% are male. (Note: Some teachers have taken more than one BEP-offered course.) Historically, however, women have been grossly underrepresented as school directors. This is also reflected in BEP-sponsored trainings. Of the 157 school directors trained and certified under the School Management Component, 132 are male and 25 are female. Nevertheless, interviews and other information provided by FHI 360 indicate that positive efforts are being made to increase the number and percentage of female school directors. Anecdotal evidence also indicates that this may be improving in some locales. For example, the administrative officer of the MED in Prizren told the evaluation team, “For the last two years, 70% of the people trained are women. Prior to 2009, there was only one female school director. Now there are 11 female directors and 3 deputy school directors among Prizren’s 54 schools.” In their anonymous questionnaires, MED officials were asked “In your view, what have been the opportunities for girls and women in the BEP program?” In response, MED personnel gave an average score of 4.0 (“good”) to this question. In the ranking section of the questionnaire, however, gender diversity was judged relatively less important and/or among the least valued elements, across all groups of respondents. Despite the low rankings, FHI 360 documentation—particularly its yearly BEP results documents—and some interviewees suggested that Kosovo’s policies, laws, and practices have accomplished, and are accomplishing, much in the area of gender diversity, though more progress is urgent and critically needed in many of the other BEP elements and activities. The BEP documents, along with interviews and focus groups conducted by the evaluation team, convey that gender diversity is being integrated into all BEP training courses, manuals, other printed materials, parent associations, student councils, and student clubs. In fact, there is a special emphasis on recruiting girls to join, and participate actively in, the Technician Clubs. In project documents and interviews with FHI 360 staff, the evaluation team noticed that many BEP activities and events focusing on girls and women tend to receive ample press coverage, principally on television and in newspapers. The evaluation team also found evidence of harmony between the BEP and GIZ-funded programs with regard to gender integration. Specifically, the MEST Director of Pre-University Education cited “gender balance” as a term of mutual usage, and importance, to the MEST, BEP, and GIZ’s Basic Education Team 6 Gender Assessment for USAID/Kosovo, Final Report, Cozzarelli, Catherine, August 15, 2012 7 Kosovo: 2014-2018 Country Development Cooperation Strategy 18 Leader. Special care has been taken, in coordination with GIZ, in the formulation of a mini-book series and early grade readers to balance female and male characters, avoid gender stereotyping, and use gender inclusive language. This includes special events such as celebration of International Women’s Day and a “Girls in Technology: Gaining Momentum” event organized in four municipalities. The evaluation team found that coordination of gender diversity contributed to fruitful, meaningful gender events that are geared positively toward changing attitudes and behaviors. FINDINGS RELATED TO QUESTION #3 How successful is BEP in integrating non-Albanian ethnic minorities in its Programming? To what extent has the program reached out to them? Why or why not? To what extent has the program reached out to Serb minority? Why or why not? USAID/Kosovo’s DO1 Sub-IRs, particularly its IR1.3 indicator (Improved Integration of Ethnic Minorities) and its three Sub-IRs, and the Mission’s Sub-IR1.4.2 (Greater Representation of Women, Youth and other Vulnerable Groups) all relate to minority groups. The evaluation team visited schools and municipalities with Gorane, Turkish, Bosnian, and Roma minorities in schools. In the Naim Frasheri School in Zazkok, the school director informed the team that in grades pre-K – 9, about half of the 883 students were ethnic Roma; furthermore, the school holds about ten percent of all Roma school children in Kosovo. The teachers in focus groups and school directors in interviews at these schools all maintained that they treat everyone equally, including non-Albanian ethnic minorities. Various teachers and MED officials spoke of “positive discrimination” aimed to assist ethnic minority children. In its professional materials, FHI 360 and its BEP colleagues have been evidently thorough in formulating and printing manuals and guidebooks in Serbian and Turkish, in addition to Albanian. The BEP training for teachers in Bosnian and Gorane is held in those languages. The evaluation team also had contact with two school psychologists in ethnic minority areas. The psychologist in the large Naim Frasheri School with a 50% Roma student population said she would greatly appreciate appropriate training to help her handle a very heavy workload; her statement was supported by the other 10 teachers in the focus group. The Roma children and their families have some special challenges, which affect school discipline and harmony. The only training the psychologist has ever had was from Save the Children five or six years ago. She collaborates informally with the other three school psychologists in her municipality, all of whom claim to be overworked also, but says “My colleague psychologists, I and our schools would really benefit from some high-quality, targeted training. We are inundated with work and some unique problems and are looking for helpful hints and practical techniques to put into practice.” In other teacher focus groups held at mixed ethnic schools, the teachers suggested that the BEP training did not take into account the mixed classrooms and “poor performing” minority groups (e.g., RAE students in Zaskok represent 50% of students), as the content provided is mainstreamed, without regard to ethnic students’ special needs. Where necessary, training is given in the ethnic minority’s language, but the training content is not substantively altered for that minority population. Overall, integration of non-Albanian ethnic minorities was judged relatively less important and/or less valued in the rank-order question of the anonymous questionnaires. Despite this low ranking, FHI 360 documents (particularly its yearly BEP results documents) and some interviewees suggest that Kosovo’s policies, laws, and practices have accomplished, and continue to accomplish, a good deal of progress with respect to integration of ethnic minorities. Nevertheless, more progress is urgently and critically needed in many of the BEP elements and activities. FINDINGS RELATED TO QUESTION #4 What is the current capacity and timeline for municipalities to become effective in assuming prescribed responsibilities under the Pre-University Education Law? How much has BEP utilized on the job training and 19 coaching with partner municipalities (in comparison to classical training) and what are the concrete results on the ground? Based on the findings of this assessment, the evaluation team believes it is not possible to provide a sufficiently accurate timeline for municipalities to become effective in assuming their prescribed responsibilities under the Pre-University Education Law. The inability to establish an accurate timeline is a result of too many factors being largely outside of the MEDs’ direct control, such as national and municipal laws and regulations, and personnel policies and assignments. To become effective, the MEDs will also require reliable and dependable funding and will most likely require additional training and human resources, which is outside of the MEDs’ direct control. Much of the BEP assistance to the MEDs has been for establishing and equipping the PDCs, and providing advice, materials, and/or modest funding for activities such as classroom makeovers. The BEP utilization of on-the-job training and coaching with partner municipalities seems to be limited to these activities, with concrete results on the classroom makeovers but fewer tangible results thus far with the PDCs. FINDINGS RELATED TO QUESTION #5 How effectively does BEP coordinate activities with other USAID and other donors’ programs? In 2013, the MEST accredited the Early Grade Reading Assessment in the Albanian language (A-EGRA) developed by the BEP. The A-EGRA is a formative and summative assessment tool that helps teachers diagnose student difficulties and provide appropriate support, while emphasizing higher order reading skills that allow students to comprehend, and learn from, the text they are reading. The A-EGRA was based on technology and research formulated over several previous years by the Research Triangle Institute and thus represents an effective coordination between past and present programs. This activity seemed beneficial to the evaluation team; it also addresses USAID’s Education Strategy which stresses greater achievement in early grade reading. Coincidentally, last year the MEST administered an A-EGRA test in a few selected schools to third-graders at the end of their school year and found that only 55% of students were proficient in language arts and only 35% demonstrated proficiency in mathematics. This was the first time the test had been given. The test can thus serve as a baseline; the evaluation team views these test results as relatively high when compared to other developing countries (e.g., Ethiopia, Malawi, Zambia, etc.). When meeting with the German official who shared this information and perspective, it was agreed that there was also room for improvement; in fact, the reading results helped compel the Kosovans’ decision to focus on improvement in this area. A-EGRA has been incorporated into BEP activities. A two-and-a-half day training course featuring the course program, assessment sheets, and administrator’s guide was developed. Initially, 54 teachers, 10 Program Master Learning Facilitators, and 3 professors from the University of Pristina’s Faculty of Education were trained to use A-EGRA. Also, teacher trainees in a later training course gathered data on Android-based tablet computers utilizing “Tangerine” open-source software prepared with USAID support, which they found preferable to the paper-based tests utilized by the initial trainees. Furthermore, FHI 360 personnel and some teachers in focus group discussions expressed interest in improving the BEP’s usage of A-EGRA, in conjunction with supporting reading skills as a basic foundation for good progress in all academic subjects and in life itself. In contrast, the evaluation team noted the low ranking (and thus limited value/importance) that questionnaire respondents across all groups assigned to “Reading Activities Involving Student and Parents.” The contrast between the focus group comments and the rank-order results thus reveal a potentially relevant discrepancy. The evaluation team has observed, first-hand, that interactive reading between a student and a parent (parent reading to child, child reading to parent) is of critical importance. The team’s first-hand experience is supported by applied research studies in many USAID-assisted countries, as well as studies in the United States and other developed countries worldwide. 20 In sum, assistance aimed at assessing and improving early grade reading is a real contribution on the BEP’s part. Future BEP activities, as well as any USAID follow-on project, should continue to build upon this productive, initial legacy by helping develop more books and supplementary reading materials, as well as emphasizing to parents and the general public the significance of reading well. The evaluation team also found evidence of effective coorindation with USAID programs in other countries. There were several learning and observational trips to nearby countries in relation to the BEP. These visits resulted in the immediate adoption or adaptation of useful features, processes, and techniques. For instance, the KEC Director informed the evaluation team that the Macedonian Center for Civic Education has an assessment component that the BEP partially adapted. The BEP’s Chief of Party (Keith Prenton) also served previously as COP in Macedonia on a reportedly successful USAID￾funded project, various features of which were incorporated into the BEP. Another good idea adapted from the Macedonian project was to produce and distribute to parents an attractive multi-colored brochure that succinctly and effectively explains why classroom-based student assessment is important and how it is being carried out in their children’s schools. This example demonstrates effective coordination via the adaptation of existing printed matter and media from other programs to the BEP. The evaluation team also searched for evidence of effective coordination between the BEP and programs funded by other donors. Since late 1999, when the U.N. Interim Administration Mission in Kosovo began its work, many international donor organizations and consultants have come to Kosovo to assist with education. The principal international donor organizations currently working in basic and secondary education in Kosovo appear to collaborate, conduct joint programming, and co-fund mainly on their own prerogative, rather than at the request of the MEST. Various key informants reported that the MEST is not proficient at donor coordination. MEST officials acknowledged that they do not have the personnel to handle and coordinate the plethora of international donor activities. However, the MEST donor coordinator—new to his position—said that he and MEST colleagues hope this situation can be rectified soon. Currently, at least three key MEST officials (the coordinators for each of BEP’s three components) participate in the regular quarterly meetings of the BEP’s Joint Oversight Committee (JOC). With respect to the BEP, MEST officials and other donor personnel commented that BEP’s Chief of Party, Dr. Keith Prenton, is easy to approach and relates well to international organizations. The World Bank representative said that Dr. Prenton came to meet with the World Bank’s Project Implementation Unit on several occasions for information exchange meetings and to explore potential collaboration. Dr. Prenton told the evaluation team that the BEP has adopted the World Bank’s idea of a School Development Plan conducted annually and that the World Bank and USAID have collaborated on BEP components 1 and 2, and to a lesser extent, on component 3. Currently, the World Bank is launching a new education project likely to be called the “Kosovo Educational System Improvement Project.” The World Bank Senior Operations Officer for Education mentioned five elements of the project: (1) the Bank will continue assisting financial reforms leading to local school financial autonomy; (2) the project will include school management with the hope that school directors will have their own autonomy; (3) the project will cover external student assessment and examination, with the grade 9 sample-based and the grade 12 matura; (4) teacher professional development will be incorporated into the project; and (5) the educational management information system (EMIS) will be a part of the project. These planned elements demonstrate similarities, and potential areas for further collaboration, between the BEP and a World Bank-funded follow-on effort. The BEP’s closest and most rewarding programmatic collaboration with another donor has been with GIZ, with whom there is an agreement for cooperation. From what GIZ and FHI 360 sources told the evaluation team, both appear to have consistently demonstrated reciprocity and dealt easily with each other, neither organization seemed fearful of losing plaudits or influence as a result of their 21 collaboration. GIZ’s Capacity Development in Basic Education Project, scheduled to operate until 2016 or 2017, complements the BEP by working with 145 schools in six municipalities (Prizren, Pristina, Kline, Kacanik, Fushe-Kosova, and Gjilan). GIZ is keenly interested in improving school management. GIZ formulated a 7-module, 21-day training course for school directors. USAID and the EU took or adopted 90% of this course, adding a few of their own elements. Thus, the BEP’s school director training is based on the GIZ modules and manuals. GIZ also established, with the MEST, comparable standards and capacity levels for school directors; BEP implementers reportedly assisted in developing these standards, as well as printing the standards for school directors. Dr. Prenton said it was FHI 360 who convinced the MEST to have (i.e., allow) this joint donor coordination and finance among GIZ, the EU, and USAID/BEP. In its meeting at the MEST, the Director of Pre-University Education (BEP coordinator for activities under component I) called the BEP one of the MEST’s strongest partner initiatives and asserted that the BEP is good for meeting the great challenge and responsibility of decentralization, and for supporting school management. He specifically pointed out that the capacity building and training have—in addition to being high quality—been in line with the standards followed by GIZ and other donors. Various teachers in focus group discussions, school directors in interviews, and several FHI 360 personnel also provided evidence that integration of gender diversity was being harmonized between the BEP and GIZ￾funded programs. Specifically, the MEST Director of Pre-University Education cited the “gender balance” as a term of mutual usage, and importance, to the MEST, BEP, and GIZ’s Basic Education Team Leader. Special care in the formulation of the mini-book series and early grade readers to balance female and male characters, avoid gender stereotyping, and use gender inclusive language were being practiced in a coordinated manner with GIZ, including special events such as celebration of International Women’s Day and a “Girls in Technology: Gaining Momentum” event organized in four municipalities. The evaluation team found that coordination of gender diversity contributed to fruitful and meaningful gender events, geared positively toward changing attitudes and behaviors. The evaluation team also found that GIZ’s program is in the process of incorporating the BEP’s classroom-based student assessment and related materials, and—with USAID’s permission—the BEP’s early grade reading materials. Also, the PDCs are utilized for BEP training as well as training from other providers, such as the GIZ. Examples of effective coordination with other donors may exist; however, the evaluation team was unable to interview the in-country UNICEF representative or the key EU education official, who was not in country during the team’s visit to Kosovo. FINDINGS RELATED TO QUESTION #6 Based on the review of BEP’s implementation and results, what recommendations are there for possible future USAID programming and/or other donors or governments in improving Kosovo education system? What recommendations are there for supporting the implementation of the reform process? The evaluation team’s review of the BEP’s implementation and results provided produced the following key recommendations for follow-on programs, or other donor initiatives, to improve education in Kosovo and supporting the country’s reform process: The evaluation team believes the PDCs are being under-utilized. Finding ways to further stimulate and encourage use of the PDCs should be considered as a component of a future project to strengthen Kosovo’s education reform and bolster the MEDs’ role in implementing the Pre-University Education Law. Any efforts that the BEP, MEST, MEDs, and other international donors can muster to assist Kosovo in collecting and writing up its native literature should be encouraged (i.e., folk tales, fairy tales, 22 fables, interesting information about Kosovo’s flora and fauna, geography, histories of notable Kosovars such artists, writers, scientists, etc., and other literature appealing to children). FINDINGS RELATED TO QUESTION #7 Apart from current coordination, in what other ways can BEP collaborate with other ongoing USAID programs? USAID/Kosovo just initiated a program titled “Inclusive Education and Integration in Kosovo,” which is being implemented by People in Need, a Czech NGO. This program will operate for 21 months, and has $270,000 in USAID funding and $70,000 in Czech funds. BEP stakeholders and implementers should consider collaborating with People In Need’s new program. FINDINGS RELATED TO QUESTION #8 Which of the identified deficiencies in the overall implementation of the program’s current objectives can be remedied in the remaining life of the program? What are recommendations and lessons learned? Question #8 is addressed comprehensively in the following section which outlines program deficiencies and recommendations. RECOMMENDATIONS The evaluation team identified various deficiencies related to implementation of the program’s three components. Below are a set of recommendations for remedying some of these deficiencies during the remaining life of the program: 1. FHI 360 should find ways to ensure that certificates are promptly awarded to teachers who receive the two-day training and present a portfolio of acceptable quality; likewise, FHI 360 should ensure that all outstanding certificates are properly awarded prior to ending the program. Failure to deliver duly earned certificates appears to be the largest complaint teachers have about the BEP. The problem appears to be occurring at the local school level. At a minimum, it should be made clear to all involved—both in the original Training-of-Trainers event and the subsequent cascade trainings held at individual schools—that the school director and school facilitator are responsible for certifying and forwarding to FHI 360 the portfolios and follow-up work of each teacher trained, all in a large notebook. It should be made especially clear to the school director and/or facilitator that waiting until all teachers have completed their work before sending finished portfolios to FHI 360/BEP is undesirable, as there will always be one or more teachers that do not complete their work. FHI 360 personnel should continue to carefully, yet fairly rapidly, verify the quality of the submissions before sending them on to MEST for certification. 2. BEP should create an online resource (e.g., SharePoint or something similar) where MEDs can exchange information with each other, inquire about problems, receive technical and pedagogical assistance, and share lessons learned and best practices in education. This recommendation should be implemented immediately. Presently, the MEDs all act alone, isolated from one another and from nearly every source that can assist them with the considerable tasks assigned to them. The MEDs have key roles in many aspects of the Kosovo pre-university education system, ostensibly with respect to BEP Component I (Subcomponent “Increased abilities of Municipal Education Departments (MEDs) to improve education and manage schools”) and BEP Component 3 (Subcomponent “Strengthen Municipal-level capacities to plan and organize professional teacher development activities”). MED leaders and employees face difficult, multi-faceted tasks that will require a number of years, much hard work, and assistance to resolve. Specifically, the PDC Coordinators will require training, assistance, and administrative and material support. 23 Perhaps current BEP corporate partners, Microsoft or Oracle, can assist in establishing this resource for the MEDs. 3. USAID should promote greater use of effective yet low-cost learning tools and supplementary materials during the remainder of the program. Conversely, USAID should reduce reliance on higher technology equipment; such equipment is expensive to purchase, maintain, upgrade, and repair; it is more easily damaged, is more difficult to utilize, and less appropriate for Kosovo’s education system. This recommendation should be implemented immediately. Low-cost options, aside from being inexpensive, can be more easily learned and utilized by both teachers and students, can be readily moved from classroom to classroom, require little maintenance or follow-up costs, and can often be made and/or repaired by teachers and students. There are many examples of effective, low-cost, tried-and-true teaching and learning materials that can be recommended, including: maps; globes; hanging and movable flipcharts for science, language arts, social studies, and other subjects; surprise/stimulation boxes; flannel boards; chalk/blackboards; nutrition charts; pencils/paper; and math manipulatives. Evaluation team members have found, in their teaching and observations in several countries, including the United States, that a simple globe of the world is a very effective and much-used learning tool that engenders young learners’ curiosity, interest, and understanding. 4. USAID and FHI 360 should further emphasize the importance of low-cost, easy-to-make booklets and other reading materials for young children, the importance of reading, and the immense benefit of parents reading with their children. Reading well is essential to success in all academic subject areas, and it is one key avenue to a meaningful, civic-oriented life. (GIZ is incorporating the BEP’s early grade readers into its Capacity Development in Basic Education Project in six major Kosovar municipalities. The BEP, together sometimes with communities and their schools, can purchase low￾cost, durable supplementary reading materials for young readers.) 5. There should be a responsible person, preferably a teacher, in each school who should be designated to coordinate with the PDC, and vice versa. This recommendation can be implemented immediately as part of the BEP’s assistance to municipalities, which already includes the establishment and effective use of the PDCs. Poor and/or costly transportation, limited free time, poor pay incentives, geography, lack of knowledge about a center’s availability, and other inconveniences create disincentives for using resource centers such as the PDCs. A person at each school (preferably a teacher) should be designated to coordinate and “market” the PDC’s assets, roles, uses, and potential benefits. At the same time, a designated PDC Coordinator should visit each of the municipality’s schools at least once a year to meet with the teachers and the school director to advertise the PDC and invite usage by the school. 6. An intervention aimed at integrating competent citizens into the municipal committees, to work with MED officials so that the most qualified candidates are hired for school director and teaching positions, would further support Kosovo’s reform process. This is already being piloted in Mitrovica and Pristina. If a successful and sustainable mechanism is found, and it can be supported by MEST, it might reduce the number of politically-appointed—and sometimes unqualified—directors and teachers. BEP and any similar USAID-assisted follow-on effort can join in this advocacy campaign as an initiative to encourage positive citizen involvement in education. 7. The BEP should provide high-quality, targeted training to school psychologists, especially in areas of Roma population, to assist them with their heavy, difficult workloads. 24 The need for reinstating psychologists and pedagogues was expressed to the evaluation team by the MEST’s Director of Pre-University Education, by teachers in various focus group discussions, and by several school directors in interviews. The school psychologist at a school in Ferezi, half of whose student body is Roma, explained during one focus group how important assessment and student counseling are for the children, for their families, and for relationships between these children and their families. The psychologist’s statement was totally supported by the other teachers participating in the focus group. The BEP should provide high-quality training for these few, selective school psychologists—many of whom have never received training—even if it means eliminating other planned, less urgently needed training for other educators. 8. Provide more BEP training kits, especially in large schools, as well as some hands-on training so more teachers know how to use them. Kit distribution and limited hands-on training can be enhanced during the remaining 16 months of the program. BEP training kits—for subjects such as mathematics, reading, language arts, and science—are highly useful for teachers who know how to utilize them. However, these kits can be quite challenging for some teachers as many do not know how to best put them into practical use. Some hands-on training is likely required, and more than one kit per subject is needed, particularly in larger schools. 9. Ensure the presence of all three BEP components in a school. The presence and integration of all three BEP components is important and can result in greater synergies and a more holistic approach compared to having only one or two BEP components. (The evaluation team was told one reason that all schools do not have all three components is that component 3 is more expensive, comparably, than components 1 and 2.) A brief review of all components implemented up to the present in the schools shows that schools are not exposed equally to all three components. To get full advantage of the school-based approach, it is advisable to introduce and implement all three components in the schools when the program expands. All three components in one school would create a greater critical mass for sustainable change at the school level, and provide better motivation for school directors and teachers. Additionally, there is strong correlation between the three components (e.g., teacher professional planning of school directors relates to the professional development of teachers, school directors’ role to evaluate teachers’ performance relates to teachers' professional development, etc.). 10. Expand the Assessment for Learning (AfL) training from two to three days, given the new teaching and learning approach inherent in this training. Many novel practices and techniques are involved in AfL training; numerous teachers, as well as the FHI 360 Assessment Officer, believe the training would be of higher quality and provide additional lasting benefits (e.g., more positive learning behaviors and actions on the part of students, teachers, and school directors) if it were three days in duration. The evaluation team agrees. 11. Encourage MEST and particularly MED personnel to visit schools more frequently (at least once a year), especially in rural areas. The evaluation team, during its limited visits and short time in￾country, found that some schools had not been visited for three years, potentially creating a feeling of disconnectedness. (Although USAID/BEP cannot directly implement this recommendation, it should advocate for it as much as possible). 12. Provide more post-training monitoring and mentoring, especially in rural schools. School-based training is ongoing; however, the evaluation team heard little concerning post-training monitoring and mentoring. Although personnel in the more urban schools appear to have fairly continuous training, training seems more sporadic with teachers in rural schools. Also, focus group discussions with teachers revealed very limited follow-up work after the teachers’ formal training. There was no evidence of regular and systematic mentoring, monitoring, and guidance to deepen learning and to adopt additional positive techniques and behavior. 25 13. Provide PDCs with more than one projector. More use could be made of some PDCs if more than one projector were available. This is one example of where there needs to be more technology. Some teachers and two PDC Coordinators mentioned this recommendation. As for lessons learned, regarding future projects that involve in-school technology, USAID should consider creating opportunities to mitigate the inequities between students from more privileged and technology savvy homes with those children having little or no at-home technology nor parents versed in technology. SPECIAL CONCERNS FOR USAID/KOSOVO CRITICAL THINKING SKILLS AND INTERACTIVE TEACHING AND LEARNING The evaluation team did not have the opportunity to research to what extent training and methodologies employed by BEP affected critical thinking skills. When focus group discussions and interviews turned to classroom-based assessment and, somewhat, in-service professional development of teachers, respondents felt that interactive teaching and learning, a more easy two-way flow of useful information between students and teachers, and a more convivial classroom atmosphere were attained due to the BEP. Some teachers and school directors said critical thinking skills are enhanced because both students and teachers feel more confident, knowledgeable and relaxed. On their anonymous questionnaires, KEC personnel were asked “To what extent, if any, is the BEP promoting critical thinking skills and creativity in teachers and students?” On a scale of 1 to 5 with 1 representing “not at all” and 5 representing “very much,” KEC personnel gave an average score or ranking of 4.2. TECHNOLOGY Technology and the provision of equipment to BEP-supported schools are major elements in the program. Technology's place in preparing Kosovar students for life and employment in the 21st century appear prominently in the great majority of photos on the covers and pages of BEP publications, often with one student displaying some piece of equipment to a student of the opposite sex. Students can undertake projects involving practical electronics and mechanics as well as computer-aided design and robotics. BEP and many participating schools have formed Technicians Clubs whereby knowledgeable students train teachers and other students on the use and repair of this equipment. Microsoft and the Oracle Corporation have aided BEP’s technology efforts with selected equipment, manuals, mounting competitions, fairs and exhibits, and with financial contributions. BEP has responded to MEST requests for technology. These efforts have been generally appreciated, particularly by school directors and MED officials. A variety of equipment has gone to each of the 25 PDCs which have partner agreements with BEP, as has equipment for the 21st Century Classrooms attached to each PDC, a room schools use for teaching science and technology classes. This equipment includes many devices and related supplies, such as computers, digital microscopes, electronic labs, flip cameras, LG47” TVs and TV mounting kits, Lenovo Thinkpad X130E notebooks, and many other items. However, several focus groups with teachers elicited the fact that they would like to use the various technology kits in their classes, but do not have the training to do so. (In one case witnessed by the team, some of this equipment could not be used due to extenuating circumstances.) The director and some teachers in a rural school in Mramor said the school received two cabinets of equipment, and supplies from BEP for chemistry and physics classes, but they have never been used because there is no water available. In our limited sample of schools visited (eight schools), the evaluation team did not see one example of BEP-provided technology being utilized. Most school directors showed the evaluation team the cabinets and classrooms where technological instruments, kits and supplies were stored, but were unable to provide information on how, or how often, they were used. In contrast, when the team visited classrooms, the teachers were observed teaching effectively and interactively with “traditional” learning aids and supplementary materials, principally made by the students with the assistance of 26 teachers (in itself a learning experience for young students). Many teachers interviewed believe—and the evaluation team agrees—that the BEP is overly dependent upon technology, much of which is not understood, nor the best option, for student learning and performance. Technology equipment and supplies were chosen early in the BEP after a group involving a British consultant, BEP’s Technology Coordinator, several MEST officials, and a school director went to the British Education and Training Technology fair in the United Kingdom. The group also visited an innovative Microsoft School. According to FHI 360’s Technology Coordinator, the BEP focus on technology was to develop 21st century skills in students. It was also a complement to a sizable MEST project to digitalize all schools with the Internet and computers. However, the MEST rescinded the solicitation and cancelled the project, apparently for a variety of reasons. When asked about follow-up training for teachers, FHI 360’s Technology Coordinator stated that BEP gave an additional one-day workshop to technology teachers to assist them with practical skills and hands-on work. He also mentioned that in 2013 the BEP carried out some follow-up activities and mentoring through random visits to schools and surveys with school directors. The evaluation team does not deny the importance of appropriate, useful technology, but believes that much of the technology and equipment deployed by the BEP is not worth the investment, and not as useful as some attractive, less sophisticated learning aids and supplementary materials. These could be flipcharts, charts and information on nutrition, science, mathematics, language arts and other topics that could be put on classroom walls and easily moved to other classrooms, or outdoors if the students are studying the natural environment. These materials have no or little maintenance costs. When interviewed, the GIZ Basic Education Team Leader stated that she firmly believes the greatest weakness of BEP is its over-reliance and over-dependence on technology. “Good teaching and learning cannot be solved by money and technology alone. Strong school management, solid teaching methods, and effective utilization of practical, relevant and appropriate manuals and supplementary materials are the keys,” she remarked. The shift from a poor school with principally blackboards and chalk to a school heavily employing semi-sophisticated technology may be too ambitious, particularly when a wide array of effective inexpensive teaching and learning materials can be purchased or made by students and teachers. The evaluation team also developed mixed views regarding the student Technician Clubs, Green Clubs, Student Councils and similar organizations. On the one hand, these organizations involve collective action, decision making, sharing tasks and other socio-civic activities that instill values and growth implied in USAID/Kosovo’s Development Objectives, particularly those in Democracy, Governance and Civil Society Strengthening. However, some of these activities, especially Technicians Clubs, appear to effectively favor children from higher socio-economic families, that are already relatively privileged, that already have computers and other technology in their homes, and that have parents who teach and/or encourage them to use technology. These children are a small minority, and as they become more technologically advanced they leave less fortunate students comparatively further behind. This issue was voiced by several school directors and by many teachers during focus groups. The MEST BEP Coordinator for component 3 also mentioned this issue, as did the GIZ and World Bank representatives. When dealing with school-based technology in future projects, USAID should consider creating opportunities to mitigate the inequities between those students who have considerable exposure to technology at home, and those students with little or no such at-home experience. INDICATORS UTILIZED BY USAID/KOSOVO AND BEP The evaluation team found the indicators utilized by USAID/Kosovo for BEP to be of special concern. Of the 42 indicators used in the Performance Monitoring Plan (PMP) for the BEP, 40 are numerical indicators. The other two deal with the degree of integration of a particular matter, and perceptions of a specific program aspect. The evaluation team finds the nearly exclusive use of numbers and numerical 27 indicators inadequate. Measurements of quality, merit, relevance, worth, value and usefulness are absent. Others shared the evaluation team’s concern about over-reliance on numerical indicators: the MEST BEP Coordinator for Component 3 pointed it out to the team, and mentioned she has frequently voiced her concern to BEP personnel, as well as at the JOC meetings. The international donor representatives interviewed by the evaluation team also believe that USAID/BEP relies too much on numerical indicators. To remedy this problem, simple, low-cost surveys addressing quality, value and usefulness of program elements could be administered to measure quality, relevance and other non-numerical indicators. A list of possible indicators for addressing quality, relevance, merit, and worth that could be used immediately and/or in a possible future USAID-assisted project similar to BEP include the following: Improved Quality of Teaching: 1. Percentage of teachers widely using continuous assessment as a pedagogical tool 2. Levels of satisfaction among teachers with newly-acquired interactive and participatory pedagogical skills 3. Degree and merit of follow-up activities to teachers and school directors previously trained 4. Surveys of pupils indicating satisfaction with teaching quality and methods Enhanced Community Participation in and Support of Local Schools: 1. Percentage of children who have a parent meet with the pupil’s teacher one-on-one at least once during the past year (for academic progress, not behavioral concerns) 2. Percentage of schools with Parents Councils active and involved in the quality and relevance of their schools 3. Parent surveys on the quality and effectiveness of management of their children’s schools Strengthened School Management: 1. Percentage of school directors receiving quality educational and administrative training 2. Percentage of primary and lower secondary schools with a School Development Plan and active School Board 3. Investments in books, teachers, teacher training and materials are increased and efficient 4. Percentage of MED officials and inspectors who receive good management training Improved Decentralized System of Education: 1. Percentage of schools reporting improved district or municipal support 2. PDCs strengthened to support quality, pertinent local-level education 3. Percentage increase in number of women recruited, trained and retained as teachers, and who are encouraged to teach in their home districts 4. Percentage increase in number of females who are recruited to become and serve well as school directors 5. Human Resources Information System helps municipalities know exactly who and where their personnel and retirees are and their status 6. Number of classrooms repaired or enhanced with local, municipal and/or USG support A-1 ANNEXES A-2 ANNEX A: LIST OF PERSONS MET AND/OR INTERVIEWED MEST PERSONNEL 1. Alush Istogu, Director, Pre-University Department, BEP Coordinator in MEST for Component 1 2. Iballe Cakaj, Head of Training sector, BEP Coordinator in MEST for Component 3 3. Ferit Idrizi, Director, Department for European Integration and Policy Coordination MEDs/PDCs 4. Diana Qarkaxhija, MED, Gjakove 5. Zeqir Meta, MED Prishtine 6. Hajri Ramadani, MED, Dragash 7. Idriz Kryeziu, PDC Coordinator, Prizren 8. Sadije Hajdini, PDC Coordinator, Ferizaj SCHOOL DIRECTORS 9. Yllka Juniku-Shehu, “Zekiria Rexha” School, Gjakove 10. Sadik Gojani, “Fehmi Agani” School, Gjakove 11. Islam Jusufi, “Rilindja” School, Keçekolle 12. Halim Gashi, “Avni Rustemi” School, Mramur 13. Flurim Qengaj, "Fetah Sulejmani,” School, Dragash 14. Naser Meleqi, “Ilmi Bahtijari” , School, Blaq I Vogel 15. Haki Gajraku, “Abdyl Frasheri,” School, Prizren 16. Basri Hasani, “Naim Frasheri,” School, Zaskok 17. Fahrije Retkoceri, “Green School,” School, Prishtine 18. Isak Bregaj, “Dardania,” School, Prishtine DEPUTY DIRECTORS 19. Flamur Godeni, “Zekiria Rexha” School, Gjakove 20. Afijete Doli, “Fehmi Agani” School, Gjakove 21. Emrush Ahmeti “Tefik Çanga” School, Ferizaj TEACHERS Focus Group in Mramur, “Avni Rustemi” School 22. Flurije Gashi, 23. Faton Sabedini 24. Hajriz Abazi 25. Veton Gashi 26. Xhelil Xhemshiti 27. Rexhep Gashi 28. Shukrije Ahmeti A-3 29. Hanife Gashi 30. Sabit Gashi 31. Sabit H. Gashi Focus Group in Keçekolle, “Rilindja” School 32. Skender Krasniqi 33. Salih Avdullahu 34. Nazmi Sahiti 35. Nazmi Islami 36. Bedrije Kasumi 37. Asllan Hasani 38. Vjollca Tahiri Focus Group in Gjakova, “Zekiria Rexha” School 39. Myrvete Mullademi 40. Diana Gaxha 41. Adrian Rudi 42. Pleurat Rudi 43. Labeate Dobruna Focus Group in Gjakova, “Fehmi Agani” School 44. Saranda Hoxha 45. Iliriana Shllaku 46. Besnik Bajraktari 47. Valbona Rrojeni 48. Mirlinda Misku 49. Ardita Misku 50. Elona Pozhegu 51. Eredeshire Metaj 52. Dafina Lama 53. Jetmira Hasimja 54. Mirlinda Lipa Rudi 55. Teuta Hoti 56. Bujare Mati 57. Arta Dalladaku 58. Miranda Shllaku 59. Nasihide Qurdina Focus Group in Dragash, “Fetah Sulejmani” School 60. Alltane Tershnjaku 61. Remzi Ramadani A-4 62. Vegim Vehapi 63. Azim Qengaj 64. Sejad Domuzeti 65. Valon Zejneli 66. Ruzhdi Shabani 67. Qazim Bojaxhiu 68. Shkelqim Hulaj 69. Sinan Aliu Focus Group in Prizren, “Avdyl Frasheri” School 70. Sadije Jakupi 71. Alidin Flugaj 72. Zana Saramati 73. Gazmend Thaqi 74. Milazim Avdylaj 75. Merita Hoxha Focus Group in Zaskok, “Naim Frasheri” School 76. Hava Lubishtani 77. Zarife Abdullahu 78. Mirlinda Çerkini 79. Nazire Aliu 80. Sevdije Osmani 81. Enver Haxhimusa 82. Hysni Çerkini 83. Naim Hamiti 84. Drita Lamaxhema 85. Irfan Granica Focus Group in Ferizaj, “Tefik Çanga” 86. Besjana Qiraxhiu 87. Emrush Ahmeti 88. Jakup Rexhepi 89. Rexhep Ismoni 90. Zymer Berisha 91. Vehbi Hajdini 92. Magbule Berisha 93. Nurije Maksuti 94. Luljeta Shabani 95. Vlora Osmani 96. Diana Merovci A-5 97. Fatusha Azemi 98. Artan Shabani KEC PERSONNEL 99. Dukagjin Pupovci, Executive Director at Kosova Education Center 100. Luljeta Demjaha, Local Teacher Development Coordinator 101. Jehona Xhaferi, Former MLF Component 3 FHI 360 PERSONNEL 102. Valmira Haxhaj Gushlla, Local School Management Coordinator; BEP COP as of April 1, 2014 103. Keith Prenton, Chief of Party (until March 31, 2014) 104. Arlinda Gashi-Bajgora, Deputy Chief of Party 105. Stephen Luke, Project Director, FHI 360/Washington, D.C. 106. Alma Kabashi, Finance Manager 107. Merima Dubova, Finance Officer 108. Edona Begu, Assessment Officer 109. Arsim Llazi, Educational Technology Coordinator USAID PERSONNEL 110. Maureen Shaukett, Mission Director 111. Matthew Laird, Acting Deputy Mission Director 112. Antigona Mustafa, Project Management Specialist—Education, AOR of BEP 113. Melita Cacaj, Monitoring and Evaluation Specialist, Program and Policy Office (PPO) 114. Afërdita Nimani, Development Assistance Assistant, PPO 115. Merita Teleqi-Devaja, Alternate AOR, BEP 116. Matthew Carpenter, Program Office and Monitoring and Evaluation, E + E Bureau, USAID/ (Washington D.C.) 117. Amy Southworth, Program Officer, PPO OTHER ENTITIES AND INDIVIDUALS 118. Vjollca Krasniqi, Gender Consultant BEP 119. Flora Kelmendi, Senior Operations Officer, World Bank Office in Kosovo 120. Dagmar Fuchs-Schmitz , Basic Education Team Leader, GIZ 121. Diar Hamiti, Student Council in "Naim Frasheri" School, Zaskok 122. Fitore Delolli, Student Council in "Naim Frasheri" School, Zaskok 123. Alma Delolli, Student Council in "Naim Frasheri" School, Zaskok A-6 ANNEX B: LIST OF DOCUMENTS REVIEWED 1. Academy for Educational Development (AED), Basic Education Program Annual Plan of Work, 1 December, 2010 – 30 September, 2011, 11/30/2010, Washington, D.C. 2. FHI Development 360 LLC (FHI 360), Basic Education Program Annual Plan of Work, 1 October, 2011 – 30 September, 2012, 2011, Washington, D.C. 3. FHI 360, Basic Education Program Annual Plan of Work, 1 October, 2012—30 September, 2013, 2012, Washington, D.C. 4. FHI 360, Basic Education Program Annual Plan of Work, 1 October, 2013—30 September, 2014, 2013, Washington, D.C. 5. FHI 360, Basic Education Program Annual Summary Progress Report: 2010-2011, 2011, Washington, D.C. 6. FHI 360, Basic Education Program Annual Summary Progress Report: 2011-2012, 2012, Washington, D.C. 7. FHI 360, Basic Education Program Annual Summary Progress Report: 2012-2013, 2013, Washington, D.C. 8. FHI 360, Basic Education Program Statistical Report 2010-11, 2011, Washington, D.C. 9. FHI 360, Basic Education Program Statistical Report 2012-2013, 2013, Washington, D.C. 10. USAID, Education: Opportunity Through Learning, USAID Education Strategy, February, 2011, Washington, D.C. 11. USAID, USAID Evaluation Policy, Bureau for Policy, Planning, and Learning, 2011, Washington, D.C. 12. USAID, One Mission One USAID, January 29, 2014, Washington, D.C. 13. USAID, 20 Years of USAID Economic Growth Assistance in Europe and Eurasia: Summary Report, Bureau for Europe and Eurasia, December, 2013, Washington, D.C. 14. USAID, USAID/Kosovo Country Development Cooperation Strategy 2014-2018, USAID/Kosovo, 2013, Pristina, Kosovo 15. World Bank, The World Bank in Kosovo: Country Snapshot, October 2013, Pristina, Kosovo and Washington, D.C. 16. U.S. Department of State, Press Statement by Secretary of State John Kerry on International Women’s Day, March 7, 2014, Washington, D.C. 17. Crouch, Louis and Gove Amber, Development, Purposes, Uses of EGRA (Early Grade Reading Assessment), presentation to Society for International Development, May 1, 2008, Research Triangle Institute, International, Washington, D.C. 18. Gove, Amber, Slifer-Mbacke, Lisa and Vinogradova, Elena, “The Early Grade Reading Assessment: Reflections and Possibilities,” PowerPoint presentation for USAID, March 18, 2014, Washington, D.C. 19. USAID, Snapshot of School Management Effectiveness (SSME): Development Update and Pilot Results, presentation by USAID on February 20, 2008, Washington, D.C. 20. USAID website and USAID/Kosovo Facebook page, “Breathing New Life into Kosovo’s Schools,” March 24, 2014 A-7 21. Ministry of Education, Science and Technology, The Law on Primary and Secondary Education in Kosovo, Prishtina, 2004 22. Ministry of Education and Science (MEST), Kosovo Education Strategic Plan 2011-2016, Prishtina 2011 23. Ministry of Education and Science (MEST), Law on Education in Municipalities, Prishtina 2008 24. Ministry of Education , Science and Technology (MEST), Kosovo Curriculum Framework, Prishtina 2011 25. MEST Administrative Instruction (5/2010) Teachers Licensing A-8 ANNEX C: ANONYMOUS QUESTIONNAIRES Anonymous Questionnaire for Kosovo Education Center (KEC) Personnel Pyetjet e Intervistës për personelin e Qendrës për Arsim të Kosovës Mid-term Evaluation of the Kosovo Basic Education Project (BEP) Vlerësim Afat-mesëm i Programit për Arsim Themelor në Kosovë (BEP) This questionnaire is anonymous and confidential. Thus, your name will not be solicited or revealed. We are attempting to objectively measure the quality, merit and worth of the BEP. Ky pyetësorë është anonim dhe konfidencial. Kështu, emri juaj nuk do të kërkohet apo zbulohet. Ne po përpiqemi që objektivisht të masim cilësinë, meritat dhe rëndësinë e BEP. Please leave blank or unanswered those questions about which you have no information. Ju lutemi të lini zbrazët ose të mos përgjigjeni ato pyetje për të cilat nuk keni informatë. 1. What is the degree to which the BEP program is helping the actual decentralization of education in Kosovo? Deri në çfarë shkalle programi BEP është duke ndihmuar decentralizimin aktual të arsimit në Kosovë? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 2. What is the value of the current contributions of the Professional Development Centers (PDCs) in municipalities? Cila është vlera e kontributeve aktuale e Qendrave të Zhvillimit Profesional (PDCs) në komuna? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 3. What is the potential value of contributions the PDCs can make in the future? Cila është vlera e mundshme e kontributeve që Qendrat e Zhvillimit Profesional mund të bëjnë në të ardhmen? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 4. To what extent is the capacity of municipalities to improve education increasing? Deri në çfarë mase është kapaciteti i komunave për të përmisuar rritjen në arsim? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm A-9 5. To what degree, if any, is the capacity of the PDCs to provide high-quality professional development activities for teachers increasing? Deri në çfarë shkalle, nëse ka, është kapaciteti i Qendrave për Zhvillim Profesional të sigurojë aktivitete të zhvillimit profesional të cilësisë së lartë për mësimdhënësit në ngritje? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 6. In your opinion, what is the quality of technical assistance provided by the BEP program? Sipas mendimit tuaj, çfarë është cilësia e asistencës teknike që ofrohet nga Programi BEP? Very low low average high very high Shumë e ulët e ulët mesatare e lartë shumë e lartë 7. To what degree, if any, is in-service teacher development improving as a result of BEP? Deri në çfarë shkalle, nëse ka, trajnimi për zhvillimin e mësimdhënësve është duke përmisuar si rezultat i BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 8. What is the degree of success thus far of the program-targeted schools under the BEP? Çfarë është deri më tani shkalla e suksesit e programit për shkolla në kuadër të BEP? None low some high very high Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 9. To what extent could the teachers’ union (SBASHK) be more positively involved in the BEP program than it is currently? Deri në çfarë mase Sindikata e Bashkuar e Arsimit (SBASHK) mund të përfshihet në programin BEP në mënyrë më pozitive sesa që është tani? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 10. What is the value or merit of BEP’s partnership with local communities, including with the “classroom makeover” renovations? Cila është vlera apo merita e partneritetit të BEP-it me komunitetet lokale, duke perfshirë edhe riparimet e klasave? None low average high very high Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm A-10 11. What is the usefulness of the PDC coordinators in each subject area? Çfarë është dobia e koordinatoreve të Qendrave për Zhvillim Profesional në secilën fushë? Very low low average high very high Shumë e ulët e ulët mesatar e lartë shumë e lartë 12. To what extent, if any, is the BEP program promoting critical thinking skills and creativity in teachers and students? Deri në çfarë mase, nëse ka, Programi BEP është duke promovuar aftësitë e të menduarit kritik dhe kreativitetin tek mësimdhënësit dhe studentët? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 13. The BEP program consists of many elements or components, some of them intentionally related. In your opinion, please rank order the importance or value of each of the 14 following components, ranging from a “1” for the component with the least value or importance to a “14” for that component you deem most important. Please use each number only once, thus covering all 14 listed components. Programi BEP përbëhet nga disa elemente dhe komponenta, disa nga ato qëllimisht të ngjashme. Sipas mendimit tuaj, ju lutem ti renditni secilën sipas rëndësisë apo vlerës nga 14 komponentat në vazhdim, duke renditur nga “1” për kompontën me më së paku vlerë apo rëndësi deri në “14” për komponentën që konsideroni si më të rëndësishmen. Ju lutem të perdorni secilin numër vetëm një herë, duke perfshirë të gjitha 14 komponentat e listuara. ______ developing standards for school management Zhvillimi i standardeve për menaxhmentin e shkollës ______ gender and gender diversity Gjinia dhe shumëllojshmëria gjinore ______ Professional Development Centers Qendrat për Zhvillim Profesional ______ integrating non-Albanian ethnic minorities more fully into the education system Përfshirja e pakicave etnike jo-Shqiptare më shumë në sistemin arsimor ______ training and professional development of School Directors Trajnimi dhe zhvillimi profesional i Drejtorëve të Shkollave ______ 21st century technology classrooms Klasat e teknologjisë të shekullit 21 A-11 ______ reading activities involving students and parents Interpretimi i aktiviteteve duke përfshirë studentët dhe prindërit ______ Student Councils and student clubs Këshillet studentore dhe klubet studentore ______ national standards in assessment for learning Standardet ndërkombëtare në vlerësimin për mësim ______ school-based facilitators in various subjects Ndihmësit për shkolla në landë të ndryshme ______ institutional strengthening and capacity development of the MEST, Municipal Education Departments and other entities Forcimi institucional dhe zhvillimi i kapaciteteve të MASHT, Departamentet e Arsimit Komunal, Qendra për Arsim e Kosovës dhe subjekte të tjera ______ school-based teacher training and professional development Trajnimi i mësimdhënësve bazuar në shkolla dhe zhvillimi profesional ______ school boards’ and citizen/parent involvement generally in the education system Përfshirja e Bordeve të shkollave dhe qytetarët/prindërit në përgjithësi në sistemin arsimor ______ other component or element (please specify) Ndonjë element apo komponentë tjetër (ju lutem veçojeni) 14. What is your overall view of the Basic Education Project, especially in regard to the KEC’s work in component three: improve the in-service professional development of teachers? Cila është pikëpamja juaj e përgjithshme për Projektin Themelor për Arsim, veçanërisht në lidhje me punën e Qendrës për Arsim të Kosovës në komponentën e tretë: trajnimi për të rritur zhvillimin profesional për mesimdhënësit? A-12 15. Please discuss the roles of technology in the BEP program. Ju lutem shqyrtoni funksionet e teknologjisë në programin BEP 16. What, in your opinion, are the major strengths and weaknesses of the BEP program? Cilat, sipas mendimit tuaj, janë anët e forta dhe të anët e dobëta të BEP programit? 17. What recommendations—general and specific—do you make to improve the BEP program? Cilat janë rekomandimet në përgjithësi dhe në veçanti që ju do të bënit për të përmisuar BEP programin? [Thank you very much for the time, attention and information you have given for this mid-term evaluation of the Basic Education Program.] [Ju faleminderit shumë për kohën, vëmendjen dhe informatat që keni dhënë për këtë vlerësim afat￾mesëm të Programit për Arsim Themelor] A-13 Anonymous Questionnaire for Ministry of Education, Science & Technology (MEST) Personnel Pyetjet e Intervistës për personelin e Ministrisë së Arsimit, Shkencës dhe Teknologjisë Mid-term Evaluation of the Kosovo Basic Education Project (BEP) Vlerësim Afat-mesëm i Programit për Arsim Themelor në Kosovë (BEP) This questionnaire is anonymous and confidential. Thus, your name will not be solicited or revealed. We are attempting to objectively measure the quality, merit and worth of the BEP. Ky pyetësorë është anonim dhe konfidencial. Kështu, emri juaj nuk do të kërkohet apo zbulohet. Ne po përpiqemi që objektivisht të masim cilësinë, meritat dhe rëndësinë e BEP. Please leave blank or unanswered those questions about which you have no information. Ju lutemi të lini zbrazët ose të mos përgjigjeni ato pyetje për të cilat nuk keni informatë. 1. What is the degree to which the BEP program is helping the actual decentralization of education in Kosovo? Deri në çfarë shkalle programi BEP është duke ndihmuar decentralizimin aktual të arsimit në Kosovë? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 2. What is the value of the current contributions of the Professional Development Centers (PDCs) in municipalities? Cila është vlera e kontributeve aktuale e Qendrave të Zhvillimit Profesional (PDCs) në komuna? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 3. What is the potential value of contributions the PDCs can make in the future? Cila është vlera e mundshme e kontributeve që Qendrat e Zhvillimit Profesional mund të bëjnë në të ardhmen? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 4. To what extent, if any, has school management improved because of the BEP program? Deri në cfarë mase, nëse ka, është përmisuar menaxhmenti në shkolla në kuadër të BEP Programit? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 5. To what degree is there sharing of information and practices among municipalities with the BEP program? Deri në cfarë shkalle është ndarja e praktikave dhe informatave ndërmjet komunave dhe BEP Programit? None a little some much very much Asnjëra e ulët afërsisht e lartë shumë e lartë A-14 6. In your opinion, what is the quality of technical assistance provided by the BEP program? Sipas mendimit tuaj, çfarë është cilesia e asistencës teknike që është ofrohet nga programi BEP? Very low low average high very high Shumë e ulët e ulët mesatare e lartë shumë e lartë 7. To what degree is in-service teacher development improving as a result of BEP? Deri në çfarë shkalle, trajnimi për zhvillimin e mësimdhënësve është duke përmisuar si rezultat i BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 8. What is the degree of success thus far of the program-targeted schools under the BEP? Cila është deri më tani shkalla e suksesit të programit për shkolla në kuadër të BEP? None low some high very high Asnjëra e ulët afërsisht e lartë shumë e lartë 9. To what extent could the teachers’ union (SBASHK) be more positively involved in the BEP program than it is currently? Deri në çfarë mase Sindikata e Bashkuar e Arsimit (SBASHK) mund të përfshihet në programin BEP në mënyrë më pozitive sesa që është tani? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 10. What is the value or merit of BEP’s citizen involvement and partnership with local communities, including with the “classroom makeover” renovations? Cila është vlera apo merita e partneritetit të BEP-it me komunitetet lokale, duke perfshirë edhe riparimet e klasave? None low average high very high Asnjëra e ulët mesatare e lartë shume e lartë 11. In your opinion, what is the quality of foreign donor cooperation and joint programming with the BEP program? Sipas mendimit tuaj, çfarë është cilësia e bashkëpunimit në mës të donatorëve të jashtëm dhe programit të përbashkët me BEP Programin? Very poor poor average good very good Shumë I dobët I Dobët mesatare I mirë Shumë I mirë 12. To what degree has the Joint Oversight Committee contributed to the BEP program? Deri në cfarë shkalle ka kontribuar Komiteti i Përbashket Mbikëqyrës në programin BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm A-15 13. In your view, what have been the opportunities for girls and women in the BEP program? Sipas pikëpamjes tuaj, cilat ishin mundësitë për vajza dhe gra në programin BEP? Very poor poor average good very good Shumë I dobët I Dobët mesatare I mirë Shumë I mire 14. The BEP program consists of many elements or components, many of them intentionally related. In your opinion, please rank order the importance or value of each of the 14 following components, ranging from a “1” for the component with the least value or importance to a “14” for that component you deem most important. Please use each number only once, thus covering all 14 listed components. Programi BEP përbëhet nga disa elemente dhe komponenta, disa nga ato qëllimisht të ngjashme. Sipas mendimit tuaj, ju lutem ti renditni secilën sipas rëndësisë apo vlerës nga 14 komponentat në vazhdim, duke renditur nga “1” për kompontën me më së paku vlerë apo rëndësi deri në “14” për komponentën që konsideroni si më të rëndësishmen. Ju lutem të perdorni secilin numër vetëm një herë, duke perfshirë të gjitha 14 komponentat e listuara. ______ developing standards for school management Zhvillimi i standardeve për menaxhmentin e shkollës ______ gender and gender diversity Gjinia dhe shumëllojshmëria gjinore ______ Professional Development Centers Qendrat për Zhvillim Profesional ______ integrating non-Albanian ethnic minorities more fully into the education system Përfshirja e pakicave etnike jo-Shqiptare më shumë në sistemin arsimor ______ training of School Directors Trajnimi i Drejtorëve të shkollave ______ 21st century technology classrooms Klasat e teknologjisë të shekullit 21 ______ reading activities involving students and parents Interpretimi i aktiviteteve duke përfshirë studentet dhe prindërit ______ Student Councils and student clubs Këshillet studentore dhe klubet studentore ______ national standards in assessment for learning Standardet ndërkombëtare në vlerësimin për mësim ______ school-based facilitators in various subjects Ndihmësit për shkolla në landë të ndryshme A-16 ______ institutional strengthening and capacity development of the MEST, Municipal Education Departments and other entities Forcimi institucional dhe zhvillimi i kapaciteteve të MASHT, Departamentet e Arsimit Komunal, Qendra për Arsim e Kosovës dhe subjekte të tjera ______ school-based teacher training Trajnimi i mësimdhënësve bazuar në shkolla ______ school boards and citizen/parent involvement generally in the education system Përfshirja e Bordeve të shkollave dhe qytetarët/prindërit në përgjithësi në sistemin arsimor ______ other component or element (please specify) Ndonjë element apo komponentë tjetër (ju lutem veçojeni) 15. What is your overall view of the Basic Education Project? Cili është mendimi juaj i përgjithshëm për Projektin për Arsim Themelor? 16. Please discuss briefly the roles of technology in the BEP program. Ju lutem shqyrtoni funksionet e teknologjisë në programin BEP 17. What, in your opinion, are the major strengths and weaknesses of the BEP program? Cilat, sipas mendimit tuaj, janë anët e forta dhe të anët e dobëta të BEP programit? 18. What recommendations—general and specific—do you make to improve the BEP program? Cilat janë rekomandimet në përgjithësi dhe në veçanti që ju do të bënit për të përmisuar BEP programin? [Thank you very much for the time, attention and information you have given for this mid-term evaluation of the Basic Education Program.] A-17 [Ju faleminderit shumë për kohën, vëmendjen dhe informatat që keni dhënë për këtë vlerësim afat￾mesëm të Programit për Arsim Themelor] Anonymous Questionnaire for School Directors Mostër e pyetjeve të intervistës për Drejtorë të Shkollave Mid-term Evaluation of the Kosovo Basic Education Project (BEP) Vlerësim Afat-mesëm i Programit për Arsim Themelor në Kosovë (BEP) This questionnaire is anonymous and confidential. Thus, your name will not be solicited or revealed. We are attempting to objectively measure the quality, merit and worth of the BEP. Ky pyetësorë është anonim dhe konfidencial. Kështu, emri juaj nuk do të kërkohet apo zbulohet. Ne po përpiqemi që objektivisht të masim cilësinë, meritat dhe rëndësinë e BEP. Please leave blank or unanswered those questions about which you have no information. Ju lutemi të lini zbrazët ose të mos përgjigjeni ato pyetje për të cilat nuk keni informatë. 1. What is the degree to which the BEP program is helping the actual decentralization of education in Kosovo? Deri në çfarë shkalle programi BEP është duke ndihmuar decentralizimin aktual të arsimit në Kosovë? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 2. What is the value of the current contributions of the Professional Development Centers (PDCs) in municipalities? Cila është vlera e kontributeve aktuale e Qendrave të Zhvillimit Profesional (PDCs) në komuna? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 3. What is the potential value of contributions the PDCs can make in the future? Cila është vlera e mundshme e kontributeve që Qendrat e Zhvillimit Profesional mund të bëjnë në të ardhmen? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 4. What is the quality of training for School Directors under the BEP program? Çfarë është cilësia e trajnimit për Drejtorët e shkollave në kuadër të BEP programit? Very poor poor average good very good Shumë I dobët I Dobët mesatar I mirë Shumë I mirë A-18 5. To what extent is the capacity of municipalities to improve education increasing? Deri në çfarë mase është kapaciteti i komunave për të përmisuar rritjen në arsim? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 6. To what degree is there sharing of information and practices among schools and municipalities with the BEP program? Deri në çfarë shkalle është ndarja praktikave dhe informatave ndërmjet shkollave dhe komunave së bashku me BEP Programin? None a little some much very much Asnjëra e ulët afërsisht e lartë shumë e lartë 7. In your opinion, what is the quality of technical assistance provided by the BEP program? Sipas mendimit tuaj, çfarë është cilesia e asistencës teknike që është ofrohet nga programi BEP? Very low low average high very high Shumë e ulët e ulët mesatare e lartë shumë e lartë 8. To what degree is in-service teacher development improving as a result of BEP? Deri në çfarë shkalle, trajnimi për zhvillimin e mësimdhënësve është duke përmisuar si rezultat i BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 9. What is the degree of success thus far of the program-targeted schools under the BEP? Cila është deri më tani shkalla suksesit të programit për shkolla në kuadër të BEP? None low some high very high Asnjëra e ulët afërsisht e lartë shumë e lartë 10. To what extent could the teachers’ union (SBASHK) be more positively involved in the BEP program than it is currently? Deri në çfarë mase Sindikata e Bashkuar e Arsimit (SBASHK) mund të përfshihet në programin BEP në mënyrë më pozitive sesa që është tani? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 11. What is the value or merit of BEP’s partnership with local communities, including with the “classroom makeover” renovations? A-19 Cila është vlera apo merita e partneritetit të BEP-it me komunitetet lokale, duke perfshirë edhe riparimet e klasave? None low average high very high Asnjëra e ulët mesatare e lartë shume e lartë 12. To what extent do you feel you have improved as a School Director because of the BEP program? Deri në çfarë mase mendoni se keni arritur përmisim si Drejtor i Shkollës në kuadër të BEP Programit? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 13. The BEP program consists of many elements or components, some of them intentionally related. In your opinion, please rank order the importance or value of each of the 14 following components, ranging from a “1” for the component with the least value or importance to a “14” for that component you deem most important. Please use each number only once, thus covering all 14 listed components. Programi BEP përbëhet nga disa elemente dhe komponenta, disa nga ato qëllimisht të ngjashme. Sipas mendimit tuaj, ju lutem ti renditni secilën sipas rëndësisë apo vlerës nga14 komponentat në vazhdim, duke renditur nga “1” për kompontën me më së paku vlerë apo rëndësi deri në “14” për komponentën që konsideroni si më të rëndësishmen. Ju lutem të perdorni secilin numër vetëm një herë, duke perfshirë të gjitha 14 komponentat e listuara. ______ developing standards for school management Zhvillimi i standardeve për menaxhmentin e shkollës ______ gender and gender diversity Gjinia dhe shumëllojshmëria gjinore ______ Professional Development Centers Qendrat për Zhvillim Profesional ______ integrating non-Albanian ethnic minorities more fully into the education system Përfshirja e pakicave etnike jo-Shqiptare më shumë në sistemin arsimor ______ training of School Directors Trajnimi i Drejtorëve të shkollave ______ 21st century technology classrooms Klasat e teknologjisë të shekullit 21 ______ reading activities involving students and parents A-20 Interpretimi i aktiviteteve duke përfshirë studentet dhe prindërit ______ Student Councils and student clubs Këshillet studentore dhe klubet studentore ______ national standards in assessment for learning Standardet ndërkombëtare në vlerësimin për mësim ______ school-based facilitators in various subjects Ndihmësit për shkolla në landë të ndryshme ______ institutional strengthening and capacity development of the MEST, Municipal Education Departments and other entities Forcimi institucional dhe zhvillimi i kapaciteteve të MASHT, Departamentet e Arsimit Komunal, Qendra për Arsim e Kosovës dhe subjekte të tjera ______ school-based teacher training Trajnimi i mësimdhënësve bazuar në shkolla ______ school boards and citizen/parent involvement generally in the education system Përfshirja e Bordeve të shkollave dhe qytetarët/prindërit në përgjithësi në sistemin arsimor ______ other component or element (please specify) Ndonjë element apo komponentë tjetër (ju lutem veçojeni) 14. What is your overall view of the Basic Education Project? Cili është mendimi juaj i përgjithshëm për Projektin për Arsim Themelor? 15. Please discuss the roles of technology in the BEP program. Ju lutem shqyrtoni funksionet e teknologjisë në programin BEP A-21 16. What, in your opinion, are the major strengths and weaknesses of the BEP program? Cilat, sipas mendimit tuaj, janë anët e forta dhe anët e dobëta të BEP programit? 17. What recommendations—general and specific—do you make to improve the BEP program? Cilat janë rekomandimet në përgjithësi dhe në veçanti që ju do të bënit për të përmisuar BEP programin? [Thank you very much for the time, attention and information you have given for this mid-term evaluation of the Basic Education Program.] [Ju faleminderit shumë për kohën, vëmendjen dhe informatat që keni dhënë për këtë vlerësim afat￾mesëm të Programit për Arsim Themelor] A-22 Anonymous Questionnaire for Teachers Pyetjet e Intervistës për Mësimdhënësit Mid-term Evaluation of the Kosovo Basic Education Project (BEP) Vlerësim Afat-mesëm i Programit për Arsim Themelor në Kosovë (BEP) This questionnaire is anonymous and confidential. Thus, your name will not be solicited or revealed. We are attempting to objectively measure the quality, merit and worth of the BEP. Ky pyetësorë është anonim dhe konfidencial. Kështu, emri juaj nuk do të kërkohet apo zbulohet. Ne po përpiqemi që objektivisht të masim cilësinë, meritat dhe rëndësinë e BEP. Please leave blank or unanswered those questions about which you have no information. Ju lutemi të lini zbrazët ose të mos përgjigjeni ato pyetje për të cilat nuk keni informatë. 1. To what degree is the BEP program helping the actual decentralization of education in Kosovo? Deri në çfarë shkalle programi BEP është duke ndihmuar decentralizimin aktual të arsimit në Kosovë? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 2. What is the value of the current contributions of the Professional Development Centers (PDCs) in municipalities? Cila është vlera e kontributeve aktuale e Qendrave të Zhvillimit Profesional (PDCs) në komuna? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 3. What is the potential value of contributions the PDCs can make in the future? Cila është vlera e mundshme e kontributeve që Qendrat e Zhvillimit Profesional mund të bëjnë në të ardhmen? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 4. To what extent is the capacity of municipalities to improve education increasing? Deri në çfare mase është kapaciteti i komunave për të përmisuar rritjen në arsim? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 5. To what degree, if any, do you feel you have improved as a teacher because of the BEP program? Deri në çfare niveli, nëse ka, mendoni qe ju si mësimdhënës keni arritur përmisim në kuadër të programit BEP? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 6. To what extent do teachers share teaching practices and methods among themselves? Deri në çfare mase mësimdhënësit ndajnë praktikat dhe metodat e mësimdhënies ndërmjet tyre? A-23 Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 7. In your opinion, what is the quality of technical assistance provided by the BEP program? Sipas mendimit tuaj, çfare është cilësia e asistencës teknike që është ofrohet nga programi BEP? Very low low average high very high Shumë e ulët e ulët mesatare e lartë shumë e lartë 8. To what extent is in-service teacher development improving as a result of BEP? Deri në çfarë shkalle, trajnimi për zhvillimin e mësimdhënësve është duke përmisuar si rezultat i BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 9. What is the degree of success thus far of the program-targeted schools under the BEP? Çfarë është deri më tani shkalla e suksesit të programit për shkolla në kuadër të BEP? None low some high very high Asnjëra e ulët afërsisht e lartë shumë e lartë 10. To what extent could the teachers’ union (SBASHK) be more positively involved in the BEP program than it is currently? Deri në çfarë mase Sindikata e Bashkuar e Arsimit (SBASHK) mund të përfshihet në programin BEP në mënyrë më pozitive sesa që është tani? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 11. To what degree has the BEP program involved parents and other citizens in schools generally and in their children’s learning specifically? Deri në çfarë shkalle programi BEP ka perfshirë prindërit dhe qytarët e tjerë në shkolla në pergjithësi dhe në të mësuarit e fëmijëve të tyre në veçanti? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftuëshem Kënaqshëm 12. In your opinion, what is the quality of teacher training and professional development under the BEP program? Sipas mendimit tuaj, cili është kualiteti i trajnimit dhe zhvillimit profesional te mësimdhënësve në kuadër të BEP programit? Very poor poor average good very good Shume i dobët I Dobët mesatar I mirë Shume i mirë 13. The BEP program consists of many elements or components, some of them intentionally related. In your opinion, please rank order the importance or value of each of the 14 following components, ranging from a “1” for the component with the least value or importance to a “14” for that A-24 component you deem most important. Please use each number only once, thus covering all 14 listed components. Programi BEP përbëhet nga disa elemente dhe komponenta, disa nga ato qëllimisht të ngjashme. Sipas mendimit tuaj, ju lutem ti renditni secilën sipas rëndësisë apo vlerës nga 14 komponentat në vazhdim, duke renditur nga “1” për kompontën me më së paku vlerë apo rëndësi deri në “14” për komponentën që konsideroni si më të rëndësishmen. Ju lutem të perdorni secilin numër vetëm një herë, duke perfshirë të gjitha 14 komponentat e listuara. ______ developing standards for school management Zhvillimi i standardeve për menaxhmentin e shkollës ______ gender and gender diversity Gjinia dhe shumëllojshmëria gjinore ______ Professional Development Centers Qendrat për Zhvillim Profesional ______ integrating non-Albanian ethnic minorities more fully into the education system Përfshirja e pakicave etnike jo-Shqiptare më shumë në sistemin arsimor ______ training of School Directors Trajnimi i Drejtorëve të shkollave ______ 21st century technology classrooms Klasat e teknologjisë të shekullit 21 ______ reading activities involving students and parents Interpretimi i aktiviteteve duke përfshirë studentet dhe prindërit ______ Student Councils and student clubs Këshillet studentore dhe klubet studentore ______ national standards in assessment for learning Standardet ndërkombëtare në vlerësimin për mësim ______ school-based facilitators in various subjects Ndihmësit për shkolla në landë të ndryshme ______ institutional strengthening and capacity development of the MEST, Municipal Education Departments, the Kosovo Education Center and other entities Forcimi institucional dhe zhvillimi i kapaciteteve të MASHT, Departamentet e Arsimit Komunal, Qendra për Arsim e Kosovës dhe subjekte të tjera ______ school-based teacher training Trajnimi i mësimdhënësve bazuar në shkolla ______ school boards and citizen/parent involvement generally in the education system Përfshirja e Bordeve të shkollave dhe qytetarët/prindërit në përgjithësi në sistemin arsimor A-25 ______ other component or element (please specify) Ndonjë element apo komponentë tjetër (ju lutem veçojeni)) 14. As a teacher, what is your overall view of the Basic Education Project? Si mësimdhënës, cila është pikëpamja juaj e përgjithshme për Projektin për Arsim Themelor? 15. Please discuss the roles of technology in the BEP program. Ju lutem shqyrtoni funksionet e teknologjisë në BEP programin. 16. What, in your opinion, are the major strengths and weaknesses of the BEP program? Cilat, sipas mendimit tuaj, janë anët e forta dhe anet e dobëta të BEP programit? 17. What recommendations—general and specific—do you make to improve the BEP program? Cilat janë rekomandimet në përgjithësi dhe në veçanti qe ju do të bënit për të përmisuar BEP programin? [Thank you very much for the time, attention and information you have given for this mid-term evaluation of the Basic Education Program.] [Ju faleminderit shumë për kohën, vëmendjen dhe informatat që keni dhënë për këtë vlerësim afat￾mesëm të Programit për Arsim Themelor] A-26 Anonymous Questionnaire for Municipal Education Department (MED) Officials Pyetjet e Intervistës per Zyrtarët e Drejtorisë Komunale të Arsimit Mid-term Evaluation of the Kosovo Basic Education Project (BEP) Vlerësim Afat-mesëm i Programit për Arsim Themelor në Kosovë (BEP) This questionnaire is anonymous and confidential. Thus, your name will not be solicited or revealed. We are attempting to objectively measure the quality, merit and worth of the BEP. Ky pyetësorë është anonim dhe konfidencial. Kështu, emri juaj nuk do të kërkohet apo zbulohet. Ne po përpiqemi që objektivisht të masim cilësinë, meritat dhe rëndësinë e BEP. Please leave blank or unanswered those questions about which you have no information. Ju lutemi të lini zbrazët ose të mos përgjigjeni ato pyetje për të cilat nuk keni informatë. 1. What is the degree to which the BEP program is helping the actual decentralization of education in Kosovo? Deri në çfarë shkalle programi BEP është duke ndihmuar decentralizimin aktual të arsimit në Kosovë? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 2. What is the value of the current contributions of the Professional Development Centers (PDCs) in municipalities? Cila është vlera e kontributeve aktuale e Qendrave të Zhvillimit Profesional (PDCs) në komuna? None a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 3. What is the potential value of contributions the PDCs can make in the future? Cila është vlera e mundshme e kontributeve që Qendrat e Zhvillimit Profesional mund të bëjnë në të ardhmen? None a little some much very much Asnjëra Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 4. To what extent, if any, is the capacity of municipalities to improve education increasing? Deri në çfarë mase, nëse ka, është kapaciteti i komunave për të përmisuar rritjen në arsim? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 5. To what degree is there sharing of information and practices among municipalities with the BEP program? Deri në çfarë shkalle është ndarja praktikave dhe informatave ndërmjet shkollave dhe komunave së bashku me BEP Programin? None a little some much very much Asnjëra e ulët afërsisht e lartë shume e lartë A-27 6. In your opinion, what is the quality of technical assistance provided by the BEP program? Sipas mendimit tuaj, çfarë është cilesia e asistencës teknike që është ofrohet nga programi BEP? Very low low average high very high Shumë e ulët e ulët mesatare e lartë shumë e lartë 7. To what degree is in-service teacher development improving as a result of BEP? Deri në çfarë shkalle, trajnimi për zhvillimin e mësimdhënësve është duke përmisuar si rezultat i BEP? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 8. What is the degree of success thus far of the program-targeted schools under the BEP? Cila eshte deri me tani shkalla suksesit te programit per shkolla ne kuader te BEP? None low some high very high Asnjëra e ulët afërsisht e lartë shumë e lartë 9. To what extent could the teachers’ union (SBASHK) be more positively involved in the BEP program than it is currently? Deri në çfarë mase Sindikata e Bashkuar e Arsimit (SBASHK) mund të përfshihet në programin BEP në mënyrë më pozitive sesa që është tani? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 10. What has been the value or merit of BEP’s citizen involvement and partnership with local communities, including with the “classroom makeover” renovations? Cila është vlera apo merita e partneritetit të BEP-it me komunitetet lokale, duke perfshirë edhe riparimet e klasave? None low average high very high Asnjera e ulet mesatare e larte shume e larte 11. To what extent, if any, has school management improved because of the BEP program? Deri në çfarë mase, nëse ka, është përmisuar menaxhmenti në shkolla në kuadër të BEP Programit? Not at all a little some much very much Aspak Shumë pak Afërsisht Mjaftueshëm Kënaqshëm 12. In your view, what have been the opportunities for girls and women in the BEP program? Sipas pikëpamjes tuaj, cilat ishin mundesitë per vajza dhe gra në programin BEP? Very poor poor average good very good Shumë I dobët I Dobët mesatare I mirë Shume I mirë A-28 13. The BEP program consists of many elements or components, some of them intentionally related. In your opinion, please rank order the importance or value of each of the 14 following components, ranging from a “1” for the component with the least value or importance to a “14” for that component you deem most important. Please use each number only once, thus covering all 14 listed components. Programi BEP përbëhet nga disa elemente dhe komponenta, disa nga ato qëllimisht të ngjashme. Sipas mendimit tuaj, ju lutem ti renditni secilën sipas rëndësisë apo vlerës nga 14 komponentat në vazhdim, duke renditur nga “1” për kompontën me më së paku vlerë apo rëndësi deri në “14” për komponentën që konsideroni si më të rëndësishmen. Ju lutem të perdorni secilin numër vetëm një herë, duke perfshirë të gjitha 14 komponentat e listuara. ______ developing standards for school management Zhvillimi i standardeve për menaxhmentin e shkollës ______ gender and gender diversity Gjinia dhe shumëllojshmëria gjinore ______ Professional Development Centers Qendrat për Zhvillim Profesional ______ integrating non-Albanian ethnic minorities more fully into the education system Përfshirja e pakicave etnike jo-Shqiptare më shumë në sistemin arsimor ______ training of School Directors Trajnimi i Drejtorëve të shkollave ______ 21st century technology classrooms Klasat e teknologjisë të shekullit 21 ______ reading activities involving students and parents Interpretimi i aktiviteteve duke përfshirë studentet dhe prindërit ______ Student Councils and student clubs Këshillet studentore dhe klubet studentore ______ national standards in assessment for learning Standardet ndërkombëtare në vlerësimin për mësim ______ school-based facilitators in various subjects Ndihmësit për shkolla në landë të ndryshme ______ institutional strengthening and capacity development of the MEST, Municipal Education Departments and other entities Forcimi institucional dhe zhvillimi i kapaciteteve të MASHT, Departamentet e Arsimit Komunal, Qendra për Arsim e Kosovës dhe subjekte të tjera ______ school-based teacher training Trajnimi i mësimdhënësve bazuar në shkolla A-29 ______ school boards and citizen/parent involvement generally in the education system Përfshirja e Bordeve të shkollave dhe qytetarët/prindërit në përgjithësi në sistemin arsimor ______ other component or element (please specify) Ndonjë element apo komponentë tjetër (ju lutem veçojeni) 14. What is your overall view of the Basic Education Project? Cili është mendimi juaj i përgjithshëm për Projektin për Arsim Themelor? 15. Please discuss the roles of technology in the BEP program. Ju lutem shqyrtoni funksionet e teknologjisë në programin BEP . 16. What, in your opinion, are the major strengths and weaknesses of the BEP program? Cilat, sipas mendimit tuaj, janë anët e forta dhe anët e dobëta të BEP programit? 17. What recommendations—general and specific—do you make to improve the BEP program? Cilat janë rekomandimet në përgjithësi dhe në veçanti që ju do të bënit për të përmisuar BEP programin? [Thank you very much for the time, attention and information you have given for this mid-term evaluation of the Basic Education Program.] [Ju faleminderit shumë për kohën, vëmendjen dhe informatat që keni dhënë për këtë vlerësim afat￾mesëm të Programit për Arsim Themelor] A-30 ANNEX D: SCOPE OF WORK A-31 A-32 A-33 A-34 A-35 A-36 A-37 U.S. Agency for International Development 1300 Pennsylvania Ave., NW Washington, D.C. 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov