June 2014 This document was created by Social Impact, Inc. under contract no. AID-OAA-M-13-00011. The views expressed herein do not necessarily reflect those of the US Government. Evaluation Report Midterm Performance Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project EVALUATION REPORT MIDTERM EVALUATION OF USAID/MACEDONIA’S INTERETHNIC INTEGRATION IN EDUCATION PROJECT This document was prepared by Zojka Naskova and Mathias Kjaer of Social Impact, Jelena Mihajlovic of USAID/Serbia, and Mirjana Popovic Valjevac of USAID/Bosnia, with support from Alison Miranda, Daniel Sabet, and Sarah Alvarez of Social Impact. For any questions related to the evaluation, please contact: Social Impact 2300 Clarendon Blvd, Suite 1000 Arlington, VA, 22201, USA Tel: (703) 465-1884 Email: mkjaer@socialimpact.com TABLE OF CONTENTS Acknowledgments ........................................................................................................................ iii Acronyms...................................................................................................................................... iv Geographic Coverage ................................................................................................................... v 1. Executive Summary ................................................................................................................. vi Evaluation Purpose ...................................................................................................................................................... vi Background ................................................................................................................................................................... vi Evaluation Questions and Methods ........................................................................................................................ vii Major Findings and Conclusions .............................................................................................................................. vii Recommendations ........................................................................................................................................................ x 2. Introduction ............................................................................................................................... 1 Country Context .......................................................................................................................................................... 1 Overview of USAID/Macedonia IIEP ........................................................................................................................ 1 3. Purpose and Scope of Evaluation ............................................................................................ 3 Evaluation Purpose ....................................................................................................................................................... 3 Methodology .................................................................................................................................................................. 3 Limitations and Threats to Validity .......................................................................................................................... 4 4. Findings ...................................................................................................................................... 6 Q1: What are the progress and effectiveness of the project to date? ............................................................ 6 Component 1: Community Outreach ................................................................................................................ 6 Component 2: Capacity Building of School Management and Teachers .................................................. 10 Component 3: Demonstration Schools ........................................................................................................... 16 Component 4: Incentives to Schools and Communities .............................................................................. 23 Q2: How well is IIEP communicating with beneficiaries, GoM, other donors, and NGOs? .................... 26 Q3: Does the project approach need to be modified in order to reflect the reality of the current ethnic relations? If so, how? .................................................................................................................................................. 27 Q4: Which selected actions and/or cross-cutting themes and corresponding activities should be further emphasized, modified, or eliminated, and why? .................................................................................................. 28 Q5: What should the implementer do to ensure sustainability? .................................................................... 29 Q6: What existing alternative approaches could lead to better results? ...................................................... 30 5. Overarching Conclusions ....................................................................................................... 33 6. Recommendations .................................................................................................................. 35 Annexes ........................................................................................................................................ 37 Annex A: Evaluation Statement of Work ............................................................................................................. 38 Annex B: Sources Reviewed .................................................................................................................................... 45 Annex C: List of People Consulted ........................................................................................................................ 47 Annex D: Data Collection Schedule ...................................................................................................................... 54 Annex E: Interview Protocols .................................................................................................................................. 55 Annex F: Survey Results from Anonymous Mini-Survey ................................................................................... 65 Annex G: Disclosure of Any Conflicts of Interest ............................................................................................. 72 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project iii ACKNOWLEDGMENTS This evaluation would not have been possible without the support and contribution of many. The team would first like to extend its gratitude to its USAID/Macedonia Contracting Officer’s Representative (COR), Ivica Vasev, and A-COR and Education Specialist Lela Jakovlevska. These two individuals provided valuable inputs at multiple stages of our evaluation effort and the success of this evaluation would not have been possible without their support. The team would also like to thank Acting Mission Director Jennifer Connolly for participating in both our in- and out-briefs and giving the team a better sense of what management purposes this evaluation will serve going forward. Next, the team would like to extend its sincere gratitude to the IIEP implementing staff at MCEC and its partner organizations. These individuals approached the evaluation with a clear interest in an honest and objective account of their activities. Despite an already busy schedule, these individuals repeatedly took time out of their days to meet with the evaluation team, help schedule interviews, track down project documents, and provide answers to any clarifying questions, many times outside of regular work hours. The team would also like to thank all the national- and municipal-level Government of Macedonia officials that made time to meet with us during fieldwork. The team would especially like to recognize Njomza Qerimi and Roman Papadimitrov for their friendly, consistent, and welcomed support to help the team better understand the complexities of the Macedonian educational system and dynamics within the Ministry of Education. The team also wishes to thank and express its appreciation to all the school administrators, teachers, students, and parents who met with us over the course of its interviews. This evaluation would not have been possible without them and the team was consistently greeted with warmth, enthusiasm, and thoughtful responses. The team would like to thank its support and home office staff. Our interpreter and logistics coordinator, Besim Dogani, worked tirelessly to fill these dual roles, arranging interviews in between breaks in interpreting to ensure that the team had a full schedule of interviews during its fieldwork. Our second interpreter, Blerina Zlatku, provided consistent and professional interpretation throughout our interviews and helped the team members understand the nuances of certain terms and expressions. Alison Miranda played a vital quality assurance role and helped manage the entire evaluation process from conception and design through writing and dissemination. Daniel Sabet served as our Senior Technical Advisor and provided consistent, detailed, and insightful support on the team’s evaluation design, sampling, data analysis, and reporting finalization. Last but not least, the team wishes to recognize Sarah Alvarez for providing not only helpful logistical support but also supporting the team’s data analysis and creating many of the graphs that appear in this report. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project iv ACRONYMS BDE Bureau for Development of Education CHRCR Center for Human Rights and Conflict Resolution COP Chief of Party COR Contracting Officer’s Representative EU European Union EUCOM United States European Command FGD Focus group discussion GoM Government of Macedonia IIE Interethnic Integration in Education IIEP Interethnic Integration in Education Project IR Intermediate result M&E Monitoring and evaluation MCEC Macedonian Civic Education Center MIM Macedonian Institute for Media MKD Macedonian denar MoES Ministry of Education and Science NGO Non-governmental organization OSCE Organization for Security and Co-operation in Europe PMP Performance Management Plan PWC Partners’ Working Council SEC State Examination Center SEI State Education Inspectorate SI Social Impact SIE Strategy for Integrated Education SIT School Integration Team SRT School Renovation Team UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government VETC Vocational Education and Training Center Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project v GEOGRAPHIC COVERAGE Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project vi 1. EXECUTIVE SUMMARY EVALUATION PURPOSE In April 2014, the United States Agency for International Development’s Mission in Macedonia (USAID/Macedonia) commissioned Social Impact (SI) to conduct a midterm performance evaluation of its four-year, $5.2 million Interethnic Integration in Education Project (IIEP). Following nearly two and a half years of implementation, the purpose of the evaluation was to provide an external assessment to (1) enhance the effectiveness of the intervention during its remaining implementation and (2) inform future USAID investments in interethnic integration. The primary intended users of this evaluation are IIEP implementing staff and relevant USAID/Macedonia education and program office staff. Secondary users include other donors and implementers working on interethnic education issues in Macedonia or the region, as well as project beneficiaries. BACKGROUND Ethnic Tension and Education in Macedonia Ethnic tensions between the Macedonian majority and the Albanian minority have posed a major obstacle to the peace and stability of Macedonia. Relations between the two groups were exacerbated by the 1999 Kosovo War, which eventually contributed to an Albanian insurgency in Macedonia in 2001. Insurgents voluntarily disarmed as part of the Ohrid Framework Agreement, and the agreement led to greater rights for ethnic Albanians and other minority ethnic groups. Nonetheless, tensions remain and are pronounced in the educational system. Schools in Macedonia have traditionally been divided along linguistic lines, eliminating the opportunity for children from different ethnic and language groups to interact and develop norms of trust and reciprocity. Instead, separation and division threatens to reduce accurate information flow across ethnic lines and to perpetuate stereotypes about “other” groups. Facing internal and external pressure, the Government of Macedonia (GoM) adopted a Strategy toward Integrated Education (SIE) in 2010. However, while adoption of the Strategy is a clear step forward, implementation faces numerous obstacles. Overview of USAID/Macedonia IIEP To help overcome these obstacles, USAID/Macedonia contracted the Macedonian Civic Education Center (MCEC) and its eight local partners to implement the four-year, $5,200,000 project to build broad public understanding of the benefits of an integrated educational system in Macedonia. It works with a variety of actors to create “the political, social, and economic environment needed for Macedonia to achieve sustained interethnic integration in schools, in other educational institutions and eventually all of society.”1 The project is implemented through four interrelated components that target all primary and secondary schools across Macedonia and build the capacity of the Ministry of Education and Science (MoES), the Bureau for Development of Education (BDE), the State Education Inspectorate (SEI), and the Vocational Education and Training Center (VETC) to support interethnic integration in education (IIE) activities. 1 IIEP Website, “About the Project.” http://pmio.mk/about-the-project/ Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project vii EVALUATION QUESTIONS AND METHODS The evaluation provides answers to two types of questions: (1) determining if the project has met its objectives and (2) recommendation questions aimed at improving the current project implementation and informing future USAID efforts in interethnic educational integration. Process and effectiveness questions: 1. What are the progress and the effectiveness of the project’s interventions to date on the project objectives? 2. How well is the project communicating with beneficiaries, the Government of Macedonia, other donors, and NGOs in the area of ethnic integration? Recommendation questions: 3. Does the project approach need to be modified in order to reflect the reality of current ethnic relations? If so, how? 4. Which selected actions and/or cross-cutting themes and corresponding activities should be further emphasized, modified, or eliminated, and why? 5. What should the implementer do to ensure sustainability? 6. What existing alternative approaches could lead to better results? The evaluation was carried out by a four-member joint SI-USAID team between April and June 2014. The Team Leader and Evaluation Specialist from SI were joined by two regional USAID colleagues from USAID/Serbia and USAID/Bosnia. The team used a simple ex-post performance evaluation design relying on a qualitatively dominant but still mixed-methods approach involving (1) a desk review; (2) semi￾structured key informant interviews; and (3) site visits to schools involving semi-structured key informant interviews, observation, focus groups, and mini-surveys. Fieldwork took place in 11 cities throughout Macedonia, and the team visited 16 schools and interviewed 387 respondents in May 2014. The evaluation was limited by the time available in country and faced three internal threats to validity (history, selection, and testing biases) and one external threat to validity (response bias). MAJOR FINDINGS AND CONCLUSIONS Overall, IIEP is an ambitious, far-reaching project that seeks to identify and rectify cultural and ethnic stereotypes hardened by conflict and generations of mistrust. It operates with a relatively modest budget to design, implement, and monitor over 87 discrete sets of activities directed at a diverse group of stakeholders at the national, municipal, and local levels and geographically spread throughout the country. Considering the complexity of the issue that the project aims to address and the scale at which the project is implemented, the project is performing remarkably well. Progress and Effectiveness of Interventions Component 1: Community Outreach. This component aims to raise awareness and promote IIE within the education community and among the general public. IIEP was carefully designed to engage, secure, and maintain the early buy-in of key GoM representatives, and has been successful in raising the awareness of the need and benefit of IIE activities with national-level education stakeholders. While the project has succeeded in conveying this information to key audiences within schools at the local level, it has had more limited success in maintaining active communication with municipal officials and parents. It has undertaken a number of activities to promote and shape media reporting of its activities and has seen an uptake in the number of positive media stories covering IIEP Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project viii activities. The project has had relatively more success reaching local rather than national media outlets and even encouraging local students, teachers, and administrators to create and share their own positive IIE stories. The project is having more difficulty in securing the interest of national-level media outlets and preventing media sensationalism, which is the norm that the project is seeking to change. Component 2: Capacity Building of School Management and Teachers. This component aims to provide structured capacity building for education institutions and professionals in support of IIE and equip teachers, school managers, school board members, parents, and municipal education officials with the skills needed to effectively carry out activities. Given the broad scope of activities, the project’s performance under this component has been mixed. The project has undertaken extensive efforts to build the initial capacity of select stakeholders throughout Macedonia. The scale of this undertaking—mobilizing the whole education system and to train teaches in all primary and secondary schools across the country in a relatively limited period of time—is impressive and itself represents a large success. Educators are taking interethnic collaboration to a higher level and beginning to develop a system for further integration. Initial success has been achieved in institutionalizing integration issues in the education system, though this process has been slow. Educators need further encouragement and support to effectively implement integration activities and overcome systemic and financial obstacles. Component 3: Demonstration Schools. This component supports six model schools to provide examples of best practices and lessons learned that will contribute to a deeper understanding of various approaches and challenges to ethnic integration in schools. IIEP is progressing well in terms of building a deeper understanding of the issues and challenges to integration within demonstration schools; however, additional work is needed to disseminate this knowledge externally. Generally, teachers were satisfied with the training received, and students report an increase in communication between groups. The project has taken a number of steps to document its learning, including plans to produce a best practices manual; however, more could be done to “demonstrate the demonstration schools.” As in the Component 2 schools, the project is finding it challenging to engage parents on a sustained basis, and activities largely involve students and teachers already supportive of integration activities. Component 4: Incentives to Schools and Communities. In collaboration with the United States European Command (EUCOM), this component offers funds for school refurbishments as an incentive for schools to participate in ethnic integration activities. The component is functioning well as an incentive for schools to consider designing and implementing their own IIE activities, particularly in encouraging the involvement of school management, School Integration Teams (SITs), and municipal representatives. However, internal United States Government (USG) pressure to begin allocating funds caused the project to shift its model of providing assistance as a reward for implemented activities to an incentive to implement activities during the first year of implementation. This was remedied following the pilot year and schools applying to receive renovations are generally proposing IIE-related activities; however, more work could be done to ensure that the proposed activities are integrated into Annual School Plans and they are genuinely implemented, and that parents, teachers, students, and the community at large are better informed that this is an incentive/award for IIEP. Communication with Beneficiaries, GoM, Other Donors, and NGOs IIEP has demonstrated a high degree of success in maintaining regular communication with key beneficiaries and national-level GoM representatives. While the project has undertaken efforts to engage and solicit support from municipal-level representatives, parents, and local businesses and chambers of commerce, these efforts have been less successful than those targeted at other audiences. These groups Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project ix are crucial for sustaining project activities given their ability to provide critical financial and non-financial support and encouragement. Relevance to Context and Current Ethnic Realities IIEP’s overarching approach is relevant to the current ethnic realities, and there is wide agreement that a holistic, national-level approach was appropriate for tackling an issue as complex as interethnic integration. The project is targeting a variety of stakeholders across multiple levels; however, the project may be further strengthened if additional effort is made to tailor activities to more localized demographics and regional interethnic dynamics instead of providing a standard model of school activities across all municipalities in Macedonia. In other words, the project would benefit if its training and activities were more sensitive to different demographics that exist in different parts of Macedonia, and the type of activities should be tailored to respect these differences and suit the local demographics. Although the project recognizes that politics plays a significant and usually destabilizing role within schools in Macedonia, it is careful to engage political parties directly and avoids reacting to individual, politically sensitive events. The project prefers to remain apolitical and instead tries to take a more proactive, holistic approach of preventing interethnic conflicts. Cross-Cutting Themes The project is demonstrating more limited success in reaching municipal-level government officials compared to its relatively high level of success reaching national-level officials. IIEP has conducted a number of awareness-raising activities at the municipal level, and it is not clear that holding additional informational events will result in municipal officials actually attending and participating in these events. Respondents also overwhelmingly conveyed their frustrations and concerns for the future sustainability of activities given the limited financial resources available to support them. Interviews with the MoES IIE Working Group members also revealed a frustration that the sequencing of project phases had been too short and prevented the project from applying lessons learned during the previous phases. On a higher level, the sentiment that the effective promotion of integration involves “a process, not a project” was expressed throughout the team’s interviews. The majority of respondents, including IIEP staff themselves, recognize that parents could be much more involved, although the project does appear to be more successful in reaching parents in its demonstration schools than in non-demonstration schools. The team noted some unrealized potential for more explicit interaction across the four IIEP components. While the vertical linkages between activities in each component are clear, the horizontal linkages between each component are much more difficult to determine. Ensuring Sustainability In the design of the project, MCEC has seriously considered sustainability and strives to build capacities and systems to this end. IIEP has built local capacity by training and engaging local educators in practice; worked with institutions such as the MoES, BDE, VETC, SEI, State Examination Center (SEC), and municipalities to develop institutional frameworks and policies in support of IIE; and developed comprehensive manuals and handbooks for training and IIE activity implementation. The Working Group for IIE has been established and two members of MoES staff have been engaged to act as point persons for IIE. Local partner NGOs have built IIE expertise and capacity to continue promoting integration in their communities. However, the project’s cascade training model presents some sustainability challenges. Project activities are often conducted in addition to other existing formal curricular and extracurricular activities, and there may be opportunity for greater efficiencies both at the project activity and the curriculum levels. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project x Alternative Approaches There have been a number of programs addressing interethnic collaboration in education in the country, each taking a slightly different approach. There was a strong consensus in interview responses that the IIEP is applying a holistic and inclusive approach that is widely accepted and appreciated by respondents. RECOMMENDATIONS Overall, IIEP is a well-designed, high-functioning project that is not in need of any major changes in design or implementation. That said, IIEP should consider the list of prioritized recommendations to strengthen its performance during the final years of implementation. 1. Continue to work to institutionalize IIE activities within the formal State education institutions. IIEP has rightly recognized that education stakeholders ultimately defer to guidance outlined in the national curriculum and criteria set by SEI. To support the implementation of this recommendation the project could, for example, (1) work with BDE to look for opportunities to “integrate” IIE activities into existing curricular and extracurricular programs; (2) support SEI with customized trainings explaining the importance and meaning of the newly developed IIE indicators; (3) continue to advise and support MoES to better understand and prioritize IIE issues, including working with the new Minister of Education; and (4) continue to work with the IIE Working Group to formally integrate its policy development activities into State structure and institutions. 2. Strategize on ways to identify and secure additional funding for IIE activities. While the project intentionally limited the amount of financial support it would provide to schools, the issues and constraints associated with limited funding cannot be ignored and present a real challenge to its future sustainability. Recognizing that the project operates on an already tight budget, IIEP could consider the following: (1) encouraging and supporting the MoES to re-establish and coordinate donor coordination meetings to discuss opportunities to support IIE activities. The project should promote its SITs as a pre-established and organized entry point to coordinate donor activities within schools; (2) providing training and mentoring to school managers, teachers, students, and parents on a range of possible fundraising activities, highlighting strategies that have proven effective in the past; (3) encouraging municipal officials and local businesses to participate and support IIE activities in their communities; (4) advocating for a MoES grant program and/or municipal budget allocations to support IIE school activities; (5) establishing conditional matching goals, possibly financed through Component 4, where IIEP would provide matching funds to schools that fundraise to certain targets; and/or (6) introducing a nationwide competition where IIEP participant schools present their activities and receive awards for best implementation or most creative ideas. 3. Continue to provide professional development opportunities and professional incentives for education institution staff and teachers. With the project finalizing its basic trainings across all schools, the project is at a good stage to focus on (1) providing increased and improved mentoring and professional development support for educators (i.e. teachers, principals, MoES staff) to strengthen their technical skills and capacity to meet expectations; (2) continued professional development opportunities for Master Trainers (already planned as Advanced Training); (3) exploring further support for education inspectors, including efforts to familiarize them with the new IE indicators and their optimal application in practice; (4) involving more teacher training institutions and other higher education institutions to learn from the project’s experiences, support teachers (through student internships, research assignments, teaching practice) and take measures to address teacher professional development issues in the short and long term; (5) working with BDE to develop plans for continued staff and teacher professional development and support; and (6) Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project xi finalizing the process of certifying IIE activities as a key incentive for teachers and working to incorporate them into reformed mechanisms for credentialing, remuneration, and professional growth. 4. Redouble efforts and devise new ways of reaching and sustaining the engagement of municipal officials in activities at the local level. The project has demonstrated considerable success in reaching key GoM officials at the national level, but it is having difficulty replicating that success at the municipal level. Given that municipal officials hold a relatively high level of influence within schools, helping to appoint school principals and determine school budgets, they are key stakeholders who can ensure the project’s future sustainability. In order to implement this recommendation, the project could consider (1) inviting municipal representatives to participate in SITs, perhaps even requiring such participation in written agreements with municipalities; (2) continuing to invite municipal representatives to dissemination events highlighting early successes; and (3) considering continuing to provide and even strengthen specialized training and follow-up mentoring for how municipal officials could better support IIEP within schools across their municipality. In order to encourage municipal officials to actively participate, IIEP could consider channeling funds through municipal budgets, which would help build municipal awareness of education issues as well as their capacities in this domain. For example, all of the monitoring and evaluation (M&E) data collection activities expected of teachers in the project could be handed over to the municipal staff. At the least, they should be included in the data collection and analysis for the purpose of future education planning in the municipality. 5. Strengthen communication and awareness-raising about early project successes, and the concepts, practice and benefits of IIE. The project demonstrated its earlier ability to communicate and raise awareness of the purpose of its activities. The project should now take the next steps and start to communicate some of its early results and lessons learned. Possible ways to do so include (1) holding additional public events to demonstrate early successes in the project’s demonstration and partner schools; (2) encouraging additional representation on SITs and municipal IIE Councils to include students, parents, municipality representatives, and local chambers of commerce; (3) organizing larger, non–school-specific events such as IIE conferences; and (4) looking for ways to more actively use information and communications technology and web-based collaborative learning tools to encourage more partner school interaction and diffusion of best practices. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 1 2. INTRODUCTION COUNTRY CONTEXT Ethnic Tension and Education in Macedonia Ethnic tensions between the Macedonian majority and the Albanian minority have posed a major obstacle to the peace and stability of Macedonia. Relations between the two groups were exacerbated by the 1999 Kosovo War, which eventually contributed to an Albanian insurgency in Macedonia in 2001. Insurgents voluntarily disarmed as part of the Ohrid Framework Agreement, and the agreement led to greater rights for ethnic Albanians and other minority ethnic groups, including the recognition of minority languages spoken by at least 20 percent of the regional population as official languages of the country. Nonetheless, tensions among ethnic groups remain in Macedonian society and politics. The educational system of a country can either exacerbate or ameliorate such ethnic tensions.2 In the Macedonian educational system, schools have traditionally been divided along linguistic lines, eliminating an opportunity for children from different ethnic and language groups to interact and develop norms of trust and reciprocity. Instead, separation and division threatens to reduce accurate information flow across ethnic lines and to perpetuate stereotypes about “other” groups. Despite the potential risk of separate schooling, there has only been limited political will to actively push for greater integration. For example, a 2009 United Nations Children’s Fund (UNICEF) report concluded that while the Government of Macedonia (GoM) had taken steps to ban discrimination, it had done little to promote integration.3 The report cites textbooks that did not focus on the country’s multiethnic history as one of many examples. Facing internal and external pressure, the GoM adopted in 2010 a Strategy toward Integrated Education (SIE), which was developed in partnership with the Organization for Security and Cooperation in Europe (OSCE). The strategy entails five elements, including (1) joint student activities, (2) mutual learning of other languages, (3) curricular and textbook revisions, (4) training for teachers on integrated instruction, and (5) management changes reflective of a decentralized multiethnic society. However, while adoption of the Strategy is a clear step forward, implementation faces numerous obstacles. For example, writing in 2013, Barbieri et al. note that no implementation, action, or monitoring plan had yet been laid out, despite the fact that the strategy had limited time remaining.4 OVERVIEW OF USAID/MACEDONIA IIEP To help overcome these obstacles, USAID/Macedonia contracted the Macedonian Civic Education Center (MCEC) to implement the four-year, USD $5,200,000 Interethnic Integration in Education Project (IIEP). IIEP is a wide-ranging project that seeks to build broad public understanding of the benefits of an integrated educational system in Macedonia. It works with a variety of actors to create “the political, social, and economic environment needed for Macedonia to achieve sustained interethnic integration in schools, in other educational institutions, and eventually all of society.”5 Its activities target all primary and secondary schools across Macedonia and actively aim to build the capacity of the Ministry of Education and Science (MoES), the Bureau for Development of Education (BDE), the State Education Inspectorate (SEI), and the 2 Edward Miguel. 2004. “Tribe or nation?” World Politics. Vol. 56: 327–362. 3 Violeta Petroska-Beska, Mirjana Najcevska, Nikolina Kenig, Safet Ballazhi, and Ana Tomovska. 2009. Multiculturalism and Interethnic Relations in Education. Skopje: UNICEF Country Office. 4 Sara Barbieri, Roska Vrgova, Jovan Bliznakovski. 2013. “Overcoming Ethnic-Based Segregation: How to Integrate Public Schools in Macedonia and Bosnia Herzegovina”. Skopje: Institute for Democracy. 5 IIEP Website, “About the Project.” http://pmio.mk/about-the-project/ Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 2 Vocational Education and Training Center (VETC) to successfully integrate Macedonia’s schools. Specifically, the project is implemented through four “mutually-supporting and interrelated components:”6 1. Community Outreach: Raise awareness and promote interethnic integration within the education community and among the general public. 2. Capacity Building of School Management and Teachers: Provide teachers, school managers, and school boards with the knowledge, attitudes, and skills needed to work in a multiethnic environment, help prevent divisions, and create conditions that promote ethnic integration. 3. Demonstration Schools: Support six model schools to provide examples of best practices and lessons learned that will contribute to a deeper understanding of various approaches and challenges to ethnic integration in schools. 4. Incentives to Schools and Communities: Offer funds for school refurbishments as an incentive for schools to participate in ethnic integration activities. With the expressed aim of involving “relevant expertise and collaborative approach in implementing project activities,”7 MCEC works with eight local non-governmental organizations (NGOs) to implement its activities: Center for Human Rights and Conflict Resolution (CHRCR), Macedonian Institute for Media (MIM), Loja, Sumnal, Youth Education Forum, PAC Multimedia, Step by Step, and My Career. In its original proposal, MCEC explained that this constellation of organizations allowed for an “integrated mix of strategies, approaches, and activities,” as well as more localized knowledge in various communities throughout Macedonia. 6 IIEP Program Description. p. 1. 7 Ibid. “Project Partners.” Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 3 3. PURPOSE AND SCOPE OF EVALUATION EVALUATION PURPOSE USAID/Macedonia commissioned Social Impact (SI) to conduct a midterm performance evaluation of IIEP to (1) provide USAID with an external assessment of the project, which will be used to enhance the effectiveness of the existing intervention through the end of the life of the project, and (2) inform future USAID investments in the area of interethnic integration. SI’s performance evaluation sought to provide answers to two types of questions: (1) evaluation questions aimed at determining if the project has met its objectives and (2) recommendation questions aimed at improving the current project implementation and informing future USAID efforts in interethnic educational integration. More specifically, these questions were: Evaluation questions: 1. What are the progress and the effectiveness of the project’s interventions to date on the project objectives? 2. How well is the project communicating with beneficiaries, the Government of Macedonia, other donors, and NGOs in the area of ethnic integration? Recommendation questions: 3. Does the project approach need to be modified in order to reflect the reality of current ethnic relations? If so, how? 4. Which selected actions and/or cross-cutting themes and corresponding activities should be further emphasized, modified, or eliminated, and why? 5. What should the implementer do to ensure sustainability? 6. What existing alternative approaches could lead to better results? METHODOLOGY The evaluation was carried out by a four-member joint SI-USAID team between April-June 2014. The Team Leader and Evaluation Specialist were provided by SI and were joined by two regional USAID colleagues from the USAID/Bosnia and USAID/Serbia missions. The team used a simple ex-post performance evaluation design relying on a qualitatively dominant but still mixed-methods approach involving (1) a desk review of available primary and secondary documents; (2) semi-structured key informant interviews; and (3) site visits to schools involving semi-structured key informant interviews, observation, focus groups, and mini-surveys. Fieldwork took place in 11 cities throughout Macedonia (Skopje, Tetovo, Gostivar, Kicevo, Struga, Ohrid, Bitola, Strumica, Radovis, Veles, and Kumanovo) from May 5–20, 2014. The team visited 16 schools and interviewed 387 respondents. Please see Annex C: People Consulted and Annex D: Fieldwork Schedule for additional detail on the places visited and people consulted for this evaluation. The evaluation team began its fieldwork and site visits together in Skopje. This allowed the team an opportunity to pilot the data collection instruments and ensure a unified approach to the interview protocols. The team then divided into two sub-teams to optimize their time in country. Each sub-team conducted fieldwork in various parts of Macedonia purposefully selected to ensure that both had exposure to communities with varying levels of ethnic and socioeconomic diversity. The teams generally conducted structured key informant and small group interviews with GoM (both national and municipal) and school Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 4 management representatives and semi-structured focus group discussions (FGDs) with teachers, students, parents, and journalists. The teams distributed an anonymous mini-survey prior to its FGDs with students to (1) capture information from students who may not be comfortable speaking in larger groups; (2) supplement the qualitative information from the FGD with quantitative data from the closed-ended mini￾survey questions; and (3) offer a starting point for discussion. The teams also used direct observation during site visits, observing school iconography and the interaction of students, teachers, and/or parents at any planned extracurricular activities. The teams also observed several trainings and Working Group meetings. The evaluation team used two approaches for its data analysis: (1) a rolling analysis conducted during fieldwork and (2) a more in-depth parallel analysis conducted after fieldwork across the complete datasets. Following each day of fieldwork, the team met to discuss emerging trends and new findings that had surfaced that day. Discussions focused on the original evaluation questions to ensure that conversations remained targeted and focused. Prior to its final day in country and out-brief with MCEC and USAID/Macedonia representatives, the team held a full-day joint analysis session to systematically identify preliminary findings, conclusions, and recommendations for each evaluation question. Upon returning from the field and prior to initial report drafting, the team conducted a more in-depth parallel analysis to examine evidence from the document review, key informant interviews, focus group discussions, direct observation, and mini-survey survey data. This allowed for a “methods triangulation,” in which the team analyzed data related to an evaluation question (and relevant indicators) using different methods in parallel and then across data collection methods and research sites. Outcomes observed were analyzed for attribution to, or at least contribution by, IIEP. The team also disaggregated its data and conducted the same analysis for data collected from different sources—e.g., women, ethnic group, government organizations, implementing partners. Through these methods, the team sought to increase the reliability and validity of the findings and conclusions presented below. LIMITATIONS AND THREATS TO VALIDITY As with any short-term performance evaluation, the team was restricted by its limited time in country. The team had a total of 14 working days in country to meet with as many stakeholders and visit as many schools as possible. While the team is generally satisfied with the substantial number of interviews and school visits it was able to conduct—the team interviewed 387 stakeholders across eight stakeholder groups and visited 16 schools—it was able to interview only a few school board members and no political party representatives. Given the extent to which politics influences school life in Macedonia, this was a significant limitation. The evaluation was also limited by the timing of the evaluation. IIEP is working to change strongly held beliefs, attitudes, and perceptions calcified by a history of conflict and mistrust. These are longer-term outcomes that would not be immediately visible following two and a half years of implementation. Thus, while the team was able to analyze the extent to which the project is progressing in meeting its objectives, the team is limited in its ability to judge the extent to which changes in attitudes and beliefs have truly taken hold and the extent to which such changes can be attributed directly to IIEP. National elections were scheduled after evaluation planning began, and though the fieldwork dates were postponed, the evaluation took place just two weeks following national elections, with the government still being negotiated and formed during the team’s fieldwork. There were several important internal threats to validity. While this evaluation did benefit from baseline data collected as part of the IIEP, it is not an impact evaluation and limited in the extent to which it can measure changes over time or attribute causality to the IIEP. In addition, there was a likely selection bias in the composition of student FGDs. The team did not request access to lists of student participants at each school in advance of fieldwork and therefore relied on teachers and school managers to help select students FGD respondents. It is likely that this resulted in an over-representation of students likely to give favorable accounts of the projects, as students selected may have been those most active in or most Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 5 positive about the activities. There was also a testing threat, as some respondents across all groups interviewed had previously participated in the project’s baseline survey or regular requests for monitoring data. (It is not clear how common this was.) Respondents may have remembered answers they had given previously to similar questions and conditioned their responses accordingly. Lastly, in terms of external threats to validity, it is likely that the study experienced a response bias, whereby respondents felt a natural tendency to provide answers that they believed the interviewer wanted to hear. This is likely to have been especially pronounced at the individual school level, with teachers and school administrators being worried about potentially losing project activities if they provided “wrong” answers. Younger students might have also been susceptible to such bias during the student FGDs. While the factors presented above do raise concerns for the internal and external validity of the evaluation findings, the evaluation benefited from strong project monitoring data, helpful secondary studies, and multiple data-collection activities. These data sources provided an opportunity for the team to triangulate its findings and limit the influence of the threats to validity mentioned above. For its immediate purposes as a mid-term performance evaluation, the team remains confident in its data regarding the implementation and progress towards the objectives described below. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 6 4. FINDINGS Q1: WHAT ARE THE PROGRESS AND EFFECTIVENESS OF THE PROJECT TO DATE? In the following sections we review each component in turn and consider the extent to which the purpose, intermediate results (IRs), and sub-intermediate results (sub-IRs) were achieved. Each section begins with a box that details the component activities. In addition to the data and analysis by the evaluation team, indicators from MCEC’s Performance Management Plan (PMP) are also presented and discussed. Component 1: Community Outreach Overview Purpose: To raise awareness and promote IIE within the education community and among the general public. This will be achieved through three IRs:  Result 1: Collaboration between key education institutions in support of implementation of IIE activities in primary and secondary school established.  Result 2: Education community on local and national level better understands the need for integrated education and its benefits for all students, teachers, and parents.  Result 3: Journalists understand the need for integrated education and its benefits for the society as a whole. Target Audience: The general public. Sub-IR–Specific Findings Sub-IR 1.1: Collaboration between key education institutions in support of implementation of interethnic integrated education activities in primary and secondary schools is established. INDICATOR 2013 TARGET 2013 ACTUAL Number of key education institutions that sign the operational agreement 4 education institutions 4 education institutions Given IIEP’s ambitious aim to work in all primary and secondary schools across Macedonia, a critical foundation to IIEP’s success was securing the buy-in of Macedonia’s four main education institutions. IIEP has achieved this important target and signed operational agreements with the MoES, BDE, VETC and SEI as planned. These agreements, reviewed by the team and discussed during interviews, provide a critical basis for outlining the extent of collaboration and identifying roles and responsibilities. Each education institution involved has a clearly defined role in the process: MoES is involved in the selection of Master Trainers; SEI prepares qualitative indicators for schools; and BDE reviews the schools’ annual curricula. Each institution also provides advisors to serve as Master Trainers to help train and mentor teachers involved in Component 2 on how to implement IIE activities. The operational agreements represent an important milestone, which was preceded by substantial effort on the part of both IIEP and USAID/Macedonia to secure key government buy-in during the early design Component 1 Activities  Outreach at national level, including launch events and trainings and workshops for journalists, SITs, students.  Outreach at local community level, including regular meetings, roundtables, school presentations, and debates and other competitions.  Cross-cutting activities, including exchange visits between schools; public presentations by successful schools; maintaining a website to enable collaboration and sharing; establishing a Partners’ Working Council (PWC) for partners to share news, make important decisions, and learn through joint capacity building. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 7 phases of the project. The team heard accounts from both USAID/Macedonia and IIEP of several consultative meetings held with the Prime Minister, Deputy Prime Minister, and Minister for Education to convince them of the need for integration activities. These meetings helped overcome some initial skepticism that the IIEP project was coming too soon after the 2001 conflict and that the country was simply not yet ready for these kinds of IIE activities. Both the project and USAID/Macedonia continue to nurture this high-level support by sharing draft annual work plans and distributing monthly briefing updates to a listserv of key government representatives. IIEP also benefits from the placement of two USAID advisors within the MoES, which is a separate activity from the project. GoM respondents interviewed acknowledged the critical role that these two advisors have played in raising the level of awareness within the Ministry of Education for the need and benefit of IIE activities. As an indication of their satisfaction and recognized utility of these two staff and to ensure sustainability of their efforts, MoES recently tasked two junior staff to be trained by the advisors prior to their scheduled departure in December 2014. While this is a positive step, interview responses indicated that these two new advisors are still junior and are not yet capable of replacing the current advisors. Interviewees also mentioned that supporting IIE activities will constitute only a relatively small portion of their portfolios and some questioned whether they would really have the capacity to provide the level of support the current advisors are providing (inferring that they would simply be too busy with other tasks and that their IIE responsibilities would fall to the side). Interviews with these USAID advisors confirm that they regard the IIEP as a close partner and resource in helping them maintain IIE as a key governmental priority. Leveraging the relationships fostered within the MoES, IIEP staff members now regularly participate in periodic meetings of the Working Group on interethnic integration at the MoES. The team observed two Working Group meetings wherein members discussed definitions and standards for IIE based on the work of IIEP, as well as how to adapt what were regarded as useful and informative M&E tools developed by the project. The Working Group is currently in the process of finalizing a PMP for a Country Strategic Document for Interethnic Integration in Education and will utilize some of the national level indicators used by IIEP. In this way, the project has moved beyond merely sharing written documents to actively supporting the interaction and collective problem-solving of key national-level education institutions on the topic of IIE. Sub-IR 1.2: The education community at the local and national level better understands the need for integrated education and its benefits for all students, teachers, and parents. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of responsiveness from key education institutions that are informed about IIE goals and activities 50% of the staff 52% of the staff Percentage of municipalities whose officials are informed about the major IEE goals and activities 70% of all municipalities 52% of all municipalities Percentage of municipalities that organize promotional IIE event for their citizens 50% of all municipalities 19% of all municipalities The project launched a significant number of awareness-raising events early in its implementation and continues to hold public events featuring activities at demonstration and partner schools. As one example, the project recently hosted a nation-wide debate tournament, which was observed by the team, whereby students from across Macedonia debated the pros and cons of integration. The project has also developed and finalized a communication strategy that consolidates key messages for public dissemination and provides a series of tools to better engage media outlets and encourage them to publish stories of IIEP activities on a variety of different platforms. The team probed on the success of the project’s outreach efforts, and a majority of respondents within each respondent group reported having seen public advertisements or news stories or having attended launch and public dissemination events. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 8 The project also maintains a comprehensive and regularly updated website that hosts a variety of information, including background materials, monthly progress and monitoring reports, photos and videos of joint-student activities, applications for renovation, calendar of events, and various other types of information. School directors, SIT members, and representatives of the MoES and OSCE praised the project’s website and refer to it as a useful resource. Although the IIEP has engaged the education community in IIEP activities, the majority of interview respondents also mentioned that there had been multiple past projects that have helped raise awareness of the need and benefits of integration of education (e.g. Italian Peace Project, Nansen Dialogue Center, various UN-supported projects, and past USAID projects). Familiarity with the concept of IIE was described by interview respondents as something natural in a multiethnic community, perhaps due in part to these previous projects. However, schools did not report and the team did not observe that educational methodologies, tools, or practices from past IIE projects remained in schools. Presumed familiarity with the concept of IIE among the project’s key stakeholders may be a factor that affects future collaboration towards achieving project results. The team noticed that the project does not have a working definition for “integration,” and at least one of the project sub-contractor interviewees spoke about the need to more clearly define “integration” and to increase this understanding. Across most respondent groups, various related but meaningfully different terms were used interchangeably: “multiethnic” versus “integrated” activities, and “assimilation” 8 versus “integration.” As noted by the project’s own monitoring data, IIEP is having difficulty maintaining the sustained engagement of key audiences within the local municipal governments. During its first year, the project failed to reach its targets for (1) the percentage of municipal officials informed about the project, defined as having at least two officials participate in IIEP promotional activities and (2) the percentage of municipalities that organize their own promotional events for their citizens. According to interviewees across all respondent groups, local municipalities exercise a significant level of influence within their respective schools, with mayors often directly appointing school principals based on their party allegiance. However, it is important to note that the project has undertaken a sizeable effort, and has been successful overall, in informing municipal education advisors about IIEP activities. In fact, prior to working in any school in the country, the project secured municipal consent, without which the activities would not have been possible. Sub-IR 1.3: Journalists understand the need for integrated education and its benefits for the society as a whole. INDICATOR 2013 TARGET 2013 ACTUAL Number of articles and/or broadcasts prepared by journalists that constructively present the IIE activities 20 204 The project has far exceeded its targeted level of media coverage in large part due to its targeted and proactive efforts. Media outreach began at the start of the project, with the component coordinator identifying key contacts within all media outlets to be engaged in IIE activities and trainings. IIEP has also developed a communication strategy for engaging media, holds monthly meetings with media editors, and prepares press releases before each event to help guide media coverage. The team also noted that the project is doing a good job of reaching a variety of audience-specific outlets. For example, the project targets MTV to better reach students, as well as Turkish, Albanian, and Macedonian-language media 8 The team recognizes that this is a sensitive term in the Macedonian context. The reader should note that this term was used in at least four interviews and was generally used when the respondents were describing earlier efforts to implement IIE activities (before the current IIEP project). “Media used to only cover when there were problems; now the project is working with them to share continued positive message.” —MCEC Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 9 outlets. The project is currently in negotiations to create and produce a new documentary series for national TV to be called “7x7.” IIEP is also demonstrating early success in engaging journalists to train teachers and students on how to generate stories about their integration activities. The team observed a media training in Ohrid and found the training to be interactive and well organized. Students and teachers from four schools were trained by journalists and project staff on various topics, including how to create a positive news story and post it to the Internet. The project and MIM staff was available to answer questions from participants. While the training is recognized to be of high quality, the team noted that IIEP does not provide much formal follow-up on these trainings due to limited funding but does review the products developed by trained students and teachers. Representatives from both the project and other education institutions noted that media coverage of integrated education has increased since the project started, with some linking this directly to the work of IIEP. However, the team found that opinions differ on the level of success between national and local media coverage. The majority of respondents feel that the project has been much more successful in engaging local media. Respondents at the local level generally contend that national media interest is mostly focused on politics and that integrated education is not a priority issue. For these respondents, this lack of prioritization prevents the general public from being familiar with IIEP. This view was not shared by all respondents, however, and those at the central level (education institutions, OSCE, MCEC, and other partners) generally contend that the project’s events are visible on national TV stations. The project staff feels that they are beginning to see a transformation in receptiveness of media: in some instances, local media had identified and published stories without probing from the project. Nonetheless, challenges remain and there is general agreement that the media is still divisive and one of the biggest obstacles to further integration. Conclusions  Overall, there is evidence that the project has succeeded in raising awareness of the need and benefit of IIE activities. At the national level, the project has benefited from the presence of two USAID-sponsored advisors within the MoES, but the project is also directly engaging with high-level GoM education representatives who regularly contribute during Working Group meetings.  While the project also appears to have successfully conveyed the needs and benefits of IIE activities at the local level with teachers, students, and to a lesser extent parents, the project has found less success in engaging local municipal officials. These officials represent a key stakeholder group for the project’s future sustainability, as local municipal governments exercise a significant level of influence within Macedonian schools.  The project has undertaken significant efforts to encourage and train journalists to produce positive news stories regarding IIE activities. While the project is able to document an increase in the volume of positive news stories, respondents widely agreed that there is still an overwhelming tendency in the media, particularly the national media, to emphasize ethnic differences and conflict rather than commonalities and successful integration. “We have regular coverage locally. Local papers, local TV, we also have a Facebook page, we realized young people read social media more, and all events are covered by the local portal. But national media does not cover us that much—they are highly politicized and that reflects in education.” —School in Kumanovo Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 10 Component 2: Capacity Building of School Management and Teachers Overview Purpose: To provide structured capacity building for education institutions and professionals in support of IIE and equip teachers, school managers, school board members, parents, and municipal education officials with the skills needed to effectively carry out activities. This will be achieved through three IRs:  Result 1: Education stakeholders’ competencies to participate actively in IIE activities are strengthened.  Result 2: Education stakeholders’ skills to support and/or implement interethnic integrated activities on school/community level are improved.  Result 3: Number and diversity of multiethnic activities within the schools and communities are increased. Target Audience: Under this component the project team works with all education institutions: MoES, BDE, VETC, SEI, municipal education officials, and teacher training institutions (pedagogical faculties) and aims to reach all primary and secondary school teachers, school managers, school board members, parents and, indirectly, students. This is the largest component of the IIEP, taking up 40 percent of the budget and reaching out to all 423 schools in the country in three phases of implementation. Key implementers are MCEC and partners, the CHRCR, Loja, and Sumnal. Sub-IR–Specific Findings Sub-IR 2.1: Education stakeholders’ competencies to participate actively in IIE activities are strengthened. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of advisors and inspectors trained to support and introduce the IIE in-school activities 70% of all advisors and inspectors 85% of all advisors and inspectors Number of advisors who are engaged in trainings for implementing IIEP school-based activities 20 21 Percentage of education inspectors trained to evaluate the IIE aspect of school performance 40% of all inspectors 96% of all (48) inspectors In line with findings from Component 1, the project has demonstrated a high level of success in engaging national-level educators, exceeding its original target by 15 percentage points to train 85 percent of national education institution advisors and inspectors. The team learned that the original training materials were developed by CHRCR in collaboration with Quincy University and have been further refined based on feedback and lessons learned from earlier project phases. Interview responses indicate that CHRCR has trained a team of 46 Master Trainers with approximately equal numbers of representatives from education institutions (MoES, BDE, and VETC) and teachers. The project has also far exceeded its targets for training State Education Inspectors on how to incorporate IIE indicators during their school evaluations. Interview responses explained that the SEI expects to Component 2 Activities  Collaborating with key educational institutions and municipalities  Conducting baseline survey  Developing training modules and training Master Trainers  Training School Integration Teams  Improving skills for teachers and professional service staff  Empowering school boards and school managers  Enabling active parent-school partnerships  Establishing a resource-sharing system Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 11 implement new evaluation criteria during the upcoming school year but, at present, it does not plan to provide additional training to its municipal inspectors. Rather, the State Inspectorate plans to send a memo with the new criteria to Regional Inspectorate Office supervisors. It is unclear whether the training previously provided will be sufficient to ensure inspectors’ adequate understanding of the concepts, methods, and processes of IIE and subsequently the assessment of schools with the new indicators. Project staff explained that IIEP will continue to provide trainings on the use of the IIE indicators and that three regional trainings have already been planned for the fall of 2014. IIEP and SEI are also currently working on designing instruments to test schools’ compliance with these indicators. Beyond these activities, the team also learned of two other key activities undertaken by the project to strengthen educators’ competencies. First, IIE topics have been introduced in the Principals’ Exam, further emphasizing IIE as an important element of education policies and management. Second, while intercultural understanding is already a general criterion in school evaluations, IIE indicators historically have not been interwoven in all evaluation categories. Going forward, IIE indicators will be incorporated across all seven categories of the integral school evaluations conducted by the SEI annually for each school. It is expected that this will formally encourage schools to make IIE activities and policies an integral part of their instruction and school life. Sub-IR 2.2: Education stakeholders’ skills to support and/or implement IIE activities on school/community level are improved. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of municipalities actively engaged in monitoring of the implementation of IIE in-school activities 60% of all municipalities 29.7% (21+4) of all municipalities Percentage of schools with boards that incorporate the IIE philosophy into their structure and functioning 50% of 251 included schools 55% Percentage of schools whose boards and principals are involved in planning and implementation of school-/ community-based interethnic projects 60% of included schools 76.97% of included schools Percentage of schools that have SIT established 70% of included schools 92% of included schools Percentage of schools whose SIT are engaged in dissemination of IIE knowledge and skills to the other teachers in their school 70% of included schools 93.95% of included schools Percentage of teachers from two-/three-language schools who are engaged in joint professional activities 20% of the teachers from 19 included two-/ three-language schools 72% of the teachers from 19 included two-/three-language schools Percentage of teachers from two-/three-language schools that implement joint activities with ethnically/linguistically mixed groups of students 40% of trained teachers from included two-/ three-language schools 30.5% of trained teachers from included two-/ three-language schools Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 12 Percentage of two-/three-language schools that have parents from different ethnic/language background engaged in joint school-/community-based IIE activities 40% of included two-/ three-language schools 54% of included (19) two-/three-language schools IIEP has undertaken an extensive number of activities focused on building the capacity of local stakeholders to implement and/or support IIE activities at all primary and secondary schools across Macedonia. Given the ambitious scope of this set of activities, it is perhaps not surprising that the project is seeing mixed results. Just as with Component 1, the project is once again finding it difficult to engage with municipal officials. According to the project’s own monitoring data, the project reached less than half of its intended number of municipalities actively monitoring the implementation of IIE activities in their schools. IIEP reports that its achievement is lower than expected due to the fact that municipalities included in Phase 2 are still at the beginning of monitoring the IIE activities in their schools. It would also be reasonable to expect a slower rate of implementation at this level as these practices are relatively new; education management responsibilities were decentralized to the municipal level only recently. At the community level, the project works with SITs in each school—composed of teachers, school administrators, technical staff, and in some cases a representative from the Parents’ Council—to design, implement, and document IIEP activities. The project relies on a cascade training approach wherein SIT members receive training from the Master Trainers and are then expected to disseminate these new approaches to their peers. The project reports that 92 percent of the included schools have established SITs. SIT members interviewed by the team appeared clear about their roles within the project and reported that the initial training and technical assistance conducted by Master Trainers were welcomed and generally appreciated. Project-developed resources (such as handbooks and manuals) were described as helpful, although a few SIT members did mention that certain key concepts were not new to them and some of the trainings could have been better targeted. The team also learned that IIEP is beginning to work with teacher training institutions, such as the Pedagogical faculty, to better incorporate IIE topics in pre-service training for teachers. Project staff report that SIT members have conducted dissemination training in 58 pilot and 191 second￾phase schools. Each Master Trainer also provides supervision and mentoring support to several schools. The team found a consensus among interviewed SIT training participants that supervision and mentoring support could be strengthened. Most participants see supervision meetings as greatly beneficial and stress that they want more mentoring support. A number of interviewees expressed concern that too much of the time at the mentoring meetings is taken up by logistics, administrative, and finance issues rather than methods and challenges of conducting IIE activities in schools. Some teachers also expressed concern with the manner in which these sessions have been conducted and sought more constructive feedback. Teachers and BDE advisors also report that mentoring is not consistently delivered but rather dependent on the individual mentor. “One thing to think about is the mentoring visits, for more constant support of the schools for implementing these activities. Not all are ready to accept this as a way of life, overcoming the needs to dominate in the environment… People have a need for contact with someone who supports them continuously.” —Principal Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 13 In addition to educators, parents and other community members play important roles in the education process. The project recognizes the importance of the parents and makes efforts to include them in IIE activities and decision￾making processes. All educators interviewed stressed the importance of reaching out to and involving and equipping parents to support their children and teachers for further interethnic integration. Parents have been involved in select activities with their children, and they have participated in some decision-making processes through the Parent Councils and school boards. However, most school staff reported a relatively low level of involvement from parents in supporting IIEP activities. Although teachers and students observe that many parents are receptive to the idea of being more engaged, parents claim to lack the time and resources to be more present in the schools. In fact, the vast majority of parents interviewed by the team were generally supportive of their children’s participation in IIEP activities, but they were often not very knowledgeable about what these activities actually entail. It is important to note that a minority of parents—and some students—reported fears and concerns about socializing with other ethnic groups (especially about “their girls” meeting “other” boys) and fears of violence and badly managed conflict. Most of the interviewed parents of pupils at schools in which different ethnic groups have class during different shifts do not trust schools and educators to care for their kids in an integrated setting. Often this was the justification provided for favoring the separation of kids by ethnicity, be it in different shifts or in different buildings. Sub-IR 2.3: The number and diversity of multiethnic activities within the schools and communities are increased. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of two-/three-language schools that organize all short-term extracurricular activities with ethnically/ linguistically mixed groups of students 50% of included two-/three-language schools 80.3% of included (19) two-/three￾language schools Percentage of two-/three-language schools that organize all long-term extracurricular activities with ethnically/linguistically mixed groups of students 50% of included two-/three-language schools 60.5% of included (19) two/three￾language schools Percentage of two-/three-language schools that introduce joint teaching of regular subjects (with linguistically mixed groups of students) 50% of included two-/three-language schools (minimum 1 subject per school) 43.4% of included two-/three-language schools (minimum 1 subject per school) Percentage of students in two-/three-language schools that participate in joint long-term extracurricular activities 30% of students from included two-/ three-language schools 11.8% of students from included two-/ three-language schools Percentage of one-language schools that are engaged in joint cross-school activities with ethnically/linguistically mixed groups of students 20% of 195 included one-language schools 20.7% of included schools Percentage of one-language schools whose staff is engaged in cross-school professional cooperation with the staff from a school using another language 20% of the included one-language schools 33.1% of the included one￾language schools “Maybe we have some issues with cooperation between parents and teachers. Bringing parents to schools, parents have not seen that they should be part of this process. We don’t have any mechanism for bringing them here. There is a rulebook about parents’ participation that speaks about involvement of parents in activities. I would change the part about cooperation about parents—I would make it a special project.” —SIT members Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 14 The project has mobilized school personnel to conduct a large number of activities and exceeded expectations for the percentage of schools conducting both short- and long-term integrated activities since the start of the project. Schools that offer instruction in more than one language are undertaking numerous activities, including joint extracurricular activities, such as holiday celebrations, field trips, and sports events, as well as occasional joint classes (with teachers and students from classes of different languages of instruction) in the subjects of foreign language, art and music, and physical education. Schools are a bit slower to integrate regular classes9 and these classes do represent a greater integration challenge than extracurricular activities due to the need for different languages of instruction. Students seem to favor extracurricular activities, reportedly because they see them as more fun, outside of the school routine, and as opportunities to spend longer periods of time together doing something they enjoy. There is wide agreement among educators that the project is helping them to better structure and document activities, many of which they already had been conducting prior to the project. These include excursions, holiday celebrations, sports activities, and joint choirs. Many point out that these activities, in the past and now, are still not always fully integrated. One illustrative example was a recent joint field trip taken by a pair of schools, where each school booked a separate bus—the Albanian group rented from an ethnic Albanian–run company and the Macedonian group from an ethnic Macedonian owner—and students traveled to the site separately. The Albanian group wanted to stop in Tetovo to visit the Colorful Mosque, and the Macedonian group did not. In another school, Albanian and Macedonian students could not agree on a destination for the senior excursion, so each group went separately to their respective destinations. The team also learned that the ethnic Albanians students wanted to travel to Turkey and Albania, while ethnic Macedonian students choose different destinations in Western Europe including Bulgaria and Greece. Interviews with schools’ management, teachers, and parents indicated that they were fine with their children not going together with “others.” While the team recognizes that this is only one illustrative example, and that other, more positive examples exist, it nevertheless demonstrates the need and importance of continuous work with all stakeholders on IIE, and that integration remains a challenge and a long-term process. At the start of the project, schools were asked to undergo a process of self-evaluation; however, it is not clear to what extent the self-evaluation findings are being used by schools to assess their progress or if these findings are being used to design the IIEP activities each school undertakes. Interviews with school representatives indicated that the respondents do not appear to be connecting self-evaluation results with IIE activity selection. The team noted that nearly all school principals interviewed could not talk about the self-evaluation results without needing to refer back to the original reports and PowerPoints. Instead they reported feeling constrained to a certain list of activities (some of which they have been conducting for some time) rather than being able to pick and choose activities to best match their self-assessed needs. The self-evaluation does provide an opportunity to assess activities previously conducted and identify ways to improve them going forward; however, it seems that teachers are missing this possible application of the self-evaluation. There are two self-evaluation processes for IIEP schools—a standard MoES self￾evaluation and the IIEP form. The project team is revising the IIEP self-evaluation forms, looking to add more detail about the type and quality of the activities. The questions on the two self-evaluation forms are reportedly different, but it may be more efficient to merge the two. 9 Classes that are part of the regular curriculum and subjects, such as math, history, and social studies. “We’ve implemented a number of activities: comparison of literature and grammar, comparison of Macedonian with Turkish, German, [and] French poets. Students were writing essays in both Turkish and Macedonian and then had discussions. Sex education classes together about AIDS and other STDs. We’re also doing some electrical work, creating manuals, performing sports activities—mountain biking, etc. Also had national day about saving energy.” —Principal in southern Macedonia Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 15 School staff participation in IIE activities has exceeded project expectations, yet teachers generally reported that they face institutional and procedural constraints on the integration of IIE activities into their already full workloads. Some teachers note that the formal curricula (Наставни Програми) do not provide space for integrated activities, and it is difficult for teachers to find time to conduct integrated activities. Most teachers feel they are implementing IIEP activities outside of their formal teaching duties and beyond their working hours. One teacher explained that he is responsible for an extracurricular math club, which is composed only of Macedonian-speaking students, so he is conducting IIEP activities in addition to this. It appeared, at times, that teachers are adding additional integrated activities instead of integrating existing activities (such as thematic extracurricular clubs), which also adds to their course burden. Either it is not clear to teachers that they could integrate existing extracurricular activities or there is not a systematic structure available to facilitate such a process. Teachers also express concern that they cannot integrate their existing curricular and extracurricular activities due to the strict project guidelines that require equal ethnic representation on each student team. This poses a logistical challenge for schools with an uneven balance of linguistic groups among their student population. In such environments, it is almost impossible to provide for equal numbers of linguistic groups in each class/activity. Equal ethnic ratios in student groups may be a good approach to testing different models, but it is hardly practical or feasible for the long term and would exclude many students. Teachers also expressed frustrations with the administrative burden of IIEP reporting requirements. As noted above, teachers participating in the IIEP have to undertake multiple self-evaluation processes as well as complete other project reporting forms. On the one hand, the project can be commended for having a very comprehensive and detailed project documentation system, and educators are learning how to better document and monitor their activities through the IIEP. This is indeed a valuable skill, as CHRCR staff note that customarily extracurricular activities are usually not documented at all. On the other hand, teacher time spent on IIEP documentation must be weighed against time and motivation needed to prepare for IIEP activities in addition to other formal responsibilities. Schools also undertake partnership activities with schools of other languages of instruction, especially schools with monolingual instruction. Schools occasionally utilize technology for collaboration, mostly between the teachers, but some also reported engaging students in dialogue with peers over email, social media, or Skype. Most activities are conducted in person, and there are significant costs associated with conducting these activities, particularly in transportation. At times parents are asked for a contribution, but many family budgets cannot absorb the costs associated with their children’s participation in field trips. Schools and other education stakeholders opine that municipal officials generally are not providing enough material support for school activities for a variety of reasons, including many competing priorities in the distribution of scarce municipal resources. Schools do receive a 23,000 MKD grant from the project to support their partnership activities, and these funds are greatly appreciated. Some schools face a high demand for partnerships—especially the less common minority-language schools. As a result, these schools tend to have multiple partners, and these funds are insufficient to support all partnership activities. Besides being limited, the partnership grant disbursal has been slower than expected by schools. Schools spend the money upfront from their own budgets (and often from teacher’s own pockets) and then engage in what they perceive to be an extremely detailed and demanding financial reporting process. A number of the schools have yet to be reimbursed for IIEP activity expenses, and that has proven to be a de-motivating factor for some to participate in additional “The teachers do not have necessary amount of time for activities—they have to prepare for lessons, prepare electronic register, prepare and print tests.” —Principal “Teachers are overburdened, have a lot of administrative requirements and less time for activities and so for the moment it is more important to fulfill their electronic register than the learning of their students and children suffer because of that.” —Master trainer Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 16 project activities. One school manager said that it refused the money so as not to have to fulfill administrative and financial requirements associated with reconciling these funds. IIEP staff note that USG reporting requirements are strict and auditable but believe that this process is helping schools learn how to do proper budgeting and reporting and to be financially accountable up to international standards. Staff felt this provides an opportunity for capacity building for schools on learning how to implement clear and transparent procurement procedures. They also noted that Loja and Sumnal have been engaged to facilitate and expedite the document preparation process. Conclusions  The project has succeeded in raising the stakeholders’ awareness of the need and benefit of IIE. Educators are increasing interethnic collaboration and beginning to develop a system for further integration. Additional work is needed to support stakeholders to effectively implement (and develop new ideas for implementing) integration activities and overcome critical systemic and financial obstacles.  Future motivation of teachers and school management to implement IIEP activities is somewhat questionable without further incentives, be it financial support from municipalities, MoES, private sponsors, or other sources, or professional certification for teachers. As discussed in sub-IR 3, a majority of teachers report frustration with the lack of incentives for participation in activities, which are seen as essentially outside their job descriptions. School staff will need more assistance to be able to tackle this issue and find ways to continue to make joint student activities a reality.  Parents play a key role in shaping children’s attitudes toward ethnicity and must be partners in the education process. There is still more to do to overcome fears and mistrust that exist among the different ethnic groups. Besides raising public awareness about integration issues, further measures are needed with parents. Looking at what parents’ needs are and what it is that could bring them to the table may be a good next step.  To date, the focus of integration activities has been on language of instruction, whereby classes or schools of one language of instruction are instructed to conduct joint activities with partners using other language of instruction. While sensible, this approach has had the unintended consequence of placing a large burden on schools with minority languages of instructions, who are asked to maintain multiple partnerships. It is also missing potential opportunities for further multicultural/multiethnic integration in monolingual instruction schools. Most schools with Macedonian language of instruction contain many ethnicities within them, like Roma, Vlachs, Serbs, Turks, etc. and could benefit from extracurricular integration activities. Component 3: Demonstration Schools Overview Purpose: Support six model schools to provide examples of best practices and lessons learned that will contribute to a deeper understanding of various approaches and challenges to ethnic integration in six demonstration schools through the achievement of four IRs:  Result 1: Communication and cooperation between students with different ethnic background are improved.  Result 2: Teachers of mixed ethnic backgrounds cooperate more effectively in their every￾day work as well as in organizing joint students’ activities in a multiethnic environment. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 17  Result 3: Parents of mixed ethnic backgrounds are involved in school/community projects.  Result 4: School policy explicitly demonstrates commitment to integrated education. Target Audience: The project aimed to engage all “key” stakeholders in the selected demonstration schools, including all teachers, school board members, principals, pedagogues, psychologists, and Parent Councils. CHRCR was the primary institution responsible for the design, implementation, and oversight of activities under this component. IIEP allocated around 30 percent of overall funding for activities under this component. Sub-IR–Specific Findings Sub-IR 3.1: Communication and cooperation between students with different ethnic backgrounds are improved. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of teachers showing evidence of working together with mixed groups of students 20% of trained teachers 52% of trained teachers Percentage of students who have participated in the joint long-term extracurricular activities 20% of students 25.72% Number of subjects taught in linguistically mixed group of students (blank) N/A Percentage of students who show decrease of negative ethnic stereotypes and prejudices towards the “others” 40% of students in the samples The pre-test questionnaire has been administered and the post-test will be administered by the end of the 2013–14 school year. Percentage of student bodies at classroom level that are democratically elected 70% of classrooms 75% Number of schools that have democratically elected student bodies (blank) 5 schools Number of student bodies’ initiatives for participation in school decision-making (blank) at least one initiative per school/paired schools The team interviewed school managers, SIT members, and students in all six demonstration schools, and it held FGDs with parents from two out of six demonstration schools. There was a strong consensus across all respondent groups that the level of communication and cooperation among students had increased as a result of project activities. While respondents from all six of the demonstration schools explained that their school had implemented similar activities under previous donor projects, the number and intensity of activities under IIEP was reportedly higher. Several respondents also mentioned that activities in the past occurred more in “parallel”—meaning that although an activity would include both Macedonian- and Albanian-speaking students, the students would only interact with those from their own Component 3 Activities  Supporting SITs  Joint student activities  Supporting joint student governments  Capacity building for teachers and professional service staff  Involving parents  Capacity building for school governance and management  Mentoring, networking, and promotion Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 18 group—and that there was significantly more “mixing” and interaction under the IIEP joint activities. A comparison of the mini-survey data from demonstration and non-demonstration schools corresponds with these findings. Figure 1 shows a discernable increase in the level of interaction between students of different language backgrounds in demonstration schools. The data shows not only increased numbers of both Macedonian- and Albanian- speaking students reporting that they interact “often” with students of different language backgrounds but also that only one student (out of a possible 48) reported “rarely” interacting with students of different language backgrounds in the demonstration schools. Figure 1: Mini-survey results: How often students of different language backgrounds interact FGDs with both Macedonian- and Albanian-speaking students revealed a high level of satisfaction with the activities, and both groups of students expressed interest and excitement to learn about the “other” ethnic group. Interestingly, in the majority (but not all) of the schools, Macedonian-speaking students were quicker to mention the opportunity to interact with other ethnic groups as a reason for volunteering for joint student activities. Albanian-speaking students would often first speak about their interest in the subject (e.g., photography, dance, or football) and then mention the opportunity to interact with other ethnic groups as a secondary reason for joining. Respondents from a majority of the demonstration schools were able to cite examples of students from IIEP joint activities interacting outside of school; however, these were often limited to Facebook interactions or relatively isolated examples of a small group of students meeting for coffee or to watch a movie “Previously, the impact was from outside to inside school; now it is from the inside of the school to the outside.” —SIT member, Ohrid “We had been working on these activities for some time but not deep, so [we] thought this was a good opportunity to deepen engagement. [The] project brought additional inclusion of more minor ethnic groups. [The] project is growing in intensity— more interaction in the second phase.” —SIT member, Karposh 3.7% 0.0% 26.3% 12.5% 40.7% 47.6% 28.9% 62.5% 55.6% 52.4% 44.7% 25.0% 0.0% 20.0% 40.0% 60.0% Demonstration Macedonian n=27 Demonstration Albanian n=21 Non-demonstration Macedonian n=38 Non-demonstration Albanian n=8 Percentage of Respondents Rarely Sometimes Often Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 19 outside of school. The majority of respondents, across all six demonstration schools—but particularly in areas with higher ethnic diversity—mentioned that language barriers, politics, and the skepticism (if not outright hostility) of community members prevented these students from interacting more regularly outside of school hours. Results from the mini-survey showed that students within demonstration schools reported interacting with students from other language background more frequently; however, the mini￾survey did not ask questions regarding the intensity or duration of this interaction (see Figure 2). Figure 2: Mini-survey results: “Outside of school, I prefer to interact only with students from my ethnic group.” In interviews with MCEC, CHRCR, and USAID/Macedonia staff, as well as Kumanovo school managers and SIT members, interviewees highlighted an important incident that illustrates both the importance of the project and the challenges it confronts. Although the official police reports are still pending, interviewees recounted an incident in which Albanian-speaking students were harassed and allegedly attacked by a group of Macedonian-speaking students on their way back from a joint-student activity. MCEC and CHRCR staff noted that the alleged perpetrators were not themselves involved in the joint￾student activities. Following the attack, several of the Albanian-speaking students reported the incident to local police, and their parents are reported to have been eager and quite vocal in pushing the police to investigate and charge the culprits. MCEC and CHRCR report that Component 3 coordinators made “over 30 visits” to Kumanovo in order to mediate the situation, and joint-student activities were eventually reinstated following a roughly one-month hiatus. Although still to be officially confirmed, the two Macedonian students alleged to have been the main perpetrators voluntarily changed schools without school management needing to expel them. The incident represents an important instance in which the project was able to overcome significant external, negative public pressure to resume joint-student activities and continue supporting communication and interaction between the two groups of Macedonian￾and Albanian-speaking students in the two schools. 7.1% 26.1% 21.1% 12.5% 17.9% 17.4% 21.1% 37.5% 46.4% 30.4% 42.1% 37.5% 28.6% 26.1% 15.8% 12.5% 0.0% 20.0% 40.0% 60.0% Demonstration Macedonian n=28 Demonstration Albanian n=23 Non-demonstration Macedonian n=28 Non-demonstration Albanian n=8 Percentage of Respondents Strongly Agree Agree Disagree Strongly Disagree Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 20 Sub-IR 3.2: Teachers of mixed ethnic backgrounds cooperate more effectively in professional work as well as in joint students’ activities in multiethnic environments. INDICATOR 2013 TARGET 2013 ACTUAL Number of joint meetings of the professional teacher bodies one meeting per semester in each school/paired schools at least one meeting in the second semester 2013 at five schools Percentage of trained teachers included in joint professional activities 30% of trained teachers 35.71% of trained teachers Interview responses from teachers and school managers offered generally positive evaluations of the training and guidance received from CHRCR on how to plan, implement, and monitor IIEP activities in the demonstration schools. While some claimed that certain topics were not as relevant to them or their particular community, all claimed that the trainings were generally professional and well organized. A majority of teachers reported having previous familiarity and experience implementing “interethnic” or “multicultural” activities but many of these respondents stated that the IIEP activities were more numerous and intense. However, just as with Component 2, all interviewees across all six demonstration schools complained about the financial and administrative burden that the IIEP activities placed on teachers. Teachers repeatedly expressed frustration at the lack of financial compensation for implementing additional activities and at having to use their own resources to purchase the materials needed for many of the activities (e.g., sports or photography equipment). Teachers also complained about repeated requests to document their activities and provide extensive monitoring data, often in “long” or “complicated” Excel documents. MCEC and CHRCR staff were asked to respond to these concerns. They explained that the project was purposefully designed to not provide financial compensation to teachers for their participation in order to encourage sustainability and the continuation of activities beyond the project. In other words, the project seeks to encourage teachers to participate because they believe in the overarching objective of promoting integration, not because of financial reward. MCEC staff also explained that materials (pens/pencils, paper, scissors) were provided for the Creative Workshops but not for other activities. Regarding the administrative burden placed on teachers, CHRCR staff reported that they were aware of this frustration and were in the process of streamlining some of the M&E data requests to include only the most necessary information. The staff also explained that part of this administrative “burden” was intentional in order to encourage teachers to better document their activities, particularly the level of actual interaction between students, as well as to “push them” and provide a sense of “we’re watching them.” Sub-IR 3.3: Parents of mixed ethnic backgrounds are involved in school/community projects. INDICATOR 2013 TARGET 2013 ACTUAL Number of ethnically mixed Parent Council meetings around issues concerning the best interest of children one meeting per semester in each school/paired schools at least one meeting per semester in five schools Percentage of trained parents of different ethnicity, participating in school activities with students 20% of trained parents 60% of trained parents Contrary to the experience in the non-demonstration schools, school managers and/or SITs from all six demonstration schools reported that parents were involved in some of the extracurricular integration activities. These qualitative responses support the project’s own monitoring data, which reports that the involvement of trained parents participating in school activities tripled the project’s initial target. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 21 However, it should be noted that the parental involvement was most often described as attendance or participation in one-off activities such as holiday decorations or food preparation, tree plantings, or end-of-school-year presentations.10 The only really repeated engagement of parents described during the interviews related to the presence of a Parent Council representative on the SITs. Responses from the mini-survey also indicated that students in demonstration schools were more likely to feel that their parents allowed or even supported their making friends with children of other language backgrounds (see Figure 3). Qualitative responses during the subsequent FGDs confirmed that many of the students felt that their parents actually encouraged them to do so. This is perhaps not surprising since these same parents allowed their children to attend the demonstration schools and signed permission slips for their involvement in joint-student activities. Figure 3: My parents do not forbid me to interact with people outside of my own ethnic group. 10 For example, as explained in one of the project’s periodic reports, “Parents got involved in one way or another. In the PS Bratstvo Edinstvo in Ohrid, parents’ involvement was marked through helping the students and teachers in decorating the school hall; in the secondary economic school in Gostivar, through preparing national dishes for the need of promotion of the project activities in the school. In Kicevo, Skopje, and Kumanovo, the parents have gathered in workshops in which together with students [they] have participated in making and/or putting decorations in the halls related to the New Year celebration.” “Every teacher has the [responsibility] to implement this kind of activity. Now we have changes in our law—State Inspectorate will begin evaluation of schools according to new indicators, so if teachers want to get high grades from their teaching, then they have to implement integration in their teaching.” —Demonstration school principal 65.4% 62.5% 44.7% 37.5% 23.1% 20.8% 26.3% 62.5% 3.8% 4.2% 7.7% 13.2% 12.5% 15.8% 0.0% 20.0% 40.0% 60.0% 80.0% Demonstration Macedonian n=26 Demonstration Albanian n=24 Non-demonstration Macedonian n=38 Non-demonstration Albanian n=8 Percentage of Respondents Strongly Agree Agree Disagree Strongly Disagree “Parents are supportive and interested. We had [a] workshop about how to engage parents. Since then we decided to do one activity, “One Day of Greening,” plus an excursion to Matka, which was an initiative by the parents. Training was great, we had exercises where we got closer, found ways to come closer to achieving collaboration.” —Demonstration school principal Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 22 Sub-IR 3.4: School policy explicitly demonstrates commitment for interethnic integrated education. INDICATOR 2013 TARGET 2013 ACTUAL Number of school board meetings around issues concerning the best interest of children one meeting per semester in each school/paired schools The activity started in 2014. Percentage of IIE activities covered by the School Annual Plans 25% of IIE activities 71.15% of IIE activities Percentage of items in the school iconography that reflect the multicultural/multilingual nature of the school/community 30% of the items (20% in the paired schools) Training for iconography took place in December of 2013. Given its emphasis on institutional and systemic transformation, changes in school policies have been slower to materialize than results in other sub-IRs. IIEP appears to have had the most success in getting its activities included in Annual School Plans, both within and outside of the demonstration schools. The project has only recently begun conducting workshops related to school iconography. However, the team’s direct observation in two demonstration schools confirmed that school iconography in shared spaces was either multilingual (as seen in Ohrid) or ethnically neutral (as seen in Karposh). The team heard several accounts of school board members receiving training from CHRCR but was not told of any explicit changes to schools policies occurring as a result. School managers, teachers, and SIT members in the demonstration schools repeatedly emphasized that any official changes in school policy would necessitate MoES, BDE, and SEI involvement. Following its official requests, the project was able to get seven BDE advisors assigned to all six demonstration schools, which provides an opportunity for raising institutional awareness within the BDE of the strengths, challenges, and monitoring instruments of the IIEP activities. The project has also held at least five large demonstration events attended by MoES, BDE, and SEI representatives showcasing the work of the project. MCEC staff explained that the project tries to hold two showcase events in the demonstration schools annually—one midway through the school year and one at the conclusion. The most widely cited example of success in changing school policy in interviews with demonstration school managers, teachers, and SITs representatives, as well as project and MoES staff, was the development of IIE-specific indicators that are planned to be introduced and used during the 2014–15 school year to assess school and teacher performance—meaning that both teachers and schools will now be explicitly assessed on their implementation of integration activities. Conclusions  The level of project success varies within and across the demonstration schools, but the project overall has been effective in increasing communication and cooperation between students, teachers, and even parents of different ethnic groups, particularly during school hours or school-specific events.  There are clear concerns among teachers and administrators regarding the lack of financial support to implement and incentivize program activities. This shortcoming risks having a demoralizing effect on teachers; however, increasing financial incentives might undermine the sustainability of efforts.  In contrast to non-demonstration schools, demonstration schools have witnessed considerable parental involvement in integration activities. Nonetheless, parental involvement has not achieved the desired depth likely required to indicate or produce meaningful cultural change. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 23  Efforts to influence school policy have not progressed as fast as other programmatic activities; however, efforts to change policy will depend on central government authorities and not just the schools themselves. Component 4: Incentives to Schools and Communities Overview Purpose: In collaboration with the United States European Command’s (EUCOM) Humanitarian Assistance Program, the project offers funds for school refurbishments as an incentive for schools to participate in ethnic integration activities. Investments are intended to, among other things, benefit persons with disabilities and achieve the following:  Result 1: Schools in need of renovation committed to the IIE goals are renovated with cost share collected by the school/ municipality/local community.  Result 2: The learning environment and accessibility in renovated schools are improved for all (including persons with disabilities).  Result 3: The capacity of school management, school boards, parents, and local communities to work jointly on improving the learning environment is increased. Target Audience: School staff, school boards, parents, and municipal officials. This component constitutes roughly 20 percent of the overall project budget. The municipality for the selected school to be renovated is required to provide at least a 10 percent cost share (monetary or in￾kind). These renovations are intended to incentivize schools to implement activities from the other three components and provide tangible examples of the benefits of interethnic cooperation to stakeholders both in and outside of the renovated schools. Sub-IR–Specific Findings Sub-IR 4.1: Schools in need of renovation that most prove to be committed to the IIE goals are renovated with cost share collected by the school/municipality/local community. INDICATOR 2013 TARGET 2013 ACTUAL Percentage of submitted applications that reflect commitment to IIEP 60% of 150 eligible schools 67% (150 schools were eligible to apply) Percentage of renovated schools that prove commitment to IIEP activities 60% of schools renovated by 2012 58.3 of schools renovated by 2012 There was widespread appreciation among all groups interviewed for the renovations planned and received under Component 4. Interviewees felt that renovations to school infrastructure were an appropriate and relevant incentive to support other components of the project. They also felt that this component was particularly useful for engaging schools that are less receptive to implementing these activities. Interviews with school managers and SIT members whose schools had received renovations confirmed that School Renovation Teams (SRTs) had been established and that at least three IIE activities were included in their school’s Annual Work Plan. Component 4 Activities  Developing School Reconstruction Teams (SRTs)  Creating effective learning environments  Involving communities in school renovation  Creating partnerships with the MoES sector for capital investments Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 24 Interviews with school managers, SIT members, and municipal representatives for schools who had received renovation confirmed that their respective municipality had provided the minimum 10 percent cost share. The project reports to have raised MKD 4,344,223 from selected schools and municipalities at the end of Year 2, representing a 24 percent cost share, well in excess of the minimum 10 percent. While the majority of feedback received was positive, two weak points were identified by respondents and the team’s direct observation. The first and relatively minor shortcoming relates to the application process and subsequent school selection during the first year. The project invited 153 primary and secondary schools from 25 municipalities to apply and received 100 applications (two of which were from partner schools not initially invited). The applications were reviewed individually by the Renovation Working Group—consisting of members from MoES, “Step by Step,” EUCOM, USAID and USAID/MoES, and IIEP—and the top ten ranked schools were selected. No site visits were conducted prior to award. Given that the final selection was based solely on the written applications received, some of the schools selected, such as PS Tefejuz in Cair, had already been recently renovated. Project staff members themselves acknowledged that the ten schools selected in Phase I were not necessarily the one with the biggest renovation needs, but, from the submitted applications, they had the best ideas for integrated activities. To its credit, the project recognized this shortcoming and changed its process in the second year. The Working Group now ranks its top 20 schools based on applications received, conducts select site visits, and then selects its final ten schools. The second and more significant shortcoming relates to the timing of the renovations. Component 4 was originally intended as a reward for those schools that had best demonstrated their commitment to IIE activities. However, as explained by project staff, EUCOM requested that the project begin spending renovation funds during the first year. This resulted in a situation in which schools were submitting applications proposing activities without having first received training from the project on the different types of possible activities or the criteria for higher-quality activities. Schools were then selected based on these applications without having demonstrated their commitment to the activities they proposed, although project staff explained that such a situation was inevitable due to the need to use funds for renovations at a very early project stage. The project is essentially meeting its target 60 percent of renovated schools proving their commitment to IIEP activities (the 1.7 percent shortcoming is negligible)— with commitment defined as schools implementing at least three different activities with ethnically/linguistically mixed groups of students and teachers in their Annual Plans. However, this target is unnecessarily low and suggests the project is only partially fulfilling the original intent of Component 4. Put another way, 41.7 percent of schools that received renovations during the pilot year did not follow up on their commitment to implement at least three of their proposed activities. Sub-IR 4.2: The learning environment and accessibility in renovated schools are improved for all (including persons with disabilities). INDICATOR 2013 TARGET 2013 ACTUAL Percentage of renovated schools that have improved the healthy/safe conditions on their premises 100% of 20 renovated schools within IIEP 100% of renovated schools within IIEP Percentage of renovated schools that have accessibility for persons with physical disabilities 100% of renovated schools within IIEP 86.3% of 22 renovated schools within IIEP Three school buildings, accounting for the missing 13.7%, cannot build ramps for technical reasons. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 25 Given that the evaluation team was not able to see for itself the state of the renovated schools prior to renovation, it is difficult for the team to comment on the extent to which conditions have improved other than by relying on photo documentation that the project has available. However, based on testimonies from students, teachers, and school managers, as well as comparison with conditions in several non￾renovated schools, the team is confident that conditions did indeed improve as a result of renovations. The team did not notice the installation of any new ramps at the schools visited, although it should be acknowledged that the sub-teams did not explicitly ask to see these during their tours (the team did see the roof and window repairs). The team did note that the only project renovation to improve accessibility for disabled students came in the form of building ramps, which is only one of several possible renovations that could improve accessibility to various areas of the schools, although it should be noted that these would have higher cost implications. Sub-IR 4.3: The capacity of school management, school boards, parents, and local community to work jointly on improving the learning environment is increased. INDICATOR 2013 TARGET 2013 ACTUAL Number of renovated schools that have successfully established SRTs 20 SRTs from 20 included schools 22 SRTs from 22 included schools Percentage of SRT members that participate in the decision-making for school renovation 80% of SRT members from 20 included schools 90% of SRT members from 22 included schools Number of renovated schools that have trained technical service staff from SRT members for school maintenance two SRT members per school (from 20 included schools) two SRT members per school (from 22 included schools) One of the evaluation sub-teams observed a maintenance training for SRTs and found the training organized, professional, well attended, and accessible to multiple language groups. Handouts and PowerPoint slides were available in three languages—Macedonian, Albanian, and Turkish—and an interpreter was available for any in-person interpretation needed. The four SRTs present were composed of five to six members, representing school management, teachers, technical staff, and Parent Councils. The training provided useful information on the importance of establishing school maintenance plans and prioritizing maintenance and renovation work. The team also found that students, parents, and community members were not aware of any renovation received by the project during interviews at two separate renovation schools. This was despite at least one of the schools having a quite prominent, fully branded sign explaining the renovations at the main school entrance. Conclusion  Component 4 is very popular, and respondents report that it has led to needed infrastructure improvements. It also offers a potentially strong incentive for encouraging integration activities and involving students, parents, and the community; however, several limitations in the initial design and implementation of the refurbishment activities weakened this incentive during the pilot year but appear to have been subsequently remedied. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 26 Q2: HOW WELL IS IIEP COMMUNICATING WITH BENEFICIARIES, GOM, OTHER DONORS, AND NGOS? Overall, the project is communicating well with key IIE stakeholders across Macedonia. The project has taken a number of steps to ensure regular communication with GoM representatives, particularly at the national level (see Component 1). Independent of the IIEP, USAID/Macedonia has hired two USAID advisors who are embedded within the MoES to serve as key resources on any issues pertaining to IIE. A large share of the advisors’ portfolio includes communicating the purpose and progress of IIEP activities to various MoES officials. The team had the opportunity to interview three high-level MoES officials, each of whom expressed appreciation and support of the role played by these two advisors and stated that they had learned a significant amount about the challenges and opportunities of IIE as a result. The team is limited in its ability to comment on how successfully the project is communicating with other donors, as it was only able to interview two other donors currently supporting similar IIE activities, OSCE and EUCOM. The OSCE representatives reported being satisfied with the communication received from the project, seemed highly aware of its various activities. However, these two representatives were part of the Working Group and therefore likely to be much more informed on the details of IIEP than many of their colleagues. While the team does not have enough data to conclude that the project is not communicating with other donors, it did not hear or read of many instances where the project concertedly engaged with other donors focused on similar issues, such as the British Council or Italian, Swiss, and Dutch embassies. OSCE representatives mentioned that USAID regularly participated in a donor coordination meeting, in fact USAID staff confirmed that they regularly initiated them, which the OSCE representatives reported was quite helpful; however, this donor coordination effort is no longer convened, reportedly due to donors’ frustration at the lack of organization and action from the MoES on various issues raised during the meetings. The project has been relatively less successful in maintaining active communication at the municipal level. To be clear, this does not refer to the project’s initial outreach efforts, which were impressive (see Component I). The municipal-level challenges are maintaining ongoing interaction with, and, in particular, financial support from, these actors. The team found that the level of awareness among municipal education advisors varied quite significantly across the country and, although the project has been remarkably successful in securing a nearly 25 percent cost share for school renovations at the local level, it has not attempted in raising funds or even in-kind support for joint-student activities within or across schools. Representatives from all but a few schools (Struga and Strumica) admitted that they did not actively engage in fundraising activities within their own communities. When asked about the possibility of raising funds from local businesses, community members, and municipal offices, many of the school managers and SIT members admitted that more could be done to raise such funds. They were cautiously optimistic that such efforts would result in additional financial or in-kind support. When asked if the project actively looks to include more local NGO partners in its activities, project staff reported that it found this difficult given its limited budget. Project staff contended that the first question asked by such potential partners was how much money the project would be able to provide to support integration activities. Finally, with the exception of most demonstration schools, the project has been less successful in communicating with parents. Here the project faces significant cultural and socioeconomic challenges with engaging parents. Many respondents reported that parents traditionally are not as interested or involved in their children’s school life as in other countries and that many parents, particularly in rural areas, do not have the funds or time to travel and participate in school activities. Several respondents also claimed that the challenge of sustaining more active engagement by parents was so significant that it necessitates “a project of its own.” That said the team found that it most cases, the project’s engagement with parents Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 27 usually takes the form of their involvement as a Parents’ Council representative on a SIT and parents’ attendance at one-off demonstration events (midyear or year-end presentations), holidays decorations, or tree-planting activities. Conclusion  The project has demonstrated a high degree of success in maintaining regular communication with key beneficiaries and national level GoM representatives. While the project has undertaken efforts to engage and solicit support from municipal level representatives, parents, and local businesses and chambers of commerce, these efforts have been less successful. Nonetheless, given their ability to provide critical financial and non-financial support and encouragement, these groups offer the potential key for schools to sustaining project activities beyond the life of the project. Q3: DOES THE PROJECT APPROACH NEED TO BE MODIFIED IN ORDER TO REFLECT THE REALITY OF THE CURRENT ETHNIC RELATIONS? IF SO, HOW? The team probed on the relevance of IIE activities throughout their interviews across all stakeholder groups. There was an overwhelming consensus that the issue of interethnic integration was of central importance to Macedonia’s future. The dominant response was that Macedonia is and has always been a multiethnic society and it was thus imperative for groups to learn to live together for the sake of the country’s future. Nearly 94 percent of students (n=111) surveyed identified the issue of integration as either “very important” or “important” to Macedonia’s future. Further, a majority, particularly among the adults interviewed, felt that the project’s comprehensive, national-level approach is appropriate for tackling such a complex issue. However, a small but still important minority of respondents (n<10), particularly among teachers and SIT members, felt that some of the training and support provided by the project could have been better targeted to their specific local contexts. This was especially true in areas where there was one particularly dominant ethnic group such as in southeast Macedonia, where the large majority of the population is traditionally Macedonian accompanied by smaller populations of Turks, Roma, and very few Albanians. These respondents claimed that many of the trainings for dealing with interethnic conflict simply were not relevant to the situations in their schools. Interestingly, nearly all of these respondents seemed to associate interethnic conflict as only occurring between Macedonians and Albanians. Given the salient and often destabilizing role that politics plays in the Macedonia context, the team noted the conspicuous absence of any activities focused on educating or even trying to mitigate the influence of political parties within schools in Macedonia—school leaders are appointed at the municipal level and are usually members of the political party in power. The team probed why the project had not more actively sought to engage with political parties. Both project and USAID/Macedonia staff responded that this was intentional and that the project took significant steps to remain apolitical. They explained that given the political context in Macedonia, engaging with one political party, even for purely informational purposes only, without the other present will quickly lead to rumors of favoritism and possibly even collusion. Respondents also explained that only the Embassy Front Office deals with political parties and that previous USAID projects have tried to engage political parties and have had limited success. Thus, while project staff agreed with an overwhelming majority of respondents that politics play an unusually strong and disruptive factor in schools, they emphasized the importance of not being reactive to individual events and remaining as far outside the realm of politics as possible. While arguably a correct strategic approach, this threat to the project remains unaddressed. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 28 Q4: WHICH SELECTED ACTIONS AND/OR CROSS-CUTTING THEMES AND CORRESPONDING ACTIVITIES SHOULD BE FURTHER EMPHASIZED, MODIFIED, OR ELIMINATED, AND WHY? One of the most prominent cross-cutting themes was the project’s relatively limited success in reaching municipal-level government officials, especially compared to its relatively high level of success reaching national-level officials. While the increase in number and geographic spread of these stakeholders presents a challenge (there are 84 municipalities in Macedonia, meaning at least 84 more individuals to reach), IIEP could better target and maintain engagement with this group. As discussed under Component 1, the project has already conducted a number of awareness-raising activities at the municipal level, and the project’s Chief of Party (COP) reports having met with “nearly every mayor across Macedonia.” While it is not clear that holding additional informational events will result in municipal officials actually attending and participating in these events, the project could look to build on an example in Strumica, where a municipal representative participates in the school’s SIT. The project has invested numerous resources in organizing and training these SITs and they now represent a key entry point for increasing the level of community awareness of the progress being made towards integration within the schools. Having municipal representatives serve on SITs would not only result in increased awareness of project activities and early results but might also encourage a level of personal buy-in from these representatives. In some cases it could result in their transformation into “champions” and support (financial or otherwise) from municipal offices. Another cross-cutting issue centers on the financial support for project activities. Respondents overwhelmingly conveyed their frustrations and concerns for the future sustainability of activities given the limited financial resources available to support them. Respondents in 12 separate interviews raised this concern, with principals and SIT members most likely to raise the issue. These respondents claimed that the MKD 23,000 received to support partner activities, though appreciated, was barely enough to cover one partner school visit. The team heard several accounts of teachers needing to pay or supply their own materials to use during creative workshops or paying travel expenses out of pocket. While a few respondents reported that IIEP had stressed the importance of their schools organizing their own fundraising activities, they claimed that the parents in their communities were often too poor to provide any kind of financial support and that their municipal governments were unlikely to make any additional funding available. The team probed on the extent to which schools had actually tried to organize such fundraising activities and representatives; all but one school admitted that they had never actually tried. As the project moves into its final years of implementation, it should consider providing specific trainings focused on presenting possible fundraising activities that schools could organize. Interview responses revealed a relative dearth of ideas from school managers and teachers, other than craft and bake sales. One useful contribution would be to develop a toolkit of example activities that schools could use to raise funds. Ideally these would be built around interethnic activities. Interviews with Working Group members also revealed a frustration with the sequencing of project phases, which have been too short and prevented the project from applying lessons learned during the previous phases. On a higher level, the sentiment that the effective promotion of integration involves “a process, not a project” was expressed throughout the team’s interviews. While respondents generally recognized that the project was laying an important and needed foundation, they were nearly unanimous in explaining that the issue of integration, specifically the integration of all the country’s primary and secondary schools, could not be achieved in a four-year project timeframe. The more active involvement of parents is another cross-cutting issue likely to impact the ultimate success of the project. Respondents generally recognized that children spend the majority of their daily lives outside of their schools and that the opinions, attitudes, and actions of their parents have significant influence in shaping children’s attitudes towards integration. Respondents report that the project is having Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 29 greater success engaging with parents from its demonstration schools, and it is likely that non￾demonstration schools could learn from these experiences. Nonetheless, the majority of respondents, including IIEP staff themselves, recognize that even in the demonstration schools parents could be much more actively involved. Ironically, the last cross-cutting issue for the project to consider is the relatively weak cross-cutting (or at least mutually supporting nature) of its four components. Two notable examples include the redesign of Component 4 to serve more as an incentive to implement IIEP activities in the future rather than a reward for successfully implementing activities under Components 2 or 3. While not as pronounced, the team also found relatively little evidence of early successes in the project’s demonstration schools and lessons learned (Component 3) influencing activities under Component 1. The team noted that while the vertical linkages between activities in each component are easy to understand, the horizontal linkages between each component are much more difficult to determine in the project’s design and logical framework. Q5: WHAT SHOULD THE IMPLEMENTER DO TO ENSURE SUSTAINABILITY? The project has made significant progress in raising awareness of the issues and communicating the need for and benefit of IIE. In the design of the project, the implementer has seriously considered sustainability and strives to build capacities and systems to this regard. IIEP has built local capacity by training and engaging local educators, working with institutions such as the MoES, BDE, VETC, SEI, SEC, and municipalities to develop institutional frameworks and policies in support of IIE and developing comprehensive manuals and handbooks for training and IIE activity implementation. The Working Group for IIE has been established at the MoES and two members of staff have been engaged to act as point persons for IIE. Furthermore, local NGOs have built IIE expertise and capacity to continue promoting integration in their communities. Nonetheless, sustainability challenges remain. For example, while designed to ensure sustainability, the project’s cascade training model presents a sustainability challenge. Even though Master Trainers are full￾time employees of the education system, they are compensated for the trainings delivered and mentoring sessions held with SIT members. The evaluation team did not find any plans to continue using Master Trainers in IIE issues beyond the life of the project. At the system level, the team found no clear plans for continued teacher professional development in IIE through existing educational institutions. The operational agreement with the BDE, for example, does mention that the BDE staff will collaborate with IIEP to support schools in “creating programs for teacher professional development in IIE” as a project activity, but the team found no evidence of how this might occur beyond the project period. The project is starting the process and building momentum, but government institutions need a system that will allow for sustained professional growth. Developing the collaborative learning skills and conflict mitigation schedules to operate in a multicultural and multilingual post-conflict environment remains a significant challenge. Despite some challenges reported with the ongoing support after trainings, teachers are trying new methods, becoming more familiar with integration, and learning to document and systematize their IIE activities better. Nonetheless, formal recognition of teachers’ efforts has been slower than expected, which, it should be noted, is due in part to delays with BDE approval and puts their continued involvement in question. Teachers are professionally evaluated by means of evidence (certificates) for trainings completed and activities conducted. The team learned that teachers have not received IIEP certificates yet, although confirmations of attendance are provided by IIEP, and there seems to be a discrepancy between their expectations and the projects’ plans and timeline. According to the Working Group for IIE, Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 30 teachers will receive a certificate after they have not only completed trainings, but also completed a full year of IIEP activities. The project management is working to have the certificates issued by the BDE so that they are formally recognized in the framework of the country’s education system. BDE approval of the certificates is still underway, yet almost all trainings have been completed and many teachers have already been implementing IIEP activities for more than a year. Teachers expressed concern at having no evidence of their efforts to show to inspectors. In addition, some teachers noted that under the current system of teacher remuneration, teachers who receive a good “grade” in their evaluation do not receive any greater reward than other teachers who have not been as active in the school, which is a de-motivating factor. A related project supported by USAID and implemented by MCEC, Teacher Professional and Career Development Project, 11 is focusing on reform of the teacher certification and reward system. While it was not clear to the team when elements of this reformed system would be rolled out, formalizing certification around IIEP teacher engagement would be one way to formalize an incentive structure for teachers’ motivation and involvement in conducting IIE activities. A majority of teachers express concern that integrated activities would have to be formally incorporated in curricula (наставни програми and насоки) in order for them to become standard practice in schools. However, curricula for individual subjects set standards but do not specify the makeup of the class. It seems that there are mixed messages amongst the different stakeholders on what exactly needs to happen for a school/teacher to organize and conduct an integrated curricular or extracurricular course. According to the VETC, schools that wish to implement an integrated elective course on a topic of their choice (usually as an elective) would present a proposal to the VETC; upon approval of the concept, they could implement the activity as proposed. Reportedly there are 18 such integrated electives in the country’s vocational school system. Gymnasiums12 and elementary schools also can propose electives (project activities) in their work plans; once they are approved, they can be implemented as such. It is a bit more complicated for curricular activities. According to BDE representatives, standards for curricular subjects are established but it is up to the principal to decide how to organize the classes—whether monolingual or multilingual. Both principals and teachers, however, appear hesitant to engage in such a process and reorganize their classes in a system that is currently organized around a single language of instruction. This is the area that will need more attention from all stakeholders in order to make the structural changes required to support true integration amongst different ethnic groups in the schools. The project has worked extensively on the integral evaluation indicators and school and municipal work plans, and this is a good first step. The recommendations section provides a list of options that the project could consider in order to further strengthen the sustainability of its activities. Q6: WHAT EXISTING ALTERNATIVE APPROACHES COULD LEAD TO BETTER RESULTS? The project applied holistic and inclusive approach by including all important stakeholders from the design phase and during the implementation of project activities. This approach was widely accepted and praised as the most appropriate one. There are and have been a number of programs addressing interethnic collaboration in education in the country, each taking slightly different approach to the issue. Examples 11 http://www.mcgo.org.mk/usaid-teacher-professional-and-career-development-project/ 12 Liberal arts high schools “We have no certificates for any of the trainings we have attended. We have asked for them at each mentoring meetings, they say it is coming. We have not certificates for this… We expect certificates—integral evaluation is coming… Inspectors do not know what IIEP is. Inspectors want paper, black and white. They will not evaluate us on activities, but on certificates.” —Teachers Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 31 are the current Strengthening Multiethnic Cooperation in Municipalities Project13 of the Pestalozzi Children’s Foundation (also implemented by MCEC), the Nansen Dialogue Model for Integration program,14 the work of OSCE, and the UN Joint Programme for Interethnic Community Dialogue, which ended the summer of 2012. The Pestalozzi project is focusing on only two school subjects and mostly civic education, and it undoubtedly complements the IIEP with the training it provides for teachers, municipal officials, and parents. The Nansen Dialogue Model is more narrowly focused on a few model schools, one of which was built for this very purpose, and it takes a whole-school approach. The UN Joint Programme addressed education as one of a number of issues related to the topic of interethnic dialogue, but it did develop some foundations for the work to come, such as guidelines and manuals for interethnic dialogue and integration in education. Of course, the OSCE was instrumental in the creation of the SIE, which has provided a strong foundation for the IIEP. In all of these programs there are some good approaches that can be used to achieve particular results. Some good examples of such initiatives are the inclusion of related interethnic understanding themes in the life skills classes, as well as some formalized programs for integrated elective15 courses both in primary and secondary schools. One such example is the World Heritage program of Freeform Classes (Слободни Часови) in vocational secondary schools developed by the VETC— a formal elective course that teachers can offer in the framework of the regular school work and salary. So called “Elective Activities” in primary schools and “Project Activities” and “Freeform Classes” in secondary schools are good vehicles for integrated activities and more could be done in that direction, together with MoES, the BDE, and the VETC. The project could focus its efforts on developing formal programs (насоки) for elective IIE courses with the MoES, BDE, and the VETC and then train teachers to offer such courses. These electives are part of the regular course load (and salary) of the teachers and as such they may be more motivated to implement them. One problem with this approach, however, is that electives mean the teacher elects for the whole class; students do not really elect their elective course. Major changes in scheduling of classes in schools will be required to enable true mixing of students in elective and other courses, and those changes may often be impossible due to shortage of space and other logistical issues. By understanding the importance of interethnic integration in shaping children’s minds, the project could consider including preschools/kindergartens. Mozaik is a preschool model which is already becoming institutionalized and it could use some help to grow. Extending the SEI IIE criteria to kindergarten evaluations could be one step, as well as helping fund wider-scale teacher training in the Mozaik model and opening new Mozaik groups in state and private kindergartens. This may be outside this project’s scope but could be considered for future interventions. A bolder and more aggressive campaign of promoting the SIE is another possible approach—it could help spread the message, but it carries its own risks of backlash if not communicated effectively across multiple audiences of varied political opinions. 13 Following an interethnic understanding and intercultural education model developed in cooperation with BDE, 18 ethnically mixed primary schools in nine municipalities implement a comprehensive program with the students through the training of school governance and management structures and teachers. These schools have accepted and practice policies for interethnic integration in education. Student activities are implemented with grade 3 students in Know your Community and grade 7 students in Civic Education classes. 14 The Nansen model for integrated education (NMIE) is a new model of education which enables integration of students, parents, and teachers from different ethnic communities through a modern upbringing and educational process. The model supports the integration of students who belong to different ethnic communities, who study together (each in their native language) in the same school facility in the same shift; it is implemented in nine primary and secondary schools throughout different municipalities in Macedonia. 15 Изборни програми, содржини и активности Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 32 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 33 5. OVERARCHING CONCLUSIONS While the preceding sections have provided detailed findings and conclusions specific to each evaluation question, this section highlights the higher-level conclusions. It seeks to provide a clearer picture of the “forest for the trees” and looks to overcome a common tendency in process-focused performance evaluation reports to overemphasize shortcomings in implementation and underemphasize relatively significant achievements. Overall, IIEP is an ambitious, far-reaching project that seeks to identify and rectify cultural and ethnic stereotypes hardened by conflict and generations of mistrust. It operates with a relatively small budget ($5.2 million over four years) to design, implement, and monitor over 87 discrete sets of activities directed at a diverse but necessarily varied group of stakeholders at the national, municipal, and local levels spread throughout the country. Considering the size and complexity of the issue that the project seeks to address—the interethnic integration of all schools at the primary and secondary level—the project is performing remarkably well. The project is divided into four distinct but mutually supporting components. Component 1 focuses on outreach and communication. The project was carefully designed to engage, secure, and maintain the early buy-in of key GoM representatives and has demonstrated a high level of success in raising the awareness of the need for and benefit of IIE activities with national-level education stakeholders. While the project has succeeded in conveying this information to key audiences within schools at the local level, it has had more limited success in maintaining active communication and engagement with municipal officials and parents. The project has undertaken a number of activities to promote and shape the reporting of its activities by the media and has seen an uptick in the number of media stories covering IIE activities. In particular, IIEP has had considerable success reaching local media outlets and even encouraging local students, teachers, and administrators to create and share their own positive IIE stories. However, the project is having more difficulty in securing the interest of national-level media outlets and encouraging these journalists not to invent, overemphasize, or sensationalize dimensions of interethnic conflict where they may not exist. Under Component 2 the project is undertaking an ambitious set of activities to build the capacity of national education institution advisors to serve as Master Trainers and build the skillsets and awareness of teachers, school managers, school board members, parents, and municipal education officials to implement IIE activities at the local level. Given the broad scope of activities, the project’s performance under this component has been mixed. The project has succeeded in raising stakeholder awareness of the need for and benefit of IIE. Educators are taking interethnic collaboration to a higher level and beginning to develop a system for further integration. Initial success has been achieved in institutionalizing integration issues in the education system, though this process has been slow. Stakeholders need further encouragement and support to effectively implement integration activities and overcome critical systemic and financial obstacles. The project provides more intense support to six model schools to provide examples of best practices and lessons learned from various approaches and challenges to ethnic integration in schools under Component 3. The project is progressing well in terms of building a deeper understanding of the issues and challenges to integration within demonstration schools; however, additional work is needed to disseminate this knowledge externally. Generally, teachers were satisfied with the training received and students report an increase in communication between groups. The project has taken a number of steps to document its learning, including plans to produce a best practices manual; however, more could be done to “demonstrate the demonstration schools.” As with the Component 2 schools, the project is finding it challenging to engage parents on a sustained basis and activities largely involve students and teachers already supportive of integration activities. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 34 Component 4 serves largely as an incentive for schools to consider designing and implementing their own IIE activities. This component is working relatively well in this regard, particularly in encouraging the involvement of school management, SITs, and municipal representatives. However, the component did not function as a reward for the successful implementation of activities during the pilot year, although this has been remedied during subsequent years. Schools applying to be part of this component are generally proposing five IIE-related activities, but more work could be done to ensure that the three proposed activities are integrated into Annual School Plans and that they are genuinely implemented, and that parents, teachers, students, and the community at large are better informed that this was an incentive/award for IIEP. The project is relevant and responsive to the context and ethnic realities present in Macedonia today; however, the disruptive influence of politics in school life, limited financing and opportunities for fundraising for activities, the segregating nature of the education system, and the lack of concrete professional and/or financial incentives to continually motivate teachers threatens the project’s sustainability in the long term. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 35 6. RECOMMENDATIONS Overall, IIEP is a well-designed, high-functioning project that is not in need of any major changes in design or implementation. Both IIEP and USAID/Macedonia staff appear familiar with many of the successes and challenges presented above and have in many instances already begun discussing ways of strengthening the project going forward. The team had the opportunity to discuss various component-specific recommendations with both IIEP and USAID/Macedonia staff. Below, the team provides a more overarching and prioritized list of actionable recommendations for the project to consider during its final years of implementation. 1. Continue to work to institutionalize IIE activities within the formal State education institutions. IIEP has rightly recognized that education stakeholders ultimately defer to guidance outlined in the national curriculum and criteria set by SEI. To support the implementation of this recommendation the project could, for example, (1) work with BDE to look for opportunities to “integrate” IIE activities into existing curricular and extracurricular programs; (2) support SEI with customized trainings that explain the importance and meaning of the newly developed IIE indicators; (3) continue to advise and support MoES to better understand and prioritize IIE issues, including working with the new Minister of Education; and (4) continue to work with the IIE Working Group to formally integrate their policy development activities into State structure and institutions. 2. Strategize on ways to identify and secure additional funding for IIE activities. While the project intentionally limited the amount of financial support it would provide to schools, the issues and constraints associated with limited funding cannot be ignored and present a real challenge to its future sustainability. Recognizing that the project operates on an already tight budget, IIEP could consider (1) encouraging and supporting the MoES to reestablish and coordinate donor coordination meetings to discuss opportunities to support IIE activities. The project should promote its SITs as a pre-established and organized entry point to coordinate donor activities within schools; (2) providing training and mentoring to school managers, teachers, students, and parents on a range of possible fundraising activities, highlighting strategies that have proven effective in the past; (3) encouraging municipal officials and local businesses to participate and support IIE activities in their communities; (4) advocating for a MoES grant program and/or municipal budget allocations to support IIE school activities; (5) establishing conditional matching goals, possibly financed through Component 4, where IIEP would provide matching funds to schools that fundraise to certain targets; and/or (6) introducing a nationwide competition in which IIEP participate schools present their activities and receive awards for best implementation or most creative ideas. 3. Continue to provide professional development opportunities and professional incentives for education institution staff and teachers. With the project finalizing its basic trainings across all schools, the project is at a good stage to focus on (1) providing increased and improved mentoring and professional development support for educators (i.e., teachers, principals, MoES staff) to strengthen their technical skills and capacity to meet expectations; (2) continued professional development opportunities for Master Trainers (already planned as Advanced Training); (3) exploring further support for education inspectors, including efforts to familiarize them with the new IE indicators and their optimal application in practice; (4) involving more teacher training institutions and other higher education institutions to learn from the project’s experiences, support teachers (through student internships, research assignments, teaching practice), and take measures to address teacher professional development issues in the short and long term; (5) working with BDE to develop plans for continued staff and teacher professional development and support; and (6) Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 36 finalizing the process of certifying IIE activities as a key incentive for teachers and working to incorporate them into reformed mechanisms for credentialing, remuneration, and professional growth. 4. Redouble efforts and devise new ways of reaching and sustaining the engagement of municipal officials in activities at the local level. The project has demonstrated considerable success in reaching key GoM officials at the national level, but it is having difficulty replicating that success at the municipal level. Given that municipal officials hold a relatively high level of influence within schools, helping to appoint school principals and determine school budgets, they are key stakeholders who can ensure the project’s future sustainability. In order to implement this recommendation, the project could consider (1) inviting municipal representatives to participate in SITs, perhaps even requiring such participation in written agreements with municipalities; (2) continuing to invite municipal representatives to dissemination events highlighting early successes; and (3) considering continuing to provide and even strengthen specialized training and follow-up mentoring for how municipal officials could better support IIEP within schools across their municipality. In order to encourage municipal officials to actively participate, IIEP could consider channeling funds through municipal budgets, which would help build municipal awareness of education issues as well as their capacities in this domain. For example, all of the M&E data collection activities expected of teachers in the project could be handed over to the municipal staff. At the least, they should be included in the data collection and analysis for the purpose of future education planning in the municipality. 5. Strengthen communication and awareness-raising about early project successes, and the concepts, practice, and benefits of IIE. The project demonstrated its earlier ability to communicate and raise awareness of the purpose of its activities. The project should now take the next steps and start to communicate some of its early results and lessons learned. Possible ways to do so include (1) holding additional public events to demonstrate early successes in the project demonstration and partner schools; (2) encouraging additional representation on SITs and municipal IIE Councils to include students, parents, municipality representatives, and local chambers of commerce; (3) organizing larger, non–school-specific events such as IIE conferences; and (4) looking for ways to more actively use ICT and web￾based collaborative learning tools to encourage more partner school interaction and diffusion of best practices. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 37 ANNEXES Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 38 ANNEX A: EVALUATION STATEMENT OF WORK STATEMENT OF WORK FOR THE PERFORMANCE EVALUATION OF USAID/MACEDONIA’S INTERETHNIC INTEGRATION IN EDUCATION PROJECT (IIEP) I. SUMMARY The United States Agency for International Development (USAID) in Macedonia seeks the services of a Contractor to perform a mid-term performance evaluation of the USAID-funded Interethnic Integration in Education Project (IIEP), award #AID-165-A-12-00002, to take place in April-May 2014. II. BACKGROUND Since its independence in 1991, Macedonia has weathered difficult economic circumstances, interethnic strife, significant “brain drain,” high unemployment, and a continuing disagreement with its neighbor, Greece, over rights to its name, which threatens Macedonia’s entry into NATO and the EU. The ethnic divide in Macedonia has existed for decades, but escalated after independence in 1991. This issue took center stage during the conflict in 2001 and is one of the primary topics treated in the Ohrid Framework Agreement (OFA), a peace deal signed by the Government of Macedonia (GoM) and ethnic Albanian representatives ending the armed conflict. There has been quantifiable progress on OFA targets such as decentralization, equitable representation, and use of languages of different ethnic communities. In education, the increase in mother-tongue curriculum and two new Albanian-language universities have increased non-majority enrollments at all education levels, but at the same time contributed to the ethnic division in education. However, the overall ethnic integration remains to be a challenge. Students and classes in the Macedonian education system have been divided by language of instruction for decades, which limits opportunities to interact. Although the curriculum is officially the same, disparate histories are taught in schools, contributing to skewing the concept of the “other” and perpetuating harmful stereotypes. Teacher training is also segregated. University programs and career training are divided along linguistic and ethnic lines, impeding professional integration and exposure to interethnic methodologies. In addition, school management and teacher employment are heavily influenced by ethnically-divisive politics. Minority students begin learning Macedonian language in fourth grade with only two classes per week. Non-majority youth face significant lingual obstacles to the labor market. Majority students rarely speak minority languages. Until few years ago, the international community’s moderate success with the scattered interethnic programs was weakened due to the GoM’s lack of a general policy, political will, and commitment to multiculturalism. However, on October 5, 2010, the Government of Macedonia voted to formally adopt a Strategy toward Integrated Education (SIE), developed in partnership with the Organization for Security and Cooperation in Europe (OSCE). The approval of the Strategy took almost a year reflecting the highly sensitive nature of this topic. USAID and the US Embassy worked tirelessly with OSCE, the Office of the High Commissioner for National Minorities (HCNM), and other donors to bring this new policy to fruition. The Strategy offers a phased approach and is organized around five thematic strands: 1) integration through joint student activities, 2) integration by increasing the mutual knowledge of other languages, 3) adjustments of curricula and textbooks, 4) improvement of teachers’ qualifications for integrated instruction, as well as 5) preparation of school management and teachers for more effective work in a decentralized multiethnic society. The GoM formed a Working Group comprised of representatives from the Ministry of Education and Science (MoES) and its agencies, as well as donors, to oversee the Strategy implementation. After nearly two dormant years after the formal adoption of the SIE, and as a result of the pressure from the international community, the MoES has recently issued a Report on the measures taken by the GoM on the SIE implementation. In addition, the MoES has shown explicit interest and commitment to developing a Performance Management Plan for ethnic integration efforts in education, an endeavor worthy of praise, although yet in a nascent stage. Three of the five thematic strands from the Strategy have been tackled by various interventions undertaken by the international community or the host government. The issue of early acquisition of the language/s spoken in respective communities is highly politicized and threatens to derail efforts. Textbooks and curricula revision is still in the very beginning, particularly the work on revision of history textbooks, yet another politically sensitive topic. Given the highly sensitive and politicized nature of these critical issues, donor support and pressure remain critical. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 39 As agreed with the GoM, USAID/Macedonia signed in December 2011 a Cooperative Agreement with a local organization, the Macedonian Civic Education Center (MCEC), to implement a four-year, $5,200,000 Interethnic Integration in Education Project (IIEP) to provide awareness building, diversity training, technical assistance, and incentives to school boards, principals, teachers, and administration officials in support of the GoM’s Strategy toward Integrated Education. IIEP works primarily on building capacities of the Ministry of Education and Science (MoES), the Bureau for Development of Education (BDE), the State Education Inspectorate (SEI) and the Vocational Education and Training Center (VETC), so they can support ethnic integration on a system level. In addition, the project works at a local level, gradually involving all primary and secondary schools, including students, teachers, school principals, as well as school board members. The project is comprised of the following four mutually-supporting components:  Component 1— Community Outreach. The objective of Component 1 is to raise awareness of the general public, especially the media for importance of interethnic integration and why it is vital for the stability of Macedonia and its accession in to the European Union.  Component 2 - Capacity Building of School Management and Teachers. The objective of Component 2 is to provide teachers, school managers, school boards with the skills they need to work in a multiethnic environment, to help prevent divisions along ethnic lines in their schools and communities, and to create conditions that promote ethnic integration.  Component 3— Demonstration Schools. The objective of component 3 is to provide more holistic and intensive interventions in six selected schools. These model schools will provide examples of best practices and lessons learned that will contribute to a deeper understanding of various approaches and challenges to ethnic integration in schools.  Component 4 - Provision of Incentives to Schools and Communities. The objective of component 4 is to provide school refurbishments as an incentive for schools to participate in ethnic integration activities. USAID has found school renovations to be highly effective incentives to motivate schools and communities to fully engage in education reforms. A second objective is to assist these communities with the improvement of the infrastructure and learning environments of their schools, including ensuring that these schools are accessible to the disabled. Component 4 has been financed by EUCOM’s Humanitarian Assistance Program, but managed by USAID. III. PURPOSE OF THE EVALUATION  The purpose of the performance evaluation is to provide USAID with an external assessment of IIEP that will be used to enhance the effectiveness of the existing intervention for the remaining performance period and in consideration for future USAID investments in the area of interethnic integration. The evaluation will: analyze the progress and effectiveness of the interventions to date, and recommend potential modifications for improvement. IV. EVALUATION QUESTIONS A. Analyze the progress and effectiveness of interventions to date 1. What are the progress and the effectiveness of project’s interventions to date on the following project objectives: Objective 1— Education Community Better Understands the Need for Ethnic Integration for the Country’s Prosperity. Result 1: Collaboration between key education institutions in support of implementation of interethnic integrated education (IIE) activities in primary and secondary school established. Result 2: Education community on local and national level better understands the need for integrated education and its benefits for all students, teachers, and parents. Result 3: Journalists understand the need for integrated education and its benefits for the society as a whole. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 40 Objective 2— Competencies for Interethnic Integration of Education (IIE) Stakeholders are Improved Result 1: Education stakeholders’ competencies to participate actively in IIE activities are strengthened. Result 2: Education stakeholders’ skills to support and/or implement IIE activities on school/community level are improved. Result 3: Number of diversity of multiethnic activities within the schools and communities is increased Objective 3— Selected Demonstration Schools Provide Successful Models, Best Practices and Lessons Learned on Ethnic Integration Result 1: Communication and cooperation between students with different ethnic background is improved. Result 2: Teachers of mixed ethnic backgrounds cooperate more effectively in their every-day work as well as in organizing joint students’ activities in a multiethnic environment. Result 3: Parents of mixed ethnic backgrounds are involved in school/community projects. Result 4: School policy explicitly demonstrates commitment for integrated education. Objective 4— Education Stakeholders Work Jointly on Improving School Facilities and Strengthening School Capacities to Work Towards Interethnic Integration. Result 1: School in need of renovation committed to the IIE goals are renovated with cost share collected by the school/municipality/local community. Result 2: The learning environment and accessibility in renovated schools is improved for all (including persons with disabilities). Result 3: The capacity of school management, school boards, parents and local communities to work jointly on improving the learning environment is increased. 2. How well is the project communicating with beneficiaries, the Government of Macedonia, other donors and NGOs in the area of ethnic integration? B. Recommend potential modifications for improvement 3. Does the project approach need to be modified in order to reflect the reality of the current ethnic relations? If so, how? 4. Which selected actions and/or cross-cutting themes and corresponding activities should be further emphasized, modified or eliminated and why? 5. What should the implementer do to ensure sustainability? 6. What alternative approaches exist which could lead to better results? V. METHODOLOGY The evaluation team, in collaboration with USAID/Macedonia, will finalize the overall evaluation methodology once in-country. However, the Mission expects that, at a minimum, the team will:  review and analyze the existing performance information from USAID’s current partners in the subject field;  conduct site visits;  meet and interview USAID project beneficiaries, partners, other donors working in the area, host government counterparts at appropriate levels;  interview USAID staff and a representative number of experts working in the sector;  propose sustainability criteria for the mechanisms introduced or developed by the project. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 41 The evaluation team will spend approximately two to three weeks in Macedonia carrying out this Statement of Work. Before arrival in country, the team members shall familiarize themselves with documentation about the IIEP and USAID’s current assistance in the Education and Civil Participation area. USAID Macedonia will ensure that this documentation is available to the team prior to their arrival in Macedonia. The literature includes at a minimum:  The IIEP Cooperative Agreement Project Description  The IIEP reports and materials: annual, quarterly and monthly reports, annual work plans, the PMP, and other thematic relevant reports.  Reports, analyses on the situation in the area of ethnic relations in education, Strategy toward Integrated Education, latest EU Progress Reports. VI. QUALIFICATIONS Evaluation Team Members: The team should be comprised of one independent (U.S., Third Country National, local) expert, Team leader, who has experience with ethnic relations in education projects, assessments and/or evaluations in Southeastern Europe, one or two local experts, preferably for the biggest ethnic communities (ethnic Macedonian and ethnic Albanian), and a USAID representative from a different mission who will both use this opportunity as a learning curve and contribute to the evaluation with her/his knowledge. Cumulatively, the team should have significant experience in evaluating development assistance and working on or evaluating projects aimed at improving ethnic integration in education. Knowledge of USAID and other donor assistance in the ethnic integration in education area is desired. All attempts should be made for the team to be comprised of an equal number of male and female members. Local team members should have excellent understanding of the Macedonian education system and ethnic relations and be able to establish contacts and communicate effectively with government officials and local communities. The expatriate expert and at least one local team member must be proficient in English. To avoid conflict of interest, none of the team members should have current or past business relationships with the Project. All Team members will be required to provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest. The Evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy (http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf) VII. SCHEDULE AND DELIVERABLES The evaluation effort should commence in April-May 2014. The Team Leader should arrive in Skopje, Macedonia, and be prepared to begin work immediately. USAID/Macedonia will provide the team with input and guidance in setting up a schedule of interviews and site visits, but the responsibility for the schedule resides with the Contractor. The schedule should be defined before the Team Leader arrives in country and should be finalized as soon as possible after the full Team is assembled in Macedonia. The draft schedule should be ready for review and discussion at the initial team planning meeting with USAID, which should take place within two days of when the team first convenes in Macedonia. The evaluation Team shall complete an Evaluation Design document as part of their technical proposal. The evaluation design will include an evaluation design matrix (including the key questions, methods and data sources used to address each question and the data analysis plan for each question), draft questionnaires and other data collection instruments or their main features, and known limitations to the evaluation design. Prior to beginning of the fieldwork in Macedonia, all team members will review background program documents to gain a firm understanding of the situation in Macedonia and the USAID IDEAS project. The Team leader will spend between twelve and fifteen working days in Macedonia. The team will interview key USAID and Project staff, beneficiaries of USAID’s assistance, representatives of the Government of Macedonia (GoM), other appropriate donor organizations providing assistance in the area, local communities, researchers who have conducted research on ethnic relations in the Macedonian schools, and other program stakeholders. Additionally, in its response, the Contractor shall propose its plan for selecting a representative number of Project activities for the evaluation team to assess. In selecting a representative number of initiatives for in-depth evaluation, the Contractor shall consider geographic, ethnic, gender, socio-economic and other relevant factors. The following is an illustrative list of those to be interviewed by the team: Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 42  Relevant USAID staff and project staff  Central Government officials (Minister and/or Deputy Minister of Education, Director of the Bureau for Development of Education, Director of the State Education Inspectorate, senior staff from the Vocational Education and Training (VET) Center.  Relevant donor community representatives (OSCE, EU, UNICEF, British Council, Norwegian Embassy, Swiss Foundation Pestalozzi) The Contractor is encouraged to identify and visit additional Macedonian organizations and groups, both formal and informal, based on its review of materials. Proposed schedule: Week of April 14— 1) Expatriate team leader should conduct literature review; 2) Local team members should conduct literature review, schedule upcoming meetings, and arrange logistics in Macedonia; 3) Draft schedule submitted to USAID Agreement Officer’s Representative (AOR). (Deliverable 1) Week of April 21— 1) Start of the fieldwork; 2) Kick-off meeting with USAID with the goal to establish clear expectations about the outcomes of the assessment and go over the goals, schedule and methodology of the assessment; 3) At the end of the week, brief USAID orally on the progress and findings to date. (Deliverable 2) Week of April 28— 1) Second week of fieldwork; 2) At the end of the week the Contractor is required to submit a detailed outline of the evaluation with substantial substance to USAID for discussion and comment, as well as brief USAID orally on the key findings and recommendations. o/a May 12— Contractor shall finalize the Evaluation and submit a penultimate draft to USAID. (Deliverable 3) a. Evaluation Report. The following sections shall be included in the document: i. Table of Contents ii. An Executive Summary— (3- 5 pages) summarizing the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). iii. Evaluation Findings (no more than 15 pages), which provides analysis and answers the questions listed above in Section IV Scope of Work. iv. Detailed Recommendations and their potential impacts v. In addition the report should meet the 76 requirements listed under the Checklist for Assessing USAID Evaluation Reports vi. Report Appendices, including: o A copy of the evaluation Statement of Work; o Cross-reference guide listing the evaluation questions from Section IV and specifying on which page the questions are answered in the report. o Team composition and study methods (1 page maximum); o A list of documents consulted, and of individuals and agencies interviewed; and o More detailed discussions of methodological or technical issues. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.). o Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team (final draft only). o All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides. o Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 43 o/a May 19 - USAID will provide the Contractor with final comments. Per the USAID evaluation policy, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.16  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by AOR.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. o/a June 30 - The Contractor shall incorporate all comments and submit the final Evaluation report to USAID. (Deliverable 4) All records from the evaluation (e.g., interview transcripts or summaries) must be provided to the COR. All quantitative data collected by the evaluation team must be provided in an electronic file in easily readable format agreed upon with the COR. The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. Based on the review and clearance provided by the Senior Education Specialist, the USAID/Macedonia COR will approve the deliverables. The Contractor shall be responsible for providing the final deliverables to USAID Macedonia via email and in hard copy. The Contractor shall also provide an electronic copy to DEC, the database of the USAID Development Experience Clearinghouse (DEC) in accordance with normal AID/W requirements. http://dec.usaid.gov VIII. PROPOSED LEVEL OF EFFORT Expatriate Evaluation Specialist Expatriate Team Leader 5 days preparation 12-15 days fieldwork 10 days follow up and report preparation = 27-30 days total Local Evaluation/Coordination Specialist(s) 16 http://www.transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 44 (Two individuals) Evaluation Expert 2 days preparation 12 days fieldwork (at least 12) 5 days follow up and report preparation = 19 days total Coordination Specialist 12-15 days fieldwork = 12-15 days total A six-day workweek is authorized. IX SPECIAL PROVISIONS Duty Post Skopje, Macedonia. Access to Classified Information The Contractor shall not have access to any Government classified material. Logistical Support The Contractor is responsible for providing all logistical support. Office space shall not be provided by USAID. The Contractor will be responsible for providing office supplies, equipment, computers, copiers, printers, etc. Translation services and vehicle rentals are the responsibility of the contractor. Supervision The team will work under the direction of the USAID/Macedonia COR, which will be assigned by a letter from the Contracting Officer. Performance Period The Evaluation will be carried out in April-May 2014. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 45 ANNEX B: SOURCES REVIEWED USAID and Partner Documents  Macedonia Civic Education Center and Ministry of Education and Science of the Republic of Macedonia to Implement the USAID Interethnic Integration in Education Project, Memorandum of Understanding, June 2012  Center for Human Rights and Conflict Resolution and the Macedonian Civic Education Center, Report and PowerPoint, Baseline Research Regarding the Interethnic Integration in the Education, 2012. Policy/Academic Documents  Jovan Bliznakovsi, Roska Vrgova, and Sara Barbieri, Overcoming Ethnic-Based Segregation: How to Integrate Public Schools in Macedonia and Bosnia and Herzegovina, November 2013.  Ministry of Education and Science of the Republic of Macedonia, Steps Towards Integrated Education in the Education System of the Republic of Macedonia, October 2010.  Steps Towards Integrated Education in the Education System of the Republic of Macedonia, MoES  Водич за изготвување на програми за слободни часови на училиштето во интегрираното образование, 2012  http://electronicintifada.net/content/important-lessons-integrated-education-state-israel/6698  „Меѓуетничка интеграција во образованието— Обука за тимови за училишна интеграција“,ПМИО, Јуни 2013  „ Училиште што ја унапредува меѓуетничката интеграција во образованието“,ПМИО, Скопје 2013  „Колку училиштето придонесува за меѓуетничка интеграција во образованието— прирачник за самоевалуација на основните и средните ичилишта.“ Декември 2012  „Демократско учество на учениците во училиштето— нацрт прирачник за наставниците.“ Скопје, октомври 2012  2Заеднички активности со групи ученици со мешан етнички/јазичен состав– нацрт прирачник за наставниците.“ Скопје, октомври 2012  Меѓуетничка интеграција во образованието— обука за дисеминација на училишно ниво.“ ПМИО 20134  IATT EQUITY AND INCLUSION IN EDUCATION; Tools to support education sector planning and evaluation  „Насоки за формирање партнерство.“ ПМИО IIEP Project Documents  USAID, IIEP Activity Plan, 2014  USAID, IIEP Performance Management Plan  USAID, IIEP Program Description  USAID, IIEP Results Framework  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 6, April –June, 2013  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 5, January –March, 2013 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 46  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 9, January –March, 2014  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 7, July –September, 2013  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 3, July–September, 2012  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 4, October –December, 2012  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 8, October –December, 2013  USAID and Macedonia Civic Education Center, Interethnic Integration in Education Project Quarterly Report No. 2, April –June, 2012 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 47 ANNEX C: LIST OF PEOPLE CONSULTED Government Institutions Name Institution/Position Names provided in draft report only. BDE BDE Ministry of Education and Science Chief of Cabinet, Ministry of Education and Science Ministry of Education and Science Ministry of Education and Science AOR, USAID AOR, USAID CSDO BDE State Education Inspectorate State Advisor, Ministry of Education and Science USAID Advisor, Ministry of Education and Science BDE USAID Advisor, Ministry of Education and Science BDE Bureau for educational development BDE Total Female Male Macedonian Albanian Turkish Bosnian International 17 13 4 12 5 0 0 0 Non-Governmental Organizations/Partner Organizations Name Organization/Position Names provided in draft report only. MCEC/IIEP, Project Assistant, Component 2 Vocational and Educational Training Center (VET), Skopje CBC Loja Open Fun Football Schools MCEC/IIEP, Assistant, Component 4 CBC Loja OSCE, Education office MCEC/IIEP, Assistant, Component 1 MCEC/IIEP, Administrative and Project Coordinator NATO liaison office, Skopje MCEC/IIEP, Assistant, Component 1 OSCE, Education office IPA - CBC expert Youth Educational Forum YEF PAC Multimedia MCEC/IIEP Coordinator, Component 4 Association for Roma Community Development SUMNAL MCEC/IIEP, Chief of Party MCEC/IIEP Finance Coordinator OXO (Environmental education), Macedonia Macedonian Institute for Media Center for Human Rights and Conflict Resolution Youth Educational Forum YEF CHRCR/IIEP, MCE Coordinator Macedonian Institute for Media Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 48 MCEC/IIEP, Coordinator, Component 2 Association for Roma Community Development SUMNAL Center for Local Development, Radovis Center for Local Development, Radovis MCEC/IIEP, Coordinator for partner organizations MCEC/IIEP Team, Component 2 Center for Human Rights and Conflict Resolution PAC Multimedia My Career Total Female Male Macedonian Albanian Turkish Bosnian International 34 21 13 23 9 1 0 1 Municipality Institutions/Representatives Name Institution/Position Names provided in draft report only. Master Trainer, Struga Municipal Education Inspector, Municipality of Bogovinje Municipal Advisor for Education, Municipality of Radovis Municipal Coordinator, Municipality of Radovis Head of Public Department, Municipality of Bogovinje Master Trainer, Ohrid Head of the sector for education, Municipality of Gostivar Office for Local Development, Municipality of Bogovinje Total Female Male Macedonian Albanian Turkish Bosnian International 8 3 5 3 5 0 0 0 Media Representatives Name Organization/Position Names provided in draft report only. Macedonian Television 2 Inbox7, News Portal Radio Free Europe 24 Vesti, TV Station ALFA TV Total Female Male Macedonian Albanian Turkish Bosnian International 5 2 3 3 0 1 1 0 Parents Name School Names provided in draft report only. P.S. “Sande Sterioski”, Kicevo P.S. “Sande Sterioski”, Kicevo SMES “Gostivar”, Gostivar P.S. “Sande Sterioski”, Kicevo P.S. “Sande Sterioski”, Kicevo PS “Liria”, Zherovjane, Municipality of Bogovinje, Tetovo SMES “Gostivar”, Gostivar PS “Gjorgji Sugarev”, Bitola PS “Gjorgji Sugarev”, Bitola Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 49 SMES “Gostivar”, Gostivar PS “Gjorgji Sugarev”, Bitola P.S. “Sande Sterioski”, Kicevo PS “Gjorgji Sugarev”, Bitola P.S. “Sande Sterioski”, Kicevo SMES “Gostivar”, Gostivar P.S. “Sande Sterioski”, Kicevo P.S. “Sande Sterioski”, Kicevo PS “Gjorgji Sugarev”, Bitola PS “Liria”, Zherovjane, Municipality of Bogovinje, Tetovo PS “Liria”, Zherovjane, Municipality of Bogovinje, Tetovo P.S. “Sande Sterioski”, Kicevo SMES “Gostivar”, Gostivar PS “Gjorgji Sugarev”, Bitola PS “Liria”, Zherovjane, Municipality of Bogovinje, Tetovo Total Female Male Macedonian Albanian Turkish Roma Bosnian 24 9 15 10 10 1 3 0 Students17 No. of students School 27 CHSC “Zdravko Cvetkovski”, Karpos 17 P.S. “Sande Sterjoski”, Kicevo 6 PS “Brakja Miladinovci” Kumanovo 15 PS “Bratstvo Edinstvo”, Ohrid 8 PS “Gjorgji Sugarev”, Bitola 6 PS “Liria”, Zherovjane, Municipality of Bogovinje, Tetovo 8 PS “Rajko Zinzifov”, Gorno Orizare, Veles 13 PS “Tefeyuz”, Cair, Skopje 6 PS “Zivko Brajkovski”, Butel, Skopje 16 S.S. Mosa Pijade, Tetovo 18 S.S. Niko Nestor, Struga 34 SMES Gostivar, Gostivar 8 SS “Kosta Susinov”, Radovis 9 SS “Nikola Karev”, Strumica Total Female Male Macedonian Albanian Turkish Roma Bosnian 191 81 110 76 73 26 4 12 17 Given their status as minors, student names are intentionally omitted for their protection. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 50 Principals and School Management Name and Surname Position in the School Names provided in draft report only. SIT, PS “Zivko Brakovski”, Butel Pedagogue, SS “Niko Nestor” Struga Sociologist, SOSU „Mosa Pijade“, Tetovo Pedagogue, PS “Liria”, Zherovjane, M. of Bogovinje Principal, CHSC “Zdravko Cvetkovski”, Karpos Psychologist, SS “Niko Nestor” Struga Secretary, SS “Niko Nestor” Struga Principal, PS “Alija Avdovic”, Batince, Skopje Shift Manager, SS “Niko Nestor” Struga Psychologist, SS “Nikola Karev”, Strumica Principal, PS Draga Stojanovska MAT, Sopishte, Skopje Principal, PS “Bajram Shabani”, Kumanovo Psychologist, SS “Niko Nestor” Struga Psychologist, SS “Niko Nestor” Struga Principal, SMES “Gostivar”, Gostivar Shift Manager, SS “Niko Nestor” Struga Shift Manager, SS “Niko Nestor” Struga Principal, PS “Sande Sterjoski”, Kicevo Social worker, SS “Nikola Karev”, Strumica Principal, SS “Niko Nestor” Struga Pedagogue, SS “Nikola Karev”, Strumica Principal, PS “Brakja Miladinovci”, Kumanovo Principal, PS “Bratstvo Edinstvo”, Ohrid Principal, PS “Liria”, Zherovjane, M. of Bogovinje Principal PS “Tefeyuz”, Cair, Skopje Principal, PS “Rajko Zinzifov”, Gorno Orizare, Veles Shift Manager, SS “Niko Nestor” Struga Assistant Principal, SS “Nikola Karev”, Strumica Secretary, PS “Liria”, Zherovjane, M. of Bogovinje Assistant Principal, SOSU „Mosa Pijade“, Tetovo Principal, SS “Nikola Karev”, Strumica Psychologist, SIT, SOSU „Mosa Pijade“, Tetovo Assistant Principal, SOSU „Mosa Pijade“, Tetovo Assistant Principal, CHSC “Zdravko Cvetkovski”, Karpos Principal, PS “Zivko Brakovski”, Butel Ass. Principal, PS “Bratstvo Edinstvo”, Ohrid Assistant Principal, SS “Niko Nestor” Struga Principal, PS “Gjorgji Sugarev”, Bitola Total Female Male Macedonian Albanian Turkish Roma Bosnian 38 18 20 18 17 2 0 1 Student Integration Teams Name and Surname Position in the School Names provided in draft report only. SIT, PS “Bajram Shabani”, Kumanovo Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos SIT, PS “Brakja Miladinovci”, Kumanovo SIT, SMES, “Gostivar” Gostivar Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 51 SIT, PS “Tefeyuz”, Cair Skopje English teacher, SIT, SOSU „Mosa Pijade“, Tetovo SIT, PS “Brakja Miladinovci”, Kumanovo SIT, PS “Gjorgji Sugarev”, Bitola SIT, SOSU „Mosa Pijade“, Tetovo Pedagogue, SIT, PS “Bratstvo Edinstvo”, Ohrid SIT, PS “Gjorgji Sugarev”, Bitola SIT, PS “Rajko Zinzifov”, Gorno Orizare, Veles Professor SIT, PS “Bratstvo Edinstvo”, Ohrid SIT, PS “Zivko Brakovski”, Butel Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos Professor SIT, PS “Bratstvo Edinstvo”, Ohrid Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje SIT, SMES, “Gostivar” Gostivar Communication teacher, SIT, SOSU „Mosa Pijade“, Tetovo SIT, PS “Tefeyuz”, Cair Skopje Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje SIT, PS “Bajram Shabani”, Kumanovo SIT, PS “Zivko Brakovski”, Butel SIT, PS “Brakja Miladinovci”, Kumanovo SIT, Coordinator, PS “Tefeyuz”, Cair Skopje SIT, PS “Tefeyuz”, Cair Skopje Mathematics teacher, SIT, SS “Kosta Susinov”, Radovis English teacher, SS “Niko Nestor” Struga Professor SIT Coordinator, CHSC “Zdravko Cvetkovski SIT, PS “Zivko Brakovski”, Butel SIT, SMES, “Gostivar” Gostivar Professor SIT, PS “Bratstvo Edinstvo”, Ohrid Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje Professor SIT, PS “Bratstvo Edinstvo”, Ohrid SIT, PS “Rajko Zinzifov”, Gorno Orizare, Veles Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos Macedonian lang. teach, SIT, SS “Kosta Susinov”, Radovis Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje Prof. SIT Ass. Coordinator, CHSC “Zdravko Cvetkovski” SIT, PS “Bajram Shabani”, Kumanovo Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos SIT, SMES, “Gostivar” Gostivar Psychology teacher, SIT, SOSU „Mosa Pijade“, Tetovo SIT, PS “Bajram Shabani”, Kumanovo SIT, PS “Rajko Zinzifov”, Gorno Orizare, Veles SIT, PS “Gjorgji Sugarev”, Bitola Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos SIT, PS “Gjorgji Sugarev”, Bitola Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje SIT, PS “Bajram Shabani”, Kumanovo President of School Board, SS “Nikola Karev”, Strumica SIT, PS “Tefeyuz”, Cair Skopje English teacher, SIT,SOSU „Mosa Pijade“, Tetovo Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 52 SIT, SMES, “Gostivar” Gostivar SIT, PS “Zivko Brakovski”, Butel SIT, PS “Tefeyuz”, Cair Skopje SIT, PS “Zivko Brakovski”, Butel SIT, PS “Brakja Miladinovci”, Kumanovo Professor SIT, CHSC “Zdravko Cvetkovski”, Karpos Teacher, PS “Liria”, Zherovjane, Municipal. of Bogovinje SIT Coordinator, SMES, “Gostivar” Gostivar SIT, PS “Rajko Zinzifov”, Gorno Orizare, Veles SIT, PS “Gjorgji Sugarev”, Bitola IT teacher, SIT, SS “Kosta Susinov”, Radovis Psychologist, SIT, SS “Kosta Susinov”, Radovis SIT, PS “Rajko Zinzifov”, Gorno Orizare, Veles SIT, PS “Brakja Miladinovci”, Kumanovo Total Female Male Macedonian Albanian Turkish Roma Bosnian 70 53 17 31 30 6 0 3 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 53 Summary of People Consulted Type of Institution Number Gender Ethnic group Female Male Macedonian Albanian Turkish Bosnian Roma Other* Governmental 17 13 04 12 05 Municipal 08 03 05 03 05 NGO/ Partner Organizations 34 21 13 23 9 01 01 Media 05 02 03 03 01 01 Principal, Management 38 18 20 18 17 02 01 SIT members 70 53 17 31 30 06 03 Parents 24 9 15 10 10 01 03 Students 191 81 110 76 73 26 12 04 Total 387 200 187 176 149 37 17 07 01 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 54 ANNEX D: DATA COLLECTION SCHEDULE Date Places Visited No. of meetings Monday may 5th USAID Mission, Skopje 4 Tuesday may 6th Ministry of Education and Science, Skopje Macedonian Civic Education Center, Skopje 4 Wednesday may 7th Ministry of Education and Science, Skopje CHSC “Zdravko Cvetkovski” Demonstration School, Karpos 5 Thursday may 8th P.S. ”Draga Stojanova”, Sopiste, Skopje P.S. “Zivko Brajkosk”, Butel P.S. “Bratstvo Edinstvo”, Ohrid MoES WG Training observation, Hotel Sileks, Ohrid 10 Friday may 9th Youth via media training, Hotel Sileks, Ohrid Renovation Workshop in PS “Bratstvo Edinstvo”, Ohrid S.S. “Niko Nestor” School, Struga Master Trainers, Ohrid, Struga Municipality of Gostivar, Sector for Education SMES “Gostivar” School visit, Gostivar Association for Democratic Initiatives, Gostivar 10 Saturday may 10th PS “Sande Sterjoski”, Kicevo PS „Liria“ Bogovinje, Zerovjane 6 Monday may 12th SOSU “Mosa Pijade”, Tetovo PS “Bajram Shabani”, Kumanovo PS “Brakja Miladinovci”, Kumanovo 7 Tuesday May 13th Secondary School “Nikola Karev”, Strumica Municipality of Strumica, Sector for Education IPA CBC Office, Strumica PS “Gorgi Sugarev”, Bitola NGO MKC, Bitola 7 Wednesday May 14th SS “Kosta Susinov,” Radovis Center for Local Development, Radovis Municipality of Radovis, Sector for Education P.S. “Rajko Zinzifov”, Veles Ministry of Education and Science WG Focus Group, Skopje 9 Thursday May 15th OSCE Skopje, Education Office PS “Tefejuz School”, Cair, Skopje PS “Alija Avdovic” School event visit, Batinci, Skopje Open Fun Football Schools, Skopje 6 Friday May 16th Bureau for the development of Education, Skopje PS “Braka Miladinovci” event, Kumanovo OXO Macedonia, Skopje Center for Human Rights and Conflict Resolution NATO Liaison Office, Skopje 6 Saturday May 17th Hotel Continental, Skopje, IIEP Coordinator Group 1 Monday May 19th State Education Inspectorate 1 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 55 ANNEX E: INTERVIEW PROTOCOLS USAID/Macedonia IIEP Evaluation Coversheet for All Protocols Date of Interview: Interviewee Name, Title: Team (Sub-Team A, B): Time Start: Time End: Interviewer(s): Location Category of Interviewee: Age Group: No. of Interviewees ___________ M _____ F______ Ethnic Group: Introduction: Good morning/afternoon and thank you for taking the time to speak with us today. As mentioned during our interview request, we are working with USAID/Macedonia to conduct a midterm evaluation of the Interethnic Integration in Education Project (IIEP). The evaluation is intended to provide an informed assessment of progress to date and recommend potential modifications, if needed. Our team has had the opportunity to review some background documents to get a better sense of the design and implementation of the project. However, these documents can only tell us so much. We would like to speak with you today to hear about your experience, in your own words, in order to help us better understand how these projects look and function “on the ground.” Confidentiality Protocol  We will collect information on individuals’ names, organizations, and positions. A list of key informants will be made available as an annex to the final evaluation report, but those names and positions will not be associated to any particular findings or statements in the report.  We may include quotes from respondents in the evaluation report, but will not link individual names, organizations, or personally identifiable information to those quotes, unless express written consent is granted by the respondent. Should the team desire to use a particular quote, photograph, or identifiable information in the report, the evaluators will contact the respondent(s) for permission to do so.  All data gathered will be used for the sole purposes of this evaluation, and will not be shared with other audiences or used for any other purpose.  Your participation in this interview is voluntary and if you do not feel comfortable answering a particular question please let us know and we will simply go on to the next question. Once again, thank you for taking the time to speak with us today. Do you have any questions for us before we get started? Inform interviewee we may follow-up with brief email survey at the end of fieldwork. Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 56 USAID/MCEC KIIs/Subcontractors (Est. Time ~45 min) Introduction 1. Before we begin, can you please tell us a bit about your involvement with IIEP? (Probe: role, duration, intensity, level of involvement) 2. Were you involved in the design of the project/proposal? If so, does the project as implemented today differ in any significant way? Progress and Effectiveness 3. IIEP is structured along four specific objectives (awareness-raising, capacity-building, demonstration, and renovation). In your opinion, has IIEP been equally successful in these objectives or has the project been stronger in some areas compared to others? 4. Overall, do you feel the project is appropriate for the context and realities of ethnic relations in Macedonia today? Why/why not? 5. In your opinion, what have been IIEP’s biggest successes so far? What have been its biggest challenges? (Probe: why?) Awareness/Communication/Dissemination 6. IIEP was designed to engage with a variety of stakeholders (GoM institutions, municipal governments, education stakeholders, journalists, other donors etc.). How would you assess IIEP’s engagement with these groups? a. Has it been more successful engaging certain audiences compared to others? b. Are there any key groups/stakeholders not presently being engaged that should be? c. How is the project received by the different stakeholders? Potential Modifications 7. Given your familiarity with the project, are there any areas of the project that you feel are in need of improvement? 8. You are one of our primary intended users for this evaluation. As such, our aim is to provide you with relevant and useful information to help you better manage/oversee this project. Is there anything in particular that you feel is important for us to explore during our fieldwork? (Emphasize limited time in￾country) Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 57 GoM Representatives [MoES, BDE, SEI, VETC, SEC, MoES Coordination Committee, Master Trainers] and Journalists (Est. Time ~45 min) *focuses on Component 1 Introduction: 1. Before we begin, please tell us a bit about your familiarity with IIEP. a. How did you first hear about the project? b. What is your role/involvement in the project? c. What do you think is the aim of the project? What type of activities does it include? d. Have you participated directly in any of these activities? (Probe: workshop/training/activities). Evaluator Rating: Based on answers above, respondent appears (1-5 with 1 not at all, 5 very familiar): 1 2 3 4 5 2. Please complete the following statement based on the options provided. In my opinion, the interethnic integration of our schools is ________ to Macedonia’s future. Very important Important Somewhat Important Not at all important No opinion a. Why did you select this answer? 3. In your opinion, what is the current state of integration in schools today? Progress and Effectiveness: 4. IIEP regards [MoES, BDE, SEI, VETC, SEC] as a partner(?) in the success of IIEP. Based on your understanding: a. Is there a formal agreement in place between [MoES, BDE, SEI, VETC, SEC] and IIEP? b. What is the nature of the cooperation between [MoES, BDE, SEI, VETC, SEC] and IIEP? Does each party have specific roles and responsibilities? c. What are [MoES, BDE, SEI, VETC, SEC]’s plans for sustaining the IIEP activities in schools after the end of the project? Have formal integration education structures and procedures been established for integrated education activities in the education system? 5. [If relevant based on answers to 1d above] You indicated that you have attended a/several IIEP event(s). a. Was/were this event(s) informative? d. Did you learn anything new? (Probe for specifics) e. Have you applied any of the concepts discussed in your work since the event? (Probe: Did the training provide information of how to introduce and sustain IIE?) f. Have you trained any School Integration Teams (SITs)? i. Before IIEP Training? ii. After IIEP Training? iii. Was there any change in how you trained the SITs? g. What are the plans of your institution for continued IIEP training and support of teachers after the end of the project? h. Please complete the following sentence using a 1-5 scale (explain scale). I found the IIEP event(s) I attended: Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 58 1 2 3 4 5 Not at all informative Somewhat informative Informative Highly informative Very informative (new concepts) 6. [For journalists] Have you written/produced any articles/broadcasts on IIE since the event? a. If so, did you witness/receive any feedback/reaction to these? b. If not, why not? (Probe: lack of public interest? Editorial pushback?) Awareness/Communication/Dissemination 7. IIEP has designed to engage with a variety of stakeholders (GoM institutions, municipal governments, education stakeholders, journalists, other donors etc.). Based on what you have seen, how would you assess IIEP’s level of engagement with these groups? a. Has it been more successful engaging certain audiences compared to others? b. Are there any key groups/stakeholders not presently being engaged that should be? 8. Do you know if IIEP advertises their activities publically? c. Have you seen IIEP mentioned in the news? In any TV/radio program? d. Have you seen any posters, advertisements, etc.? Potential Modifications 9. Given your familiarity with the project, are there any areas of the project that you feel are in need of improvement? 10. If you had the chance to design your own IIE project, would you design a similar project to IIEP or would you make any significant changes? Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 59 Local School Administrators (School Boards, Municipal Officials) (Est. Time ~45 min) Introduction: 1. Before we begin, please tell us a bit about your familiarity with IIEP. a. How did you first hear about the project? b. What do you think is the aim of the project? What type of activities does it include? c. Have you participated directly in any of these activities? (Probe: workshop/training/activities). d. Have you attended any similar, non-IIEP events? Evaluator Rating: Based on answers above, respondent appears (1-5 with 1 not at all, 5 very familiar): 1 2 3 4 5 2. Please complete the following statement based on the options provided. In my opinion, the interethnic integration of our schools is ________ to Macedonia’s future. Very important Important Somewhat Important Not at all important No opinion a. Why did you select this answer? 3. What do you understand under the term integrated education? 4. In your opinion, what is the current state of integration in schools today? Progress and Effectiveness: 5. [If relevant based on answers to 1c above] You indicated that you have attended a/several IIEP event(s). a. Was/were this event(s) informative? b. Did you learn anything new? (Probe for specifics) c. Have you applied any of the concepts discussed in your work since the event? (Probe: Did the training provide information of how to introduce and sustain IIE?) d. Please complete the following sentence using a 1-5 scale (explain scale). I found the IIEP event(s) I attended: 1 2 3 4 5 Not at all informative Somewhat informative Informative Highly informative Very informative (new concepts) 6. [If relevant based on 2a above] You mentioned that you think IIE is an important goal to work towards. Have you designed, implemented, or been involved in any activities in the past 12 months intended to encourage IIE? a. If yes, what did these look like? (Probe: groups involved, intensity, duration, type of activity). b. If no, why not? (Probe: funding? Lack interest? Other priorities?) 7. Have you seen any differences in receptiveness to IIE from students/teachers/parents? a. If yes, what could be changed to better reach the less receptive groups? 8. What are some of the most important challenges to consider when designing/implementing IIE activities? In your opinion, is IIEP meeting these challenges? (Probe: funding, political influence, community interest, physical space). 9. Have you seen any evidence/examples that IIE activities are working? Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 60 10. Have you seen any unanticipated results (positive or negative) from the implementation of IIEP activities? 11. What are your institution’s plans for sustaining the IIEP activities in your community after the end of the project? Have formal integration education structures and procedures been established for integrated education activities in the community? Awareness/Communication/Dissemination 12. IIEP was designed to engage with a variety of stakeholders (GoM institutions, municipal governments, education stakeholders, journalists, other donors etc.). Based on what you have seen, how would you assess IIEP’s level of engagement with these groups? a. Has it been more successful engaging certain audiences compared to others? b. Are there any key groups/stakeholders not presently being engaged that should be? 13. Do you know if IIEP advertises their activities publically? a. Have you seen IIEP mentioned in the news? b. Have you seen any posters, advertisements, etc.? Potential Modifications 14. Given your familiarity with the project, are there any areas of the project that you feel are in need of improvement? 15. If you had the chance to design your own IIE project, would you design a similar project to IIEP or would you make any significant changes? Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 61 School Principals and Teachers KII Protocol (Est. Time ~45 min) Introduction: 1. Warm Up: Before we begin, please tell us a bit about yourself. 2. Please tell us a bit about your school. a. Do students from different ethnic background interact regularly? a. Have you noticed a change in the interaction between students of different backgrounds? (Probe: what did this look like? Was this within the past 18 months?). b. Iconography [Optional]: Can you please tell us about the decorations around your school. What types of posters/pictures/symbols/charts are hung on the wall? Why? c. Does your school have a SIT? 1. Do you know who is on it? 2. What do they do? 3. Have you seen any results? 3. Please complete the following statement based on options provided. In my opinion, the interethnic integration of our schools is ________ to Macedonia’s future. Very important Important Somewhat Important Not at all important No opinion Why: Progress and Effectiveness: 4. Do you participate in any joint activities with a teacher from different language background? If yes, please describe. (Probe: can be both teaching and professional development). 5. IIEP supports both short-term (field trips, performance, tournaments) and longer-term mixed activities. Do you think one type works better than the other? Why? 6. How do your school administrators feel about IIE? Are they supporting the integration of students i. What about your fellow teachers? ii. Students? iii. Parents of students? iv. Municipal officials? 7. Have you seen any evidence/examples that IIE activities are achieving their objectives? 8. Have you seen any unanticipated results (positive or negative) from the implementation of IIEP activities? 9. What do you think are the biggest challenges to implementing IIE activities? 10.Do you think that IIE activities will continue after the project funding ends? If yes or no, what makes you think that? What do you suggest to do to ensure that IIE continues in the long term? Awareness/Communication/Dissemination 9. Do you know if IIEP advertises their activities publically? a. Have you seen IIEP mentioned in the news? b. Have you seen any posters, advertisements, etc.? Potential Modifications Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 62 10. Given your familiarity with the project, are there any areas of the project that you feel are in need of improvement? 11. If you had the chance to design your own IIE project, would you design a similar project to IIEP or would you make any significant changes? What changes would you make? Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 63 FGD with Students (Est. Time ~80 min) Mini-Survey Questions: 1. In my opinion, the integration of our schools is ________ to Macedonia’s future. Very important Important Somewhat Important Not at all important No opinion 2. It is ________ for students to be taught in their mother tongue. Very important Important Somewhat Important Not at all important No opinion 3. It is _______ for me to learn another language other than my mother tongue.(Probe which one: Macedonian, Albanian, Turkish, Serbian) Very important Important Somewhat Important Not at all important No opinion 4. In my school, students from different language backgrounds ______ interact. Never Rarely Sometimes Often Not applicable 5. Outside of school I prefer to only interact with people from my ethnic group. Strongly agree Agree Disagree Strongly Disagree No opinion 6. The teachers at my school work to make sure that the school is a safe and open space for students of all language groups to learn. Strongly agree Agree Disagree Strongly Disagree No opinion 7. My parents do not forbid me to interact with people outside my own ethnic group. Strongly agree Agree Disagree Strongly Disagree No opinion Discussion Questions: 1. A majority of you responded that integration is/is not important for Macedonia’s future. What are some of the reasons why this may be the case? i. Does anyone disagree? 2. Please tell us a bit about your school. Do students from different language groups interact on a regular basis? What does this interaction most often look like? 3. Do you think that your school is a welcoming place for people from different language groups? (Probe: teachers/principal supportive? Iconography and school atmosphere?). 4. What about outside of school? Are you encouraged to interact with people from different groups? 5. Have you participated in any IIE activities? Which? How often? What has been your experience with such activities? How did you like them? What did you think of them? 6. What do you think of activities where students of different ethnic groups learn together? Are they better or worse and why? 7. Has the implementation of the IIE project in your school made any changes, positive or negative to the school atmosphere and the community as a whole? 8. If you were asked to design a school to encourage integration what would it look like? Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 64 FGD with Parents (Est. Time ~80 min) Mini-Survey Questions: 1. I believe that my child will learn most in school when surrounded by children only from his/her language group. Strongly agree Agree Disagree Strongly Disagree No opinion 2. In my opinion, the integration of our schools is ________ to Macedonia’s future. Very important Important Somewhat Important Not at all important No opinion 3. I believe it is important for parents of different ethnic backgrounds to be involved in school and community projects? Strongly agree Agree Disagree Strongly Disagree No opinion 4. I encourage my child to interact with people outside my own ethnic group. Strongly agree Agree Disagree Strongly Disagree No opinion 5. In my community, ethnic integration is not an important issue. Strongly agree Agree Disagree Strongly Disagree No opinion 6. As a parent, I feel that my child is less safe in a school with students from a different ethnic group. Strongly agree Agree Disagree Strongly Disagree No opinion Discussion Questions: 1. A majority of you responded that integration is/is not important for Macedonia’s future. What are some of the reasons why this may be the case? i. Does anyone disagree? 2. Please tell us a bit about your children’s school. Do children have the opportunity to interact with children from other ethnic groups during the school day? What about during extracurricular activities? 3. Please tell us about any extracurricular activities that you children participate in. Was this organized by the school or parents or both? Do parents from different groups participate? 4. Outside of school-organized activities, do children have the opportunity to interact with children from other ethnic groups? 5. Part of our task is to help USAID understand if IIEP is working or if anything needs to be improved. How would you respond? Are you satisfied with the results you’ve seen or would you make any changes? If you would make changes, what would they look like Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 65 ANNEX F: SURVEY RESULTS FROM ANONYMOUS MINI-SURVEY Responses disaggregated by language of instruction and demonstration schools vs. non￾demonstration schools. 71.4% 83.3% 52.6% 37.5% 21.4% 16.7% 34.2% 62.5% 7.1% 0.0% 13.2% 0.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=28 Demonstration Albanian n=24 Non-demonstration Macedonian n=38 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q1. How important is the integration of schools to your future Very Important Important Somewhat Important 77.8% 62.5% 51.4% 62.5% 22.2% 33.3% 35.1% 25.0% 0.0% 4.2% 13.5% 12.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=27 Demonstration Albanian n=24 Non-demonstration Macedonian n=37 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q2. How important is it for students to be taught in their mother tongue? Very Important Important Somewhat Important Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 66 44.4% 45.8% 51.4% 25.0% 44.4% 54.2% 27.0% 62.5% 11.1% 0.0% 18.9% 12.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=27 Demonstration Albanian n=24 Non-demonstration Macedonian n=37 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q3. Importance of learning another language other than mother tongue Very Important Important Somewhat Important 3.7% 0.0% 26.3% 12.5% 40.7% 47.6% 28.9% 62.5% 55.6% 52.4% 44.7% 25.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=27 Demonstration Albanian n=21 Non-demonstration Macedonian n=38 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q4. How often do students of different language backgrounds interact Rarely Sometimes Often Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 67 7.1% 26.1% 21.1% 12.5% 17.9% 17.4% 21.1% 37.5% 46.4% 30.4% 42.1% 37.5% 28.6% 26.1% 15.8% 12.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=28 Demonstration Albanian n=23 Non-demonstration Macedonian n=28 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q5. Outside of school, I prefer to only interact with students from the same my ethnic group Strongly Agree Agree Disagree Strongly Disagree 39.3% 50.0% 36.1% 75.0% 57.1% 45.8% 52.8% 25.0% 3.6% 4.2% 5.6% 0.0% 0.0% 0.0% 5.6% 0.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=28 Demonstration Albanian n=24 Non-demonstration Macedonian n=36 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q6. The teachers at my school work to make sure that the school is a safe and open space for students of all language groups to learn Strongly Agree Agree Disagree Strongly Disagree Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 68 Responses across all schools disaggregated by language of instruction. 65.4% 62.5% 44.7% 37.5% 23.1% 20.8% 26.3% 62.5% 3.8% 4.2% 13.2% 0.0% 7.7% 12.5% 15.8% 0.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Demonstration Macedonian n=26 Demonstration Albanian n=24 Non-demonstration Macedonian n=38 Non-demonstration Albanian n=8 Percent of Respondents Respondent Group Q7. My parents do not forbid me to interact with people outside my own ethnic group Strongly Agree Agree Disagree Strongly Disagree 74% 59% 26% 30% 0% 11% 0% 20% 40% 60% 80% 100% Albanian Macedonian Percent of Respondents Language of instruction Q1. The integration of schools is ______ to Macedonia's future Very Important Important Somewhat Important Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 69 62.5% 62.5% 31.3% 29.7% 6.3% 7.8% 0.0% 0.0% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Perecent of Respondents Albanian Macedonian Language of Instruction Q2. It is______for students to be taught in their mother tongue Very important Important Somewhat Important Not at all 40.6% 47.7% 56.3% 33.8% 3.1% 15.4% 3.1% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Albanian Macedonian Percent of Respondents Language of Instruction Q3. It is_______ (Macedonian, Albanian, Turkish, Serbian) for me to learn another language other than my mother tongue Very Important Somewhat Important Not at all Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 70 0.0% 0.0% 4.3% 16.9% 65.2% 30.4% 33.8% 49.2% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Albanian Macedonian Percent of Respondents Language of Instruction Q4. In my school, students from different language backgrounds ______ interact. Never Rarely Sometimes Often 8.7% 15.2% 17.4% 19.7% 43.5% 43.9% 30.4% 21.2% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Albanian Macedonian Percent of Respondents Language of Instruction Q5. Outside of school I prefer to only interact with people from my ethnic group. Strongly Agree Agree Disagree Strongly Disagree Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 71 52.2% 37.5% 43.5% 54.7% 4.3% 6.3% 0.0% 1.6% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Albanian Macedonian Percent of Respondents Language of Instruction Q6. The teachers at my school work to make sure that the school is a safe and open space for students of all language groups to learn Strongly Agree Agree Disagree Strongly Disagree 52.2% 53.1% 30.4% 25.0% 4.3% 13.0% 9.4% 12.5% 0.0% 20.0% 40.0% 60.0% 80.0% 100.0% Albanian Macedonian Percent of Respondents Language of Instruction Q7. My parents do not forbid me to interact with people outside my own ethnic group. Strongly Agree Agree Disagree Strongly Disagree Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 72 ANNEX G: DISCLOSURE OF ANY CONFLICTS OF INTEREST Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 73 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 74 Evaluation Report Midterm Evaluation of USAID/Macedonia’s Interethnic Integration in Education Project 75